Skip to main content
Version: 1.0

Setup Guide

KubeOpera is a GitOps system: every environment's full state — infrastructure add-ons, the database, and every platform service — is declared as Kubernetes manifests in a Git repository, and FluxCD continuously reconciles the cluster to match. There is no imperative install script that "deploys KubeOpera"; there is a directory that describes an environment, and Flux, once pointed at it, makes the cluster match that description and keeps it that way.

A standalone Helm chart also exists (covered near the end of this guide as a secondary option) — it's useful for a narrow case (a quick evaluation install with no GitOps setup at all), but it only covers the application services, not the cluster add-ons cert-manager/ingress-nginx/etc. depend on being Flux-managed, so it isn't the recommended way to stand up a real environment.

How an environment is structured​

A KubeOpera environment is a directory under clusters/<environment>/ in the deployment repository (development, staging, and production each have one today). Flux reconciles the whole thing from a single root Kustomization:

clusters/production/
├── flux-system/          Flux's own controllers (source, kustomize, helm, notification)
├── infrastructure/
│   ├── sources/           HelmRepository/GitRepository definitions the rest of the tree pulls from
│   ├── networking/        cert-manager, ingress-nginx, external-dns — as HelmReleases
│   ├── storage/           StorageClasses
│   ├── monitoring/        kube-prometheus-stack, Grafana
│   ├── security/          Kyverno, Falco
│   └── compute/           aws-node-termination-handler
├── cert-manager-config/   ClusterIssuers (depends on cert-manager's CRDs existing first)
├── database/              PostgreSQL
└── apps/                  one Kustomization per platform service, e.g.:
    apps/auth-service.yaml:
      path: ./fluxcd/apps/auth-service/overlays/production
      sourceRef: {kind: GitRepository, name: flux-system}

This is the important structural fact: cert-manager, ingress-nginx, and every other cluster add-on are managed by Flux exactly the same way the application services are — as HelmRelease or Kustomization objects in this tree, reconciled on the same interval, with the same drift correction. They are not a separate manual install step that happens before Flux takes over; they're part of what Flux installs. The only things that exist before Flux does are the handful of node-level components Flux itself structurally depends on to run at all — see Node bootstrap below.

RabbitMQ is the one exception worth knowing about: rather than a shared platform component, it's defined inside apps/security-api/base/ and rides along with that service's own Kustomization instead of getting a dedicated entry in infrastructure/.

Installing a new environment​

1. Provision infrastructure​

Terraform (infrastructure/terraform/environments/<env>/) provisions the VPC, a bastion, control-plane/etcd/worker EC2 instances, and — depending on terraform.tfvars — a worker Auto Scaling Group, via the vpc/security/compute modules. setup-terraform-backend.sh idempotently creates the S3 state bucket first.

2. Bootstrap the nodes​

Before Flux can run, a node needs pod networking, a way to identify itself to the cloud provider, and a way to provision volumes — none of which Flux can install for itself, since Flux's own controllers need them just to schedule. configure-nodes.sh then install-kubernetes.sh (run from the bastion) handle this: OS/containerd/kubelet setup, kubeadm init on the first control-plane node, Calico CNI, the AWS Cloud Controller Manager, and the EBS CSI driver.

kubeadm init uses an explicit JoinConfiguration/InitConfiguration rather than the bare --print-join-command one-liner, specifically so kubeletExtraArgs.cloud-provider: external gets set — omitting it silently breaks AWS Cloud Controller Manager node registration.

Metrics Server is also installed at this stage, as an operational convenience rather than a hard Flux dependency. Everything else — ingress-nginx included — belongs to step 4, not here.

3. Install Flux​

Install Flux itself via its Helm chart:

helm repo add fluxcd-community https://fluxcd-community.github.io/helm-charts
helm install flux-system fluxcd-community/flux2 -n flux-system --create-namespace

Then point it at this environment's directory by creating a GitRepository and a root Kustomization referencing clusters/<env>/:

flux create source git flux-system \
  --url=https://github.com/<org>/gitops-iac \
  --branch=main \
  --namespace=flux-system

flux create kustomization flux-system \
  --source=GitRepository/flux-system \
  --path="./fluxcd/clusters/<env>" \
  --prune=true \
  --namespace=flux-system

flux bootstrap github --owner=<org> --repository=gitops-iac --path=fluxcd/clusters/<env> --token-auth does the equivalent in one command, generating and committing the flux-system/ manifests for you — it's the method development, staging, and production were originally set up with, and remains a valid alternative. Either way, the outcome is the same: Flux's controllers running, watching this repository.

4. Reconciliation takes it from here​

Once the root Kustomization exists, Flux reconciles the entire tree shown above with no further manual steps: cert-manager and ingress-nginx come up, cert-manager-config applies the ClusterIssuer once cert-manager's CRDs exist, PostgreSQL starts, and every service under apps/ deploys via its own Kustomization, pointed at its overlays/<env>/ directory. A committed change to any of these — a new image tag, a config value, a replica count — reaches the cluster on Flux's next reconcile, with no kubectl apply involved.

Secrets in this tree are managed as Sealed Secrets: encrypted before being committed to Git, decryptable only by the in-cluster sealed-secrets controller, so nothing sensitive is ever readable from the repository itself.

Creating a new environment​

This is automated: Cluster Management's Connect Existing and Create New flows both generate a genuinely new, customer-specific clusters/<slug>/ tree for you, rather than assuming one already exists. The generator (tools/generate-gitops-env) reads clusters/development/ as its reference — the only complete environment tree in this repository — and produces:

  • The infrastructure layer (cert-manager, ingress-nginx, storage classes, monitoring, security, sealed-secrets) copied over, with external-dns repointed at the new domain/region.
  • A fresh, per-service ingress hostname (<hostPrefix>.<your-domain>) for every service that has one, via an overlay patch — never by editing the shared base/ manifests every environment reads.
  • Freshly generated Postgres/RabbitMQ credentials and a container-registry pull secret, sealed with Sealed Secrets before they ever touch Git — never copied from clusters/development/'s own (real, plaintext) values.

This runs as a two-pass job (gitops-scaffolder, driven by host-cluster-controller's GeneratingGitOpsTree phase): the first pass pushes just the infrastructure layer and waits for sealed-secrets to actually come up on the target cluster before sealing anything, then a second pass generates and pushes the rest. The result is committed to either the platform's own shared gitops-iac repository (the default) or a genuinely self-hosting customer's own repository and credentials, supplied at cluster-creation time.

Creating the initial admin account​

auth-service's own migrations seed the Super Admin role only — not an account. When you create an environment through Cluster Management with an admin email/username set, host-cluster-controller does this for you automatically, once the environment's apps have actually reconciled — the generated password lands in a Secret on the host cluster, referenced from the cluster's own status. Otherwise, create the account by hand by calling the same internal bootstrap endpoint once, after authapi's pod is up:

kubectl exec -n kubeopera-core deploy/authapi -- \
  curl -s -X POST http://localhost:8082/internal/bootstrap/super-admin \
  -H "X-Api-Key: $INTERNAL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"email":"you@example.com","username":"admin","password":"a strong password"}'

$INTERNAL_API_KEY is authapi-secrets' INTERNAL_API_KEY value (kubectl get secret authapi-secrets -n kubeopera-core -o jsonpath='{.data.INTERNAL_API_KEY}' | base64 -d). The same endpoint is the correct way to reset a lost super-admin password later — it upserts by email and is safe to call again at any time.

Verifying an install​

flux get kustomizations         # did Flux fetch and apply the latest commit?
flux get sources git
kubectl get pods -n kubeopera-core     # most services share this namespace
kubectl logs deploy/<deployment-name> -n kubeopera-core

flux get kustomizations is worth checking before kubectl get pods when something doesn't look right — it's the layer above "is the pod running" that's easy to forget, and it tells you whether Flux even applied what you expect it to have applied.

Alternative: the standalone Helm chart​

This is not the recommended way to run a real environment — use Cluster Management instead, which handles cert-manager/ingress-nginx/storage and everything else below as real GitOps state, the same way this guide's primary path does. This chart only covers the application services, so anything relying on those cluster add-ons has to be provisioned separately if you use it. It's useful for a narrow case: a quick evaluation install on a cluster that already has those add-ons, with no GitOps repository or Flux involved at all.

gitops/fluxcd/tools/generate-chart generates two Helm charts directly from the same Kustomize manifests Flux applies: kubeopera-crds (the platform's CRDs — HostCluster, Tenant, KubeOperaApp, AdvisorAgent) and kubeopera (the platform services — 42 of ~44 in scope today). A CI check regenerates both on every change to the source manifests and fails the build on drift, so the charts can't silently fall out of sync with what GitOps actually deploys.

helm install kubeopera-crds ./charts/kubeopera-crds
helm install kubeopera ./charts/kubeopera -f my-values.yaml

This installs directly against whatever kubeconfig context is active — no Git repository, no Flux, no bastion. It works against a cluster built with steps 1–2 above, or any other Kubernetes cluster.

Requirements, the same ones the GitOps path relies on Flux to install, since Helm doesn't install them for you:

  • An Ingress controller with an IngressClass matching global.ingress.className
  • cert-manager, with a ClusterIssuer matching global.ingress.clusterIssuer (defaults to letsencrypt-prod-dns)
  • A StorageClass matching whatever k8s-optimizer's persistence values resolve to (the chart's current default, fast-ssd, doesn't exist on any environment in this repository yet and should be overridden)

What the chart does differently from the raw Kustomize manifests it's generated from:

  • Every credential (global.postgres, global.jwt, global.authMasterKey, global.rabbitmq, and each service's existingSecret) either points at a Secret you provide, or is generated once on first install and never touched again on helm upgrade. The source manifests ship real plaintext-base64 Secret objects for local development; the chart never carries those values forward.
  • A post-install hook calls the same bootstrap endpoint described above, using global.adminBootstrap.email/.username and a generated password — helm install's NOTES.txt output tells you how to retrieve it, or warns loudly if you left both unset.
  • Image references only get global.imageRegistry prepended when the source image actually used it — a public image like nginx:1.27-alpine is never rewritten.

Every rendered manifest is validated with helm lint and kubectl apply --dry-run, including against a live cluster's real API server. A full install on a genuinely empty cluster, start to finish, hasn't been run yet.

Managing additional clusters​

Everything above gets one cluster running KubeOpera — it doesn't register that cluster with KubeOpera itself. Once you can log in, open Cluster Management (/clusters):

  • Register This Cluster — one click, inspects the cluster KubeOpera is already running on using its own in-cluster ServiceAccount.
  • Connect Existing — link a different, already-running cluster by kubeconfig; KubeOpera bootstraps Flux on it, generates a real, customer-specific GitOps tree for it (see Creating a new environment above), and points that Flux instance at it.
  • Create New — provision a new AWS cluster from inside the UI: choose vanilla kubeadm or managed EKS, region, size, a stored credential — then the same tree-generation and Flux hand-off as Connect Existing. See Cluster Management for what this automates and what it doesn't yet.

Next Steps​