Skip to main content
Version: 2.0

Setup Guide

This guide explains how a KubeOpera environment is put together and how to build one from bare infrastructure. If you just want KubeOpera running, start with the Installation tutorial — come back here when you want to understand or customize what it did.

KubeOpera is a GitOps system​

A KubeOpera environment is a directory in Git. It declares everything the environment runs — cluster add-ons, databases, the message bus and every platform service — as Kubernetes manifests, and Flux continuously makes the cluster match it.

That has practical consequences:

  • Every change is a commit. Upgrading a service, changing a setting or adding a replica is a pull request, reviewed and auditable.
  • Drift is corrected automatically. A manual kubectl edit is reverted on Flux's next reconcile — change Git instead.
  • Environments are reproducible. Staging and production differ only where their directories say they do.

How an environment is structured​

Each environment lives under fluxcd/clusters/<environment>/ in your GitOps repository. Flux reconciles the whole tree from one root Kustomization:

clusters/production/
├── flux-system/           Flux's own controllers
├── infrastructure/
│   ├── sources/           HelmRepository and GitRepository definitions
│   ├── networking/        cert-manager, ingress-nginx, external-dns
│   ├── storage/           StorageClasses
│   ├── monitoring/        kube-prometheus-stack, Grafana, Loki, Jaeger
│   ├── security/          Kyverno, Falco, Sealed Secrets
│   ├── messaging/         RabbitMQ
│   └── compute/           Karpenter, aws-node-termination-handler
├── cert-manager-config/   ClusterIssuers (applied after cert-manager's CRDs exist)
├── database/              PostgreSQL
└── apps/                  one Kustomization per KubeOpera service, for example:
    apps/auth-service.yaml:
      path: ./fluxcd/apps/auth-service/overlays/production
      sourceRef: {kind: GitRepository, name: flux-system}

Cluster add-ons are managed exactly like KubeOpera's own services — as HelmRelease and Kustomization objects in this tree, reconciled on the same interval with the same drift correction. Only the few components Flux itself needs in order to run (networking, cloud integration, storage — see Step 2) are installed before Flux.

Services: base and overlays​

Each service's manifests are split into:

  • base/ — what's identical in every environment;
  • overlays/<environment>/ — what legitimately differs: replicas, resources, hostnames and settings, applied as Kustomize patches.

Put environment differences in overlays, never in base/. See Advanced configuration.

Building an environment from scratch​

If you already have a Kubernetes cluster, skip to Step 3. Otherwise, KubeOpera's Terraform and bootstrap scripts build one on AWS for you.

1. Provision infrastructure​

Terraform in infrastructure/terraform/environments/<env>/ creates the network, a bastion host, control-plane and worker instances, and (optionally) a worker Auto Scaling Group, using the vpc, security and compute modules.

cd infrastructure/terraform/environments/<env>
./setup-terraform-backend.sh     # creates the S3 state bucket (safe to re-run)
terraform init
terraform apply -var-file=terraform.tfvars
tip

Prefer a managed control plane? Use the eks module instead of compute, or let KubeOpera provision EKS or GKE clusters for you from Cluster Management.

2. Bootstrap the nodes​

Before Flux can run, nodes need a container runtime, pod networking, a cloud integration and storage. From the bastion, run:

./configure-nodes.sh      # OS, containerd and kubelet
./install-kubernetes.sh   # kubeadm, Calico, AWS Cloud Controller Manager, EBS CSI driver, Metrics Server

install-kubernetes.sh initializes the control plane with an explicit kubeadm configuration that sets cloud-provider: external, so nodes register correctly with the AWS Cloud Controller Manager.

Everything else — including ingress — is installed by Flux in the next steps.

3. Install Flux​

Generate an environment directory (if you haven't already — see Installation), then bootstrap Flux against it:

flux bootstrap github \
  --owner=<org> \
  --repository=gitops-iac \
  --path=fluxcd/clusters/<env> \
  --token-auth

Prefer to install Flux with Helm and wire it up yourself? That works too:

helm repo add fluxcd-community https://fluxcd-community.github.io/helm-charts
helm install flux-system fluxcd-community/flux2 -n flux-system --create-namespace

flux create source git flux-system \
  --url=https://github.com/<org>/gitops-iac --branch=main --namespace=flux-system
flux create kustomization flux-system \
  --source=GitRepository/flux-system --path="./fluxcd/clusters/<env>" \
  --prune=true --namespace=flux-system

4. Let Flux reconcile​

Once the root Kustomization exists, Flux does the rest, in dependency order:

  1. cert-manager, ingress-nginx and the other add-ons;
  2. cert-manager-config applies the ClusterIssuers;
  3. PostgreSQL and RabbitMQ start;
  4. every service under apps/ deploys from its overlays/<env>/ directory.

From then on, a committed change — a new image tag, a setting, a replica count — reaches the cluster on the next reconcile.

5. Create the first administrator​

kubectl exec -n kubeopera-core deploy/authapi -- \
  kubeopera-admin create-super-admin --email you@example.com --username admin

The command prints a generated password; you'll change it at first sign-in. Run it again at any time to reset a lost administrator password.

Environments created from Cluster Management with an administrator email do this for you automatically.

Secrets​

Nothing sensitive is committed in plain text. Database passwords, signing keys and API keys are Sealed Secrets: encrypted for a specific cluster before they're committed, and decryptable only by that cluster's controller.

To add or change a secret:

kubectl create secret generic my-secret -n kubeopera-core \
  --from-literal=API_KEY=… --dry-run=client -o yaml \
  | kubeseal --format yaml > apps/my-service/overlays/<env>/my-secret.sealed.yaml
git add … && git commit -m "Rotate my-service API key" && git push

Credentials that users manage at runtime — AI provider keys, registry credentials, cloud credentials — are stored encrypted by auth-service instead. See Configuration: where secrets live.

Creating more environments​

You don't need to hand-write environment directories. Cluster Management's Connect Existing and Create New flows generate a complete, cluster-specific environment for you, using the development environment as the reference:

  • the infrastructure layer, with external-dns pointed at the new cluster's domain and region;
  • a hostname for every service under the new domain, added as overlay patches;
  • fresh PostgreSQL and RabbitMQ credentials and a registry pull secret, sealed before they touch Git.

The generator (gitops-scaffolder, driven by host-cluster-controller) runs in two passes: it first pushes the infrastructure layer and waits for Sealed Secrets to come up on the new cluster, then seals and pushes everything else. The result is committed to the platform's shared GitOps repository or to your own repository, as you choose when creating the cluster.

Verifying an environment​

flux get kustomizations        # has Flux applied the latest commit?
flux get sources git           # can Flux reach the repository?
kubectl get pods -n kubeopera-core
kubectl logs deploy/<service> -n kubeopera-core

Check flux get kustomizations first when something looks wrong: it tells you whether Flux has even applied what you expect, which is the layer above "is the pod running?".

The Helm alternative​

The KubeOpera Helm charts are generated from the same manifests Flux applies, and a CI check keeps them in sync:

  • kubeopera-crds — the platform's CRDs (HostCluster, Tenant, KubeOperaApp, AdvisorAgent);
  • kubeopera — every platform service.
helm install kubeopera-crds kubeopera/kubeopera-crds
helm install kubeopera kubeopera/kubeopera -n kubeopera-core --create-namespace -f my-values.yaml

The charts install KubeOpera's services, not cluster add-ons, so the cluster needs:

  • an ingress controller with an IngressClass matching global.ingress.className;
  • cert-manager with a ClusterIssuer matching global.ingress.clusterIssuer;
  • a StorageClass for persistent volumes (global.storageClass, defaulting to the cluster's default class).

How the chart handles credentials:

  • Each credential (global.postgres, global.jwt, global.authMasterKey, global.rabbitmq, and each service's existingSecret) either references a Secret you provide or is generated once on first install and kept on upgrade.
  • If you set global.adminBootstrap.email and .username, a post-install hook creates the first administrator; helm install prints how to read the password.
  • Images are pulled from global.imageRegistry when you set it, so you can mirror KubeOpera's images into your own registry.

Next steps​