Host Cluster Controller

Host Cluster Controller is a controller-runtime operator that watches HostCluster CRs and does the actual work behind Cluster Management: validating a connected cluster's kubeconfig, driving a fresh AWS provisioning run via cluster-provisioner, and — for every onboarding path — bootstrapping Flux on the target and pointing it at KubeOpera's own GitOps repository. It's the closest architectural sibling to Tenant Controller: same finalizer-gated reconcile-loop shape, same "CRD status is a state machine" pattern, applied to whole clusters instead of per-customer vClusters.

CRD: HostCluster

Group/Version: kubeopera.io/v1alpha1 · Kind: HostCluster

apiVersion: kubeopera.io/v1alpha1
kind: HostCluster
metadata:
  name: 7e9d3a1b-...              # the clusters.id row this CR belongs to
  namespace: kubeopera-system
spec:
  clusterID: "7e9d3a1b-..."
  mode: connect_existing          # | provision_vanilla | provision_eks
  provider: aws                   # free string, not enum-backed -- see below
  region: eu-north-1
  connectKubeconfigSecretRef: hostcluster-7e9d3a1b-input-kubeconfig
  provisioning:                   # only for mode=provision_vanilla/provision_eks
    topology: vanilla_kubeadm     # | eks
    controlPlaneCount: 1           # vanilla_kubeadm only; EKS manages its own control plane
    nodeCount: 2
    instanceType: t3.medium
    kubernetesVersion: "1.31.0"
    credentialRef: <cloud_credentials.id in auth-service>
    terraformEnvironment: cluster-7e9d3a1b-...
  gitOpsRepoURL: https://github.com/ochestra-tech/gitops-iac.git
  gitOpsPath: fluxcd/clusters/my-cluster
  gitCredentialsSecretRef: gitops-iac-credentials
  domain: my-cluster.acme.com       # required -- feeds every generated service's ingress hostname
  registryPullSecretRef: hostcluster-7e9d3a1b-registry-pull   # optional
  adminEmail: admin@acme.com        # optional -- both unset skips automatic admin-account creation
  adminUsername: admin
status:
  phase: Ready
  kubeconfigSecret: hostcluster-7e9d3a1b-kubeconfig
  fluxHealthy: true
  kubeOperaInstalled: false        # not yet actually set by any reconcile path
  adminPasswordSecretRef: hostcluster-7e9d3a1b-admin-password
  message: "flux healthy, pointed at fluxcd/clusters/my-cluster. super-admin account admin@acme.com created -- retrieve the password from secret kubeopera-system/hostcluster-7e9d3a1b-admin-password"
  lastReconciledAt: "2026-09-10T00:35:00Z"

spec.provider is deliberately a free string rather than a Go-const-backed enum in the CRD's generated schema — only aws has a real implementation, but adding gcp/azure later shouldn't require a CRD migration (the same enum-completeness lesson the Tenant CRD's own status.phase field learned the hard way, applied proactively here).

Reconcile State Machine

HostCluster CR created
          │
    ┌─────▼──────┐
    │  Pending   │
    └─────┬──────┘
          │  mode == connect_existing?
     ┌────┴────┐
     │         │
┌────▼───┐ ┌───▼────────────┐
│Validat-│ │  Provisioning  │  ← mode=provision_vanilla/provision_eks only
│ingAcc- │ │  (Tier 2, see  │
│ess     │ │  below)        │
└────┬───┘ └───┬────────────┘
     │ Discovery().ServerVersion()      │ poll cluster-provisioner,
     │ retries w/ backoff, 5min give-up │ fetch kubeconfig once succeeded
     └────────────┬─────────────────────┘
                   │  both converge on the same kubeconfig-in-hand state
          ┌────────▼─────────┐
          │ BootstrappingFlux│  ← install Flux via its own Helm chart,
          └────────┬─────────┘     poll IsHealthy(), seed git creds Secret
                   │
          ┌────────▼──────────────┐
          │ GeneratingGitOpsTree  │  ← delegate to gitops-scaffolder: generate
          └────────┬──────────────┘     + push a real tree at spec.gitOpsPath
                   │
          ┌────────▼──────────┐
          │InstallingKubeOpera│  ← EnsureGitRepository + EnsureKustomization
          └────────┬──────────┘     on the target, pointed at the now-real tree;
                   │                  once "apps" is Ready, one-time admin bootstrap
             ┌─────▼──────┐
             │   Ready    │  (idempotent -- further reconciles no-op)
             └────────────┘
          (on error at any step)
             ┌────────────┐
             │   Failed   │  ← patch spec/status to re-trigger; no automatic retry
             └────────────┘

Ready no longer means only "Flux is healthy and pointed somewhere" — since GeneratingGitOpsTree guarantees spec.gitOpsPath holds a real, complete tree before InstallingKubeOpera ever points Flux at it, Ready here means Flux is healthy, pointed at that real tree, and (once InstallingKubeOpera observes the tree's own apps Kustomization reporting Ready=True) the platform's application layer has itself converged. It still isn't a guarantee that every individual service inside that layer is healthy — that's a finer-grained concern than this CR tracks — but it's a meaningfully stronger claim than before GeneratingGitOpsTree existed, when spec.gitOpsPath was simply assumed to already be populated. This mirrors Tenant.status.phase = Ready's general shape (CR status reflects the reconciler's own scope, not every downstream service's health) without repeating its earlier, weaker scoping.

Provisioning (Tier 2): delegating to cluster-provisioner

reconcileProvisioning doesn't run Terraform itself — a controller-runtime reconcile loop expects each pass to be cheap, and a real apply can run 10-40+ minutes. Instead it's a thin HTTP client to cluster-provisioner:

1. First entry (no job tracked yet): POST /api/v1/provisions with the CR's own spec.provisioning fields. The returned job ID is stored as an annotation (kubeopera.io/provision-job-id) on the CR itself, not held in controller memory — this specifically survives a controller pod restart mid-provision, the same way ValidatingAccess's own give-up timer is tracked via annotation.
2. Every subsequent reconcile: GET /api/v1/provisions/{id} to poll status. queued/planning/applying just update the CR's own message and requeue; failed moves the CR to Failed with the real Terraform error text attached; succeeded triggers step 3.
3. GET /api/v1/provisions/{id}/kubeconfig — fetched fresh, exactly once, the moment the job succeeds. Never cached anywhere upstream of this call; cluster-provisioner doesn't persist kubeconfig content either (see its own page).
4. The fetched kubeconfig is stored via the same SecretClient.StoreKubeconfig a Connect-Existing kubeconfig uses, and the CR moves straight to BootstrappingFlux — from this point on, a provisioned cluster and a connected one are handled by identical code.

GitOps Tree Generation: delegating to gitops-scaffolder

reconcileGeneratingGitOpsTree follows reconcileProvisioning's exact shape — a thin HTTP client to a separate service (gitops-scaffolder), tracked via a job-ID annotation (kubeopera.io/scaffold-job-id) that survives a controller pod restart, rather than running the (multi-minute) generation work inline:

1. First entry: POST /api/v1/scaffolds with the CR's own slug (derived from spec.gitOpsPath), domain, region, the stored kubeconfig Secret's name, and — when set — spec.gitCredentialsSecretRef/spec.registryPullSecretRef.
2. Every subsequent reconcile: GET /api/v1/scaffolds/{id} to poll status; failed moves the CR to Failed with the real underlying error; succeeded triggers InstallingKubeOpera.

gitops-scaffolder itself runs the actual work as a Kubernetes Job (scaffold-runner, in kubeopera-provisioning): clone the target git repository, run tools/generate-gitops-env once with -infra-only (cert-manager, ingress-nginx, storage, monitoring, security, sealed-secrets — nothing sealed yet, since sealed-secrets itself doesn't exist on the target until this step reconciles), commit and push, poll the target's own sealed-secrets controller until it's Available, fetch its public certificate through the Kubernetes API's service-proxy subresource (no port-forward), then run the generator a second time — now sealing fresh Postgres/RabbitMQ credentials and a registry pull secret against that real cert — and push again. See Setup Guide for what the generated tree actually contains.

Admin Bootstrap

Once InstallingKubeOpera observes the generated tree's own apps Kustomization reporting Ready=True, and only when both spec.adminEmail/spec.adminUsername are set, a one-time call creates the platform's initial super-admin account: a random password is generated, authapi's internal /internal/bootstrap/super-admin endpoint is called on the target (again via the Kubernetes API's service-proxy subresource, using authapi-secrets' own INTERNAL_API_KEY read directly from the target), and the generated password is stored in a Secret named hostcluster-<clusterID>-admin-password on the host cluster (status.adminPasswordSecretRef records the name). Guarded to run exactly once per CR — the bootstrap endpoint is a real account upsert, so calling it a second time with a fresh random password would silently change the account's real password out from under whoever already has the first one. Leaving both fields unset is a deliberate, supported choice; status.message says so plainly rather than silently producing an install nobody can log into.

Kubeconfig Storage

Every path (connect or provision) ends with a working kubeconfig stored the same way:

kubectl get secret hostcluster-<cluster-id>-kubeconfig \
  -n kubeopera-system \
  -o jsonpath='{.data.kubeconfig}' | base64 -d

Labelled kubeopera.io/host-cluster-id={clusterID} and kubeopera.io/managed-by=host-cluster-controller. Unlike Tenant's own kubeconfig handling (which rewrites a vCluster's server: field to an in-cluster short-DNS form, since a vCluster is always reachable from inside the host cluster's own network), a connected or provisioned external cluster has no such guarantee, so its server: field is stored close to as-supplied/as-produced — whatever address the cluster actually exposes.

An EKS-provisioned cluster's stored kubeconfig is the one real structural exception: its users[].user.exec block runs a small companion binary, /eks-token-helper (shipped in this controller's own container image — see below), rather than embedding a static credential, since EKS authentication is always a short-lived, AWS-signed token rather than a client certificate.

RBAC

Deliberately namespace-scoped (Role, not ClusterRole) — a real, considered difference from tenant-controller's own broader RBAC. This controller only ever touches its own HostCluster CRs and Secrets, both confined to kubeopera-system; everything it does to a target cluster goes through a client built from that target's own kubeconfig, using whatever access that kubeconfig's identity already has there, never this ServiceAccount's own permissions.

rules:
  - apiGroups: [kubeopera.io]
    resources: [hostclusters]
    verbs: [get, list, watch, update, patch]
  - apiGroups: [kubeopera.io]
    resources: [hostclusters/status]
    verbs: [get, update, patch]
  - apiGroups: [""]
    resources: [secrets]
    verbs: [get, list, watch, create, update, patch, delete]

Making this narrower Role actually sufficient required one non-obvious fix: controller-runtime's manager caches/watches a Namespaced CRD at cluster scope by default regardless of where its instances live, which 403's against a namespace-scoped Role. The manager is explicitly configured with Cache.DefaultNamespaces: {"kubeopera-system": {}} to scope its watch correctly — confirmed live: without this the whole manager crash-loops on cache-sync timeout.

Environment Variables

Variable	Default	Description
ENVIRONMENT	development	Enables dev-mode structured logging
CLUSTER_PROVISIONER_BASE_URL	(unset)	cluster-provisioner's in-cluster Service URL. Unset means Tier 2 (provision_vanilla/provision_eks) fails fast and clearly rather than hanging — Tier 1 (connect_existing) doesn't need this at all.
CLUSTER_PROVISIONER_API_KEY	(unset)	Shared secret with cluster-provisioner's own API_KEY — the same trusted-internal-caller pattern used between build-service and cicd-gateway.
GITOPS_SCAFFOLDER_BASE_URL	(unset)	gitops-scaffolder's in-cluster Service URL. Unset means GeneratingGitOpsTree fails fast and clearly rather than hanging — every mode now passes through this phase, unlike CLUSTER_PROVISIONER_BASE_URL, which only Tier 2 needs.
GITOPS_SCAFFOLDER_API_KEY	(unset)	Shared secret with gitops-scaffolder's own API_KEY.

Deployment

Runs as a single-replica Deployment in kubeopera-system, gcr.io/distroless/static:nonroot base image — no shell, no package manager, deliberately minimal. Its container image is a multi-stage Docker build producing two binaries: the controller itself, and eks-token-helper — a small, separate Go module (its own go.mod, isolated dependency tree) that mints EKS bearer tokens via sigs.k8s.io/aws-iam-authenticator/pkg/token, the same library aws eks get-token itself is built on. It lives as a genuinely separate module rather than a package inside this controller because its one real dependency pulls k8s.io/client-go forward by several minor versions — bundling it directly into this module's own go.mod would have forced that same jump onto the controller everywhere else it's used.

kubectl logs -n kubeopera-system -l app=host-cluster-controller -f

Health and readiness probes on :8081/healthz / :8081/readyz; Prometheus metrics on :8080/metrics via controller-runtime's built-in metrics server.