App Controller

Repo: o-apps/app-controller · Metrics port: 8080 · Health probe port: 8081

App Controller is a Kubernetes operator, built on controller-runtime rather than the hexagonal HTTP-service pattern most of this platform's other backends follow — it has no REST API of its own and does no direct database work. Its entire job is to watch KubeOperaApp custom resources on the host cluster and turn each one into a real, running application inside the correct tenant vCluster, entirely through Git and Flux rather than by talking to Kubernetes directly. KubeOpera API is the only thing that ever creates or edits a KubeOperaApp CR; this controller is what actually acts on it.

Why GitOps, and not a direct apply

Every tenant's cluster already runs Flux for its own reason — it's how tenant-controller bootstraps a vCluster in the first place — so reusing it for application deployment keeps one deployment mechanism across the whole platform instead of two. It also means an app's desired state always has a durable, auditable history in Git, independent of whatever's currently running, and that a rollback (documented on the KubeOpera API page) is nothing more than pushing an older desired state back to Git and letting Flux converge to it.

What Reconcile actually does

When a KubeOperaApp CR is created or changed, the controller resolves which tenant vCluster it belongs to — either by CloudSpace ID alone (falling back to whichever vCluster in that CloudSpace is Ready first, for CRs created before per-vCluster targeting existed) or, when the CR carries a tenantCRName, by fetching that exact Tenant CR and verifying its CloudSpace ownership label matches before building a client from it. That resolved client is a real Kubernetes client for the tenant's own vCluster, entirely separate from the host-cluster client the controller uses for itself.

With both clients available, it ensures the app's namespace exists directly inside the tenant vCluster — rather than waiting for Flux to eventually apply a git-committed Namespace manifest, which removes an ordering dependency later steps would otherwise have on Flux's own reconcile timing. If the CR's spec.registry.secretRef names a registry-credentials Secret (created host-side by KubeOpera API when a tenant supplies private-registry credentials, or by the per-tenant registry provisioning flow for Kaniko-built images), the controller copies that Secret, by name and contents, into the same namespace inside the tenant vCluster, so the generated Pod spec's imagePullSecrets reference resolves there.

The application spec on the CR is then translated into a full Kubernetes manifest set — Deployment, Service, and, when TLS or a custom domain is requested, an Ingress carrying the cert-manager.io/cluster-issuer annotation — and pushed to the tenant apps Git repository. The controller ensures a Flux GitRepository and Kustomization exist inside the tenant vCluster pointing at that path, with prune: true set so that removing or deleting the manifests later cleans up cleanly. Flux itself, running inside the vCluster, is what actually applies the manifests; the controller's job ends once they're committed and Flux is pointed at them.

Deletion is finalizer-based, not fire-and-forget

Every KubeOperaApp CR carries a kubeopera.io/app-cleanup finalizer, added by the controller on its first normal reconcile. When the CR is deleted, Kubernetes doesn't remove it immediately — it sets a deletion timestamp and waits for every finalizer to be removed. The controller sees that timestamp, resolves the tenant vCluster client the same way it would for a normal reconcile, and deletes that app's Flux Kustomization and GitRepository inside the vCluster. Because those Kustomizations are already configured with prune: true, deleting the Kustomization alone cascades to deleting the Deployment, Service, and Ingress it applied — there's no need to delete each resource type individually. Only once that cleanup succeeds does the controller remove the finalizer, which is what finally lets the CR itself disappear. If the tenant vCluster is briefly unreachable, the existing failure-retry pattern (RequeueAfter: 30s) covers it; the resources being already gone is treated as success, not an error, so a retry after partial cleanup can't get stuck.

This finalizer is what makes app deletion in the dashboard mean what it says. Before it existed, deleting a KubeOperaApp CR removed the record but left Flux still reconciling the old manifests indefinitely — the app kept running, invisibly, until someone noticed and cleaned it up by hand. One real limitation remains, called out rather than silently left: the git-committed manifest files themselves aren't removed by this cleanup, only the live Kustomization/GitRepository that were reconciling them — those files are inert once nothing points at them, but they do accumulate in the tenant apps repository over time.

Configuration

This controller takes almost no environment-specific configuration of its own — nearly everything it needs (which Git repository, which branch, which credentials Secret, whether TLS is requested) travels on the KubeOperaApp CR itself, set by KubeOpera API at creation time. It starts with the standard controller-runtime manager flags (--metrics-bind-address, default :8080; --health-probe-bind-address, default :8081, serving /healthz and /readyz) and an ENVIRONMENT variable for logging verbosity. Its RBAC grants cluster-wide access to kubeoperaapps (including delete, needed for the finalizer cleanup above) and to Secrets, plus read access to Tenant CRs and their exported kubeconfig Secrets so it can build a client for each tenant's vCluster.