GitOps (ArgoCD)

Cluster state is declarative in Git and reconciled by ArgoCD. Deploys happen by committing manifests (or image tags) — not by kubectl apply. Git is the single source of truth for the long-lived workloads; the cluster continuously self-heals toward it.

Most of this page is the design; the installation lives elsewhere

Argo CD is deployed and running, and it manages itself (gitops-apps/control/argocd.yaml, chart argo-cd 10.3.2). It is just not here: the manifests live in a second repository, Agentfy/gitops, which until recently was named nowhere in this one. Start at Two repositories — it draws the boundary and lists what runs today, file by file. Every section below now says which of its decisions the installation took and which it did not.

The boundary — what ArgoCD does and does not manage

This is the key decision for an ephemeral-runtime platform:

ArgoCD manages (GitOps)	ArgoCD does NOT manage
Persistent Deployments: api, app, admin, LightRAG	the ephemeral worker Jobs (browser runs inside them)
Services, Ingress, cert-manager, NetworkPolicies, ResourceQuotas, namespaces	individual AgentRuntimeSession Jobs
Redis / Postgres (via operator or manifests)	anything created per task at runtime
The worker's ServiceAccount + RBAC, the runner image ref, RuntimeProfile presets (ConfigMap)	—

The worker Jobs are created imperatively by the runtime/worker slice via the k8s API (hundreds of times, scale-to-zero). GitOps must not own them — Argo would constantly see them as "drift" and prune them. Argo owns the worker's deployment artifacts (image, RBAC, profiles, NetworkPolicy template); the runtime manager owns the Job instances.

Likewise ArgoCD ≠ Argo Workflows — we don't use Workflows for the worker (Native Jobs instead).

Repo & layout

The split happened. The manifests are not in agentfy2/k8s/; they are in Agentfy/gitops, and that is where every claim on this page is checked. What is actually there:

Agentfy/gitops/
├── bootstrap/            applied by hand ONCE: clusterissuer, argo ingress, root-app
├── gitops-apps/
│   ├── control/          Applications on the control cluster (argocd, rancher, vault, cert-manager)
│   └── dev/ · prod/      Applications per contour (cert-manager, ESO, external-dns, reloader)
│       └── manifests/    RAW manifests, delivered INTO the workload cluster
├── agentfy-apps/dev/     one Application per product service (api, app, admin, redis)
└── helm/                 the charts themselves: agentfy-{api,app,admin,redis}

One rule from that repository is worth carrying here, because breaking it fails silently: gitops-apps/<env>/ may hold only Argo Application objects — those must exist on the control cluster. A raw ClusterIssuer or ExternalSecret goes under gitops-apps/<env>/manifests/, which a separate Application (manifests-app.yaml, directory.recurse: true) delivers into the workload cluster.

Superseded — the layout this project planned, and why it is kept

The tree below was the plan: charts in k8s/charts/, values in k8s/values/<env>/, Applications in k8s/argocd/, all in this repository, with "an option to split into a dedicated agentfy-deploy repo later". The option was taken. It is kept rather than deleted because the shape it argues for — one chart per component, per-env values, app-of-apps — is the shape Agentfy/gitops uses; only the addresses changed. The one component in it that still does not exist anywhere is LightRAG.

Helm — one chart per component + per-env values:

k8s/
├── charts/
│   ├── api/ app/ admin/ lightrag/   # chart per app (templates/ + values.yaml)
│   ├── worker/        # SA + RBAC + NetworkPolicy template + runtimeProfiles (chart)
│   ├── platform/      # Redis, Postgres (CNPG), namespaces, ResourceQuota/LimitRange
│   └── networking/    # Ingress, cert-manager Issuer, default-deny NetworkPolicies
├── values/
│   ├── dev/  <app>.yaml     # replicas, resources, hostnames, image tags
│   └── prod/ <app>.yaml
└── argocd/
    ├── root.yaml      # the app-of-apps root Application
    └── apps/          # one ArgoCD Application per chart (source.helm.valueFiles → values/<env>)

(Shared snippets via a small base/library chart or chart dependencies; secrets are not in values — see below.)

App-of-apps

This is the one part the installation took whole. The root Application is bootstrap/root-app.yaml; it watches gitops-apps/control on main, and every other Application is reached from there — including two that are themselves app-of-apps (agentfy-dev-apps, agentfy-prod-apps, each watching its contour's folder) and one that carries the product (agentfy-dev-services → agentfy-apps/dev/). Adding a service is still one child manifest. ApplicationSet is not used; the folders are duplicated per contour by hand.

Delivery flow

push code → build & push image (ghcr.io/agentfy/<app>) → bump image.tag in
            Agentfy/gitops helm/<chart>/values.dev.yaml → commit → ArgoCD syncs

The half that is real is the right-hand side: the tag genuinely is one line in helm/agentfy-{api,app,admin}/values.dev.yaml, Git is the source of truth, and a deploy is revertable by git revert.

The left-hand side is not verified. Neither repository contains a workflow that builds or pushes an application image — Agentfy/gitops's three GitHub Actions workflows are ansible-lint and two manual Ansible runs, and this repository has no .github/ at all — yet the values files pin real tags, so the images exist. Who builds them is written down nowhere that could be checked. Argo CD Image Updater is not installed; the tag bumps are commits.

Secrets in Git (never plaintext)

The manifests reference infra credentials (DB URL, Redis URL, the secrets KEK, Claude API key, Resend key, registry creds) — never committed in clear. Use one of:

• Sealed Secrets (Bitnami) — not adopted. It was the MVP recommendation; the reason it lost is that it keeps the ciphertext in Git, so rotating a value is a commit and a controller key is a single point of loss.
• SOPS + age — not adopted, same objection plus an Argo plugin to maintain.
• External Secrets Operator — this is the choice, and it is running. Chart external-secrets 2.9.0 in both contours (gitops-apps/{dev,prod}/external-secrets.yaml), reading a HashiCorp Vault that runs on the control cluster (gitops-apps/control/vault.yaml, chart vault 0.34.0, standalone, 5Gi, vault.agentfy.ai). The store is a ClusterSecretStore named vault — KV v2 at path secret, kubernetes auth on mount kubernetes-dev / kubernetes-prod, role eso (gitops-apps/{dev,prod}/manifests/external-secret*/clustersecretstore.yaml).

Nothing is encrypted into Git at all: a value lives in Vault, an ExternalSecret names its path, and an ordinary Kubernetes Secret appears. Working examples — externalsecret-external-dns.yaml (AWS keys for ExternalDNS, refreshed hourly), externalsecret-ghcr.yaml (the registry pull secret) and the api's own, which helm/agentfy-api/ templates from secrets.vaultKey.

Note: agent secrets are not here — they live encrypted in Postgres (see Agent secrets). GitOps only handles platform/infra creds.

Sync policy

• dev: automated sync + selfHeal + prune — fully hands-off. As designed.
• prod: planned as PR-gated with a manual window for risky changes. Today it is not: every Application in Agentfy/gitops, prod included, carries automated: {prune: true, selfHeal: true}. The gate is the PR into that repository's main, and nothing else.
• Sync waves are used, and they are the ordering that exists: 0 for redis and the platform charts, 1 for the api and the raw manifests, 2 for app and admin.
• Migrations: planned as a PreSync hook Job. Today prisma migrate deploy runs as an initContainer on the api Deployment (helm/agentfy-api/templates/deployment.yaml, gated by runMigrations). The difference is not cosmetic — an initContainer runs once per pod and races itself if replicas > 1, which is part of why the api chart pins strategy: Recreate and one replica.

Bootstrap (the one imperative step)

1. Install ArgoCD into the argocd namespace (Helm/manifest) — once.
2. Apply the root app-of-apps Application → Argo pulls everything else from Git.
3. From then on, the cluster (optionally including ArgoCD itself) is managed by Git.

That is what happened. In Agentfy/gitops it is a single kubectl apply -f bootstrap/ against the control cluster, and the folder holds exactly three files: the Let's Encrypt ClusterIssuer, the Ingress for the Argo CD UI (argo.agentfy.ai), and root-app.yaml. Argo CD does manage itself from there — gitops-apps/control/argocd.yaml — with ignoreDifferences on argocd-secret so the generated admin password is not fought over.

Reuse from Ranch

cleanslice/ranch/k8s is a working reference — ~80% a template for us:

• argocd/app-of-apps.yaml — exactly this app-of-apps pattern (one Application per component, automated: prune+selfHeal). Copy structure; change repoURL/paths/namespaces.
• platform/lightrag/* — a working LightRAG deploy on Postgres with pgvector + AGE (LIGHTRAG_{KV,VECTOR,GRAPH,DOC_STATUS}_STORAGE=PG*) — our exact chosen backend.
• browser-pool-image/ — Ranch's Browserless + JWT-noVNC live-takeover image; we instead bundle Playwright + Chromium directly into the worker image (browser runs in-pod, no separate pool), but the noVNC live-takeover trick is worth borrowing into the worker.
• infrastructure/cnpg.yaml + database/pg-cluster.yaml — CloudNativePG for the main app DB.
• platform/{api,app,admin}/*, deploy/ (cert-issuer, ghcr-pullsecret), local/ bootstrap.

Adapt — the one real difference: Ranch runs agents via Argo Workflows (templates/rbac.yaml SA + workflows RBAC + agent-workflow.manifest.ts). We use Native k8s Jobs → drop the Workflows install/RBAC; keep the RBAC shape (platform/api/rbac.yaml: API ServiceAccount in platform + Role in the workers namespace) but with batch/jobs create/delete verbs for our runtime/worker manager.

None of this section has been executed. It is a survey of what to borrow, written before Agentfy/gitops existed. What the installation actually took from it is the app-of-apps shape and the ghcr pull secret; LightRAG, CNPG and the workers nodepool below are not deployed anywhere — the live Postgres runs on its own Hetzner VMs (ansible/roles/postgres), not on CNPG.

Gotchas captured: CNPG's stock image lacks Apache AGE (won't override shared_preload_libraries) → Ranch runs LightRAG's Postgres on gzdaniel/postgres-for-rag (pgvector

• AGE) — so in practice two Postgres (app DB on CNPG; LightRAG/AGE DB separate) until a custom CNPG-with-AGE image. Hetzner: storageClassName: hcloud-volumes, PVCs pin to worker nodes; node node-role: workers + toleration workload=worker:NoSchedule.

What stays imperative / runtime

• worker Jobs and AgentRuntimeSessions — created by the runtime manager per task.
• Agent data (agents, memory, chat, secrets) — in Postgres, not Git.
• Autoscaling of the workers nodepool — cluster-autoscaler reacts to pending Jobs.

GitOps gives us reproducible, auditable, revertable platform state; the ephemeral agent runtime stays dynamic on top of it.