Cluster & nodes

How the Hetzner k3s cluster is laid out, why the control plane runs on its own node(s), and how autoscaling grows and shrinks both pods and nodes — automatically, in both directions.

What is running today has a different shape

This page is a design, and the clusters that actually serve traffic were not built from it. In Agentfy/gitops: terraform/mgmt.tf and terraform/db.tf raise one management node and two PostgreSQL nodes on a flat Hetzner network, and ansible/k3s.yml puts a single-node k3s on the management node — that is the control cluster, where Argo CD, Rancher and Vault live. The dev and prod workload clusters are provisioned through Rancher (terraform/loadbalancer.tf records that the lb-target label is applied by Rancher), which is why every workload Application targets https://10.0.1.3:6443 or https://10.0.1.9:6443.

So today: no three tainted pools, no cluster-autoscaler, no CloudNativePG, and no workers nodepool anywhere. Postgres runs on its own VMs (ansible/roles/postgres); Redis is a plain Deployment (helm/agentfy-redis). Which repository holds what: Two repositories.

Three node pools (not two)

The cluster has three pools, each with a distinct job. The control plane is its own pool — it does not share nodes with application workloads.

Pool	Runs	Taint / selector	Schedulable by apps?	Scaling
control-plane	k3s server: API server · etcd · scheduler · controller-manager	node-role.kubernetes.io/control-plane:NoSchedule	No — tainted, isolated	Fixed: 1 (dev) / 3 (prod HA)
core (always-on)	api, app, admin, LightRAG, Postgres (CNPG), Redis, Ingress, ArgoCD, monitoring	default (schedulable)	Yes	HPA per app + cluster-autoscaler on the pool
workers (ephemeral)	worker Jobs (bash/fs/browser in-pod/heavy)	node-role: workers + workload=worker:NoSchedule	Only Jobs that tolerate it	Scale-to-zero; cluster-autoscaler 0 → N

                    ┌──────────────────────────────────────────┐
                    │  control-plane pool  (tainted, isolated)   │
                    │  api-server · etcd · scheduler · ctrl-mgr  │   1 (dev) / 3 (prod, HA quorum)
                    └──────────────────────────────────────────┘
                                       │ manages
        ┌──────────────────────────────┴──────────────────────────────┐
        ▼                                                              ▼
┌────────────────────────────────────┐            ┌────────────────────────────────────┐
│  core pool  (always-on)             │            │  workers pool  (ephemeral, tainted) │
│  api / app / admin (HPA Deployments)│            │  worker Jobs — one task, then exit   │
│  LightRAG (browser runs in worker)  │            │  scale-to-zero between tasks         │
│  Postgres (CNPG) · Redis · Ingress  │            │  cluster-autoscaler 0 → N            │
│  ArgoCD · Loki/Grafana              │            │  node-role: workers · NoSchedule tol │
│  cluster-autoscaler grows the pool  │            └────────────────────────────────────┘
└────────────────────────────────────┘

Why the control plane needs its own node(s)

A k3s server is the brain of the cluster (API server + etcd). It must stay responsive even when workloads spike — and agent workloads spike hard (a runaway task, a heavy LightRAG ingest, a browser pool under load). Isolating it:

• etcd is disk- and latency-sensitive. Co-scheduling a noisy pod that starves CPU or I/O makes etcd miss heartbeats → leader elections, slow/failed API calls → the whole cluster goes blind.
• API-server availability must be independent of workload pressure. If a node packed with agent Jobs also runs the API server, an overloaded node can take down control of the entire cluster.
• HA via quorum. Production runs 3 control-plane nodes (odd number → etcd quorum survives one node loss). Dev runs 1 (no HA, cheap). Never run 2 (no quorum benefit, just cost).
• Taint, don't trust. The pool is tainted control-plane:NoSchedule so nothing lands there by accident — isolation is enforced by the scheduler, not by convention.

This is the answer to "the control plane should have its own node": yes — a dedicated, tainted control-plane pool, 1 node in dev and 3 in prod. The old "two pools" description folded it into core; it is now split out.

Autoscaling — three independent layers

Scaling happens at three levels that compose. Two are about pods, one is about nodes — and each scales up and down automatically.

Layer	Unit it resizes	Direction	Applies to
HPA (Horizontal Pod Autoscaler)	replica count of a Deployment	up/down on CPU·mem·custom metric	api, app, admin, LightRAG
VPA (Vertical Pod Autoscaler)	per-pod requests/limits	right-sizes up/down	stateful + right-sizing (in-place on k8s ≥1.33, no restart)
Cluster Autoscaler	node count of a pool	add when pods are Pending, remove when nodes idle	core and workers pools

api (and app/admin) — horizontal + nodes

api is a stateless Deployment (agent state is loaded per request, no per-agent pod), so it scales cleanly:

• HPA: minReplicas 2 → maxReplicas N, target ~60% CPU plus a custom metric (in-flight requests / orchestrator queue depth). Traffic up → more replicas; traffic down → fewer (back to 2).
• Cluster-autoscaler on core: when HPA wants more replicas than fit on current nodes, a new core node is added; when replicas shrink and a node empties, it's removed.
• VPA (recommend/auto): keeps requests honest so HPA's percentage math reflects reality and the scheduler doesn't over- or under-pack nodes.

workers — node scale-to-zero, driven by demand

The worker is a Job, not a Deployment (one task, then exit), so it isn't HPA-scaled — demand is the number of queued tasks:

• runtime/worker creates one Job per task (imperatively, not GitOps).
• Cluster-autoscaler scales the workers pool 0 → N off Pending Job pods, and back to 0 when there's no work (ttlSecondsAfterFinished + the idle reaper clean up finished sessions).
• Right-sized Job requests (≈100m CPU — see incident below) let the autoscaler pack tasks densely and only grow a node when genuinely needed.
• Later (P2, not MVP): a KEDA ScaledJob can scale workers directly off BullMQ queue depth for sharper reaction. Deferred per the locked "no KEDA for MVP" decision — HPA + cluster-autoscaler cover MVP.

Why this matters — the June 2026 incident

A real failure this design prevents: agent Jobs requested 500m CPU but used ~10m. Fourteen of them reserved the workers node to 100% (7700m) while it sat at 6% real usage — so new Jobs hung in Pending (FailedScheduling: Insufficient cpu) and the agent wouldn't start.

Two layers of this strategy fix it directly:

1. Right-sized requests (500m → 100m) — the scheduler's reservation math now reflects reality. VPA automates this going forward; on k3s ≥1.33 the in-place resize applies it with no pod restart (exactly how the incident was remediated live).
2. Cluster-autoscaler — a genuinely full workers pool now grows a node instead of hanging Jobs in Pending indefinitely.

Sizing (rough MVP → prod)

Pool	Dev	Prod	Notes
control-plane	1× CPX21	3× CPX31 (fast NVMe)	etcd wants CPU + low-latency disk; HA quorum at 3
core	1–2× CPX41	3+× CCX/CPX (autoscaled)	Postgres/Redis/LightRAG live here; size for steady load
workers	0–1× CPX31	0 → N× CPX41/CCX (autoscaled)	scale-to-zero; bigger nodes for heavy/browser profiles

Per-tenant ResourceQuota + LimitRange cap each team's namespace, and a default LimitRange ensures no pod runs without requests (an unbounded pod breaks autoscaler math and re-creates the incident above).

Provisioning

• IaC (kube-hetzner Terraform / hcloud) defines the three pools; membership is set by node labels + taints (node-role: workers, control-plane:NoSchedule).
• Cluster-autoscaler is wired to the Hetzner cloud provider per pool (min/max node counts); workers min = 0.
• GitOps (ArgoCD) then deploys the workloads on top — see GitOps. The autoscalers themselves (CA, VPA, metrics-server) are part of the platform chart.

The IaC in the first bullet is this repository's terraform/, and it has not raised the clusters that serve traffic — those came from Agentfy/gitops, in the shape described in the box at the top of this page. The last bullet is the one that did happen, in a different repository: Argo CD deploys the workloads.

Locally, it is a real cluster

Development does not run against a stub. A k3d cluster — real k3s inside Docker — is raised by ./.superset/k3d.sh up, and the difference from production is the kubeconfig and the namespace, not the client, the manifests or the RBAC.

Three properties are what make it usable while several people work at once:

• One cluster, a namespace per workspace. Six k3d clusters on one laptop is six virtual machines; one cluster with a namespace each costs a fraction of that and still isolates.
• Its kubeconfig is a separate file. The bring-up never touches ~/.kube/config — a tool that repoints your default context is a tool that eventually points a command at production.
• A pod can reach the api on the host. The cluster's DNS is given host.k3d.internal and host.docker.internal, so the address a worker dials back on is an ordinary hostname rather than a special case. Without that, none of the rest of this works locally.

Every step is idempotent: run up twice and the second run finds everything and changes nothing. The cluster itself is shared, so stopping or deleting it stops it for everyone on the machine — clean removes only your own namespace.

See also

• Resources & implementation — the full inventory of deployables and datastores.
• Worker on Kubernetes — the Job lifecycle the workers pool runs.
• Cloud strategy — when (if ever) any of this moves to AWS.