The thing that bugged me about the old pipeline

Earlier posts (Deployment, Security) described how a push to main got from GitHub to the cluster. The honest summary: a hosted runner sat outside the cluster, and the Kubernetes API was firewalled to the world. So every deploy did a little dance — temporarily open the control-plane firewall, run kubectl apply, then close it again, even on failure.

It worked, and it always closed the door behind itself. But two things nagged:

1. A hosted runner can’t see a private cluster. The only way to let it in was to widen the API firewall for the length of the job. Brief, automated, restored every time — but still a door that opened on every deploy.
2. Hosted CI minutes aren’t free. On a private repo you get a monthly allowance and then pay by the minute. A runner that spins up, waits on a flaky control plane, deploys, and tears down burns minutes for work that’s mostly waiting.

The fix for both: stop reaching in from outside. Run the deploy from inside the cluster.

One pod per job, then it’s gone

The deploy step now runs on a self-hosted runner that lives in the cluster, managed by Actions Runner Controller (ARC) and its runner scale sets. The model is the important part:

• A tiny listener watches the GitHub Actions job queue. That’s the only always-on piece, and it’s small enough to disappear into spare capacity.
• When a job that targets the in-cluster runner is queued, the controller creates a fresh pod for it.
• The job runs in that pod — a clean, brand-new environment every time.
• When the job finishes, the pod is deleted. No reuse, no leftover state.

So the runner is ephemeral: it exists only while a job is running, and there is exactly one job per pod. Between deploys there are zero runner pods.

push to main
   │
   ├── build job        →  hosted runner (builds the site + image, pushes it)
   │
   └── deploy job       →  in-cluster runner
                              │  controller spins up a pod
                              │  pod applies the manifests from *inside*
                              │  pod is deleted
                              ▼
                           back to zero runners

Because the deploy now talks to the API server from inside the cluster, the firewall never opens. The control-plane ACL dance is gone entirely — there’s no step left that touches it.

Why this is cheaper

The whole point of “scale to zero” is that idle costs nothing.

• No standing runner. A classic self-hosted runner is a VM you pay for around the clock, whether it’s building anything or not. Here the runner pool has a floor of zero. Deploys are infrequent and short, so the pool is empty almost all the time.
• No new nodes. The runner pods ride the spare capacity already sitting on the cluster’s existing nodes. There’s no dedicated CI machine and no extra node to keep warm — the deploy borrows headroom for a minute and gives it back.
• Less metered hosted time. Only the lightweight build still runs on a hosted runner (more on that below), and it’s quick. The slow part of the old pipeline — a hosted runner waiting on the control plane after re-opening the firewall — simply doesn’t exist anymore.

The steady-state cost of the whole runner system is one little listener pod. Everything else is summoned on demand and billed in pod-seconds against capacity that was already paid for.

Builds stay outside; deploys move inside

A deliberate split: building an image and deploying it are different jobs with different needs.

• The build doesn’t need the cluster at all — it compiles the static site, bakes a container image, and pushes it to the registry. That stays on a hosted runner. Keeping it there means no container-build machinery has to run inside the cluster.
• The deploy is the only part that needs to reach the Kubernetes API, so that’s the part that moved in-cluster.

Splitting them this way keeps the in-cluster runner small and simple, and it leaves a clean upgrade path: if hosted build minutes ever become the thing worth trimming, the build can move in-cluster later by flipping where that job runs — no rewrite required.

What a deploy looks like now

End to end, on a push to main:

1. Build (hosted): build the Hugo site, build and push the container image. Takes seconds.
2. Deploy (in-cluster): the controller spins up a runner pod. It applies the manifests and rolls out the new version — talking to the API server over the in-cluster network, no firewall change.
3. Cleanup: the pod is deleted; the pool returns to zero.

No ACL to open, no ACL to restore, no waiting for a freshly-allowlisted IP to propagate across the control plane. The deploy is just… a deploy.

The same in-cluster runner handles the other apps that share this cluster, too — each one’s deploy lands on a throwaway pod the same way.

A note on secrets and access

I’ll keep this light (the Security post goes deeper). Two things worth saying: the deploy identifies itself to the cluster from inside, using a scoped, in-cluster identity rather than a credential shipped in from the outside — and the keys it needs to unlock app config live in the cluster too, so they don’t travel out to a hosted runner to do their job. Less moving across the boundary, fewer doors to open.

The trade I like

The old pipeline traded a brief, automated firewall opening for the ability to deploy from anywhere. The new one trades nothing — the deploy runs where the cluster already is. It’s cheaper (idle is free), simpler (no ACL choreography), and the runner that does it is gone the moment it’s done.

Index