Homelab Rebuild Part 16: Handing the Cluster to FluxCD

Intro

Until today, every app in the cluster got to the cluster by hand: a helm upgrade per release, or a kubectl apply of a few manifests. The values were in Git, but nothing made sure the cluster matched them. Part 15 showed how that goes wrong: one kubectl apply -f on the wrong file created a second Postgres cluster in default, and it ran for three hours before I noticed.

So I handed the cluster to FluxCD: Git is now the only way a change gets in. This post covers why Flux over Argo CD, how I converted 20 app folders, how Flux took over the running Helm releases without reinstalling them, and the Proxmox cluster split that happened halfway through.

TL;DR: Flux’s helm-controller adopts releases installed with the Helm CLI as an ordinary upgrade, so the takeover restarted almost nothing. Kustomize’s helmCharts: doesn’t work in Flux; each chart became a HelmRelease with its values file turned into a ConfigMap. Every app started suspended and was switched on in small batches. During one batch, the three Proxmox hosts lost each other for 22 minutes. The cluster recovered on its own, one OpenBao pod needed a restart, and I still don’t know the cause.

The Situation

• k0s cluster: three controllers and five workers on three Proxmox hosts (pve-01, pve-02, pve-03)
• 20 app folders under kubernetes/apps/, 15 of them with a Helm chart
• Each app folder had a kustomization.yaml with a helmCharts: block. That block was only used to render locally; the real installs were helm upgrade --install from each app’s README
• Some manifests were never in a kustomization at all and were applied by hand: cert-manager’s ClusterIssuers, the External Secrets ClusterSecretStore, Kyverno policies, Grafana dashboards and alert rules
• Helm 4.2.1 on my laptop

Flux or Argo CD

I’m the only person deploying, memory on these hosts is tight, and everything was already a Helm release. Flux fits. Headlamp, which already runs in the cluster, has a Flux plugin if I want a UI.

Setup

Layout

clusters/homelab/
  flux-system/
    gotk-components.yaml   # flux install --export
    gotk-sync.yaml         # GitRepository + root Kustomization
    kustomization.yaml     # resource patches
  apps.yaml                # one Flux Kustomization per kubernetes/apps/<app>
kubernetes/apps/<app>/     # unchanged place for each app

I installed Flux 2.9.6 with only the three controllers I need: no notification controller, no image automation. The export goes into Git, and a kustomize patch sets requests and limits to the cluster’s policy (Flux’s defaults are 1 CPU and 1 GiB per controller):

The GitRepository reads the repo over SSH with a read-only deploy key and only packs the two folders Flux needs:

spec:
  url: ssh://[email protected]/<org>/infrastructure
  ref:
    branch: main
  secretRef:
    name: flux-system
  ignore: |
    /*
    !/clusters
    !/kubernetes

One Flux Kustomization per app

apps.yaml has 20 entries like this one:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: headlamp
  namespace: flux-system
spec:
  suspend: true
  interval: 10m
  retryInterval: 2m
  path: ./kubernetes/apps/headlamp
  prune: true
  wait: false
  sourceRef:
    kind: GitRepository
    name: flux-system

prune: true deletes anything removed from Git. That’s what I want for most objects, but not for namespaces: removing an app folder by mistake would delete its namespace, its PVCs and the data on them. Every namespace.yaml now carries an opt-out:

metadata:
  annotations:
    kustomize.toolkit.fluxcd.io/prune: disabled

helmCharts becomes HelmRelease

Flux’s kustomize-controller doesn’t inflate helmCharts:. A short script turned each block into a chart source and a HelmRelease, keeping the same release name, namespace and chart version as the live release, so Flux would see an existing release to upgrade, not a new one to install:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: headlamp
  namespace: headlamp
spec:
  interval: 1h
  releaseName: headlamp
  chart:
    spec:
      chart: headlamp
      version: 0.45.0
      sourceRef:
        kind: HelmRepository
        name: kubernetes-sigs-github-io-headlamp
  valuesFrom:
    - kind: ConfigMap
      name: headlamp-values
      valuesKey: values.yaml

The values files didn’t move. Kustomize turns each one into a ConfigMap, and the watch label tells helm-controller to upgrade as soon as the values change instead of waiting for the next interval:

configMapGenerator:
  - name: headlamp-values
    namespace: headlamp
    files:
      - values.yaml=values.yaml
generatorOptions:
  disableNameSuffixHash: true
  labels:
    reconcile.fluxcd.io/watch: Enabled

Two charts come from OCI registries (Actions Runner Controller, and External Secrets, whose HTTP chart repo timed out in Part 15). Those use an OCIRepository and chartRef instead.

A few per-app settings carried over lessons from earlier parts:

• Kyverno: install.disableWait and upgrade.disableWait, the equivalent of the --wait=hookOnly that got it unstuck in Part 15
• Cilium: upgrade.disableWait, and no automatic rollback; for the CNI I’d rather fix forward
• Tailscale operator: dropped a valuesInline block that duplicated a key already in values.yaml

The manifests that used to be applied by hand went into their app’s kustomization, including the Grafana dashboards, now ConfigMaps generated with the grafana_dashboard: "1" label the sidecar looks for.

A staged takeover

Every Flux Kustomization started with suspend: true, so merging the PR changed nothing in the apps. Then each batch was a one-line commit flipping suspend to false, followed by a check of what had restarted:

1. headlamp alone, to test adoption
2. local-path, LimitRanges, cert-manager, External Secrets, external-dns, gateway, KEDA, the CloudNativePG operator and the Postgres cluster
3. monitoring, Pi-hole, cloudflared, Actions Runner Controller, Tailscale operator
4. tetragon, Kyverno, OpenBao and Cilium, one at a time

Headlamp answered the main question. The release went from revision 3 (my last helm upgrade) to revision 4, made by Flux, and the pod didn’t restart:

REVISION  UPDATED                   STATUS      CHART            DESCRIPTION
3         Wed Oct 14 07:38:56 2026  superseded  headlamp-0.45.0  Upgrade complete
4         Wed Oct 14 11:26:27 2026  deployed    headlamp-0.45.0  Upgrade complete

Issue 1: Cluster-scoped objects got a namespace

Symptom

Rendering the Kyverno folder after the conversion:

kind: GeneratingPolicy
metadata:
  name: default-deny-ingress
  namespace: kyverno

Cause

A top-level namespace: in kustomization.yaml sets the namespace on every object kustomize doesn’t know to be cluster-scoped. It knows built-in kinds like ClusterRoleBinding, but not custom ones. Kyverno’s policies, cert-manager’s ClusterIssuer, External Secrets’ ClusterSecretStore and Tailscale’s ProxyClass are all cluster-scoped custom resources.

Fix

No top-level namespace: in those four folders. The namespaced objects there (the generated HelmRelease and ConfigMaps, a Certificate, an ExternalSecret) set metadata.namespace themselves.

Issue 2: The Flux CLI couldn’t verify TLS

Symptom

failed to get server groups: Get "https://<api-proxy>/api":
tls: failed to verify certificate: SecPolicyCreateSSL error: 0

Cause

I run tools on my laptop inside a sandbox (nono), and Go binaries on macOS need access to the system keychain to verify certificates. kubectl worked; the flux CLI didn’t.

Fix

The only thing I needed the CLI for was flux create secret git. The same Secret is easy to build by hand, and the private key never leaves the cluster and a temporary folder that’s deleted right after:

ssh-keygen -q -t ed25519 -N "" -C flux-homelab -f ./identity
ssh-keyscan -t ed25519,ecdsa,rsa github.com > ./known_hosts
kubectl -n flux-system create secret generic flux-system \
  --from-file=identity=./identity \
  --from-file=identity.pub=./identity.pub \
  --from-file=known_hosts=./known_hosts
rm ./identity
gh repo deploy-key add ./identity.pub --repo <org>/infrastructure --title flux-homelab

gh repo deploy-key add creates read-only keys by default, which is all Flux needs.

One more small thing: the first kubectl apply -k clusters/homelab/flux-system failed for the GitRepository and Kustomization with no matches for kind ... ensure CRDs are installed first, because the CRDs from the same apply weren’t established yet. Running it a second time fixed it.

Issue 3: The few pods that did restart

Symptom

Most takeovers restarted nothing. A handful of pods rolled once.

Cause

Each one had a different, harmless reason:

Fix

Nothing to fix. They happen once, on the first Flux upgrade. The two apps I worried about most were the quietest: OpenBao (revision 7) upgraded with no field conflict on the injector’s webhook caBundle, which needed --server-side=false with the Helm 4 CLI in Part 15, and Cilium (revision 6) left the agent and Envoy DaemonSets at generation 7, so no node lost networking.

Issue 4: The hosts lost each other for 22 minutes

Symptom

A few minutes after the batch 3 commit, kubectl through the Tailscale API proxy started timing out. From pve-01’s corosync log:

11:39:47 corosync[1359]: [TOTEM ] Token has not been received in 2737 ms
11:39:54 corosync[1359]: [TOTEM ] A processor failed, forming new configuration: token timed out (3650ms)
11:40:09 corosync[1359]: [QUORUM] Sync left[2]: 2 3
11:40:09 corosync[1359]: [QUORUM] This node is within the non-primary component and will NOT provide any services.

Each host saw only itself (Nodes: 1, Quorate: No), and the links kept coming up for a few seconds and dropping again.

Inside the k0s cluster:

• kube-apiserver on controller-03 grew to 2.3 GiB RSS in a 3 GiB VM, with 12 MiB available and no swap
• SSH to controller-01 and controller-02 timed out during the banner exchange
• Controllers that lost their leases restarted: cert-manager, the CloudNativePG operator (10 times), KEDA, Kyverno, cilium-operator (9), the Flux controllers, and one Postgres instance (4)

What it was not

Cause

Unknown. At 12:01:38 all three hosts formed a full membership again on their own, and corosync has been stable since. The split started right after the batch 3 commit (monitoring, Pi-hole, cloudflared, ARC, Tailscale operator), but I found nothing that connects those upgrades to host-to-host traffic on the LAN. I’m leaving it as an open question rather than guessing, and watching the corosync logs for a repeat.

Fix

Proxmox and k0s recovered by themselves, with two exceptions:

• openbao-2 (the Raft voter on the pve-03 worker) was hung: no log lines after 11:51, and its readiness probe (bao status, 3 second timeout) kept failing. openbao-0 and openbao-1 kept quorum the whole time. Deleting the pod brought it back unsealed as a standby.
• My laptop lost Tailscale’s DNS for *.ts.net (no such host for the API proxy, and no ts.net resolver in scutil --dns). Reconnecting Tailscale fixed it.

After the split, kube-apiserver was back to 1.4 to 1.8 GiB on all three controllers. The memory spike came from the split, not from Flux.

Update (2026-10-14, later that day): controller-03 ran out of memory again without any split, so the split was not the root cause; the 3 GiB controllers were too small for kube-apiserver. See Part 18 .

Result

• 20 apps reconciled from main; a change is a commit and a push, then flux reconcile
• No more helm upgrade or kubectl apply by hand; to debug something hands-on, I suspend that app’s Kustomization first
• The takeover restarted only the pods in Issue 3; the riskiest releases (OpenBao, Cilium) restarted nothing
• Flux uses about 270 MiB: helm-controller 119 MiB, kustomize-controller 103 MiB, source-controller 49 MiB

Lessons

1. Flux adopts Helm CLI releases. Same release name, namespace and chart version, and the first reconcile is an ordinary helm upgrade. No reinstall, no downtime.
2. Start suspended, then switch on in batches. Merging the GitOps setup changed nothing; each batch was a one-line commit I could check or revert.
3. Protect namespaces from pruning. One annotation stops a deleted folder from deleting the data under it.
4. Kustomize doesn’t know CRD scopes. A top-level namespace: lands on cluster-scoped custom resources too.
5. Some restarts on takeover are bookkeeping. Old restartedAt annotations and OCI digests in chart labels both change the pod template once.
6. Rule out the obvious before blaming the timing. The split began right after a change I made, and the hosts couldn’t ping each other. But ping was blocked by design, and nothing linked the change to the LAN. Some causes stay open.

Series

This post is part of Homelab Rebuild:

1. Part 1: Upgrading Proxmox VE 8 to 9
2. Part 2: Starting Over When the IaC Is Broken
3. Part 3: When Tailscale Breaks Your Cilium Gateway
4. Part 4: Living on the Tailnet
5. Part 5: Secrets Without Secrets in Git
6. Part 6: Certificates, DNS and a Private CA
7. Part 7: Self-Hosted CI and Getting Back to IaC
8. Part 8: Stateful Workloads on Unequal Hardware
9. Part 9: Going Public Safely with Cloudflare Tunnel
10. Part 10: When the Hypervisor Went Quiet
11. Part 11: Hardening the Cluster in Five Layers
12. Part 12: Observability with Coroot, and Losing pve-01
13. Part 13: The Host That Kept Losing Power
14. Part 14: Observability, Take Two with OpenTelemetry
15. Part 15: Requests and Limits for Everything
16. Part 16: Handing the Cluster to FluxCD (this post)
17. Part 17: Making the Guardrails Earn Their Keep
18. Part 18: When the API Server Outgrew Its VM
19. Part 19: Logging In to OpenBao with Cloudflare Access and Google

References

• Flux: Installation
• Flux: Helm Releases
• Flux: Kustomizations
• Flux: OCI Repositories
• Argo CD: Overview
• Kustomize: namespace
• Proxmox VE: Cluster Manager
• Proxmox VE: Firewall