kubernetes: enabling Hubble Relay at cluster creation deadlocks the cilium and coredns HelmReleases #5315

Describe the bug

A tenant Kubernetes cluster created with Hubble Relay enabled through addons.cilium.valuesOverride never finishes installing its addons. The -cilium and -coredns HelmReleases wait on each other:

• hubble-relay connects to hubble-peer.cozy-cilium.svc.cluster.local, so it needs cluster DNS to become Ready;
• <release>-coredns has dependsOn: <release>-cilium (packages/apps/kubernetes/templates/helmreleases/coredns.yaml), so CoreDNS is not installed until the Cilium release is Ready;
• the Cilium release waits for every workload of the chart to be Ready, hubble-relay included.

CoreDNS is never installed, hubble-relay never becomes Ready, and the Cilium HelmRelease retries forever (remediation.retries: -1). Every addon that depends on Cilium (CSI, cert-manager, metrics-server, the VictoriaMetrics operator, monitoring-agents, VPA) stays in DependencyNotReady. Meanwhile the Kubernetes application reports Ready=True and WorkloadsReady=True.

This only happens when Relay is enabled before CoreDNS is installed, which in practice means at cluster creation. Enabling Relay on a running cluster works, because CoreDNS is already there.

Environment

• Cozystack v1.6.5 (kubernetes chart 0.0.0+7a70bb118f33). The templates involved are unchanged on main (56224a1).
• Provider: on-prem, tenant cluster on KubeVirt, Kubernetes v1.35, Talos v1.13.6

To Reproduce

1. Create a Kubernetes application with:

   addons:
     cilium:
       valuesOverride:
         cilium:
           hubble:
             enabled: true
             relay:
               enabled: true

2. Wait for the worker nodes to join.
3. kubectl get hr -n <tenant-ns> | grep <release>

Expected behaviour

The addons install and the cluster converges, as it does without Relay.

Actual behaviour

<release>-cilium   False  Helm upgrade failed for release cozy-cilium/cilium with chart [email protected]+134b2615a93e: failed early due to stalled resources: [Deployment/cozy-cilium/hubble-relay status: 'Failed']
<release>-coredns  False  dependency '<tenant-ns>/<release>-cilium' is not ready
<release>-csi      False  dependency '<tenant-ns>/<release>-cilium' is not ready
...

status.upgradeFailures went past 170 within two hours. Once the Deployment reaches ProgressDeadlineExceeded, Helm's kstatus check fails each retry early.

Logs

hubble-relay in the tenant cluster (0/1 Running, restarting):

level=info msg="Failed to create peer notify client for peers change notification; will try again after the timeout has expired" subsys=hubble-relay error="rpc error: code = Unavailable desc = dns: A record lookup error: lookup hubble-peer.cozy-cilium.svc.cluster.local. on 10.95.0.10:53: read udp 10.243.0.61:37995->10.95.0.10:53: i/o timeout" connectionTimeout=30s

There is no Service or Deployment in kube-system apart from the konnectivity agents.

Workaround

Remove the dependency once, so that CoreDNS installs:

kubectl -n <tenant-ns> patch hr <release>-coredns --type=json -p '[{"op":"remove","path":"/spec/dependsOn"}]'
kubectl -n <tenant-ns> annotate hr <release>-coredns reconcile.fluxcd.io/requestedAt="$(date -u +%FT%TZ)" --overwrite

CoreDNS, then Cilium, became Ready within about two minutes, and the other addons followed. The parent release restores the dependsOn on its next reconcile, which is harmless once CoreDNS is installed.

Possible fixes

• Drop the -coredns → -cilium dependency. Without a CNI, CoreDNS pods stay Pending, so the CoreDNS release only waits within its own timeout and retries until Cilium is up. It does not need Flux ordering.
• Or keep the ordering and stop the Cilium release from waiting on DNS-dependent components (for example disableWait, or excluding hubble-relay from the wait). This hides real failures of the agent DaemonSet, so it is less attractive.
• At a minimum, document that Hubble Relay must be enabled after the cluster is up (docs: Hubble enablement snippet doesn't say where to put it (Package values nesting) #2772 already covers where Hubble values go).

Additional context

The Kubernetes application stayed Ready/WorkloadsReady for the whole time, so nothing in the platform pointed at the failure. That is a separate gap in the WorkloadMonitor and is not covered by this issue.

Checklist

• I have checked the documentation
• I have searched for similar issues
• I have included all required information
• I have provided clear steps to reproduce
• I have included relevant logs