Backport #3887 (bootstrap-token Job out of the Helm hook) to release-1.6 #4539

What happens

#3875 is closed and the fix, #3887, is merged. But the fix landed on main only: it is not in v1.6.3 and not in v1.6.4. Both tags still ship the Job as a Helm hook.

packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml, identical in v1.6.3 and v1.6.4:

  annotations:
    "helm.sh/hook": post-install,post-upgrade
    "helm.sh/hook-weight": "10"
    "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded

git tag --contains on the merge commit returns v1.7.0-alpha.1 and nothing from the 1.6 line.

Why it matters

The deadlock is deterministic, not flaky, and it takes out every tenant Kubernetes cluster on the platform at once:

1. The chart upgrade waits for MachineDeployment and the child HelmReleases to become ready.
2. Workers cannot register, because the bootstrap-token-<id> Secret is only created by the Job.
3. The Job is a post-upgrade hook, so it runs after the wait succeeds.
4. The wait cannot succeed without nodes.

The upgrade times out, Flux retries, and the cycle repeats. Observed state:

NAME                     CLUSTER   NODENAME   PHASE         AGE
<cluster>-md0-xxxxx-yyy  <cluster>             Provisioned   3h
<cluster>-md0-xxxxx-zzz  <cluster>             Provisioned   3h

VMIs Running with IPs assigned, guest cluster reporting No resources found for nodes, and no bootstrap-token Secret anywhere in the guest kube-system.

Anyone on 1.6.3 hitting this and reading the issue tracker will find #3875 closed, upgrade to 1.6.4 expecting the fix, and land in exactly the same state. That is the specific reason I am filing this rather than commenting on #3875.

Workaround

Create the Secret by hand, reproducing what the Job does. Token material comes from <release>-talos-secrets (bootstrapTokenId, bootstrapTokenSecret), applied against the guest cluster with the admin kubeconfig from <release>-admin-kubeconfig:

apiVersion: v1
kind: Secret
metadata:
  name: bootstrap-token-<id>
  namespace: kube-system
type: bootstrap.kubernetes.io/token
stringData:
  token-id: <id>
  token-secret: <secret>
  usage-bootstrap-authentication: "true"
  usage-bootstrap-signing: "true"
  auth-extra-groups: "system:bootstrappers:kubeadm:default-node-token"

Nodes registered within 90 seconds of the Secret appearing, on all three affected tenants.

Two further things bite after that, worth knowing for anyone following the same path. MachineHealthCheck remediation, on by default at 50% since 1.6.0, deletes the workers that failed to come up, which is fine. But the cluster autoscaler can be left in Scale-up timed out for node group ... after 15m16s backoff, holding the MachineDeployment at zero replicas while pods sit pending; restarting the autoscaler Pod clears it.

Ask

Backport #3887 to release-1.6, or document in the 1.6 release notes that tenant Kubernetes upgrades need the Secret created manually.

Environment

Cozystack v1.5.4 to v1.6.3, then v1.6.4. Bare metal, Talos, several tenants running managed Kubernetes.