Talos worker nodes fail to register: bootstrap-token secret missing from workload cluster kube-system #3875

Summary

Fresh Talos worker VMs come up healthy, apid responds, kubelet service is Running/OK, but nodes never register in the workload cluster. All API calls from kubelet are silently rejected as system:anonymous, including the CSR that would give kubelet its client certificate. Machines stay Provisioned indefinitely; the workload cluster ends up with zero Nodes.

Root cause: the workload cluster's kube-system namespace does not contain the bootstrap-token-<id> Secret that matches the token embedded in the TalosConfig bootstrap data. The token is generated and referenced by TalosConfig (cluster.token: <id>.<secret>), but nothing on the Cozystack side creates the corresponding bootstrap-token Secret inside the workload cluster.

Symptom overlaps with #3513 (0 nodes registered, machines Provisioned, VMIs healthy), but the failure mode is deterministic on affected clusters, not flaky. #3513 tracks other manifestations of the same "no nodes register" outcome.

Reproduction / evidence

On an affected tenant Kubernetes CR (a prod cluster that ran for weeks with the same misconfig):

$ kubectl --context host --namespace tenant-x get machines
NAME                CLUSTER         NODENAME   PROVIDERID           PHASE         AGE   VERSION
ht-apps-abc-def                                kubevirt://...       Provisioned   11d   v1.33.13
ht-apps-abc-ghi                                kubevirt://...       Provisioned   11d   v1.33.13

$ kubectl --context tenantkubeconfig get nodes
No resources found

$ kubectl --context host --namespace tenant-x get vmi
# all Running, agent connected, IPs assigned

talosctl service kubelet on each worker:

STATE    Running
HEALTH   OK
EVENTS   [Waiting]: task "kubelet" failed: exit code 1
         (last log "No valid client certificate is found and the server is
          responsive, exiting.")

talosctl logs kubelet (looping every ~7s):

"Failed to ensure lease exists":
  leases.coordination.k8s.io "ht-apps-abc-def" is forbidden:
  User "system:anonymous" cannot get resource "leases" ...

"Unable to register node with API server":
  nodes "ht-apps-abc-def" is forbidden:
  User "system:anonymous" cannot get resource "nodes" ...

"Failed while requesting a signed certificate from the control plane":
  cannot create certificate signing request: Unauthorized

bootstrap-token present in the TalosConfig secret:

$ kubectl --context host --namespace tenant-x \
    get secret ht-apps-abc-def-bootstrap-data \
    --output=jsonpath='{.data.value}' | base64 -d | grep -E 'token:'
    token: <id>.<secret>          # machine.token (Talos join)
    token: mcvbbi.<secret>        # cluster.token (kubelet bootstrap)  <- this one

Same mcvbbi id is in ${release}-talos-secrets under bootstrapTokenId.

bootstrap-token-mcvbbi Secret in workload kube-system:

$ kubectl --context tenantkubeconfig --namespace kube-system \
    get secret --field-selector=type=bootstrap.kubernetes.io/token
No resources found in kube-system namespace.

Comparison with a working cluster

Same chart version, another tenant that has been healthy since day one:

$ kubectl --context tenantkubeconfig-staging --namespace kube-system \
    get secret --field-selector=type=bootstrap.kubernetes.io/token
NAME                     TYPE                            DATA   AGE
bootstrap-token-ac7loi   bootstrap.kubernetes.io/token   6      11d

ac7loi matches bootstrapTokenId in that tenant's ${release}-talos-secrets. The difference between working and broken clusters is exactly whether this Secret exists.

Fix that unblocks the cluster

Manually create the missing Secret in the workload cluster:

apiVersion: v1
kind: Secret
metadata:
  name: bootstrap-token-<id>
  namespace: kube-system
type: bootstrap.kubernetes.io/token
stringData:
  token-id: <id>
  token-secret: <secret>
  usage-bootstrap-authentication: "true"
  usage-bootstrap-signing: "true"
  auth-extra-groups: system:bootstrappers:kubeadm:default-node-token

Values taken from ${release}-talos-secrets.data.bootstrapTokenId / bootstrapTokenSecret.

After applying: kubelet immediately transitions to authenticated, CSR is auto-approved by kubeadm:node-autoapprove-bootstrap (present in the workload cluster), nodes become Ready, workloads move.

Suggested cozystack change

Either:

• talos-reconcile Job should also kubectl apply a bootstrap-token Secret against the workload cluster (it already assembles the same bootstrapTokenId/Secret values), or

• The tenant Kubernetes chart should provision this Secret at cluster-init time and re-provision it if the underlying ${release}-talos-secrets rotates.

Currently the CSR-approver ClusterRoleBinding (kubeadm:node-autoapprove-bootstrap) is present but useless without the matching bootstrap-token Secret.

Environment

• Cozystack v1.6.x, Kamaji-backed tenant Kubernetes clusters
• Chart version: 0.0.0+7c18a8ec4e4d
• Reproduced on two independent tenant clusters (long-running prod, not a fresh e2e install)
• Not reproduced on a sibling tenant that was created earlier and has the Secret present

Notes

• Related: E2E: tenant worker bring-up misses the node-Ready budget with zero Nodes registered — survived #3455's fsync fix and the 12m→18m raise #3513 (same "0 nodes registered" outcome — different root cause though; E2E: tenant worker bring-up misses the node-Ready budget with zero Nodes registered — survived #3455's fsync fix and the 12m→18m raise #3513's traces point at pull/CRI issues before kubelet even connects, this one is post-connect auth failure)
• Related: talos-reconcile Job is OOMKilled at 256Mi and never completes #3519 (talos-reconcile Job OOM at 256Mi — orthogonal to this bug, but blocks the reconcile Job that could have applied the bootstrap-token fix)