Cluster identity is keyed by owner/name, not an immutable id (#3339): decide the migration

Question (owner decision needed)

Cluster identity is keyed by name: the CA subject is `k8s-cluster:/` (`internal/server/ca_provider_server.go`). A cluster that is deleted and recreated under the same name inherits the old identity (#3339).

Already parked on #3634 ("cluster identity: immutable id and re-mint for READY clusters, needs the dual-accept window").

Default if no answer: issue an immutable id per cluster at creation and key identity on it; READY clusters are re-minted behind a dual-accept window (old name-based subject accepted until the re-mint is done).

Acceptance criteria (once confirmed)

New clusters get an immutable id; the CA subject uses it.
A recreated cluster under the same name does not inherit the old identity (test).
READY clusters: dual-accept until re-mint; the window's end is recorded on this issue.

Part of #3339.

Owner accepted the default for this question (go with defaults for all three, 2026-11-12). Proceeding under the default; a PR will state the default as applied.

agent-ff167aad-haiku is starting processing it (default applied: immutable id per cluster; dual-accept window for READY clusters)

agent-ff167aad-haiku: scoped, not yet implemented. The change touches the Cluster protobuf message (regenerated), both cluster store implementations and their schema, the client-certificate mint (cluster_ca_pki.go MintClientCert) and the CA interceptor that maps the CN to the caller, plus the dual-accept window. The re-mint for READY clusters is an operational rollout (nodes need new certificates), so it cannot be verified in this sandbox. Proposed staging, each a reviewable PR: (1) add an immutable cluster_id to the record and store it for new clusters only, no behaviour change; (2) mint and verify the subject with the id, accepting the legacy name-only subject for pre-existing clusters; (3) a re-mint flow and the window end, recorded here. Waiting for a go-ahead on stage 1.

agent-ff167aad-haiku: starting stage 1 (immutable cluster_id on new clusters, no behaviour change)

agent-ff167aad-haiku: stage 1 is already in place on main, so no code change. New clusters get a uuid at create (internal/server/cluster_server.go), and both stores persist it: the Postgres k8s_clusters.id is a non-null primary key and MemStore keeps the same record. What is still missing is stage 2: the CA subject (k8s-cluster:/) and the client certificate still use owner and name, not the id. Stage 2 is next and needs a go-ahead.

agent-ff167aad-haiku: starting stage 2 (subject and certificate carry the cluster id; name-only subjects accepted for pre-existing clusters)

agent-ff167aad-haiku: stage 2 in PR #3706 (CI pending). New certificates bind to the cluster id; name-only certificates still accepted until stage 3. Landing is for the owner.

test(cluster): tag the subject test for the daemon build, like the co…

agent-ff167aad-haiku: starting stage 3 (re-mint ready clusters onto the id-bearing subject, then close the name-only window)

agent-ff167aad-haiku: stage 3 scoped, not implemented. Two findings change the plan. (1) ReconcileOnce sets a cluster to ERROR on any reconcile failure (internal/server/cluster_reconciler.go), and the CA mint/deploy only runs on the settle path for clusters not yet READY, so a re-mint for READY clusters needs its own pass that never writes ERROR. (2) Knowing which clusters still hold a name-only certificate needs a per-cluster marker, a schema change in both stores. Proposed split: 3a = marker column (both stores) + a re-mint pass for READY clusters that deploys the id-bearing certificate and sets the marker, plus a count of clusters still on the legacy subject; name-only certificates still accepted. 3b = reject name-only subjects, in a separate PR, only once the count is zero. Waiting for a go-ahead on 3a; 3b needs its own decision.

agent-ff167aad-haiku: starting stage 3a (ca-identity marker, re-mint pass for ready clusters, legacy count; name-only still accepted)

agent-ff167aad-haiku: stage 3a in PR #3711 (CI pending): a per-cluster CA-id marker, a re-mint pass for ready clusters that never sets ERROR, and a log count of clusters still on the name-only subject. Name-only certificates still accepted; rejection (3b) is a separate decision.

fix(cluster): cluster certificates bind to the immutable cluster id (F…

feat(cluster): re-issue ready clusters the id-bearing autoscaler cred…

QA review of the stage 3b brief (reject name-only cluster certificates)

Reviewed against main at v0.137.1 (stage 3a, #3711, is in v0.137.0). Read-only review; no code changed. Items 1-3 are the real risk and should be settled before the 3b PR is queued.

Gate and rollout risks

The gate ignores clusters that are not ready. WHERE state = 'ready' AND ca_id_bound = false skips degraded, error and provisioning clusters. The re-mint pass skips them too (cluster_reconciler.go, remintCAIdentities). A cluster that was ready, got a name-only credential, then went degraded or error can still have ca_id_bound = false; its autoscaler is locked out when 3b ships. The settle path re-mints on recovery, so it heals later, but the autoscaler is down until then. Suggested gate: WHERE state <> 'deleting' AND ca_id_bound = false, or at least report the non-ready count separately.
ca_id_bound = true means "deployed", not "in use". I found no restart or rollout logic for the CA deploy in internal/cluster or internal/server (it may live elsewhere). If the running autoscaler keeps the old certificate until it restarts, the gate can read 0 while a cluster still presents a name-only subject, which is exactly the lockout the gate exists to prevent. Please confirm that deployCA makes the running process reload the credential.
The gate is per daemon and per moment. It must read 0 on every daemon that runs clusters, and again immediately before the release is deployed, not only when the PR is opened. A restore from backup could also reintroduce ca_id_bound = false rows.
The log line cannot show "the window is closed". "CA identity: N ready cluster(s) still on the name-only subject" prints only when this pass had failures, and never at zero. If mintCACert or deployCA is nil the pass returns silently. For brief item 4, either log an explicit "closed" line or drop the line; only the SQL can prove closure.

Brief vs. code

TestBoundClusterRefusesARecreatedCluster is in ca_subject_id_test.go, not ca_provider_server_test.go. Both files need review, plus ca_subject_id_test.go round-tripping an empty id through parseCASubject (should keep working; the parser stays lenient).
ca_provider_server_test.go builds a name-only subject at two places. One of them uses ScopeClustersRead, so it is a scope-refusal test; after converting it to an id-bearing subject, confirm it still fails for the scope reason and not the subject reason.
Four more RPCs call caIdentity without boundCluster and will refuse name-only subjects after the change. Cover at least one in a test.
In boundCluster, the id != "" guard becomes dead; make the id-match unconditional. Assert PermissionDenied, that the message says to re-issue the credential, and that it does not echo the owner, name or subject.

Process

"Open the PR but do not queue it": do not arm auto-merge on this PR.
Branch from current origin/main.
Changelog fragment goes under changelog.d/ per its README, with the one-time re-issue note in generic wording.
Checks: go vet with and without -tags containarium_client; new and edited tests keep //go:build !containarium_client; show the new refusal test failing on main and passing with the change.

QA follow-up: item 2 verified. `DeployCA` does not restart a running autoscaler

Checked on main at v0.137.1.

What DeployCA does (pkg/core/cluster/manager.go, DeployCA): overwrites the client cert, key, CA cert and cloud-config files on the control-plane VM, pushes the bootstrap script, and runs it.

What the script does (pkg/core/cluster/bootstrap.go, RenderCAUnitScript): rewrites the unit file (identical content), then systemctl daemon-reload and systemctl enable --now k3s-cluster-autoscaler.service.

enable --now only starts a unit that is not already active; on a running unit it is a no-op.
daemon-reload does not restart units.
The ExecStartPre steps (image pull, kill and remove the old container) run only when the unit starts.
Restart=always applies only after the process exits.

Consequence: on a ready cluster the new credential lands on disk, but the running autoscaler keeps the credential it loaded at start. The re-mint pass then sets ca_id_bound = true as soon as the script exits 0 (cluster_reconciler.go, remintCAIdentities). So the stage 3b gate can read 0 while a cluster's autoscaler still presents a name-only subject, and 3b would lock it out.

Not verified: I believe the upstream externalgrpc client loads the key pair once at startup, but that code is not in this repo. If it reloads the files per connection there is no problem. The existing tests stub deployCA (cluster_ca_remint_test.go), so nothing covers this.

Clusters would heal only whenever the autoscaler or its VM next restarts, which is not deterministic.

Suggested fix (prerequisite for 3b, or part of it): make the bootstrap restart the unit when the credential files changed (a systemctl restart after enable, or a checksum compare). Alternatively, set ca_id_bound only after confirming the unit's ActiveEnterTimestamp is newer than the certificate file.

Live check: on any ready cluster's control-plane VM, compare systemctl show k3s-cluster-autoscaler -p ActiveEnterTimestamp with the mtime of the client certificate file. If the unit started before the file was written, the old credential is still in use.

Filed #3718 for the autoscaler restart gap (DeployCA does not restart a running autoscaler). It blocks queueing stage 3b.

Filed #3720: clusters already re-minted by 3a need a one-time forced re-deploy so their running autoscaler picks up the new credential (the fix in #3719 only restarts on a deploy, and the re-mint pass skips bound clusters). Blocks queueing stage 3b.

agent-ff167aad-haiku: starting stage 3b PR (reject name-only certificates). Opened unqueued; the rollout gate must pass on a live daemon before it is queued.

agent-ff167aad-haiku: stage 3b PR #3724 is open and NOT queued (CI pending). It refuses name-only cluster certificates. Queue only after the rollout gate passes on a live daemon: no rows from the SQL in the PR body.

fix(cluster): restart the autoscaler when its credential changes (Foo…

fix(cluster): refuse name-only cluster machine certificates (#3689 st…

test(cluster): mint id-bearing certificates in the CA mTLS tests (#3689…

Navigation Menu

Repository navigation

Cluster identity is keyed by owner/name, not an immutable id (#3339): decide the migration #3689

Question (owner decision needed)

Acceptance criteria (once confirmed)

QA review of the stage 3b brief (reject name-only cluster certificates)

Gate and rollout risks

Brief vs. code

Process

QA follow-up: item 2 verified. `DeployCA` does not restart a running autoscaler

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Cluster identity is keyed by owner/name, not an immutable id (#3339): decide the migration #3689

Description

Question (owner decision needed)

Acceptance criteria (once confirmed)

Activity

hsinatfootprintai commented on Nov 12, 2026

hsinatfootprintai commented on Nov 13, 2026

hsinatfootprintai commented on Nov 13, 2026

hsinatfootprintai commented on Nov 13, 2026

hsinatfootprintai commented on Nov 13, 2026

hsinatfootprintai commented on Nov 13, 2026

hsinatfootprintai commented on Nov 13, 2026

hsinatfootprintai commented on Nov 13, 2026

hsinatfootprintai commented on Nov 13, 2026

hsinatfootprintai commented on Nov 13, 2026

hsinatfootprintai commented on Nov 13, 2026

hsinatfootprintai commented on Nov 14, 2026

QA review of the stage 3b brief (reject name-only cluster certificates)

Gate and rollout risks

Brief vs. code

Process

hsinatfootprintai commented on Nov 14, 2026

QA follow-up: item 2 verified. DeployCA does not restart a running autoscaler

hsinatfootprintai commented on Nov 14, 2026

hsinatfootprintai commented on Nov 14, 2026

hsinatfootprintai commented on Nov 14, 2026

hsinatfootprintai commented on Nov 14, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

QA follow-up: item 2 verified. `DeployCA` does not restart a running autoscaler