Engine-agnostic CA extraction controller for the ca.crt-only trust anchor #3286

Context

The merged design proposal unified-tls-pki settles the consume contract for TLS trust anchors: every engine, regardless of who owns its CA, exposes its trust anchor through one canonical object — a Secret named <release>.tenant-ca containing only ca.crt (the DOT is load-bearing: the dash form is ambiguous — for a release foo it collides with an operator-generated CA Secret of a neighbouring release named foo-tenant, and a dot cannot appear in operator-generated Secret names), stamped by the controller with internal.cozystack.io/tenant-ca: "true" and made tenant-visible through the lineage webhook's tenantresource verdict. Kafka's key-free <release>-clients-ca-cert is the reference shape — not the name (see below). The two labels are not interchangeable: the controller stamps tenant-ca, which the generic spec.secrets selector matches; tenantresource is the webhook's authoritative output and cannot be forged by stamping it statically.

Two engines already meet that contract: kafka, where Strimzi publishes a key-free CA natively, and redis, where the forked operator emits one (PR #2729). Every other engine has a key-bearing CA Secret and delivers no ca.crt to the tenant at all. That is one uniform delivery gap, and §5 of the proposal answers it with a single engine-agnostic extraction controller. That controller has no home today — this issue is it. Nothing in main implements it: publish-ca-cert has zero occurrences in the tree.

Why a controller and not something cheaper

The proposal rejects every cheaper mechanism, and the reasons are load-bearing:

• Helm lookup — runs at render time and is invisible to the Flux digest, so a chart would not re-render when an operator-created CA appears asynchronously. PR [platform] Replace Helm lookup with valuesFrom mechanism #1787 already moved the global-values channel off lookup for the same reason.
• A custom valuesFrom on the application HelmRelease — expectedValuesFrom() pins every application HelmRelease to the single cozystack-values Secret and overwrites drift (internal/controller/applicationdefinition_helmreconciler.go).
• Forking each upstream operator — the redis path. Owning forks of CloudNativePG, PSMDB, mariadb-operator, opster and the RabbitMQ cluster operator is far more surface than one controller.
• trust-manager — its Bundle sources read from its own trust namespace, not from a per-release tenant namespace.
• External Secrets Operator — mechanically sound, but ESO is opt-in and not a default platform component, so it cannot be assumed installed. Recorded as the retirement path if that ever changes.

Scope

Watch Secrets labelled internal.cozystack.io/publish-ca-cert: "true"; an optional internal.cozystack.io/publish-ca-cert-key annotation names the key to lift, defaulting to ca.crt. A second required label, internal.cozystack.io/publish-ca-cert-release, carries the release name — the platform runs cert-manager with enableCertificateOwnerRef: false, so a cert-manager Secret has no ownerReference and its release cannot be inferred from the lineage graph; it must be stated explicitly. For the cert-manager-minting charts all three markers ride Certificate.spec.secretTemplate.labels. The label, not the name, is the contract — CA Secret names are non-uniform across engines, and <release>-ca-cert is overloaded to the point of meaning opposite things on redis and mongodb — which is also why it is not the projection's name.

For each labelled source, upsert an intra-namespace type: Opaque Secret named <release>.tenant-ca holding only ca.crt, re-copying on every source change so a CA rotation propagates without a chart re-render.

Four behaviours are load-bearing:

1. Key off the label and the content, never the name. Verify the lifted value carries no -----BEGIN … PRIVATE KEY----- header, and fail closed if it does.
2. Tolerate operator-created, asynchronous sources. CNPG and PSMDB create their CA Secret at runtime, well after the chart renders. The controller waits on the watch event; it must not error or busy-loop before the source exists.
3. Sanitize at write time, not only at render time. Copy a single whitelisted key and re-assert the no-private-key check on every write. Never copy Data wholesale.
4. Owner-ref the projected Secret to the release's HelmRelease. Garbage collection is then native, with no prune logic. Do not owner-ref the source Secret. Note the apps.cozystack.io kinds are virtual — a tenant Application is stored as a HelmRelease — so there is no app CR in etcd to reference; the HelmRelease is the only real object in the chain, and deleting the app CR deletes it, which cascades to the projection. BlockOwnerDeletion stays false so a trust anchor never delays an application's teardown.

Never overwrite a Secret the controller did not create; surface a name collision as a Warning Event rather than failing silently.

Reuse

The skeleton is the wildcard-secret reconciler from PR #2990 (internal/controller/wildcardsecret/reconciler.go), with one structural difference: that controller replicates cross-namespace, while this one is strictly intra-namespace and per-release.

Tenant marking needs no per-release templating — one generic spec.secrets entry with matchLabels: {internal.cozystack.io/tenant-ca: "true"}, stamped by the controller on every projected Secret, covers every engine. The lineage admission webhook (internal/lineagecontrollerwebhook/webhook.go) remains the authority on the tenantresource label; a statically stamped label does not survive on its own. The private-key guard already exists in cozy-lib.tls.caCertSecret (packages/library/cozy-lib/templates/_tls.tpl, PR #2989).

The irreducibly new work is the extraction step itself: read one key, write a key-free copy, re-copy on rotation.

Source selection: two legs

The open question originally recorded here — whether operators that create their own CA Secret can be made to label it — has been answered. Neither CloudNativePG nor Percona PSMDB can, so source selection is two-legged and both legs are first-class:

1. Label-driven (the default). Watch Secrets carrying internal.cozystack.io/publish-ca-cert: "true". This is how the cert-manager-minting engines opt in, stamping the label through Certificate.spec.secretTemplate.labels.
2. Name-driven fallback. Watch a named Secret in the release namespace, with no label on the source at all. Required for postgres (CNPG) and mongodb (PSMDB), whose operators will not label their output. Keep this declarative — a small per-engine configuration of source name and key to lift, so that adding an engine is data rather than code.

Projection, sanitization, the private-key guard, owner-ref and collision handling are identical on both legs; only source discovery differs. The write path must not fork — that is exactly where a key-leak bug would hide.

The private-key guard matters more on the fallback leg rather than less. CNPG's <cluster>-ca genuinely carries ca.key, and PSMDB's <release>-ca-cert genuinely carries a private key, so on that leg the source is key-bearing by construction. A test must prove that a source holding both ca.crt and ca.key yields a projected Secret holding ca.crt alone.

The evidence for both verdicts is in the comments. A possible escape from the fallback for postgres only (CNPG bring-your-own-CA) is tracked as #3295; it does not remove the need for the mechanism, because mongodb forces it regardless.

Acceptance

• A tenant can read ca.crt for any converged engine, and cannot read any object carrying tls.key or ca.key.
• A CA rotation on the source propagates to the projected Secret without a chart re-render.
• Deleting the application instance garbage-collects the projected Secret, with no prune logic.
• The projected Secret is never written over a pre-existing foreign Secret of the same name.

Part of #2811. Blocks #2814.