postgres: init-job hook mounts operator-managed superuser secret without waiting, racing CNPG on fresh install #3921

Description

packages/apps/postgres/templates/init-job.yaml defines <release>-init-job as a post-install,post-upgrade Helm hook (weight -5). The Job mounts the <release>-superuser secret both as a volume and via env.secretKeyRef:

• volume secret -> secretName: <release>-superuser (no optional: true)
• env PGUSER/PGPASSWORD -> secretKeyRef on <release>-superuser

This secret is not rendered by the chart. It is created asynchronously by the CNPG operator (the chart only sets enableSuperuserAccess: true on the Cluster in db.yaml). On a fresh install the hook Pod can therefore start before CNPG has materialized the secret, opening a FailedMount / CreateContainerConfigError window. There is no initContainer wait and no optional: true guard.

The kubelet retries the mount, so this is usually a startup delay rather than a hard deadlock. But because Helm blocks on hook completion, a slow CNPG bootstrap can burn the hook timeout and roll the release back. The file header already documents a concrete incarnation of this class (hook timing out during bootstrap.recovery, Helm rolling the upgrade back, CNPG panicking in InitInfo.loadBackup), currently worked around symptomatically by skipping the Job entirely under bootstrap.enabled.

Secondary lifecycle issue: as a post-upgrade hook, the Job never runs while the release is stuck in UpgradeFailed, so role/database convergence silently stops until the release recovers.

Why this matters

This is the same anti-pattern already fixed twice in apps/kubernetes: a hook Job depending on an operator-created resource with no ordering guarantee (talos-reconcile #3145, bootstrap-token #3875 / #3887). The umbrella #2497 tracks the root cause of hooks used in place of dependency management.

Suggested fix

Follow the shape of #3887:

• move the Job out of the Helm hook lifecycle into a main-phase resource (the release already tolerates this pattern),
• add an initContainer that waits for <release>-superuser,
• mark the secret volume optional: true so the Pod waits in init rather than failing the mount,
• give the Job a content-hash name suffix + ttlSecondsAfterFinished for self-healing.

Affected

• packages/apps/postgres/templates/init-job.yaml