Repository navigation
fix(postgres): run init-job in main phase and wait for the superuser secret - #3923
IvanHunters wants to merge 2 commits into
Conversation
github-actions
Bot
added
area/database
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…secret The users/databases/roles reconciler Job mounted the CNPG-managed <release>-superuser secret as a post-install,post-upgrade hook. The secret is created asynchronously by the operator, so the hook Pod raced it and hit FailedMount / CreateContainerConfigError on a fresh install, and as a post-upgrade hook it stopped reconciling entirely while the release was UpgradeFailed. Move the Job to the main phase and gate it with a wait-for-superuser init container behind an optional secret volume, so the Pod waits in init until the operator publishes the secret instead of failing the mount. The Job name now carries a content-hash suffix because a main-phase immutable Job cannot be patched in place across revisions. The bootstrap.recovery skip and the ttlSecondsAfterFinished GC behaviour are preserved. Signed-off-by: IvanHunters <[email protected]>
6241bd5 to
ed1bc76
Compare
The init-job now carries a content-hash name suffix, so consumers that waited on the fixed name `<release>-init-job` broke: - the chainsaw postgres suite asserted and deleted the Job by name - the backup example (examples/backups/postgres/run-all.sh) waited on it Give the Job stable selector labels (component=init-job, instance=<release>) and match it by those everywhere instead of by name. Signed-off-by: IvanHunters <[email protected]>
|
This PR has had no activity for 60 days and was marked |
github-actions
Bot
added
the
lifecycle/stale
|
Closed because no activity followed the |
github-actions
Bot
added
the
lifecycle/rotten
What this PR does
Fixes #3921.
The postgres chart's
<release>-init-job(the users/databases/roles reconciler) mounted the CNPG-managed<release>-superusersecret as apost-install,post-upgradeHelm hook. That secret is created asynchronously by the operator, which caused two problems:FailedMount/CreateContainerConfigError, and Helm blocks on hook completion, so a slow CNPG bootstrap could burn the hook timeout and roll the release back.post-upgradehook the Job never ran while the release wasUpgradeFailed, so role/database convergence silently stopped until the release recovered.This is the same anti-pattern already addressed in
apps/kubernetes(talos-reconcile #3145, bootstrap-token #3875 / #3887), tracked by the umbrella #2497.Change
wait-for-superuserinit container behind anoptional: truesecret volume, so the Pod waits in the init phase until the operator publishes the secret instead of failing the mount. The init container polls the mounted file (no API access, no RBAC), with a deadline below the HelmRelease install timeout so a stuck bootstrap surfaces as CrashLoopBackOff.bootstrap.recoveryskip and thettlSecondsAfterFinishedGC.Out of scope: this PR does not add
helm-install-disable-waitto the postgres AppDef, so a main-phase Job still gates release readiness the same way the hook did. Changing readiness semantics is a broader decision left to a separate discussion.Existing installs
An upgrade replaces the old hook Job (
<release>-init-job) with a new main-phase Job (<release>-init-job-<hash>). The old hook Job is no longer managed by the release, but it self-collects via its existingttlSecondsAfterFinishedonce finished.Downstream repositories
The diff only touches the chart's Job template and its unit test;
values.yaml, the values schema, and CRDs are unchanged, so no downstream repository tracks it.Release note