Repository navigation
feat: keep defaulting and validation working when admission webhooks are unavailable - #10874
Conversation
|
❗ By default, the pull request is configured to backport to all release branches.
|
9de997a to
6706f53
Compare
GellertBabel
left a comment
There was a problem hiding this comment.
I think without the else path the system might have short-lived state where the status is not correct on the CR.
625b0c2 to
d8abcef
Compare
|
/test |
|
@leonardoce, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/27538979619 |
maxlengdell
left a comment
There was a problem hiding this comment.
Looks good, tried it locally and it works as expected. Added optional comment
d88754e to
78ad5fe
Compare
9b7408b to
fe3c32c
Compare
|
@mnencia, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/27776841214 |
…are unavailable When admission webhooks are not installed or fail to respond, resources previously skipped defaulting and validation entirely — leaving the operator to reconcile invalid or incomplete specs. Resources are now defaulted and validated during reconciliation as a fallback. When a resource is missing defaults, the operator either applies them directly or waits for the webhook to do so. When validation fails, the reason is surfaced as an admission error on the resource status and reconciliation stops, so users can see exactly why their resource was rejected even if the webhook never ran. Signed-off-by: Leonardo Cecchi <[email protected]>
Co-authored-by: Gellért Bábel <[email protected]> Signed-off-by: Leonardo Cecchi <[email protected]>
Co-authored-by: Gellért Bábel <[email protected]> Signed-off-by: Leonardo Cecchi <[email protected]>
Signed-off-by: Leonardo Cecchi <[email protected]>
Signed-off-by: Leonardo Cecchi <[email protected]>
Signed-off-by: Niccolò Fei <[email protected]>
The reconcile-loop guard wired the raw validators, so the cnpg.io/validation: disabled break-glass annotation was ignored and the guard halted reconciliation even when the served webhook accepted the object. Wrap the guard validators in the same bypassable validator used by the served webhook. Signed-off-by: Marco Nenciarini <[email protected]>
A backup that failed admission validation stayed in the "invalid backup definition" phase even after its definition was corrected. The start gates only fire from an empty or pending phase, so the backup never started again. Reset the phase when the validation error is cleared. The error message is cleared by the start and pending transitions rather than when the validation error is reset, so the change is carried into the persisted status instead of only mutating the in-memory patch base. Signed-off-by: Marco Nenciarini <[email protected]>
The operator now always manages the webhook serving certificate when it owns the PKI (WEBHOOK_CERT_DIR unset), so the always-running webhook server has a valid certificate to serve. The renamed MANAGE_WEBHOOK_CONFIGURATIONS setting controls only whether the CA bundle is injected into the mutating and validating webhook configurations, so it can be turned off when those are managed externally. Signed-off-by: Marco Nenciarini <[email protected]>
The guard wrote the full validation error into the world-readable status, which can embed field values such as object-store URLs, external-cluster connection parameters and secret names. Persist only the offending field paths and keep the full error in the operator logs. Signed-off-by: Marco Nenciarini <[email protected]>
Some cluster validators passed whole spec structs as the invalid value, serializing object-store URLs, connection parameters and the replica promotion token into the error message. Use field.Required and field.Forbidden, or a scalar value, instead. Signed-off-by: Marco Nenciarini <[email protected]>
Document the ApplyChanges semantics on AdmissionParams, fix a log message typo, and drop a dead reference to a non-existent Database.Default(). Signed-off-by: Marco Nenciarini <[email protected]>
Also add the MANAGE_WEBHOOK_CONFIGURATIONS operator setting to the configuration reference. Signed-off-by: Marco Nenciarini <[email protected]>
The field carried omitempty but no +optional marker, so the generated CRD reference listed it as required. Align it with the ScheduledBackup error field. Signed-off-by: Marco Nenciarini <[email protected]>
The guard cleared the admission error only in memory when validation passed again. The Pooler and ScheduledBackup reconcilers either skip the status write when nothing else changed, or build their patch base from the already-mutated object, so the cleared error never reached the API server and a stale error stuck on the status after the spec was corrected. The guard now persists the cleared status when it owns writes and an error was recorded. Types whose error is cleared by their own reconciler, Cluster through the phase machinery and Database through its status fields, report no admission error to the guard so it does not race that logic. Signed-off-by: Marco Nenciarini <[email protected]>
The admission guard ran before the terminal-phase short-circuit in the Backup reconciler, so a backup that had already reached Completed or Failed was re-validated on every reconcile. Some validation checks depend on the runtime environment rather than the immutable spec (for example the presence of the VolumeSnapshot CRD via HaveVolumeSnapshot), so a transient environment change would flip a Completed/Failed record to the "invalid backup definition" phase and destroy the historical result. Because the recorded terminal phase was overwritten in place, clearing the admission error later reset the phase to empty rather than restoring it, which could also re-trigger an already-finished backup. Move the guard call after the terminal-phase switch so immutable, finished backups are never re-admitted. The DefinitionInvalid recovery path is unaffected: that phase is not terminal, so a fixed backup still flows into the guard and gets cleared. Signed-off-by: Gabriele Bartolini <[email protected]> Assisted-by: Claude Opus 4.8
When the admission guard runs in a reconciler that does not own writes to the object (ApplyChanges is false, for example the instance manager reconciling a Cluster), a validation failure returned a reconcile.TerminalError. A terminal error stops controller-runtime from requeueing, so the reconciler would not retry on its own and could stall until an unrelated event happened to wake it up again. Such a reconciler can neither fix the spec nor record the error in the status: the only sensible action is to wait for the owning reconciler to apply a correction. Requeue after a short delay instead, matching the behavior already used by the defaulting path when ApplyChanges is false. Signed-off-by: Gabriele Bartolini <[email protected]> Assisted-by: Claude Opus 4.8
When the admission guard recorded an invalid Cluster definition it set the phase to "Invalid cluster definition" directly, bypassing RegisterPhase. RegisterPhase normally updates the readiness condition together with the phase, so going around it left the Ready condition untouched: a cluster that was healthy before could keep reporting Ready=True while its definition was invalid. Update the readiness condition to False in SetAdmissionError, mirroring SetClusterReadyCondition, so the persisted status is consistent. The phase machinery still owns clearing the error on the next healthy loop. Signed-off-by: Gabriele Bartolini <[email protected]> Assisted-by: Claude Opus 4.8
|
With the help of Claude, I pushed three fixes for the clear correctness issues; the rest are discussion points. Fixes pushed:
Discussion points (not addressed):
|
##### [\`v1.30.0\`](https://github.com/cloudnative-pg/cloudnative-pg/releases/tag/v1.30.0) **Release date:** Jun 29, 2026 ##### Important changes - Updated the deprecation notice for native (in-tree) Barman Cloud support to reflect that it will now be removed in CloudNativePG 1.31.0, rather than 1.30.0. Users are still encouraged to migrate to the Barman Cloud Plugin. ([#11083](cloudnative-pg/cloudnative-pg#11083)) <!-- 1.29 1.28 --> - The `cluster` reference is now immutable on the `Database`, `Pooler`, `Publication`, `Subscription`, and `ScheduledBackup` resources. Pointing one of these objects at a different cluster has no well-defined semantics and previously left the controllers in an inconsistent state; the update is now rejected at the API server via a CEL validation rule. ([#10743](cloudnative-pg/cloudnative-pg#10743)) <!-- 1.29 1.28 1.25 --> ##### Features - **Primary `Lease` for safe primary election**: introduced a Kubernetes `Lease` object (named after the cluster) that acts as a mutex serializing primary promotion: the instance manager must hold the lease before acting as primary and releases it on clean shutdown so replicas can promote without waiting for the full TTL. Timings are configurable via the new `.spec.primaryLease` stanza. The lease is a promotion gate, not a fence. Primary isolation remains responsible for fencing. ([#10627](cloudnative-pg/cloudnative-pg#10627)) - **`DatabaseRole` CRD for declarative role management**: introduced a `DatabaseRole` custom resource that manages a PostgreSQL role as a standalone Kubernetes object, instead of declaring it inline in the `Cluster`'s `.spec.managed.roles` stanza. Each role gets its own lifecycle, status, and RBAC, which suits GitOps workflows and lets role definitions live next to the applications that own them. The spec reuses the same `RoleConfiguration` structure as the inline method, so migrating a role is a matter of moving the stanza into its own manifest. A `databaseRoleReclaimPolicy` field (`retain`, the default, or `delete`) controls what happens to the role when the resource is deleted, mirroring persistent volumes. ([#6155](cloudnative-pg/cloudnative-pg#6155)) - **TLS client certificates for declarative roles**: a `DatabaseRole` can now include a `clientCertificate` block to have the operator automatically generate and renew a TLS client certificate, signed by the cluster's client CA and stored in a `<databaserole-name>-client-cert` Secret. This enables password-free PostgreSQL `cert` authentication; the Secret is cleaned up when the feature is disabled or the `DatabaseRole` is deleted. ([#10896](cloudnative-pg/cloudnative-pg#10896)) - **PgBouncer image management via image catalogs**: the `Pooler` resource can now reference an entry in an `ImageCatalog` or `ClusterImageCatalog` through the new `spec.pgbouncer.imageCatalogRef` field, centralizing PgBouncer image management. When a catalog entry is updated, all referencing `Poolers` are automatically reconciled and roll out the new image without any change to their spec. The resolved image is reported in `status.image`, and a new `status.phase` (`active`, `paused`, `inactive`, or `failed`), also surfaced as a `Phase` column in `kubectl get pooler`, summarizes the lifecycle. ([#10568](cloudnative-pg/cloudnative-pg#10568)) ##### Enhancements - Enabled `pg_upgrade` in-place major upgrades to PostgreSQL 19 or later for clusters that use Image Volume extensions, building on the extension-path support added to `pg_upgrade` in PostgreSQL 19. During the upgrade `Job`, the source- and target-version extension images are mounted side by side, so the old server keeps its libraries and a failed upgrade reverts cleanly. ([#10366](cloudnative-pg/cloudnative-pg#10366)) <!-- 1.29 --> - Added TLS support for the `Pooler` metrics endpoint via `.spec.monitoring.tls.enabled`. When enabled, the metrics server is served over HTTPS, reusing the certificate and key from `.spec.pgbouncer.clientTLSSecret` and reloading it on every handshake to support rotation without a restart; the generated `PodMonitor` scrapes over `https` accordingly. ([#10466](cloudnative-pg/cloudnative-pg#10466)) - Added a label selector to the `Cluster` scale subresource (`status.selector`), making a `Cluster` a valid `targetRef` for the Vertical Pod Autoscaler (VPA) and Horizontal Pod Autoscaler (HPA), which can now map a `Cluster` to its instance pods. Contributed by [@sebv004](https://github.com/sebv004). ([#8996](cloudnative-pg/cloudnative-pg#8996)) <!-- 1.29 1.28 1.25 --> - The operator now emits a `Warning` `PrimaryStatusCheckFailed` event on the `Cluster` when the primary pod is `Ready` from the kubelet perspective but the operator's `/pg/status` check fails and failover is deferred, giving users visibility into the deferral via `kubectl describe cluster`. ([#10509](cloudnative-pg/cloudnative-pg#10509)) <!-- 1.29 1.28 1.25 --> - Added the `ENABLE_WEBHOOK_NAMESPACE_SUFFIX` flag, which suffixes the operator's webhook configuration names with `-<OPERATOR_NAMESPACE>` so that multiple operator instances can coexist on the same cluster. The operator only looks up these configurations; users must create and maintain them. Contributed by [@maxlengdell](https://github.com/maxlengdell). ([#10420](cloudnative-pg/cloudnative-pg#10420)) <!-- 1.29 1.28 1.25 --> - The operator now reloads a CNPG-i plugin automatically when its pods are rolled: it watches the `EndpointSlices` backing plugin `Services` and re-enqueues every cluster using the plugin once the new pods become `Ready`, so an upgraded plugin is picked up without waiting for the next resync. ([#10836](cloudnative-pg/cloudnative-pg#10836)) <!-- 1.29 1.28 1.25 --> - Instance serial numbers are now assigned by reusing the lowest free slot among existing instance names, instead of always incrementing a global counter. Pod and PVC names stay stable across instance recreation (for example, an instance recreated after a node drain comes back with the same name), and serials freed by deleted instances are reclaimed. A new `Initialized` cluster condition reports whether the cluster has completed its first bootstrap, and `status.latestGeneratedNode` is deprecated: it is no longer written, but is preserved on the CRD for backward compatibility. ([#10548](cloudnative-pg/cloudnative-pg#10548)) - Defaulting and validation now run during reconciliation as a fallback when admission webhooks are unavailable, or configured to ignore failures, so the operator no longer reconciles invalid or incomplete specs. Missing defaults are applied directly, and validation failures are surfaced in the resource status instead of failing silently later. ([#10874](cloudnative-pg/cloudnative-pg#10874)) ##### Security - **`CVE-2026-55769` / `GHSA-x8c2-3p4r-v9r6`: `search_path` pinning on operator-issued connections**: a database owner could plant overloaded built-in operators in the `public` schema and alter the `search_path` so that operator introspection probes, running as the cluster superuser, resolved those overloads before `pg_catalog`, a `CWE-426` privilege-escalation chain (same class as `CVE-2018-1058`) that could lead to in-pod RCE via `COPY ... FROM PROGRAM`. The operator now pins `search_path = pg_catalog, public, pg_temp` on every pooled connection so it ships in the startup message and takes precedence over tenant-controlled defaults. ([#10774](cloudnative-pg/cloudnative-pg#10774), [`GHSA-x8c2-3p4r-v9r6`](GHSA-x8c2-3p4r-v9r6)) <!-- 1.29 1.28 1.25 --> - **`GHSA-7qwx-x8ff-3px9`: authenticated operator-to-instance-manager calls**: the instance manager's remote webserver relied on network isolation rather than authentication for its operator-only control endpoints, so any party able to reach the pod's status port could invoke them, disrupting backup orchestration and WAL archival and reading operational metadata. (The upgrade endpoint is SHA-256-pinned, so this did not permit arbitrary code execution.) The operator now generates an in-memory ECDSA P-256 client certificate at startup and reconciles its SHA-256 fingerprint into the cluster status; the instance manager rejects requests to sensitive endpoints that do not present a matching certificate. This hardening is not backported; earlier releases should continue to restrict the status port with a `NetworkPolicy`. ([#10579](cloudnative-pg/cloudnative-pg#10579), [`GHSA-7qwx-x8ff-3px9`](GHSA-7qwx-x8ff-3px9)) - **`CVE-2026-55765` / `GHSA-w3gf-xc94-wvmj`: operator-side SCRAM-SHA-256 password encoding**: the operator now SCRAM-SHA-256 encodes cleartext role passwords before issuing `CREATE`/`ALTER ROLE ... PASSWORD`, so the literal PostgreSQL parses (and that extensions such as `pg_stat_statements` or `pgaudit` may capture) is the SCRAM verifier rather than the cleartext secret. Pre-hashed (MD5 or SCRAM) values are forwarded unchanged, and the per-Secret annotation `cnpg.io/passwordPassthrough: "enabled"` opts out. ([#10724](cloudnative-pg/cloudnative-pg#10724), [`GHSA-w3gf-xc94-wvmj`](GHSA-w3gf-xc94-wvmj)) <!-- 1.29 1.28 1.25 --> ##### Changes - Added support for Kubernetes 1.36. ([#10900](cloudnative-pg/cloudnative-pg#10900)) <!-- 1.29 1.28 --> - Updated the default PostgreSQL version to 18.4. ([#10719](cloudnative-pg/cloudnative-pg#10719)) <!-- 1.29 1.28 1.25 --> - Updated the Kubernetes versions used to test the operator on public cloud providers. ([#10720](cloudnative-pg/cloudnative-pg#10720), <!-- 1.29 1.28 1.25 --> [#10563](cloudnative-pg/cloudnative-pg#10563), <!-- 1.29 1.28 1.25 --> [#11033](cloudnative-pg/cloudnative-pg#11033)) <!-- 1.29 1.28 1.25 --> ##### Fixes - Fixed declarative `Database`, `Publication`, and `Subscription` objects reporting a stale primary-side status forever after their cluster was demoted to a replica; the controller now re-checks the replica condition and watches the `Cluster` so a demotion is detected promptly. ([#10871](cloudnative-pg/cloudnative-pg#10871)) <!-- 1.29 1.28 1.25 --> - Fixed non-sequential pod names (for example `-1`, `-3`) caused by the instance serial counter being advanced before the corresponding `Job` and PVCs were created; the bump is now persisted only after those resources exist. ([#10491](cloudnative-pg/cloudnative-pg#10491)) <!-- 1.29 1.28 1.25 --> - Fixed deletion of a `Database`, `Publication`, or `Subscription` getting stuck in `Terminating` on a replica cluster, where the replica gate ran before the finalizer reconciler and the finalizer was never released. On a replica the PostgreSQL object is left to the primary cluster. ([#10853](cloudnative-pg/cloudnative-pg#10853)) <!-- 1.29 1.28 1.25 --> - Fixed a conflicting duplicate `Database` or `Subscription` with a `delete` reclaim policy dropping the PostgreSQL object owned by the surviving CR; the drop is now gated on a recorded reconciliation. ([#10870](cloudnative-pg/cloudnative-pg#10870)) <!-- 1.29 1.28 1.25 --> - Fixed the `postgres` superuser being left locked out after superuser access was disabled and then re-enabled, because the cached secret version was not invalidated and the password was never re-applied. Diagnosed by [@mhartmann-jaconi](https://github.com/mhartmann-jaconi). ([#10834](cloudnative-pg/cloudnative-pg#10834)) <!-- 1.29 1.28 1.25 --> - Fixed backups getting stuck in the `started` phase when the instance manager running them was restarted (for example by the in-place upgrade following an operator upgrade) before the backup reached `running`; the reconciliation is now rescheduled so the lost session is detected. ([#10859](cloudnative-pg/cloudnative-pg#10859)) <!-- 1.29 1.28 --> - Fixed `exec`/`attach` streaming to negotiate WebSocket with a SPDY fallback, restoring compatibility both with Kubernetes versions that have removed SPDY and with platforms such as OpenShift that reject WebSocket exec upgrades. Contributed by [@bartscheers](https://github.com/bartscheers). ([#10876](cloudnative-pg/cloudnative-pg#10876), [#10933](cloudnative-pg/cloudnative-pg#10933)) <!-- 1.29 --> - Fixed resource leaks when concurrent `Backup` objects raced: backups now run in strict creation-time order, so an already-executing backup is never preempted by a newer one and its replication slot and PostgreSQL session are no longer orphaned on the primary. Contributed by [@GabriFedi97](https://github.com/GabriFedi97). ([#10747](cloudnative-pg/cloudnative-pg#10747)) <!-- 1.29 1.28 1.25 --> - Fixed role reconciliation clearing the password on a PostgreSQL role when the referenced Secret could not be fetched; the role is now left untouched until the Secret becomes available, and per-action errors are aggregated for better visibility. ([#10053](cloudnative-pg/cloudnative-pg#10053)) <!-- 1.29 1.28 1.25 --> - Fixed a bootstrap failure where a metrics-exporter setup error (commonly a duplicate-key race with the controller) rolled back `streaming_replica` creation and wedged replica joins. The metrics-exporter step now runs in a separate transaction. Contributed by [@BlaiseAntony](https://github.com/BlaiseAntony). ([#10749](cloudnative-pg/cloudnative-pg#10749)) <!-- 1.29 1.28 1.25 --> - Fixed a `ScheduledBackup` controller loop that occurred when a `Backup` was created but its status patch never landed; the controller now adopts an existing `Backup` for the next iteration instead of looping on `AlreadyExists`. ([#10612](cloudnative-pg/cloudnative-pg#10612)) <!-- 1.29 1.28 1.25 --> - Fixed a nil-pointer panic when reconciling a `Pooler` whose `Cluster` has been deleted. ([#10667](cloudnative-pg/cloudnative-pg#10667)) <!-- 1.29 1.28 --> - Fixed bootstrap log handling so that all named log pipes (`postgres`, `postgres.csv`, and `postgres.json`) get consumers during `WithActiveInstance`, preventing regular files from being created in place of the named pipes. ([#10043](cloudnative-pg/cloudnative-pg#10043)) <!-- 1.29 1.28 1.25 --> - Fixed generation of invalid IPv6 URLs by wrapping the address in square brackets. Contributed by [@Infinoid](https://github.com/Infinoid). ([#10682](cloudnative-pg/cloudnative-pg#10682)) <!-- 1.29 1.28 1.25 --> - Fixed an external cluster plugin still being treated as active when its configuration set `enabled: false`. ([#10932](cloudnative-pg/cloudnative-pg#10932)) <!-- 1.29 1.28 1.25 --> - Fixed a race during bootstrap recovery from an object store where the restore job could read a stale `Cluster` (primary not yet recorded and timeline still unset) and have its `.history` files rejected by the split-brain guard. When this happened, recovery stopped at the base backup's timeline and silently dropped transactions committed on later timelines. History files are now allowed while the cluster timeline is unset. Contributed by [@dennispidun](https://github.com/dennispidun). ([#10818](cloudnative-pg/cloudnative-pg#10818)) <!-- 1.29 1.28 1.25 --> - Fixed a race where deleting an instance's PVCs could leave the instance permanently stuck: if the data PVC was removed while a WAL PVC was still terminating, the operator recreated the instance bound to the terminating volume, leaving the Pod unschedulable and blocking all further reconciliation. The operator now waits for terminating PVCs to be fully removed before recreating or reattaching an instance, and surfaces the wait through a log line and the cluster phase. ([#11017](cloudnative-pg/cloudnative-pg#11017)) - Fixed a cache race during cluster creation when the server and client CA resolve to the same Secret (the default): a stale informer cache triggered a redundant `Create` that failed with `AlreadyExists` and could leave the cluster stuck in `Unable to create required cluster objects`. The operator now reuses the already-fetched CA Secret when the names match. ([#10989](cloudnative-pg/cloudnative-pg#10989)) <!-- 1.29 1.28 1.25 --> - Fixed the `pg_basebackup` bootstrap path overwriting or failing on a pre-existing `PGDATA` (for example after a replica Pod restart) by enforcing the same pre-flight directory check already applied by the other bootstrap methods; this also protects statically provisioned PVCs from being silently overwritten. ([#11006](cloudnative-pg/cloudnative-pg#11006)) <!-- 1.29 1.28 1.25 --> - Fixed the `Cluster` phase flapping between `Healthy` and a plugin-failure phase when a post-reconcile plugin hook returned an error; the `Healthy` phase is now registered as the last step of a successful reconciliation, so a loop that ends in a plugin error never reports `Healthy`. Contributed by [@GabriFedi97](https://github.com/GabriFedi97). ([#10421](cloudnative-pg/cloudnative-pg#10421)) <!-- 1.29 1.28 1.25 --> - Fixed stale certificate data and partial reads after an external server's Secret was rotated (for example a CA bundle shrinking from two certificates to one): the file is now written atomically, so libpq always reads either the old or the new value, never a mix. Contributed by [@Anand-240](https://github.com/Anand-240). ([#10975](cloudnative-pg/cloudnative-pg#10975)) <!-- 1.29 1.28 1.25 --> - Fixed plugin connectivity to use the plugin `Service` FQDN instead of its short name, avoiding failures when a cluster-level proxy is automatically injected into pods. Contributed by [@kdautrey](https://github.com/kdautrey). ([#10921](cloudnative-pg/cloudnative-pg#10921)) <!-- 1.29 1.28 1.25 --> - Fixed excessive operator log noise from the per-request `Cluster` create/update validation webhook messages, now logged at `debug` instead of `info`. ([#10984](cloudnative-pg/cloudnative-pg#10984)) <!-- 1.29 1.28 1.25 --> - Fixed `spec.postgresql.parameters` accepting keys that are not valid PostgreSQL parameter names, which could inject arbitrary directives into `postgresql.conf`; key names are now validated by the webhook. ([#11029](cloudnative-pg/cloudnative-pg#11029)) <!-- 1.29 1.28 1.25 --> - Fixed a switchover deadlock when a WAL-archiver plugin was enabled on an existing cluster: with `primaryUpdateMethod: switchover` the primary could not be rolled out because a clean demotion needs the archiver sidecar that is still missing. The operator now recreates the primary Pod in place so the sidecar is injected and archiving resumes. The check also covers plugins that inject the archiver as a native sidecar (an init container with `restartPolicy: Always`), such as the Barman Cloud plugin. ([#11032](cloudnative-pg/cloudnative-pg#11032), <!-- 1.29 1.28 1.25 --> [#11059](cloudnative-pg/cloudnative-pg#11059)) <!-- 1.29 1.28 1.25 --> - Fixed a cluster staying in `Setting up primary` indefinitely when the instance-creation Job exhausted its backoff limit; the operator now detects the terminal Job failure and marks the cluster unrecoverable, naming the failed Job and pointing to its logs. ([#11035](cloudnative-pg/cloudnative-pg#11035)) <!-- 1.29 1.28 1.25 --> - Fixed a first-primary bootstrap deadlock where a status-patch conflict after the data PVC was created but before the initialization Job was started left the orphan Pending PVC counted as an instance, blocking the bootstrap gate; the PVC-state reconciler now recreates the bootstrap Job reusing the assigned serial. ([#11039](cloudnative-pg/cloudnative-pg#11039)) <!-- 1.29 1.28 1.25 --> - Fixed external cluster names and secret selector references being joined into filesystem paths without validation, letting a `..` component or path separator escape the external secrets directory when the instance manager dumps connection material; these values are now rejected at the validating webhook and re-checked at the write site. Reported by [@r0binak](https://github.com/r0binak). ([#11045](cloudnative-pg/cloudnative-pg#11045)) <!-- 1.29 1.28 1.25 --> - Fixed a backup getting stuck in `pending` forever: the concurrent-backup gate ran on every reconcile and could overwrite an already-completed phase written asynchronously by the instance manager. The gate now runs only while the backup phase is still unset or `pending`. ([#11056](cloudnative-pg/cloudnative-pg#11056)) <!-- 1.29 --> - Fixed a declarative `VolumeSnapshot` backup being permanently marked as failed when a stale cache made the operator re-create a snapshot it had already provisioned, failing with `AlreadyExists`. The operator now tolerates the collision when the existing snapshot carries this backup's label and adopts it; a collision with a foreign snapshot still surfaces as an error. ([#11071](cloudnative-pg/cloudnative-pg#11071)) <!-- 1.29 1.28 1.25 --> - Fixed a volume snapshot backup being discarded on a transient instance-manager connection error (for example a dial timeout from a brief pod-network disruption) during the finalize step, even when its snapshots were already provisioned; such network errors are now retried instead of treated as terminal. ([#11069](cloudnative-pg/cloudnative-pg#11069)) <!-- 1.29 1.28 1.25 --> - Fixed a replica switchover losing its `status.demotionToken` when a reconcile was requeued between storing the token and cleaning up the transition metadata (for example a cleanup patch failing against a flaky webhook); the empty no-change token is no longer patched back over the stored value. ([#11075](cloudnative-pg/cloudnative-pg#11075)) <!-- 1.29 1.28 1.25 --> - `cnpg` plugin: - Fixed `kubectl cnpg psql` on Windows, where execution relied on a Unix-only system call and failed with "not supported by windows"; Windows now launches `kubectl exec` as a child process. Contributed by [@Utkarsh-sharma47](https://github.com/Utkarsh-sharma47). ([#10972](cloudnative-pg/cloudnative-pg#10972)) <!-- 1.29 1.28 1.25 --> - Fixed an unbounded memory leak in `kubectl cnpg logs -f` on busy clusters, where a per-log-group timer was never released; timers are now reused across iterations. Contributed by [@Anand-240](https://github.com/Anand-240). ([#10976](cloudnative-pg/cloudnative-pg#10976)) <!-- 1.29 1.28 1.25 --> ##### Supported versions - Kubernetes 1.36, 1.35, and 1.34 - PostgreSQL 18, 17, 16, 15, and 14 - PostgreSQL 18.4 is the default image - PostgreSQL 14 support ends on November 12, 2026 --- ##### [\`v1.29.2\`](https://github.com/cloudnative-pg/cloudnative-pg/releases/tag/v1.29.2) **Release date:** Jun 29, 2026 ##### Important changes - Updated the deprecation notice for native (in-tree) Barman Cloud support to reflect that it will now be removed in CloudNativePG 1.31.0, rather than 1.30.0. Users are still encouraged to migrate to the Barman Cloud Plugin. ([#11083](cloudnative-pg/cloudnative-pg#11083)) <!-- 1.29 1.28 --> - The `cluster` reference is now immutable on the `Database`, `Pooler`, `Publication`, `Subscription`, and `ScheduledBackup` resources. Pointing one of these objects at a different cluster has no well-defined semantics and previously left the controllers in an inconsistent state; the update is now rejected at the API server via a CEL validation rule. ([#10743](cloudnative-pg/cloudnative-pg#10743)) <!-- 1.29 1.28 1.25 --> ##### Enhancements - Enabled `pg_upgrade` in-place major upgrades to PostgreSQL 19 or later for clusters that use Image Volume extensions, building on the extension-path support added to `pg_upgrade` in PostgreSQL 19. During the upgrade `Job`, the source- and target-version extension images are mounted side by side, so the old server keeps its libraries and a failed upgrade reverts cleanly. ([#10366](cloudnative-pg/cloudnative-pg#10366)) <!-- 1.29 --> - Added a label selector to the `Cluster` scale subresource (`status.selector`), making a `Cluster` a valid `targetRef` for the Vertical Pod Autoscaler (VPA) and Horizontal Pod Autoscaler (HPA), which can now map a `Cluster` to its instance pods. Contributed by [@sebv004](https://github.com/sebv004). ([#8996](cloudnative-pg/cloudnative-pg#8996)) <!-- 1.29 1.28 1.25 --> - The operator now emits a `Warning` `PrimaryStatusCheckFailed` event on the `Cluster` when the primary pod is `Ready` from the kubelet perspective but the operator's `/pg/status` check fails and failover is deferred, giving users visibility into the deferral via `kubectl describe cluster`. ([#10509](cloudnative-pg/cloudnative-pg#10509)) <!-- 1.29 1.28 1.25 --> - The operator now reloads a CNPG-i plugin automatically when its pods are rolled: it watches the `EndpointSlices` backing plugin `Services` and re-enqueues every cluster using the plugin once the new pods become `Ready`, so an upgraded plugin is picked up without waiting for the next resync. ([#10836](cloudnative-pg/cloudnative-pg#10836)) <!-- 1.29 1.28 1.25 --> ##### Security and Supply Chain - **`CVE-2026-55769` / `GHSA-x8c2-3p4r-v9r6`: `search_path` pinning on operator-issued connections**: a database owner could plant overloaded built-in operators in the `public` schema and alter the `search_path` so that operator introspection probes, running as the cluster superuser, resolved those overloads before `pg_catalog`, a `CWE-426` privilege-escalation chain (same class as `CVE-2018-1058`) that could lead to in-pod RCE via `COPY ... FROM PROGRAM`. The operator now pins `search_path = pg_catalog, public, pg_temp` on every pooled connection so it ships in the startup message and takes precedence over tenant-controlled defaults. ([#10774](cloudnative-pg/cloudnative-pg#10774), [`GHSA-x8c2-3p4r-v9r6`](GHSA-x8c2-3p4r-v9r6)) <!-- 1.29 1.28 1.25 --> - **`CVE-2026-55765` / `GHSA-w3gf-xc94-wvmj`: operator-side SCRAM-SHA-256 password encoding**: the operator now SCRAM-SHA-256 encodes cleartext role passwords before issuing `CREATE`/`ALTER ROLE ... PASSWORD`, so the literal PostgreSQL parses (and that extensions such as `pg_stat_statements` or `pgaudit` may capture) is the SCRAM verifier rather than the cleartext secret. Pre-hashed (MD5 or SCRAM) values are forwarded unchanged, and the per-Secret annotation `cnpg.io/passwordPassthrough: "enabled"` opts out. ([#10724](cloudnative-pg/cloudnative-pg#10724), [`GHSA-w3gf-xc94-wvmj`](GHSA-w3gf-xc94-wvmj)) <!-- 1.29 1.28 1.25 --> ##### Changes - Added support for Kubernetes 1.36. ([#10900](cloudnative-pg/cloudnative-pg#10900)) <!-- 1.29 1.28 --> - Updated the default PostgreSQL version to 18.4. ([#10719](cloudnative-pg/cloudnative-pg#10719)) <!-- 1.29 1.28 1.25 --> - Updated the Kubernetes versions used to test the operator on public cloud providers. ([#10720](cloudnative-pg/cloudnative-pg#10720), <!-- 1.29 1.28 1.25 --> [#10563](cloudnative-pg/cloudnative-pg#10563), <!-- 1.29 1.28 1.25 --> [#11033](cloudnative-pg/cloudnative-pg#11033)) <!-- 1.29 1.28 1.25 --> ##### Fixes - Fixed `spec.postgresql.parameters` accepting keys that are not valid PostgreSQL parameter names, which could inject arbitrary directives into `postgresql.conf`; key names are now validated by the webhook. ([#11029](cloudnative-pg/cloudnative-pg#11029)) <!-- 1.29 1.28 1.25 --> - Fixed declarative `Database`, `Publication`, and `Subscription` objects reporting a stale primary-side status forever after their cluster was demoted to a replica; the controller now re-checks the replica condition and watches the `Cluster` so a demotion is detected promptly. ([#10871](cloudnative-pg/cloudnative-pg#10871)) <!-- 1.29 1.28 1.25 --> - Fixed non-sequential pod names (for example `-1`, `-3`) caused by the instance serial counter being advanced before the corresponding `Job` and PVCs were created; the bump is now persisted only after those resources exist. ([#10491](cloudnative-pg/cloudnative-pg#10491)) <!-- 1.29 1.28 1.25 --> - Fixed a switchover deadlock when a WAL-archiver plugin was enabled on an existing cluster: with `primaryUpdateMethod: switchover` the primary could not be rolled out because a clean demotion needs the archiver sidecar that is still missing. The operator now recreates the primary Pod in place so the sidecar is injected and archiving resumes. The check also covers plugins that inject the archiver as a native sidecar (an init container with `restartPolicy: Always`), such as the Barman Cloud plugin. ([#11032](cloudnative-pg/cloudnative-pg#11032), <!-- 1.29 1.28 1.25 --> [#11059](cloudnative-pg/cloudnative-pg#11059)) <!-- 1.29 1.28 1.25 --> - Fixed a cluster staying in `Setting up primary` indefinitely when the instance-creation Job exhausted its backoff limit; the operator now detects the terminal Job failure and marks the cluster unrecoverable, naming the failed Job and pointing to its logs. ([#11035](cloudnative-pg/cloudnative-pg#11035)) <!-- 1.29 1.28 1.25 --> - Fixed deletion of a `Database`, `Publication`, or `Subscription` getting stuck in `Terminating` on a replica cluster, where the replica gate ran before the finalizer reconciler and the finalizer was never released. On a replica the PostgreSQL object is left to the primary cluster. ([#10853](cloudnative-pg/cloudnative-pg#10853)) <!-- 1.29 1.28 1.25 --> - Fixed a conflicting duplicate `Database` or `Subscription` with a `delete` reclaim policy dropping the PostgreSQL object owned by the surviving CR; the drop is now gated on a recorded reconciliation. ([#10870](cloudnative-pg/cloudnative-pg#10870)) <!-- 1.29 1.28 1.25 --> - Fixed the `postgres` superuser being left locked out after superuser access was disabled and then re-enabled, because the cached secret version was not invalidated and the password was never re-applied. Diagnosed by [@mhartmann-jaconi](https://github.com/mhartmann-jaconi). ([#10834](cloudnative-pg/cloudnative-pg#10834)) <!-- 1.29 1.28 1.25 --> - Fixed backups getting stuck in the `started` phase when the instance manager running them was restarted (for example by the in-place upgrade following an operator upgrade) before the backup reached `running`; the reconciliation is now rescheduled so the lost session is detected. ([#10859](cloudnative-pg/cloudnative-pg#10859)) <!-- 1.29 1.28 --> - Fixed `exec`/`attach` streaming to negotiate WebSocket with a SPDY fallback, restoring compatibility both with Kubernetes versions that have removed SPDY and with platforms such as OpenShift that reject WebSocket exec upgrades. Contributed by [@bartscheers](https://github.com/bartscheers). ([#10876](cloudnative-pg/cloudnative-pg#10876), [#10933](cloudnative-pg/cloudnative-pg#10933)) <!-- 1.29 --> - Fixed resource leaks when concurrent `Backup` objects raced: backups now run in strict creation-time order, so an already-executing backup is never preempted by a newer one and its replication slot and PostgreSQL session are no longer orphaned on the primary. Contributed by [@GabriFedi97](https://github.com/GabriFedi97). ([#10747](cloudnative-pg/cloudnative-pg#10747)) <!-- 1.29 1.28 1.25 --> - Fixed role reconciliation clearing the password on a PostgreSQL role when the referenced Secret could not be fetched; the role is now left untouched until the Secret becomes available, and per-action errors are aggregated for better visibility. ([#10053](cloudnative-pg/cloudnative-pg#10053)) <!-- 1.29 1.28 1.25 --> - Fixed a bootstrap failure where a metrics-exporter setup error (commonly a duplicate-key race with the controller) rolled back `streaming_replica` creation and wedged replica joins. The metrics-exporter step now runs in a separate transaction. Contributed by [@BlaiseAntony](https://github.com/BlaiseAntony). ([#10749](cloudnative-pg/cloudnative-pg#10749)) <!-- 1.29 1.28 1.25 --> - Fixed a `ScheduledBackup` controller loop that occurred when a `Backup` was created but its status patch never landed; the controller now adopts an existing `Backup` for the next iteration instead of looping on `AlreadyExists`. ([#10612](cloudnative-pg/cloudnative-pg#10612)) <!-- 1.29 1.28 1.25 --> - Fixed a nil-pointer panic when reconciling a `Pooler` whose `Cluster` has been deleted. ([#10667](cloudnative-pg/cloudnative-pg#10667)) <!-- 1.29 1.28 --> - Fixed bootstrap log handling so that all named log pipes (`postgres`, `postgres.csv`, and `postgres.json`) get consumers during `WithActiveInstance`, preventing regular files from being created in place of the named pipes. ([#10043](cloudnative-pg/cloudnative-pg#10043)) <!-- 1.29 1.28 1.25 --> - Fixed generation of invalid IPv6 URLs by wrapping the address in square brackets. Contributed by [@Infinoid](https://github.com/Infinoid). ([#10682](cloudnative-pg/cloudnative-pg#10682)) <!-- 1.29 1.28 1.25 --> - Fixed an external cluster plugin still being treated as active when its configuration set `enabled: false`. ([#10932](cloudnative-pg/cloudnative-pg#10932)) <!-- 1.29 1.28 1.25 --> - Fixed a race during bootstrap recovery from an object store where the restore job could read a stale `Cluster` (primary not yet recorded and timeline still unset) and have its `.history` files rejected by the split-brain guard. When this happened, recovery stopped at the base backup's timeline and silently dropped transactions committed on later timelines. History files are now allowed while the cluster timeline is unset. Contributed by [@dennispidun](https://github.com/dennispidun). ([#10818](cloudnative-pg/cloudnative-pg#10818)) <!-- 1.29 1.28 1.25 --> - Fixed a cache race during cluster creation when the server and client CA resolve to the same Secret (the default): a stale informer cache triggered a redundant `Create` that failed with `AlreadyExists` and could leave the cluster stuck in `Unable to create required cluster objects`. The operator now reuses the already-fetched CA Secret when the names match. ([#10989](cloudnative-pg/cloudnative-pg#10989)) <!-- 1.29 1.28 1.25 --> - Fixed the `pg_basebackup` bootstrap path overwriting or failing on a pre-existing `PGDATA` (for example after a replica Pod restart) by enforcing the same pre-flight directory check already applied by the other bootstrap methods; this also protects statically provisioned PVCs from being silently overwritten. ([#11006](cloudnative-pg/cloudnative-pg#11006)) <!-- 1.29 1.28 1.25 --> - Fixed the `Cluster` phase flapping between `Healthy` and a plugin-failure phase when a post-reconcile plugin hook returned an error; the `Healthy` phase is now registered as the last step of a successful reconciliation, so a loop that ends in a plugin error never reports `Healthy`. Contributed by [@GabriFedi97](https://github.com/GabriFedi97). ([#10421](cloudnative-pg/cloudnative-pg#10421)) <!-- 1.29 1.28 1.25 --> - Fixed stale certificate data and partial reads after an external server's Secret was rotated (for example a CA bundle shrinking from two certificates to one): the file is now written atomically, so libpq always reads either the old or the new value, never a mix. Contributed by [@Anand-240](https://github.com/Anand-240). ([#10975](cloudnative-pg/cloudnative-pg#10975)) <!-- 1.29 1.28 1.25 --> - Fixed plugin connectivity to use the plugin `Service` FQDN instead of its short name, avoiding failures when a cluster-level proxy is automatically injected into pods. Contributed by [@kdautrey](https://github.com/kdautrey). ([#10921](cloudnative-pg/cloudnative-pg#10921)) <!-- 1.29 1.28 1.25 --> - Fixed excessive operator log noise from the per-request `Cluster` create/update validation webhook messages, now logged at `debug` instead of `info`. ([#10984](cloudnative-pg/cloudnative-pg#10984)) <!-- 1.29 1.28 1.25 --> - Fixed a first-primary bootstrap deadlock where a status-patch conflict after the data PVC was created but before the initialization Job was started left the orphan Pending PVC counted as an instance, blocking the bootstrap gate; the PVC-state reconciler now recreates the bootstrap Job reusing the assigned serial. ([#11039](cloudnative-pg/cloudnative-pg#11039)) <!-- 1.29 1.28 1.25 --> - Fixed external cluster names and secret selector references being joined into filesystem paths without validation, letting a `..` component or path separator escape the external secrets directory when the instance manager dumps connection material; these values are now rejected at the validating webhook and re-checked at the write site. Reported by [@r0binak](https://github.com/r0binak). ([#11045](cloudnative-pg/cloudnative-pg#11045)) <!-- 1.29 1.28 1.25 --> - Fixed a backup getting stuck in `pending` forever: the concurrent-backup gate ran on every reconcile and could overwrite an already-completed phase written asynchronously by the instance manager. The gate now runs only while the backup phase is still unset or `pending`. ([#11056](cloudnative-pg/cloudnative-pg#11056)) <!-- 1.29 --> - Fixed a declarative `VolumeSnapshot` backup being permanently marked as failed when a stale cache made the operator re-create a snapshot it had already provisioned, failing with `AlreadyExists`. The operator now tolerates the collision when the existing snapshot carries this backup's label and adopts it; a collision with a foreign snapshot still surfaces as an error. ([#11071](cloudnative-pg/cloudnative-pg#11071)) <!-- 1.29 1.28 1.25 --> - Fixed a volume snapshot backup being discarded on a transient instance-manager connection error (for example a dial timeout from a brief pod-network disruption) during the finalize step, even when its snapshots were already provisioned; such network errors are now retried instead of treated as terminal. ([#11069](cloudnative-pg/cloudnative-pg#11069)) <!-- 1.29 1.28 1.25 --> - Fixed a replica switchover losing its `status.demotionToken` when a reconcile was requeued between storing the token and cleaning up the transition metadata (for example a cleanup patch failing against a flaky webhook); the empty no-change token is no longer patched back over the stored value. ([#11075](cloudnative-pg/cloudnative-pg#11075)) <!-- 1.29 1.28 1.25 --> - `cnpg` plugin: - Fixed `kubectl cnpg psql` on Windows, where execution relied on a Unix-only system call and failed with "not supported by windows"; Windows now launches `kubectl exec` as a child process. Contributed by [@Utkarsh-sharma47](https://github.com/Utkarsh-sharma47). ([#10972](cloudnative-pg/cloudnative-pg#10972)) <!-- 1.29 1.28 1.25 --> - Fixed an unbounded memory leak in `kubectl cnpg logs -f` on busy clusters, where a per-log-group timer was never released; timers are now reused across iterations. Contributed by [@Anand-240](https://github.com/Anand-240). ([#10976](cloudnative-pg/cloudnative-pg#10976)) <!-- 1.29 1.28 1.25 -->
When admission webhooks are not installed (or configured to ignore failures), resources previously skipped defaulting and validation entirely, leaving the operator to reconcile invalid or incomplete specs.
Resources are now defaulted and validated during reconciliation as a fallback. When a resource is missing defaults, the operator either applies them directly or waits for the webhook to do so. When validation fails, the offending fields are surfaced in the resource status, and reconciliation stops, with full details in the operator logs, so users can see what was rejected even if the webhook never ran.
Webhook serving-certificate management is also decoupled from CA-bundle injection: the new
MANAGE_WEBHOOK_CONFIGURATIONSoption (enabled by default) lets webhook configurations be managed externally, e.g., by cert-manager's CA injector or by GitOps, while the operator continues rotating the serving cert.Closes #10942