Skip to content

fix(slots): cleanup replication slots on replicas when features disabled - #9381

Merged
leonardoce merged 4 commits into
mainfrom
dev/9380
Dec 15, 2025
Merged

leonardoce merged 4 commits into
mainfrom
dev/9380

Conversation

@mnencia

@mnencia mnencia commented Dec 7, 2025 •

Copy link
Copy Markdown
Member

When both replicationSlots.highAvailability.enabled and replicationSlots.synchronizeReplicas.enabled are disabled, the Replicator now performs cleanup by calling reconcile() before continuing the loop, ensuring user-defined replication slots are properly removed from replica pods.

This change also adds webhook validation to prevent users from removing
the synchronizeReplicas configuration section while it is still enabled.
This ensures that cleanup logic can properly execute before the
configuration is removed, preventing orphaned replication slots on
replicas.

Closes #9380

TODO

  • document the known limitations: when excludedPatterns is modified the operator isn't able to compare the difference with the previous this my cause some lingering rep slots

@mnencia
mnencia requested a review from a team December 7, 2025 10:53
@dosubot dosubot Bot added the size:XS This PR changes 0-9 lines, ignoring generated files. label Dec 7, 2025
@cnpg-bot cnpg-bot added backport-requested ◀️ This pull request should be backported to all supported releases release-1.25 labels Dec 7, 2025
@github-actions

github-actions Bot commented Dec 7, 2025

Copy link
Copy Markdown
Contributor

❗ By default, the pull request is configured to backport to all release branches.

  • To stop backporting this pr, remove the label: backport-requested ◀️ or add the label 'do not backport'
  • To stop backporting this pr to a certain release branch, remove the specific branch label: release-x.y

@dosubot dosubot Bot added bug 🐛 Something isn't working ok to merge 👌 This PR can be merged labels Dec 7, 2025
@mnencia

mnencia commented Dec 7, 2025

Copy link
Copy Markdown
Member Author

/test

@github-actions

github-actions Bot commented Dec 7, 2025

Copy link
Copy Markdown
Contributor

@mnencia mnencia removed the ok to merge 👌 This PR can be merged label Dec 7, 2025
@cnpg-bot cnpg-bot added the ok to merge 👌 This PR can be merged label Dec 7, 2025
@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:XS This PR changes 0-9 lines, ignoring generated files. labels Dec 10, 2025
@armru

armru commented Dec 10, 2025

Copy link
Copy Markdown
Member

/test limit=local

@github-actions

Copy link
Copy Markdown
Contributor

@dosubot dosubot Bot added the lgtm This PR has been approved by a maintainer label Dec 10, 2025
Copilot AI review requested due to automatic review settings December 11, 2025 09:36

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes cleanup of user-defined replication slots on replica pods when replication slot features are disabled. It adds webhook validation to prevent configuration removal while features are enabled and ensures cleanup reconciliation runs before the replicator continues its loop.

Key Changes:

  • Modified replication slot runner to execute cleanup reconciliation when features are disabled
  • Added webhook validation preventing removal of synchronizeReplicas configuration while enabled
  • Enhanced test coverage for new synchronizeReplicas validation rules

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
internal/management/controller/slots/runner/runner.go Added cleanup reconciliation call when replication features are disabled to properly remove user-defined slots from replicas
internal/webhook/v1/cluster_webhook.go Added validation to prevent removing synchronizeReplicas configuration section while the feature is enabled, ensuring cleanup can execute properly
internal/webhook/v1/cluster_webhook_test.go Added comprehensive test cases for new synchronizeReplicas validation rules, including scenarios for removing configuration sections and disabling features

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread internal/webhook/v1/cluster_webhook.go Resolved
@mnencia
mnencia force-pushed the dev/9380 branch 2 times, most recently from 8fe975a to a49b60a Compare December 11, 2025 10:04
mnencia and others added 3 commits December 15, 2025 13:18
When both replicationSlots.highAvailability.enabled and
replicationSlots.synchronizeReplicas.enabled are disabled, the
Replicator now performs cleanup by calling reconcile() before
continuing the loop, ensuring user-defined replication slots
are properly removed from replica pods.

Closes #9380

Signed-off-by: Marco Nenciarini <[email protected]>
This change also adds webhook validation to prevent users from removing
the synchronizeReplicas configuration section while it is still enabled.
This ensures that cleanup logic can properly execute before the
configuration is removed, preventing orphaned replication slots on
replicas.

Signed-off-by: Armando Ruocco <[email protected]>
Signed-off-by: Marco Nenciarini <[email protected]>
Signed-off-by: Marco Nenciarini <[email protected]>
@leonardoce

leonardoce commented Dec 15, 2025 •

Copy link
Copy Markdown
Contributor

After a chat with @armru we found that the improved webhook doesn't really check prevent the mistake from happening because it will run after the defaulting webhook, and the defaulting webhook will just set the cluster.spec.replicationSlots.highAvailability and cluster.spec.replicationSlots.synchronizeReplicas if they are not set.

The condition in https://github.com/cloudnative-pg/cloudnative-pg/pull/9381/files#diff-30bf0aa461c01460908ae409672d7d5be2a1d9065620671ea75ef81a5e9cec03R2311 won't ever be met because of that.

To cover this, we need to keep the old version of the cluster.spec.replicationSlots.highAvailability stanza in the cluster status to clean up only the replication slots that are managed by the operator. This will be addressed in a subsequent commit.

@leonardoce
leonardoce merged commit b7c9690 into main Dec 15, 2025
31 of 34 checks passed
@leonardoce
leonardoce deleted the dev/9380 branch December 15, 2025 15:15
cnpg-bot pushed a commit that referenced this pull request Dec 15, 2025
…led (#9381)

When both replicationSlots.highAvailability.enabled and
replicationSlots.synchronizeReplicas.enabled are disabled, the
Replicator now performs cleanup by calling reconcile() before continuing
the loop, ensuring user-defined replication slots are properly removed
from replica pods.

This change also adds webhook validation to prevent users from removing
the synchronizeReplicas configuration section while it is still enabled.
This ensures that cleanup logic can properly execute before the
configuration is removed, preventing orphaned replication slots on
replicas.

Closes #9380

Signed-off-by: Marco Nenciarini <[email protected]>
Signed-off-by: Armando Ruocco <[email protected]>
Co-authored-by: Armando Ruocco <[email protected]>
(cherry picked from commit b7c9690)
cnpg-bot pushed a commit that referenced this pull request Dec 15, 2025
…led (#9381)

When both replicationSlots.highAvailability.enabled and
replicationSlots.synchronizeReplicas.enabled are disabled, the
Replicator now performs cleanup by calling reconcile() before continuing
the loop, ensuring user-defined replication slots are properly removed
from replica pods.

This change also adds webhook validation to prevent users from removing
the synchronizeReplicas configuration section while it is still enabled.
This ensures that cleanup logic can properly execute before the
configuration is removed, preventing orphaned replication slots on
replicas.

Closes #9380

Signed-off-by: Marco Nenciarini <[email protected]>
Signed-off-by: Armando Ruocco <[email protected]>
Co-authored-by: Armando Ruocco <[email protected]>
(cherry picked from commit b7c9690)
leonardoce pushed a commit that referenced this pull request Dec 15, 2025
…led (#9381)

When both replicationSlots.highAvailability.enabled and
replicationSlots.synchronizeReplicas.enabled are disabled, the
Replicator now performs cleanup by calling reconcile() before continuing
the loop, ensuring user-defined replication slots are properly removed
from replica pods.

This change also adds webhook validation to prevent users from removing
the synchronizeReplicas configuration section while it is still enabled.
This ensures that cleanup logic can properly execute before the
configuration is removed, preventing orphaned replication slots on
replicas.

Closes #9380

Signed-off-by: Marco Nenciarini <[email protected]>
Signed-off-by: Armando Ruocco <[email protected]>
Co-authored-by: Armando Ruocco <[email protected]>
(cherry picked from commit b7c9690)
mnencia added a commit that referenced this pull request Jan 20, 2026
…led (#9381)

When both replicationSlots.highAvailability.enabled and
replicationSlots.synchronizeReplicas.enabled are disabled, the
Replicator now performs cleanup by calling reconcile() before continuing
the loop, ensuring user-defined replication slots are properly removed
from replica pods.

This change also adds webhook validation to prevent users from removing
the synchronizeReplicas configuration section while it is still enabled.
This ensures that cleanup logic can properly execute before the
configuration is removed, preventing orphaned replication slots on
replicas.

Closes #9380

Signed-off-by: Marco Nenciarini <[email protected]>
Signed-off-by: Armando Ruocco <[email protected]>
Co-authored-by: Armando Ruocco <[email protected]>
(cherry picked from commit b7c9690)
renovate Bot added a commit to sdwilsh/ansible-playbooks that referenced this pull request Mar 26, 2026
##### [\`v1.28.1\`](https://github.com/cloudnative-pg/cloudnative-pg/releases/tag/v1.28.1)

**Release date:** Feb 5, 2026

##### Enhancements

- Added support for Azure's `DefaultAzureCredential` authentication mechanism for backup and recovery operations. This can be enabled by setting `azureCredentials.useDefaultAzureCredentials: true` in the backup configuration, simplifying authentication in Azure environments without requiring explicit storage account keys or SAS tokens. ([#9468](cloudnative-pg/cloudnative-pg#9468)) <!-- 1.27 1.25 -->

##### Fixes

- Fixed validation of PostgreSQL extension names containing underscores (e.g., `pg_partman`, `pg_ivm`). Extension names with underscores are automatically sanitized to use hyphens for Kubernetes volume names while preserving the original name in mount paths. Webhook validation prevents naming conflicts after sanitization. Contributed by [@shusaan](https://github.com/shusaan). ([#9386](cloudnative-pg/cloudnative-pg#9386)) <!-- 1.27 -->

- Fixed a critical issue where the `TimelineID` in the cluster status was not reset to 1 after a major version upgrade. Because `pg_upgrade` initializes a new timeline, keeping the old ID (e.g., timeline 2) caused replicas to attempt to restore incompatible history files from object storage, leading to fatal "requested timeline is not a child of this server's history" errors. ([#9830](cloudnative-pg/cloudnative-pg#9830)) <!-- 1.27 -->

- Fixed an issue where stale TLS status fields in the `Pooler` were not cleared after being removed from the specification. This was particularly critical when upgrading to v1.28.0, where the `ServerTLS` field was repurposed, causing PgBouncer to use incorrect certificates and resulting in "unsupported certificate" errors that blocked all application connectivity. The operator now explicitly clears `ServerCA`, `ClientCA`, `ClientTLS`, and `ServerTLS` status fields when they are no longer configured. ([#9397](cloudnative-pg/cloudnative-pg#9397))

- Fixed a bug where replicas could enter a crash-loop by attempting to download timeline history files from future timelines. This occurred when stale files remained in the WAL archive from a previous cluster life, and replicas would incorrectly try to fetch them during recovery. ([#9650](cloudnative-pg/cloudnative-pg#9650)) <!-- 1.27 1.25 -->

- Fixed a race condition in `replica_cluster` setups during designated primary transitions, preventing transient "no primary" states in the replica cluster. ([#9601](cloudnative-pg/cloudnative-pg#9601)) <!-- 1.27 1.25 -->

- The backup controller now uses the unique instance session ID to detect instance manager restarts. This prevents the operator from incorrectly assuming a backup is still progressing if the underlying container has crashed and restarted, which previously led to orphaned backup objects. ([#9370](cloudnative-pg/cloudnative-pg#9370)) <!-- 1.27 -->

- Fixed a validation gap in Azure object store configurations where the `storageAccount` was not required when using explicit credentials (such as a storage key or SAS token). The operator now enforces that a storage account name is provided in these cases and that `connectionString` is mutually exclusive with other authentication parameters. ([#9604](cloudnative-pg/cloudnative-pg#9604)) <!-- 1.27 1.25 -->

- Optimized the deletion path so the operator begins cleaning up resources immediately when a cluster is marked for deletion. This significantly reduces the time a cluster remains in `Terminating` status while waiting for internal reconciliation loops. ([#9555](cloudnative-pg/cloudnative-pg#9555)) <!-- 1.27 1.25 -->

- Fixed an issue where replication slots were not properly dropped from replicas when the feature was disabled or the cluster was reconfigured. This ensures that unused slots do not cause WAL build-up on the primary. ([#9381](cloudnative-pg/cloudnative-pg#9381)) <!-- 1.27 1.25 -->

- Fixed an issue where `imagePullSecrets` were not added to the `ServiceAccount` created for the `Pooler`. Previously, these secrets were applied to the Deployment but not the SA, which caused image pull failures in restricted environments using certain security policies. ([#9427](cloudnative-pg/cloudnative-pg#9427)) <!-- 1.27 1.25 -->

- Added a check to verify ownership before the operator deletes a `PodMonitor`. This prevents the operator from accidentally deleting manually managed monitoring resources that happen to share a name with expected CNPG resources. Contributed by [@juliamertz](https://github.com/juliamertz). ([#9340](cloudnative-pg/cloudnative-pg#9340)) <!-- 1.27 1.25 -->

- Fixed a bug where `pg_stat_archiver` metrics would continue to report stale data on standby instances after a switchover. The exporter now skips these metrics on standbys, as PostgreSQL only provides valid archiver stats on the primary. ([#9411](cloudnative-pg/cloudnative-pg#9411)) <!-- 1.27 1.25 -->

- Clarified the interpretation of timestamp formats for recovery `targetTime`. Timestamps provided without an explicit timezone are now consistently interpreted as UTC. Contributed by [@pchovelon](https://github.com/pchovelon). ([#8937](cloudnative-pg/cloudnative-pg#8937)) <!-- 1.27 1.25 -->

- Fixed backup status updates to prevent "resource has been modified" errors during concurrent updates. ([#9551](cloudnative-pg/cloudnative-pg#9551)) <!-- 1.27 1.25 -->

- Fixed event reporting to use the correct pod name when a backup pod is not found. ([#9552](cloudnative-pg/cloudnative-pg#9552)) <!-- 1.27 1.25 -->

- Improved performance of scheduled backup operations for clusters with a very high number of historical backups. ([#9489](cloudnative-pg/cloudnative-pg#9489)) <!-- 1.27 1.25 -->

- Fixed error handling when removing finalizers on `Database` objects. ([#9431](cloudnative-pg/cloudnative-pg#9431)) <!-- 1.27 1.25 -->

- `cnpg` plugin:

  - Updated the `status` command to display "Disabled" when the `skipWalArchiving` annotation is present on a cluster. This replaces confusing "starting up" or "unknown" states when WAL archiving is intentionally bypassed. ([#9709](cloudnative-pg/cloudnative-pg#9709)) <!-- 1.27 1.25 -->

  - Fixed the `logs --follow` command to continue polling for new pods instead of exiting prematurely when all current log streams complete. ([#9599](cloudnative-pg/cloudnative-pg#9599)) <!-- 1.27 1.25 -->
sdwilsh pushed a commit to sdwilsh/ansible-playbooks that referenced this pull request Mar 26, 2026
##### [\`v1.28.1\`](https://github.com/cloudnative-pg/cloudnative-pg/releases/tag/v1.28.1)

**Release date:** Feb 5, 2026

##### Enhancements

- Added support for Azure's `DefaultAzureCredential` authentication mechanism for backup and recovery operations. This can be enabled by setting `azureCredentials.useDefaultAzureCredentials: true` in the backup configuration, simplifying authentication in Azure environments without requiring explicit storage account keys or SAS tokens. ([#9468](cloudnative-pg/cloudnative-pg#9468)) <!-- 1.27 1.25 -->

##### Fixes

- Fixed validation of PostgreSQL extension names containing underscores (e.g., `pg_partman`, `pg_ivm`). Extension names with underscores are automatically sanitized to use hyphens for Kubernetes volume names while preserving the original name in mount paths. Webhook validation prevents naming conflicts after sanitization. Contributed by [@shusaan](https://github.com/shusaan). ([#9386](cloudnative-pg/cloudnative-pg#9386)) <!-- 1.27 -->

- Fixed a critical issue where the `TimelineID` in the cluster status was not reset to 1 after a major version upgrade. Because `pg_upgrade` initializes a new timeline, keeping the old ID (e.g., timeline 2) caused replicas to attempt to restore incompatible history files from object storage, leading to fatal "requested timeline is not a child of this server's history" errors. ([#9830](cloudnative-pg/cloudnative-pg#9830)) <!-- 1.27 -->

- Fixed an issue where stale TLS status fields in the `Pooler` were not cleared after being removed from the specification. This was particularly critical when upgrading to v1.28.0, where the `ServerTLS` field was repurposed, causing PgBouncer to use incorrect certificates and resulting in "unsupported certificate" errors that blocked all application connectivity. The operator now explicitly clears `ServerCA`, `ClientCA`, `ClientTLS`, and `ServerTLS` status fields when they are no longer configured. ([#9397](cloudnative-pg/cloudnative-pg#9397))

- Fixed a bug where replicas could enter a crash-loop by attempting to download timeline history files from future timelines. This occurred when stale files remained in the WAL archive from a previous cluster life, and replicas would incorrectly try to fetch them during recovery. ([#9650](cloudnative-pg/cloudnative-pg#9650)) <!-- 1.27 1.25 -->

- Fixed a race condition in `replica_cluster` setups during designated primary transitions, preventing transient "no primary" states in the replica cluster. ([#9601](cloudnative-pg/cloudnative-pg#9601)) <!-- 1.27 1.25 -->

- The backup controller now uses the unique instance session ID to detect instance manager restarts. This prevents the operator from incorrectly assuming a backup is still progressing if the underlying container has crashed and restarted, which previously led to orphaned backup objects. ([#9370](cloudnative-pg/cloudnative-pg#9370)) <!-- 1.27 -->

- Fixed a validation gap in Azure object store configurations where the `storageAccount` was not required when using explicit credentials (such as a storage key or SAS token). The operator now enforces that a storage account name is provided in these cases and that `connectionString` is mutually exclusive with other authentication parameters. ([#9604](cloudnative-pg/cloudnative-pg#9604)) <!-- 1.27 1.25 -->

- Optimized the deletion path so the operator begins cleaning up resources immediately when a cluster is marked for deletion. This significantly reduces the time a cluster remains in `Terminating` status while waiting for internal reconciliation loops. ([#9555](cloudnative-pg/cloudnative-pg#9555)) <!-- 1.27 1.25 -->

- Fixed an issue where replication slots were not properly dropped from replicas when the feature was disabled or the cluster was reconfigured. This ensures that unused slots do not cause WAL build-up on the primary. ([#9381](cloudnative-pg/cloudnative-pg#9381)) <!-- 1.27 1.25 -->

- Fixed an issue where `imagePullSecrets` were not added to the `ServiceAccount` created for the `Pooler`. Previously, these secrets were applied to the Deployment but not the SA, which caused image pull failures in restricted environments using certain security policies. ([#9427](cloudnative-pg/cloudnative-pg#9427)) <!-- 1.27 1.25 -->

- Added a check to verify ownership before the operator deletes a `PodMonitor`. This prevents the operator from accidentally deleting manually managed monitoring resources that happen to share a name with expected CNPG resources. Contributed by [@juliamertz](https://github.com/juliamertz). ([#9340](cloudnative-pg/cloudnative-pg#9340)) <!-- 1.27 1.25 -->

- Fixed a bug where `pg_stat_archiver` metrics would continue to report stale data on standby instances after a switchover. The exporter now skips these metrics on standbys, as PostgreSQL only provides valid archiver stats on the primary. ([#9411](cloudnative-pg/cloudnative-pg#9411)) <!-- 1.27 1.25 -->

- Clarified the interpretation of timestamp formats for recovery `targetTime`. Timestamps provided without an explicit timezone are now consistently interpreted as UTC. Contributed by [@pchovelon](https://github.com/pchovelon). ([#8937](cloudnative-pg/cloudnative-pg#8937)) <!-- 1.27 1.25 -->

- Fixed backup status updates to prevent "resource has been modified" errors during concurrent updates. ([#9551](cloudnative-pg/cloudnative-pg#9551)) <!-- 1.27 1.25 -->

- Fixed event reporting to use the correct pod name when a backup pod is not found. ([#9552](cloudnative-pg/cloudnative-pg#9552)) <!-- 1.27 1.25 -->

- Improved performance of scheduled backup operations for clusters with a very high number of historical backups. ([#9489](cloudnative-pg/cloudnative-pg#9489)) <!-- 1.27 1.25 -->

- Fixed error handling when removing finalizers on `Database` objects. ([#9431](cloudnative-pg/cloudnative-pg#9431)) <!-- 1.27 1.25 -->

- `cnpg` plugin:

  - Updated the `status` command to display "Disabled" when the `skipWalArchiving` annotation is present on a cluster. This replaces confusing "starting up" or "unknown" states when WAL archiving is intentionally bypassed. ([#9709](cloudnative-pg/cloudnative-pg#9709)) <!-- 1.27 1.25 -->

  - Fixed the `logs --follow` command to continue polling for new pods instead of exiting prematurely when all current log streams complete. ([#9599](cloudnative-pg/cloudnative-pg#9599)) <!-- 1.27 1.25 -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-requested ◀️ This pull request should be backported to all supported releases bug 🐛 Something isn't working lgtm This PR has been approved by a maintainer ok to merge 👌 This PR can be merged release-1.28 size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Replication slots not cleaned up on replicas when both HA and synchronizeReplicas are disabled

5 participants