Skip to content

fix(backup): use instance session ID to detect instance manager restarts - #9370

Merged
mnencia merged 7 commits into
mainfrom
dev/fix-backup-stuck
Jan 16, 2026
Merged

mnencia merged 7 commits into
mainfrom
dev/fix-backup-stuck

Conversation

@armru

@armru armru commented Dec 4, 2025 •

Copy link
Copy Markdown
Member

This fix addresses an issue where backups would hang indefinitely during operator releases. When the operator and barman plugin were updated simultaneously, the backup goroutine running in the instance manager would be killed (due to the instance manager binary being replaced via exec()), but the operator would still think the backup was running because the container ID hadn't changed.

The fix introduces a deterministic mechanism to detect instance manager restarts by tracking the SessionID

@armru
armru requested a review from a team December 4, 2025 09:46
@armru armru added no-issue do not backport This PR must not be backported - it will be in the next minor release labels Dec 4, 2025
@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. bug 🐛 Something isn't working ok to merge 👌 This PR can be merged labels Dec 4, 2025
@armru

armru commented Dec 4, 2025

Copy link
Copy Markdown
Member Author

/test

@github-actions

github-actions Bot commented Dec 4, 2025

Copy link
Copy Markdown
Contributor

@armru
armru force-pushed the dev/fix-backup-stuck branch from 1d9f291 to e2ceb17 Compare December 4, 2025 10:27
@armru
armru requested a review from jsilvela as a code owner December 4, 2025 10:27
@armru

armru commented Dec 4, 2025

Copy link
Copy Markdown
Member Author

/test

@github-actions

github-actions Bot commented Dec 4, 2025

Copy link
Copy Markdown
Contributor

@armru
armru force-pushed the dev/fix-backup-stuck branch from e2ceb17 to 30d2b47 Compare December 16, 2025 12:39
@armru armru changed the title fix(backup): detect instance manager hash change fix(backup): use instance session ID to detect instance manager restarts Dec 16, 2025
@armru

armru commented Dec 16, 2025

Copy link
Copy Markdown
Member Author

/test

@github-actions

Copy link
Copy Markdown
Contributor

@mnencia
mnencia force-pushed the dev/fix-backup-stuck branch from 6568549 to eea3f0b Compare December 22, 2025 13:15
@mnencia

mnencia commented Dec 22, 2025

Copy link
Copy Markdown
Member

/test

@github-actions

Copy link
Copy Markdown
Contributor

@dosubot dosubot Bot added the lgtm This PR has been approved by a maintainer label Dec 22, 2025
@jbattiato
jbattiato force-pushed the dev/fix-backup-stuck branch from cc85f68 to ac94a7c Compare December 24, 2025 17:35
@mnencia
mnencia force-pushed the dev/fix-backup-stuck branch 2 times, most recently from 36534cc to 30ee553 Compare January 15, 2026 13:08
@mnencia

mnencia commented Jan 15, 2026

Copy link
Copy Markdown
Member

/test ft=backup-restore

@github-actions

Copy link
Copy Markdown
Contributor

armru and others added 3 commits January 16, 2026 12:59
This fix addresses an issue where backups would hang indefinitely
during operator releases. When the operator and barman plugin were updated
simultaneously, the backup goroutine running in the instance manager would
be killed (due to the instance manager binary being replaced via exec()),
but the operator would still think the backup was running because the
container ID hadn't changed.

The fix introduces a deterministic mechanism to detect instance manager
restarts by tracking the ExecutableHash:

- Added `ExecutableHash` field to `InstanceID` struct in backup status
- When a backup starts, the instance manager's ExecutableHash is stored
- On each reconcile, the stored hash is compared with the current hash
- If the hashes differ, the instance manager was restarted/upgraded and
  the backup process is dead, marking the backup as FAILED

Signed-off-by: Armando Ruocco <[email protected]>
This commit improves the detection of instance manager restarts during
backup operations by introducing an instance session ID instead of relying
on the executable hash alone.

Signed-off-by: Armando Ruocco <[email protected]>
Remove fallback that returned partial status without SessionID.
Return retriable error when status is unavailable to ensure
SessionID is always present for restart detection.

Signed-off-by: Marco Nenciarini <[email protected]>
Remove the guard condition that prevented isInstanceManagerRestarted()
from being called when SessionID is empty. The function itself correctly
handles the empty SessionID case by detecting operator upgrades during
running backups.

Signed-off-by: Marco Nenciarini <[email protected]>
@jbattiato
jbattiato force-pushed the dev/fix-backup-stuck branch from c6b757d to 82d0789 Compare January 16, 2026 11:59
@mnencia mnencia added backport-requested ◀️ This pull request should be backported to all supported releases release-1.25 release-1.28 and removed do not backport This PR must not be backported - it will be in the next minor release labels Jan 16, 2026
@mnencia
mnencia merged commit 102d785 into main Jan 16, 2026
44 checks passed
@mnencia
mnencia deleted the dev/fix-backup-stuck branch January 16, 2026 12:59
mnencia added a commit that referenced this pull request Jan 16, 2026
…rts (#9370)

This fix addresses an issue where backups would hang indefinitely during
operator releases. When the operator and barman plugin were updated
simultaneously, the backup goroutine running in the instance manager
would be killed (due to the instance manager binary being replaced via
exec()), but the operator would still think the backup was running
because the container ID hadn't changed.

The fix introduces a deterministic mechanism to detect instance manager
restarts by tracking the SessionID

Signed-off-by: Armando Ruocco <[email protected]>
Signed-off-by: Marco Nenciarini <[email protected]>
Co-authored-by: Marco Nenciarini <[email protected]>
(cherry picked from commit 102d785)
mnencia added a commit that referenced this pull request Jan 16, 2026
…rts (#9370)

This fix addresses an issue where backups would hang indefinitely during
operator releases. When the operator and barman plugin were updated
simultaneously, the backup goroutine running in the instance manager
would be killed (due to the instance manager binary being replaced via
exec()), but the operator would still think the backup was running
because the container ID hadn't changed.

The fix introduces a deterministic mechanism to detect instance manager
restarts by tracking the SessionID

Signed-off-by: Armando Ruocco <[email protected]>
Signed-off-by: Marco Nenciarini <[email protected]>
Co-authored-by: Marco Nenciarini <[email protected]>
(cherry picked from commit 102d785)
renovate Bot added a commit to sdwilsh/ansible-playbooks that referenced this pull request Mar 26, 2026
##### [\`v1.28.1\`](https://github.com/cloudnative-pg/cloudnative-pg/releases/tag/v1.28.1)

**Release date:** Feb 5, 2026

##### Enhancements

- Added support for Azure's `DefaultAzureCredential` authentication mechanism for backup and recovery operations. This can be enabled by setting `azureCredentials.useDefaultAzureCredentials: true` in the backup configuration, simplifying authentication in Azure environments without requiring explicit storage account keys or SAS tokens. ([#9468](cloudnative-pg/cloudnative-pg#9468)) <!-- 1.27 1.25 -->

##### Fixes

- Fixed validation of PostgreSQL extension names containing underscores (e.g., `pg_partman`, `pg_ivm`). Extension names with underscores are automatically sanitized to use hyphens for Kubernetes volume names while preserving the original name in mount paths. Webhook validation prevents naming conflicts after sanitization. Contributed by [@shusaan](https://github.com/shusaan). ([#9386](cloudnative-pg/cloudnative-pg#9386)) <!-- 1.27 -->

- Fixed a critical issue where the `TimelineID` in the cluster status was not reset to 1 after a major version upgrade. Because `pg_upgrade` initializes a new timeline, keeping the old ID (e.g., timeline 2) caused replicas to attempt to restore incompatible history files from object storage, leading to fatal "requested timeline is not a child of this server's history" errors. ([#9830](cloudnative-pg/cloudnative-pg#9830)) <!-- 1.27 -->

- Fixed an issue where stale TLS status fields in the `Pooler` were not cleared after being removed from the specification. This was particularly critical when upgrading to v1.28.0, where the `ServerTLS` field was repurposed, causing PgBouncer to use incorrect certificates and resulting in "unsupported certificate" errors that blocked all application connectivity. The operator now explicitly clears `ServerCA`, `ClientCA`, `ClientTLS`, and `ServerTLS` status fields when they are no longer configured. ([#9397](cloudnative-pg/cloudnative-pg#9397))

- Fixed a bug where replicas could enter a crash-loop by attempting to download timeline history files from future timelines. This occurred when stale files remained in the WAL archive from a previous cluster life, and replicas would incorrectly try to fetch them during recovery. ([#9650](cloudnative-pg/cloudnative-pg#9650)) <!-- 1.27 1.25 -->

- Fixed a race condition in `replica_cluster` setups during designated primary transitions, preventing transient "no primary" states in the replica cluster. ([#9601](cloudnative-pg/cloudnative-pg#9601)) <!-- 1.27 1.25 -->

- The backup controller now uses the unique instance session ID to detect instance manager restarts. This prevents the operator from incorrectly assuming a backup is still progressing if the underlying container has crashed and restarted, which previously led to orphaned backup objects. ([#9370](cloudnative-pg/cloudnative-pg#9370)) <!-- 1.27 -->

- Fixed a validation gap in Azure object store configurations where the `storageAccount` was not required when using explicit credentials (such as a storage key or SAS token). The operator now enforces that a storage account name is provided in these cases and that `connectionString` is mutually exclusive with other authentication parameters. ([#9604](cloudnative-pg/cloudnative-pg#9604)) <!-- 1.27 1.25 -->

- Optimized the deletion path so the operator begins cleaning up resources immediately when a cluster is marked for deletion. This significantly reduces the time a cluster remains in `Terminating` status while waiting for internal reconciliation loops. ([#9555](cloudnative-pg/cloudnative-pg#9555)) <!-- 1.27 1.25 -->

- Fixed an issue where replication slots were not properly dropped from replicas when the feature was disabled or the cluster was reconfigured. This ensures that unused slots do not cause WAL build-up on the primary. ([#9381](cloudnative-pg/cloudnative-pg#9381)) <!-- 1.27 1.25 -->

- Fixed an issue where `imagePullSecrets` were not added to the `ServiceAccount` created for the `Pooler`. Previously, these secrets were applied to the Deployment but not the SA, which caused image pull failures in restricted environments using certain security policies. ([#9427](cloudnative-pg/cloudnative-pg#9427)) <!-- 1.27 1.25 -->

- Added a check to verify ownership before the operator deletes a `PodMonitor`. This prevents the operator from accidentally deleting manually managed monitoring resources that happen to share a name with expected CNPG resources. Contributed by [@juliamertz](https://github.com/juliamertz). ([#9340](cloudnative-pg/cloudnative-pg#9340)) <!-- 1.27 1.25 -->

- Fixed a bug where `pg_stat_archiver` metrics would continue to report stale data on standby instances after a switchover. The exporter now skips these metrics on standbys, as PostgreSQL only provides valid archiver stats on the primary. ([#9411](cloudnative-pg/cloudnative-pg#9411)) <!-- 1.27 1.25 -->

- Clarified the interpretation of timestamp formats for recovery `targetTime`. Timestamps provided without an explicit timezone are now consistently interpreted as UTC. Contributed by [@pchovelon](https://github.com/pchovelon). ([#8937](cloudnative-pg/cloudnative-pg#8937)) <!-- 1.27 1.25 -->

- Fixed backup status updates to prevent "resource has been modified" errors during concurrent updates. ([#9551](cloudnative-pg/cloudnative-pg#9551)) <!-- 1.27 1.25 -->

- Fixed event reporting to use the correct pod name when a backup pod is not found. ([#9552](cloudnative-pg/cloudnative-pg#9552)) <!-- 1.27 1.25 -->

- Improved performance of scheduled backup operations for clusters with a very high number of historical backups. ([#9489](cloudnative-pg/cloudnative-pg#9489)) <!-- 1.27 1.25 -->

- Fixed error handling when removing finalizers on `Database` objects. ([#9431](cloudnative-pg/cloudnative-pg#9431)) <!-- 1.27 1.25 -->

- `cnpg` plugin:

  - Updated the `status` command to display "Disabled" when the `skipWalArchiving` annotation is present on a cluster. This replaces confusing "starting up" or "unknown" states when WAL archiving is intentionally bypassed. ([#9709](cloudnative-pg/cloudnative-pg#9709)) <!-- 1.27 1.25 -->

  - Fixed the `logs --follow` command to continue polling for new pods instead of exiting prematurely when all current log streams complete. ([#9599](cloudnative-pg/cloudnative-pg#9599)) <!-- 1.27 1.25 -->
sdwilsh pushed a commit to sdwilsh/ansible-playbooks that referenced this pull request Mar 26, 2026
##### [\`v1.28.1\`](https://github.com/cloudnative-pg/cloudnative-pg/releases/tag/v1.28.1)

**Release date:** Feb 5, 2026

##### Enhancements

- Added support for Azure's `DefaultAzureCredential` authentication mechanism for backup and recovery operations. This can be enabled by setting `azureCredentials.useDefaultAzureCredentials: true` in the backup configuration, simplifying authentication in Azure environments without requiring explicit storage account keys or SAS tokens. ([#9468](cloudnative-pg/cloudnative-pg#9468)) <!-- 1.27 1.25 -->

##### Fixes

- Fixed validation of PostgreSQL extension names containing underscores (e.g., `pg_partman`, `pg_ivm`). Extension names with underscores are automatically sanitized to use hyphens for Kubernetes volume names while preserving the original name in mount paths. Webhook validation prevents naming conflicts after sanitization. Contributed by [@shusaan](https://github.com/shusaan). ([#9386](cloudnative-pg/cloudnative-pg#9386)) <!-- 1.27 -->

- Fixed a critical issue where the `TimelineID` in the cluster status was not reset to 1 after a major version upgrade. Because `pg_upgrade` initializes a new timeline, keeping the old ID (e.g., timeline 2) caused replicas to attempt to restore incompatible history files from object storage, leading to fatal "requested timeline is not a child of this server's history" errors. ([#9830](cloudnative-pg/cloudnative-pg#9830)) <!-- 1.27 -->

- Fixed an issue where stale TLS status fields in the `Pooler` were not cleared after being removed from the specification. This was particularly critical when upgrading to v1.28.0, where the `ServerTLS` field was repurposed, causing PgBouncer to use incorrect certificates and resulting in "unsupported certificate" errors that blocked all application connectivity. The operator now explicitly clears `ServerCA`, `ClientCA`, `ClientTLS`, and `ServerTLS` status fields when they are no longer configured. ([#9397](cloudnative-pg/cloudnative-pg#9397))

- Fixed a bug where replicas could enter a crash-loop by attempting to download timeline history files from future timelines. This occurred when stale files remained in the WAL archive from a previous cluster life, and replicas would incorrectly try to fetch them during recovery. ([#9650](cloudnative-pg/cloudnative-pg#9650)) <!-- 1.27 1.25 -->

- Fixed a race condition in `replica_cluster` setups during designated primary transitions, preventing transient "no primary" states in the replica cluster. ([#9601](cloudnative-pg/cloudnative-pg#9601)) <!-- 1.27 1.25 -->

- The backup controller now uses the unique instance session ID to detect instance manager restarts. This prevents the operator from incorrectly assuming a backup is still progressing if the underlying container has crashed and restarted, which previously led to orphaned backup objects. ([#9370](cloudnative-pg/cloudnative-pg#9370)) <!-- 1.27 -->

- Fixed a validation gap in Azure object store configurations where the `storageAccount` was not required when using explicit credentials (such as a storage key or SAS token). The operator now enforces that a storage account name is provided in these cases and that `connectionString` is mutually exclusive with other authentication parameters. ([#9604](cloudnative-pg/cloudnative-pg#9604)) <!-- 1.27 1.25 -->

- Optimized the deletion path so the operator begins cleaning up resources immediately when a cluster is marked for deletion. This significantly reduces the time a cluster remains in `Terminating` status while waiting for internal reconciliation loops. ([#9555](cloudnative-pg/cloudnative-pg#9555)) <!-- 1.27 1.25 -->

- Fixed an issue where replication slots were not properly dropped from replicas when the feature was disabled or the cluster was reconfigured. This ensures that unused slots do not cause WAL build-up on the primary. ([#9381](cloudnative-pg/cloudnative-pg#9381)) <!-- 1.27 1.25 -->

- Fixed an issue where `imagePullSecrets` were not added to the `ServiceAccount` created for the `Pooler`. Previously, these secrets were applied to the Deployment but not the SA, which caused image pull failures in restricted environments using certain security policies. ([#9427](cloudnative-pg/cloudnative-pg#9427)) <!-- 1.27 1.25 -->

- Added a check to verify ownership before the operator deletes a `PodMonitor`. This prevents the operator from accidentally deleting manually managed monitoring resources that happen to share a name with expected CNPG resources. Contributed by [@juliamertz](https://github.com/juliamertz). ([#9340](cloudnative-pg/cloudnative-pg#9340)) <!-- 1.27 1.25 -->

- Fixed a bug where `pg_stat_archiver` metrics would continue to report stale data on standby instances after a switchover. The exporter now skips these metrics on standbys, as PostgreSQL only provides valid archiver stats on the primary. ([#9411](cloudnative-pg/cloudnative-pg#9411)) <!-- 1.27 1.25 -->

- Clarified the interpretation of timestamp formats for recovery `targetTime`. Timestamps provided without an explicit timezone are now consistently interpreted as UTC. Contributed by [@pchovelon](https://github.com/pchovelon). ([#8937](cloudnative-pg/cloudnative-pg#8937)) <!-- 1.27 1.25 -->

- Fixed backup status updates to prevent "resource has been modified" errors during concurrent updates. ([#9551](cloudnative-pg/cloudnative-pg#9551)) <!-- 1.27 1.25 -->

- Fixed event reporting to use the correct pod name when a backup pod is not found. ([#9552](cloudnative-pg/cloudnative-pg#9552)) <!-- 1.27 1.25 -->

- Improved performance of scheduled backup operations for clusters with a very high number of historical backups. ([#9489](cloudnative-pg/cloudnative-pg#9489)) <!-- 1.27 1.25 -->

- Fixed error handling when removing finalizers on `Database` objects. ([#9431](cloudnative-pg/cloudnative-pg#9431)) <!-- 1.27 1.25 -->

- `cnpg` plugin:

  - Updated the `status` command to display "Disabled" when the `skipWalArchiving` annotation is present on a cluster. This replaces confusing "starting up" or "unknown" states when WAL archiving is intentionally bypassed. ([#9709](cloudnative-pg/cloudnative-pg#9709)) <!-- 1.27 1.25 -->

  - Fixed the `logs --follow` command to continue polling for new pods instead of exiting prematurely when all current log streams complete. ([#9599](cloudnative-pg/cloudnative-pg#9599)) <!-- 1.27 1.25 -->
@dosubot dosubot Bot mentioned this pull request Mar 30, 2026
4 tasks done
armru added a commit that referenced this pull request Jun 9, 2026
A backup in the "started" phase only moves forward through the status
updates of the instance manager that is running it. When that instance
manager is restarted before the backup reaches the "running" phase, for
example by the in-place upgrade that follows an operator upgrade, no
update ever arrives: the controller scheduled no requeue after starting
the backup, and the watch on clusters could not reschedule it either, so
the backup stayed in the "started" phase forever and the session ID
check introduced by #9370 never had a chance to run.

Requeue the reconciliation while a backup is in the "started" phase so
the lost instance manager session is detected and the backup is marked
as failed.

The cluster-to-backup mapper also enqueued the running backups of every
cluster in the namespace except the one that changed; invert that
condition. The predicate gating those events only matched clusters with
an in-core backup section, which clusters backed up through a CNPG-I
plugin do not have; match those too.

Signed-off-by: Armando Ruocco <[email protected]>
armru added a commit that referenced this pull request Jun 9, 2026
A backup in the "started" phase only moves forward through the status
updates of the instance manager that is running it. When that instance
manager is restarted before the backup reaches the "running" phase, for
example by the in-place upgrade that follows an operator upgrade, no
update ever arrives and nothing reschedules the backup: the controller
scheduled no requeue after starting it, and the watch on clusters could
not re-enqueue it either, so the backup stayed in the "started" phase
and the session ID check introduced by #9370 never had a chance to run.

Requeue the reconciliation while a backup is in the "started" phase so
the lost instance manager session is detected and the backup is marked
as failed. The backup is passed to startBackupManagedByInstance by
pointer so the phase transition stays visible to that check.

The cluster-to-backup mapper also enqueued the running backups of every
cluster in the namespace except the one that changed; invert that
condition, and include "started" backups in the lookup so the cluster
event caused by the failure that orphaned a backup is the earliest
chance to detect it. The predicate gating those events only matched
clusters with an in-core backup section, which clusters backed up
through a CNPG-I plugin do not have; match those too.

Signed-off-by: Armando Ruocco <[email protected]>
NiccoloFei pushed a commit that referenced this pull request Jun 11, 2026
A backup in the "started" phase only moves forward through the status
updates of the instance manager that is running it. When that instance
manager is restarted before the backup reaches the "running" phase, for
example by the in-place upgrade that follows an operator upgrade, no
update ever arrives and nothing reschedules the backup: the controller
scheduled no requeue after starting it, and the watch on clusters could
not re-enqueue it either, so the backup stayed in the "started" phase
and the session ID check introduced by #9370 never had a chance to run.

Requeue the reconciliation while a backup is in the "started" phase so
the lost instance manager session is detected and the backup is marked
as failed. The backup is passed to startBackupManagedByInstance by
pointer so the phase transition stays visible to that check.

The cluster-to-backup mapper also enqueued the running backups of every
cluster in the namespace except the one that changed; invert that
condition, and include "started" backups in the lookup so the cluster
event caused by the failure that orphaned a backup is the earliest
chance to detect it. The predicate gating those events only matched
clusters with an in-core backup section, which clusters backed up
through a CNPG-I plugin do not have; match those too.

Signed-off-by: Armando Ruocco <[email protected]>
gbartolini pushed a commit that referenced this pull request Jun 15, 2026
A backup in the "started" phase only moves forward through the status
updates of the instance manager that is running it. When that instance
manager is restarted before the backup reaches the "running" phase, for
example by the in-place upgrade that follows an operator upgrade, no
update ever arrives and nothing reschedules the backup: the controller
scheduled no requeue after starting it, and the watch on clusters could
not re-enqueue it either, so the backup stayed in the "started" phase
and the session ID check introduced by #9370 never had a chance to run.

Requeue the reconciliation while a backup is in the "started" phase so
the lost instance manager session is detected and the backup is marked
as failed. The backup is passed to startBackupManagedByInstance by
pointer so the phase transition stays visible to that check.

The cluster-to-backup mapper also enqueued the running backups of every
cluster in the namespace except the one that changed; invert that
condition, and include "started" backups in the lookup so the cluster
event caused by the failure that orphaned a backup is the earliest
chance to detect it. The predicate gating those events only matched
clusters with an in-core backup section, which clusters backed up
through a CNPG-I plugin do not have; match those too.

Signed-off-by: Armando Ruocco <[email protected]>
NiccoloFei pushed a commit that referenced this pull request Jun 15, 2026
A backup in the "started" phase only moves forward through the status
updates of the instance manager that is running it. When that instance
manager is restarted before the backup reaches the "running" phase, for
example by the in-place upgrade that follows an operator upgrade, no
update ever arrives and nothing reschedules the backup: the controller
scheduled no requeue after starting it, and the watch on clusters could
not re-enqueue it either, so the backup stayed in the "started" phase
and the session ID check introduced by #9370 never had a chance to run.

Requeue the reconciliation while a backup is in the "started" phase so
the lost instance manager session is detected and the backup is marked
as failed. The backup is passed to startBackupManagedByInstance by
pointer so the phase transition stays visible to that check.

The cluster-to-backup mapper also enqueued the running backups of every
cluster in the namespace except the one that changed; invert that
condition, and include "started" backups in the lookup so the cluster
event caused by the failure that orphaned a backup is the earliest
chance to detect it. The predicate gating those events only matched
clusters with an in-core backup section, which clusters backed up
through a CNPG-I plugin do not have; match those too.

Signed-off-by: Armando Ruocco <[email protected]>
NiccoloFei pushed a commit that referenced this pull request Jun 15, 2026
A backup in the "started" phase only moves forward through the status
updates of the instance manager that is running it. When that instance
manager is restarted before the backup reaches the "running" phase, for
example by the in-place upgrade that follows an operator upgrade, no
update ever arrives and nothing reschedules the backup: the controller
scheduled no requeue after starting it, and the watch on clusters could
not re-enqueue it either, so the backup stayed in the "started" phase
and the session ID check introduced by #9370 never had a chance to run.

Requeue the reconciliation while a backup is in the "started" phase so
the lost instance manager session is detected and the backup is marked
as failed. The backup is passed to startBackupManagedByInstance by
pointer so the phase transition stays visible to that check.

The cluster-to-backup mapper also enqueued the running backups of every
cluster in the namespace except the one that changed; invert that
condition, and include "started" backups in the lookup so the cluster
event caused by the failure that orphaned a backup is the earliest
chance to detect it. The predicate gating those events only matched
clusters with an in-core backup section, which clusters backed up
through a CNPG-I plugin do not have; match those too.

Signed-off-by: Armando Ruocco <[email protected]>
mnencia pushed a commit that referenced this pull request Jun 15, 2026
A backup in the "started" phase only moves forward through the status
updates of the instance manager that is running it. When that instance
manager is restarted before the backup reaches the "running" phase, for
example by the in-place upgrade that follows an operator upgrade, no
update ever arrives and nothing reschedules the backup: the controller
scheduled no requeue after starting it, and the watch on clusters could
not re-enqueue it either, so the backup stayed in the "started" phase
and the session ID check introduced by #9370 never had a chance to run.

Requeue the reconciliation while a backup is in the "started" phase so
the lost instance manager session is detected and the backup is marked
as failed. The backup is passed to startBackupManagedByInstance by
pointer so the phase transition stays visible to that check.

The cluster-to-backup mapper also enqueued the running backups of every
cluster in the namespace except the one that changed; invert that
condition, and include "started" backups in the lookup so the cluster
event caused by the failure that orphaned a backup is the earliest
chance to detect it. The predicate gating those events only matched
clusters with an in-core backup section, which clusters backed up
through a CNPG-I plugin do not have; match those too.

Signed-off-by: Armando Ruocco <[email protected]>
mnencia added a commit that referenced this pull request Jun 15, 2026
A backup in the "started" phase only moves forward through the status
updates of the instance manager that is running it. When that instance
manager is restarted before the backup reaches the "running" phase, for
example by the in-place upgrade that follows an operator upgrade, no
update ever arrives and nothing reschedules the backup: the controller
scheduled no requeue after starting it, and the watch on clusters could
not re-enqueue it either, so the backup stayed in the "started" phase
and the session ID check introduced by #9370 never had a chance to run.

Requeue the reconciliation while a backup is in the "started" phase so
the lost instance manager session is detected and the backup is marked
as failed. The backup is now passed to startBackupManagedByInstance by
pointer so the phase transition stays visible to that check.

The cluster-to-backup mapper, the other path that could have re-enqueued
the backup, had two further problems. It enqueued the running backups of
every cluster in the namespace except the one that actually changed, so
invert that condition; and it only considered "running" backups, so
include "started" ones too, since the cluster event from the failure
that orphaned the backup is the earliest chance to detect it. The
predicate gating those events only matched clusters with an in-core
backup section, which clusters backed up through a CNPG-I plugin do not
have; match those too.

Signed-off-by: Armando Ruocco <[email protected]>
Signed-off-by: Marco Nenciarini <[email protected]>
Co-authored-by: Marco Nenciarini <[email protected]>
cnpg-bot pushed a commit that referenced this pull request Jun 15, 2026
A backup in the "started" phase only moves forward through the status
updates of the instance manager that is running it. When that instance
manager is restarted before the backup reaches the "running" phase, for
example by the in-place upgrade that follows an operator upgrade, no
update ever arrives and nothing reschedules the backup: the controller
scheduled no requeue after starting it, and the watch on clusters could
not re-enqueue it either, so the backup stayed in the "started" phase
and the session ID check introduced by #9370 never had a chance to run.

Requeue the reconciliation while a backup is in the "started" phase so
the lost instance manager session is detected and the backup is marked
as failed. The backup is now passed to startBackupManagedByInstance by
pointer so the phase transition stays visible to that check.

The cluster-to-backup mapper, the other path that could have re-enqueued
the backup, had two further problems. It enqueued the running backups of
every cluster in the namespace except the one that actually changed, so
invert that condition; and it only considered "running" backups, so
include "started" ones too, since the cluster event from the failure
that orphaned the backup is the earliest chance to detect it. The
predicate gating those events only matched clusters with an in-core
backup section, which clusters backed up through a CNPG-I plugin do not
have; match those too.

Signed-off-by: Armando Ruocco <[email protected]>
Signed-off-by: Marco Nenciarini <[email protected]>
Co-authored-by: Marco Nenciarini <[email protected]>
(cherry picked from commit 6b916d0)
mnencia added a commit that referenced this pull request Jun 15, 2026
A backup in the "started" phase only moves forward through the status
updates of the instance manager that is running it. When that instance
manager is restarted before the backup reaches the "running" phase, for
example by the in-place upgrade that follows an operator upgrade, no
update ever arrives and nothing reschedules the backup: the controller
scheduled no requeue after starting it, and the watch on clusters could
not re-enqueue it either, so the backup stayed in the "started" phase
and the session ID check introduced by #9370 never had a chance to run.

Requeue the reconciliation while a backup is in the "started" phase so
the lost instance manager session is detected and the backup is marked
as failed. The backup is now passed to startBackupManagedByInstance by
pointer so the phase transition stays visible to that check.

The cluster-to-backup mapper, the other path that could have re-enqueued
the backup, had two further problems. It enqueued the running backups of
every cluster in the namespace except the one that actually changed, so
invert that condition; and it only considered "running" backups, so
include "started" ones too, since the cluster event from the failure
that orphaned the backup is the earliest chance to detect it. The
predicate gating those events only matched clusters with an in-core
backup section, which clusters backed up through a CNPG-I plugin do not
have; match those too.

Signed-off-by: Armando Ruocco <[email protected]>
Signed-off-by: Marco Nenciarini <[email protected]>
Co-authored-by: Marco Nenciarini <[email protected]>
(cherry picked from commit 6b916d0)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-requested ◀️ This pull request should be backported to all supported releases bug 🐛 Something isn't working lgtm This PR has been approved by a maintainer no-issue ok to merge 👌 This PR can be merged release-1.28 size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants