Skip to content

fix(major-upgrade): reset TimelineID to 1 after upgrade - #9830

Merged
mnencia merged 4 commits into
mainfrom
dev/9764
Feb 3, 2026
Merged

mnencia merged 4 commits into
mainfrom
dev/9764

Conversation

@mnencia

@mnencia mnencia commented Feb 2, 2026 •

Copy link
Copy Markdown
Member

After a major version upgrade, pg_upgrade creates a new database system with a new System ID and resets the timeline to 1. However, cluster.Status.TimelineID retained its old value (e.g., timeline 2), causing replicas to restore incompatible timeline history files from object storage.

This resulted in PostgreSQL fatal errors:

"requested timeline 2 is not a child of this server's history"
"Latest checkpoint in file backup_label is at 0/13000080 on timeline 1, but in the history of the requested timeline, the server forked off from that timeline at 0/70226C8."

The fix sets cluster.Status.TimelineID to 1 after major upgrade completion, ensuring validateTimelineHistoryFile() blocks incompatible timeline history files (e.g., 00000002.history).

E2E tests verify this by performing a switchover to timeline 2 before the major upgrade, then confirming TimelineID is reset to 1 after upgrade completion.

Closes #9764

@cnpg-bot cnpg-bot added backport-requested ◀️ This pull request should be backported to all supported releases release-1.25 release-1.28 labels Feb 2, 2026
@github-actions

github-actions Bot commented Feb 2, 2026

Copy link
Copy Markdown
Contributor

❗ By default, the pull request is configured to backport to all release branches.

  • To stop backporting this pr, remove the label: backport-requested ◀️ or add the label 'do not backport'
  • To stop backporting this pr to a certain release branch, remove the specific branch label: release-x.y

@mnencia

mnencia commented Feb 2, 2026

Copy link
Copy Markdown
Member Author

/test

@github-actions

github-actions Bot commented Feb 2, 2026

Copy link
Copy Markdown
Contributor

@mnencia
mnencia marked this pull request as ready for review February 2, 2026 16:13
@mnencia
mnencia requested a review from a team February 2, 2026 16:13
@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. bug 🐛 Something isn't working labels Feb 2, 2026
@mnencia

mnencia commented Feb 2, 2026

Copy link
Copy Markdown
Member Author

/test ft=postgres-major-upgrade

@github-actions

github-actions Bot commented Feb 2, 2026

Copy link
Copy Markdown
Contributor

@mnencia
mnencia force-pushed the dev/9764 branch 3 times, most recently from fe9639c to f59b9bb Compare February 2, 2026 17:06
@mnencia

mnencia commented Feb 2, 2026

Copy link
Copy Markdown
Member Author

/test ft=postgres-major-upgrade

@github-actions

github-actions Bot commented Feb 2, 2026

Copy link
Copy Markdown
Contributor

@mnencia
mnencia force-pushed the dev/9764 branch 2 times, most recently from 3f47dd3 to 141533d Compare February 2, 2026 17:27
@cnpg-bot cnpg-bot added the ok to merge 👌 This PR can be merged label Feb 2, 2026
@mnencia
mnencia force-pushed the dev/9764 branch 2 times, most recently from bf9d378 to 78317af Compare February 2, 2026 17:33
mnencia and others added 3 commits February 3, 2026 11:03
After a major version upgrade, pg_upgrade creates a new database
system with a new System ID and resets the timeline to 1.
However, cluster.Status.TimelineID retained its old value (e.g.,
timeline 2), causing replicas to restore incompatible timeline
history files from object storage.

This resulted in PostgreSQL fatal errors:

    "requested timeline 2 is not a child of this server's history"
    "Latest checkpoint in file backup_label is at 0/13000080 on
     timeline 1, but in the history of the requested timeline,
     the server forked off from that timeline at 0/70226C8."

The fix sets cluster.Status.TimelineID to 1 after major upgrade
completion, ensuring validateTimelineHistoryFile() blocks
incompatible timeline history files (e.g., 00000002.history).

E2E tests verify this by performing a switchover to timeline 2
before the major upgrade, then confirming TimelineID is reset to
1 after upgrade completion.

Closes #9764

Signed-off-by: Marco Nenciarini <[email protected]>
Add explicit test verification for TimelineID being reset to 1
after major upgrade completion in majorVersionUpgradeHandleCompletion.

Signed-off-by: Armando Ruocco <[email protected]>
Signed-off-by: Gabriele Bartolini <[email protected]>
@dosubot dosubot Bot added the lgtm This PR has been approved by a maintainer label Feb 3, 2026
@mnencia
mnencia merged commit a2f9d47 into main Feb 3, 2026
36 checks passed
@mnencia
mnencia deleted the dev/9764 branch February 3, 2026 12:17
cnpg-bot pushed a commit that referenced this pull request Feb 3, 2026
After a major version upgrade, pg_upgrade creates a new database system
with a new System ID and resets the timeline to 1. However,
cluster.Status.TimelineID retained its old value (e.g., timeline 2),
causing replicas to restore incompatible timeline history files from
object storage.

This resulted in PostgreSQL fatal errors:

    "requested timeline 2 is not a child of this server's history"

    "Latest checkpoint in file backup_label is at 0/13000080 on timeline 1,
     but in the history of the requested timeline, the server forked off from
     that timeline at 0/70226C8."

The fix sets cluster.Status.TimelineID to 1 after major upgrade
completion, ensuring validateTimelineHistoryFile() blocks incompatible
timeline history files (e.g., 00000002.history).

E2E tests verify this by performing a switchover to timeline 2 before
the major upgrade, then confirming TimelineID is reset to 1 after
upgrade completion.

Closes #9764

Signed-off-by: Marco Nenciarini <[email protected]>
Signed-off-by: Armando Ruocco <[email protected]>
Signed-off-by: Gabriele Bartolini <[email protected]>
Co-authored-by: Armando Ruocco <[email protected]>
Co-authored-by: Gabriele Bartolini <[email protected]>
(cherry picked from commit a2f9d47)
cnpg-bot pushed a commit that referenced this pull request Feb 3, 2026
After a major version upgrade, pg_upgrade creates a new database system
with a new System ID and resets the timeline to 1. However,
cluster.Status.TimelineID retained its old value (e.g., timeline 2),
causing replicas to restore incompatible timeline history files from
object storage.

This resulted in PostgreSQL fatal errors:

    "requested timeline 2 is not a child of this server's history"

    "Latest checkpoint in file backup_label is at 0/13000080 on timeline 1,
     but in the history of the requested timeline, the server forked off from
     that timeline at 0/70226C8."

The fix sets cluster.Status.TimelineID to 1 after major upgrade
completion, ensuring validateTimelineHistoryFile() blocks incompatible
timeline history files (e.g., 00000002.history).

E2E tests verify this by performing a switchover to timeline 2 before
the major upgrade, then confirming TimelineID is reset to 1 after
upgrade completion.

Closes #9764

Signed-off-by: Marco Nenciarini <[email protected]>
Signed-off-by: Armando Ruocco <[email protected]>
Signed-off-by: Gabriele Bartolini <[email protected]>
Co-authored-by: Armando Ruocco <[email protected]>
Co-authored-by: Gabriele Bartolini <[email protected]>
(cherry picked from commit a2f9d47)
renovate Bot added a commit to sdwilsh/ansible-playbooks that referenced this pull request Mar 26, 2026
##### [\`v1.28.1\`](https://github.com/cloudnative-pg/cloudnative-pg/releases/tag/v1.28.1)

**Release date:** Feb 5, 2026

##### Enhancements

- Added support for Azure's `DefaultAzureCredential` authentication mechanism for backup and recovery operations. This can be enabled by setting `azureCredentials.useDefaultAzureCredentials: true` in the backup configuration, simplifying authentication in Azure environments without requiring explicit storage account keys or SAS tokens. ([#9468](cloudnative-pg/cloudnative-pg#9468)) <!-- 1.27 1.25 -->

##### Fixes

- Fixed validation of PostgreSQL extension names containing underscores (e.g., `pg_partman`, `pg_ivm`). Extension names with underscores are automatically sanitized to use hyphens for Kubernetes volume names while preserving the original name in mount paths. Webhook validation prevents naming conflicts after sanitization. Contributed by [@shusaan](https://github.com/shusaan). ([#9386](cloudnative-pg/cloudnative-pg#9386)) <!-- 1.27 -->

- Fixed a critical issue where the `TimelineID` in the cluster status was not reset to 1 after a major version upgrade. Because `pg_upgrade` initializes a new timeline, keeping the old ID (e.g., timeline 2) caused replicas to attempt to restore incompatible history files from object storage, leading to fatal "requested timeline is not a child of this server's history" errors. ([#9830](cloudnative-pg/cloudnative-pg#9830)) <!-- 1.27 -->

- Fixed an issue where stale TLS status fields in the `Pooler` were not cleared after being removed from the specification. This was particularly critical when upgrading to v1.28.0, where the `ServerTLS` field was repurposed, causing PgBouncer to use incorrect certificates and resulting in "unsupported certificate" errors that blocked all application connectivity. The operator now explicitly clears `ServerCA`, `ClientCA`, `ClientTLS`, and `ServerTLS` status fields when they are no longer configured. ([#9397](cloudnative-pg/cloudnative-pg#9397))

- Fixed a bug where replicas could enter a crash-loop by attempting to download timeline history files from future timelines. This occurred when stale files remained in the WAL archive from a previous cluster life, and replicas would incorrectly try to fetch them during recovery. ([#9650](cloudnative-pg/cloudnative-pg#9650)) <!-- 1.27 1.25 -->

- Fixed a race condition in `replica_cluster` setups during designated primary transitions, preventing transient "no primary" states in the replica cluster. ([#9601](cloudnative-pg/cloudnative-pg#9601)) <!-- 1.27 1.25 -->

- The backup controller now uses the unique instance session ID to detect instance manager restarts. This prevents the operator from incorrectly assuming a backup is still progressing if the underlying container has crashed and restarted, which previously led to orphaned backup objects. ([#9370](cloudnative-pg/cloudnative-pg#9370)) <!-- 1.27 -->

- Fixed a validation gap in Azure object store configurations where the `storageAccount` was not required when using explicit credentials (such as a storage key or SAS token). The operator now enforces that a storage account name is provided in these cases and that `connectionString` is mutually exclusive with other authentication parameters. ([#9604](cloudnative-pg/cloudnative-pg#9604)) <!-- 1.27 1.25 -->

- Optimized the deletion path so the operator begins cleaning up resources immediately when a cluster is marked for deletion. This significantly reduces the time a cluster remains in `Terminating` status while waiting for internal reconciliation loops. ([#9555](cloudnative-pg/cloudnative-pg#9555)) <!-- 1.27 1.25 -->

- Fixed an issue where replication slots were not properly dropped from replicas when the feature was disabled or the cluster was reconfigured. This ensures that unused slots do not cause WAL build-up on the primary. ([#9381](cloudnative-pg/cloudnative-pg#9381)) <!-- 1.27 1.25 -->

- Fixed an issue where `imagePullSecrets` were not added to the `ServiceAccount` created for the `Pooler`. Previously, these secrets were applied to the Deployment but not the SA, which caused image pull failures in restricted environments using certain security policies. ([#9427](cloudnative-pg/cloudnative-pg#9427)) <!-- 1.27 1.25 -->

- Added a check to verify ownership before the operator deletes a `PodMonitor`. This prevents the operator from accidentally deleting manually managed monitoring resources that happen to share a name with expected CNPG resources. Contributed by [@juliamertz](https://github.com/juliamertz). ([#9340](cloudnative-pg/cloudnative-pg#9340)) <!-- 1.27 1.25 -->

- Fixed a bug where `pg_stat_archiver` metrics would continue to report stale data on standby instances after a switchover. The exporter now skips these metrics on standbys, as PostgreSQL only provides valid archiver stats on the primary. ([#9411](cloudnative-pg/cloudnative-pg#9411)) <!-- 1.27 1.25 -->

- Clarified the interpretation of timestamp formats for recovery `targetTime`. Timestamps provided without an explicit timezone are now consistently interpreted as UTC. Contributed by [@pchovelon](https://github.com/pchovelon). ([#8937](cloudnative-pg/cloudnative-pg#8937)) <!-- 1.27 1.25 -->

- Fixed backup status updates to prevent "resource has been modified" errors during concurrent updates. ([#9551](cloudnative-pg/cloudnative-pg#9551)) <!-- 1.27 1.25 -->

- Fixed event reporting to use the correct pod name when a backup pod is not found. ([#9552](cloudnative-pg/cloudnative-pg#9552)) <!-- 1.27 1.25 -->

- Improved performance of scheduled backup operations for clusters with a very high number of historical backups. ([#9489](cloudnative-pg/cloudnative-pg#9489)) <!-- 1.27 1.25 -->

- Fixed error handling when removing finalizers on `Database` objects. ([#9431](cloudnative-pg/cloudnative-pg#9431)) <!-- 1.27 1.25 -->

- `cnpg` plugin:

  - Updated the `status` command to display "Disabled" when the `skipWalArchiving` annotation is present on a cluster. This replaces confusing "starting up" or "unknown" states when WAL archiving is intentionally bypassed. ([#9709](cloudnative-pg/cloudnative-pg#9709)) <!-- 1.27 1.25 -->

  - Fixed the `logs --follow` command to continue polling for new pods instead of exiting prematurely when all current log streams complete. ([#9599](cloudnative-pg/cloudnative-pg#9599)) <!-- 1.27 1.25 -->
sdwilsh pushed a commit to sdwilsh/ansible-playbooks that referenced this pull request Mar 26, 2026
##### [\`v1.28.1\`](https://github.com/cloudnative-pg/cloudnative-pg/releases/tag/v1.28.1)

**Release date:** Feb 5, 2026

##### Enhancements

- Added support for Azure's `DefaultAzureCredential` authentication mechanism for backup and recovery operations. This can be enabled by setting `azureCredentials.useDefaultAzureCredentials: true` in the backup configuration, simplifying authentication in Azure environments without requiring explicit storage account keys or SAS tokens. ([#9468](cloudnative-pg/cloudnative-pg#9468)) <!-- 1.27 1.25 -->

##### Fixes

- Fixed validation of PostgreSQL extension names containing underscores (e.g., `pg_partman`, `pg_ivm`). Extension names with underscores are automatically sanitized to use hyphens for Kubernetes volume names while preserving the original name in mount paths. Webhook validation prevents naming conflicts after sanitization. Contributed by [@shusaan](https://github.com/shusaan). ([#9386](cloudnative-pg/cloudnative-pg#9386)) <!-- 1.27 -->

- Fixed a critical issue where the `TimelineID` in the cluster status was not reset to 1 after a major version upgrade. Because `pg_upgrade` initializes a new timeline, keeping the old ID (e.g., timeline 2) caused replicas to attempt to restore incompatible history files from object storage, leading to fatal "requested timeline is not a child of this server's history" errors. ([#9830](cloudnative-pg/cloudnative-pg#9830)) <!-- 1.27 -->

- Fixed an issue where stale TLS status fields in the `Pooler` were not cleared after being removed from the specification. This was particularly critical when upgrading to v1.28.0, where the `ServerTLS` field was repurposed, causing PgBouncer to use incorrect certificates and resulting in "unsupported certificate" errors that blocked all application connectivity. The operator now explicitly clears `ServerCA`, `ClientCA`, `ClientTLS`, and `ServerTLS` status fields when they are no longer configured. ([#9397](cloudnative-pg/cloudnative-pg#9397))

- Fixed a bug where replicas could enter a crash-loop by attempting to download timeline history files from future timelines. This occurred when stale files remained in the WAL archive from a previous cluster life, and replicas would incorrectly try to fetch them during recovery. ([#9650](cloudnative-pg/cloudnative-pg#9650)) <!-- 1.27 1.25 -->

- Fixed a race condition in `replica_cluster` setups during designated primary transitions, preventing transient "no primary" states in the replica cluster. ([#9601](cloudnative-pg/cloudnative-pg#9601)) <!-- 1.27 1.25 -->

- The backup controller now uses the unique instance session ID to detect instance manager restarts. This prevents the operator from incorrectly assuming a backup is still progressing if the underlying container has crashed and restarted, which previously led to orphaned backup objects. ([#9370](cloudnative-pg/cloudnative-pg#9370)) <!-- 1.27 -->

- Fixed a validation gap in Azure object store configurations where the `storageAccount` was not required when using explicit credentials (such as a storage key or SAS token). The operator now enforces that a storage account name is provided in these cases and that `connectionString` is mutually exclusive with other authentication parameters. ([#9604](cloudnative-pg/cloudnative-pg#9604)) <!-- 1.27 1.25 -->

- Optimized the deletion path so the operator begins cleaning up resources immediately when a cluster is marked for deletion. This significantly reduces the time a cluster remains in `Terminating` status while waiting for internal reconciliation loops. ([#9555](cloudnative-pg/cloudnative-pg#9555)) <!-- 1.27 1.25 -->

- Fixed an issue where replication slots were not properly dropped from replicas when the feature was disabled or the cluster was reconfigured. This ensures that unused slots do not cause WAL build-up on the primary. ([#9381](cloudnative-pg/cloudnative-pg#9381)) <!-- 1.27 1.25 -->

- Fixed an issue where `imagePullSecrets` were not added to the `ServiceAccount` created for the `Pooler`. Previously, these secrets were applied to the Deployment but not the SA, which caused image pull failures in restricted environments using certain security policies. ([#9427](cloudnative-pg/cloudnative-pg#9427)) <!-- 1.27 1.25 -->

- Added a check to verify ownership before the operator deletes a `PodMonitor`. This prevents the operator from accidentally deleting manually managed monitoring resources that happen to share a name with expected CNPG resources. Contributed by [@juliamertz](https://github.com/juliamertz). ([#9340](cloudnative-pg/cloudnative-pg#9340)) <!-- 1.27 1.25 -->

- Fixed a bug where `pg_stat_archiver` metrics would continue to report stale data on standby instances after a switchover. The exporter now skips these metrics on standbys, as PostgreSQL only provides valid archiver stats on the primary. ([#9411](cloudnative-pg/cloudnative-pg#9411)) <!-- 1.27 1.25 -->

- Clarified the interpretation of timestamp formats for recovery `targetTime`. Timestamps provided without an explicit timezone are now consistently interpreted as UTC. Contributed by [@pchovelon](https://github.com/pchovelon). ([#8937](cloudnative-pg/cloudnative-pg#8937)) <!-- 1.27 1.25 -->

- Fixed backup status updates to prevent "resource has been modified" errors during concurrent updates. ([#9551](cloudnative-pg/cloudnative-pg#9551)) <!-- 1.27 1.25 -->

- Fixed event reporting to use the correct pod name when a backup pod is not found. ([#9552](cloudnative-pg/cloudnative-pg#9552)) <!-- 1.27 1.25 -->

- Improved performance of scheduled backup operations for clusters with a very high number of historical backups. ([#9489](cloudnative-pg/cloudnative-pg#9489)) <!-- 1.27 1.25 -->

- Fixed error handling when removing finalizers on `Database` objects. ([#9431](cloudnative-pg/cloudnative-pg#9431)) <!-- 1.27 1.25 -->

- `cnpg` plugin:

  - Updated the `status` command to display "Disabled" when the `skipWalArchiving` annotation is present on a cluster. This replaces confusing "starting up" or "unknown" states when WAL archiving is intentionally bypassed. ([#9709](cloudnative-pg/cloudnative-pg#9709)) <!-- 1.27 1.25 -->

  - Fixed the `logs --follow` command to continue polling for new pods instead of exiting prematurely when all current log streams complete. ([#9599](cloudnative-pg/cloudnative-pg#9599)) <!-- 1.27 1.25 -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-requested ◀️ This pull request should be backported to all supported releases bug 🐛 Something isn't working lgtm This PR has been approved by a maintainer ok to merge 👌 This PR can be merged release-1.28 size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Problem upgrading on a restored cluster.

5 participants