Repository navigation
Conversation
dosubot
Bot
added
the
size:S
cnpg-bot
added
backport-requested ◀️
Contributor
|
❗ By default, the pull request is configured to backport to all release branches.
|
redbaron
force-pushed
the
major-upgrade-failure-rollback
branch
2 times, most recently
from
December 1, 2025 22:30
a2bdc0c to
625135e
Compare
NiccoloFei
force-pushed
the
major-upgrade-failure-rollback
branch
2 times, most recently
from
February 25, 2026 16:08
8891664 to
0abcc65
Compare
4 tasks done
NiccoloFei
force-pushed
the
major-upgrade-failure-rollback
branch
2 times, most recently
from
February 27, 2026 14:53
a01219d to
ea3e0a6
Compare
NiccoloFei
requested review from
NiccoloFei,
jsilvela and
litaocdl
as code owners
February 27, 2026 19:09
dosubot
Bot
added
size:L
NiccoloFei
force-pushed
the
major-upgrade-failure-rollback
branch
from
February 27, 2026 19:10
275df96 to
ee198a9
Compare
Collaborator
|
/test |
Contributor
|
@NiccoloFei, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/22500207879 |
Unless PGData version was updated (as recorded by the status), allow image rollback to the previous version. Signed-off-by: Maxim Ivanov <[email protected]>
Signed-off-by: Niccolò Fei <[email protected]>
NiccoloFei
force-pushed
the
major-upgrade-failure-rollback
branch
from
March 2, 2026 13:15
ee198a9 to
fc490ee
Compare
Collaborator
|
/test |
Contributor
|
@NiccoloFei, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/22577660156 |
Contributor
|
@NiccoloFei, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/22587955136 |
Member
|
The E2E tests added by @NiccoloFei highlighted that the initial approach was not working due to the race between the job recreating the pod and the rollback. I experimented with a more radical approach by making the job run only once (BackoffLimit=0) and automatically handling the job deletion when the user reverts the image: #10104 |
mnencia
added a commit
that referenced
this pull request
Mar 9, 2026
Set BackoffLimit=0 on the major upgrade job so Kubernetes does not retry a failed pg_upgrade (retries won't produce a different result and the retry pods hold the primary PVC, blocking recovery). When the upgrade job exists but has not completed, the reconciler now checks whether the user rolled back the image to the previous major version. If so it deletes the job with foreground propagation and requeues, allowing the cluster to restart on the original version without manual intervention. Move the majorupgrade.Reconcile call above the running-jobs guard in reconcileResources so the reconciler can act on failed upgrade jobs that would otherwise block the controller indefinitely. Inspired by Maxim Ivanov's initial rollback approach in #9344. Signed-off-by: Marco Nenciarini <[email protected]>
mnencia
added a commit
that referenced
this pull request
Mar 9, 2026
Add a rollback scenario to the major upgrade E2E suite. The test creates a cluster with the pgvector extension, attempts an upgrade to a minimal image that lacks the extension (causing pg_upgrade to fail), then reverts the image and verifies the operator automatically cleans up the failed job and the cluster recovers on the original version with its timeline unchanged. Based on Niccolò Fei's E2E test work in #9344. Co-authored-by: Niccolò Fei <[email protected]> Signed-off-by: Marco Nenciarini <[email protected]>
Member
|
Hi @redbaron, thank you for your work on this — it was instrumental in shaping the final solution. As mentioned above, the E2E tests revealed a race condition with the default BackoffLimit that led us to take a different approach in #10104: disabling job retries entirely (BackoffLimit=0) and having the operator handle the cleanup automatically. Since that PR supersedes this one, I'm going to close it. Thanks again for the contribution! |
Contributor
|
❗ By default, the pull request is configured to backport to all release branches.
|
mnencia
added a commit
that referenced
this pull request
Mar 9, 2026
…#10104) Set BackoffLimit=0 on the major upgrade job so Kubernetes does not retry a failed pg_upgrade. When the user reverts the image after a failed upgrade, the operator automatically deletes the failed job and lets the cluster restart on the original version. Inspired by Maxim Ivanov's initial rollback approach in #9344. Closes #9128 Signed-off-by: Marco Nenciarini <[email protected]> Signed-off-by: Armando Ruocco <[email protected]> Co-authored-by: Niccolò Fei <[email protected]> Co-authored-by: Armando Ruocco <[email protected]>
mnencia
added a commit
that referenced
this pull request
Mar 9, 2026
…#10104) Set BackoffLimit=0 on the major upgrade job so Kubernetes does not retry a failed pg_upgrade. When the user reverts the image after a failed upgrade, the operator automatically deletes the failed job and lets the cluster restart on the original version. Inspired by Maxim Ivanov's initial rollback approach in #9344. Closes #9128 Signed-off-by: Marco Nenciarini <[email protected]> Signed-off-by: Armando Ruocco <[email protected]> Co-authored-by: Niccolò Fei <[email protected]> Co-authored-by: Armando Ruocco <[email protected]> (cherry picked from commit 5b7b799)
mnencia
added a commit
that referenced
this pull request
Mar 9, 2026
…#10104) Set BackoffLimit=0 on the major upgrade job so Kubernetes does not retry a failed pg_upgrade. When the user reverts the image after a failed upgrade, the operator automatically deletes the failed job and lets the cluster restart on the original version. Inspired by Maxim Ivanov's initial rollback approach in #9344. Closes #9128 Signed-off-by: Marco Nenciarini <[email protected]> Signed-off-by: Armando Ruocco <[email protected]> Co-authored-by: Niccolò Fei <[email protected]> Co-authored-by: Armando Ruocco <[email protected]> (cherry picked from commit 5b7b799)
4 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Unless PGData version was updated (as recorded by the status), allow image rollback to the previous version.
Fixes #9128