Repository navigation
[Backport release-1.6] fix(backups): hold a BackupJob while its application is being restored - #5066
Merged
Merged
Conversation
Nothing on the backup path looks at restores. A Plan creates its BackupJobs on schedule, and the BackupJob reconciler dispatches them to their driver whatever else is happening to the application. For CNPG that is not merely a backup of a half-restored state: every BackupJob reconcile re-applies the Cluster's barman plugin, and on a cluster under an in-place restore it can land after the chart has re-rendered the Cluster with its new serverName and put it back on the source's. The restored cluster then archives onto the source's WAL prefix and its recovery fails with "WAL archive check failed ... Expected empty archive". This was reproduced with a backup triggered a few seconds after an in-place restore, and an hourly plan overlaps a restore of up to thirty minutes about half the time. A BackupJob that has not started yet now waits while a non-terminal RestoreJob in its namespace writes into the same application, resolved as the drivers resolve it: the Backup's applicationRef, with the target's kind and name when a targetApplicationRef is set, and the target's API group only when it names one. A RestoreJob whose Backup is gone holds nothing, since the restore controller fails it on its next pass. options.targetNamespace is honoured only for a Backup taken by the Velero strategy, the one driver that reads it; the CNPG and Job-based drivers ignore the option and restore in place, so their restores hold whatever it says. The held job stays Pending with a Ready=False RestoreInProgress condition naming the restore and is requeued, rather than failed, so a scheduled run still happens once the restore ends. A failed write of that status is returned, so the manager retries it. When the restore ends the condition is removed before the job starts, since it would otherwise read "waiting" until the driver next writes Ready, which for a healthy run is when the backup completes. A run that has already started is left alone: its driver owns state the gate would strand. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <[email protected]> (cherry picked from commit af8cf74)
The Velero driver resolves a restore's target from the Backup's applicationRef, overriding only the kind and name, and restores the Backup's resources. A targetApplicationRef.apiGroup never reaches it. The restore gate honoured that group all the same, so a Velero RestoreJob naming another group let a BackupJob of the application actually being restored start against it, and held one of an application the restore never touches. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <[email protected]> (cherry picked from commit 28e42fb)
1 of 12 tasks
Contributor
|
Important Review skippedBot user detected. To trigger a single review, invoke the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Backport of #4544 to
release-1.6.