You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit e494009
Browse filesBrowse the repository at this point in the historyBrowse files
authored
fix: report failed instance creation jobs instead of waiting forever (#11035)
When bootstrapping, restoring, or adding a replica, an instance creation
Job can exhaust its backoff limit and end in a BackoffLimitExceeded
failure. When that happened the cluster stayed in a transient creation
phase (for example "Setting up primary") forever: the operator counted
the failed Job as still running, so reconciliation kept waiting on it
and never reported the failure.
A Job is treated as running until it reports one successful completion.
A Job that has permanently failed never reaches that state, so it looked
the same as one still in progress, and the running-job guard returned
early on every reconciliation without updating the phase.
This change detects the terminal JobFailed condition and moves the
cluster into the existing unrecoverable phase, with a reason that names
the failed Job and points to its logs. Detection keys off the JobFailed
condition, not the failed-pod count, so a Job that is still retrying
within its backoff limit is left alone. The check runs after the major
version upgrade reconciliation, which owns its own Job lifecycle, so
upgrade Jobs are not affected.
The failed Job and its Pods are left in place so their logs stay
available. There is no single recipe to recover: the cause has to be
found in the logs and fixed before the instance can be created again.
Unit tests cover the new Job failure detection, and a controller test
verifies that a Job past its backoff limit moves the cluster to the
unrecoverable phase.
Closes#11042
Refs #10733
Signed-off-by: Armando Ruocco <[email protected]>
Signed-off-by: Leonardo Cecchi <[email protected]>
Signed-off-by: Marco Nenciarini <[email protected]>
Co-authored-by: Leonardo Cecchi <[email protected]>
Co-authored-by: Marco Nenciarini <[email protected]>
0 commit comments