fix(orchestration): mark non-terminal tasks Canceled on scheduler deadlock (#1879)#1894
Merged
fix(orchestration): mark non-terminal tasks Canceled on scheduler deadlock (#1879)#1894
Conversation
…dlock (#1879) When the scheduler detected a deadlock (no running or ready tasks, graph not complete), it set GraphStatus::Failed but left individual tasks in their previous status (Pending/Skipped). The message formatter then reported "Plan failed. 0/N tasks failed:" — accurate but misleading. Fix: mirror the cancel_all() pattern — iterate non-terminal tasks and set them to TaskStatus::Canceled at deadlock time. Update format_plan_done_message() to distinguish three cases: - Pure deadlock (0 failed, N canceled): "Plan canceled. N/M tasks did not run." - Mixed (failed + canceled): "Plan failed. X/M tasks failed, Y canceled:" - Normal failure (failed only): original message unchanged Add debug_assert! to make the self.running invariant explicit. Add tracing::warn! for the impossible empty-both edge case.
github-actions
bot
added
documentation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
TaskStatus::Canceled(mirrorscancel_all()pattern), instead of leaving them inPending/Skippedformat_plan_done_message()distinguishes three cases: pure deadlock → "Plan canceled. N/M tasks did not run.", mixed failure+cancellation → "Plan failed. X/M tasks failed, Y canceled:", normal failure → original message unchangeddebug_assert!(self.running.is_empty())to make the deadlock-branch invariant explicittracing::warn!for the impossible empty-failed+empty-canceled edge caseRoot cause
When the deadlock branch fired it set
GraphStatus::Failedbut never updated individual task statuses. The message formatter countedTaskStatus::Failedtasks (zero), producing "Plan failed. 0/N tasks failed:" — correct but misleading.Test plan
cargo +nightly fmt --check— passcargo clippy --workspace --features full -- -D warnings— passcargo nextest run --config-file .github/nextest.toml --workspace --features full --lib --bins— 5975/5975 passedtest_deadlock_marks_non_terminal_tasks_canceled,test_deadlock_not_triggered_when_task_running(scheduler.rs)finalize_plan_execution_deadlock_emits_cancelled_message,finalize_plan_execution_mixed_failed_and_cancelled(agent/tests.rs)Closes #1879