Fix pipeline stress test: stale completions and busy state#282
Merged
Conversation
Gate _emit_node_state on is_generation_current() to prevent stale task completions from corrupting DAG node state. Check both has_pending_work() and has_tasks() in is_busy since they can diverge when artifact_created events arrive before task finalization. Skip process_graph() for stale generation completions. Add _busy_reason() diagnostic for settle-phase debugging.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extracted pipeline-only fixes from #278 that address the
test_pipeline_stress_random_invalidationsstress test failures._emit_node_stateonis_generation_current()— prevents stale task completions from corrupting DAG node state (bothstep_stage.pyandworkpiece_stage.py)has_pending_work()andhas_tasks()inis_busy—artifact_createdevents can set DAG nodes to VALID before the worker sends its final "done" message, causinghas_pending_work()to return False while tasks are still liveprocess_graph()for stale generation completions — avoids processing work from outdated data generations_busy_reason()diagnostic — logs why the pipeline reports busy during settle phase, for easier debuggingtest_pipeline_context.py— reflects the newis_busysemantics (active context only, not all contexts)Tested: stress test passes locally (~2min), all pipeline context tests pass.
Note: the full stress test reliability also depends on the tasker/worker pool changes in #278 (stuck worker detection, early cancel). This PR contains the pipeline-side fixes that are independent and useful on their own.