@@ -9,35 +9,37 @@ graph nodes and tracks dynamic nodes spawned by `ctx.run_node()`.
99Workflow manages two kinds of child nodes through a unified scheduler (` DynamicNodeScheduler ` ):
1010
1111- ** Static (graph) nodes** — declared in ` edges ` , compiled into a
12- ` Graph ` . Scanned upfront into ` _LoopState.recovered_executions ` by
13- ` ReplayManager.scan_workflow_events(ctx) ` . When scheduled by the
14- orchestration loop, ` _start_node_task ` checks ` check_interception ` on any
15- recovered execution to mark ` _LoopState.replayed_nodes ` and pre-populate
16- ` _LoopState.runs[node_path] ` ( ` DynamicNodeRun ` with ` is_static=True ` ), then
17- delegates to ` ctx._run_node_internal ` ( ` DynamicNodeScheduler ` ). Tracked in
18- ` _LoopState.nodes ` by node name ( ` NodeState ` ).
12+ ` Graph ` . Scanned upfront by ` ReplayManager.scan_workflow_events(ctx) ` and
13+ seeded into ` _LoopState.runs[node_path] ` ( ` DynamicNodeRun ` ) . When scheduled
14+ by the orchestration loop, ` _start_node_task ` marks the entry with
15+ ` is_static=True ` , points ` _LoopState.static_runs[node_name] ` at that same
16+ ` DynamicNodeRun ` , checks ` _classify_static_replay ` to populate
17+ ` _LoopState.replayed_nodes ` , and delegates to ` ctx._run_node_internal `
18+ ( ` DynamicNodeScheduler ` ).
1919- ** Dynamic nodes** — spawned at runtime via ` ctx.run_node() ` from
2020 inside a graph node's ` _run_impl ` . Lazily rehydrated from ` ReplayManager `
2121 by ` DynamicNodeScheduler._rehydrate_from_events ` and tracked in
2222 ` _LoopState.runs ` by full ` node_path ` (` DynamicNodeRun ` ,
23- which holds ` state: NodeState ` , ` output ` , ` task ` , ` transfer_to_agent ` ,
24- ` recovered_state ` , and ` is_static ` ). Managed by ` DynamicNodeScheduler ` .
23+ which holds ` state: NodeState ` , ` output ` , ` route ` , ` branch ` , ` task ` ,
24+ ` transfer_to_agent ` , ` recovered_state ` , and ` is_static ` ). Managed by
25+ ` DynamicNodeScheduler ` .
2526
26- Static and dynamic nodes share the same ` DynamicNodeScheduler ` , replay
27- interception (` check_interception ` / ` ReplayManager ` ), and
28- ` _LoopState.interrupt_ids ` set, so the Workflow sees a unified view of
29- execution and all pending interrupts.
27+ Static and dynamic nodes share the same ` DynamicNodeScheduler ` , ` DynamicNodeRun `
28+ outcome recording (` record_outcome ` ), replay interception
29+ (` check_interception ` / ` ReplayManager ` ), and ` _LoopState.interrupt_ids ` set,
30+ so the Workflow sees a single unified view of execution and all pending
31+ interrupts.
3032
3133## Implementing a graph node
3234
3335A graph node is a regular BaseNode placed in a Workflow's edges.
34- In ` _start_node_task ` , the Workflow checks ` check_interception ` against
35- ` _LoopState.recovered_executions ` (to record ` _LoopState.replayed_nodes ` and
36- seed ` _LoopState.runs ` with ` is_static=True ` ) and dispatches
36+ In ` _start_node_task ` , the Workflow classifies any pre-seeded ` DynamicNodeRun `
37+ in ` _LoopState.runs ` (to record ` _LoopState.replayed_nodes ` ), indexes it under
38+ ` _LoopState.static_runs[node_name] ` , and dispatches
3739` ctx._run_node_internal ` to ` DynamicNodeScheduler ` , which either returns a
3840replayed mock ` Context ` (` create_mock_context ` ) or wraps the node in a
39- ` NodeRunner ` with a child ` Context ` , reading ` ctx.output ` , ` ctx.route ` , and
40- ` ctx.interrupt_ids ` after it completes .
41+ ` NodeRunner ` with a child ` Context ` ; ` _handle_completion ` then records the
42+ final outcome on the shared ` DynamicNodeRun ` in deterministic loop order .
4143
4244** Output** — two paths. At most one per execution. The Workflow
4345reads the output to pass downstream.
@@ -158,13 +160,17 @@ Within a single step (`_execute_step`), when replay is enabled:
158160 concurrent ` run.task ` , or calls `check_interception(node=curr_node,
159161 recovered=run.recovered_state, current_run=run)`:
160162
161- 1 . ** Same-turn dedup / waiting (` current_run ` present and ` not current_run.is_static ` )** —
163+ 1 . ** Same-turn dedup / failed / waiting (` current_run ` present and ` not current_run.is_static ` )** —
162164 if ` current_run.state.status == COMPLETED ` , returns ` should_run=False ` with
163- ` current_run.output ` and ` current_run.transfer_to_agent ` . If ` WAITING ` with
164- ` interrupts ` , returns ` should_run=False ` with those ` interrupts ` . (Every
165- static dispatch gets a fresh ` run_id ` , so loop edges never reuse a
166- ` node_path ` ; static runs have ` is_static=True ` so no-outcome static nodes
167- fast-forward in step 7 via ` should_run = (current_run is not None and not current_run.is_static) ` .)
165+ ` current_run.output ` , ` current_run.route ` , and
166+ ` current_run.transfer_to_agent ` . If ` FAILED ` , returns ` should_run=True `
167+ with ` resume_inputs=recovered.resolved_responses ` so a dynamic run that
168+ failed in the current turn reruns instead of consulting stale recovered
169+ events. If ` WAITING ` with ` interrupts ` , returns ` should_run=False ` with
170+ those ` interrupts ` . (Every static dispatch gets a fresh ` run_id ` , so loop
171+ edges never reuse a ` node_path ` ; static runs have ` is_static=True ` so
172+ no-outcome static nodes fast-forward in step 7 via
173+ ` should_run = (current_run is not None and not current_run.is_static) ` .)
168174
1691752 . ** Nested ` Workflow ` (` isinstance(node, Workflow) ` )** — before checking
170176 completion or interrupts, ` check_interception ` immediately returns
@@ -175,7 +181,12 @@ Within a single step (`_execute_step`), when replay is enabled:
175181
1761823 . ** Waiting / Partial resume (` recovered.interrupt_ids - recovered.resolved_ids ` non-empty)** —
177183 - If ` node.rerun_on_resume ` is ` True ` and ` recovered.resolved_ids ` is
178- non-empty (** partial resume** ): returns ` should_run=True ` with
184+ non-empty (** partial resume** ): if ` recovered.finished_after_resume ` is
185+ ` True ` and the node either recorded a result (` recovered.output ` ,
186+ ` recovered.route ` , or ` recovered.transfer_to_agent ` ) or has
187+ ` not node.wait_for_output ` , fast-forwards (` should_run=False ` ) with the
188+ rehydrated ` output ` , ` route ` , and ` transfer_to_agent ` (and no
189+ ` interrupts ` ); otherwise returns ` should_run=True ` with
179190 ` resume_inputs=recovered.resolved_responses ` .
180191 - Otherwise: returns ` should_run=False ` with ` interrupts=unresolved ` ,
181192 propagating unresolved interrupt IDs to ` _state.interrupt_ids ` via a
@@ -201,9 +212,14 @@ Within a single step (`_execute_step`), when replay is enabled:
201212 ` resume_inputs=recovered.resolved_responses ` .
202213
2032147 . ** No output, route, or interrupts in recovered events** —
204- - If ` node.wait_for_output ` or ` node.rerun_on_resume ` : returns
215+ - If ` node.wait_for_output ` and the node is a ` BaseAgent ` (task or chat
216+ mode), or ` node.rerun_on_resume ` with ` wait_for_output ` : returns
205217 ` should_run=True ` with ` resume_inputs=recovered.resolved_responses ` .
206- - Otherwise: returns ` should_run = (current_run is not None and not current_run.is_static) `
218+ - Else if ` node.rerun_on_resume ` : fast-forwards (` should_run=False ` ) when
219+ ` recovered.finished_after_resume ` , otherwise returns ` should_run=True `
220+ with ` resume_inputs ` .
221+ - Otherwise (including any other ` wait_for_output ` node, whose next trigger
222+ starts a new run): returns ` should_run = (current_run is not None and not current_run.is_static) `
207223 (static nodes that completed with ` None ` output fast-forward with
208224 ` should_run=False ` , while dynamic nodes with no recorded outcome
209225 re-execute with ` should_run=True ` ).
@@ -212,7 +228,8 @@ Within a single step (`_execute_step`), when replay is enabled:
212228
213229When a dynamic child interrupts:
214230
215- 1 . ` DynamicNodeScheduler._record_result ` sets the child's ` DynamicNodeRun `
231+ 1 . ** Dynamic child interrupts** :
232+ ` DynamicNodeRun.record_outcome ` sets the child's ` DynamicNodeRun `
216233 status to WAITING and adds its interrupt IDs to
217234 ` _LoopState.interrupt_ids ` .
2182352 . ` ctx.run_node() ` checks ` child_ctx.interrupt_ids ` . If non-empty,
@@ -349,24 +366,30 @@ Key behaviors:
349366On every run, ` Workflow ` reconstructs progress from session events in three
350367stages:
351368
352- 1 . ** Upfront static scan (` ReplayManager.scan_workflow_events ` )** — during
369+ 1 . ** Upfront child scan (` ReplayManager.scan_workflow_events ` )** — during
353370 ` _run_impl ` SETUP, ` ReplayManager ` scans the invocation's events and
354- populates ` loop_state.recovered_executions ` (keyed by ` node_name@run_id ` )
355- with outputs, routes, interrupts, resolved function responses, error codes,
356- and transfers for direct static children. ` Workflow ` then seeds ` START `
371+ ` Workflow ` seeds ` _LoopState.runs[node_path] ` with ` DynamicNodeRun ` entries
372+ carrying outputs, routes, interrupts, resolved function responses, error
373+ codes, and transfers for direct children. ` Workflow ` then seeds ` START `
357374 triggers normally (` _seed_start_triggers ` ).
3583752 . ** Static node interception in ` _start_node_task ` and ` DynamicNodeScheduler ` ** —
359376 when ` _schedule_ready_nodes ` pops a trigger for a static node,
360- ` _start_node_task ` looks up ` recovered = loop_state.recovered_executions.get(f"{node_name}@{run_id}") ` :
361- - If ` recovered ` is present, ` _start_node_task ` calls
362- ` check_interception(node=node, recovered=recovered) ` to check whether the
377+ ` _start_node_task ` looks up or creates ` run = loop_state.runs[node_path] ` ,
378+ sets ` run.is_static = True ` , and registers ` loop_state.static_runs[node_name] = run ` :
379+ - If ` run.recovered_state ` is present, ` _start_node_task ` uses
380+ ` _classify_static_replay ` (` check_interception ` ) to determine whether the
363381 run will fast-forward (` not result.should_run ` and not a resolved transfer
364382 interrupt), adding ` node_name ` to ` loop_state.replayed_nodes ` (so
365383 checkpoint emission is skipped in ` _schedule_ready_nodes ` and on the
366384 ` COMPLETED ` path of ` _handle_completion ` ; ` _handle_completion ` still emits
367- a checkpoint if a replayed node comes back ` WAITING ` ) and seeding
368- ` loop_state.runs[node_path] ` with
369- ` DynamicNodeRun(state=NodeState(run_id=run_id), recovered_state=recovered, is_static=True) ` .
385+ a checkpoint if a replayed node comes back ` WAITING ` ).
386+ ` _at_concurrency_limit ` excludes ` replayed_nodes ` from the
387+ ` max_concurrency ` count. A resolved-transfer fast-forward is not in that
388+ set because it runs its transfer target once the replay barrier releases
389+ it, so under ` max_concurrency ` ` _schedule_ready_nodes ` keeps its trigger
390+ buffered (` _is_held_transfer ` ) until its barrier key is released and then
391+ counts it like any running node. If nothing else is pending, it starts
392+ one held node anyway so the barrier reports the divergence.
370393 - ` _start_node_task ` then dispatches ` ctx._run_node_internal ` to
371394 ` DynamicNodeScheduler._check_existing_run ` , which runs
372395 ` check_interception(node=curr_node, recovered=run.recovered_state, current_run=run) ` :
@@ -377,25 +400,30 @@ stages:
377400 - ** Completed non-` Workflow ` node (or ` rerun_on_resume=False ` with all interrupts resolved)** :
378401 returns ` should_run=False ` ; ` _check_existing_run ` builds a mock
379402 ` Context ` (` create_mock_context ` ) and awaits
380- ` ReplayManager.wait_sequence ` so ` _handle_completion ` can re-surface
381- replayed output (` _maybe_reemit_replayed_output ` ) and buffer downstream
382- triggers without running ` NodeRunner ` .
403+ ` ReplayManager.wait_sequence ` so ` _handle_completion ` can record the
404+ outcome on ` run ` , re-surface replayed output
405+ (` _maybe_reemit_replayed_output ` ), and buffer downstream triggers without
406+ running ` NodeRunner ` .
383407 - ** Interrupted with unresolved interrupts** : if ` rerun_on_resume=True `
384408 and ` recovered.resolved_ids ` is non-empty (** partial resume** ),
409+ fast-forwards (` should_run=False ` ) with the rehydrated ` output ` ,
410+ ` route ` , and ` transfer_to_agent ` when ` recovered.finished_after_resume `
411+ is ` True ` (and a result was recorded or ` not node.wait_for_output ` ), or
385412 re-executes (` should_run=True ` ) via ` NodeRunner ` with
386- ` resume_inputs=recovered.resolved_responses ` ; otherwise stays WAITING
387- (` should_run=False ` with ` interrupts ` populated) and records unresolved
388- IDs in ` _LoopState.interrupt_ids ` .
413+ ` resume_inputs=recovered.resolved_responses ` otherwise; if not a partial
414+ resume on a ` rerun_on_resume ` node, stays WAITING (` should_run=False `
415+ with ` interrupts ` populated) and records unresolved IDs in
416+ ` _LoopState.interrupt_ids ` .
389417 - ** Interrupted with all interrupts resolved (` rerun_on_resume=True ` ) or prior failure (` error_code ` )** :
390418 re-executes (` should_run=True ` ) via ` NodeRunner ` with
391419 ` resume_inputs=recovered.resolved_responses ` .
392- 3 . ** Dynamic node lazy rehydration (` _rehydrate_from_events ` )** — dynamic
393- children are not scanned upfront into ` recovered_executions ` . When a
394- re-executing parent calls ` ctx.run_node() ` ,
395- ` DynamicNodeScheduler._execute_step ` lazily populates
396- ` _LoopState.runs[node_path] ` via ` _rehydrate_from_events ` and applies the
397- same ` check_interception ` rules in ` _check_existing_run ` (with
398- ` current_run=run ` also deduplicating same-turn completed or waiting runs).
420+ 3 . ** Dynamic node lazy rehydration (` _rehydrate_from_events ` )** — descendant
421+ dynamic children not already present in ` _LoopState.runs ` are lazily
422+ rehydrated when a re-executing parent calls ` ctx.run_node() ` .
423+ ` DynamicNodeScheduler._execute_step ` populates ` _LoopState.runs[node_path] `
424+ via ` _rehydrate_from_events ` and applies the same ` check_interception ` rules
425+ in ` _check_existing_run ` (with ` current_run=run ` also deduplicating
426+ same-turn completed or waiting runs).
399427
400428## Key design rules for node authors
401429
0 commit comments