Skip to content

Commit 9c01b02

Browse files
committed
fix(workflow): record runs that finish without a result so resume replays them
A node run that finished with no output, route or transfer and emitted no event left nothing in the session showing it ran, so resuming after a RequestInput ran it (and its side effects) again. The same applied to ctx.run_node children and to children of a retried sub-workflow. NodeRunner now emits a completion marker (NodeInfo.completed) for such a run. The resume scan records it on the child's scan state, and check_interception fast-forwards a dynamic run that has one instead of running it again. Fixes google#7666
1 parent 9687bae commit 9c01b02

9 files changed

Lines changed: 247 additions & 11 deletions

File tree

‎docs/guides/workflow/retry_config/index.md‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -103,7 +103,7 @@ inner = Workflow(
103103
)
104104
```
105105

106-
A retried sub-workflow does not redo work it already finished. Children that produced an output or a state change are replayed from the session's events, so only the failed node and whatever follows it run again. A child that produced neither leaves nothing to replay and runs again on every attempt, so give a side-effecting node an output or a state change if repeating it would be harmful.
106+
A retried sub-workflow does not redo work it already finished. Finished children are replayed from the session's events, so only the failed node and whatever follows it run again. This includes a child that finished without an output or a state change: ADK records that it finished, so its side effects are not repeated.
107107

108108
Setting `RetryConfig` on the node that can fail is still the narrower option, and it retries just that node rather than the whole sub-workflow.
109109

@@ -124,7 +124,6 @@ test_retry_config = RetryConfig(
124124

125125
- **The attempt counter does not survive an interrupt.** It is **not** persisted. If the workflow is interrupted, whether by a human-in-the-loop input downstream or by the application restarting, and is later resumed, a node that has to run again starts counting from 1 with its full budget of attempts back.
126126
- **Retries are local to the node.** They happen inside that node's execution and nowhere else. Once a node has used up its attempts it enters the `FAILED` state, and the workflow execution fails with it, or follows whatever error handling you have configured.
127-
- **A retried `Workflow` runs some finished children again.** A child that produced no output and no state change runs again, because the replay has no record that it completed. See [Retrying a nested workflow](#retrying-a-nested-workflow).
128127
- **A cancelled child can be replayed as finished.** A child that was cancelled right after it emitted an event without an output, such as a progress message, looks finished to the replay. The retry then hands back no output for it instead of running it again. A parallel worker item that is cancelled because another item failed is the usual case.
129128

130129
## Related samples

‎src/google/adk/events/event.py‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,15 @@ class NodeInfo(BaseModel):
7575
carries the output value.
7676
"""
7777

78+
completed: bool | None = None
79+
"""When True, the node run at ``path`` finished without emitting any other
80+
event and without an output, route or transfer.
81+
82+
Such a run leaves no other trace in the session, so this event records that
83+
it finished. On resume the run is replayed from history instead of running
84+
again.
85+
"""
86+
7887
@property
7988
def run_id(self) -> str:
8089
"""The run ID of the node that generated the event."""

‎src/google/adk/workflow/_node_runner.py‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -142,6 +142,11 @@ def __init__(
142142
self._prior_output = prior_output
143143
self._prior_interrupt_ids = prior_interrupt_ids
144144

145+
# Whether the current attempt has put an event of its own in the session.
146+
self._emitted_in_attempt = False
147+
# Whether the current attempt stopped on a child that is still waiting.
148+
self._interrupted_in_attempt = False
149+
145150
@property
146151
def run_id(self) -> str:
147152
"""The run ID assigned to this node run."""
@@ -163,6 +168,8 @@ async def run(
163168
ctx = self._create_child_context(
164169
resume_inputs, attempt_count=attempt_count
165170
)
171+
self._emitted_in_attempt = False
172+
self._interrupted_in_attempt = False
166173
# A retry after a dynamic child failure rewinds to this point, so it
167174
# replays only the children this attempt completed.
168175
counters_at_start = self._snapshot_dynamic_children(ctx)
@@ -182,6 +189,7 @@ async def run(
182189
ctx.error, attempt_count
183190
):
184191
await self._flush_output_and_deltas(ctx)
192+
await self._record_completion_without_result(ctx)
185193
logger.debug("node %s end.", ctx.node_path)
186194
return ctx
187195
except asyncio.CancelledError:
@@ -425,6 +433,7 @@ async def _execute_node(
425433
# A dynamic child interrupted via ctx.run_node().
426434
# Tool calls have their own Context, so copy their interrupt IDs too.
427435
ctx._interrupt_ids.update(exc.interrupt_ids)
436+
self._interrupted_in_attempt = True
428437

429438
async def _run_node_loop(self, ctx: Context, node_input: Any) -> None:
430439
"""Iterate node.run(), track events in context, and enqueue them."""
@@ -503,6 +512,7 @@ async def _enqueue_event(self, event: Event, ctx: Context) -> None:
503512
if not event.partial:
504513
self._flush_deltas(event, ctx)
505514
await ctx._invocation_context._enqueue_event(event)
515+
self._emitted_in_attempt = True
506516

507517
if event.output is not None:
508518
ctx._output_emitted = True
@@ -537,11 +547,47 @@ async def _flush_output_and_deltas(self, ctx: Context) -> None:
537547

538548
self._enrich_event(event, ctx)
539549
await ctx._invocation_context._enqueue_event(event)
550+
self._emitted_in_attempt = True
540551
if has_deferred_output:
541552
ctx._output_emitted = True
542553
if has_unflushed_route:
543554
ctx._route_emitted = True
544555

556+
async def _record_completion_without_result(self, ctx: Context) -> None:
557+
"""Emit a completion marker for a run that finished leaving no trace.
558+
559+
A run that emitted no event and has no output, route or transfer leaves
560+
nothing in the session that shows it ran, so resume would run it (and its
561+
side effects) again. The marker lets the scan replay it instead.
562+
"""
563+
from ..events.event import Event
564+
from ._workflow import Workflow
565+
566+
if (
567+
self._emitted_in_attempt
568+
or self._interrupted_in_attempt
569+
# A nested workflow always reruns and replays its own children.
570+
or isinstance(self._node, Workflow)
571+
or ctx.error is not None
572+
or ctx.interrupt_ids
573+
or ctx.output is not None
574+
or ctx._output_delegated
575+
or ctx.route is not None
576+
or ctx.actions.transfer_to_agent is not None
577+
# Finishing without output means a wait_for_output node is still
578+
# waiting for more input, not that it is done.
579+
or self._node.wait_for_output
580+
# Tests wrap Context around invocation-context stubs that lack
581+
# is_aborted or return a truthy Mock; neither means aborted.
582+
or getattr(ctx._invocation_context, "is_aborted", False) is True
583+
):
584+
return
585+
586+
event = Event()
587+
event.node_info.completed = True
588+
self._enrich_event(event, ctx)
589+
await ctx._invocation_context._enqueue_event(event)
590+
545591
def _flush_deltas(self, event: Event, ctx: Context) -> None:
546592
"""Move pending state/artifact deltas from ctx onto the event."""
547593
from ..events.event_actions import EventActions

‎src/google/adk/workflow/utils/_rehydration_utils.py‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,6 +94,14 @@ class _ChildScanState:
9494
finished_after_resume: bool = False
9595
"""Whether the child emitted a direct completion event since its last answer (or start)."""
9696

97+
completed: bool = False
98+
"""Whether the child's run recorded that it finished without a result.
99+
100+
Set from a completion marker (see ``_is_completion_marker``). Unlike
101+
``finished_after_resume``, which any direct non-interrupt event sets, this
102+
is only set once the run has actually ended.
103+
"""
104+
97105

98106
def _wrap_response(value: Any) -> dict[str, Any]:
99107
"""Wraps a value into a dict suitable for FunctionResponse.response.
@@ -551,6 +559,11 @@ def get_owner_key(event_path_builder: _NodePathBuilder) -> str | None:
551559
if schema_json:
552560
schemas_by_id[interrupt_id] = schema_json
553561

562+
if interrupt_ids_to_process:
563+
child.completed = False
564+
elif is_direct and _is_completion_marker(event):
565+
child.completed = True
566+
554567
if is_direct:
555568
child.finished_after_resume = (
556569
not interrupt_ids_to_process
@@ -563,6 +576,11 @@ def get_owner_key(event_path_builder: _NodePathBuilder) -> str | None:
563576
return scan_states
564577

565578

579+
def _is_completion_marker(event: Event) -> bool:
580+
"""Returns whether ``event`` records a run that finished without a result."""
581+
return bool(event.node_info and getattr(event.node_info, 'completed', None))
582+
583+
566584
def _event_has_result(event: Event) -> bool:
567585
"""Returns whether ``event`` carries a node output, route, or transfer."""
568586
return (

‎src/google/adk/workflow/utils/_replay_interceptor.py‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -191,9 +191,14 @@ def check_interception(
191191
should_run = True
192192
resume_inputs = recovered.resolved_responses
193193
else:
194-
# Allow fresh execution for crashed/timeout dynamic nodes;
195-
# static nodes with no outcome (e.g. return None) should be fast-forwarded.
196-
should_run = current_run is not None and not current_run.is_static
194+
# Allow fresh execution for crashed/timeout dynamic nodes, but not for
195+
# dynamic nodes that recorded finishing without a result; static nodes
196+
# with no outcome (e.g. return None) should be fast-forwarded.
197+
should_run = (
198+
current_run is not None
199+
and not current_run.is_static
200+
and not recovered.completed
201+
)
197202

198203
return InterceptionResult(
199204
should_run=should_run,

‎tests/unittests/workflow/test_workflow_dynamic_nodes.py‎

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -773,9 +773,6 @@ async def parent(*, ctx, node_input):
773773

774774

775775
@pytest.mark.asyncio
776-
@pytest.mark.xfail(
777-
reason='No completion marker event for None output (event design flaw)'
778-
)
779776
async def test_dynamic_node_none_output_not_rerun():
780777
"""Dynamic child that completed with None output is not re-run.
781778
@@ -786,9 +783,6 @@ async def test_dynamic_node_none_output_not_rerun():
786783
Assert:
787784
- On Run 3, A should NOT re-run (it already completed).
788785
- A should return None (cached), B resumes.
789-
- Currently fails because A's None completion leaves no
790-
trace in session events — the lazy scan thinks A still
791-
needs to re-run.
792786
"""
793787
run_count_a = [0]
794788

‎tests/unittests/workflow/test_workflow_failures.py‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1294,6 +1294,38 @@ def flaky_node(ctx: Context, node_input: Any):
12941294
assert completed_attempts == 1
12951295

12961296

1297+
@pytest.mark.asyncio
1298+
async def test_retry_config_on_nested_workflow_replays_child_without_output():
1299+
"""A retried sub-workflow does not redo a child that finished with no output."""
1300+
side_effect_attempts = 0
1301+
flaky_attempts = 0
1302+
1303+
@node()
1304+
def side_effect_node(ctx: Context):
1305+
nonlocal side_effect_attempts
1306+
side_effect_attempts += 1
1307+
1308+
@node()
1309+
def flaky_node(ctx: Context, node_input: Any):
1310+
nonlocal flaky_attempts
1311+
flaky_attempts += 1
1312+
if flaky_attempts < 3:
1313+
raise CustomError('Node failed')
1314+
yield 'recovered'
1315+
1316+
inner_wf = Workflow(
1317+
name='inner_wf',
1318+
edges=[(START, side_effect_node, flaky_node)],
1319+
retry_config=RetryConfig(max_attempts=3, initial_delay=0.0, jitter=0.0),
1320+
)
1321+
outer_wf = Workflow(name='outer_wf', edges=[(START, inner_wf)])
1322+
1323+
await _run_workflow(outer_wf)
1324+
1325+
assert flaky_attempts == 3
1326+
assert side_effect_attempts == 1
1327+
1328+
12971329
@pytest.mark.asyncio
12981330
async def test_retry_config_on_outer_workflow_retries_nested_failure():
12991331
"""A failure inside a sub-workflow is a failure of the outer workflow too."""

‎tests/unittests/workflow/test_workflow_hitl.py‎

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3081,3 +3081,116 @@ def collect(node_input: dict[str, Any]) -> str:
30813081
assert b_runs == 1
30823082
outputs3 = [e.output for e in events3 if e.output is not None]
30833083
assert outputs3[-1] == 'approved:True|from_b'
3084+
3085+
3086+
async def _run_and_answer_first_request(
3087+
runner: testing_utils.InMemoryRunner,
3088+
) -> list[Event]:
3089+
"""Runs the workflow once, then answers the request it paused on."""
3090+
events1 = await runner.run_async('start')
3091+
[interrupt_id] = [
3092+
interrupt_id
3093+
for event in workflow_testing_utils.get_request_input_events(events1)
3094+
for interrupt_id in extract_event_interrupt_ids(event)
3095+
]
3096+
return await runner.run_async(
3097+
new_message=testing_utils.UserContent(
3098+
create_request_input_response(interrupt_id, {'ok': True})
3099+
),
3100+
invocation_id=events1[0].invocation_id,
3101+
)
3102+
3103+
3104+
def _app(name: str, root: Workflow, resumable: bool) -> App:
3105+
return App(
3106+
name=name,
3107+
root_agent=root,
3108+
resumability_config=(
3109+
ResumabilityConfig(is_resumable=True) if resumable else None
3110+
),
3111+
)
3112+
3113+
3114+
@pytest.mark.parametrize('resumable', [False, True])
3115+
@pytest.mark.asyncio
3116+
async def test_node_returning_none_is_not_rerun_on_resume(resumable: bool):
3117+
"""A node that finished without output is replayed, not run again."""
3118+
emails_sent = []
3119+
3120+
def send_email():
3121+
emails_sent.append('email')
3122+
3123+
def ask():
3124+
return RequestInput(message='Approve?')
3125+
3126+
def finish(node_input: Any):
3127+
return node_input
3128+
3129+
wf = Workflow(name='wf', edges=[(START, send_email, ask, finish)])
3130+
runner = testing_utils.InMemoryRunner(
3131+
app=_app('none_output_seq', wf, resumable)
3132+
)
3133+
3134+
events2 = await _run_and_answer_first_request(runner)
3135+
3136+
assert emails_sent == ['email']
3137+
assert [
3138+
e.output for e in events2 if e.node_info.name == 'finish' and e.output
3139+
] == [{'ok': True}]
3140+
3141+
3142+
@pytest.mark.parametrize('resumable', [False, True])
3143+
@pytest.mark.asyncio
3144+
async def test_parallel_node_returning_none_is_not_rerun_on_resume(
3145+
resumable: bool,
3146+
):
3147+
"""A parallel branch that finished without output is not run again."""
3148+
emails_sent = []
3149+
3150+
def send_email():
3151+
emails_sent.append('email')
3152+
3153+
def ask():
3154+
return RequestInput(message='Approve?')
3155+
3156+
wf = Workflow(name='wf', edges=[(START, send_email), (START, ask)])
3157+
runner = testing_utils.InMemoryRunner(
3158+
app=_app('none_output_parallel', wf, resumable)
3159+
)
3160+
3161+
await _run_and_answer_first_request(runner)
3162+
3163+
assert emails_sent == ['email']
3164+
3165+
3166+
@pytest.mark.parametrize('resumable', [False, True])
3167+
@pytest.mark.asyncio
3168+
async def test_dynamic_child_returning_none_is_not_rerun_on_resume(
3169+
resumable: bool,
3170+
):
3171+
"""A ctx.run_node child that finished without output is not run again."""
3172+
emails_sent = []
3173+
3174+
@node
3175+
def send_email():
3176+
emails_sent.append('email')
3177+
3178+
@node
3179+
def ask():
3180+
return RequestInput(message='Approve?')
3181+
3182+
@node(rerun_on_resume=True)
3183+
async def parent(*, ctx: Context, node_input: Any):
3184+
await ctx.run_node(send_email)
3185+
answer = await ctx.run_node(ask)
3186+
yield answer
3187+
3188+
wf = Workflow(name='wf', edges=[(START, parent)])
3189+
runner = testing_utils.InMemoryRunner(
3190+
app=_app('none_output_dynamic', wf, resumable)
3191+
)
3192+
3193+
events2 = await _run_and_answer_first_request(runner)
3194+
3195+
assert emails_sent == ['email']
3196+
assert {'ok': True} in [e.output for e in events2 if e.output is not None]

‎tests/unittests/workflow/utils/test_replay_interceptor.py‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -396,6 +396,26 @@ def test_cross_turn_no_outcome_waiting_agent_or_rerun_node_reruns(
396396
assert result.should_run
397397

398398

399+
def test_cross_turn_dynamic_run_recorded_completed_fast_forwards():
400+
"""Case 6 replays a dynamic run whose completion marker was recorded."""
401+
recovered = _ChildScanState(run_id='1', completed=True)
402+
node = BaseNode(name='node', rerun_on_resume=False, wait_for_output=False)
403+
dynamic_run = DynamicNodeRun(
404+
state=NodeState(run_id='1'),
405+
recovered_state=recovered,
406+
is_static=False,
407+
)
408+
409+
result = check_interception(
410+
node=node,
411+
recovered=recovered,
412+
current_run=dynamic_run,
413+
)
414+
415+
assert not result.should_run
416+
assert result.output is None
417+
418+
399419
# --- create_mock_context ---
400420

401421

0 commit comments

Comments
 (0)