Repository navigation
Description
Cross-repo note: Proposes a change to
google/adk-python; filed here for tracking. Stage 1 of google/adk-python#6529 (NODE_OUTPUT/NODE_ERROR) landed in04b8b727on 2026-08-10. This RFC was re-verified against upstreammainat3f21e891on 2026-08-11. A prototype exists in caohy1988/adk-python#8, but the review below changes its event contract; the branch should not be rebased unchanged. The proposal remains plugin-only and additive: noworkflow/execution change and no BigQuery schema-column change.
RFC: Workflow-node observation windows (WORKFLOW_NODE_FIRST_OBSERVED / WORKFLOW_NODE_OBSERVATION_SUMMARY) for the BigQuery Agent Analytics Plugin
A producer-side event-observation contract for workflow nodes, deliberately separate from framework-exact node lifecycle events.
The BQAA plugin's contract is that the BigQuery row is the contract. Stage 1 made workflow-node results visible. This RFC standardizes what the plugin can prove from
on_event_callback: which node path it first observed during one runner call, the first/last source events it observed, whether it observed interrupt or node-error evidence, and which terminal plugin callback drained the segment. It does not call those facts node start, completion, failure, or wall-time latency.
Summary
Add two event types, two typed views, and one plugin-local segment key:
WORKFLOW_NODE_FIRST_OBSERVED— emitted immediately on the first event with a non-emptynode_info.pathobserved during one runner call. It carries the existing C1 node envelope, the firstsource_event_id,boundary_basis='event_observation', and a newworkflow_segment_id.WORKFLOW_NODE_OBSERVATION_SUMMARY— emitted from whichever terminal callback the runner reaches (after_run_callbackoron_run_error_callback) for every node observed in that runner call. It carries first/last source-event IDs and timestamps, event count, signed source-event span, direct/descendant interrupt evidence, node-error observation count, drain callback, and the same segment ID.workflow_segment_ididentifies a runner call. Current resumable ADK execution can reuseinvocation_idacross pause/resume calls. The plugin therefore creates a UUID inbefore_run_callback;(workflow_segment_id, node_path)is the pair key.invocation_idremains the logical invocation identifier and may span multiple segments.- Interrupt evidence is positive data. A non-empty
long_running_tool_idsobservation is sticky for that node during the segment, matchingNodeRunner's additive interrupt set. A tracked ancestor recordsdescendant_interrupt_observed=true. The producer does not encode pause as an absent row. - Terminal callback is not node outcome.
after_run_callbackalso runs when a caller stops consuming the async stream and the root task is cancelled;on_run_error_callbackcan represent node and non-node failures. The summary recordsdrain_callback='after_run' | 'on_run_error'without translating either intocompletedorfailed. - No back-dated rows. Synthetic row
timestampremains emission time. Source-event times live in explicit typed-view columns, so ingestion order, row time, and represented source time are not conflated. - Invocation-local state with stale-owner protection.
before_run_callbackalways installs a fresh accumulator and segment UUID. Both terminal callbacks drain and clear it infinally; a laterbefore_run_callbackreplaces stale state if cancellation or abandonment skipped terminal cleanup.
The summary view exposes the event span without a join:
SELECT
app_name,
session_id,
invocation_id,
workflow_segment_id,
node_path,
node_run_id,
source_event_span_ms,
node_interrupt_observed,
descendant_interrupt_observed,
node_error_observed_count,
drain_callback
FROM v_workflow_node_observation_summaryFor live diagnostics, a FIRST_OBSERVED row without a summary means only summary not observed yet. It must not be presented as proof that a node is running, paused, or stuck: the terminal row may be delayed, filtered, dropped, or never emitted.
Problem
| Approach | What it can truthfully provide | Silent nodes? | Cost / constraint |
|---|---|---|---|
Stage 1 only (NODE_OUTPUT / NODE_ERROR) |
Result/error points | No | Already landed |
| Canonical aggregate view over existing BQAA rows | First/last timestamps for rows the plugin already materializes | No | Cheapest; misses source events that produce no BQAA row, has no stable runner-call key when resumable calls reuse invocation_id, and cannot recover discarded interrupt evidence |
Existing invoke_node / invoke_workflow OpenTelemetry spans |
Exact execution timing in principle | Yes, when spans are recorded | Global OTel plumbing rather than a BQAA plugin contract; plain-node spans omit node.path / run_id, and silent nodes have no associated event IDs for correlation |
Public before_node / after_node / on_node_error callbacks around NodeRunner |
Framework-exact lifecycle and outcome | Yes | Requires a framework API and explicit retry/resume/replay/cancellation semantics |
| This RFC — producer-side observation window | All events seen by on_event_callback, runner-call segmentation, positive interrupt/error evidence |
No | Up to two extra rows per observed node plus invocation-local state |
The view-only alternative is the baseline, not a strawman. Choose this RFC only if consumers need at least one fact unavailable from existing persisted rows: complete on_event_callback observation coverage, an explicit runner-call segment key, or sticky interrupt evidence. If those facts are not needed, ship only the aggregate view.
Why this matters
- One producer definition of an observation window. Multi-event generator, streaming, and agent-backed nodes get consistent first/last source-event provenance without every dashboard inventing its own aggregation.
- Resumable runner calls become separable.
workflow_segment_idprevents the many-to-many joins produced by(invocation_id, node_path)when the same invocation and node run resume across multiple calls. - Interrupt evidence becomes queryable without an anti-join. Consumers can distinguish a directly observed interrupt, an affected tracked ancestor, and a segment with no interrupt evidence.
- The exact lifecycle vocabulary stays trustworthy.
WORKFLOW_NODE_STARTING/WORKFLOW_NODE_COMPLETEDremain available for future framework callbacks that observe the real NodeRunner boundary.
This RFC does not close the FunctionNode wall-time request in #6529. That remains the exact-lifecycle follow-up below.
Positioning & priority (honest valuation — read this first)
This is workflow-node event-span analytics, not node-latency instrumentation.
-
A single-output FunctionNode has a zero source-event span. The function may run for minutes before emitting its only output; none of that time is observable here.
-
First/last event is not start/end. Queue time, work before the first event, and work after the last event are absent.
-
Silent nodes remain invisible. This includes a node that emits no event and a
wait_for_output=Truenode that reachesWAITINGwithout emitting anything. -
WAITINGis not fully observable. An event-emitting node can later becomeWAITINGwithoutlong_running_tool_ids; the summary correctly says no interrupt was observed, not that the node completed. -
Replay is observation, not execution.
_maybe_reemit_replayed_outputcan enqueue a fresh event for a fast-forwarded node that did not execute in the new runner call. No current event field identifies that case. -
Agent nodes can have two telemetry families. A workflow
Agentmay emit exactAGENT_*rows and observation-derivedWORKFLOW_NODE_*rows. Consumers must not add their durations. -
Source-event timestamps can tie or regress. A one-event span is zero; custom/replayed timestamps can make the signed span negative. The view preserves the source facts rather than clamping them.
-
A missing summary is ambiguous. Delivery loss, filtering, cancellation, abandoned generators, and an in-progress call can all produce the same stored shape. Only positive summary fields are state claims.
-
Default row cost follows existing BQAA filtering. With no event allow/deny configuration, both new types are enabled and add up to two rows per observed node per runner call. Operators can deny either type; doing so intentionally weakens the paired contract.
-
Strategic value: MEDIUM. The contract standardizes graph-event analytics and segment identity but does not provide the FunctionNode latency originally requested.
-
Immediate value: HIGH for multi-event and interrupting nodes; LOW for ordinary single-output FunctionNodes. Before implementation, name at least one production query or dashboard that needs facts unavailable from the canonical aggregate-view baseline.
-
Risk: LOW-MEDIUM. No table schema migration, but row volume and invocation-local state grow with the number of distinct node paths observed in one runner call, and summary rows drain in one terminal burst.
Row contract
Both typed views retain the common columns (timestamp, event_id, session_id, invocation_id, trace/span IDs, status, and error fields) and add:
| View | Additional columns |
|---|---|
v_workflow_node_first_observed |
app_name, workflow_segment_id, node_path, node_run_id, node_parent_run_id, source_event_id, source_event_timestamp, boundary_basis |
v_workflow_node_observation_summary |
app_name, workflow_segment_id, node identity, first_source_event_id, last_source_event_id, first_source_event_timestamp, last_source_event_timestamp, source_event_span_ms, observed_event_count, node_interrupt_observed, descendant_interrupt_observed, node_error_observed_count, drain_callback, boundary_basis |
All added fields ride in existing JSON payload columns and are projected by _EVENT_VIEW_DEFS; there is no top-level BigQuery schema migration. boundary_basis='event_observation' appears on both event types so a future exact lifecycle contract never silently mixes with this one.
Key Design Decisions
- Use observation names, not lifecycle names. Public analytics event names are effectively irreversible. Documentation cannot make
STARTINGmean “possibly the final output event,” and a future producer must not change the semantics of an existing event type from approximate to exact. - Create a plugin segment ID in
before_run_callback.invocation_idis not a runner-call boundary in resumable execution. A random UUID is identity only; segment chronology comes from the associatedFIRST_OBSERVED/summary row timestamps. - Aggregate every path-bearing event before event-type filtering. The accumulator sees every event delivered to this plugin's
on_event_callback, even when that event produces no ordinary BQAA row. Filtering controls written rows, not what “observed” means. - Keep row time and source-event time separate.
timestampanswers when the synthetic row was emitted. Explicit source-event timestamp columns answer what source interval the summary represents. NoEventData.timestamp_overrideis needed. - Emit summaries for interrupt-observed nodes. Positive direct/descendant flags are more reliable than absence. A summary is not a completion row and does not close a logical node run across resume segments.
- Count node errors without attributing terminal failure.
node_error_observed_countis factual. ANODE_ERRORmay be a retryable attempt, and the escaped run error loseserror_node_path; matching exception type/message cannot prove which parallel node caused termination. - Deterministic drain order is first-observation order. Summary rows are enqueued before the plugin's
INVOCATION_COMPLETEDorINVOCATION_ERRORrow when the corresponding callback is reached. - Never synthesize an empty-path node. The default
node_info.path == ''remains outside this contract. - Preserve delivery semantics. Each synthetic row gets the existing per-row
event_idbefore enqueue; retry of that queued row preserves the ID. Consumers in default write mode still apply the documentedevent_iddeduplication rule.
Verification plan
- Segmentation: a resumable pause/resume that reuses
invocation_idandnode.pathproduces distinctworkflow_segment_idvalues and no cross-segment join. - Observation: first path-bearing event emits one
FIRST_OBSERVED; later events update the summary only; empty-path events produce neither type; accumulation happens before source-event filtering. - Summary: one-event node reports count 1 and span 0; multi-event node preserves first/last IDs and timestamps; regressing timestamps remain negative; row
timestampstays drain time. - Interrupts: evidence is sticky; direct leaf and tracked strict ancestors are distinguished; siblings remain false; every tracked node still receives a summary.
- Errors/retries: each qualifying node error increments the count; successful retry remains an observation count, not a failed status; parallel identical exceptions never produce node-failure attribution.
- Terminal paths: normal return, consumer early-close/
GeneratorExit, and escapedExceptionrecord the actual callback that drained. Cancellation/abandonment that reaches neither callback produces no summary; the nextbefore_run_callbackreplaces stale state. - Replay: fast-forward output re-emission is tested/documented as an observation with no execution claim.
- Overlap: workflow Agent nodes can produce both
AGENT_*and observation rows without changing existing events. - Isolation/scale: concurrent task invocations do not share state; sequential same-
invocation_idrunner calls reset segment ownership; looped node paths make the state/row bound explicit; terminal drain order is stable. - Views/config: exact projected column lists above;
boundary_basison both; event allow/deny behavior; payload-column denylist follows existing view projection; syntheticevent_idsurvives batch retry. - Regression: non-workflow invocations produce no observation rows.
Deferred (each needs its own design decision)
- Framework-exact node lifecycle. Add node-native callbacks (or another supported exporter contract) around
NodeRunnerwith defined retry, resume, replay, cancellation, and short-circuit semantics. Only this can truthfully emitWORKFLOW_NODE_STARTING/WORKFLOW_NODE_COMPLETEDand measure wall time, including silent nodes. workflow_node_status='failed'. Current upstream now has a reliable invocation-error callback, but not reliable node attribution:DynamicNodeFailErroris unwrapped beforeon_run_error_callback,error_node_pathis lost, retry attempts emitNODE_ERROR, and parallel nodes may raise identical exception type/message pairs. Reserve the value until the terminal signal carries node-path or source-event causality.- Replay attribution. Mark
_maybe_reemit_replayed_outputand other fast-forward events so consumers can separate execution from observation replay. - All
WAITINGstates.wait_for_output=Truewithout interrupt IDs is internal workflow state, not an event-visible fact. - Imperative in-node tool instrumentation. Tools called directly inside a
FunctionNodebody still do not traverse normal tool callbacks. - Cross-segment logical-run stitching. A paused node can reuse
invocation_id, path, and run ID across segment IDs. Logical wall duration requires a pause-exclusion policy and explicit segment ordering.
Implementation gate
Before reviving the prototype, validate one concrete consumer query against real Stage 1 data and the canonical aggregate-view baseline. Proceed with producer rows only if the query requires the new segment key, complete on_event_callback coverage, or positive interrupt evidence. Otherwise ship the view-only alternative and leave the exact lifecycle work for the node-callback RFC.
Links
- Upstream report and lifecycle follow-up context: google/adk-python#6529
- Stage 1 landing:
google/adk-python@04b8b727 - Upstream code reviewed for this revision:
google/adk-python@3f21e891 - Prototype that requires contract changes: caohy1988/adk-python#8
Activity
[-]RFC: Workflow-node boundary events (WORKFLOW_NODE_STARTING / WORKFLOW_NODE_COMPLETED) — Stage 2 of adk-python#6529[/-][+]RFC: Workflow-node observation windows — plugin-only Stage 2A of adk-python#6529[/+]