Skip to content

RFC: Workflow-node observation windows — plugin-only Stage 2A of adk-python#6529 #98

Open

Description

@caohy1988

Cross-repo note: Proposes a change to google/adk-python; filed here for tracking. Stage 1 of google/adk-python#6529 (NODE_OUTPUT / NODE_ERROR) landed in 04b8b727 on 2026-08-10. This RFC was re-verified against upstream main at 3f21e891 on 2026-08-11. A prototype exists in caohy1988/adk-python#8, but the review below changes its event contract; the branch should not be rebased unchanged. The proposal remains plugin-only and additive: no workflow/ execution change and no BigQuery schema-column change.

RFC: Workflow-node observation windows (WORKFLOW_NODE_FIRST_OBSERVED / WORKFLOW_NODE_OBSERVATION_SUMMARY) for the BigQuery Agent Analytics Plugin

A producer-side event-observation contract for workflow nodes, deliberately separate from framework-exact node lifecycle events.

The BQAA plugin's contract is that the BigQuery row is the contract. Stage 1 made workflow-node results visible. This RFC standardizes what the plugin can prove from on_event_callback: which node path it first observed during one runner call, the first/last source events it observed, whether it observed interrupt or node-error evidence, and which terminal plugin callback drained the segment. It does not call those facts node start, completion, failure, or wall-time latency.


Summary

Add two event types, two typed views, and one plugin-local segment key:

  1. WORKFLOW_NODE_FIRST_OBSERVED — emitted immediately on the first event with a non-empty node_info.path observed during one runner call. It carries the existing C1 node envelope, the first source_event_id, boundary_basis='event_observation', and a new workflow_segment_id.
  2. WORKFLOW_NODE_OBSERVATION_SUMMARY — emitted from whichever terminal callback the runner reaches (after_run_callback or on_run_error_callback) for every node observed in that runner call. It carries first/last source-event IDs and timestamps, event count, signed source-event span, direct/descendant interrupt evidence, node-error observation count, drain callback, and the same segment ID.
  3. workflow_segment_id identifies a runner call. Current resumable ADK execution can reuse invocation_id across pause/resume calls. The plugin therefore creates a UUID in before_run_callback; (workflow_segment_id, node_path) is the pair key. invocation_id remains the logical invocation identifier and may span multiple segments.
  4. Interrupt evidence is positive data. A non-empty long_running_tool_ids observation is sticky for that node during the segment, matching NodeRunner's additive interrupt set. A tracked ancestor records descendant_interrupt_observed=true. The producer does not encode pause as an absent row.
  5. Terminal callback is not node outcome. after_run_callback also runs when a caller stops consuming the async stream and the root task is cancelled; on_run_error_callback can represent node and non-node failures. The summary records drain_callback='after_run' | 'on_run_error' without translating either into completed or failed.
  6. No back-dated rows. Synthetic row timestamp remains emission time. Source-event times live in explicit typed-view columns, so ingestion order, row time, and represented source time are not conflated.
  7. Invocation-local state with stale-owner protection. before_run_callback always installs a fresh accumulator and segment UUID. Both terminal callbacks drain and clear it in finally; a later before_run_callback replaces stale state if cancellation or abandonment skipped terminal cleanup.

The summary view exposes the event span without a join:

SELECT
  app_name,
  session_id,
  invocation_id,
  workflow_segment_id,
  node_path,
  node_run_id,
  source_event_span_ms,
  node_interrupt_observed,
  descendant_interrupt_observed,
  node_error_observed_count,
  drain_callback
FROM v_workflow_node_observation_summary

For live diagnostics, a FIRST_OBSERVED row without a summary means only summary not observed yet. It must not be presented as proof that a node is running, paused, or stuck: the terminal row may be delayed, filtered, dropped, or never emitted.

Problem

Approach What it can truthfully provide Silent nodes? Cost / constraint
Stage 1 only (NODE_OUTPUT / NODE_ERROR) Result/error points No Already landed
Canonical aggregate view over existing BQAA rows First/last timestamps for rows the plugin already materializes No Cheapest; misses source events that produce no BQAA row, has no stable runner-call key when resumable calls reuse invocation_id, and cannot recover discarded interrupt evidence
Existing invoke_node / invoke_workflow OpenTelemetry spans Exact execution timing in principle Yes, when spans are recorded Global OTel plumbing rather than a BQAA plugin contract; plain-node spans omit node.path / run_id, and silent nodes have no associated event IDs for correlation
Public before_node / after_node / on_node_error callbacks around NodeRunner Framework-exact lifecycle and outcome Yes Requires a framework API and explicit retry/resume/replay/cancellation semantics
This RFC — producer-side observation window All events seen by on_event_callback, runner-call segmentation, positive interrupt/error evidence No Up to two extra rows per observed node plus invocation-local state

The view-only alternative is the baseline, not a strawman. Choose this RFC only if consumers need at least one fact unavailable from existing persisted rows: complete on_event_callback observation coverage, an explicit runner-call segment key, or sticky interrupt evidence. If those facts are not needed, ship only the aggregate view.

Why this matters

  • One producer definition of an observation window. Multi-event generator, streaming, and agent-backed nodes get consistent first/last source-event provenance without every dashboard inventing its own aggregation.
  • Resumable runner calls become separable. workflow_segment_id prevents the many-to-many joins produced by (invocation_id, node_path) when the same invocation and node run resume across multiple calls.
  • Interrupt evidence becomes queryable without an anti-join. Consumers can distinguish a directly observed interrupt, an affected tracked ancestor, and a segment with no interrupt evidence.
  • The exact lifecycle vocabulary stays trustworthy. WORKFLOW_NODE_STARTING / WORKFLOW_NODE_COMPLETED remain available for future framework callbacks that observe the real NodeRunner boundary.

This RFC does not close the FunctionNode wall-time request in #6529. That remains the exact-lifecycle follow-up below.

Positioning & priority (honest valuation — read this first)

This is workflow-node event-span analytics, not node-latency instrumentation.

  • A single-output FunctionNode has a zero source-event span. The function may run for minutes before emitting its only output; none of that time is observable here.

  • First/last event is not start/end. Queue time, work before the first event, and work after the last event are absent.

  • Silent nodes remain invisible. This includes a node that emits no event and a wait_for_output=True node that reaches WAITING without emitting anything.

  • WAITING is not fully observable. An event-emitting node can later become WAITING without long_running_tool_ids; the summary correctly says no interrupt was observed, not that the node completed.

  • Replay is observation, not execution. _maybe_reemit_replayed_output can enqueue a fresh event for a fast-forwarded node that did not execute in the new runner call. No current event field identifies that case.

  • Agent nodes can have two telemetry families. A workflow Agent may emit exact AGENT_* rows and observation-derived WORKFLOW_NODE_* rows. Consumers must not add their durations.

  • Source-event timestamps can tie or regress. A one-event span is zero; custom/replayed timestamps can make the signed span negative. The view preserves the source facts rather than clamping them.

  • A missing summary is ambiguous. Delivery loss, filtering, cancellation, abandoned generators, and an in-progress call can all produce the same stored shape. Only positive summary fields are state claims.

  • Default row cost follows existing BQAA filtering. With no event allow/deny configuration, both new types are enabled and add up to two rows per observed node per runner call. Operators can deny either type; doing so intentionally weakens the paired contract.

  • Strategic value: MEDIUM. The contract standardizes graph-event analytics and segment identity but does not provide the FunctionNode latency originally requested.

  • Immediate value: HIGH for multi-event and interrupting nodes; LOW for ordinary single-output FunctionNodes. Before implementation, name at least one production query or dashboard that needs facts unavailable from the canonical aggregate-view baseline.

  • Risk: LOW-MEDIUM. No table schema migration, but row volume and invocation-local state grow with the number of distinct node paths observed in one runner call, and summary rows drain in one terminal burst.

Row contract

Both typed views retain the common columns (timestamp, event_id, session_id, invocation_id, trace/span IDs, status, and error fields) and add:

View Additional columns
v_workflow_node_first_observed app_name, workflow_segment_id, node_path, node_run_id, node_parent_run_id, source_event_id, source_event_timestamp, boundary_basis
v_workflow_node_observation_summary app_name, workflow_segment_id, node identity, first_source_event_id, last_source_event_id, first_source_event_timestamp, last_source_event_timestamp, source_event_span_ms, observed_event_count, node_interrupt_observed, descendant_interrupt_observed, node_error_observed_count, drain_callback, boundary_basis

All added fields ride in existing JSON payload columns and are projected by _EVENT_VIEW_DEFS; there is no top-level BigQuery schema migration. boundary_basis='event_observation' appears on both event types so a future exact lifecycle contract never silently mixes with this one.

Key Design Decisions

  • Use observation names, not lifecycle names. Public analytics event names are effectively irreversible. Documentation cannot make STARTING mean “possibly the final output event,” and a future producer must not change the semantics of an existing event type from approximate to exact.
  • Create a plugin segment ID in before_run_callback. invocation_id is not a runner-call boundary in resumable execution. A random UUID is identity only; segment chronology comes from the associated FIRST_OBSERVED/summary row timestamps.
  • Aggregate every path-bearing event before event-type filtering. The accumulator sees every event delivered to this plugin's on_event_callback, even when that event produces no ordinary BQAA row. Filtering controls written rows, not what “observed” means.
  • Keep row time and source-event time separate. timestamp answers when the synthetic row was emitted. Explicit source-event timestamp columns answer what source interval the summary represents. No EventData.timestamp_override is needed.
  • Emit summaries for interrupt-observed nodes. Positive direct/descendant flags are more reliable than absence. A summary is not a completion row and does not close a logical node run across resume segments.
  • Count node errors without attributing terminal failure. node_error_observed_count is factual. A NODE_ERROR may be a retryable attempt, and the escaped run error loses error_node_path; matching exception type/message cannot prove which parallel node caused termination.
  • Deterministic drain order is first-observation order. Summary rows are enqueued before the plugin's INVOCATION_COMPLETED or INVOCATION_ERROR row when the corresponding callback is reached.
  • Never synthesize an empty-path node. The default node_info.path == '' remains outside this contract.
  • Preserve delivery semantics. Each synthetic row gets the existing per-row event_id before enqueue; retry of that queued row preserves the ID. Consumers in default write mode still apply the documented event_id deduplication rule.

Verification plan

  • Segmentation: a resumable pause/resume that reuses invocation_id and node.path produces distinct workflow_segment_id values and no cross-segment join.
  • Observation: first path-bearing event emits one FIRST_OBSERVED; later events update the summary only; empty-path events produce neither type; accumulation happens before source-event filtering.
  • Summary: one-event node reports count 1 and span 0; multi-event node preserves first/last IDs and timestamps; regressing timestamps remain negative; row timestamp stays drain time.
  • Interrupts: evidence is sticky; direct leaf and tracked strict ancestors are distinguished; siblings remain false; every tracked node still receives a summary.
  • Errors/retries: each qualifying node error increments the count; successful retry remains an observation count, not a failed status; parallel identical exceptions never produce node-failure attribution.
  • Terminal paths: normal return, consumer early-close/GeneratorExit, and escaped Exception record the actual callback that drained. Cancellation/abandonment that reaches neither callback produces no summary; the next before_run_callback replaces stale state.
  • Replay: fast-forward output re-emission is tested/documented as an observation with no execution claim.
  • Overlap: workflow Agent nodes can produce both AGENT_* and observation rows without changing existing events.
  • Isolation/scale: concurrent task invocations do not share state; sequential same-invocation_id runner calls reset segment ownership; looped node paths make the state/row bound explicit; terminal drain order is stable.
  • Views/config: exact projected column lists above; boundary_basis on both; event allow/deny behavior; payload-column denylist follows existing view projection; synthetic event_id survives batch retry.
  • Regression: non-workflow invocations produce no observation rows.

Deferred (each needs its own design decision)

  1. Framework-exact node lifecycle. Add node-native callbacks (or another supported exporter contract) around NodeRunner with defined retry, resume, replay, cancellation, and short-circuit semantics. Only this can truthfully emit WORKFLOW_NODE_STARTING / WORKFLOW_NODE_COMPLETED and measure wall time, including silent nodes.
  2. workflow_node_status='failed'. Current upstream now has a reliable invocation-error callback, but not reliable node attribution: DynamicNodeFailError is unwrapped before on_run_error_callback, error_node_path is lost, retry attempts emit NODE_ERROR, and parallel nodes may raise identical exception type/message pairs. Reserve the value until the terminal signal carries node-path or source-event causality.
  3. Replay attribution. Mark _maybe_reemit_replayed_output and other fast-forward events so consumers can separate execution from observation replay.
  4. All WAITING states. wait_for_output=True without interrupt IDs is internal workflow state, not an event-visible fact.
  5. Imperative in-node tool instrumentation. Tools called directly inside a FunctionNode body still do not traverse normal tool callbacks.
  6. Cross-segment logical-run stitching. A paused node can reuse invocation_id, path, and run ID across segment IDs. Logical wall duration requires a pause-exclusion policy and explicit segment ordering.

Implementation gate

Before reviving the prototype, validate one concrete consumer query against real Stage 1 data and the canonical aggregate-view baseline. Proceed with producer rows only if the query requires the new segment key, complete on_event_callback coverage, or positive interrupt evidence. Otherwise ship the view-only alternative and leave the exact lifecycle work for the node-callback RFC.

Links

Activity

  1. changed the title [-]RFC: Workflow-node boundary events (WORKFLOW_NODE_STARTING / WORKFLOW_NODE_COMPLETED) — Stage 2 of adk-python#6529[/-] [+]RFC: Workflow-node observation windows — plugin-only Stage 2A of adk-python#6529[/+] on Aug 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions