Skip to content

Consolidate subagent liveness onto one actor-owned watchdog (addition through subtraction) #1472

Description

@Aaronontheweb

Motivation

A single debugging session into "subagents fail to launch" (#1467) surfaced bugs at the seams between ~6 independent mechanisms that govern one subagent spawn:

  • a liveness classification (Opaque vs SelfMonitoring) resolved via GetLivenessMode with a silent ?? ToolLivenessMode.Opaque fallback;
  • three watchdog budget modes (ResetOnItem / FirstItemOnly / WallClock);
  • two watchdog implementations: the off-thread, volatile StreamingToolWatchdog and the actor-scheduled ProcessingWatchdog;
  • the subagent's three budgets (prefill / inter-delta / no-progress);
  • a SuspendsInactivityWatchdog flag threaded through the activity stream to pause the clock during human approval;
  • two logging systems (Akka async vs MEL sync) with an AsyncLocal routing gate.

Every defect found lived at an interaction, not inside a single part: the WallClock branch silently ignoring the approval-pause, the spawn watchdog regression (#1467), and the per-session log gap (#1468). The failures were invisible, intermittent, and plausible-looking — the signature of too many interacting parts, not one mistake.

Goal: make the system more robust by removing moving parts, not adding more.

Target design

  1. One watchdog, and it's actor state. Liveness is the actor's job: a plain field on the actor, advanced by the actor's own scheduler. The stream consumer just sends progress / blocked / done messages; the actor alone decides life and death. Deletes the second watchdog — the off-thread StreamingToolWatchdog, its polling timer, and all the Volatile shared state.

  2. One liveness rule, not three modes + a classification. A thing is alive if it's making progress or explicitly blocked on a human; it dies only after N seconds of genuine no-progress. That single rule replaces ResetOnItem/FirstItemOnly/WallClock, the Opaque/SelfMonitoring enum, and the silent ?? Opaque fallback. No tool declares a "mode."

  3. "Blocked on a human" is a state, not a flag. Waiting on approval transitions to an explicit WaitingForApproval state that pauses the clock and resumes on resolve. Deletes SuspendsInactivityWatchdog and its threading through a streaming channel into a second watchdog — and makes "approvals never touch the clock" true by construction.

  4. The parent never supervises the child's clock. Once spawned, a subagent owns its lifecycle; the parent's spawn_agent call just awaits the result (plus user/turn cancellation). Deletes the startup-window budget, the parent-side watchdog, and the activity that exists only to keep the parent from killing the child. One owner of "is the subagent alive": the subagent.

  5. One log path, tagged at the source. Lifecycle events (spawn → dispatch → block → resume → outcome) are emitted once at the actor boundary through a single sink that carries the session id from where it's known. Deletes the Akka-vs-MEL split and the AsyncLocal gate that only works on the right thread. If it happened, it's in the transcript.

What gets deleted

  • StreamingToolWatchdog (off-thread polling/volatile path) — folded into actor-owned liveness.
  • ToolWatchdogResetMode (the three modes) — replaced by the single liveness rule.
  • ToolLivenessMode + IToolExecutor.GetLivenessMode + the ?? Opaque fallback.
  • ToolActivityUpdate.SuspendsInactivityWatchdog (flag) — replaced by an explicit WaitingForApproval state.
  • the parent-side spawn_agent budget / startup window.
  • the second log path / AsyncLocal gate for subagent lifecycle visibility.

Invariants that become true by construction

  • A healthy subagent is never killed.
  • Approvals (and any human-blocked wait) never affect the clock.
  • Every spawn outcome is visible in the session transcript.

Keep — essential complexity, not removable

  • Tool/stream consumption runs off the actor mailbox (can't block the actor).
  • Subagents own their own liveness.
  • Human-in-the-loop approval.

Migration order (incremental — each step independently shippable)

  1. Stop the bleeding (done): honor the approval-pause in all watchdog modes + a mode-matrix regression test that fails on the old code (branch fix/watchdog-approval-suspend).
  2. Approval as state: make "blocked on a human" a first-class WaitingForApproval state with explicit pause/resume; remove SuspendsInactivityWatchdog from the streaming path. Add an actor-level approval-pause test (hold approval past the budget → no timeout; genuine stall → timeout).
  3. Collapse the rule: fold the three budget modes + ToolLivenessMode into one "progress-or-blocked, else no-progress" rule; delete the silent ?? Opaque (fail loud).
  4. Drop parent supervision: the parent awaits the result + cancellation only; delete the spawn_agent startup-window budget.
  5. One watchdog: replace the off-thread volatile StreamingToolWatchdog with actor-owned liveness (consumer sends progress/blocked/done; actor decides). Biggest step — last, after the rule is simplified.
  6. One log path: emit lifecycle once at the actor boundary through one session-tagged sink; remove the AsyncLocal-gated second path.

Related

This is the repo constitution talking back: "avoid manual synchronization unless absolutely necessary," "no silent fallbacks," "reuse before you add." This subsystem is where those rules quietly eroded; consolidating restores them.

Metadata

Metadata

Assignees

No one assigned

    Labels

    cleanupCode quality improvements and tech debt reductionreliabilityRetries, resilience, graceful degradationsubagentsspawn_agent, SubAgentActor, definition loader, discovery context layer, and related features

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions