Motivation
A single debugging session into "subagents fail to launch" (#1467) surfaced bugs at the seams between ~6 independent mechanisms that govern one subagent spawn:
- a liveness classification (
Opaque vs SelfMonitoring) resolved via GetLivenessMode with a silent ?? ToolLivenessMode.Opaque fallback;
- three watchdog budget modes (
ResetOnItem / FirstItemOnly / WallClock);
- two watchdog implementations: the off-thread, volatile
StreamingToolWatchdog and the actor-scheduled ProcessingWatchdog;
- the subagent's three budgets (prefill / inter-delta / no-progress);
- a
SuspendsInactivityWatchdog flag threaded through the activity stream to pause the clock during human approval;
- two logging systems (Akka async vs MEL sync) with an AsyncLocal routing gate.
Every defect found lived at an interaction, not inside a single part: the WallClock branch silently ignoring the approval-pause, the spawn watchdog regression (#1467), and the per-session log gap (#1468). The failures were invisible, intermittent, and plausible-looking — the signature of too many interacting parts, not one mistake.
Goal: make the system more robust by removing moving parts, not adding more.
Target design
-
One watchdog, and it's actor state. Liveness is the actor's job: a plain field on the actor, advanced by the actor's own scheduler. The stream consumer just sends progress / blocked / done messages; the actor alone decides life and death. Deletes the second watchdog — the off-thread StreamingToolWatchdog, its polling timer, and all the Volatile shared state.
-
One liveness rule, not three modes + a classification. A thing is alive if it's making progress or explicitly blocked on a human; it dies only after N seconds of genuine no-progress. That single rule replaces ResetOnItem/FirstItemOnly/WallClock, the Opaque/SelfMonitoring enum, and the silent ?? Opaque fallback. No tool declares a "mode."
-
"Blocked on a human" is a state, not a flag. Waiting on approval transitions to an explicit WaitingForApproval state that pauses the clock and resumes on resolve. Deletes SuspendsInactivityWatchdog and its threading through a streaming channel into a second watchdog — and makes "approvals never touch the clock" true by construction.
-
The parent never supervises the child's clock. Once spawned, a subagent owns its lifecycle; the parent's spawn_agent call just awaits the result (plus user/turn cancellation). Deletes the startup-window budget, the parent-side watchdog, and the activity that exists only to keep the parent from killing the child. One owner of "is the subagent alive": the subagent.
-
One log path, tagged at the source. Lifecycle events (spawn → dispatch → block → resume → outcome) are emitted once at the actor boundary through a single sink that carries the session id from where it's known. Deletes the Akka-vs-MEL split and the AsyncLocal gate that only works on the right thread. If it happened, it's in the transcript.
What gets deleted
StreamingToolWatchdog (off-thread polling/volatile path) — folded into actor-owned liveness.
ToolWatchdogResetMode (the three modes) — replaced by the single liveness rule.
ToolLivenessMode + IToolExecutor.GetLivenessMode + the ?? Opaque fallback.
ToolActivityUpdate.SuspendsInactivityWatchdog (flag) — replaced by an explicit WaitingForApproval state.
- the parent-side
spawn_agent budget / startup window.
- the second log path / AsyncLocal gate for subagent lifecycle visibility.
Invariants that become true by construction
- A healthy subagent is never killed.
- Approvals (and any human-blocked wait) never affect the clock.
- Every spawn outcome is visible in the session transcript.
Keep — essential complexity, not removable
- Tool/stream consumption runs off the actor mailbox (can't block the actor).
- Subagents own their own liveness.
- Human-in-the-loop approval.
Migration order (incremental — each step independently shippable)
- Stop the bleeding (done): honor the approval-pause in all watchdog modes + a mode-matrix regression test that fails on the old code (branch
fix/watchdog-approval-suspend).
- Approval as state: make "blocked on a human" a first-class
WaitingForApproval state with explicit pause/resume; remove SuspendsInactivityWatchdog from the streaming path. Add an actor-level approval-pause test (hold approval past the budget → no timeout; genuine stall → timeout).
- Collapse the rule: fold the three budget modes +
ToolLivenessMode into one "progress-or-blocked, else no-progress" rule; delete the silent ?? Opaque (fail loud).
- Drop parent supervision: the parent awaits the result + cancellation only; delete the
spawn_agent startup-window budget.
- One watchdog: replace the off-thread volatile
StreamingToolWatchdog with actor-owned liveness (consumer sends progress/blocked/done; actor decides). Biggest step — last, after the rule is simplified.
- One log path: emit lifecycle once at the actor boundary through one session-tagged sink; remove the AsyncLocal-gated second path.
Related
This is the repo constitution talking back: "avoid manual synchronization unless absolutely necessary," "no silent fallbacks," "reuse before you add." This subsystem is where those rules quietly eroded; consolidating restores them.
Motivation
A single debugging session into "subagents fail to launch" (#1467) surfaced bugs at the seams between ~6 independent mechanisms that govern one subagent spawn:
OpaquevsSelfMonitoring) resolved viaGetLivenessModewith a silent?? ToolLivenessMode.Opaquefallback;ResetOnItem/FirstItemOnly/WallClock);StreamingToolWatchdogand the actor-scheduledProcessingWatchdog;SuspendsInactivityWatchdogflag threaded through the activity stream to pause the clock during human approval;Every defect found lived at an interaction, not inside a single part: the
WallClockbranch silently ignoring the approval-pause, the spawn watchdog regression (#1467), and the per-session log gap (#1468). The failures were invisible, intermittent, and plausible-looking — the signature of too many interacting parts, not one mistake.Goal: make the system more robust by removing moving parts, not adding more.
Target design
One watchdog, and it's actor state. Liveness is the actor's job: a plain field on the actor, advanced by the actor's own scheduler. The stream consumer just sends
progress/blocked/donemessages; the actor alone decides life and death. Deletes the second watchdog — the off-threadStreamingToolWatchdog, its polling timer, and all theVolatileshared state.One liveness rule, not three modes + a classification. A thing is alive if it's making progress or explicitly blocked on a human; it dies only after N seconds of genuine no-progress. That single rule replaces
ResetOnItem/FirstItemOnly/WallClock, theOpaque/SelfMonitoringenum, and the silent?? Opaquefallback. No tool declares a "mode.""Blocked on a human" is a state, not a flag. Waiting on approval transitions to an explicit
WaitingForApprovalstate that pauses the clock and resumes on resolve. DeletesSuspendsInactivityWatchdogand its threading through a streaming channel into a second watchdog — and makes "approvals never touch the clock" true by construction.The parent never supervises the child's clock. Once spawned, a subagent owns its lifecycle; the parent's
spawn_agentcall just awaits the result (plus user/turn cancellation). Deletes the startup-window budget, the parent-side watchdog, and the activity that exists only to keep the parent from killing the child. One owner of "is the subagent alive": the subagent.One log path, tagged at the source. Lifecycle events (spawn → dispatch → block → resume → outcome) are emitted once at the actor boundary through a single sink that carries the session id from where it's known. Deletes the Akka-vs-MEL split and the AsyncLocal gate that only works on the right thread. If it happened, it's in the transcript.
What gets deleted
StreamingToolWatchdog(off-thread polling/volatile path) — folded into actor-owned liveness.ToolWatchdogResetMode(the three modes) — replaced by the single liveness rule.ToolLivenessMode+IToolExecutor.GetLivenessMode+ the?? Opaquefallback.ToolActivityUpdate.SuspendsInactivityWatchdog(flag) — replaced by an explicitWaitingForApprovalstate.spawn_agentbudget / startup window.Invariants that become true by construction
Keep — essential complexity, not removable
Migration order (incremental — each step independently shippable)
fix/watchdog-approval-suspend).WaitingForApprovalstate with explicit pause/resume; removeSuspendsInactivityWatchdogfrom the streaming path. Add an actor-level approval-pause test (hold approval past the budget → no timeout; genuine stall → timeout).ToolLivenessModeinto one "progress-or-blocked, else no-progress" rule; delete the silent?? Opaque(fail loud).spawn_agentstartup-window budget.StreamingToolWatchdogwith actor-owned liveness (consumer sends progress/blocked/done; actor decides). Biggest step — last, after the rule is simplified.Related
fix/watchdog-approval-suspend— the targeted line-109 fix + mode-matrix test (step 1).This is the repo constitution talking back: "avoid manual synchronization unless absolutely necessary," "no silent fallbacks," "reuse before you add." This subsystem is where those rules quietly eroded; consolidating restores them.