What started this

A spawn_agent call was killed mid-run by the parent session's generic per-tool inactivity watchdog (SessionConfig.ToolExecutionTimeout, default 90s):

Error executing tool: Tool 'spawn_agent' produced no activity for 90s and was stopped.
It may be stuck — please try again, or simplify the request.

Three identically-shaped sub-agents spawned at once:

Two ran well past 90s of wall-clock and survived — so this is an inactivity timeout that resets on activity, not a wall-clock cap. The third opened a >90s window with no forwarded activity and was killed from the outside while healthy; its own (much larger, progress-aware) internal watchdogs never fired. It had nothing to do with inference capacity — the trigger is silence on a signaling channel, not slow execution.

That symptom is a single instance of a general problem, and this issue is about the general problem, not a point fix.

The real problem

Knowing when a tool call has stalled — and sub-agents are tool calls, no different from shell_execute, web_fetch, or an MCP call. From the session's seat they are all the same thing: an opaque delegated operation that should eventually return a result and might not.

The confusion at the root of the timeout sprawl:

• We detect "no output." We mean "no progress." Those differ during prefill/queue waits, tool execution, and human-approval waits — all silent, none stalled. Silence is not evidence of death.
• A timeout is a bet that "silent longer than X ⇒ dead." Layering bets with different X values doesn't make the inference smarter — the tightest, dumbest bet just fires first and kills healthy work. That is the mechanism by which adding timeouts has made this worse.
• The current per-tool inactivity watchdog is wrong in both directions. It resets on incidental output (shell stdout, sub-agent pings are pumped into it as keepalive), so it over-kills the quiet-but-healthy (a sub-agent in prefill, a compile that prints nothing for two minutes) and under-detects the noisy-but-wedged (a livelock spewing output resets it forever). Output is an accidental proxy for progress.
• The real cleavage is observability, not leaf-vs-nested. A tool can either report its own progress or it can't. Sub-agents are the most observable tool we have (in-process, structured progress, a no-progress watchdog that distinguishes "keepalives but no tokens" from real work). They are the exemplar of the contract every tool should expose — not an exception to be exempted.

Every tool call shares one lifecycle: startup latency (blind) → making progress → maybe wedged → done. The only irreducibly-blind window is time-to-first-sign-of-life; after that, stall detection should be progress-driven.

Goal

Make stall detection uniform across all tool calls and keyed on lost forward progress, observed at the layer that can actually see it — never inferred from output silence by the orchestration layer.

In the target state:

• The session owns budget and cancellation, not stall detection.
• A tool that can report progress is trusted while it reports; the session consumes that signal.
• A genuinely opaque tool is bounded by one explicit, caller-meaningful wall-clock budget — an honest bet, sized per operation, generous by default.
• No layer runs a tight inactivity timer fed by incidental output. That dishonest middle path is the bug.
• The irreducible guessing surface shrinks to time-to-first-sign-of-life, sized per-op and per-environment.

Where the friction and conflicts are

• Heterogeneous tool surface. Tools span fully observable (in-process sub-agent) to fully opaque (third-party MCP, plain HTTP). No single signal every tool can emit; the model has to span both gracefully.
• The progress contract can't be mandatory. Existing tools don't implement it and third-party tools can't be forced to. So a fallback is required — and the only honest fallback is the wall-clock bet, which reintroduces the very thing we're minimizing, just scoped correctly.
• "Progress" is not "activity," and can be faked. Heartbeats/output don't prove forward motion (livelock, a sub-agent looping the same call, a model streaming tokens forever without converging). The contract must mean forward motion, not noise — and convergence/looping may be a separate concern (iteration budgets) from stall.
• Detection vs response are conflated today. The watchdog both detects and kills the whole tool call. With nesting, a stall inside a sub-agent shouldn't necessarily nuke the parent turn — there's an open question of who decides the response (kill the leaf, abort the turn, surface to the user) and at what granularity.
• Who watches the watcher. "Delegate liveness downward" is too strong on its own: if the owning layer is itself wedged (e.g., a starved mailbox so its own timer never fires), the parent still needs a coarse backstop. So the parent keeps a generous, absolute wall-clock backstop as defense-in-depth — just not a tight, dishonest inactivity proxy.
• The first-sign-of-life bet can't be eliminated, only bounded. For an opaque tool the whole call may be blind, so some wall-clock bet is unavoidable; the work is to shrink its scope and make it honest, not to pretend it away.
• Config surface and backward compatibility. ~7 overlapping timeout knobs exist today (TurnLlmTimeout, FirstTokenTimeout, PrefillTimeout, NoProgressTimeout, ToolExecutionTimeout, SidecarLlmTimeout, plus the sub-agent mirror set). Collapsing them must respect existing config/schema and the reality that fast-hosted vs slow self-hosted backends need different bets.
• Don't throw away the good part. The session/sub-agent already has a progress-aware two-phase watchdog (prefill → inter-delta → keepalive-immune no-progress). That is the exemplar to generalize, not replace; the friction is reconciling it with the dumb tool-layer inactivity timer without adding a third mechanism.

What we probably need (direction, not design)

• A first-class notion of progress that is distinct from output — forward motion the worker asserts, not bytes observed.
• A uniform tool liveness contract with a graceful fallback: a tool either reports progress (and is governed by that) or is treated as opaque (and is governed by an explicit wall-clock budget). Legacy/third-party tools default to the fallback.
• Instrument matched to observability: consume progress where it exists, place one honest bet where it doesn't, and never infer progress from incidental output.
• Separation of stall detection from stall response/escalation, so the layer that detects isn't forced to also decide the blast radius.
• A coarse, honest wall-clock backstop at the parent for defense-in-depth, explicitly not a tight inactivity timer.
• Consolidation of the timeout knobs into this model, plus an invariant that forbids two layers independently inactivity-timing the same self-governing operation, and forbids an outer budget tighter than an inner one (the inversion that produced the symptom above).

Secondary observations (carry-along, not the core)

• The per-call override knob (_timeout_seconds) is easy to get wrong: the model first tried Timeout_seconds (rejected as unknown arg), then set _timeout_seconds: 90 — equal to the default, widening nothing. A symptom of pushing the liveness decision up to a layer that has to be told the right number.
• After the kill, retried sub-agents reported success=True in ~5–7s but wrote no files — a separate failure mode (a sub-agent returning a final answer without doing the work).
• Observability gap: the detailed per-iteration sub-agent trace did not land in the daemon log for this run (only HttpClient cleanup noise appeared in the failing window), so "which silent window stalled" was not answerable from logs. Whatever the model becomes, the stalled-vs-slow distinction needs to be reconstructable after the fact.