Skip to content

Orchestration: DagScheduler deadlock when SubAgentManager concurrency exhausted before plan execution #1619

@bug-ops

Description

@bug-ops

Description

When a sub-agent is spawned during the planning phase (via an LLM tool call), it occupies the only available concurrency slot (max_concurrent=1). When /plan confirm triggers execution, spawn_for_task for all DAG tasks immediately fails with ConcurrencyLimitReached and the plan never progresses.

Root Cause

  1. LLM uses spawn_subagent tool during planning — sub-agent (e.g. rust-architect) occupies slot 1/1
  2. Plan execution starts, DagScheduler issues Spawn actions for task_id=1, task_id=2
  3. spawn_for_task fails: ConcurrencyLimit { active: 1, max: 1 }
  4. Tasks revert to Ready; DagScheduler spins at 250ms (see Orchestration: DagScheduler busy-spins at 250ms when SubAgentManager is saturated #1618)
  5. The external sub-agent may complete eventually, but during the spin the user sees no progress

Observed Log

INFO  sub-agent spawned task_id="8ce69290-..." def_name="rust-architect"  ← planning phase
ERROR spawn_for_task failed error=concurrency limit reached (active: 1, max: 1) task_id=1  ← execution

Fix Directions

  1. Short-term: Document that [agents] max_concurrent must be set high enough (e.g., N+1 where N = max orchestration parallel tasks) when orchestration is enabled.
  2. Medium-term: Reserve concurrency slots for orchestration tasks at plan-confirm time so planning-phase sub-agents cannot starve them.
  3. Long-term: Separate concurrency budgets: one pool for LLM tool-call sub-agents, one pool for orchestration task agents.

Severity: High

Plan execution silently fails to make progress when the concurrency limit is set to default (1). User sees "Confirmed. Executing plan..." but no tasks complete.

Discovered

Continuous improvement session 2026-03-13, v0.14.3

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions