Skip to content

[Bug] sessions_spawn children execute serially despite subagents.maxConcurrent=12 (v2026.6.5) #91940

Description

@wangwllu

EDIT 2026-06-10: Root cause turned out to not be the subagent lane — that lane is correctly 12-wide and sessions_spawn dispatches into it correctly. The real serialization happens one layer down in the Claude CLI backend (resolveCliRunQueueKey collapsing all fresh sessions onto a per-workspace key). See correction comment and the underlying-bug issue #91946. The "Source-side observations" section below still describes the wiring accurately but the conclusion it hints at (lazy lane init / lane=1) is wrong — please skip to #91946 for the actual root cause and proposed fixes.


Summary

sessions_spawn children execute strictly serially even when both agents.defaults.maxConcurrent (8) and agents.defaults.subagents.maxConcurrent (12) are configured well above the spawn count. Spawning 4 children in the same turn produces serial execution: each child's first CLI turn completes before the next child's CLI turn starts.

This affects any orchestration that fans out to N subagents expecting parallel work — comprehensive reviews, multi-modal sweeps, parallel codegen workers, etc.

Version

  • OpenClaw 2026.6.5
  • claude-cli runtime, model anthropic/Claude-Opus-4.7-hq (routed to claude-cli)
  • macOS 25.4.0 (Darwin arm64), Node v26.0.0
  • Install: npm global

Repro

In a single main-session turn, spawn 4 subagents whose work is trivial (each one just prints two timestamps with a sleep 4 in between):

sessions_spawn(task: "Print PING-A start=$(date +%H:%M:%S.%3N) pid=$$. sleep 4. Print PING-A end=$(date +%H:%M:%S.%3N).", taskName: "lane_test_a")
sessions_spawn(task: "...PING-B...", taskName: "lane_test_b")
sessions_spawn(task: "...PING-C...", taskName: "lane_test_c")
sessions_spawn(task: "...PING-D...", taskName: "lane_test_d")

Each child should finish in ~5–10s. With subagents.maxConcurrent=12, all 4 should overlap and the wall clock should be ~slowest-single-child.

Expected

4 children run concurrently. PING-A/B/C/D start= timestamps should be within ~1s of each other.

Actual

Children run strictly one-at-a-time, ~12 seconds apart.

PING-A start=20:41:29.199    end=20:41:33.259
PING-B start=20:41:41.3xx    end=20:41:45.3xx
PING-C start=20:41:53.3xx    end=20:41:57.3xx
PING-D start=20:42:15.524    end=20:42:19.579

Wall clock from first spawn (20:41:12) to last child end (20:42:19): ~67 seconds for 4 children whose pure work was 4s each.

Gateway log evidence

The gateway log shows each cli exec for child N only fires after child N-1's turn ends:

20:41:12.945  cli exec     model=Claude-Opus-4.7-hq    ← child A starts
20:41:12.947  claude live session start  activeSessions=2
20:41:31.991  claude live session turn   durationMs=88115   ← child A's first turn ends (88s of CLI startup + 4s work)
20:41:35.869  cli exec     model=Claude-Opus-4.7-hq    ← child B starts (~4s after A's turn ends)
20:41:35.869  claude live session start  activeSessions=3
20:41:48.272  claude live session turn   durationMs=12400   ← child B turn ends
20:41:48.279  cli exec                                      ← child C starts
20:41:48.279  claude live session start  activeSessions=4
20:42:00.217  claude live session turn   durationMs=11936
20:42:00.221  cli exec                                      ← child D starts
20:42:00.221  claude live session start  activeSessions=5
20:42:24.965  claude live session turn   durationMs=24743

activeSessions does count up to 5, so the runtime is technically capable of holding multiple live CLI sessions — but the queue is releasing them one-at-a-time.

Configuration

{
  "agents": {
    "defaults": {
      "maxConcurrent": 8,
      "subagents": { "maxConcurrent": 12, "archiveAfterMinutes": 60 }
    }
  }
}

openclaw config get:

agents.defaults.maxConcurrent             = 8
agents.defaults.subagents.maxConcurrent   = 12

Gateway was restarted after the config change. applyGatewayLaneConcurrency should have called setCommandLaneConcurrency("subagent", 12).

Source-side observations

Read against dist/ (v2026.6.5):

  • dist/openclaw-tools-YEJwiczM.js:11750 — the sessions_spawn direct dispatch correctly passes lane: AGENT_LANE_SUBAGENT ("subagent") when calling the gateway agent method.
  • dist/embedded-agent-CpxCz9I4.js:2128–2168 — every embedded run double-enqueues: enqueueSession(() => enqueueGlobal(actualWork)). sessionLane resolves from the child's session key (agent:main:subagent:<UUID> — unique per child), globalLane resolves to the lane string passed from spawn ("subagent").
  • dist/lanes-CI0_P-yC.jsAGENT_LANE_SUBAGENT = "subagent".
  • dist/agent-limits-DGV0ALs8.jsresolveSubagentMaxConcurrent(cfg) reads cfg.agents.defaults.subagents.maxConcurrent, defaults to 8.
  • dist/hook-client-ip-config-DQhhfgV0.js:5–11applyGatewayLaneConcurrency is the only path that sets the subagent lane; it runs at gateway startup (server.impl-Btmg89EG.js:1976) and on config reload (server-reload-handlers-B5rJ13f1.js:529). No other code paths call setCommandLaneConcurrency("subagent", ...).
  • dist/command-queue-CWrLXrgW.js:94 — default lane.maxConcurrent = 1 for any lane created lazily via getLaneState(...).

So the wiring looks correct on paper, but the observed throughput says the lane is effectively maxConcurrent = 1 for subagent. Possible mechanisms:

  1. resolveGlobalSingleton(COMMAND_QUEUE_STATE_KEY, ...) may be initializing the subagent lane lazily on first enqueueCommandInLane("subagent", ...) before applyGatewayLaneConcurrency runs (or under a different module copy / different singleton key, leaving the lane at the default 1). This would also be invisible to config get, since that only inspects config, not the lane snapshot.
  2. Some ordering issue between subagentSpawnDeps.dispatchGatewayMethodInProcess callers and applyGatewayLaneConcurrency.
  3. The expectFinal: true semantics on callSubagentGateway are holding the spawner main lane in some way that appears like subagent serialization but is actually sessionLane(main) blocking — but the same main lane held 4 spawn tool calls successfully within 3s (12.9s → 15.9s in the log), so this is unlikely.

Other concurrency-related issues today

For context, the same gateway showed 28 × release_lane … released=0, 38 × stuck session recovery, 10 × abort_embedded_run events in the day before this test — all consistent with reports in #48488, #89766, #84903.

Repro context

  • Single direct DM session as parent (agent:main:telegram:direct:<uid>). Not a forum group, no other heavy load. Daily memory file confirms the 8/12 config has been applied since 2026-06-05.

Workaround

Spawning the same kind of work via cron (isolated session, scheduled at immediately) does run in parallel, since cron uses a separate cron-nested lane bound by resolveCronMaxConcurrentRuns. This is what users hitting this end up doing — but cron has a very different lifecycle (one-shot scheduled, no streaming back to parent), so it's not a drop-in replacement.

Asks

  1. Confirm whether applyGatewayLaneConcurrency actually takes effect for the subagent lane on a running gateway — i.e., add a gateway lanes introspection command that snapshots the live lane state (max + active + queued), so users can verify without source-diving.
  2. If lazy getLaneState is the issue: ensure subagent/main lane state is preallocated with the configured concurrency at startup, before anything can enqueueCommandInLane(...) against it.
  3. Document the actual semantics of agents.defaults.subagents.maxConcurrent in the runtime schema (currently types.openclaw-fYj4Ft14.d.ts:397 says "Default: 1" but the code says default 8 — at minimum the docstring is wrong).

Happy to paste the full gateway log slice if useful.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.impact:otherThis issue has meaningful maintainer-visible impact outside the owned taxonomy.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions