Summary

Problem: A single long-running nested agent run (e.g. an ACP Claude Code run on session A) blocks every other nested operation across the gateway, because all nested agent runs share one global queue with maxConcurrent=1. In the reported incident, a 7-14 minute ACP run on ebao-next blocked three other agent sessions (ebao, ebao-vue, and follow-up nested work) for 7-14 minutes each. Assistant responses generated during the backlog were never delivered to Discord.

Why: CommandLane.Nested = "nested" is a single lane name used by every nested operation — sessions_send, runAgentStep, ACP dispatches — regardless of which session they target. The command queue keyed off that bare string serializes everything behind a single head-of-line run.

What changed: Nested lanes are now scoped to the target session key. sessions_send, runAgentStep, and the A2A ping-pong flow now compute their lane as nested:<targetSessionKey>. A long run on session A still serializes subsequent nested work against A (concurrency invariant preserved), but nested work against B, C, D now runs in parallel on their own lanes.

What did NOT change:

• CommandLane.Nested is still exported; callers without a known target session (legacy cron paths via resolveNestedAgentLane(lane)) still fall back to the unscoped lane.
• isNestedAgentLane(lane) treats both nested and nested:<key> as nested for delivery logging, metrics, and future switchboards.
• No concurrency invariants were weakened. Each per-session lane still has maxConcurrent=1 by default (same as before for the unscoped nested lane).
• Session delivery context corruption and delivery retry/DLQ (two other sub-fixes the reporter suggested) are out of scope for this PR — see "Out of scope" below.

Change Type

Bug fix (reliability / data loss prevention).

Scope

• agents (src/agents)
• plugins / providers
• channels
• gateway
• cli / web-ui / apps

Linked Issue

Closes #67502.

Root Cause

Architecture before this PR:

1. resolveNestedAgentLane(lane?) in src/agents/lanes.ts returned the bare string \"nested\" for every caller that did not supply an explicit lane.
2. Every nested call site (src/agents/tools/sessions-send-tool.ts:279, src/agents/tools/sessions-send-tool.a2a.ts:95,126, src/agents/tools/agent-step.ts:39) passed lane: AGENT_LANE_NESTED directly.
3. src/process/command-queue.ts keys LaneState by lane name in a Map<string, LaneState> (getLaneState(lane)) with a default maxConcurrent: 1.
4. Nested lane concurrency is never reconfigured anywhere (src/gateway/server-lanes.ts only sets Cron, Main, Subagent concurrency).

Result: every nested run across every session contends on a single lane with a hard cap of 1 in-flight task. When one session's nested work takes ~10 minutes (ACP + Claude Code, long tool loops), every other session's follow-up work queues behind it and surfaces as lane wait exceeded: lane=nested waitedMs=576497 queueAhead=1.

Fix

1. Added resolveNestedAgentLaneForSession(sessionKey) in src/agents/lanes.ts. Returns nested:<sessionKey> when a key is supplied, and falls back to the unscoped nested lane when it is empty (preserves cron/legacy semantics).
2. Added isNestedAgentLane(lane) so delivery logging (src/agents/command/delivery.ts:302) and any future consumer can recognise both the unscoped and scoped variants.
3. Updated every production call site that has a target session key to use resolveNestedAgentLaneForSession:
   • src/agents/tools/sessions-send-tool.ts (top-level sessions_send)
   • src/agents/tools/sessions-send-tool.a2a.ts (A2A ping-pong replies + announce step)
   • src/agents/tools/agent-step.ts (default lane when caller does not override)
4. Changed the identity comparison in src/agents/command/delivery.ts from opts.lane === AGENT_LANE_NESTED to isNestedAgentLane(opts.lane) so nested logging still fires on scoped lanes.
5. Downgraded the exported AGENT_LANE_NESTED and AGENT_LANE_SUBAGENT constants to plain string types so callers that compare against arbitrary lane labels (per-session suffixes, user-supplied lanes) do not trip no-unsafe-enum-comparison.

Regression Test Plan

• Coverage level: Unit tests in the owning helper test file, plus updated assertions in downstream consumer tests.
• Target tests added:
  • src/agents/lanes.test.ts — new describe blocks resolveNestedAgentLaneForSession (#67502) (5 cases) and isNestedAgentLane (5 cases), covering: fallback to unscoped when no key, per-session scoping, distinct lanes for distinct sessions, deterministic output, whitespace trimming, substring-guard against accidental matches like \"deeply-nested-lane\" / \"nestedfoo\".
  • src/commands/agent.delivery.test.ts — new case verifying [agent:nested] log prefix fires for scoped lanes (nested:agent:ebao-next:discord:channel:1), not just the unscoped lane.
• Updated test assertions (behaviour invariant, not regression):
  • src/agents/openclaw-tools.sessions.test.ts — two assertions now match expect.stringMatching(/^nested(?::|$)/) instead of the literal \"nested\".
  • src/gateway/server.sessions-send.test.ts — E2E assertion matches the same regex.

User-visible Changes

Users running multiple agents with sessions_send(timeoutSeconds: 0) no longer see cross-agent stalls when one agent is running a long nested tool loop (ACP, Claude Code, subagent chains). Gateway logs continue to show [agent:nested] prefixed output regardless of the new lane suffix.

Diagram

Before                              After
-------                             -------
                                     session-A nested lane (nested:A)
                                         | [task 1] -> [task 2] -> ...
All nested runs                     session-B nested lane (nested:B)
  |                                     | [task 1] -> [task 2] -> ...
  v   one global \"nested\" lane     session-C nested lane (nested:C)
 queue: A1 A2 B1 C1 A3 B2 ...          | [task 1] -> [task 2] -> ...
 max=1 | head-of-line blocks       (each maxConcurrent=1; concurrent across sessions)

Security Impact

• Adds or changes permissions/capabilities? No.
• Reads, writes, or persists secrets? No.
• Opens new network endpoints or outbound calls? No.
• Changes code execution boundaries (sandbox, exec, MCP)? No.
• Widens data visibility or cross-user scope? No. Per-session lane scoping strengthens isolation.

Repro + Verification

• Environment: OpenClaw 2026.4.14 (323493f), Linux 6.17.0-20-generic, Node v24.14.1, Discord persistent group sessions, ACP runtime: acpx + Claude Code.
• Steps: Set up three persistent Discord-bound agent sessions. Have the first dispatch tasks to the other two via sessions_send(timeoutSeconds: 0). The two targets each invoke a 7-14 minute ACP coding task. Meanwhile, the orchestrator agent also needs to emit a summary.
• Expected (after fix): Each target's ACP run executes on nested:<its-own-sessionKey>. The orchestrator's follow-up runs on nested:<orchestrator-sessionKey>. No cross-session blocking.
• Actual (before fix): All three contend on nested; the orchestrator's summary is queued behind both ACP runs and surfaces minutes late or not at all.

Evidence

Failing before: targeting distinct sessions still produced a single Map<string, LaneState> entry \"nested\"; existing tests literal-matched lane: \"nested\" so cross-session isolation wasn't covered by the suite.

Passing after:

• pnpm test src/agents/lanes.test.ts — 13 tests green (8 new, 5 existing).
• pnpm test src/agents/tools/sessions-send-tool.a2a.test.ts src/agents/command/delivery.test.ts src/agents/openclaw-tools.sessions.test.ts — 20 tests green across the touched consumers.
• pnpm test src/gateway/server.sessions-send.test.ts src/commands/agent.delivery.test.ts — 14 tests green with the updated pattern matcher + new per-session nested lane logging case.
• pnpm test src/cron/isolated-agent.lane.test.ts — 3 tests green (cron still uses the unscoped fallback path).
• Broader pnpm test 'src/agents/**/*.test.ts' sweep — 3841 tests green, 2 unrelated SSRF/network failures in web-fetch.provider-fallback.test.ts (environment-specific; pre-exist on upstream main).

Human Verification

• Verified: targeted unit tests; full pnpm test 'src/agents/**/*.test.ts' sweep except unrelated SSRF tests; pnpm lint clean on touched files (unrelated pre-existing failures in extensions/qa-lab and extensions/discord not introduced by this change); pnpm tsgo clean on touched files (unrelated pre-existing failures in extensions/discord/src/monitor/gateway-plugin.* not introduced by this change).
• Not verified: full pnpm build — fails on upstream main with an unrelated @clawdbot/lobster/core rolldown resolution error; did not address.
• Edge cases considered: empty / whitespace-only session keys (falls back to unscoped), cron paths using resolveNestedAgentLane(lane) (unchanged; still unscoped fallback), substring guards to ensure lane names like \"deeply-nested-lane\" or \"nestedfoo\" are not misclassified.

Review Conversations

• All Greptile findings addressed
• All ChatGPT Codex findings addressed

Compatibility / Migration

• Backward compatible: Yes. AGENT_LANE_NESTED is still exported; callers that continue to pass it directly keep their existing (unscoped) behaviour. Cron integration (resolveNestedAgentLane) was intentionally left unchanged.
• Config changes: None.
• Migration required: None.

Risks and Mitigations

• Risk: The lane-state Map grows as new session keys appear. Lane state is ~200 bytes per entry and is never GC'd today.

  • Mitigation: Same memory-growth story already exists for session:* lanes (src/agents/pi-embedded-runner/lanes.ts). The realistic upper bound is the number of active sessions, which is already bounded. A future "reap idle nested lanes" pass can be added if it becomes a concern; deliberately out of scope for this fix.

• Risk: A lane label collision between the scoped nested lane and a user-supplied lane named \"nested:something\".

  • Mitigation: resolveNestedAgentLane still honours caller-supplied lanes verbatim (first-call-wins via ?? resolveNestedAgentLaneForSession(...)), and the new prefix nested: is namespaced to mirror the existing session:<key> convention.

• Risk: Out-of-scope sub-issues from the same report (delivery-context corruption where the session's channel field degrades from \"discord\" to \"webchat\", and the absence of delivery retry / DLQ) remain.

  • Mitigation: Those are genuinely architectural follow-ups. This PR addresses the root lane-congestion cause the reporter documented first and lists the remaining two fixes as explicit future work in [Bug]: Assistant response lost during nested lane congestion + delivery context corruption #67502.

AI-assisted: This PR was developed with AI assistance.