Skip to content

[Bug]: command-queue module-scoped state duplicated by code-splitting, maxConcurrent config ignored #41901

@best

Description

@best

Bug type

Behavior bug (incorrect output/state without crash)

Summary

agents.defaults.maxConcurrent configuration is completely ignored. All agent requests are processed serially (maxConcurrent=1) regardless of configured value. Root cause: src/process/command-queue.ts stores runtime state (lanes, gatewayDraining, nextTaskId) as module-scoped variables, which get duplicated into 10 independent chunks by tsdown code-splitting. setCommandLaneConcurrency() only updates one chunk's state; actual message processing uses different chunks with default maxConcurrent=1.

Steps to reproduce

  1. Set agents.defaults.maxConcurrent: 10 in openclaw.json
  2. Send messages to two channels simultaneously (e.g. Telegram + Discord)
  3. Observe second message waits for first to complete (serial processing)
  4. Verify in dist:
rg -l 'let nextTaskId = 1;' dist/
# Shows 10 independent copies of command-queue state

Expected behavior

Configured maxConcurrent value is respected. Multiple channels process messages concurrently up to the configured limit.

Actual behavior

All requests serialize through maxConcurrent=1 regardless of configuration. Additionally:

  • resetAllLanes() (SIGUSR1) only resets one chunk's lanes, leaving stale activeTaskIds in other chunks
  • markGatewayDraining() is invisible to pi-embedded chunks during shutdown
# Gateway log showing serial blocking despite maxConcurrent=10:
lane wait exceeded: lane=main waitedMs=66810 queueAhead=0

# 10 chunks with independent state copies:
dist/compact-BIp3lbnh.js
dist/pi-embedded-CJVNBk0y.js
dist/pi-embedded-DePf5reQ.js
dist/plugin-sdk/dispatch-C5fHAX5a.js
dist/plugin-sdk/dispatch-D36HVShU.js
dist/plugin-sdk/dispatch-DHll-mp1.js
dist/plugin-sdk/dispatch-DJpqKByO.js
dist/plugin-sdk/dispatch-vBEK_Alg.js
dist/plugin-sdk/reply-DYZ7va08.js
dist/reply-BZQv8rey.js

OpenClaw version

2026.3.8 (verified still present in 2026.3.9 source)

Operating system

Linux 6.12.73+deb13-amd64 (x64)

Install method

npm global

Model

Any (bug is in queue infrastructure, model-independent)

Provider / routing chain

Any (bug is pre-provider)

Logs, screenshots, and evidence

OpenClaw already solved this exact code-splitting state duplication problem in two other modules:

src/hooks/internal-hooks.ts (lines 176-192):

// Uses a globalThis singleton so that registerInternalHook and
// triggerInternalHook always share the same Map even when the bundler
// emits multiple copies of this module into separate chunks (bundle splitting).
const _g = globalThis as typeof globalThis & {
  __openclaw_internal_hook_handlers__?: Map<string, InternalHookHandler[]>;
};
const handlers = (_g.__openclaw_internal_hook_handlers__ ??= new Map<string, InternalHookHandler[]>());

src/context-engine/registry.ts (lines 15-33):

const CONTEXT_ENGINE_REGISTRY_STATE = Symbol.for("openclaw.contextEngineRegistryState");
// Keep context-engine registrations process-global so duplicated dist chunks
// still share one registry map at runtime.
function getContextEngineRegistryState(): ContextEngineRegistryState {
  const globalState = globalThis as typeof globalThis & { ... };
  return (globalState[CONTEXT_ENGINE_REGISTRY_STATE] ??= { engines: new Map() });
}

command-queue.ts is the only remaining module with shared runtime state that lacks this treatment.

Impact and severity

Affected: All users with multiple channels or maxConcurrent > 1
Severity: High (silently ignores user configuration, forces serial processing)
Frequency: 100% reproducible
Consequence: Cross-channel blocking, degraded throughput, SIGUSR1 restart may leave stale tasks, gateway draining invisible to pi-embedded chunks

Additional information

Related: #16055 (same symptom reported by multiple users, root cause not identified), #1159 (closed feature request for parallel processing).

Temporary workaround: manually patch maxConcurrent: 1 to desired value in dist/pi-embedded-*.js after each upgrade, then restart gateway.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions