Skip to content

Discord channel lanes intermittently held after run completes — "lane wait exceeded" with stale per-session jsonl mtime #72421

Description

@jessewunderlich

Symptom

User-sent Discord messages stop being processed in a channel even though the gateway is healthy and other channels work. Diagnostic log emits repeatedly:

lane wait exceeded: lane=session:agent:main:discord:channel:<id> waitedMs=<>200000 queueAhead=0|1

with no concurrent LLM activity (per-session <sessionId>.jsonl mtime is 5+ minutes stale, gateway CPU low). Manually deleting the session entry from agents/main/sessions/sessions.json and restarting unblocks the channel; new messages then process normally.

Repro hypothesis

Lane release path appears to miss a branch when a run completes via cron / heartbeat / system-event entry rather than a user-initiated reply. Multiple paths in the bundled dist do call sessionLock.release() (e.g. compact, wait-for-idle-before-flush, transcript-rewrite, selection), but at least one terminal path leaves the in-memory lock held without an exception/abort. Fresh user messages then pile up behind a lane that thinks it is still processing.

Concrete observation: in a 24h window we saw lane wait exceeded entries for 3 distinct Discord channels with queueAhead=0 (head-of-queue stuck) and waitedMs ranging 220s–600s, with the per-session jsonl untouched for the entire wait period.

Environment

  • OpenClaw 2026.4.24 (homebrew)
  • Node 25.9.0
  • macOS arm64
  • Discord plugin, multiple guilds/channels, several main-session crons feeding through

Repro steps (best-effort)

  1. Configure several payload.kind=systemEvent crons targeted at sessionTarget=main plus current-bound agentTurn jobs that share Discord channel session keys.
  2. Run for hours.
  3. Eventually one channel goes silent: queued user messages don't trigger a reply, gateway is otherwise healthy.

I have not been able to produce a tight unit-test repro — it appears to be a race between a run terminating and the lock-release callback in certain delivery paths.

Workaround

cp ~/.openclaw/agents/main/sessions/sessions.json{,.bak}
python3 - <<'PY'
import json
p = '<sessions.json path>'
d = json.load(open(p))
del d['agent:main:discord:channel:<channel-id>']
json.dump(d, open(p,'w'), indent=2)
PY
openclaw gateway restart

I've shipped a detection-only watchdog on my side that scans the diagnostic log for lane wait exceeded and alerts to a Discord ops channel — happy to PR that as a CLI subcommand if useful.

What would help

  1. Confirm whether sessionLock.release() is invoked from every terminal path of runAgentTurn / processSystemEvent — particularly the cron-delivered and heartbeat-delivered branches.
  2. Add a defensive watchdog inside the gateway: any lane held >N minutes with no in-flight LLM call should be auto-released (with a warning log).
  3. Expose lane state via openclaw status --lanes for easier diagnosis.

Happy to provide gateway logs / sessions.json snapshots privately if helpful.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions