Bug type

Performance / latency

Summary

User-initiated Discord DM turns on 2026.5.20 consistently take ~50–65 s end-to-end, but only ~3–15 s of that is the actual LLM call. The remaining ~48 s is a silent gap between embedded run done and lane task done in which the gateway emits zero log entries even at OPENCLAW_LOG_LEVEL=trace beyond a periodic heartbeat. The session lane completes on its own (no lane wait exceeded, no stuck session warning), but the reply is held for ~48 s before delivery.

The behavior is reproducible on every Discord DM turn, regardless of message length, model latency, or TTS configuration.

Environment

• OpenClaw 2026.5.20
• Node v24.14.0
• OS: Synology DSM, kernel 4.4.302+ (cgroup v1), 4 CPUs, 64 GB RAM, no container CPU/memory limit
• Gateway: node dist/index.js gateway --bind lan --port 18789 --verbose
• OPENCLAW_LOG_LEVEL=trace (set for diagnostic capture)
• OPENCLAW_CODEX_APP_SERVER_SANDBOX=danger-full-access
• Channels: Discord (DM mode, mention not required)
• Plugins enabled: browser, codex, discord, elevenlabs, memory-core, searxng (7/94)
• Model: openai/gpt-5.5 via codex-app-server (OAuth)

Observed timeline (typical turn — TIMING_TEST_69)

16:17:01.016   discord: inbound id=...
16:17:01.x     [discord-preflight + routing]                   ~50 ms
16:17:07.x     [plugins] loading codex/openai/browser/...      ~1–6 s  (plugin re-load)
16:17:07.650   [agent/embedded] trace:embedded-run startup stages:
                 phase=attempt-dispatch totalMs=574
                 stages=workspace:2ms,runtime-plugins:148ms,hooks:1ms,
                 model-resolution:314ms,auth:90ms,context-engine:6ms,
                 attempt-workspace:11ms,attempt-prompt:0ms,
                 attempt-runtime-plan:2ms,attempt-dispatch:0ms
16:17:07.802   liveness warning: reasons=event_loop_delay
                 eventLoopDelayMaxMs=2246 eventLoopUtilization=0.222
16:17:08.846   codex app-server stderr: ERROR Codex could not find bubblewrap
16:17:10.248   run registered
16:17:16.603   embedded run done: durationMs=9515 aborted=false
16:17:37.802   heartbeat: webhooks=0/0/0 active=0 waiting=0 queued=0
                                                  ↑ ← only log entry in 48 s
16:18:04.698   lane task done: lane=main durationMs=57615 active=0 queued=0
16:18:04.703   lane task done: lane=session:agent:main:main durationMs=57633
16:18:04.816   message processed: ... outcome=completed duration=63489ms
16:18:06.889   discord: delivered 1 reply

End-to-end: 65.9 s. Embedded LLM: 9.5 s. Silent post-LLM gap: ~48 s.

What we ruled out

• TTS — Set messages.tts.auto=off, gateway restarted, behavior unchanged. Same ~48 s gap. (messages.tts.providers.elevenlabs plugin still loaded; the silence isn't TTS-bound.)
• The LLM — durationMs from embedded run done ranges 3.3–15 s across three test turns. All within normal gpt-5.5 latency.
• Plugin loading / pre-stream prep — attempt-dispatch totalMs=574. The stream-ready phase prep that [Bug]: 2026.4.29 embedded agent reply latency ~40-47s before stream-ready; packaged channel deps missing #75650 talks about does not appear in our traces; we're on the post-2026.4.29 codepath.
• Infrastructure — 4 idle CPUs, no memory pressure, no cgroup limits. cAdvisor shows the container averages ~2 % CPU during these turns.
• Stuck-session detection — stuck session watchdog does not fire (lane completes within its threshold). No lane wait exceeded either.

Three additional observations

1. shouldWarnEmbeddedRunStageSummary thresholds never trigger. The bundled code (dist/selection-BmjEdnnA.js) defines:

   const EMBEDDED_RUN_STAGE_WARN_TOTAL_MS = 1e4;   // 10s
   const EMBEDDED_RUN_STAGE_WARN_STAGE_MS = 5e3;   // 5s

   Our 48-65 s totals should warn 5× over, yet no [trace:embedded-run] prep stages warning is ever emitted at any log level. Either emitPrepStageSummary is never invoked on the user-turn code path, or shouldWarn is computed against a different time window (per-stage rather than total elapsed).

2. OPENCLAW_CODEX_APP_SERVER_SANDBOX=danger-full-access does not suppress the bubblewrap probe. Every turn the codex app-server emits:

   ERROR codex_app_server: Codex could not find bubblewrap on PATH.
   Install bubblewrap with your OS package manager.
   

   …twice per turn (separated by ~7 s). The env var is honored (codex runs without sandbox), but the underlying check that produces the stderr message isn't gated by it.

3. liveness warning: event_loop_delay eventLoopDelayMaxMs=2246 fires inside the embedded-run window. The Node event loop is being blocked for ~2.2 s by something synchronous. This is concerning independent of the post-LLM gap.

Repro

In any clean 2026.5.20 install with Discord enabled and OPENCLAW_LOG_LEVEL=trace:

1. DM the bot any short message (e.g. hi).
2. Wait for reply.
3. grep -E "discord: inbound|embedded run done|lane task done|discord: delivered" /tmp/openclaw/openclaw-<date>.log for the turn.

Expected: lane task done durationMs ≈ embedded run done durationMs + a few hundred ms.
Actual: lane task done durationMs ≈ embedded run done durationMs + ~45–55 s with no intervening log activity.

Related (but distinct)

• Discord channel lanes intermittently held after run completes — "lane wait exceeded" with stale per-session jsonl mtime #72421 (closed) — Discord channel lanes intermittently held permanently after run completes, requiring sessions.json surgery. Same lane subsystem, but their lane never recovers without intervention; ours completes after ~48 s.
• [Bug]: Embedded-run bootstrap-context stage variance causes 20-37s heartbeat fires on 5.20 (regression from 5.6; not fixed by 5.12 PR #80449 or 5.20 PRs #83477/#83858) #85783 (open) — Embedded-run bootstrap-context stage variance causing 20-37 s heartbeat fires on 5.20. Different phase (stream-ready prep), different trigger (heartbeats, not user turns). Possibly same root cause in the embedded runner — both surface as silent variable delay on 5.20.
• [Bug]: 2026.4.29 embedded agent reply latency ~40-47s before stream-ready; packaged channel deps missing #75650 (closed) — Pre-stream 39-47 s embedded-run prep on 2026.4.29. Fixed for the phase=stream-ready path; the symptom has now apparently relocated to a post-embedded run done phase that has no equivalent trace instrumentation.
• [Bug]: Stuck processing sessions are detected but never aborted — gateway requires external restart to recover #71127 / [Bug]: Agent processing lane can stall for minutes without timeout recovery, plus memory-core dreaming cron race condition on Gateway restart #73581 (both closed) — Lane stall recovery. Our lane doesn't stall (it completes).

What would help

1. Trace logging for the post-embedded run done lane pipeline. Whatever is happening between embedded run done and lane task done is fully silent — even at trace, even with --verbose. The user/operator has no insight into the 48 s.
2. Fix the warning emission. EMBEDDED_RUN_STAGE_WARN_TOTAL_MS=10000 is meaningless if our 60 s+ totals don't trigger it. Either the snapshot isn't taken on this path or the comparison is against the wrong field.
3. Gate the bubblewrap stderr emission on OPENCLAW_CODEX_APP_SERVER_SANDBOX=danger-full-access (or equivalently document why it must continue to emit).