Bug type

Regression (worked before, now fails)

Beta release blocker

No (workaround: stay on 5.6)

Summary

OpenClaw's embedded agent runtime exhibits 20-37 second heartbeat fire latencies on 5.20, driven primarily by variance in the bootstrap-context stage (timings range 1-23 seconds across consecutive fires of the same agent class). This regression first manifested in our production environment on 5.12 (caused S45 rollback), is NOT fixed in 5.20 despite multiple related PRs (#80449, #83477, #83858), and is NOT present in 5.6.

Lab reproduction (clean 5.20 container)

Setup:

# Use any clean Linux base image with Node 22.22.2 + [email protected]
docker run -d --name lab -e OPENROUTER_API_KEY=<real-key> \
                       -e TELEGRAM_BOT_TOKEN=fake \
                       -e OPENCLAW_GATEWAY_TOKEN=fake \
                       <base-image> sleep 7200
docker exec lab npm i -g [email protected]

Config (key edits to clean clawdbot-like config):

{
  "agents": {
    "list": [
      {
        "id": "main",
        "model": { "primary": "openrouter/google/gemini-2.5-flash" }
        // ↑ Avoid openai-codex if no OAuth in lab; eliminates boot-md retry interference
      }
      // Plus 5 more agents (self-ops, career-intel, content-brain, etc.)
    ],
    "defaults": {
      "heartbeat": {
        "every": "1m",           // ← fast cycle for testing
        "model": "openrouter/google/gemini-2.5-flash",
        "target": "telegram",
        "lightContext": true,
        "isolatedSession": true
      }
    }
  }
}

Run:

docker exec -d lab sh -c "openclaw gateway > /tmp/gw.log 2>&1 &"
# wait ~220s
docker exec lab grep "trace:embedded-run" /tmp/gw.log

Empirical results (220s observation window, 6 agents × heartbeat.every=1m)

23 trace:embedded-run log entries captured (13 stream-ready phase + 10 attempt-dispatch phase).

Stream-ready phase totalMs — 13 samples sorted ascending (seconds):

Summary: 9/13 ≥ 20s (69%); 5/13 ≥ 28s (38%); MAX 37.1s; Median 20.4s; Mean ~23.9s.

Stage-level breakdown of one slow sample (totalMs=28141):

phase=stream-ready totalMs=28141 stages=
  workspace-sandbox:757ms@757ms,
  skills:0ms@757ms,
  core-plugin-tools:1271ms@2028ms,
  bootstrap-context:21151ms@23179ms,     ← LONG POLE (75% of total time)
  bundle-tools:534ms@23713ms,
  system-prompt:4ms@23717ms,
  session-resource-loader:2761ms@26478ms,
  agent-session:1ms@26479ms,
  stream-setup:1662ms@28141ms

bootstrap-context stage timings — 13 samples sorted (milliseconds):

1033, 1595, 1637, 2452, 2490, 3382, 3416, 4276, 6640, 13838, 20825, 21151, 22963

Bimodal distribution: 8 fast (1-7s) + 5 slow (13-23s). The slow tail is what drives the 20-37s total heartbeat fire latencies.

Confirmation these are heartbeat fires (not boot-md retries)

Liveness warning at 16:34:10 explicitly tagged the work:

[diagnostic] liveness warning: reasons=event_loop_delay,cpu interval=30s 
eventLoopDelayP99Ms=4183.8 eventLoopDelayMaxMs=7923 eventLoopUtilization=0.895
work=[active=agent:self-ops:main:heartbeat(processing/model_call,q=0,age=4s last=model_call:started)
     |agent:career-intel:main:heartbeat(processing/model_call,q=0,age=4s last=model_call:started)]

Both agent:self-ops:main:heartbeat and agent:career-intel:main:heartbeat were in flight, both at model_call:started for 4 seconds.

Triangulation against 5.6 (no bug)

On the same VPS, our production container running OpenClaw 2026.5.6 (uptime 2 days post-rollback from 5.12):

• Total trace:embedded-run entries over 2 days: 1 (phase=attempt-dispatch totalMs=5649, 5.6s normal completion)
• Zero phase=stream-ready entries in 2 days
• Zero heartbeat-related error logs
• Zero event_loop_delay warnings with heartbeat in work=[...]

Compared to 5.20 lab: 23 traces in 220 seconds; 5.6 production: 1 trace in 172,800 seconds. 5.6 is silent and fast; 5.20 emits frequently and is slow.

Our rollback evidence (5.12 → 5.6, 21s cutover) showed immediate restoration of normal heartbeat behavior — strong evidence that 5.6 doesn't have this bug class.

Regression timeline

What didn't fix it

• PR fix(agents): escalate LLM idle timeout to model fallback after profile rotation #80449 ("escalate LLM idle watchdog through profile rotation+model fallback", merged 2026-05-13, in 5.12+) — fallback chain is NOT being triggered for this code path. Our config has explicit fallbacks (gemini-3-flash-preview, mistral-small-2603) but next=none consistently appears in [model-fallback/decision] logs even when bootstrap-context exceeds 20s.
• PR Filter heartbeat response-tool transcript artifacts #83477 ("Filter heartbeat response-tool transcript artifacts", in 5.20) — addresses heartbeat artifact filtering in NORMAL prompts; doesn't affect heartbeat-fire latency itself.
• PR fix(memory): close local embedding providers on timeout #83858 ("close local embedding providers on timeout", listed in 5.20 release notes; shows CLOSED in gh pr view but mentioned in notes) — addresses memory leaks not latency; we're not using local embeddings (use OpenRouter's /embeddings endpoint).

Expected behavior

bootstrap-context stage should complete within a bounded time (similar to other phases like workspace-sandbox 0.7-1.9s or session-resource-loader 0.5-3s). On 5.6 this appears to be the case (no slow traces over 2 days production uptime).

Actual behavior (5.20)

bootstrap-context stage shows bimodal distribution with a slow mode at 13-23 seconds. This drives total heartbeat fire latency to 20-37 seconds, well exceeding any reasonable heartbeat budget.

Environment

• OpenClaw 2026.5.20 (commit e510042)
• Node 22.22.2
• Linux container (Ubuntu base, x86_64)
• OpenRouter as model provider (openrouter/google/gemini-2.5-flash for heartbeat + main agent in lab; production main agent uses openai-codex/gpt-5.5 via ChatGPT Plus OAuth but heartbeat is identical)
• 10 plugins loaded (active-memory, browser, canvas, device-pair, file-transfer, google, memory-core, phone-control, talk-voice, telegram)
• 6 agents configured with agents.defaults.heartbeat propagating per-agent

Questions for maintainers

1. Is there a known cause for bootstrap-context stage variance in 5.7-5.20 range?
2. Are there config knobs (timeout, max-files, context-budget) that could bound bootstrap-context execution time?
3. Why doesn't PR fix(agents): escalate LLM idle timeout to model fallback after profile rotation #80449's idle-watchdog escalation trigger for this path? (We see next=none in fallback decision logs even on 30s+ fires.)
4. Recommended path forward: file follow-up PR for bootstrap-context time budget, OR is 5.21+ already addressing this?

Workaround

Stay on 2026.5.6 indefinitely. Service-level impact is observability noise (~1,440 daily heartbeat error log entries with 60m cadence × 6 agents on the broken versions), not service-breaking, but it undermines monitoring trust.

Related issues / PRs

• PR fix(agents): escalate LLM idle timeout to model fallback after profile rotation #80449 (5.13) — supposed to fix this class, but next=none indicates it's not firing
• PR Filter heartbeat response-tool transcript artifacts #83477 (5.20) — heartbeat artifact filtering (separate concern)
• PR fix(memory): close local embedding providers on timeout #83858 (5.20 release notes, CLOSED in gh) — embedding cleanup (separate concern)
• Issue [Bug] Event loop freeze (9+ min) + Discord heartbeat timeout + local embeddings regression #75577 (closed 2026-05-01) — different symptom (event loop freeze 9+ min) but related code area