Bug type

Regression (worked before, now fails)

Beta release blocker

No

Summary

After upgrading from v2026.4.23 to v2026.4.26, every chat turn's embedded-runner prep phase takes ~50+ seconds (LLM call ~3s; total ~76s) on ARM hardware. Befor upgrade was like 15s.

Steps to reproduce

1. On an ARM SBC (orangepi4pro, Node 24.15.0), install OpenClaw v2026.4.26 (also reproduces on v2026.4.27).
2. Configure a single LLM provider (minimax) and a single channel (telegram); leave bundled plugins
   (brave/duckduckgo/exa/firecrawl/google/minimax/moonshot/ollama/perplexity/searxng/tavily/xai) at their default enabled-by-default state.
3. Start the gateway: pnpm openclaw gateway.
4. Wait for startup to settle, then send one chat message via Telegram.
5. Observe: the gateway logs [plugins] loading from .../dist/extensions//index.js for every bundled plugin on every turn, the [trace:embedded-run] prep
   stages line reports totalMs ≈ 53000 with core-plugin-tools, system-prompt, and stream-setup each ≈ 15000ms, and the user-perceived turn takes ~75-80 seconds
   end-to-end while the LLM call portion is ~3 seconds.
6. Repeat on the same gateway process (no restart): subsequent turns show the same ~50s prep, confirming the work is per-turn rather than one-time startup.

Expected behavior

On the same hardware running v2026.4.23 (the last release widely reported as stable, per community threads referenced in #60528 and the piunikaweb 2026-04-29
article), each chat turn completed in roughly 5-10 seconds end-to-end with the LLM call being the dominant portion. The intended caching pattern is also
documented in src/plugins/CLAUDE.md:

▎ Cache concept: metadata stays fresh unless a caller owns an explicit PluginMetadataSnapshot, PluginLookUpTable, or manifest registry for the current flow. Do
▎ not add persistent metadata caches for discovery, manifest registries, installed-index reconstruction, owner lookup, model suppression, provider policy,
▎ public-artifact metadata, or similar control-plane answers.

Per that guidance, the embedded-runner per-turn entry points (loadAgentToolResultMiddlewaresForRuntime, resolvePluginTools, resolveRuntimePluginRegistry,
createOpenClawCodingTools) should be invoked with a caller-owned manifestRegistry / PluginRegistrySnapshot that is built once per session (or per process) and
reused across turns, instead of each call re-deriving discovery, manifest registry, installed-plugin index, package-root walks, and per-provider
auth-profile-store reads from disk on every turn. The expected [trace:embedded-run] prep stages totalMs on this hardware is in the 5000-8000ms range, comparable
to what an opt-in process-wide cache produces today (validated at 8257ms on this same hardware: see sergeyksv/openclaw survival/plugin-cache).

Actual behavior

Each chat turn takes ~75-80 seconds end-to-end on the orangepi4pro, with the LLM call accounting for only ~3 seconds. Direct evidence:

1. Embedded runner stage trace — emitted because total exceeds the EMBEDDED_RUN_STAGE_WARN_TOTAL_MS = 10_000 / EMBEDDED_RUN_STAGE_WARN_STAGE_MS = 5_000
   thresholds in src/agents/pi-embedded-runner/run/attempt-stage-timing.ts:

[agent/embedded] [trace:embedded-run] prep stages:
phase=stream-ready totalMs=53561 stages=
workspace-sandbox:21ms
skills:1ms
core-plugin-tools:14976ms
bootstrap-context:51ms
bundle-tools:3438ms
system-prompt:15783ms
session-resource-loader:3445ms
agent-session:2ms
stream-setup:15844ms

A separate startup-phase trace adds another ~22.8s (model-resolution 6.5s, auth 7.0s, attempt-dispatch 9.3s) on the first turn after gateway start.

2. Per-turn plugin re-import — every chat turn the gateway logs:

[plugins] loading brave from /home/sergey/openclaw/dist/extensions/brave/index.js
[plugins] loading duckduckgo from /home/sergey/openclaw/dist/extensions/duckduckgo/index.js
... (all 12 bundled extensions) ...
[plugins] loaded 12 plugin(s) (12 attempted) in 189.9ms

This fires on every turn, not only at startup.

3. strace evidence — the same plugin manifest files are re-opened hundreds of times per turn:

openat .../dist-runtime/extensions/brave/package.json
openat .../dist-runtime/extensions/brave/openclaw.plugin.json
openat .../dist-runtime/extensions/brave/index.js
openat .../dist-runtime/extensions/browser/package.json
... [cycles through every plugin, then repeats] ...

4. node --prof summary (single chat turn):

[Summary]
ticks total nonlib name
5388 8.6% 21.7% JavaScript
19464 31.2% 78.3% C++
3377 5.4% 13.6% GC
37468 60.1% Shared libraries

[C++ entry points] (top)
3233 39.6% syscall@@glibc
1208 14.8% access@@glibc
936 11.5% __open@@glibc
771 9.4% __read@@glibc

[Bottom up (heavy) profile] (hottest JS leaf)
3391 ticks findPackageRootSync (~5.4% of total CPU)
<- resolveOpenClawPackageRootSync
<- resolveBundledPluginsDir
<- resolvePluginSourceRoots
<- discoverOpenClawPlugins (78.6%)
<- resolvePluginCacheInputs (21.4%)

discoverOpenClawPlugins is invoked 700+ times per chat turn with identical inputs (verified by adding a process-wide stack-trace probe to the function entry).

5. Per-factory breakdown of core-plugin-tools (instrumented):

[openclaw-tools-timing]
session-agent=0ms image-tool=612ms image-generate=1263ms
video-generate=1547ms music-generate=1730ms pdf-tool=2665ms
web-search=2841ms web-fetch=2842ms message-tool=2852ms
nodes-tool=2852ms tools-array=2858ms plugin-tools=2919ms

createPdfTool ~935ms, createImageGenerateTool ~651ms, createImageTool ~612ms — all dominated by hasAuthForProvider (src/agents/tools/model-config.helpers.ts:37)
which reads auth-profiles.json from disk per provider, called dozens of times per tool factory.

No user-visible error is raised — the system functions correctly, just very slowly. The 30-60s wait per chat turn is the only externally visible symptom.

OpenClaw version

2026.4.26

Operating system

Ubuntu 24.04

Install method

No response

Model

minimax

Provider / routing chain

openclaw->minimax

Additional provider/model setup details

No response

Logs, screenshots, and evidence

Impact and severity

Who is affected: Everyone. Confirmed slow on ARM SBC (orangepi4pro), Intel i7, and Apple M2. Earlier in this debugging cycle, before per-plugin disabling was
applied as a mitigation, turn time was on the order of minutes; selectively disabling unused bundled plugins is currently the only thing keeping turns finite
for most users.

Severity: BLOCKER. Every chat turn pays ~75-80s of plugin/runtime overhead with the LLM portion only ~3s — the product is functionally unusable for interactive
chat unless the user happens to be on hardware so fast it makes the regression appear merely "annoying" rather than "broken."

How often: Every single chat turn, deterministically. No warm-up effect within a process.

Practical consequence: OpenClaw cannot be used for its primary function on any reasonable hardware as of v2026.4.5+. Workarounds in the wild (rollback to
v2026.4.23 — see piunikaweb 2026-04-29, or manually disabling every bundled plugin not strictly needed) sacrifice features and ship daily breakage risk. A
working opt-in cache mitigation (sergeyksv/openclaw survival/plugin-cache) recovers ~85% of the latency without provider, network, or LLM-side changes — proving
the cost is purely repeated work and the regression is recoverable.

Additional information

Summary

Since approximately v2026.4.5, the embedded runner's per-chat-turn prep
phase re-evaluates process-lifetime data (plugin discovery, manifest
registry, installed-plugin index, registry snapshot, package root,
bundled plugins dir, auth-profile store) on every turn. On fast x86
dev hardware this manifests as a 3-8 second prep; on ARM SBCs it's
30-60 seconds, with the LLM call itself being only 2-3 seconds of that.

This is likely the same root cause underlying #60528, #62051, #71938
and the chokidar-loader regression in #73176. Filing as a separate
issue because I have concrete trace evidence pinpointing the eight
hot points and a reference patch that recovers ~85% of the latency.

Environment

• OpenClaw: 2026.4.27 (adc20fe), reproduces on every release since ~2026.4.5
• Hardware: orangepi4pro (ARM)
• Provider: minimax
• Channel: telegram
• Node: 24.15.0

Evidence

[trace:embedded-run] prep stages — single chat turn after warm startup

Unpatched (current main):

totalMs=53561 stages=
workspace-sandbox:21ms
skills:1ms
core-plugin-tools:14976ms <-- 15s
bootstrap-context:51ms
bundle-tools:3438ms
system-prompt:15783ms <-- 16s
session-resource-loader:3445ms
agent-session:2ms
stream-setup:15844ms <-- 16s

Three independent stages all ~15s with similar shape — characteristic
signature of three call sites each re-walking the same shared
dependency graph (plugin registry / auth profile resolution).

With the cache patch (linked below):

totalMs=8257 stages=
core-plugin-tools:2580ms (-83%)
bundle-tools:495ms (-86%)
system-prompt:2326ms (-85%)
session-resource-loader:545ms (-84%)
stream-setup:2290ms (-86%)

End-to-end chat turn including LLM call drops from ~76s to ~11s.

node --prof baseline (unpatched)

[Summary]
ticks total nonlib name
5388 8.6% 21.7% JavaScript
19464 31.2% 78.3% C++
3377 5.4% 13.6% GC
37468 60.1% Shared libraries

[C++ entry points] top
3233 39.6% syscall@@glibc
1208 14.8% access@@glibc
936 11.5% __open@@glibc
771 9.4% __read@@glibc

[Bottom up (heavy) profile] top JS leaves
3391 ticks (~5.4% of total) findPackageRootSync
<- resolveOpenClawPackageRootSync
<- resolveBundledPluginsDir
<- resolvePluginSourceRoots
<- discoverOpenClawPlugins (78.6%)
<- resolvePluginCacheInputs (21.4%)

discoverOpenClawPlugins is invoked 700+ times per chat turn with
identical inputs (verified with stack-trace probe).

Where each second goes (instrumented and confirmed)

1. hasAuthForProvider (model-config.helpers.ts:37) re-reads
   auth-profiles.json from disk per provider. Called ~30 times per
   createPdfTool / createImageTool / createImageGenerateTool
   construction.
2. resolvePluginToolRegistry (tools.ts:99) → resolveRuntimePluginRegistry
   per turn; falls through to loadOpenClawPlugins on cacheKey miss.
3. discoverOpenClawPlugins (discovery.ts:845) runs synchronous
   readdirSync + readFileSync on every plugin manifest each call.
4. resolveBundledPluginsDir → resolveOpenClawPackageRootSync →
   findPackageRootSync walks fs from cwd looking for package.json
   on every call.
5. Legacy doctor migrations (legacy-config-compat.ts) call into
   plugin manifest registry on every config read.

Root cause (best guess)

src/plugins/CLAUDE.md documents the intended pattern:

Cache concept: metadata stays fresh unless a caller owns an explicit
PluginMetadataSnapshot, PluginLookUpTable, or manifest registry
for the current flow. Do not add persistent metadata caches for
discovery, manifest registries, installed-index reconstruction...

The seam parameters (manifestRegistry, index, candidates) exist
on most call sites — loadAgentToolResultMiddlewaresForRuntime,
loadPluginManifestRegistry, loadInstalledPluginIndex,
loadPluginRegistrySnapshot, loadOpenClawPlugins — but no caller
in the agent runtime threads a snapshot through. Each per-turn entry
point re-derives the registry from disk.

Why this isn't visible to maintainers

On M-series Macs / fast x86, the re-derivation is fast enough that
it reads as 3-8s prep, which sits below the
EMBEDDED_RUN_STAGE_WARN_TOTAL_MS = 10_000 warn threshold and
EMBEDDED_RUN_STAGE_WARN_STAGE_MS = 5_000 per-stage threshold in
attempt-stage-timing.ts. No CI assertion runs on slow hardware.
Tests follow the agents/CLAUDE.md guideline of using lightweight
artifacts instead of cold-loading plugin runtime, so the test suite
never exercises the per-turn cold path. The trace infrastructure exists
for debugging, not monitoring, so the regression goes unnoticed
across daily releases.

Mitigation (env-gated, no behavior change when unset)

I maintain a survival branch with eight process-lifetime caches at the
hot points above, opt-in via OPENCLAW_DIRTY_DISCOVERY_CACHE=1:

https://github.com/sergeyksv/openclaw/tree/survival/plugin-cache

The commit body is the canonical reference for the eight cache layers,
per-layer rationale, and per-stage measurements. This is not offered
as an upstream PR — src/plugins/CLAUDE.md explicitly forbids
persistent metadata caches at this layer. It exists so that other users
hitting this regression on slow hardware can apply a working patch
while waiting for the proper fix.

Proper fix

Plumb a PluginRegistrySnapshot (or PluginMetadataSnapshot) through
the per-turn call chain, owned by the gateway / agent runtime for the
process lifetime:

• loadAgentToolResultMiddlewaresForRuntime({manifestRegistry}) ←
  caller already accepts this; nothing passes it
• resolvePluginTools({snapshot}) and
  resolveRuntimePluginRegistry({snapshot}) similar
• createOpenClawCodingTools should not call hasAuthForProvider per
  provider during construction; either move the enumeration into
  execute() (lazy) or use static capability descriptors per the
  agents/tools/CLAUDE.md guideline

This is a 200+ line PR across ~15 files. I am not in a position to
write that PR (small fork, no insight into invariants the snapshot
must satisfy), but I'm happy to share more diagnostic detail or
help reproduce.

Asks

1. Confirm whether this duplicates [Bug] High pre-processing latency (8-12s) on webchat after gateway restart, not in v2026.2.26 #60528 or should stay separate.
2. If a perf gate on a CPU-budget runner with a real chat turn could be
   added to CI, the next regression of this shape would be caught
   automatically. Currently nothing surfaces it.
3. The trace warn thresholds (10000ms total / 5000ms stage) seem high
   given the LLM portion is ~3s. Lowering to e.g. 3000ms / 1500ms
   would make the trace useful for proactive monitoring rather than
   only post-hoc investigation.