Skip to content

Gateway sync-FS plugin discovery blocks event loop on every channels.status / channel restart #78100

Description

@fede-kamel

Symptom

On slow filesystems (WSL2 9P, NFS, container overlays, encrypted volumes) the gateway exhibits long event-loop blocks and inbound dispatch latencies of 17–25s. WhatsApp Web channels also disconnect repeatedly with 408 timeouts, leading to a self-reinforcing reconnect loop.

Repro

  • WSL2 with locally-built pnpm gateway:watch
  • Linked WhatsApp account
  • Send a self DM via pnpm openclaw message send --channel whatsapp --target +<self> --message "hi"
  • Observe inbound web messagecli exec gap of 17–25s in /tmp/openclaw/openclaw-*.log
  • Observe eventLoopDelayMaxMs=6446ms in liveness warnings during dispatch
  • Observe WhatsApp Web connection closed (status 408) shortly after

Root cause

loadPluginRegistrySnapshotWithMetadata (src/plugins/plugin-registry-snapshot.ts) is invoked from channels.status polls and channel-restart paths. Each invocation re-walks the filesystem synchronously via realpathSync, statSync, readFileSync to verify manifest hashes, package.json signatures, and bundled-root paths.

23-minute CPU profile (node --cpu-prof on the gateway process):

Total CPU time: 1380s
  661s  native::(idle)         48%  (waiting on sync syscalls)
  380s  native::lstat          27.6%
  105s  native::stat            7.6%
   46s  native::fstat           3.3%
   46s  native::close           3.3%

Top JS callers of those syscalls:

65s   resolveBoundaryPathSync       (boundary-path resolver)
47s   safeRealpathSync via discovery checkSourceEscapesRoot
41s   installed-plugin-index resolvePackageJsonPath
38s   installed-plugin-index resolvePackageJsonRelativePath
37s   discoverInDirectory (plugin walker)
17s   getPathKindSync
15s   readLexicalStat
30s   openVerifiedFileSync (3 sites)
11s   isFilesystemMountPoint
13s   safeFileSignature / safeHashFile (manifest re-hashing)
4s    resolveComparablePath in loadPluginRegistrySnapshotWithMetadata

Cascading WhatsApp disconnects

Baileys' keepAliveIntervalMs is 25s. When the event loop blocks for 30s+ during channels.whatsapp.start-account or other discovery-heavy phases, the WS keepalive misses its window and WA returns 408. The disconnect triggers a channel restart, which re-runs start-account, which re-runs discovery, which blocks again, and so on. Block-to-disconnect correlation in one session:

disconnect preceding event-loop block
17:19:04 57s
17:41:52 67s
18:07:31 56s
16:08:03 107s

Proposed fixes

  1. (this PR/follow-up) Coalesce loadPluginRegistrySnapshotWithMetadata calls within a short TTL window
  2. Async-ify the sync-FS work in plugin discovery + boundary-path resolution + installed-plugin-index rebuild so the event loop is not blocked
  3. Process-lifetime memoization for safeRealpathSync / boundary-path resolution where path topology is stable

The repo's src/plugins/CLAUDE.md currently says "Do not add persistent metadata caches for discovery, manifest registries, installed-index reconstruction" — this guidance keeps the design freshness-first, but on slow filesystems the literal cost of "always fresh" is the event loop. Worth an architectural conversation about mtime-gated caching vs. no caching.

Workarounds

  • Run on native Linux/macOS (FS calls 10–50× faster than WSL2 9P)
  • Use Telegram or other push-based channels (no WS keepalive sensitivity)

Affects

All channels and CLI commands on slow filesystems. WhatsApp Web is the most visible symptom because of its keepalive sensitivity.

Metadata

Metadata

Assignees

Labels

P1High-priority user-facing bug, regression, or broken workflow.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.maturity:stableIssue affects a taxonomy feature currently scored M4/M5.

Type

No type

Fields

Priority

None yet

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions