#92241: Gateway holds stale module import paths after update/rollback — inbound messages silently dropped (ERR_MODULE_NOT_FOUND) by wings1029 · Pull Request #92351 · openclaw/openclaw

wings1029 · 2026-06-12T05:40:20Z

Summary

Guard lazy reply-module loaders against ERR_MODULE_NOT_FOUND caused by stale
dist chunk hashes after an in-place npm install -g upgrade or rollback.
Previously, a running gateway process would silently drop all inbound messages
because the lazy import() resolved to chunk names that no longer existed on
disk. Now isDistRotationError detects the stale-dist condition, the guard
emits an operator-visible warning pointing to systemctl restart, and the
error propagates so the failure is no longer silent.

Root Cause

After npm install -g openclaw@<version>, the dist/ tree gets new chunk
hashes (e.g. cleanup-DlVQZQex.js → cleanup-DbGY5-v-.js). The running
gateway process still holds the old Node.js module cache, so a lazy dynamic
import() for a module that transitively references the old hash hits
ERR_MODULE_NOT_FOUND. The failure surfaces only when an inbound message
arrives and triggers the lazy load — health checks and channel connections
look fine, so the operator sees a healthy gateway silently dropping messages.

The fix adds a narrow detection function (isDistRotationError) in the shared
lazy-promise module and wraps the seven lazy-load call sites in
get-reply.ts with a guardedLoad helper that clears the stale cache and logs
the restart instruction.

Real behavior proof

Behavior addressed: isDistRotationError discriminates ERR_MODULE_NOT_FOUND /
MODULE_NOT_FOUND errors whose message references openclaw/dist/ (the tell
for a hashed chunk stale after update/rollback). guardedLoad catches that
condition, clears the stale loader promise, emits an operator-visible
error + warn log pair, and re-throws so the inbound delivery pipeline can
surface the failure instead of silently dropping messages.

Real environment tested: Linux 4.19.112-2.el8.x86_64, Node.js v22.22.0, tsx 4.x,
openclaw workspace import of production lazy-promise and infra/errors modules.

Exact steps or command run after this patch:

node scripts/run-vitest.mjs src/shared/lazy-promise.test.ts
./node_modules/.bin/tsx scripts/repro-92241.ts

Evidence after fix:

lazy-promise.test.ts: 11 tests passed (5 existing + 6 new isDistRotationError cases).
scripts/repro-92241.ts: 16 assertions passed, 0 failed, exercising positive/negative
detection, guardedLoad cache-clear, operator-warning log emission, error propagation,
and non-rotation pass-through.

Observed result after fix:

=== isDistRotationError: positive cases ===
  ✓ Linux ERR_MODULE_NOT_FOUND with openclaw/dist path → true
  ✓ Windows MODULE_NOT_FOUND with openclaw\dist path → true
  ✓ Hashed chunk ERR_MODULE_NOT_FOUND in dist → true

=== isDistRotationError: negative cases ===
  ✓ ERR_MODULE_NOT_FOUND outside openclaw → false
  ✓ ENOENT error → false
  ✓ Error without code → false
  ✓ null → false
  ✓ undefined → false
  ✓ string → false

=== guardedLoad: behavior verification ===
  ✓ guardedLoad: normal load returns value
  ✓ guardedLoad: normal load dedupes (single call)

--- dist rotation guard triggers (expected error output below) ---
[ERROR] auto-reply/reply-loader: bundled module changed under running gateway
after update/rollback — restart required (lazy module "stale-module" failed:
Cannot find module '/usr/lib/node_modules/openclaw/dist/chunks/stale-hash.js')
[WARN] auto-reply/reply-loader: run "systemctl --user restart
openclaw-gateway.service" (or equivalent) to reload dist modules
  ✓ guardedLoad: dist rotation error re-thrown
  ✓ guardedLoad: dist rotation clears stale cache
  ✓ guardedLoad: dist rotation propagated to caller
--- non-rotation error (expected pass-through, no clear) ---
  ✓ guardedLoad: non-rotation error does NOT clear cache
  ✓ guardedLoad: non-rotation error still re-thrown

=== Results: 16 passed, 0 failed ===

What was not tested: running an actual npm install -g upgrade/rollback while
the gateway is live with inflight Signal messages. That requires a full systemd
integration setup and is not feasible in this verification environment.

Regression Test Plan

pnpm test -- --run src/shared/lazy-promise.test.ts passes (11 tests)
tsx scripts/repro-92241.ts passes (16 assertions, 0 failed)
Change is minimal: 3 files, +83/-10, no new dependencies

AI-assisted: built with Claude Code

…s after update/rollback After an in-place npm install -g update or rollback, dist chunk hashes rotate while the gateway process is still running. Dynamic import() of lazy reply modules that transitively reference old hashed chunks fails with ERR_MODULE_NOT_FOUND, and the unprotected call sites silently swallow the error so inbound messages disappear. - Add isDistRotationError() classifier to detect ERR_MODULE_NOT_FOUND / MODULE_NOT_FOUND errors that reference openclaw/dist paths. - Add guardedLoad() wrapper that catches dist-rotation errors, clears the stale loader cache, logs a visible operator warning, and re-throws so the inbound delivery fails visibly instead of silently. - Route all seven reply-path lazy loaders through guardedLoad(). This is defense-in-depth: the operator gets a clear "restart required" message instead of discovering the silent message drop hours later. Closes openclaw#92241 Co-Authored-By: Claude <[email protected]>

clawsweeper · 2026-06-12T05:42:21Z

Codex review: needs real behavior proof before merge. Reviewed June 15, 2026, 4:15 PM ET / 20:15 UTC.

Summary
The PR adds an isDistRotationError classifier, wraps get-reply.ts lazy reply loaders with restart-required logging, and adds unit tests plus a standalone repro script.

PR surface: Source +70, Tests +66, Other +188. Total +324 across 4 files.

Reproducibility: yes. at source level: the linked rollback report plus current lazy import boundaries show a credible stale hashed chunk path after package replacement. No live current-main package-swap reproduction was established in this read-only review.

Review metrics: 2 noteworthy metrics.

Reply Lazy-Import Coverage: 7 inner loaders guarded; 1 outer resolver loader unguarded. The reported stale-dist failure can occur before the new reply-local guard is reached.
Operator Restart Guidance: 1 systemd command added. Supervisor-specific guidance can misdirect operators on other supported gateway launch modes.

Merge readiness
Overall: 🧂 unranked krab
Proof: 🦪 silver shellfish
Patch quality: 🧂 unranked krab
Result: blocked until stronger real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

[P1] Add redacted live update or rollback proof through an actual inbound message.
Move stale-dist handling to a lifecycle or outer dispatch boundary that covers the resolver import.
Replace the systemd-specific warning with supervisor-neutral restart guidance.

Proof guidance:

[P1] Needs stronger real behavior proof before merge: The PR supplies terminal output from unit and synthetic checks, but not a live installed gateway updated or rolled back with an actual inbound message; add redacted live terminal/log proof and update the PR body for re-review, or ask a maintainer to comment @clawsweeper re-review.

Risk before merge

[P1] The guard only covers inner get-reply.ts loaders; dispatch-from-config.ts can still hit the same stale-dist import before the new warning is reachable.
[P1] A rotated dist tree is terminal until process restart, but the loader path can retry and re-log on later inbound messages instead of moving to one process-owned recovery state.
[P1] The warning names a systemd user-service command even though supported gateways may run under launchd, containers, foreground processes, or custom supervisors.
[P1] The supplied proof does not show a live installed gateway updated or rolled back while handling an actual inbound message.

Maintainer options:

Consolidate Recovery Before Merge (recommended)
Move stale-dist handling to a boundary that covers the outer resolver and inner reply loaders, and make the terminal failure single-shot or process-owned.
Accept A Partial Diagnostic Deliberately
Maintainers could merge this as reply-local defense-in-depth only after live proof and an explicit rationale for leaving sibling imports outside the guard.
Pause For Lifecycle Design
If lifecycle owners want update or restart enforcement instead of a second auto-reply recovery policy, pause this branch while the canonical issue owns the fix shape.

Next step before merge

[P1] The remaining blocker is maintainer choice of the durable lifecycle/recovery boundary plus contributor live proof, not a narrow automated patch.

Security
Cleared: The diff adds local TypeScript detection, tests, and a repro script without dependency, lockfile, workflow, credential, permission, or package-resolution changes.

Review findings

[P1] Cover the outer reply runtime import — src/auto-reply/reply/get-reply.ts:141-158
[P1] Do not retry a terminal stale-dist loader — src/auto-reply/reply/get-reply.ts:125
[P2] Use supervisor-neutral restart guidance — src/auto-reply/reply/get-reply.ts:130-131

Review details

Best possible solution:

Centralize stale-dist detection and recovery at a process or outer reply-dispatch boundary that covers the full inbound path, emits supervisor-neutral guidance, avoids repeated retries against deleted chunks, and is backed by live package-swap inbound-message proof.

Do we have a high-confidence way to reproduce the issue?

Yes at source level: the linked rollback report plus current lazy import boundaries show a credible stale hashed chunk path after package replacement. No live current-main package-swap reproduction was established in this read-only review.

Is this the best way to solve the issue?

No. The classifier is useful, but the proposed fix is too narrow; the maintainable solution belongs at a lifecycle or outer dispatch boundary that covers every stale-dist import path.

Full review comments:

[P1] Cover the outer reply runtime import — src/auto-reply/reply/get-reply.ts:141-158
These guards only run after dispatch-from-config.ts has loaded get-reply-from-config.runtime.js. That outer lazy import can encounter the same removed dist chunk first, so inbound delivery can still fail before the restart warning; move the guard to a boundary that covers the outer resolver too.
Confidence: 0.95
[P1] Do not retry a terminal stale-dist loader — src/auto-reply/reply/get-reply.ts:125
For a rotated dist tree, clearing or evicting the loader cannot make the deleted chunk exist. Because lazy-promise already evicts rejected loads by default, each later inbound turn can retry, fail, and warn again until restart; cache a terminal state or trigger one process-owned recovery path.
Confidence: 0.94
[P2] Use supervisor-neutral restart guidance — src/auto-reply/reply/get-reply.ts:130-131
The new warning tells operators to run a systemd user-service command, but OpenClaw gateways also run under launchd, containers, foreground processes, and custom supervisors. Emit portable guidance or route through the existing restart owner.
Confidence: 0.92

Overall correctness: patch is incorrect
Overall confidence: 0.95

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 32af1c069727.

Label changes

Label justifications:

P1: The PR targets a gateway regression where a healthy-looking process can silently lose inbound messages after update or rollback.
merge-risk: 🚨 message-delivery: The diff changes central inbound reply loader failure behavior while leaving an earlier reply-dispatch import exposed to the same stale-dist failure.
merge-risk: 🚨 availability: The proposed recovery runs while the process and installed dist tree are inconsistent and can repeatedly fail until restart.
rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🦪 silver shellfish and patch quality is 🧂 unranked krab.
status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: The PR supplies terminal output from unit and synthetic checks, but not a live installed gateway updated or rolled back with an actual inbound message; add redacted live terminal/log proof and update the PR body for re-review, or ask a maintainer to comment @clawsweeper re-review.

Evidence reviewed

PR surface:

Source +70, Tests +66, Other +188. Total +324 across 4 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	2	78	8	+70
Tests	1	68	2	+66
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	1	188	0	+188
Total	4	334	10	+324

What I checked:

PR guard location: The potential merge result wraps seven inner get-reply.ts lazy loaders with guardedLoad, logs restart-required guidance, and rethrows matched dist-rotation errors. (src/auto-reply/reply/get-reply.ts:120, bb05532f0fcc)
Outer resolver remains unguarded: The same potential merge still lazy-loads get-reply-from-config.runtime.js in dispatch-from-config.ts before any new get-reply.ts guard can run. (src/auto-reply/reply/dispatch-from-config.ts:2858, bb05532f0fcc)
Rejected lazy imports retry by default: createLazyPromiseLoader evicts rejected promises unless cacheRejections is true, so a deleted dist chunk can be retried on later inbound messages even after the PR calls loader.clear(). (src/shared/lazy-promise.ts:64, bb05532f0fcc)
Current lifecycle precedent: Current main already treats in-place dist rotation as a process lifecycle concern by eagerly resolving the gateway lifecycle runtime before signal listeners. (src/cli/gateway-cli/run-loop.ts:111, 32af1c069727)
Related merged recovery history: Merged pull request fix(gateway): eager-load lifecycle runtime to survive in-place upgrades #84890 introduced the adjacent gateway lifecycle eager-load recovery behavior for the same old-process/new-dist failure class. (src/cli/gateway-cli/run-loop.ts, 4a9138556ea6)
Canonical broader stale-hash issue remains open: Related issue Auto-update can leave running gateway with stale hashed bundle imports #85844 remains open and the latest retest reports a stale @openclaw/codex compact chunk after v2026.6.6, so the durable update/recovery policy is not settled by this PR.

Likely related people:

myps6415: Authored the merged gateway lifecycle eager-loading fix for the same in-place package-swap and stale chunk failure class. (role: adjacent recovery contributor; confidence: high; commits: 4a9138556ea6; files: src/cli/gateway-cli/run-loop.ts)
vincentkoc: Merged the lifecycle recovery PR and appears in recent auto-reply history around the affected reply pipeline. (role: merger and recent adjacent contributor; confidence: medium; commits: 4a9138556ea6, 60e818f563d8; files: src/cli/gateway-cli/run-loop.ts, src/auto-reply/reply/get-reply.ts)
steipete: GitHub file history ties this handle to the lazy-loader abstraction and repeated auto-reply/shared-loader maintenance. (role: historical area contributor; confidence: medium; commits: db06fcd990ff, 00d8d7ead059; files: src/shared/lazy-promise.ts, src/auto-reply/reply/get-reply.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

…otation guard - Add 6 isDistRotationError test cases in lazy-promise.test.ts (positive, negative, boundary: null, undefined, non-object) - Add repro-92241.ts proof script importing production modules - Demonstrates guardedLoad detects ERR_MODULE_NOT_FOUND in dist/, clears stale cache, logs operator warning, re-throws Closes openclaw#92241 Co-Authored-By: Claude <[email protected]>

wings1029 · 2026-06-12T06:43:24Z

@clawsweeper re-review

clawsweeper · 2026-06-12T06:43:27Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Failed
Detail: The targeted re-review did not finish cleanly. Check the workflow run for details.
Run: https://github.com/openclaw/clawsweeper/actions/runs/27399480030
Updated: 2026-06-12T06:55:34.111Z

- src/shared/lazy-promise.ts: add {} around early-return bodies in isDistRotationError (curly rule) - scripts/repro-92241.ts: remove redundant boolean-literal comparisons, use empty catch binding Co-Authored-By: Claude <[email protected]>

wings1029 · 2026-06-12T07:11:00Z

@clawsweeper re-review

clawsweeper · 2026-06-12T07:11:03Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Failed
Detail: The targeted re-review did not finish cleanly. Check the workflow run for details.
Run: https://github.com/openclaw/clawsweeper/actions/runs/27401666312
Updated: 2026-06-12T07:39:12.235Z

wings1029 · 2026-06-12T07:33:11Z

@clawsweeper re-review

Co-Authored-By: Claude <[email protected]>

wings1029 · 2026-06-12T08:54:49Z

@clawsweeper re-review

clawsweeper · 2026-06-12T08:54:51Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Complete
Detail: The targeted re-review finished, the durable review comment was updated, and the synced verdict was routed.
Run: https://github.com/openclaw/clawsweeper/actions/runs/27405549521
Updated: 2026-06-12T09:01:32.379Z

… importer path Previously the regex matched openclaw/dist/ anywhere in the error message, which caused false positives when a third-party dependency was missing but the importer happened to be under openclaw/dist/. Now extract the missing module path from the Node.js ESM error format ("Cannot find module 'TARGET' imported from IMPORTER") and verify that the target itself is under the dist tree. - Add P1 regression test: third-party dep missing, importer in dist → false - Update positive test cases to use realistic "imported from" format Closes openclaw#92241 Co-Authored-By: Claude <[email protected]>

wings1029 · 2026-06-12T09:06:22Z

@clawsweeper re-review

clawsweeper · 2026-06-12T09:06:24Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Complete
Detail: The targeted re-review finished, the durable review comment was updated, and the synced verdict was routed.
Run: https://github.com/openclaw/clawsweeper/actions/runs/27406121280
Updated: 2026-06-12T09:13:46.799Z

openclaw-barnacle Bot added size: S triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 12, 2026

openclaw-barnacle Bot added scripts Repository scripts size: M and removed size: S labels Jun 12, 2026

openclaw-barnacle Bot added proof: supplied External PR includes structured after-fix real behavior proof. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 12, 2026

chore: trigger CI re-run

5db2ace

Co-Authored-By: Claude <[email protected]>

clawsweeper Bot mentioned this pull request Jun 15, 2026

Auto-update can leave running gateway with stale hashed bundle imports #85844

Open

clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Jun 15, 2026

wings1029 closed this Jun 17, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

#92241: Gateway holds stale module import paths after update/rollback — inbound messages silently dropped (ERR_MODULE_NOT_FOUND)#92351

#92241: Gateway holds stale module import paths after update/rollback — inbound messages silently dropped (ERR_MODULE_NOT_FOUND)#92351
wings1029 wants to merge 5 commits into
openclaw:mainfrom
wings1029:fix/issue-92241--bug---gateway-holds-stale-module-import-

wings1029 commented Jun 12, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jun 12, 2026 •

edited

Loading

Uh oh!

wings1029 commented Jun 12, 2026

Uh oh!

clawsweeper Bot commented Jun 12, 2026 •

edited

Loading

Uh oh!

wings1029 commented Jun 12, 2026

Uh oh!

clawsweeper Bot commented Jun 12, 2026 •

edited

Loading

Uh oh!

wings1029 commented Jun 12, 2026

Uh oh!

wings1029 commented Jun 12, 2026

Uh oh!

clawsweeper Bot commented Jun 12, 2026 •

edited

Loading

Uh oh!

wings1029 commented Jun 12, 2026

Uh oh!

clawsweeper Bot commented Jun 12, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

wings1029 commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Root Cause

Real behavior proof

Regression Test Plan

Uh oh!

clawsweeper Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wings1029 commented Jun 12, 2026

Uh oh!

clawsweeper Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wings1029 commented Jun 12, 2026

Uh oh!

clawsweeper Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wings1029 commented Jun 12, 2026

Uh oh!

wings1029 commented Jun 12, 2026

Uh oh!

clawsweeper Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wings1029 commented Jun 12, 2026

Uh oh!

clawsweeper Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

wings1029 commented Jun 12, 2026 •

edited

Loading

clawsweeper Bot commented Jun 12, 2026 •

edited

Loading

clawsweeper Bot commented Jun 12, 2026 •

edited

Loading

clawsweeper Bot commented Jun 12, 2026 •

edited

Loading

clawsweeper Bot commented Jun 12, 2026 •

edited

Loading

clawsweeper Bot commented Jun 12, 2026 •

edited

Loading