Skip to content

fix(infra/restart): exclude ancestor pids from stale-gateway cleanup#68517

Merged
steipete merged 2 commits into
openclaw:mainfrom
openperf:fix/weixin-sidecar-ancestor-cleanup
Apr 18, 2026
Merged

fix(infra/restart): exclude ancestor pids from stale-gateway cleanup#68517
steipete merged 2 commits into
openclaw:mainfrom
openperf:fix/weixin-sidecar-ancestor-cleanup

Conversation

@openperf

@openperf openperf commented Apr 18, 2026

Copy link
Copy Markdown
Member

Summary

  • Problem: The openclaw-weixin sidecar invokes cleanStaleGatewayProcessesSync() during its init path. The Unix filter in src/infra/restart-stale-pids.ts excluded only process.pid, and the two Windows filters did the same. Because the sidecar and its parent gateway share the openclaw command-name signature and the parent is listening on the configured gateway port, the parent's PID passed the filter and was SIGTERM'd. The supervisor restarted the gateway, the gateway re-spawned the sidecar, the cleanup ran again — an unbounded restart loop on ~45 s cadence, 100 % reproducible on main (2026.4.14 and 2026.4.15).

  • Root Cause: The pid !== process.pid filter encodes an invariant: "stale-gateway cleanup must never terminate a process whose death cascades into the caller." That invariant was applied only to the caller itself. Killing a direct ancestor is functionally identical to killing the caller — cgroup cleanup reaps the child, the supervisor (systemd / launchctl) respawns the ancestor, and the child re-enters the same cleanup call site. The same applies transitively to any ancestor whose death propagates via the supervisor. The existing guard was therefore not wrong, merely incomplete: it protected one node of the lineage where it needed to protect the whole chain.

  • Fix: Complete the self-exclusion invariant at the filter layer so the cleanup primitive is safe regardless of caller context. A new helper getSelfAndAncestorPidsSync() returns the set of PIDs comprising the current process, its process.ppid (always via Node's syscall — no spawn, cross-platform), and on Linux the transitive ancestor chain read from /proc/<pid>/status. parsePidsFromLsofOutput (Unix) and filterVerifiedWindowsGatewayPids / filterVerifiedWindowsGatewayPidsResult (Windows) both consult this set. The walk is bounded (MAX_ANCESTOR_WALK_DEPTH = 32), cycle-safe (early break on pids.has(parent)), and degrades silently on any /proc read failure — a deliberate product decision for hardened containers (hidepid=2 / gVisor / AppArmor), documented in the readParentPidFromProc catch-branch comment and locked by the "leaves the gateway grandparent in the kill list when /proc truncates the walk" regression test so the degraded outcome is explicit, not accidental. The fix is chosen over "detect misuse and refuse to run" or "patch the third-party sidecar call site" because (a) it repairs a design-level invariant rather than adding a detector, (b) it protects every present and future caller of the exported function, and (c) it has zero behavioural effect on the three legitimate supervisor-context callers (launchd.ts, restart.ts, gateway-cli/run.ts) whose ancestor chains are systemd/launchctl/shell and never listen on the gateway port.

  • What changed:

    • src/infra/restart-stale-pids.ts:
      • New import readFileSync from node:fs.
      • New constant MAX_ANCESTOR_WALK_DEPTH = 32.
      • New helper readParentPidFromProc(pid) — Linux /proc/<pid>/status parse, returns number | null. The catch branch carries a comment documenting that a null return truncates the ancestor walk at the current hop; for the direct gateway → sidecar topology this is safe because process.ppid is captured unconditionally without a /proc read, but a deeper gateway → plugin-host → sidecar chain on hidepid=2 / gVisor / AppArmor-locked namespaces leaves the gateway grandparent in the kill list. The degraded outcome is pinned by a regression test rather than silently allowed.
      • New helper getSelfAndAncestorPidsSync() — returns Set<number> of self + direct parent + transitive ancestors (Linux). The check admits any positive ancestor PID including 1, because a containerised gateway is frequently the namespace entrypoint (PID 1); excluding 1 unconditionally would recreate the [Bug]: openclaw-weixin sidecar kills main gateway #68451 loop on every containerised install. Written in early-return style (no dead inner guards).
      • parsePidsFromLsofOutput replaces the pid !== process.pid filter with !excluded.has(pid) where excluded = getSelfAndAncestorPidsSync(). Its JSDoc is updated to describe the Linux /proc behaviour (no more "pure function — no I/O" claim).
      • filterVerifiedWindowsGatewayPids and filterVerifiedWindowsGatewayPidsResult apply the same exclusion set before argv verification, so the parent gateway PID is dropped before readWindowsProcessArgsSync is even asked about it.
      • __testing export is unchanged — the existing setSleepSyncOverride / setDateNowOverride / callSleepSyncRaw remain; no new hooks were added. The ancestor walk is exercised by test-only stubbing of process.ppid and vi.mock('node:fs'), keeping the runtime-reachable mutation surface at zero.
    • src/infra/restart-stale-pids.test.ts:
      • vi.mock('node:fs') intercepts readFileSync so the /proc ancestor lookup can be driven deterministically. A precise as typeof actual.readFileSync cast satisfies tsgo's overload check without reaching for any.
      • mockReadFileSync hoisted mock + beforeEach that installs an ENOENT default (simulating a restricted /proc — the view a non-privileged process has under hidepid=2). Tests that need a specific /proc payload override with mockImplementation.
      • withStubbedPpid(ppid, fn) helper — Object.defineProperty(process, 'ppid', ...) + descriptor restore in finally, matching the pattern already used for process.platform stubbing in this file. No __testing mutation surface is introduced.
      • Five new regression tests:
        1. "excludes ancestor pids so a sidecar cannot kill its parent gateway — regression for [Bug]: openclaw-weixin sidecar kills main gateway #68451" — the reported direct-parent topology on the Unix lsof path.
        2. "excludes the full ancestor chain, not just the direct parent — deeper nesting" — it.skipIf(platform !== 'linux'), drives the real /proc walk with a mocked two-hop chain terminated by PPid:\t0.
        3. "excludes PID 1 when the direct parent gateway is the container entrypoint — container topology" — pins the > 0 guard (an earlier > 1 draft would have reopened the loop for containerised installs).
        4. "leaves the gateway grandparent in the kill list when /proc truncates the walk — documented degradation on hidepid/gVisor hosts" — it.skipIf(platform !== 'linux'), pins the degraded behaviour so a future refactor cannot silently regress further.
        5. "excludes ancestor pids on Windows too — [Bug]: openclaw-weixin sidecar kills main gateway #68451 regression mirror for the win32 path" — Windows filter via process.platform stub, asserting parent exclusion happens before the argv-verification step.
  • What did NOT change (scope boundary):

    • No change to the three existing call sites (launchd.ts, restart.ts, gateway-cli/run.ts) — the fix lives at the filter layer.
    • No change to lsof spawn logic, terminateStaleProcessesSync, waitForPortFreeSync, pollPortOnce, or any port/poll-budget timing constant.
    • No change to isGatewayArgv or the Windows argv-verification step; ancestor exclusion happens before that inspection, so argv lookups are never issued against caller-lineage PIDs.
    • No change to the __testing surface. No new runtime-reachable override — in particular, no setAncestorPidProviderOverride (an earlier iteration proposed one and it was removed per review feedback so no in-process module can neutralise ancestor exclusion at runtime).
    • No change to public API signatures. cleanStaleGatewayProcessesSync(portOverride?) keeps its exact contract.
    • No new dependencies. readFileSync is a Node built-in already transitively present in the module graph.
    • No any types introduced — new surfaces are typed (number | null, Set<number>, helpers) and the one unavoidable cast (as typeof actual.readFileSync) is a precise retype against the real module export, not a widening to any.

Reproduction

  1. Deploy openclaw 2026.4.14 or 2026.4.15 behind a supervisor (systemd, launchd, or equivalent with Restart=always).
  2. Install @tencent-weixin/openclaw-weixin v2.1.3 or v2.1.8 and enable it in plugins.allow.
  3. Start the gateway via the supervisor.
  4. Observe the gateway crash-loop at ~45 s cadence: systemd-cgls shows the gateway and sidecar in a tight respawn cycle; journalctl -u openclaw shows repeated EADDRINUSE / SIGTERM / exit-1 entries.
  5. Disable the plugin to confirm the loop stops.

With this patch applied, step 4 no longer loops: the sidecar's cleanup filter drops the parent gateway PID before any SIGTERM is sent, and the gateway remains stable.

Risk / Mitigation

  • Risk: The new getSelfAndAncestorPidsSync() is called from parsePidsFromLsofOutput, which executes once per lsof poll inside waitForPortFreeSync (up to ~40 calls per cleanup budget). On Linux each call may read up to 32 /proc/<pid>/status files; on macOS/Windows it is in-memory only.
  • Mitigation: /proc reads are readFileSync against a virtual filesystem — no disk I/O, no syscalls to external binaries. Worst-case arithmetic: 40 polls × ≤ 32 reads × few hundred bytes each ≈ well under 10 ms per cleanup, within noise of the existing spawnSync("lsof", …) overhead. Every read is wrapped in try/catch returning null; the walk also breaks on any null result, so a restricted /proc (user namespace, AppArmor, seccomp) degrades to "only self + ppid excluded" rather than failing. Existing tests pass unchanged because their synthetic stalePid = process.pid + N values are never in the caller's real ancestor chain; new tests drive the walk deterministically via withStubbedPpid and vi.mock('node:fs').
  • Risk: Path-based file read (/proc/${pid}/status) could be flagged by a security review for path injection.
  • Mitigation: pid is always an integer produced by process.ppid or by Number.parseInt of a /proc-sourced decimal field, filtered by Number.isFinite(parsed) && parsed > 0. No user input, no string concatenation from untrusted data, no shell — the value is always a positive integer interpolated into a fixed path shape.
  • Risk: Hardened-container topologies (hidepid=2, gVisor, AppArmor-locked namespaces) restrict /proc so the walk truncates at process.ppid, leaving a 3-level gateway → plugin-host → sidecar chain with the gateway grandparent outside the exclusion set.
  • Mitigation: Documented in code on the readParentPidFromProc catch branch and pinned by the "leaves the gateway grandparent in the kill list when /proc truncates the walk" regression test. The reported [Bug]: openclaw-weixin sidecar kills main gateway #68451 case is direct-parent (gateway → sidecar) and is fully covered on every platform via process.ppid without depending on /proc. A fuller fix for deeper chains on hardened hosts (macOS/Windows ancestor walk, pidfd-based Linux walk, or privileged cmdline probe) is tracked out of band.

Change Type (select all)

  • Bug fix

Scope (select all touched areas)

  • Gateway
  • Supervisor / restart logic (src/infra/restart-stale-pids.ts)
  • Cross-platform port cleanup (Unix lsof + Windows PowerShell/netstat)
  • Tests (src/infra/restart-stale-pids.test.ts)

Linked Issue/PR

Fixes #68451

@openclaw-barnacle openclaw-barnacle Bot added size: S maintainer Maintainer-authored PR labels Apr 18, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d26748ccca

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/infra/restart-stale-pids.ts Outdated
@greptile-apps

greptile-apps Bot commented Apr 18, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes an infinite supervisor restart loop caused by cleanStaleGatewayProcessesSync() SIGTERMing a parent gateway process when called from a child sidecar (openclaw-weixin). The fix extends the existing self-exclusion invariant to cover the full ancestor PID chain by introducing getSelfAndAncestorPidsSync(), applied consistently across the lsof, Windows sync, and Windows result filter paths.

The implementation is correct — bounded walk, cycle detection, graceful /proc failure handling — and the three new regression tests are hermetic and well-targeted. One minor documentation note: the parsePidsFromLsofOutput JSDoc retains the phrase "Pure function — no I/O" even though the function now delegates to getSelfAndAncestorPidsSync(), which performs /proc reads on Linux.

Confidence Score: 5/5

Safe to merge; the only finding is a stale doc comment that does not affect runtime behaviour.

All findings are P2. The logic is correct, the ancestor walk is bounded and cycle-safe, error paths degrade gracefully, tests cover all three filter code paths, and the public API is unchanged.

No files require special attention beyond the minor comment update in src/infra/restart-stale-pids.ts.

Prompt To Fix All With AI
This is a comment left during a code review.
Path: src/infra/restart-stale-pids.ts
Line: 147-150

Comment:
**Stale "Pure function — no I/O" comment**

The retained `Pure function — no I/O` claim is no longer accurate after this PR. `parsePidsFromLsofOutput` now calls `getSelfAndAncestorPidsSync()`, which reads up to 32 `/proc/<pid>/status` files on Linux. Future contributors relying on this comment to reason about testing, memoization, or call-site constraints will be misled.

```suggestion
/**
 * Parse openclaw gateway PIDs from lsof -Fpc stdout.
 * Excludes the current process and its ancestors
 * (see `getSelfAndAncestorPidsSync` for the full rationale).
 * Note: performs /proc reads on Linux via getSelfAndAncestorPidsSync.
 */
```

How can I resolve this? If you propose a fix, please make it concise.

Reviews (1): Last reviewed commit: "fix(infra/restart): exclude ancestor pid..." | Re-trigger Greptile

Comment thread src/infra/restart-stale-pids.ts Outdated
@openperf
openperf force-pushed the fix/weixin-sidecar-ancestor-cleanup branch 2 times, most recently from bb1a8e1 to 0e79318 Compare April 18, 2026 11:22
@openperf
openperf force-pushed the fix/weixin-sidecar-ancestor-cleanup branch 4 times, most recently from 84eec58 to 5118a62 Compare April 18, 2026 13:08
@steipete
steipete force-pushed the fix/weixin-sidecar-ancestor-cleanup branch from 5118a62 to fef3036 Compare April 18, 2026 16:54
@steipete
steipete force-pushed the fix/weixin-sidecar-ancestor-cleanup branch from fef3036 to 7520a06 Compare April 18, 2026 16:54
openperf and others added 2 commits April 18, 2026 17:59
The stale-gateway cleanup filter already refused to kill process.pid —
acknowledging the invariant that terminating a process whose death
cascades into the caller is never safe. That invariant was applied only
to the caller itself, not to its ancestors, which is why the
openclaw-weixin sidecar triggered an unbounded restart loop: the
sidecar's cleanup SIGTERM'd its parent gateway, the supervisor
restarted the gateway, the gateway re-spawned the sidecar, the cleanup
ran again.

Complete the invariant by excluding the full self+ancestor PID set in
both the lsof (Unix) and PowerShell/netstat (Windows) cleanup paths.
Walk uses process.ppid unconditionally (Node built-in, no spawn) and
/proc/<pid>/status on Linux for transitive ancestors, with graceful
degradation where /proc is unavailable.
@steipete
steipete force-pushed the fix/weixin-sidecar-ancestor-cleanup branch from 7520a06 to 45e0b44 Compare April 18, 2026 16:59
@steipete
steipete merged commit 76891c9 into openclaw:main Apr 18, 2026
47 checks passed
@steipete

Copy link
Copy Markdown
Contributor

Landed via rebase. Thanks @openperf.

Commits:

This should close the stale-gateway cleanup restart loop from #68451.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

maintainer Maintainer-authored PR size: M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: openclaw-weixin sidecar kills main gateway

2 participants