Bug Description
When an agent run hangs (e.g., LLM API timeout, rate-limit stall), the session write lock (.jsonl.lock) is never automatically cleaned up if the lock is held by the gateway process itself. This causes all subsequent agent activity to fail indefinitely with session file locked (timeout 10000ms) until the container is manually restarted or the lock file is manually deleted.
Root Cause
In src/agents/session-write-lock.ts, the stale lock detection logic (lines ~165-172) checks two conditions before cleaning a lock:
isAlive(pid) — uses process.kill(pid, 0) to check if the owning process is alive
- Age-based — lock older than
staleMs (default 30 minutes)
The problem: when the gateway process (e.g., pid 14 inside Docker) acquires a lock for an embedded agent run, and that run hangs during an API call, the lock is held by the same process that will try to acquire it for subsequent requests. Since isAlive(14) always returns true (the gateway IS alive), the stale detection never fires, even after 30 minutes — because by that point, the hung operation has completed or timed out internally but the lock file was never cleaned up in the error path.
The stale timeout only helps when the lock-holding process has died. When the process is alive but the lock was leaked (e.g., an unhandled rejection in the agent run path that skipped the finally block, or a SIGKILL during the lock hold), isAlive returns true and the lock persists forever.
Impact
- All WhatsApp/Telegram auto-replies fail
- All heartbeat agent runs fail
- All cron jobs targeting the main session fail
- The gateway appears healthy (container running, WhatsApp connected) but the agent is completely non-functional
- Only fix is manual intervention: delete the lock file + restart
Observed Timeline
- Lock file created at
10:21:06 UTC by pid 14 (the gateway)
- User messages at
10:25 UTC trigger agent run → session file locked (timeout 10000ms) on both primary and fallback models
- Lock never auto-cleaned because pid 14 is alive
- Required manual
rm of lock file + container restart to recover
Suggested Fix
Consider one or more of:
-
Self-pid detection: If payload.pid === process.pid and the lock is older than a shorter threshold (e.g., 2-5 minutes), treat it as stale. The gateway process shouldn't hold a single session lock for minutes — if it does, something went wrong.
-
Lock lease with renewal: Instead of a static lock file, use a lease-based approach where the lock must be periodically renewed. If not renewed within N seconds, it's automatically considered stale.
-
Configurable stale timeout: Allow staleMs to be configured via openclaw.json or environment variable so operators can tune it for their deployment.
-
Watchdog thread: A background interval that scans for locks older than N minutes held by the current process and releases them.
Environment
- OpenClaw
2026.2.15 → 2026.2.19-2
- Docker deployment, single gateway container
- Node 22.22.0
- Lock file path:
/home/node/.openclaw/agents/main/sessions/<sessionId>.jsonl.lock
Relevant Code
src/agents/session-write-lock.ts (lock acquisition, stale detection, cleanup)
src/agents/pi-embedded-runner/run/attempt.ts:387 (lock usage in agent runs)
Bug Description
When an agent run hangs (e.g., LLM API timeout, rate-limit stall), the session write lock (
.jsonl.lock) is never automatically cleaned up if the lock is held by the gateway process itself. This causes all subsequent agent activity to fail indefinitely withsession file locked (timeout 10000ms)until the container is manually restarted or the lock file is manually deleted.Root Cause
In
src/agents/session-write-lock.ts, the stale lock detection logic (lines ~165-172) checks two conditions before cleaning a lock:isAlive(pid)— usesprocess.kill(pid, 0)to check if the owning process is alivestaleMs(default 30 minutes)The problem: when the gateway process (e.g., pid 14 inside Docker) acquires a lock for an embedded agent run, and that run hangs during an API call, the lock is held by the same process that will try to acquire it for subsequent requests. Since
isAlive(14)always returnstrue(the gateway IS alive), the stale detection never fires, even after 30 minutes — because by that point, the hung operation has completed or timed out internally but the lock file was never cleaned up in the error path.The stale timeout only helps when the lock-holding process has died. When the process is alive but the lock was leaked (e.g., an unhandled rejection in the agent run path that skipped the
finallyblock, or aSIGKILLduring the lock hold),isAlivereturns true and the lock persists forever.Impact
Observed Timeline
10:21:06 UTCby pid 14 (the gateway)10:25 UTCtrigger agent run →session file locked (timeout 10000ms)on both primary and fallback modelsrmof lock file + container restart to recoverSuggested Fix
Consider one or more of:
Self-pid detection: If
payload.pid === process.pidand the lock is older than a shorter threshold (e.g., 2-5 minutes), treat it as stale. The gateway process shouldn't hold a single session lock for minutes — if it does, something went wrong.Lock lease with renewal: Instead of a static lock file, use a lease-based approach where the lock must be periodically renewed. If not renewed within N seconds, it's automatically considered stale.
Configurable stale timeout: Allow
staleMsto be configured viaopenclaw.jsonor environment variable so operators can tune it for their deployment.Watchdog thread: A background interval that scans for locks older than N minutes held by the current process and releases them.
Environment
2026.2.15→2026.2.19-2/home/node/.openclaw/agents/main/sessions/<sessionId>.jsonl.lockRelevant Code
src/agents/session-write-lock.ts(lock acquisition, stale detection, cleanup)src/agents/pi-embedded-runner/run/attempt.ts:387(lock usage in agent runs)