Skip to content

Session lock never auto-cleans when held by the gateway process itself #21783

Description

@pvrooyen

Bug Description

When an agent run hangs (e.g., LLM API timeout, rate-limit stall), the session write lock (.jsonl.lock) is never automatically cleaned up if the lock is held by the gateway process itself. This causes all subsequent agent activity to fail indefinitely with session file locked (timeout 10000ms) until the container is manually restarted or the lock file is manually deleted.

Root Cause

In src/agents/session-write-lock.ts, the stale lock detection logic (lines ~165-172) checks two conditions before cleaning a lock:

  1. isAlive(pid) — uses process.kill(pid, 0) to check if the owning process is alive
  2. Age-based — lock older than staleMs (default 30 minutes)

The problem: when the gateway process (e.g., pid 14 inside Docker) acquires a lock for an embedded agent run, and that run hangs during an API call, the lock is held by the same process that will try to acquire it for subsequent requests. Since isAlive(14) always returns true (the gateway IS alive), the stale detection never fires, even after 30 minutes — because by that point, the hung operation has completed or timed out internally but the lock file was never cleaned up in the error path.

The stale timeout only helps when the lock-holding process has died. When the process is alive but the lock was leaked (e.g., an unhandled rejection in the agent run path that skipped the finally block, or a SIGKILL during the lock hold), isAlive returns true and the lock persists forever.

Impact

  • All WhatsApp/Telegram auto-replies fail
  • All heartbeat agent runs fail
  • All cron jobs targeting the main session fail
  • The gateway appears healthy (container running, WhatsApp connected) but the agent is completely non-functional
  • Only fix is manual intervention: delete the lock file + restart

Observed Timeline

  1. Lock file created at 10:21:06 UTC by pid 14 (the gateway)
  2. User messages at 10:25 UTC trigger agent run → session file locked (timeout 10000ms) on both primary and fallback models
  3. Lock never auto-cleaned because pid 14 is alive
  4. Required manual rm of lock file + container restart to recover

Suggested Fix

Consider one or more of:

  1. Self-pid detection: If payload.pid === process.pid and the lock is older than a shorter threshold (e.g., 2-5 minutes), treat it as stale. The gateway process shouldn't hold a single session lock for minutes — if it does, something went wrong.

  2. Lock lease with renewal: Instead of a static lock file, use a lease-based approach where the lock must be periodically renewed. If not renewed within N seconds, it's automatically considered stale.

  3. Configurable stale timeout: Allow staleMs to be configured via openclaw.json or environment variable so operators can tune it for their deployment.

  4. Watchdog thread: A background interval that scans for locks older than N minutes held by the current process and releases them.

Environment

  • OpenClaw 2026.2.152026.2.19-2
  • Docker deployment, single gateway container
  • Node 22.22.0
  • Lock file path: /home/node/.openclaw/agents/main/sessions/<sessionId>.jsonl.lock

Relevant Code

  • src/agents/session-write-lock.ts (lock acquisition, stale detection, cleanup)
  • src/agents/pi-embedded-runner/run/attempt.ts:387 (lock usage in agent runs)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions