Skip to content

🐛 Session lock files not released after cron job completion (race condition) #31749

Description

@Wasslamenace

Problem

Session lock files () are not being released after cron job completion, causing complete system paralysis.

Symptoms

  • Cron job acquires session lock for write operation
  • Lock is never released after run completion (even when isError=false)
  • All subsequent message/run attempts timeout after 10s with "session file locked" error
  • System becomes completely unresponsive until manual intervention
  • Subsequent cron runs fail with cascading timeouts (backoff 30s ineffective)

Root Cause

Bug in session lifecycle management:

  • Lock acquired in sessions.ts for write operations
  • Lock release logic not triggered on run completion (possible race condition)
  • Announce/delivery mechanism tries to access session while still locked

Reproduction

  1. Create cron job with sessionTarget="main" and payload.kind="systemEvent"
  2. Wait for job to execute
  3. Try to send message to session → timeout 10s
  4. Check agents/main/sessions/*.lock → orphan lock file exists

Impact

Critical — Complete system paralysis requiring manual intervention:

# Manual recovery (only option)
rm -f /data/.openclaw/agents/main/sessions/*.lock
rm -f /data/.openclaw/agents/main/sessions/*.jsonl  # May be corrupted
docker restart <container>

Occurrences

  • 2026-03-02 00:43 — First incident (session 594d661f...c07.jsonl)
  • 2026-03-02 14:59 — Second incident (session 615691e5...5758.jsonl)
  • Frequency — ~2x in 14 hours (recurring)

Expected Behavior

Lock should be released immediately after run completion, regardless of:

  • isError status (true/false)
  • Delivery mode (none/announce/webhook)
  • Exception/crash during run

Environment

  • OpenClaw version — 2026.2.23 (npm install)
  • Runtime — Docker container (Hostinger VPS)
  • OS — Linux 6.8.0-101-generic (x64)
  • Node — v22.22.0

Workaround Applied

Created aggressive lock guardian monitoring:

  • Check every 15min (down from 1h)
  • Auto-cleanup locks >30min (down from 120min)
  • Backup sessions before removal

Workaround mitigates impact but does not fix root cause.

Proposed Fix

  1. Add finally block in session write operations to guarantee lock release
  2. Add timeout for lock acquisition (fail fast if lock held >60s)
  3. Add lock age check before acquiring (refuse if existing lock >5min)
  4. Log lock acquisition/release for debugging

Related Code

Likely affected files:

  • packages/runtime/src/sessions.ts (session lifecycle)
  • packages/runtime/src/cron.ts (cron job execution)
  • Lock acquisition/release logic

Request

Please investigate session lock lifecycle management and add guaranteed lock release on run completion.

This is a critical production bug causing complete system outages.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions