Skip to content

[Bug]: Three async interleaving races in cron, command-queue, and plugin-binding subsystems #64065

Description

@lihaokun

Bug type

Behavior bug (incorrect output/state without crash)

Beta release blocker

No

Summary

Formal concurrency interleaving analysis (predictive race detection via RAPID) identified three async races across src/process/command-queue.ts, src/plugins/hook-runner-global.ts, and src/plugins/conversation-binding.ts. All three were confirmed by instrumenting real source code and running reproduction scripts with node --import tsx/esm. One is a true data race (approvals cache), two are confirmed-but-mitigated by existing design patterns.

Steps to reproduce

These are async interleaving issues found via formal analysis, not runtime crash reports. Reproduction scripts exercise real source code:

Race A — resetAllLanes stale task resolution (src/process/command-queue.ts:337-354)

# Reproduction: stale task Promises resolve after generation bump
node --import tsx/esm race-analysis/repro-race3-reset-generation.ts
# Result: 20/20 — stale tasks resolve with original result after resetAllLanes()

When resetAllLanes() bumps state.generation while an async task is between await entry.task() (line 195) and completeTask() (line 196), the generation check correctly ignores the stale completion. However, entry.resolve(result) (line 204) still fires — the caller receives a result from a task whose lane was already invalidated.

Race B — Hook runner init/read ordering (src/plugins/hook-runner-global.ts:32-75)

node --import tsx/esm race-analysis/repro-race4-hook-runner-init.ts
# Result: 20/20 — readers see null hookRunner before initializeGlobalHookRunner completes

initializeGlobalHookRunner() writes state.registry (line 35) then state.hookRunner (line 36). Concurrent readers via getGlobalHookRunner() / hasGlobalHooks() see null before init completes. The null-safe API (?., ?? false) handles this correctly.

Race C — Approvals cache coherency (src/plugins/conversation-binding.ts:366-391)

node --import tsx/esm race-analysis/repro-race7-approvals-cache.ts
# Result: 20/20 — concurrent saves both complete with no serialization

saveApprovals() writes approvalsCache and approvalsLoaded synchronously (lines 370-371), then awaits writeJsonAtomic() (line 372). During this await:

  • A concurrent getApprovals() reads the in-memory cache (correct)
  • A second concurrent saveApprovals() overwrites the in-memory cache
  • The first writer's writeJsonAtomic() completes and persists stale data to disk, while the in-memory cache already has the second writer's data

This creates a disk/memory divergence: the file on disk has writer-1's data, approvalsCache in memory has writer-2's data. On next process restart, the stale file is loaded.

Expected behavior

  • Race A: After resetAllLanes(), invalidated task Promises should not deliver results to callers, or callers should be aware the result is from a stale generation.
  • Race B: getGlobalHookRunner() should not return null after initializeGlobalHookRunner() has been called, or the init-before-use contract should be documented.
  • Race C: Concurrent saveApprovals() calls should be serialized so the file on disk always reflects the latest in-memory state.

Actual behavior

  • Race A: Stale task Promises resolve normally. The generation counter in completeTask() prevents re-pumping, but the Promise resolution still reaches the caller. Mitigated by existing design — callers use Promise.allSettled after reset.
  • Race B: Readers see null before init. Mitigated by existing null-safe API patterns (?.hasHooks(), ?? false).
  • Race C: Two concurrent saveApprovals() both complete. The last writer wins in memory, but the first writer's writeJsonAtomic() may persist stale data to ~/.openclaw/plugin-binding-approvals.json. Not mitigated — potential data loss on restart.

OpenClaw version

2026.4.9 (commit e673efe, main branch)

Operating system

Linux (WSL2, analysis environment)

Install method

pnpm dev (source analysis)

Model

N/A (static analysis, not model-dependent)

Provider / routing chain

N/A

Additional provider/model setup details

N/A — this is a static/formal analysis finding, not dependent on any specific provider or model configuration.

Logs, screenshots, and evidence

Analysis methodology: Source code instrumented with trace recording calls (traceEmit) at shared variable access points. Test drivers import real modules via node --import tsx/esm. Traces analyzed by RAPID predictive race detection engine (HB, LockSet, WCP, SyncPreserving algorithms).

RAPID results for the three races:

Race Trace HB SP Repro Rate
A (resetAllLanes + gen) Scene 1 75 42 20/20
B (hook runner init) Scene 3 18 4 20/20
C (approvals cache) Scene 6 4 3 20/20

Note: Many RAPID-detected race pairs within these traces are false positives due to JS run-to-completion semantics (synchronous code blocks cannot be interleaved). The races listed above are specifically the ones that occur across await boundaries and are confirmed reproducible.

Impact and severity

  • Race A: Low. Mitigated by generation counter + Promise.allSettled. Only affects SIGUSR1 in-process restart edge case.
  • Race B: Low. Mitigated by null-safe API. Only affects the brief window between process start and plugin init.
  • Race C: Medium. Not mitigated. Concurrent plugin binding approval operations can cause plugin-binding-approvals.json to contain stale data. On restart, approvals granted by the last writer may be lost. Frequency depends on how often concurrent binding requests occur.

Additional information

Full analysis artifacts (instrumented source, traces, reproduction scripts) are available in the race-analysis/ directory on branch race-analysis.

Found by Haokun Li, Xiaoxue Ma @LittleS321, and Claude Code via formal concurrency interleaving analysis.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Normal backlog priority with limited blast radius.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:data-lossCan lose, corrupt, or silently drop user/session/config data.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions