fix(core): recover sessions and jobs from transient storage failures - #53704

Why

When the host disk filled for ~90 seconds during active work, SQLite WAL buffered transaction pages until COMMIT failed with SQLITE_FULL. Durable event writes crashed in-flight turns, the subsequent SessionEvent.Execution.Failed publish also failed and was dropped, and those sessions kept their write-ahead execution claims and appeared busy forever. At the same time, background jobs persisted their outcome inside Job.settle before resolving job.done, so a failed KV write left finished jobs stuck in "running" state in memory, hung their waiters, silently dropped background subagent parent notifications, and caused prompt admissions to return 500s.

What Changes

sequenceDiagram
    participant Runner as SessionRunner / Job
    participant Exec as SessionExecution / SubagentJob
    participant Bus as Bus / KV
    participant DB as SQLite (WAL)

    rect rgb(60, 35, 35)
    Note over Runner,DB: Before: 90 s full disk
    Runner->>Bus: publish(StepEvent)
    Bus->>DB: COMMIT
    DB-->>Bus: SQLITE_FULL (rollback)
    Bus-->>Exec: Defect (SQLITE_FULL)
    Exec->>Bus: publish(Execution.Failed) + release claim
    Bus->>DB: COMMIT
    DB-->>Bus: SQLITE_FULL (rollback)
    Note over Exec,DB: Terminal event dropped; claim stays held and session looks busy forever
    end

    rect rgb(30, 55, 40)
    Note over Runner,DB: After: < 30 s full disk
    Runner->>Bus: publish(StepEvent)
    loop StorageRetry.write (up to 30 s)
        Bus->>DB: BEGIN IMMEDIATE ... COMMIT
        DB-->>Bus: SQLITE_FULL (rollback)
    end
    Note over DB: Disk space frees
    Bus->>DB: BEGIN IMMEDIATE ... COMMIT
    DB-->>Bus: OK
    Bus-->>Runner: Committed once, turn continues
    end

    rect rgb(35, 45, 65)
    Note over Runner,DB: After: > 30 s full disk (e.g. 90 s)
    Runner->>Bus: publish(StepEvent)
    Bus-->>Exec: Exhausted after 30 s (SQLITE_FULL)
    loop StorageRetry.settlement (interruptible, up to 15 min)
        Exec->>Bus: publish(Execution.Failed { type: "storage" }) + release claim
        Bus->>DB: COMMIT
        DB-->>Bus: SQLITE_FULL (rollback)
    end
    Note over DB: Disk space frees
    Exec->>Bus: publish(Execution.Failed { type: "storage" }) + release claim
    Bus->>DB: COMMIT
    DB-->>Bus: OK (claim released, session marked failed)
    end

Storage Recovery & Tradeoffs

• Visible failure instead of auto-resume: When a turn fails because mid-turn writes exhausted the 30 s StorageRetry.write window, it records SessionEvent.Execution.Failed with { type: "storage" } and releases the execution claim once storage recovers. It does not auto-resume the turn after storage returns, avoiding re-executing tool calls with external side effects. If the server shuts down while waiting to record the terminal event, the interruptible settlement retry exits immediately and leaves the write-ahead claim intact for normal restart recovery.
• Retry windows: StorageRetry.write (100 ms exponential backoff capped at 2 s, up to 30 s) is sized to keep HTTP requests and active steps responsive while riding out short stalls. StorageRetry.settlement (1 s exponential backoff capped at 10 s, up to 15 min) is reserved for terminal session outcomes and subagent completion notices that nothing else will re-drive.
• SQLITE_IOERR and double-commit risk: StorageRetry classifies SQLITE_FULL, SQLITE_BUSY, SQLITE_LOCKED, and SQLITE_IOERR (including node:sqlite's ERR_SQLITE_ERROR with primary codes 5, 6, 10, 13) as transient. SQLite automatically rolls back the transaction on SQLITE_FULL, SQLITE_BUSY, SQLITE_LOCKED, and most SQLITE_IOERR failures, making whole-transaction replay atomic. However, an I/O error occurring late in COMMIT after WAL frames have flushed to disk can theoretically report SQLITE_IOERR after the transaction has already become durable, which carries a small risk of applying a retried transaction twice.

Scope

This PR covers @opencode/core durable event transactions (Bus), session lifecycle settlement (SessionExecution, toSessionError), in-memory vs durable background job settlement (Job), background subagent parent notifications (SubagentJob), and the StorageFault SQLite full-disk test harness.

Verification

bun run --cwd packages/core typecheck
bunx oxlint packages/core/src/bus.ts packages/core/src/database/storage-retry.ts packages/core/src/job.ts packages/core/src/session/execution.ts packages/core/src/session/subagent-job.ts packages/core/src/session/to-session-error.ts packages/core/test/lib/storage-fault.ts packages/core/test/session-execution.test.ts packages/core/test/session-runner.test.ts packages/core/test/storage-retry.test.ts
bun run --cwd packages/core test test/storage-retry.test.ts test/bus.test.ts test/bus-session-routing.test.ts test/job.test.ts test/kv.test.ts test/tool-subagent.test.ts test/session-error.test.ts test/session-execution.test.ts test/session-run-coordinator.test.ts test/session-runner.test.ts test/session-store.test.ts test/session-prompt.test.ts test/session-wait.test.ts test/session-shell.test.ts
bun run --cwd packages/core test test/database.test.ts test/database-drizzle.test.ts test/session-projector.test.ts test/session-log.test.ts test/session-runner-retry.test.ts test/session-runner-recorded.test.ts test/session-runner-message.test.ts test/session-runner-tool-events.test.ts test/session-step.test.ts test/session-revert.test.ts test/session-remove.test.ts test/session-move.test.ts test/session-owned.test.ts test/session-view.test.ts test/session-compact.test.ts test/session-compaction.test.ts test/session-activation.test.ts test/session-create.test.ts

• packages/core typecheck and repo-wide bun run check (lint + 36-package turbo typecheck) pass with 0 errors.
• 448 tests across the 14 storage, bus, job, subagent, and session execution/runner suites pass, plus 197 tests across the 18 additional database and session suites.
• In the full packages/core suite, test/plugin/provider-digitalocean.test.ts fails 6 OAuth callback tests with ConnectionRefused against http://localhost:1456/auth/*; verified the identical 6 failures on unmodified origin/v2 (34cf183c4c).