Skip to content

Commit 0db4cbb

Browse files
test(daemon): seeded concurrency torture lane for session/lease/lock invariants
Refs #1416 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
1 parent 8cce0ef commit 0db4cbb

8 files changed

Lines changed: 1274 additions & 0 deletions

File tree

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
name: Concurrency Torture Nightly
2+
3+
# Seeded concurrency torture lane for session/lease/lock invariants (#1416,
4+
# umbrella #1412 Track A). N concurrent clients drive randomized-but-seeded
5+
# interleavings of open/mutate/close/takeover/kill against the real SessionStore
6+
# + LeaseRegistry, with all concurrency routed through a deterministic scheduler
7+
# so a seed fully determines execution order. Deterministic and offline — no
8+
# devices, simulators, or wall-clock stress (those are out of scope for #1416).
9+
#
10+
# Scheduled + manual only: the PR gate already runs a fast default sweep via the
11+
# Node integration lane; this nightly sweeps a much larger seed range to keep
12+
# mining for ordering bugs. A failure prints the seed and the exact
13+
# `TORTURE_SEED=<n> pnpm test:concurrency-torture` replay command.
14+
15+
on:
16+
schedule:
17+
- cron: '0 5 * * *'
18+
workflow_dispatch:
19+
inputs:
20+
runs:
21+
description: 'Number of seeded interleavings to sweep'
22+
required: false
23+
default: '5000'
24+
seed-start:
25+
description: 'First seed in the sweep'
26+
required: false
27+
default: '0'
28+
29+
permissions:
30+
contents: read
31+
32+
concurrency:
33+
group: ci-${{ github.workflow }}-${{ github.ref }}
34+
cancel-in-progress: true
35+
36+
jobs:
37+
torture:
38+
name: Session/lease/lock torture sweep
39+
runs-on: ubuntu-latest
40+
timeout-minutes: 20
41+
env:
42+
TORTURE_RUNS: ${{ github.event.inputs.runs || '5000' }}
43+
TORTURE_SEED_START: ${{ github.event.inputs.seed-start || '0' }}
44+
steps:
45+
- name: Checkout
46+
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
47+
48+
- name: Setup toolchain
49+
uses: ./.github/actions/setup-node-pnpm
50+
51+
- name: Run concurrency torture sweep
52+
run: pnpm test:concurrency-torture

‎docs/agents/testing.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -189,6 +189,33 @@ AGENT_DEVICE_WEB_E2E=1 pnpm test:smoke:web
189189

190190
The test is skipped unless `AGENT_DEVICE_WEB_E2E=1` is set. The test runs `agent-device web setup` and `agent-device web doctor` with an isolated state directory before opening the fixture URL, so it verifies the public managed-backend setup path instead of relying on a global `agent-browser`. CI runs the lane on Node 24 because the managed backend requires Node >= 24. Failure artifacts, daemon state, and browser config are written under `test/artifacts/web/`.
191191

192+
## Concurrency torture lane
193+
194+
`test/integration/concurrency-torture.test.ts` (#1416, umbrella #1412 Track A) runs N concurrent
195+
clients through randomized-but-**seeded** interleavings of open/mutate/close/takeover/kill against
196+
the real `SessionStore` + `LeaseRegistry` (plus an in-memory device-claim model). After every run it
197+
asserts: no leaked leases or claims, no cross-session state bleed, every lock released after owner
198+
death, the session store stays consistent, and same-device critical sections never overlap (this
199+
pins the router's same-device open serialization under 100+ interleavings).
200+
201+
A seed alone cannot reproduce Promise/event-loop interleavings, so **all** concurrency is routed
202+
through a deterministic scheduler (`concurrency-torture/deterministic-scheduler.ts`) — an
203+
instrumented dispatcher that is the sole source of ordering (which fiber steps next, and which waiter
204+
wins a contended lock). A seed therefore fully determines execution order. What is real vs modeled,
205+
and why the production `withKeyedLock` is not driven directly, is documented at the top of
206+
`concurrency-torture/harness.ts`.
207+
208+
```bash
209+
pnpm test:concurrency-torture # default sweep (TORTURE_RUNS=128 seeds from 0)
210+
TORTURE_SEED=1234 pnpm test:concurrency-torture # replay ONE seed's exact interleaving (seed-replay flag)
211+
TORTURE_RUNS=5000 TORTURE_SEED_START=0 pnpm test:concurrency-torture # widen the sweep
212+
```
213+
214+
Every failure prints the offending seed and the exact `TORTURE_SEED=<n> pnpm test:concurrency-torture`
215+
replay command. The PR gate runs the fast default sweep through the Node integration lane
216+
(`test:integration:node`); the `Concurrency Torture Nightly` workflow sweeps a much larger seed range
217+
on schedule. Optional knobs: `TORTURE_CLIENTS`, `TORTURE_OPS`.
218+
192219
## Speed rules (experiment-backed, 2026-07-04)
193220

194221
Measured on the full unit suite (340 files, 3,210 tests, 48s wall at ~7x parallelism):

‎package.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -161,6 +161,7 @@
161161
"test:smoke": "node --test test/integration/smoke-*.test.ts",
162162
"test:integration:node": "node --test test/integration/*.test.ts",
163163
"test:integration": "pnpm test:integration:node && pnpm test:integration:provider",
164+
"test:concurrency-torture": "node --test test/integration/concurrency-torture.test.ts",
164165
"test:replay:ios": "node --experimental-strip-types src/bin.ts test test/integration/replays/ios/simulator",
165166
"test:replay:ios-device": "node --experimental-strip-types src/bin.ts test test/integration/replays/ios/device",
166167
"test:replay:android": "node --experimental-strip-types src/bin.ts test test/integration/replays/android",
Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
// Seeded concurrency torture lane for session / lease / lock invariants (#1416,
2+
// umbrella #1412 Track A).
3+
//
4+
// N concurrent clients drive randomized-but-SEEDED interleavings of
5+
// open / mutate / close / takeover / kill against fake providers (the real
6+
// `SessionStore` + `LeaseRegistry`, an in-memory device-claim model), with ALL
7+
// concurrency routed through a deterministic scheduler so a seed fully
8+
// determines execution order. After every run the harness asserts:
9+
// - no leaked leases or claims,
10+
// - no cross-session state bleed,
11+
// - every lock released after owner death,
12+
// - the session store stays consistent,
13+
// - same-device critical sections never overlap (this pins the router's
14+
// same-device open serialization under many interleavings).
15+
//
16+
// Seed replay (documented in docs/agents/testing.md):
17+
// TORTURE_SEED=1234 pnpm test:concurrency-torture
18+
// replays that exact interleaving deterministically. Otherwise the lane sweeps
19+
// TORTURE_RUNS seeds (default 128, ≥100 to satisfy the acceptance bar) starting
20+
// at TORTURE_SEED_START (default 0). Any failure prints the seed and the exact
21+
// replay command.
22+
23+
import test from 'node:test';
24+
import assert from 'node:assert/strict';
25+
26+
import { runTorture, type TortureRunResult } from './concurrency-torture/harness.ts';
27+
28+
function intFromEnv(name: string, fallback: number): number {
29+
const raw = process.env[name]?.trim();
30+
if (!raw) return fallback;
31+
const parsed = Number(raw);
32+
if (!Number.isInteger(parsed) || parsed < 0) {
33+
throw new Error(`${name} must be a non-negative integer, got ${raw}`);
34+
}
35+
return parsed;
36+
}
37+
38+
function optionalIntFromEnv(name: string): number | undefined {
39+
const raw = process.env[name]?.trim();
40+
if (!raw) return undefined;
41+
const parsed = Number(raw);
42+
if (!Number.isInteger(parsed) || parsed <= 0) {
43+
throw new Error(`${name} must be a positive integer, got ${raw}`);
44+
}
45+
return parsed;
46+
}
47+
48+
function replayHint(seed: number): string {
49+
return `Replay this exact interleaving with: TORTURE_SEED=${seed} pnpm test:concurrency-torture`;
50+
}
51+
52+
function assertClean(result: TortureRunResult): void {
53+
if (result.failures.length === 0) return;
54+
const lines = result.failures.map((f) => ` - [${f.invariant}] ${f.detail}`);
55+
assert.fail(
56+
`Concurrency invariants violated on seed ${result.seed} ` +
57+
`(${result.clients} clients, ${result.ops} ops, ${result.scheduleLength} scheduled steps):\n` +
58+
`${lines.join('\n')}\n${replayHint(result.seed)}`,
59+
);
60+
}
61+
62+
const explicitSeed = optionalIntFromEnv('TORTURE_SEED');
63+
const clients = optionalIntFromEnv('TORTURE_CLIENTS');
64+
const opsPerClient = optionalIntFromEnv('TORTURE_OPS');
65+
66+
if (explicitSeed !== undefined) {
67+
test(`concurrency torture — replay seed ${explicitSeed}`, async () => {
68+
const result = await runTorture({ seed: explicitSeed, clients, opsPerClient });
69+
// Determinism self-check: the same seed must reproduce the same schedule.
70+
const replay = await runTorture({ seed: explicitSeed, clients, opsPerClient });
71+
assert.equal(
72+
replay.scheduleLength,
73+
result.scheduleLength,
74+
`seed ${explicitSeed} produced a different schedule length on replay ` +
75+
`(${result.scheduleLength} vs ${replay.scheduleLength}) — non-determinism`,
76+
);
77+
assertClean(result);
78+
});
79+
} else {
80+
const runs = intFromEnv('TORTURE_RUNS', 128);
81+
const seedStart = intFromEnv('TORTURE_SEED_START', 0);
82+
83+
test(`concurrency torture — ${runs} seeded interleavings from ${seedStart}`, async () => {
84+
let exercisedSerialization = false;
85+
for (let i = 0; i < runs; i += 1) {
86+
const seed = seedStart + i;
87+
const result = await runTorture({ seed, clients, opsPerClient });
88+
assertClean(result);
89+
if (Object.values(result.perDeviceMaxConcurrency).some((max) => max >= 1)) {
90+
exercisedSerialization = true;
91+
}
92+
}
93+
// Sanity: the lane must actually enter device critical sections, otherwise a
94+
// regression could hollow it out into a no-op that always "passes".
95+
assert.ok(
96+
exercisedSerialization,
97+
'no device critical section was exercised across the sweep — the lane is not testing serialization',
98+
);
99+
});
100+
}
Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
// In-memory model of the advisory host-global device claim (`src/daemon/device-claims.ts`).
2+
//
3+
// The production claim is a filesystem lock file guarded by a process lock, keyed
4+
// by `canonicalLocalDeviceKey`. That is real I/O and a real OS lock — neither is
5+
// seed-reproducible, and the torture lane is explicitly not wall-clock/I/O stress
6+
// (#1416 out-of-scope). This model preserves the *invariants* the harness asserts:
7+
// - at most one live claim per device key;
8+
// - a claim is released only by its owner (ownerToken match), mirroring
9+
// `clearAdvisoryDeviceClaim`'s token/identity guard;
10+
// - a dead owner's claim must be reclaimable (owner-death reap).
11+
// Mutual exclusion of the acquire/clear critical sections is provided by the
12+
// scheduler mutex in the harness, not here, so this stays a plain synchronous map.
13+
14+
import crypto from 'node:crypto';
15+
16+
export type AdvisoryClaimOwnership = {
17+
deviceKey: string;
18+
ownerToken: string;
19+
session: string;
20+
};
21+
22+
type ClaimRecord = {
23+
deviceKey: string;
24+
ownerToken: string;
25+
session: string;
26+
};
27+
28+
export type ClaimAcquireResult =
29+
| { ownership: AdvisoryClaimOwnership; conflict?: undefined }
30+
| { ownership?: undefined; conflict: { session: string } };
31+
32+
export class InMemoryClaimRegistry {
33+
private readonly claims = new Map<string, ClaimRecord>();
34+
35+
/**
36+
* Mirrors `acquireAdvisoryDeviceClaim`: re-acquiring your own claim is
37+
* idempotent, a foreign live claim is reported as a conflict, and a free key
38+
* is claimed with a fresh owner token.
39+
*/
40+
acquire(deviceKey: string, session: string): ClaimAcquireResult {
41+
const existing = this.claims.get(deviceKey);
42+
if (existing) {
43+
if (existing.session === session) {
44+
return { ownership: { deviceKey, ownerToken: existing.ownerToken, session } };
45+
}
46+
return { conflict: { session: existing.session } };
47+
}
48+
const record: ClaimRecord = { deviceKey, ownerToken: crypto.randomUUID(), session };
49+
this.claims.set(deviceKey, record);
50+
return { ownership: { deviceKey, ownerToken: record.ownerToken, session } };
51+
}
52+
53+
/** Mirrors `clearAdvisoryDeviceClaim`: only the token owner may clear. */
54+
clear(ownership: AdvisoryClaimOwnership | undefined): void {
55+
if (!ownership) return;
56+
const existing = this.claims.get(ownership.deviceKey);
57+
if (!existing || existing.ownerToken !== ownership.ownerToken) return;
58+
this.claims.delete(ownership.deviceKey);
59+
}
60+
61+
/** Owner of `deviceKey`, or undefined when free. */
62+
ownerSession(deviceKey: string): string | undefined {
63+
return this.claims.get(deviceKey)?.session;
64+
}
65+
66+
/** Every live claim, for invariant checks. */
67+
snapshot(): { deviceKey: string; session: string }[] {
68+
return [...this.claims.values()].map((claim) => ({
69+
deviceKey: claim.deviceKey,
70+
session: claim.session,
71+
}));
72+
}
73+
}

0 commit comments

Comments
 (0)