Skip to content

Commit f9e423f

Browse files
test(daemon): drive torture lane through real lock plan + review fixes
- derive each op's lock plan from production resolveRequestExecutionLockKeys via a fake device-inventory provider, so reverting the router's same-device serialization trips the overlap invariant (verified) - assert exact replay: full scheduler trace, terminal outcome, contention - assert real same-device lock contention in the sweep + a forced 2-client case - split harness into bindings/invariants/envelope modules (all <500 LOC) - emit #1430 scheduled-lane envelope (schema/SHA/hash/seed range/duration/result) and upload it from the nightly workflow Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
1 parent 72f48d6 commit f9e423f

8 files changed

Lines changed: 741 additions & 383 deletions

File tree

‎.github/workflows/concurrency-torture-nightly.yml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,10 @@ jobs:
4141
env:
4242
TORTURE_RUNS: ${{ github.event.inputs.runs || '5000' }}
4343
TORTURE_SEED_START: ${{ github.event.inputs.seed-start || '0' }}
44+
# Standard scheduled-lane artifact envelope (#1430): schemaVersion, commit
45+
# SHA, tool/config hashes, seed range, duration, and result. Emitted by the
46+
# test and uploaded below so the observatory can detect a lane going dark.
47+
TORTURE_ENVELOPE: ${{ github.workspace }}/torture-results/envelope.json
4448
steps:
4549
- name: Checkout
4650
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
@@ -50,3 +54,11 @@ jobs:
5054

5155
- name: Run concurrency torture sweep
5256
run: pnpm test:concurrency-torture
57+
58+
- name: Upload torture lane envelope
59+
if: always()
60+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
61+
with:
62+
name: concurrency-torture-envelope
63+
path: torture-results/
64+
if-no-files-found: warn

‎docs/agents/testing.md‎

Lines changed: 22 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -125,20 +125,38 @@ pins the router's same-device open serialization under 100+ interleavings).
125125
A seed alone cannot reproduce Promise/event-loop interleavings, so **all** concurrency is routed
126126
through a deterministic scheduler (`concurrency-torture/deterministic-scheduler.ts`) — an
127127
instrumented dispatcher that is the sole source of ordering (which fiber steps next, and which waiter
128-
wins a contended lock). A seed therefore fully determines execution order. What is real vs modeled,
129-
and why the production `withKeyedLock` is not driven directly, is documented at the top of
130-
`concurrency-torture/harness.ts`.
128+
wins a contended lock). A seed therefore fully determines execution order.
129+
130+
Each operation's lock plan is **not** hand-written: it comes from the production router primitive
131+
`resolveRequestExecutionLockKeys` (`src/daemon/request-binding.ts`), driven with a fake device
132+
inventory through the production `withDeviceInventoryProvider` seam
133+
(`concurrency-torture/bindings.ts`). Only the mutex *grant* is modeled by the scheduler, because
134+
`withKeyedLock`'s native microtask hand-off cannot be reproduced from a seed. Consequently reverting
135+
the router's same-device serialization changes the derived plan and trips the overlap invariant — the
136+
lane is genuinely coupled to production lock resolution, not a duplicate of it. **Real:**
137+
`SessionStore` and `LeaseRegistry`. **Modeled:** the advisory device claim (`InMemoryClaimRegistry`)
138+
and process "kill" — the production claim is a filesystem/OS lock and real process death, both out of
139+
scope for this scheduling lane and covered by their own unit tests. The full real-vs-modeled boundary
140+
is documented at the top of `concurrency-torture/harness.ts`.
131141

132142
```bash
133143
pnpm test:concurrency-torture # default sweep (TORTURE_RUNS=128 seeds from 0)
134144
TORTURE_SEED=1234 pnpm test:concurrency-torture # replay ONE seed's exact interleaving (seed-replay flag)
135145
TORTURE_RUNS=5000 TORTURE_SEED_START=0 pnpm test:concurrency-torture # widen the sweep
136146
```
137147

148+
Replay is exact: a given seed reproduces the whole scheduler trace (`traceSignature`), the terminal
149+
invariant outcome, and the contention profile — the replay test asserts equality on all three, not
150+
just schedule length. The sweep also asserts real same-device lock *contention* occurred (two clients
151+
parked on one `device:` lock), and a dedicated forced two-client same-device test drives that
152+
contention deterministically.
153+
138154
Every failure prints the offending seed and the exact `TORTURE_SEED=<n> pnpm test:concurrency-torture`
139155
replay command. The PR gate runs the fast default sweep through the Node integration lane
140156
(`test:integration:node`); the `Concurrency Torture Nightly` workflow sweeps a much larger seed range
141-
on schedule. Optional knobs: `TORTURE_CLIENTS`, `TORTURE_OPS`.
157+
on schedule and, per #1430, emits a machine-readable envelope (schema version, commit SHA, tool/config
158+
hash, seed range, duration, result) via `TORTURE_ENVELOPE=<path>`, uploaded as the
159+
`concurrency-torture-envelope` artifact. Optional knobs: `TORTURE_CLIENTS`, `TORTURE_OPS`.
142160

143161
## Speed rules (experiment-backed, 2026-07-04)
144162

‎test/integration/concurrency-torture.test.ts‎

Lines changed: 83 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -5,25 +5,28 @@
55
// open / mutate / close / takeover / kill against fake providers (the real
66
// `SessionStore` + `LeaseRegistry`, an in-memory device-claim model), with ALL
77
// concurrency routed through a deterministic scheduler so a seed fully
8-
// determines execution order. After every run the harness asserts:
8+
// determines execution order. Each operation's lock plan is derived from the
9+
// production router primitive `resolveRequestExecutionLockKeys`, so reverting
10+
// the router's same-device serialization trips the overlap invariant. After
11+
// every run the harness asserts:
912
// - no leaked leases or claims,
1013
// - no cross-session state bleed,
1114
// - every lock released after owner death,
1215
// - the session store stays consistent,
13-
// - same-device critical sections never overlap (this pins the router's
14-
// same-device open serialization under many interleavings).
16+
// - same-device critical sections never overlap.
1517
//
1618
// Seed replay (documented in docs/agents/testing.md):
1719
// TORTURE_SEED=1234 pnpm test:concurrency-torture
18-
// replays that exact interleaving deterministically. Otherwise the lane sweeps
19-
// TORTURE_RUNS seeds (default 128, ≥100 to satisfy the acceptance bar) starting
20-
// at TORTURE_SEED_START (default 0). Any failure prints the seed and the exact
21-
// replay command.
20+
// replays that exact interleaving deterministically — the whole scheduler trace,
21+
// not just its length, is asserted equal. Otherwise the lane sweeps TORTURE_RUNS
22+
// seeds (default 128, ≥100 to satisfy the acceptance bar) starting at
23+
// TORTURE_SEED_START (default 0). Any failure prints the seed and replay command.
2224

2325
import test from 'node:test';
2426
import assert from 'node:assert/strict';
2527

26-
import { runTorture, type TortureRunResult } from './concurrency-torture/harness.ts';
28+
import { runTorture, type ClientOp, type TortureRunResult } from './concurrency-torture/harness.ts';
29+
import { buildEnvelope, writeEnvelopeIfRequested } from './concurrency-torture/envelope.ts';
2730

2831
function intFromEnv(name: string, fallback: number): number {
2932
const raw = process.env[name]?.trim();
@@ -66,13 +69,23 @@ const opsPerClient = optionalIntFromEnv('TORTURE_OPS');
6669
if (explicitSeed !== undefined) {
6770
test(`concurrency torture — replay seed ${explicitSeed}`, async () => {
6871
const result = await runTorture({ seed: explicitSeed, clients, opsPerClient });
69-
// Determinism self-check: the same seed must reproduce the same schedule.
72+
// Determinism self-check: the same seed must reproduce the EXACT schedule,
73+
// the same terminal invariant outcome, and the same contention profile.
7074
const replay = await runTorture({ seed: explicitSeed, clients, opsPerClient });
7175
assert.equal(
72-
replay.scheduleLength,
73-
result.scheduleLength,
74-
`seed ${explicitSeed} produced a different schedule length on replay ` +
75-
`(${result.scheduleLength} vs ${replay.scheduleLength}) — non-determinism`,
76+
replay.traceSignature,
77+
result.traceSignature,
78+
`seed ${explicitSeed} produced a different scheduler trace on replay — non-determinism`,
79+
);
80+
assert.deepEqual(
81+
replay.failures,
82+
result.failures,
83+
`seed ${explicitSeed} produced a different invariant outcome on replay — non-determinism`,
84+
);
85+
assert.equal(
86+
replay.deviceContention,
87+
result.deviceContention,
88+
`seed ${explicitSeed} produced different device contention on replay — non-determinism`,
7689
);
7790
assertClean(result);
7891
});
@@ -81,20 +94,66 @@ if (explicitSeed !== undefined) {
8194
const seedStart = intFromEnv('TORTURE_SEED_START', 0);
8295

8396
test(`concurrency torture — ${runs} seeded interleavings from ${seedStart}`, async () => {
84-
let exercisedSerialization = false;
85-
for (let i = 0; i < runs; i += 1) {
86-
const seed = seedStart + i;
87-
const result = await runTorture({ seed, clients, opsPerClient });
88-
assertClean(result);
89-
if (Object.values(result.perDeviceMaxConcurrency).some((max) => max >= 1)) {
90-
exercisedSerialization = true;
97+
const started = Date.now();
98+
let totalContention = 0;
99+
let failed = false;
100+
try {
101+
for (let i = 0; i < runs; i += 1) {
102+
const seed = seedStart + i;
103+
const result = await runTorture({ seed, clients, opsPerClient });
104+
totalContention += result.deviceContention;
105+
if (result.failures.length > 0) failed = true;
106+
assertClean(result);
91107
}
108+
// The sweep must actually PRODUCE same-device contention — two clients
109+
// parked on one device lock — or the lane is not exercising serialization
110+
// and a broken lock could pass unnoticed.
111+
assert.ok(
112+
totalContention > 0,
113+
'no same-device lock contention occurred across the sweep — the lane is not testing serialization',
114+
);
115+
} catch (error) {
116+
failed = true;
117+
throw error;
118+
} finally {
119+
const envelope = buildEnvelope({
120+
seedStart,
121+
runs,
122+
durationMs: Date.now() - started,
123+
result: failed ? 'fail' : 'pass',
124+
});
125+
const written = writeEnvelopeIfRequested(envelope);
126+
if (written) console.log(`torture envelope → ${written}\n${JSON.stringify(envelope)}`);
127+
}
128+
});
129+
130+
// Forced same-device contention: two clients repeatedly open THE SAME pinned
131+
// device. This deterministically drives both onto one `device:` lock so the
132+
// overlap invariant is exercised, not just present. If the router stopped
133+
// serializing same-device opens, `deviceContention` here would collapse and
134+
// the overlap invariant would fire.
135+
test('concurrency torture — forced two-client same-device contention', async () => {
136+
const program: ClientOp[][] = [
137+
['open', 'mutate', 'close', 'open', 'mutate'],
138+
['open', 'mutate', 'close', 'open', 'mutate'],
139+
];
140+
let contendedSeeds = 0;
141+
for (let seed = 0; seed < 40; seed += 1) {
142+
const result = await runTorture({ seed, program });
143+
assertClean(result);
144+
const perDeviceMax = Object.values(result.perDeviceMaxConcurrency);
145+
assert.ok(
146+
perDeviceMax.every((max) => max <= 1),
147+
`seed ${seed}: same-device critical sections overlapped — ${JSON.stringify(
148+
result.perDeviceMaxConcurrency,
149+
)}`,
150+
);
151+
if (result.deviceContention > 0) contendedSeeds += 1;
92152
}
93-
// Sanity: the lane must actually enter device critical sections, otherwise a
94-
// regression could hollow it out into a no-op that always "passes".
95153
assert.ok(
96-
exercisedSerialization,
97-
'no device critical section was exercised across the sweep — the lane is not testing serialization',
154+
contendedSeeds > 0,
155+
'two clients never actually contended for the pinned device across 40 seeds — ' +
156+
'the forced-contention scenario is not exercising the device lock',
98157
);
99158
});
100159
}
Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
// Production lock-plan + device bindings for the concurrency torture lane (#1416).
2+
//
3+
// The lane does NOT hand-write which locks an operation takes. It derives the
4+
// plan from the REAL router primitive `resolveRequestExecutionLockKeys`
5+
// (src/daemon/request-binding.ts), driven with a fake in-memory device
6+
// inventory via the production `withDeviceInventoryProvider` seam. This is what
7+
// makes the same-device serialization invariant revert-sensitive: if production
8+
// stops returning a `device:` key (e.g. the router's same-device open
9+
// serialization is reverted), the derived plan changes and the lane's overlap
10+
// invariant fires. Only the mutex GRANT is modeled (by the deterministic
11+
// scheduler) because `withKeyedLock`'s native microtask hand-off cannot be
12+
// reproduced from a seed.
13+
14+
import type { DeviceInfo } from '../../../src/kernel/device.ts';
15+
import type { CommandFlags } from '../../../src/core/dispatch-context.ts';
16+
import type { DaemonRequest } from '../../../src/daemon/types.ts';
17+
import type { SessionStore } from '../../../src/daemon/session-store.ts';
18+
import { resolveRequestExecutionLockKeys } from '../../../src/daemon/request-binding.ts';
19+
import { withDeviceInventoryProvider } from '../../../src/core/dispatch-resolve.ts';
20+
21+
import type { LockKey } from './deterministic-scheduler.ts';
22+
23+
export const DEVICE_POOL: readonly DeviceInfo[] = [
24+
{
25+
platform: 'apple',
26+
id: 'sim-a',
27+
name: 'iPhone A',
28+
kind: 'simulator',
29+
appleOs: 'ios',
30+
booted: true,
31+
},
32+
{
33+
platform: 'apple',
34+
id: 'sim-b',
35+
name: 'iPhone B',
36+
kind: 'simulator',
37+
appleOs: 'ios',
38+
booted: true,
39+
},
40+
{ platform: 'android', id: 'emu-c', name: 'Pixel C', kind: 'emulator', booted: true },
41+
];
42+
43+
/** The lease scope for one opened session, kept so the harness can release it. */
44+
export type LeaseScope = {
45+
leaseId: string;
46+
tenantId: string;
47+
runId: string;
48+
leaseBackend: 'ios-simulator';
49+
deviceKey: string;
50+
clientId: string;
51+
};
52+
53+
/**
54+
* Shadow record of one session instance the harness opened. `instanceId` is
55+
* unique across the whole run even when a session NAME is reused, so a stale
56+
* reference can never be mistaken for a live one.
57+
*/
58+
export type SessionInstance = {
59+
instanceId: number;
60+
name: string;
61+
deviceId: string;
62+
deviceKey: string;
63+
lease: LeaseScope;
64+
claim: { deviceKey: string; ownerToken: string; session: string };
65+
ownerClient: number;
66+
dead: boolean;
67+
reaped: boolean;
68+
mutations: number;
69+
};
70+
71+
/** The advisory-claim device key the daemon derives for a resolved device. */
72+
export function deviceClaimKey(device: DeviceInfo): string {
73+
return `local:${device.platform}:${device.appleOs ?? 'none'}:${device.id}`;
74+
}
75+
76+
/** An explicit device selector that resolves to exactly `device` in the pool. */
77+
function selectorFlagsForDevice(device: DeviceInfo): CommandFlags {
78+
return device.platform === 'android'
79+
? ({ platform: 'android', serial: device.id } as CommandFlags)
80+
: ({ platform: 'ios', udid: device.id } as CommandFlags);
81+
}
82+
83+
function openRequest(device: DeviceInfo, sessionName: string): DaemonRequest {
84+
return {
85+
command: 'open',
86+
positionals: [],
87+
token: 'torture',
88+
session: sessionName,
89+
flags: selectorFlagsForDevice(device),
90+
} as DaemonRequest;
91+
}
92+
93+
function existingSessionRequest(sessionName: string): DaemonRequest {
94+
return {
95+
command: 'is',
96+
positionals: [],
97+
token: 'torture',
98+
session: sessionName,
99+
flags: {} as CommandFlags,
100+
} as DaemonRequest;
101+
}
102+
103+
async function withPool<T>(task: () => Promise<T>): Promise<T> {
104+
return await withDeviceInventoryProvider(async () => [...DEVICE_POOL], task);
105+
}
106+
107+
/**
108+
* Lock plan for a fresh open of `device` under `sessionName`, computed by the
109+
* production router (`[session:<name>, device:<id>]` when the device resolves).
110+
*/
111+
export async function resolveOpenLockPlan(
112+
sessionStore: SessionStore,
113+
sessionName: string,
114+
device: DeviceInfo,
115+
): Promise<LockKey[]> {
116+
return (await withPool(() =>
117+
resolveRequestExecutionLockKeys({
118+
req: openRequest(device, sessionName),
119+
sessionName,
120+
sessionStore,
121+
}),
122+
)) as LockKey[];
123+
}
124+
125+
/**
126+
* Lock plan for an operation on an already-open session, computed by the
127+
* production router (`[device:<id>]`, read from the stored session's device).
128+
*/
129+
export async function resolveExistingLockPlan(
130+
sessionStore: SessionStore,
131+
sessionName: string,
132+
): Promise<LockKey[]> {
133+
return (await withPool(() =>
134+
resolveRequestExecutionLockKeys({
135+
req: existingSessionRequest(sessionName),
136+
sessionName,
137+
sessionStore,
138+
}),
139+
)) as LockKey[];
140+
}

‎test/integration/concurrency-torture/deterministic-scheduler.ts‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,17 @@ export type SchedulerChoice =
6464
| { kind: 'step'; fiber: number; label?: string }
6565
| { kind: 'grant'; fiber: number; key: LockKey };
6666

67+
/** Serialize a decision trace to a canonical string for exact replay comparison. */
68+
export function serializeTrace(trace: readonly SchedulerChoice[]): string {
69+
return trace
70+
.map((choice) =>
71+
choice.kind === 'step'
72+
? `s:${choice.fiber}:${choice.label ?? ''}`
73+
: `g:${choice.fiber}:${choice.key}`,
74+
)
75+
.join('|');
76+
}
77+
6778
export class SchedulerDeadlockError extends Error {
6879
readonly liveFibers: readonly number[];
6980
readonly heldLocks: readonly { key: LockKey; holder: number; waiters: number[] }[];
@@ -91,6 +102,9 @@ export class DeterministicScheduler {
91102
private turn: Deferred<void> | null = null;
92103
private firstError: unknown;
93104
private readonly choiceLog: SchedulerChoice[] = [];
105+
// How many times acquiring a key had to PARK because it was already held —
106+
// i.e. genuine contention actually occurred (not just that a lock was taken).
107+
private readonly contention = new Map<LockKey, number>();
94108

95109
constructor(pickIndex: (bound: number) => number) {
96110
this.pickIndex = pickIndex;
@@ -101,6 +115,11 @@ export class DeterministicScheduler {
101115
return this.choiceLog;
102116
}
103117

118+
/** Per-key count of acquisitions that parked on a held lock (observed contention). */
119+
get contentionByKey(): ReadonlyMap<LockKey, number> {
120+
return this.contention;
121+
}
122+
104123
/**
105124
* Run every fiber to completion under seeded interleaving. Resolves once all
106125
* fibers finish; rejects with the first fiber error, or a
@@ -182,6 +201,7 @@ export class DeterministicScheduler {
182201
const gate = deferred<void>();
183202
this.wake.set(id, gate.resolve);
184203
mutex.waiters.add(id);
204+
this.contention.set(key, (this.contention.get(key) ?? 0) + 1);
185205
this.resolveTurn();
186206
return gate.promise;
187207
};

0 commit comments

Comments
 (0)