Skip to content

Commit 0890152

Browse files
test(daemon): gate lock plan on shouldLockSessionExecution; sweep replay + forced-device contention; whole-lane envelope
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
1 parent f5abcf6 commit 0890152

4 files changed

Lines changed: 185 additions & 116 deletions

File tree

‎docs/agents/testing.md‎

Lines changed: 20 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -127,13 +127,16 @@ through a deterministic scheduler (`concurrency-torture/deterministic-scheduler.
127127
instrumented dispatcher that is the sole source of ordering (which fiber steps next, and which waiter
128128
wins a contended lock). A seed therefore fully determines execution order.
129129

130-
Each operation's lock plan is **not** hand-written: it comes from the production router primitive
131-
`resolveRequestExecutionLockKeys` (`src/daemon/request-binding.ts`), driven with a fake device
132-
inventory through the production `withDeviceInventoryProvider` seam
130+
Each operation's lock plan is **not** hand-written: it is built exactly as the daemon builds it in
131+
`createRequestExecutionScope` — gate on the production decision `shouldLockSessionExecution(command)`
132+
(`src/daemon/daemon-command-registry.ts`), and only then resolve keys via the production router
133+
primitive `resolveRequestExecutionLockKeys` (`src/daemon/request-binding.ts`), driven with a fake
134+
device inventory through the production `withDeviceInventoryProvider` seam
133135
(`concurrency-torture/bindings.ts`). Only the mutex *grant* is modeled by the scheduler, because
134136
`withKeyedLock`'s native microtask hand-off cannot be reproduced from a seed. Consequently reverting
135-
the router's same-device serialization changes the derived plan and trips the overlap invariant — the
136-
lane is genuinely coupled to production lock resolution, not a duplicate of it. **Real:**
137+
*either* production decision — exempting a command from execution locking, or dropping the `device:`
138+
key — changes the derived plan and trips the overlap invariant, so the lane is genuinely coupled to
139+
production lock resolution, not a duplicate of it. **Real:**
137140
`SessionStore` and `LeaseRegistry`. **Modeled:** the advisory device claim (`InMemoryClaimRegistry`)
138141
and process "kill" — the production claim is a filesystem/OS lock and real process death, both out of
139142
scope for this scheduling lane and covered by their own unit tests. The full real-vs-modeled boundary
@@ -146,17 +149,24 @@ TORTURE_RUNS=5000 TORTURE_SEED_START=0 pnpm test:concurrency-torture # widen t
146149
```
147150

148151
Replay is exact: a given seed reproduces the whole scheduler trace (`traceSignature`), the terminal
149-
invariant outcome, and the contention profile — the replay test asserts equality on all three, not
150-
just schedule length. The sweep also asserts real same-device lock *contention* occurred (two clients
151-
parked on one `device:` lock), and a dedicated forced two-client same-device test drives that
152-
contention deterministically.
152+
invariant outcome, and the contention profile — equality on all three is asserted not just under
153+
`TORTURE_SEED` but for **every seed in the normal sweep** (each seed is re-run and compared), so
154+
non-determinism is caught on the ordinary CI/nightly path. The sweep also asserts real same-device
155+
lock *contention* occurred (two clients parked on one `device:` lock), and a dedicated forced
156+
two-client same-device test pins both clients to one device via `pinnedDevice` so they cannot land on
157+
different devices, driving that contention deterministically.
153158

154159
Every failure prints the offending seed and the exact `TORTURE_SEED=<n> pnpm test:concurrency-torture`
155160
replay command. The PR gate runs the fast default sweep through the Node integration lane
156161
(`test:integration:node`); the `Concurrency Torture Nightly` workflow sweeps a much larger seed range
157162
on schedule and, per #1430, emits a machine-readable envelope (schema version, commit SHA, tool/config
158163
hash, seed range, duration, result) via `TORTURE_ENVELOPE=<path>`, uploaded as the
159-
`concurrency-torture-envelope` artifact. Optional knobs: `TORTURE_CLIENTS`, `TORTURE_OPS`.
164+
`concurrency-torture-envelope` artifact. The envelope is written once, after **all** lane tests
165+
settle, and reports `fail` if any of them (sweep, replay self-check, or forced-contention guardrail)
166+
failed — a later-failing guardrail can never be published as a passing envelope. The #1430 scheduled
167+
health job that watches lane freshness/last-success across workflows is that issue's own deliverable;
168+
this lane is born consumable by it (standard envelope + a `schedule:` trigger the job auto-discovers).
169+
Optional knobs: `TORTURE_CLIENTS`, `TORTURE_OPS`.
160170

161171
## Speed rules (experiment-backed, 2026-07-04)
162172

‎test/integration/concurrency-torture.test.ts‎

Lines changed: 109 additions & 81 deletions
Original file line numberDiff line numberDiff line change
@@ -22,12 +22,60 @@
2222
// seeds (default 128, ≥100 to satisfy the acceptance bar) starting at
2323
// TORTURE_SEED_START (default 0). Any failure prints the seed and replay command.
2424

25-
import test from 'node:test';
25+
import test, { after } from 'node:test';
2626
import assert from 'node:assert/strict';
2727

2828
import { runTorture, type ClientOp, type TortureRunResult } from './concurrency-torture/harness.ts';
29+
import { DEVICE_POOL } from './concurrency-torture/bindings.ts';
2930
import { buildEnvelope, writeEnvelopeIfRequested } from './concurrency-torture/envelope.ts';
3031

32+
// The #1430 envelope must report the outcome of the WHOLE lane, not just the
33+
// sweep test. Every test records its result here and the envelope is written
34+
// once, after all tests settle, so a later failing guardrail can never be
35+
// published as a passing envelope.
36+
type EnvelopeRun = { seedStart: number; runs: number; durationMs: number };
37+
let laneFailed = false;
38+
let sweepRun: EnvelopeRun | undefined;
39+
40+
async function guard(body: () => Promise<void>): Promise<void> {
41+
try {
42+
await body();
43+
} catch (error) {
44+
laneFailed = true;
45+
throw error;
46+
}
47+
}
48+
49+
after(() => {
50+
if (!sweepRun) return; // only the sweep lane (nightly) publishes an envelope
51+
const envelope = buildEnvelope({ ...sweepRun, result: laneFailed ? 'fail' : 'pass' });
52+
const written = writeEnvelopeIfRequested(envelope);
53+
if (written) console.log(`torture envelope → ${written}\n${JSON.stringify(envelope)}`);
54+
});
55+
56+
/**
57+
* Assert a seed replays bit-for-bit: same ordered scheduler trace, same terminal
58+
* invariant outcome, same contention profile. Run for EVERY seed in the sweep so
59+
* determinism is proven on the normal CI/nightly path, not only under TORTURE_SEED.
60+
*/
61+
function assertDeterministicReplay(result: TortureRunResult, replay: TortureRunResult): void {
62+
assert.equal(
63+
replay.traceSignature,
64+
result.traceSignature,
65+
`seed ${result.seed} produced a different scheduler trace on replay — non-determinism`,
66+
);
67+
assert.deepEqual(
68+
replay.failures,
69+
result.failures,
70+
`seed ${result.seed} produced a different invariant outcome on replay — non-determinism`,
71+
);
72+
assert.equal(
73+
replay.deviceContention,
74+
result.deviceContention,
75+
`seed ${result.seed} produced different device contention on replay — non-determinism`,
76+
);
77+
}
78+
3179
function intFromEnv(name: string, fallback: number): number {
3280
const raw = process.env[name]?.trim();
3381
if (!raw) return fallback;
@@ -67,93 +115,73 @@ const clients = optionalIntFromEnv('TORTURE_CLIENTS');
67115
const opsPerClient = optionalIntFromEnv('TORTURE_OPS');
68116

69117
if (explicitSeed !== undefined) {
70-
test(`concurrency torture — replay seed ${explicitSeed}`, async () => {
71-
const result = await runTorture({ seed: explicitSeed, clients, opsPerClient });
72-
// Determinism self-check: the same seed must reproduce the EXACT schedule,
73-
// the same terminal invariant outcome, and the same contention profile.
74-
const replay = await runTorture({ seed: explicitSeed, clients, opsPerClient });
75-
assert.equal(
76-
replay.traceSignature,
77-
result.traceSignature,
78-
`seed ${explicitSeed} produced a different scheduler trace on replay — non-determinism`,
79-
);
80-
assert.deepEqual(
81-
replay.failures,
82-
result.failures,
83-
`seed ${explicitSeed} produced a different invariant outcome on replay — non-determinism`,
84-
);
85-
assert.equal(
86-
replay.deviceContention,
87-
result.deviceContention,
88-
`seed ${explicitSeed} produced different device contention on replay — non-determinism`,
89-
);
90-
assertClean(result);
91-
});
118+
test(`concurrency torture — replay seed ${explicitSeed}`, () =>
119+
guard(async () => {
120+
const result = await runTorture({ seed: explicitSeed, clients, opsPerClient });
121+
const replay = await runTorture({ seed: explicitSeed, clients, opsPerClient });
122+
assertDeterministicReplay(result, replay);
123+
assertClean(result);
124+
}));
92125
} else {
93126
const runs = intFromEnv('TORTURE_RUNS', 128);
94127
const seedStart = intFromEnv('TORTURE_SEED_START', 0);
95128

96-
test(`concurrency torture — ${runs} seeded interleavings from ${seedStart}`, async () => {
97-
const started = Date.now();
98-
let totalContention = 0;
99-
let failed = false;
100-
try {
101-
for (let i = 0; i < runs; i += 1) {
102-
const seed = seedStart + i;
103-
const result = await runTorture({ seed, clients, opsPerClient });
104-
totalContention += result.deviceContention;
105-
if (result.failures.length > 0) failed = true;
106-
assertClean(result);
129+
test(`concurrency torture — ${runs} seeded interleavings from ${seedStart}`, () =>
130+
guard(async () => {
131+
const started = Date.now();
132+
let totalContention = 0;
133+
try {
134+
for (let i = 0; i < runs; i += 1) {
135+
const seed = seedStart + i;
136+
const result = await runTorture({ seed, clients, opsPerClient });
137+
totalContention += result.deviceContention;
138+
// Prove exact replay on the NORMAL sweep path (not just TORTURE_SEED):
139+
// re-run the seed and assert the whole trace + outcome + contention match.
140+
assertDeterministicReplay(result, await runTorture({ seed, clients, opsPerClient }));
141+
assertClean(result);
142+
}
143+
// The sweep must actually PRODUCE same-device contention — two clients
144+
// parked on one device lock — or the lane is not exercising serialization
145+
// and a broken lock could pass unnoticed.
146+
assert.ok(
147+
totalContention > 0,
148+
'no same-device lock contention occurred across the sweep — the lane is not testing serialization',
149+
);
150+
} finally {
151+
sweepRun = { seedStart, runs, durationMs: Date.now() - started };
107152
}
108-
// The sweep must actually PRODUCE same-device contention — two clients
109-
// parked on one device lock — or the lane is not exercising serialization
110-
// and a broken lock could pass unnoticed.
111-
assert.ok(
112-
totalContention > 0,
113-
'no same-device lock contention occurred across the sweep — the lane is not testing serialization',
114-
);
115-
} catch (error) {
116-
failed = true;
117-
throw error;
118-
} finally {
119-
const envelope = buildEnvelope({
120-
seedStart,
121-
runs,
122-
durationMs: Date.now() - started,
123-
result: failed ? 'fail' : 'pass',
124-
});
125-
const written = writeEnvelopeIfRequested(envelope);
126-
if (written) console.log(`torture envelope → ${written}\n${JSON.stringify(envelope)}`);
127-
}
128-
});
153+
}));
129154

130155
// Forced same-device contention: two clients repeatedly open THE SAME pinned
131-
// device. This deterministically drives both onto one `device:` lock so the
132-
// overlap invariant is exercised, not just present. If the router stopped
133-
// serializing same-device opens, `deviceContention` here would collapse and
134-
// the overlap invariant would fire.
135-
test('concurrency torture — forced two-client same-device contention', async () => {
136-
const program: ClientOp[][] = [
137-
['open', 'mutate', 'close', 'open', 'mutate'],
138-
['open', 'mutate', 'close', 'open', 'mutate'],
139-
];
140-
let contendedSeeds = 0;
141-
for (let seed = 0; seed < 40; seed += 1) {
142-
const result = await runTorture({ seed, program });
143-
assertClean(result);
144-
const perDeviceMax = Object.values(result.perDeviceMaxConcurrency);
156+
// device (via `pinnedDevice`, so `doOpen` cannot randomly pick a different one).
157+
// This deterministically drives both onto one `device:` lock so the overlap
158+
// invariant is exercised, not just present. If the router stopped serializing
159+
// same-device opens, `deviceContention` here would collapse and the overlap
160+
// invariant would fire.
161+
test('concurrency torture — forced two-client same-device contention', () =>
162+
guard(async () => {
163+
const pinnedDevice = DEVICE_POOL[0];
164+
const program: ClientOp[][] = [
165+
['open', 'mutate', 'close', 'open', 'mutate'],
166+
['open', 'mutate', 'close', 'open', 'mutate'],
167+
];
168+
let contendedSeeds = 0;
169+
for (let seed = 0; seed < 40; seed += 1) {
170+
const result = await runTorture({ seed, program, pinnedDevice });
171+
assertClean(result);
172+
const perDeviceMax = Object.values(result.perDeviceMaxConcurrency);
173+
assert.ok(
174+
perDeviceMax.every((max) => max <= 1),
175+
`seed ${seed}: same-device critical sections overlapped — ${JSON.stringify(
176+
result.perDeviceMaxConcurrency,
177+
)}`,
178+
);
179+
if (result.deviceContention > 0) contendedSeeds += 1;
180+
}
145181
assert.ok(
146-
perDeviceMax.every((max) => max <= 1),
147-
`seed ${seed}: same-device critical sections overlapped — ${JSON.stringify(
148-
result.perDeviceMaxConcurrency,
149-
)}`,
182+
contendedSeeds > 0,
183+
'two clients never actually contended for the pinned device across 40 seeds — ' +
184+
'the forced-contention scenario is not exercising the device lock',
150185
);
151-
if (result.deviceContention > 0) contendedSeeds += 1;
152-
}
153-
assert.ok(
154-
contendedSeeds > 0,
155-
'two clients never actually contended for the pinned device across 40 seeds — ' +
156-
'the forced-contention scenario is not exercising the device lock',
157-
);
158-
});
186+
}));
159187
}

‎test/integration/concurrency-torture/bindings.ts‎

Lines changed: 51 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -7,19 +7,34 @@
77
// makes the same-device serialization invariant revert-sensitive: if production
88
// stops returning a `device:` key (e.g. the router's same-device open
99
// serialization is reverted), the derived plan changes and the lane's overlap
10-
// invariant fires. Only the mutex GRANT is modeled (by the deterministic
11-
// scheduler) because `withKeyedLock`'s native microtask hand-off cannot be
12-
// reproduced from a seed.
10+
// invariant fires.
11+
//
12+
// The plan is built the SAME way the daemon builds it in
13+
// `createRequestExecutionScope`: gate on the production decision
14+
// `shouldLockSessionExecution(command)`, and only then resolve keys via
15+
// `resolveRequestExecutionLockKeys`. So the lane is revert-sensitive to BOTH
16+
// production decision points — exempting a command from execution locking, or
17+
// dropping the device key — even though the mutex GRANT itself stays modeled
18+
// (by the deterministic scheduler) because `withKeyedLock`'s native microtask
19+
// hand-off cannot be reproduced from a seed.
1320

1421
import type { DeviceInfo } from '../../../src/kernel/device.ts';
1522
import type { CommandFlags } from '../../../src/core/dispatch-context.ts';
1623
import type { DaemonRequest } from '../../../src/daemon/types.ts';
1724
import type { SessionStore } from '../../../src/daemon/session-store.ts';
1825
import { resolveRequestExecutionLockKeys } from '../../../src/daemon/request-binding.ts';
26+
import { shouldLockSessionExecution } from '../../../src/daemon/daemon-command-registry.ts';
27+
import { PUBLIC_COMMANDS } from '../../../src/command-catalog.ts';
1928
import { withDeviceInventoryProvider } from '../../../src/core/dispatch-resolve.ts';
2029

2130
import type { LockKey } from './deterministic-scheduler.ts';
2231

32+
// Production command each modeled op routes as, so the lane consumes the real
33+
// `shouldLockSessionExecution` gate rather than assuming every op locks.
34+
const OPEN_COMMAND = PUBLIC_COMMANDS.open;
35+
export const MUTATE_COMMAND = PUBLIC_COMMANDS.click;
36+
export const CLOSE_COMMAND = PUBLIC_COMMANDS.close;
37+
2338
export const DEVICE_POOL: readonly DeviceInfo[] = [
2439
{
2540
platform: 'apple',
@@ -82,17 +97,17 @@ function selectorFlagsForDevice(device: DeviceInfo): CommandFlags {
8297

8398
function openRequest(device: DeviceInfo, sessionName: string): DaemonRequest {
8499
return {
85-
command: 'open',
100+
command: OPEN_COMMAND,
86101
positionals: [],
87102
token: 'torture',
88103
session: sessionName,
89104
flags: selectorFlagsForDevice(device),
90105
} as DaemonRequest;
91106
}
92107

93-
function existingSessionRequest(sessionName: string): DaemonRequest {
108+
function existingSessionRequest(command: string, sessionName: string): DaemonRequest {
94109
return {
95-
command: 'is',
110+
command,
96111
positionals: [],
97112
token: 'torture',
98113
session: sessionName,
@@ -104,6 +119,23 @@ async function withPool<T>(task: () => Promise<T>): Promise<T> {
104119
return await withDeviceInventoryProvider(async () => [...DEVICE_POOL], task);
105120
}
106121

122+
/**
123+
* Resolve a request's execution lock plan EXACTLY as `createRequestExecutionScope`
124+
* does: skip locking entirely when the command is execution-lock-exempt, else
125+
* resolve the keys through the production router. Reverting either production
126+
* decision changes what this returns.
127+
*/
128+
async function resolveExecutionLockPlan(
129+
req: DaemonRequest,
130+
sessionName: string,
131+
sessionStore: SessionStore,
132+
): Promise<LockKey[]> {
133+
if (!shouldLockSessionExecution(req.command)) return [];
134+
return (await withPool(() =>
135+
resolveRequestExecutionLockKeys({ req, sessionName, sessionStore }),
136+
)) as LockKey[];
137+
}
138+
107139
/**
108140
* Lock plan for a fresh open of `device` under `sessionName`, computed by the
109141
* production router (`[session:<name>, device:<id>]` when the device resolves).
@@ -113,28 +145,25 @@ export async function resolveOpenLockPlan(
113145
sessionName: string,
114146
device: DeviceInfo,
115147
): Promise<LockKey[]> {
116-
return (await withPool(() =>
117-
resolveRequestExecutionLockKeys({
118-
req: openRequest(device, sessionName),
119-
sessionName,
120-
sessionStore,
121-
}),
122-
)) as LockKey[];
148+
return await resolveExecutionLockPlan(
149+
openRequest(device, sessionName),
150+
sessionName,
151+
sessionStore,
152+
);
123153
}
124154

125155
/**
126-
* Lock plan for an operation on an already-open session, computed by the
127-
* production router (`[device:<id>]`, read from the stored session's device).
156+
* Lock plan for `command` on an already-open session, computed by the production
157+
* router (`[device:<id>]`, read from the stored session's device).
128158
*/
129159
export async function resolveExistingLockPlan(
130160
sessionStore: SessionStore,
131161
sessionName: string,
162+
command: string,
132163
): Promise<LockKey[]> {
133-
return (await withPool(() =>
134-
resolveRequestExecutionLockKeys({
135-
req: existingSessionRequest(sessionName),
136-
sessionName,
137-
sessionStore,
138-
}),
139-
)) as LockKey[];
164+
return await resolveExecutionLockPlan(
165+
existingSessionRequest(command, sessionName),
166+
sessionName,
167+
sessionStore,
168+
);
140169
}

0 commit comments

Comments
 (0)