Skip to content

Commit dbc4f2f

Browse files
thymikeeclaude
andauthored
chore(test): start the subprocess-stub kill-criterion experiment (#1823) (#2007)
Deletes the serialized `subprocess-stub` Vitest project and drops SUBPROCESS_STUB_TESTS from unit-core's exclude, so its two real spawners (client-metro.test.ts, harness.test.ts — corpus-replay.test.ts already left for fuzz-worker in #1994) run un-serialized in the default forks pool per #1823's own kill criterion. Revert if a timeout-shaped failure shows up before 20 consecutive CI runs pass clean. The files stay excluded from the mutation lane (SERIALIZED_TESTS): that exclusion is about mutant-rerun cost, independent of Vitest project structure. Updated the comments/docs/scripts that described the old project by name so none of them assert a project that no longer exists. Claude-Session: https://claude.ai/code/session_015YPgKE1xmjdqh7T1q987DA Co-authored-by: Claude <[email protected]>
1 parent 893ce4b commit dbc4f2f

8 files changed

Lines changed: 38 additions & 36 deletions

File tree

‎docs/agents/pull-requests.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
code-quality/dead-code risk is relevant, CI guards are green, no conflict markers or unmerged
77
paths remain.
88
- A local unit-only run is not CI-green. Use `pnpm test:unit` for the repo unit bundle, or
9-
`vitest run --project unit-core --project subprocess-stub --project fuzz-worker` directly.
9+
`vitest run --project unit-core --project fuzz-worker` directly.
1010
The **Integration Tests** and **Coverage** jobs run the `provider-integration` project —
1111
verify those green on the actual PR head.
1212
- Device-facing behavior is not merge-ready without real simulator/emulator/device evidence for the

‎docs/agents/testing.md‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -159,12 +159,13 @@ Lock plans come from the production request-lock decisions — never hand-author
159159
modeled boundary is documented in the harness module, and every failure prints its exact replay
160160
command.
161161

162-
## The `subprocess-stub` project
162+
## Real-subprocess-spawn tests
163163

164-
`SUBPROCESS_STUB_TESTS` enumerates the few files that spawn real subprocesses per case. They run in
165-
a serialized Vitest project so host contention cannot turn internal budgets into generic timeouts.
166-
Membership requires naming the real spawned process; environment isolation alone does not qualify.
167-
There is no unit-test retry layer — fix or remove flakes.
164+
`SUBPROCESS_STUB_TESTS` enumerates the few files that spawn a real subprocess per case. They ran
165+
serialized in their own Vitest project until #1823's kill criterion: now un-serialized in
166+
`unit-core`'s default forks pool, reverted if a timeout-shaped failure appears within 20 consecutive
167+
CI runs. Still excluded from the mutation lane either way. There is no unit-test retry layer — fix
168+
or remove flakes.
168169

169170
## Speed rules
170171

‎package.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -171,7 +171,7 @@
171171
"test-app:maestro:ios": "pnpm test-app:maestro --platform ios",
172172
"test-app:maestro:android": "pnpm test-app:maestro --platform android",
173173
"test": "pnpm test:unit",
174-
"test:unit": "vitest run --project unit-core --project subprocess-stub --project fuzz-worker",
174+
"test:unit": "vitest run --project unit-core --project fuzz-worker",
175175
"test:maestro-compat": "vitest run --project unit-core packages/maestro src/daemon/adapters/maestro src/compat/__tests__/replay-input.test.ts",
176176
"test:coverage": "vitest run --coverage --project=!fuzz-worker && pnpm test:fuzz-worker",
177177
"test:coverage:ci": "vitest run --coverage --project=!fuzz-worker && pnpm test:fuzz-worker",

‎scripts/fuzz/harness.test.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -70,7 +70,7 @@ describe('fuzz invariant classifier', () => {
7070

7171
describe('fuzz harness self-check', () => {
7272
// One run asserts both the report and its envelope: a second full self-check would cost five
73-
// more worker startups in the serialized subprocess-stub project (#1823) for no new signal.
73+
// more real worker startups (#1823) for no new signal.
7474
it('catches every seeded violation kind and writes the self-check envelope', () => {
7575
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'fuzz-selfcheck-'));
7676
const { status, stdout } = runHarness([

‎scripts/mutation/test-scope.ts‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,9 +9,10 @@
99
// own static module graph.
1010
//
1111
// Two files are removed from whatever Vitest returns:
12-
// - the subprocess-stub group (it spawns stubbed binaries and waits real
13-
// subprocess/retry/poll time — out of scope by the issue's constraint, and
14-
// thousands of mutant runs would turn it into timeout noise);
12+
// - the real-subprocess-spawn tests (SUBPROCESS_STUB_TESTS in vitest.config.ts —
13+
// spawns stubbed binaries and waits real subprocess/retry/poll time, out of scope
14+
// by the issue's constraint, and thousands of mutant runs would turn it into
15+
// timeout noise regardless of whether Vitest itself still serializes it, #1823);
1516
// - tests that cannot run in the thread pool Stryker's vitest runner forces:
1617
// the in-process CLI-capture tests (`process.chdir` throws in a worker
1718
// thread) and the `node:worker_threads` PNG pipeline tests (a worker inside

‎src/__tests__/test-utils/fake-adb.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ export type FakeAdbProviderExtras = AndroidAdbProvider extends infer P
3737
* installs per request and the provider-scenario lane exercises. Prefer this
3838
* over PATH-stub subprocess helpers (`withMockedAdb`): no PATH
3939
* mutation, no spawns, no real subprocess waits, so converted files can leave
40-
* the serialized `subprocess-stub` project (#1823).
40+
* SUBPROCESS_STUB_TESTS in vitest.config.ts (#1823).
4141
*
4242
* The fake `exec` receives device-scoped args without a leading
4343
* `-s <serial>`: scoped providers are per-device, and raw `runCmd('adb', …)`

‎vitest.config.ts‎

Lines changed: 22 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -2,12 +2,15 @@ import { defineConfig } from 'vitest/config';
22
import { resolveVitestMaxWorkers } from './scripts/lib/vitest-concurrency.ts';
33
import slowTestGateReporter from './scripts/vitest-slow-test-reporter.ts';
44

5-
// Files that spawn a real subprocess per case, so under broad file parallelism the
6-
// spawns get starved past an internal budget and production returns a generic
7-
// timeout instead of the asserted error. The subprocess-stub project below runs
8-
// them one at a time to bound that contention; per-file `process.env` isolation is
9-
// already delivered by `pool: forks` + `isolate: true` on every project.
10-
// Membership and the project's deletion test live in issue #1823.
5+
// Files that spawn a real subprocess per case. They used to run one at a time in
6+
// their own serialized `subprocess-stub` project so broad file parallelism couldn't
7+
// starve a spawn past its internal budget and turn it into a generic timeout.
8+
// #1823 is now running that project's own kill criterion: un-serialized here in
9+
// `unit-core`'s default forks pool, watched for 20 consecutive CI runs with no
10+
// timeout-shaped failure. Revert (restore the project, restore this list to
11+
// unit-core's exclude) the moment one appears. Still excluded from the mutation
12+
// lane via SERIALIZED_TESTS below regardless of this experiment's outcome —
13+
// thousands of mutant reruns times a real spawn per case is timeout noise either way.
1114
const SUBPROCESS_STUB_TESTS: readonly string[] = [
1215
// Stubs npx plus the package managers and spawns a real Metro dev server per case.
1316
'src/__tests__/client-metro.test.ts',
@@ -48,9 +51,11 @@ const FUZZ_WORKER_TESTS: readonly string[] = [
4851
'scripts/fuzz/corpus-replay.test.ts',
4952
];
5053
/**
51-
* Everything the serialized projects own, which is what the fast lane must not also collect.
52-
* The two lists above stay module-local: this union is the whole cross-file surface, and the
53-
* mutation lane wants exactly it — every test the root config declines to run in parallel.
54+
* Every test the mutation lane must not collect: a real per-case subprocess spawn is
55+
* timeout noise under thousands of mutant reruns, independent of whether Vitest also
56+
* serializes it — `fuzz-worker` still does; `subprocess-stub`'s former members no
57+
* longer do (#1823). The two lists above stay module-local: this union is the whole
58+
* cross-file surface, and the mutation lane wants exactly it.
5459
*/
5560
export const SERIALIZED_TESTS: readonly string[] = [...SUBPROCESS_STUB_TESTS, ...FUZZ_WORKER_TESTS];
5661

@@ -114,6 +119,10 @@ export default defineConfig({
114119
include: [
115120
'src/**/*.test.ts',
116121
'packages/*/src/**/*.test.ts',
122+
// The subprocess watchdog self-check (#1823): spawns a real node subprocess per
123+
// case, one hangs on purpose (#1414). Formerly a `subprocess-stub` member; see
124+
// SUBPROCESS_STUB_TESTS above for the kill-criterion experiment this rides.
125+
'scripts/fuzz/harness.test.ts',
117126
// The validation fuzz generators' expectation gates (#1781 B2): in-process, no
118127
// subprocess or worker, so they ride the fast lane unlike their serialized siblings.
119128
'scripts/fuzz/validation-arbitraries.test.ts',
@@ -179,25 +188,15 @@ export default defineConfig({
179188
// The Maestro conformance oracle runs via `node --test` in its own CI
180189
// job (scripts/maestro-conformance), like the layering guard.
181190
],
182-
exclude: [...SERIALIZED_TESTS],
191+
exclude: [...FUZZ_WORKER_TESTS],
183192
setupFiles: SETUP_FILES,
184193
},
185194
},
186195
{
187196
test: {
188-
name: 'subprocess-stub',
189-
include: [...SUBPROCESS_STUB_TESTS],
190-
setupFiles: SETUP_FILES,
191-
fileParallelism: false,
192-
isolate: true,
193-
maxWorkers: 1,
194-
},
195-
},
196-
{
197-
test: {
198-
// Same serialization as its sibling above, for the same contention reason: the
199-
// per-case watchdog budget is real wall clock. The project exists so the coverage
200-
// run can leave it out, not to run it differently.
197+
// Serialized for the same contention reason `subprocess-stub` used to be (#1823):
198+
// the per-case watchdog budget is real wall clock. The project exists so the
199+
// coverage run can leave it out (see the comment above), not to run it differently.
201200
name: 'fuzz-worker',
202201
include: [...FUZZ_WORKER_TESTS],
203202
setupFiles: SETUP_FILES,

‎vitest.mutation.config.ts‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,8 @@ const workspaceAliases = workspaceSourceAliases(repoRoot);
2424
// (`vitest related` over the mutated files) and hands it over through
2525
// AGENT_DEVICE_MUTATION_TEST_FILES; the fallback is the deterministic unit suite,
2626
// which keeps `pnpm exec stryker run` usable by hand. Excluded either way: the
27-
// serialized groups (subprocess-stub and fuzz-worker) and the CLI-capture tests — see
27+
// real-subprocess-spawn tests (SERIALIZED_TESTS — #1823, only `fuzz-worker` still
28+
// runs them in a serialized Vitest project) and the CLI-capture tests — see
2829
// scripts/mutation/test-scope.ts for why, and why excluding them cannot hide a
2930
// surviving mutant.
3031
const scope = readTestScope();

0 commit comments

Comments
 (0)