Skip to content

Commit bca12c8

Browse files
refactor(fuzz): fast-check generators over the shared hazard list, drop the bundled lane-health work (#1414)
- Strip scripts/scheduled-lane/* and scheduled-lane-health.yml: that watcher is #1430's own deliverable and collides with PR #1439's implementation of the same lane. What this lane owes (a per-run envelope) moves into scripts/fuzz/envelope.ts. - Rebase onto #1437 and rebuild the generator layer on fast-check: cases come from arbitraries sharing SELECTOR_VALUE_HAZARDS with the property suite, and counterexamples are shrunk, so a failure names a minimal input plus fast-check's seed/path instead of a 20k-char random string. - Route harness.test.ts into the serialized subprocess-stub project. - Drop the AGENT_DEVICE_FUZZ_STARTUP_DELAY_MS test seam: the ready handshake is now proven by a case budget far below real worker startup. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
1 parent 417a391 commit bca12c8

24 files changed

Lines changed: 503 additions & 1255 deletions

‎.github/workflows/replays-nightly.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -57,9 +57,9 @@ jobs:
5757
--seed "$FUZZ_SEED" \
5858
--artifact-dir .tmp/fuzz/run
5959
60-
# Uploaded on pass as well as failure: each subdirectory of .tmp/fuzz holds the standard
61-
# scheduled-lane run-envelope.json (schemaVersion, commit/ref/run provenance, seed, config,
62-
# per-target durations, result), which is what freshness/health monitoring reads.
60+
# Uploaded on pass as well as failure: each subdirectory of .tmp/fuzz holds a
61+
# run-envelope.json (schemaVersion, commit/ref/run provenance, seed, config, per-target
62+
# durations, result), which is what freshness monitoring reads.
6363
- name: Upload run envelopes and failing cases
6464
if: always()
6565
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2

‎.github/workflows/scheduled-lane-health.yml‎

Lines changed: 0 additions & 58 deletions
This file was deleted.

‎docs/agents/testing.md‎

Lines changed: 11 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -136,29 +136,17 @@ pnpm fuzz:parsers --self-check # require the harness to stil
136136

137137
The generating run is nightly (`Parser Fuzz Lane` in `.github/workflows/replays-nightly.yml`, seeded
138138
by the run number). Every terminal path — pass, fail, `--self-check`, or a crash in the harness
139-
itself — writes `<artifact-dir>/run-envelope.json` in the standard scheduled-lane shape
140-
(`scripts/scheduled-lane/envelope.ts`: `schemaVersion`, `lane`, `result`, commit/ref/workflow-run
141-
provenance, seed and config, plus a lane-specific `details` payload — here per-target
142-
cases/failures/durations, failures, and repro commands). The nightly self-check and fuzz steps write
143-
to separate artifact subdirectories and both run unconditionally, so freshness/health monitoring
144-
(#1430) always finds an envelope; the step summary prints each one it finds and never fails on a
145-
missing file. Other scheduled lanes adopt the same writer rather than defining a second shape.
146-
147-
### Scheduled lane health
148-
149-
`pnpm lanes:health` is the consumer side of the envelope contract (#1430): it derives the lane list
150-
from `schedule:`-triggered workflows in `.github/workflows/` (never a hand-maintained list), reads
151-
each lane's recent scheduled runs from the GitHub API, and classifies it from recorded run fields —
152-
`healthy`, `failing` (two consecutive failed cadences), `dark` (no run within two cadences), or
153-
`pending` (no history to judge, e.g. a lane not on the default branch yet). A lane with no scheduled
154-
runs is judged against the commit that added its `schedule:` trigger, not the workflow's creation
155-
date, so adding a schedule to an old workflow still gets the full two-cadence grace — and an
156-
unreadable schedule date (shallow clone) stays `pending` rather than alerting. Freshness fields land
157-
in
158-
`lane-health.json` for the repo-health snapshot; an unhealthy lane opens or pings one tracking issue.
159-
It runs as `Scheduled Lane Health` (`.github/workflows/scheduled-lane-health.yml`) and locally with
160-
`--dry-run`, which never touches issues. Classification lives in
161-
`scripts/scheduled-lane/health-model.ts` and is unit-tested per category.
139+
itself — writes `<artifact-dir>/run-envelope.json` (`scripts/fuzz/envelope.ts`: `schemaVersion`,
140+
`lane`, `result`, commit/ref/workflow-run provenance, seed and config, plus per-target
141+
cases/failures/durations, failures, and repro commands), so a lane that goes dark or fails for weeks
142+
leaves a machine-readable trail. The self-check and fuzz steps write to separate artifact
143+
subdirectories and both run unconditionally; the step summary prints each envelope it finds and never
144+
fails on a missing file. The cross-lane version of this contract is #1430's own deliverable.
145+
146+
Cases come from fast-check arbitraries (`scripts/fuzz/arbitraries.ts`) built on the hazard vocabulary
147+
shared with `src/__tests__/test-utils/property-arbitraries.ts`, so a hazard added for the property
148+
suite reaches the fuzz lane too — and a counterexample is reported **shrunk**, with fast-check's seed
149+
and replay path printed alongside the saved artifact.
162150

163151
A nightly discovery reaches the unit lane by promotion, not hand-editing: the printed
164152
`promote:` command re-runs the downloaded artifact and appends it to

‎package.json‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -109,10 +109,9 @@
109109
"perf:ios": "node --experimental-strip-types scripts/perf/run.ts --platform ios",
110110
"perf:android": "node --experimental-strip-types scripts/perf/run.ts --platform android",
111111
"lint": "oxlint . --deny-warnings",
112-
"format": "node ./node_modules/oxfmt/bin/oxfmt --write src test skills scripts/fuzz scripts/scheduled-lane scripts/help-conformance-bench.mjs scripts/help-conformance-case-checks.mjs scripts/help-conformance-command-validator.ts scripts/help-conformance-expectations.mjs scripts/help-conformance-plan-validator.mjs scripts/help-conformance-runner-output.mjs scripts/help-conformance-summary.mjs scripts/help-conformance-cases.mjs scripts/help-conformance-sample-outputs.mjs scripts/__tests__/help-conformance-bench.test.ts scripts/__tests__/help-conformance-sample-outputs.test.ts scripts/__tests__/help-conformance-topic-coverage.test.ts package.json tsconfig.json tsconfig.lib.json tsdown.config.ts vitest.config.ts .github/actions/setup-node-pnpm/action.yml .oxlintrc.json .oxfmtrc.json '!test/skillgym/.skillgym-results/**'",
113-
"format:check": "node ./node_modules/oxfmt/bin/oxfmt --check src test skills scripts/fuzz scripts/scheduled-lane scripts/help-conformance-bench.mjs scripts/help-conformance-case-checks.mjs scripts/help-conformance-command-validator.ts scripts/help-conformance-expectations.mjs scripts/help-conformance-plan-validator.mjs scripts/help-conformance-runner-output.mjs scripts/help-conformance-summary.mjs scripts/help-conformance-cases.mjs scripts/help-conformance-sample-outputs.mjs scripts/__tests__/help-conformance-bench.test.ts scripts/__tests__/help-conformance-sample-outputs.test.ts scripts/__tests__/help-conformance-topic-coverage.test.ts package.json tsconfig.json tsconfig.lib.json tsdown.config.ts vitest.config.ts .github/actions/setup-node-pnpm/action.yml .oxlintrc.json .oxfmtrc.json '!test/skillgym/.skillgym-results/**'",
112+
"format": "node ./node_modules/oxfmt/bin/oxfmt --write src test skills scripts/fuzz scripts/help-conformance-bench.mjs scripts/help-conformance-case-checks.mjs scripts/help-conformance-command-validator.ts scripts/help-conformance-expectations.mjs scripts/help-conformance-plan-validator.mjs scripts/help-conformance-runner-output.mjs scripts/help-conformance-summary.mjs scripts/help-conformance-cases.mjs scripts/help-conformance-sample-outputs.mjs scripts/__tests__/help-conformance-bench.test.ts scripts/__tests__/help-conformance-sample-outputs.test.ts scripts/__tests__/help-conformance-topic-coverage.test.ts package.json tsconfig.json tsconfig.lib.json tsdown.config.ts vitest.config.ts .github/actions/setup-node-pnpm/action.yml .oxlintrc.json .oxfmtrc.json '!test/skillgym/.skillgym-results/**'",
113+
"format:check": "node ./node_modules/oxfmt/bin/oxfmt --check src test skills scripts/fuzz scripts/help-conformance-bench.mjs scripts/help-conformance-case-checks.mjs scripts/help-conformance-command-validator.ts scripts/help-conformance-expectations.mjs scripts/help-conformance-plan-validator.mjs scripts/help-conformance-runner-output.mjs scripts/help-conformance-summary.mjs scripts/help-conformance-cases.mjs scripts/help-conformance-sample-outputs.mjs scripts/__tests__/help-conformance-bench.test.ts scripts/__tests__/help-conformance-sample-outputs.test.ts scripts/__tests__/help-conformance-topic-coverage.test.ts package.json tsconfig.json tsconfig.lib.json tsdown.config.ts vitest.config.ts .github/actions/setup-node-pnpm/action.yml .oxlintrc.json .oxfmtrc.json '!test/skillgym/.skillgym-results/**'",
114114
"fuzz:parsers": "node --experimental-strip-types scripts/fuzz/run.ts",
115-
"lanes:health": "node --experimental-strip-types scripts/scheduled-lane/health.ts",
116115
"fallow": "fallow audit --base origin/main",
117116
"fallow:all": "fallow --summary",
118117
"fallow:baseline": "(fallow dead-code --save-baseline fallow-baselines/dead-code.json --summary || true) && (fallow health --save-baseline fallow-baselines/health.json --summary || true)",

‎scripts/fuzz/arbitraries.ts‎

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
// Case generators for the parser fuzz lane (#1414).
2+
//
3+
// fast-check rather than a bespoke PRNG, for two reasons the lane depends on: a reported
4+
// counterexample is the SHRUNK input (a hand-rolled mutator reports the 20k-character random one),
5+
// and the hazard vocabulary is the one #1437's property suite already curates — a hazard added
6+
// there for a round-trip property reaches the fuzzer without being retyped here.
7+
//
8+
// Cases stay near the grammar's edge on purpose: a well-formed base with hostile chunks spliced in
9+
// reaches deep parser branches that uniformly random noise never gets past the first token of.
10+
11+
import fc from 'fast-check';
12+
import {
13+
replayScriptArb,
14+
SELECTOR_VALUE_HAZARDS,
15+
selectorChainArb,
16+
} from '../../src/__tests__/test-utils/property-arbitraries.ts';
17+
import type { FuzzTarget, FuzzTargetName } from './target-types.ts';
18+
19+
/**
20+
* Hazards a valid-input property cannot use — they exist to be *rejected*: structural JSON/YAML
21+
* punctuation, delimiter lookalikes, prototype-pollution keys, numeric edges, bidi/zero-width
22+
* controls. The shared list above carries the ones valid inputs must also survive.
23+
*/
24+
const REJECTION_HAZARDS = [
25+
'`',
26+
'==',
27+
'&&',
28+
'--',
29+
'---',
30+
'#',
31+
':',
32+
',',
33+
'{',
34+
'}',
35+
'[',
36+
']',
37+
'(',
38+
')',
39+
'${',
40+
'${}',
41+
'@',
42+
'~=',
43+
'*',
44+
'\r\n',
45+
'\u0000',
46+
'\u200b',
47+
'\u202e',
48+
'\ufeff',
49+
'-0',
50+
'NaN',
51+
'Infinity',
52+
'1e999',
53+
'9007199254740993',
54+
'null',
55+
'undefined',
56+
'__proto__',
57+
'constructor',
58+
] as const;
59+
60+
const hazardArb: fc.Arbitrary<string> = fc.constantFrom(
61+
...SELECTOR_VALUE_HAZARDS,
62+
...REJECTION_HAZARDS,
63+
);
64+
65+
/** A base string with 1–4 hazards spliced in at shrinkable positions. */
66+
function corrupted(base: fc.Arbitrary<string>): fc.Arbitrary<string> {
67+
return fc
68+
.tuple(
69+
base,
70+
fc.array(fc.tuple(hazardArb, fc.nat({ max: 4096 })), { minLength: 1, maxLength: 4 }),
71+
)
72+
.map(([text, edits]) =>
73+
edits.reduce((current, [chunk, at]) => {
74+
const index = at % (current.length + 1);
75+
return current.slice(0, index) + chunk + current.slice(index);
76+
}, text),
77+
);
78+
}
79+
80+
/** A long run of one hazard — regex-backtracking and quadratic-scan bait. */
81+
const repeatedHazardArb: fc.Arbitrary<string> = fc
82+
.tuple(hazardArb, fc.integer({ min: 50, max: 400 }))
83+
.map(([chunk, times]) => chunk.repeat(times));
84+
85+
/** Input that is not trying to look like the grammar at all. */
86+
const noiseArb: fc.Arbitrary<string> = fc.oneof(
87+
fc.string({ maxLength: 40 }),
88+
fc.string({ unit: 'binary', maxLength: 40 }),
89+
fc.array(hazardArb, { maxLength: 8 }).map((parts) => parts.join('')),
90+
repeatedHazardArb,
91+
);
92+
93+
/**
94+
* Bases borrowed from the property suite, which generates *valid* inputs: corrupting a
95+
* grammar-correct script or selector chain is how the fuzzer reaches branches past the first
96+
* rejection. Targets without one generate from their own seed list.
97+
*/
98+
const STRUCTURED_BASES: Partial<Record<FuzzTargetName, fc.Arbitrary<string>>> = {
99+
selector: selectorChainArb.map((chain) => chain.expression),
100+
'replay-script': replayScriptArb,
101+
'batch-steps': fc.json({ maxDepth: 3 }),
102+
};
103+
104+
/** The case distribution for one target: mostly near-miss, some valid, some pure noise. */
105+
export function arbitraryForTarget(target: FuzzTarget): fc.Arbitrary<string> {
106+
const seeded = fc.constantFrom(...target.seeds);
107+
const structured = STRUCTURED_BASES[target.name];
108+
const base = structured === undefined ? seeded : fc.oneof(seeded, structured);
109+
return fc.oneof(
110+
{ weight: 6, arbitrary: corrupted(base) },
111+
{ weight: 2, arbitrary: base },
112+
{ weight: 2, arbitrary: noiseArb },
113+
);
114+
}

‎scripts/fuzz/corpus-replay.test.ts‎

Lines changed: 13 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -5,12 +5,13 @@
55
// Cases run synchronously here on purpose: a corpus case that hangs would hang the unit
66
// suite, which is exactly the signal (the nightly lane is where hangs are diagnosed).
77

8+
import fc from 'fast-check';
89
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
10+
import { arbitraryForTarget } from './arbitraries.ts';
911
import { readCorpus } from './corpus.ts';
1012
import { checkCase } from './invariant.ts';
1113
import { getFuzzTarget } from './registry.ts';
1214
import { FUZZ_TARGETS } from './targets.ts';
13-
import { generateCases } from './mutate.ts';
1415

1516
describe('parser fuzz regression corpus', () => {
1617
const corpus = readCorpus();
@@ -50,15 +51,18 @@ describe('parser fuzz regression corpus', () => {
5051
});
5152

5253
describe('fuzz case generation', () => {
53-
it('is deterministic for a seed, so a repro command reproduces', () => {
54-
const seeds = ['text=Login', 'label="Sign in" && role=button'];
55-
expect(generateCases(seeds, 32, 7)).toEqual(generateCases(seeds, 32, 7));
56-
expect(generateCases(seeds, 32, 7)).not.toEqual(generateCases(seeds, 32, 8));
54+
const target = getFuzzTarget('selector');
55+
56+
it('is deterministic for a seed, so a reported counterexample replays', () => {
57+
const sample = (seed: number) => fc.sample(arbitraryForTarget(target), { numRuns: 32, seed });
58+
expect(sample(7)).toEqual(sample(7));
59+
expect(sample(7)).not.toEqual(sample(8));
5760
});
5861

59-
it('always covers the verbatim seeds before mutating', () => {
60-
const seeds = ['a', 'b', 'c'];
61-
expect(generateCases(seeds, 1, 1)).toEqual(seeds);
62-
expect(generateCases(seeds, 10, 1).slice(0, 3)).toEqual(seeds);
62+
it('generates strings the target can be fed directly', () => {
63+
for (const input of fc.sample(arbitraryForTarget(target), { numRuns: 64, seed: 1 })) {
64+
expect(typeof input).toBe('string');
65+
expect(checkCase(target, input)).toBeNull();
66+
}
6367
});
6468
});

‎scripts/fuzz/envelope.ts‎

Lines changed: 66 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,17 @@
1-
// Parser fuzz lane's mapping onto the shared scheduled-lane envelope (#1414, #1430).
1+
// Run envelope for the parser fuzz lane (#1414).
22
//
3-
// The shape lives in scripts/scheduled-lane/envelope.ts so freshness/health monitoring reads one
4-
// envelope contract across lanes; this module only supplies the fuzz-specific `details` payload
5-
// and guarantees an envelope exists for *every* terminal path — pass, fail, or self-check.
6-
7-
import {
8-
buildLaneEnvelope,
9-
type LaneEnvelope,
10-
writeLaneEnvelope,
11-
} from '../scheduled-lane/envelope.ts';
3+
// A scheduled lane goes dark quietly: it can stop running, or fail for weeks, while PR CI stays
4+
// green. Freshness monitoring therefore needs one machine-readable record per run — green runs
5+
// included — describing which commit, tool, config, and seed produced the verdict. This lane
6+
// writes that record on *every* terminal path: pass, fail, self-check, or a crash in the harness.
7+
// The cross-lane version of this contract is #1430's deliverable; this is only what this lane owes.
8+
9+
import fs from 'node:fs';
10+
import path from 'node:path';
1211
import type { FuzzFailure } from './invariant.ts';
1312

13+
const SCHEMA_VERSION = 1;
14+
const FILENAME = 'run-envelope.json';
1415
const LANE = 'parser-fuzz';
1516
const TOOL = 'scripts/fuzz/run.ts';
1617

@@ -31,7 +32,46 @@ export type FuzzEnvelopeDetails = {
3132
reproCommands: string[];
3233
};
3334

34-
export type FuzzRunEnvelope = LaneEnvelope<FuzzEnvelopeDetails>;
35+
export type FuzzRunEnvelope = {
36+
schemaVersion: number;
37+
lane: string;
38+
/** `error` is for a run that could not complete itself (crash, bad config). */
39+
result: 'pass' | 'fail' | 'error';
40+
startedAt: string;
41+
finishedAt: string;
42+
durationMs: number;
43+
provenance: {
44+
commitSha: string | null;
45+
ref: string | null;
46+
workflow: string | null;
47+
workflowRunId: string | null;
48+
workflowRunNumber: string | null;
49+
workflowRunAttempt: string | null;
50+
nodeVersion: string;
51+
tool: string;
52+
};
53+
/** Everything that decides what the run did: seed, sizes, budgets, selected work. */
54+
config: Record<string, unknown>;
55+
details: FuzzEnvelopeDetails;
56+
};
57+
58+
function envOrNull(name: string): string | null {
59+
return process.env[name] ?? null;
60+
}
61+
62+
/** GitHub Actions exports these; a local run simply records `null`. */
63+
function provenanceFromEnv(): FuzzRunEnvelope['provenance'] {
64+
return {
65+
commitSha: envOrNull('GITHUB_SHA'),
66+
ref: envOrNull('GITHUB_REF'),
67+
workflow: envOrNull('GITHUB_WORKFLOW'),
68+
workflowRunId: envOrNull('GITHUB_RUN_ID'),
69+
workflowRunNumber: envOrNull('GITHUB_RUN_NUMBER'),
70+
workflowRunAttempt: envOrNull('GITHUB_RUN_ATTEMPT'),
71+
nodeVersion: process.version,
72+
tool: TOOL,
73+
};
74+
}
3575

3676
/** Writes the envelope for one fuzz run into `artifactDir`; returns its path. */
3777
export function writeFuzzEnvelope(input: {
@@ -42,16 +82,19 @@ export function writeFuzzEnvelope(input: {
4282
config: Record<string, unknown>;
4383
details: FuzzEnvelopeDetails;
4484
}): string {
45-
return writeLaneEnvelope(
46-
input.artifactDir,
47-
buildLaneEnvelope({
48-
lane: LANE,
49-
tool: TOOL,
50-
result: input.result,
51-
startedAt: input.startedAt,
52-
finishedAt: input.finishedAt,
53-
config: { mode: input.details.mode, ...input.config },
54-
details: input.details,
55-
}),
56-
);
85+
const envelope: FuzzRunEnvelope = {
86+
schemaVersion: SCHEMA_VERSION,
87+
lane: LANE,
88+
result: input.result,
89+
startedAt: new Date(input.startedAt).toISOString(),
90+
finishedAt: new Date(input.finishedAt).toISOString(),
91+
durationMs: input.finishedAt - input.startedAt,
92+
provenance: provenanceFromEnv(),
93+
config: { mode: input.details.mode, ...input.config },
94+
details: input.details,
95+
};
96+
fs.mkdirSync(input.artifactDir, { recursive: true });
97+
const file = path.join(input.artifactDir, FILENAME);
98+
fs.writeFileSync(file, `${JSON.stringify(envelope, null, 2)}\n`);
99+
return file;
57100
}

0 commit comments

Comments
 (0)