Skip to content

Commit 6b792a9

Browse files
ryansolidclaude
andauthored
test(web): async app generator — generated async apps, liveness oracle, equivalence pairs, regression gate (#4371)
* test(web): async app generator — generated Loading/Errored/Show trees over async memos, with safety and liveness oracles An opt-in campaign (ASYNC_GEN=1, pnpm async-gen) that generates small apps (signals, sync and async memos with manual flights, latest()/isPending() reads, Loading / Errored / Show / leaves with grouped or split attribute effects) and scripts (writes, actions, settling flights in any order, flush / microtask / task). Oracles: nothing throws or logs, no unhandled rejection, no hole shows undefined/NaN, and once every flight has settled the screen equals a fresh synchronous render of the final inputs. Survey, shrink and minimize modes; the minimizer reduces the app as well as the script. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_0117BoWKJDzwaZPgaGWE181b * test(web): async-gen equivalence pairs (grouping, identity memo, inert Errored, store twin, flush mode, owned memos) ASYNC_GEN_PAIRS=E1,E2,E2e,E3,E6,E7 runs each scenario a second time rewritten without changing its meaning and compares the screens at every settle point (after the first render, each task step and the end); differences are grouped by what each side shows. Scenarios also choose whether memos are created at module level or inside the root component. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_0117BoWKJDzwaZPgaGWE181b * test(web): async-gen regression gate against a recorded baseline 1,500 generated scenarios (seed 3289, ~15 s) run with the web suite; each case's safety and liveness laws are compared to baseline.json. A case that newly breaks a law fails the suite (a change broke a neighbouring shape); a recorded failure that no longer reproduces is reported, and ASYNC_GEN_UPDATE=1 records it. The baseline holds today's 20 failing cases (#4362-#4367 shapes). Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_0117BoWKJDzwaZPgaGWE181b * test(web): async-gen hydration phase Each generated scenario is emitted as a TSX module, rendered through renderToStream (server half), then hydrated chunk by chunk (client half). Oracles: no refetch of a read memo before the first write, no console.error or hydration warning, hydrated screen equals a fresh CSR render, then the scenario's steps and final liveness. Opt-in via ASYNC_GEN_HY=1 (pnpm async-gen:hydrate); minimize-hydration.mts reduces a failing case through both halves. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_0117BoWKJDzwaZPgaGWE181b * test(web): async-gen differential mode; fix test-types ASYNC_GEN_RECORD on a reference build writes every case's screens, final screen and broken laws; ASYNC_GEN_DIFF on a candidate build replays the same seed and fails on any difference. Both the client campaign and the hydration campaign take it: the gate for the reader-seat carve. Same-build record/diff is clean (client 500 cases, hydration 300). minimize-hydration.mts loads minimize.ts through a computed specifier so tsc (no allowImportingTsExtensions) accepts it; runCase's return type names screens/final. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_0117BoWKJDzwaZPgaGWE181b * test(web): async-gen pairs E8 (birth seat) and E9 (latest in a derive) E8 re-creates the tree's readers in the flush that applies each write and compares screens where nothing is in flight; E9 reads each latest(x) the tree makes through a memo of it. Both from the carve audit's seat pairs. First run (1,000 cases): #4377, #4378. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_0117BoWKJDzwaZPgaGWE181b --------- Co-authored-by: Claude <[email protected]>
1 parent 56683ea commit 6b792a9

13 files changed

Lines changed: 1811 additions & 0 deletions

File tree

‎packages/web/.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
test/async-gen/__gen__/

‎packages/web/package.json‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -367,6 +367,8 @@
367367
"test": "vitest run && vitest run --config vite.config.server.mjs && vitest run --config vite.config.hydrate.mjs",
368368
"test:server": "vitest run --config vite.config.server.mjs",
369369
"consistency:fuzz": "CONSISTENCY_FUZZ=1 vitest run --config vite.config.hydrate.mjs test/consistency/harness",
370+
"async-gen": "ASYNC_GEN=1 vitest run test/async-gen",
371+
"async-gen:hydrate": "ASYNC_GEN_HY=1 vitest run --config vite.config.server.mjs test/server/async-gen.gen.spec.tsx && ASYNC_GEN_HY=1 vitest run --config vite.config.hydrate.mjs test/consistency/async-gen",
370372
"coverage": "vitest run --coverage",
371373
"bench": "vitest bench --run",
372374
"bench:server": "vitest bench --run --config vite.config.server-bench.mjs",
Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
{
2+
"seed": 3289,
3+
"cases": 1500,
4+
"failing": {
5+
"112": [
6+
"safety/undefined-value"
7+
],
8+
"296": [
9+
"safety/undefined-value"
10+
],
11+
"310": [
12+
"safety/undefined-value"
13+
],
14+
"349": [
15+
"safety/undefined-value"
16+
],
17+
"465": [
18+
"liveness/final-screen"
19+
],
20+
"662": [
21+
"liveness/final-screen"
22+
],
23+
"752": [
24+
"safety/undefined-value"
25+
],
26+
"766": [
27+
"liveness/final-screen"
28+
],
29+
"858": [
30+
"safety/unhandled-rejection"
31+
],
32+
"864": [
33+
"safety/undefined-value"
34+
],
35+
"1046": [
36+
"liveness/final-screen"
37+
],
38+
"1091": [
39+
"liveness/final-screen"
40+
],
41+
"1188": [
42+
"liveness/final-screen"
43+
],
44+
"1222": [
45+
"liveness/final-screen"
46+
],
47+
"1237": [
48+
"liveness/final-screen"
49+
],
50+
"1258": [
51+
"liveness/final-screen"
52+
],
53+
"1270": [
54+
"safety/undefined-value"
55+
],
56+
"1307": [
57+
"safety/undefined-value"
58+
],
59+
"1383": [
60+
"liveness/final-screen"
61+
],
62+
"1455": [
63+
"safety/undefined-value"
64+
]
65+
}
66+
}
Lines changed: 208 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,208 @@
1+
/**
2+
* @vitest-environment jsdom
3+
*
4+
* The async app generator's campaign. Off by default; run with
5+
*
6+
* ASYNC_GEN=1 [ASYNC_GEN_SEED=3289] [ASYNC_GEN_CASES=200]
7+
* [ASYNC_GEN_IGNORE=final-screen,console-error] [ASYNC_GEN_MODE=survey|shrink]
8+
* npx vitest run test/async-gen
9+
*
10+
* ASYNC_GEN_RECORD / ASYNC_GEN_DIFF (survey mode): see differential.ts.
11+
*
12+
* survey runs every case and tallies findings by law and signature; shrink
13+
* stops on the first finding outside ASYNC_GEN_IGNORE and prints the reduced
14+
* scenario as JSON for replay.spec.ts.
15+
*/
16+
import { describe, expect, test } from "vitest";
17+
import fc from "fast-check";
18+
import { recordOrDiff, type Trace } from "./differential.js";
19+
import { minimize } from "./minimize.js";
20+
import { runScenario, type Finding, type Result, type Variant } from "./run.js";
21+
import { describeScenario, scenarioArb, type Scenario } from "./scenario.js";
22+
23+
const FUZZ = !!process.env.ASYNC_GEN;
24+
const SEED = Number(process.env.ASYNC_GEN_SEED ?? 3289);
25+
const CASES = Number(process.env.ASYNC_GEN_CASES ?? 200);
26+
const IGNORE = new Set((process.env.ASYNC_GEN_IGNORE ?? "").split(",").filter(Boolean));
27+
const MODE = process.env.ASYNC_GEN_MODE ?? "survey";
28+
29+
const PAIRS = (process.env.ASYNC_GEN_PAIRS ?? "").split(",").filter(Boolean);
30+
31+
const out = (s: string) => process.stdout.write(s + "\n");
32+
33+
/** The equivalence pairs: two runs of one scenario that must show the same
34+
* screen at every settle point (or, `finalOnly`, once everything settled). */
35+
const PAIR_DEFS: Record<
36+
string,
37+
{ a: Variant; b: Variant; finalOnly?: boolean; quietOnly?: boolean }
38+
> = {
39+
E1: { a: { grouped: true }, b: { split: true } },
40+
E2: { a: {}, b: { identity: true } },
41+
E2e: { a: {}, b: { inertErrored: true } },
42+
E3: { a: {}, b: { store: true } },
43+
E6: { a: {}, b: { autoFlush: true }, finalOnly: true },
44+
E7: { a: { owned: false }, b: { owned: true } },
45+
// The reader seat (carve audit): a reader born before a write vs one born
46+
// in the flush that applies it, compared where nothing is in flight (a
47+
// fresh reader of a pending node may show a fallback the old one holds);
48+
// `latest(x)` read in a derivation vs at the leaf.
49+
E8: { a: {}, b: { remount: true }, quietOnly: true },
50+
E9: { a: {}, b: { latestDerive: true } }
51+
};
52+
53+
/** What a screen shows, coarsely: an error, a fallback, or content. */
54+
const shows = (x: string) =>
55+
/\[(ROOT-)?E\]/.test(x)
56+
? "error"
57+
: /\[(ROOT-)?L\]/.test(x)
58+
? "loading"
59+
: /\[F\]/.test(x)
60+
? "show-fallback"
61+
: "content";
62+
/** How two screens differ, for grouping: by what they show, then by which
63+
* kind of read disagrees (value / latest / isPending). */
64+
function kind(x: string, y: string): string {
65+
const [a, b] = [shows(x), shows(y)];
66+
if (a !== b) return `${a} vs ${b}`;
67+
const reads = (s: string) =>
68+
new Map([...s.matchAll(/([vlp])([sm]\d):([^ <]*)/g)].map(m => [m[1] + m[2], m[3]]));
69+
const [ra, rb] = [reads(x), reads(y)];
70+
const diff = new Set<string>();
71+
for (const [k, v] of ra)
72+
if (rb.get(k) !== v) diff.add(k[0] === "v" ? "value" : k[0] === "l" ? "latest" : "isPending");
73+
for (const k of rb.keys()) if (!ra.has(k)) diff.add("presence");
74+
return diff.size ? `${a}: ${[...diff].sort().join("+")} differ` : `${a}: structure differs`;
75+
}
76+
77+
/** Runs the scenario's pairs; returns their findings, plus the base run's. */
78+
async function runWithPairs(sc: Scenario): Promise<{ base: Result; findings: Finding[] }> {
79+
const base = await runScenario(sc);
80+
const findings = [...base.findings];
81+
for (const name of PAIRS) {
82+
const def = PAIR_DEFS[name];
83+
const a = Object.keys(def.a).length ? await runScenario(sc, def.a) : base;
84+
const b = await runScenario(sc, def.b);
85+
const diff = def.finalOnly
86+
? a.final !== b.final
87+
? a.screens.length - 1
88+
: -1
89+
: a.screens.findIndex(
90+
(x, i) =>
91+
a.settled[i] &&
92+
b.settled[i] &&
93+
(!def.quietOnly || (a.quiet[i] && b.quiet[i])) &&
94+
x !== b.screens[i]
95+
);
96+
const at = diff >= 0 ? diff : a.final !== b.final ? a.screens.length : -1;
97+
if (at >= 0) {
98+
const x = at < a.screens.length ? a.screens[at] : a.final;
99+
const y = at < b.screens.length ? b.screens[at] : b.final;
100+
findings.push({
101+
oracle: "equivalence",
102+
law: name,
103+
step: at - 1,
104+
detail: `${kind(x, y)} :: ${x} ≠ ${y}`
105+
});
106+
}
107+
}
108+
return { base, findings };
109+
}
110+
const MATCH = process.env.ASYNC_GEN_MATCH ? new RegExp(process.env.ASYNC_GEN_MATCH) : null;
111+
const real = (fs: Finding[]) =>
112+
fs.filter(f => !IGNORE.has(f.law) && (!MATCH || MATCH.test(`${f.oracle}/${f.law}: ${f.detail}`)));
113+
/** A signature groups findings by law and the first words of the detail,
114+
* with numbers and values stripped. */
115+
const signature = (f: Finding) =>
116+
f.oracle === "equivalence"
117+
? `${f.oracle}/${f.law}: ${f.detail.slice(0, f.detail.indexOf(" :: "))}`
118+
: `${f.oracle}/${f.law}: ${f.detail.replace(/\d+/g, "#").replace(/≠.*/, "≠ …").slice(0, 90)}`;
119+
120+
describe("async-gen", () => {
121+
test("smoke: a settled async memo under Loading shows its value", async () => {
122+
const result = await runScenario({
123+
signals: [1],
124+
memos: [{ async: true, deps: [{ src: { k: "s", i: 0 }, how: "plain" }], poison: null }],
125+
tree: {
126+
t: "leaf",
127+
text: [{ how: "plain", src: { k: "m", i: 0 } }],
128+
attrs: [],
129+
grouped: true
130+
},
131+
steps: [{ t: "settle", pick: 0 }, { t: "task" }, { t: "write", s: 0, v: 2 }, { t: "task" }]
132+
});
133+
expect(result.findings).toEqual([]);
134+
expect(result.final).toBe("<span>vm0:3 </span>");
135+
});
136+
});
137+
138+
describe.skipIf(!FUZZ)("async-gen campaign", () => {
139+
test(`seed ${SEED}, ${CASES} cases, mode ${MODE}`, { timeout: 0 }, async () => {
140+
const arb = scenarioArb();
141+
if (MODE === "shrink") {
142+
try {
143+
await fc.assert(
144+
fc.asyncProperty(arb, async sc => {
145+
const r = await runWithPairs(sc);
146+
const f = real(r.findings)[0];
147+
if (f)
148+
throw new Error(
149+
`${f.oracle}/${f.law} @${f.step}: ${f.detail}\n${r.base.description}\n${JSON.stringify(sc)}`
150+
);
151+
}),
152+
{ seed: SEED, numRuns: CASES, endOnFailure: true }
153+
);
154+
out(`async-gen: seed ${SEED}, ${CASES} cases, no finding outside [${[...IGNORE]}]`);
155+
} catch (e: any) {
156+
out(String(e?.message ?? e));
157+
throw e;
158+
}
159+
return;
160+
}
161+
const tally = new Map<string, { count: number; first: Scenario; detail: string }>();
162+
let failing = 0;
163+
const traces: Record<number, Trace> = {};
164+
let idx = 0;
165+
for (const sc of fc.sample(arb, { seed: SEED, numRuns: CASES })) {
166+
const r = await runWithPairs(sc);
167+
traces[idx++] = {
168+
screens: r.base.screens,
169+
final: r.base.final,
170+
laws: [...new Set(r.findings.map(f => `${f.oracle}/${f.law}`))].sort()
171+
};
172+
const fs = real(r.findings);
173+
if (fs.length) failing++;
174+
const seen = new Set<string>();
175+
for (const f of fs) {
176+
const sig = signature(f);
177+
if (seen.has(sig)) continue;
178+
seen.add(sig);
179+
const t = tally.get(sig);
180+
if (t) t.count++;
181+
else tally.set(sig, { count: 1, first: sc, detail: `@${f.step} ${f.detail}` });
182+
}
183+
}
184+
const diffs = recordOrDiff(`client${PAIRS.length ? `+${PAIRS}` : ""}`, SEED, CASES, traces);
185+
if (diffs.length)
186+
out(`async-gen differential: ${diffs.length} cases differ\n ${diffs.join("\n ")}`);
187+
out(
188+
`async-gen survey: seed ${SEED}, ${CASES} cases, ${failing} with findings, ${tally.size} signatures`
189+
);
190+
for (const [sig, t] of [...tally].sort((a, b) => b[1].count - a[1].count)) {
191+
out(` ${t.count}× ${sig}`);
192+
out(` ${describeScenario(t.first)}`);
193+
out(` ${t.detail.slice(0, 300)}`);
194+
if (MODE !== "minimize") continue;
195+
// Reduce while some finding keeps the same oracle/law.
196+
const law = sig.slice(0, sig.indexOf(":"));
197+
const min = await minimize(t.first, async sc =>
198+
real((await runWithPairs(sc)).findings).some(f => `${f.oracle}/${f.law}` === law)
199+
);
200+
const r = await runWithPairs(min);
201+
const f = real(r.findings).find(f => `${f.oracle}/${f.law}` === law)!;
202+
out(` MIN ${r.base.description}`);
203+
out(` MIN @${f.step} ${f.detail.slice(0, 300)}`);
204+
out(` MIN ${JSON.stringify(min)}`);
205+
}
206+
expect(diffs, "cases that differ from ASYNC_GEN_DIFF's recording").toEqual([]);
207+
});
208+
});
Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
/**
2+
* Differential mode: one build records every case's trace (each screen, the
3+
* final screen, the laws it broke), another replays the same seed and fails
4+
* on any difference. The gate for a rewrite that must not change behaviour:
5+
*
6+
* ASYNC_GEN_RECORD=/tmp/next.json (on the reference build)
7+
* ASYNC_GEN_DIFF=/tmp/next.json (on the candidate build, same seed and cases)
8+
*
9+
* Both campaigns (client: campaign.spec.ts, hydration: hydrate.spec.tsx)
10+
* take the two variables; a difference names the case, the first screen
11+
* that differs and both sides.
12+
*/
13+
import { readFileSync, writeFileSync } from "node:fs";
14+
15+
export type Trace = { screens: string[]; final: string; laws: string[] };
16+
type Recording = { kind: string; seed: number; cases: number; traces: Record<number, Trace> };
17+
18+
const RECORD = process.env.ASYNC_GEN_RECORD;
19+
const DIFF = process.env.ASYNC_GEN_DIFF;
20+
export const differential = !!(RECORD || DIFF);
21+
22+
/** Writes the recording, or returns every difference from it (empty when
23+
* recording or when the traces match). */
24+
export function recordOrDiff(
25+
kind: string,
26+
seed: number,
27+
cases: number,
28+
traces: Record<number, Trace>
29+
): string[] {
30+
if (RECORD) {
31+
writeFileSync(RECORD, JSON.stringify({ kind, seed, cases, traces } satisfies Recording));
32+
return [];
33+
}
34+
if (!DIFF) return [];
35+
const ref = JSON.parse(readFileSync(DIFF, "utf8")) as Recording;
36+
if (ref.kind !== kind || ref.seed !== seed || ref.cases !== cases)
37+
return [
38+
`the recording is ${ref.kind} seed ${ref.seed} × ${ref.cases}, this run ${kind} seed ${seed} × ${cases}`
39+
];
40+
const out: string[] = [];
41+
for (const i of new Set([...Object.keys(ref.traces), ...Object.keys(traces)].map(Number))) {
42+
const [a, b] = [ref.traces[i], traces[i]];
43+
if (!a || !b) {
44+
out.push(`#${i}: ${a ? "missing in this run" : "missing in the recording"}`);
45+
continue;
46+
}
47+
const at = a.screens.findIndex((x, j) => x !== b.screens[j]);
48+
if (at >= 0 || a.screens.length !== b.screens.length)
49+
out.push(
50+
`#${i} screen ${at >= 0 ? at : a.screens.length}: ${a.screens[at] ?? "(none)"} → ${b.screens[at] ?? "(none)"}`
51+
);
52+
else if (a.final !== b.final) out.push(`#${i} final: ${a.final} → ${b.final}`);
53+
else if (a.laws.join() !== b.laws.join()) out.push(`#${i} laws: [${a.laws}] → [${b.laws}]`);
54+
}
55+
return out;
56+
}

0 commit comments

Comments
 (0)