Skip to content

Commit 390531c

Browse files
feat(scripts): scenario runner that proves a fix in the real app
A scenario is one file: state, an optional replay transcript, and steps. `scenario.ts run` plays it in a real server and browser with the replay agent as the model; `prove` runs it on a base and a head commit and reports failsOnBase and passesOnHead with captures side by side. Co-Authored-By: Claude Fable 5.2 <[email protected]>
1 parent 08af12d commit 390531c

16 files changed

Lines changed: 1145 additions & 6 deletions

File tree

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@ apps/mobile/.android-cache/
3838
artifacts/app-store/screenshots/
3939
artifacts/mobile-e2e/
4040
artifacts/mobile-behavior/
41+
artifacts/scenarios/
4142
.github/pr-assets/
4243
native/**/target/
4344
node_modules/

‎AGENTS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -108,7 +108,7 @@ An empty database is a bad test. Seed your worktree's `.t3` with a copy of real
108108
- [Testing](docs/internals/testing.md) carries these same test rules, plus review guidance, for review bots. Change both together.
109109
- **Do not run repo-wide checks.** No `vp check`, no `vp run -r test`, no `vp run -r typecheck` unless I ask. CI owns the full suite.
110110
- Backend behavior changes ship with focused tests for that behavior.
111-
- Proof for a bug fix depends on the bug. Logic, data, state, and protocol bugs get a failing test first (the fewest that fail on `main` for that reason, in an existing case of the module's test file where one fits), then the fix. Purely visual bugs (spacing, colour, alignment, a layout shift) get before/after captures as their proof and no test: do not add UI tests for nitpicks, and do not ask for one in review when a capture already shows it. A server-only fix with no visible surface ships the test plus a short runtime transcript.
111+
- Proof for a bug fix depends on the bug. Logic, data, state, and protocol bugs get a failing test first (the fewest that fail on `main` for that reason, in an existing case of the module's test file where one fits), then the fix. Purely visual bugs (spacing, colour, alignment, a layout shift) get before/after captures as their proof and no test: do not add UI tests for nitpicks, and do not ask for one in review when a capture already shows it. A server-only fix with no visible surface ships the test plus a short runtime transcript. For a bug a user can reproduce in the web client, `node scripts/scenario.ts prove` shows it failing on base and passing on the fix in the real app; see [Scenarios](docs/operations/scenarios.md).
112112
- The server is event-sourced, and side effects run after the command commits. In tests, drain the effect worker (`OrchestrationEffectWorkerV2.drain`) or await the specific persisted event or `Deferred` that marks the milestone. Never wait on sleeps or polling. A test that needs a timeout to pass is wrong. Concretely:
113113
- `it.effect` runs on a `TestClock`; `Effect.sleep` only advances through `TestClock.adjust`. `it.live` is wall-clock and only for subprocess or network fixtures.
114114
- No `yieldNow` or `sleep` polling loops. Await the `Deferred`, take from the `Queue`, or drain the worker.

‎apps/server/scripts/dev-replay-agent.ts‎

Lines changed: 20 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -172,11 +172,21 @@ function permissionRejectedToolCallId(request: LogicalFrame, answer: unknown): s
172172
}
173173

174174
function runAgent(): void {
175+
// `--transcript <file>` serves that one recording as the only model, for the scenario runner.
176+
const transcriptFlag = process.argv.indexOf("--transcript");
177+
const transcript = transcriptFlag === -1 ? undefined : process.argv[transcriptFlag + 1];
178+
const scenarios: ReadonlyArray<{
179+
readonly id: string;
180+
readonly name: string;
181+
readonly file: string;
182+
}> =
183+
transcript === undefined ? SCENARIOS : [{ id: "scenario", name: "Scenario", file: transcript }];
184+
const defaultScenario = scenarios[0]!;
175185
const turnsByScenario = new Map<string, ReadonlyArray<ReplayTurn>>();
176186
const loadTurns = (scenarioId: string) => {
177187
const cached = turnsByScenario.get(scenarioId);
178188
if (cached !== undefined) return cached;
179-
const scenario = SCENARIOS.find((candidate) => candidate.id === scenarioId) ?? SCENARIOS[0];
189+
const scenario = scenarios.find((candidate) => candidate.id === scenarioId) ?? defaultScenario;
180190
const turns = parseReplayTurns(
181191
NodeFS.readFileSync(new URL(scenario.file, import.meta.url), "utf8"),
182192
);
@@ -291,12 +301,17 @@ function runAgent(): void {
291301
const sessionId =
292302
message.method === "session/load" ? String(params.sessionId) : NodeCrypto.randomUUID();
293303
const cwd = typeof params.cwd === "string" ? params.cwd : process.cwd();
294-
sessions.set(sessionId, { cwd, model: SCENARIOS[0].id, promptCount: 0, cancel: undefined });
304+
sessions.set(sessionId, {
305+
cwd,
306+
model: defaultScenario.id,
307+
promptCount: 0,
308+
cancel: undefined,
309+
});
295310
return {
296311
...(message.method === "session/new" ? { sessionId } : {}),
297312
models: {
298-
currentModelId: SCENARIOS[0].id,
299-
availableModels: SCENARIOS.map((scenario) => ({
313+
currentModelId: defaultScenario.id,
314+
availableModels: scenarios.map((scenario) => ({
300315
modelId: scenario.id,
301316
name: scenario.name,
302317
})),
@@ -305,7 +320,7 @@ function runAgent(): void {
305320
}
306321
case "session/set_model": {
307322
const session = sessions.get(String(params.sessionId));
308-
if (!SCENARIOS.some((scenario) => scenario.id === params.modelId)) {
323+
if (!scenarios.some((scenario) => scenario.id === params.modelId)) {
309324
throw Object.assign(new Error(`Unknown replay scenario ${String(params.modelId)}`), {
310325
code: -32602,
311326
});

‎docs/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,7 @@ source alone does not explain. Most code changes do not need an internal documen
6565
### Runbooks
6666

6767
- [Development and local builds](./operations/development.md)
68+
- [Scenarios: real-app proof for a fix](./operations/scenarios.md)
6869
- [T3 Connect setup](./operations/connect-setup.md)
6970
- [Release](./operations/release.md)
7071
- [Observability](./operations/observability.md)

‎docs/operations/scenarios.md‎

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
# Scenarios
2+
3+
A scenario shows one behavior in the real app: a real server on a temporary T3 home, real HTTP
4+
and WebSocket, and a real browser on the real web client. The replay agent stands in for the
5+
model, so a run is free and repeatable. Use one as the proof for a bug fix: it fails on the base
6+
commit and passes on the fix.
7+
8+
Scenarios live in [`scripts/scenarios`](../../scripts/scenarios), one file each. The runner is
9+
[`scripts/scenario.ts`](../../scripts/scenario.ts).
10+
11+
## Write one
12+
13+
```ts
14+
import { defineScenario } from "../scenario-runner/scenario.ts";
15+
16+
const submit = { role: "button", name: "Continue (Enter)" } as const;
17+
18+
export default defineScenario({
19+
title: "Tab moves from the add-project Git URL field to its submit button",
20+
state: { projects: [{ title: "demo" }] },
21+
steps: [
22+
{
23+
name: "Open Add project and choose Git URL",
24+
do: async (ui) => {
25+
await ui.click({ role: "button", name: "Add project" });
26+
await ui.click("Git URL");
27+
},
28+
expect: (ui) => ui.focused({ role: "combobox" }),
29+
},
30+
{ name: "Press Tab", do: (ui) => ui.press("Tab"), expect: (ui) => ui.focused(submit) },
31+
],
32+
});
33+
```
34+
35+
A scenario has three parts. [`scenario.ts`](../../scripts/scenario-runner/scenario.ts) holds the
36+
types.
37+
38+
- **State** is what exists before the browser opens. `projects` become git repositories holding
39+
their `files`. `threads` are created through the app's own CLI, and each of their `messages`
40+
plays one turn of the transcript, so their history is real events, not inserted rows. The
41+
app marks those messages as sent by an agent.
42+
`settings` is merged into the server's `settings.json`.
43+
- **Transcript** is the recorded session the replay agent plays, one turn per message, in the
44+
`registry_transcript.ndjson` format of the
45+
[replay fixtures](../../apps/server/src/orchestration-v2/testkit/fixtures). Give a path beside
46+
the scenario or from the repository root. Without one, a message plays the
47+
[showcase turn](../../apps/server/scripts/dev-replay/showcase_transcript.ndjson). New threads
48+
default to the replay agent, so a step only types and sends.
49+
- **Steps** are what the user does (`do`) and what must then be true (`expect`). Expectations
50+
wait up to 15 seconds to become true. A step fails only when its `expect` throws. A throw in
51+
`do` means the scenario could not run, which proves nothing, so keep checks out of `do`.
52+
53+
Point at controls the way a user finds them: visible text as a string, or `{ role, name }`,
54+
`{ label }`, `{ placeholder }`. A string `name` matches the whole accessible name; pass a regular
55+
expression for a partial match. `ui.page` is the Playwright page for anything else, such as
56+
seeding browser storage or reading a scroll position.
57+
58+
Write the scenario so that it fails for the reported reason. Run it on the base commit and read
59+
the failing expectation and the capture before you trust it.
60+
61+
## Run one
62+
63+
```sh
64+
node scripts/scenario.ts run add-project-tab-reaches-submit
65+
```
66+
67+
It prints a JSON result and writes it to `artifacts/scenarios/<name>/result.json` with a capture
68+
per step, `run.webm`, and the app's log. `--app-dir <checkout>` runs the app from another
69+
checkout, and `--out <dir>` moves the output. `status` is `passed`, `failed` (an expectation did
70+
not hold; the step carries it in `message`), or `error` (the scenario could not run). The exit
71+
code is 0, 1, or 2 in that order.
72+
73+
A run starts the full dev stack, so it takes a minute or more and wants the memory of
74+
`vp run dev`. Run one at a time on a shared machine. The browser is the Chromium that
75+
`node scripts/node_modules/playwright-core/cli.js install --only-shell chromium` installs.
76+
77+
## Prove a fix
78+
79+
```sh
80+
node scripts/scenario.ts prove add-project-tab-reaches-submit \
81+
--base e509db3a2f^ --head e509db3a2f --upload-command "ghdrop upload"
82+
```
83+
84+
`prove` runs the scenario on both commits and writes `<name>.proof.json` and `<name>.proof.md`
85+
beside the captures. The JSON carries `failsOnBase`, `passesOnHead`, `baseSha`, `headSha`,
86+
`verdict`, the `steps` with each side's status and capture, `captures`, and `markdown`: a table
87+
with base and head captures side by side, each labelled with its commit, ready for a pull request
88+
description. `--upload-command` names a command that takes the capture paths and prints one URL
89+
per line; the Markdown then embeds those URLs. `--head` defaults to `HEAD`.
90+
91+
It exits 0 only when both `failsOnBase` and `passesOnHead` are true. A scenario that passes on
92+
base does not reproduce the bug, and the verdict says so. `failsOnBase` is also false when the
93+
scenario could not run on base, because a broken run is not evidence.
94+
95+
The scenario and the runner always come from your checkout. Each commit's app runs from a
96+
temporary worktree beside your checkout, installed with `vp i` and removed afterwards. A commit
97+
that is your clean `HEAD` runs in place. This lets a new scenario prove a fix against a base that
98+
predates it.
99+
100+
## Other tiers
101+
102+
Only the web client with the replay agent runs today. The format leaves room for two more
103+
without rewriting scenarios:
104+
105+
- **Live provider.** Same state and steps. The runner would set the default model to a real
106+
provider's cheapest model instead of the replay agent and skip the transcript. Expectations on
107+
exact reply text would not hold, so a scenario meant for both tiers asserts on structure.
108+
- **Mobile.** Same state and transcript, against a server the leased device pairs with, as
109+
[mobile behavior](../../scripts/mobile-behavior/README.md) does. Steps that use only the `ui`
110+
methods can map onto agent-device accessibility selectors. Steps that use `ui.page` or `css`
111+
targets are web-only.
112+
113+
## In CI
114+
115+
No job runs scenarios. Never run all of them on every pull request: each boots the app. To run
116+
the ones a branch adds or changes:
117+
118+
```sh
119+
git diff --name-only --diff-filter=AM origin/main...HEAD -- scripts/scenarios \
120+
| xargs -r -n1 node scripts/scenario.ts run
121+
```
122+
123+
A job for this needs the Chromium install step the existing browser-test jobs in
124+
[`ci.yml`](../../.github/workflows/ci.yml) use.
125+
126+
## What a scenario cannot cover
127+
128+
- Desktop shell behavior (menus, windows, IPC) and the mobile app.
129+
- Provider-specific behavior. The replay agent speaks generic ACP: session updates, permission
130+
requests, and questions. Vendor frames and real tool side effects are not replayed.
131+
- State no command can create, such as a run stopped by a usage limit. A step can seed browser
132+
storage through `ui.page`, but there is no database fixture hook.
133+
- Timing. Replay streaming is simulated, so a scenario cannot show a performance regression.
134+
- Purely visual bugs. A capture shows them, but an expectation about pixels is not one this
135+
format can state; use before and after captures as the proof.

‎knip.jsonc‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,8 @@
3131
"linux-after-pack.ts",
3232
"mobile-behavior/run.ts",
3333
"mobile-native-client.ts",
34+
// The scenario runner imports scenario files by path.
35+
"scenarios/*.ts",
3436
"setup-worktree.ts",
3537
"sign-macos.ts",
3638
"prepare-server-compile-cache.cjs",

‎package.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@
4040
"mobile:install": "node scripts/mobile-native-client.ts ensure",
4141
"mobile:behavior": "node scripts/mobile-behavior.ts",
4242
"mobile:e2e": "node scripts/mobile-e2e.ts",
43+
"scenario": "node scripts/scenario.ts",
4344
"test": "vp run -r test",
4445
"test:cached": "node scripts/test-cached.ts",
4546
"test:perf:v2-wire": "vp test run apps/server/src/orchestration-v2/WireProjection.test.ts apps/server/src/orchestration-v2/threadHistoryPaging.test.ts apps/server/src/orchestration-v2/ThreadStream.test.ts apps/server/src/orchestration-v2/ThreadTransportPerformance.test.ts apps/server/src/orchestration-v2/ShellStream.test.ts apps/server/src/orchestration-v2/CommandPolicy.test.ts apps/server/src/orchestration-v2/runtimeLayer.test.ts apps/server/src/ws.test.ts packages/contracts/src/orchestrationV2.test.ts packages/client-runtime/src/operations/commands.test.ts packages/client-runtime/src/operations/commands.performance.test.ts packages/client-runtime/src/state/boundedThreadSnapshotHttp.test.ts packages/client-runtime/src/state/threadHistoryController.test.ts packages/client-runtime/src/state/threadHistoryMerge.test.ts packages/client-runtime/src/state/threads-sync.test.ts packages/client-runtime/src/state/shellReducer.test.ts",

0 commit comments

Comments
 (0)