|
| 1 | +# Scenarios |
| 2 | + |
| 3 | +A scenario shows one behavior in the real app: a real server on a temporary T3 home, real HTTP |
| 4 | +and WebSocket, and a real browser on the real web client. The replay agent stands in for the |
| 5 | +model, so a run is free and repeatable. Use one as the proof for a bug fix: it fails on the base |
| 6 | +commit and passes on the fix. |
| 7 | + |
| 8 | +Scenarios live in [`scripts/scenarios`](../../scripts/scenarios), one file each. The runner is |
| 9 | +[`scripts/scenario.ts`](../../scripts/scenario.ts). |
| 10 | + |
| 11 | +## Write one |
| 12 | + |
| 13 | +```ts |
| 14 | +import { defineScenario } from "../scenario-runner/scenario.ts"; |
| 15 | + |
| 16 | +const submit = { role: "button", name: "Continue (Enter)" } as const; |
| 17 | + |
| 18 | +export default defineScenario({ |
| 19 | + title: "Tab moves from the add-project Git URL field to its submit button", |
| 20 | + state: { projects: [{ title: "demo" }] }, |
| 21 | + steps: [ |
| 22 | + { |
| 23 | + name: "Open Add project and choose Git URL", |
| 24 | + do: async (ui) => { |
| 25 | + await ui.click({ role: "button", name: "Add project" }); |
| 26 | + await ui.click("Git URL"); |
| 27 | + }, |
| 28 | + expect: (ui) => ui.focused({ role: "combobox" }), |
| 29 | + }, |
| 30 | + { name: "Press Tab", do: (ui) => ui.press("Tab"), expect: (ui) => ui.focused(submit) }, |
| 31 | + ], |
| 32 | +}); |
| 33 | +``` |
| 34 | + |
| 35 | +A scenario has three parts. [`scenario.ts`](../../scripts/scenario-runner/scenario.ts) holds the |
| 36 | +types. |
| 37 | + |
| 38 | +- **State** is what exists before the browser opens. `projects` become git repositories holding |
| 39 | + their `files`. `threads` are created through the app's own CLI, and each of their `messages` |
| 40 | + plays one turn of the transcript, so their history is real events, not inserted rows. The |
| 41 | + app marks those messages as sent by an agent. |
| 42 | + `settings` is merged into the server's `settings.json`. |
| 43 | +- **Transcript** is the recorded session the replay agent plays, one turn per message, in the |
| 44 | + `registry_transcript.ndjson` format of the |
| 45 | + [replay fixtures](../../apps/server/src/orchestration-v2/testkit/fixtures). Give a path beside |
| 46 | + the scenario or from the repository root. Without one, a message plays the |
| 47 | + [showcase turn](../../apps/server/scripts/dev-replay/showcase_transcript.ndjson). New threads |
| 48 | + default to the replay agent, so a step only types and sends. |
| 49 | +- **Steps** are what the user does (`do`) and what must then be true (`expect`). Expectations |
| 50 | + wait up to 15 seconds to become true. A step fails only when its `expect` throws. A throw in |
| 51 | + `do` means the scenario could not run, which proves nothing, so keep checks out of `do`. |
| 52 | + |
| 53 | +Point at controls the way a user finds them: visible text as a string, or `{ role, name }`, |
| 54 | +`{ label }`, `{ placeholder }`. A string `name` matches the whole accessible name; pass a regular |
| 55 | +expression for a partial match. `ui.page` is the Playwright page for anything else, such as |
| 56 | +seeding browser storage or reading a scroll position. |
| 57 | + |
| 58 | +Write the scenario so that it fails for the reported reason. Run it on the base commit and read |
| 59 | +the failing expectation and the capture before you trust it. |
| 60 | + |
| 61 | +## Run one |
| 62 | + |
| 63 | +```sh |
| 64 | +node scripts/scenario.ts run add-project-tab-reaches-submit |
| 65 | +``` |
| 66 | + |
| 67 | +It prints a JSON result and writes it to `artifacts/scenarios/<name>/result.json` with a capture |
| 68 | +per step, `run.webm`, and the app's log. `--app-dir <checkout>` runs the app from another |
| 69 | +checkout, and `--out <dir>` moves the output. `status` is `passed`, `failed` (an expectation did |
| 70 | +not hold; the step carries it in `message`), or `error` (the scenario could not run). The exit |
| 71 | +code is 0, 1, or 2 in that order. |
| 72 | + |
| 73 | +A run starts the full dev stack, so it takes a minute or more and wants the memory of |
| 74 | +`vp run dev`. Run one at a time on a shared machine. The browser is the Chromium that |
| 75 | +`node scripts/node_modules/playwright-core/cli.js install --only-shell chromium` installs. |
| 76 | + |
| 77 | +## Prove a fix |
| 78 | + |
| 79 | +```sh |
| 80 | +node scripts/scenario.ts prove add-project-tab-reaches-submit \ |
| 81 | + --base e509db3a2f^ --head e509db3a2f --upload-command "ghdrop upload" |
| 82 | +``` |
| 83 | + |
| 84 | +`prove` runs the scenario on both commits and writes `<name>.proof.json` and `<name>.proof.md` |
| 85 | +beside the captures. The JSON carries `failsOnBase`, `passesOnHead`, `baseSha`, `headSha`, |
| 86 | +`verdict`, the `steps` with each side's status and capture, `captures`, and `markdown`: a table |
| 87 | +with base and head captures side by side, each labelled with its commit, ready for a pull request |
| 88 | +description. `--upload-command` names a command that takes the capture paths and prints one URL |
| 89 | +per line; the Markdown then embeds those URLs. `--head` defaults to `HEAD`. |
| 90 | + |
| 91 | +It exits 0 only when both `failsOnBase` and `passesOnHead` are true. A scenario that passes on |
| 92 | +base does not reproduce the bug, and the verdict says so. `failsOnBase` is also false when the |
| 93 | +scenario could not run on base, because a broken run is not evidence. |
| 94 | + |
| 95 | +The scenario and the runner always come from your checkout. Each commit's app runs from a |
| 96 | +temporary worktree beside your checkout, installed with `vp i` and removed afterwards. A commit |
| 97 | +that is your clean `HEAD` runs in place. This lets a new scenario prove a fix against a base that |
| 98 | +predates it. |
| 99 | + |
| 100 | +## Other tiers |
| 101 | + |
| 102 | +Only the web client with the replay agent runs today. The format leaves room for two more |
| 103 | +without rewriting scenarios: |
| 104 | + |
| 105 | +- **Live provider.** Same state and steps. The runner would set the default model to a real |
| 106 | + provider's cheapest model instead of the replay agent and skip the transcript. Expectations on |
| 107 | + exact reply text would not hold, so a scenario meant for both tiers asserts on structure. |
| 108 | +- **Mobile.** Same state and transcript, against a server the leased device pairs with, as |
| 109 | + [mobile behavior](../../scripts/mobile-behavior/README.md) does. Steps that use only the `ui` |
| 110 | + methods can map onto agent-device accessibility selectors. Steps that use `ui.page` or `css` |
| 111 | + targets are web-only. |
| 112 | + |
| 113 | +## In CI |
| 114 | + |
| 115 | +No job runs scenarios. Never run all of them on every pull request: each boots the app. To run |
| 116 | +the ones a branch adds or changes: |
| 117 | + |
| 118 | +```sh |
| 119 | +git diff --name-only --diff-filter=AM origin/main...HEAD -- scripts/scenarios \ |
| 120 | + | xargs -r -n1 node scripts/scenario.ts run |
| 121 | +``` |
| 122 | + |
| 123 | +A job for this needs the Chromium install step the existing browser-test jobs in |
| 124 | +[`ci.yml`](../../.github/workflows/ci.yml) use. |
| 125 | + |
| 126 | +## What a scenario cannot cover |
| 127 | + |
| 128 | +- Desktop shell behavior (menus, windows, IPC) and the mobile app. |
| 129 | +- Provider-specific behavior. The replay agent speaks generic ACP: session updates, permission |
| 130 | + requests, and questions. Vendor frames and real tool side effects are not replayed. |
| 131 | +- State no command can create, such as a run stopped by a usage limit. A step can seed browser |
| 132 | + storage through `ui.page`, but there is no database fixture hook. |
| 133 | +- Timing. Replay streaming is simulated, so a scenario cannot show a performance regression. |
| 134 | +- Purely visual bugs. A capture shows them, but an expectation about pixels is not one this |
| 135 | + format can state; use before and after captures as the proof. |
0 commit comments