Skip to content

Commit 1da9efe

Browse files
fix(scripts): scenarios reach the screen and say why they could not run
t3.open waits for the pairing page to sign in before it navigates, and takes a thread from t3.thread, so a scenario opens a thread by its link instead of through the sidebar. scenarios:prove runs one target, replaces a worktree a killed run left, writes each side as it ends, and exits 3 when the app ran and the scenario stopped before its assertion. T3_SCENARIO_INHERIT_ENV passes variables to the app. Co-Authored-By: Claude Fable 5.2 <[email protected]>
1 parent 2dd207a commit 1da9efe

6 files changed

Lines changed: 196 additions & 29 deletions

File tree

‎docs/operations/scenarios.md‎

Lines changed: 47 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -41,8 +41,10 @@ adds one fixture for what e2e cannot know about T3 Code:
4141

4242
- `t3.project(title, files)` creates a git repository and adds it as a project.
4343
- `t3.thread({ project, title, messages })` starts a thread and plays one replayed turn per
44-
message before returning, so the thread has real history. `model` picks the recording.
45-
- `t3.open(path)` opens the app, already signed in.
44+
message before returning, so the thread has real history. `model` picks the recording. It
45+
returns the thread, which `t3.open` takes.
46+
- `t3.open(path)` opens the app, already signed in, at a path. `t3.open(thread)` opens it on a
47+
thread.
4648

4749
State is created through the app's own CLI, so it is real events and works on a base commit
4850
whose schema differs. New threads default to the replay agent's `showcase` recording: type a
@@ -61,6 +63,30 @@ could not run.
6163
Keep `agent.act` and `agent.assert` out of scenarios: they call a model, and no model is
6264
configured.
6365

66+
### Getting to the screen
67+
68+
Most scenarios that could not run stopped on the way to the screen the bug is on, not on the
69+
bug. Three rules avoid that:
70+
71+
- **Open by link, not by sidebar.** `await t3.open(thread)` lands on the thread
72+
([`thread-opens-by-link.e2e.ts`](../../scripts/scenarios/thread-opens-by-link.e2e.ts)), and
73+
`await t3.open("/settings/general")` on a settings page. A sidebar row is a button that covers
74+
its own title, so a tap on `getByText("Earlier work")` is intercepted. When the sidebar is what
75+
the scenario is about, target the row's button: `screen.getByRole("button", "Earlier work")`.
76+
- **Let `t3.open` do the navigating.** It waits until the pairing page has signed in before it
77+
goes anywhere. A second `app.open` straight after the pairing link cancels the sign-in and
78+
shows the pairing page again.
79+
- **Seed before opening.** Create projects and threads first, then call `t3.open`. With no
80+
project the app opens its first-run page instead of the one the scenario expects.
81+
82+
The app starts with only `PATH`, `HOME` and the temp directory in its environment, whatever the
83+
shell that ran the scenario has set. Name what it needs in `T3_SCENARIO_INHERIT_ENV`: exact
84+
names or prefixes ending in `*`, separated by commas.
85+
86+
```sh
87+
T3_SCENARIO_INHERIT_ENV='SHELL,MY_FIXTURE_*' vp run scenarios:prove scripts/scenarios/name.e2e.ts --base main
88+
```
89+
6490
## Run
6591

6692
```sh
@@ -102,12 +128,27 @@ commits and compares the two reports. It prints, and writes to `scripts/.e2e/pro
102128
its commit, ready for a pull request description. `--upload-command` names a command that takes
103129
the capture paths and prints one URL per line. `--head` defaults to `HEAD`.
104130

105-
It exits 0 only when the scenario fails on base and passes on head, 1 when it ran and did not
106-
prove the fix, and 2 when it could not run. A scenario that passes on base does not reproduce the
107-
bug, and the verdict says so.
131+
It exits 0 only when the scenario fails on base and passes on head, and 1 when it ran and did not
132+
prove the fix. A scenario that passes on base does not reproduce the bug, and the verdict says so.
133+
134+
Two more codes mean there is no verdict on the behavior, and `couldNotRun` in the proof says
135+
which:
136+
137+
- **3, `scenario`:** the app ran and the scenario stopped before its assertion, usually on a
138+
control it did not find. Read that side's `message` and `steps`, fix the scenario, and run it
139+
again.
140+
- **2, `app`:** the app did not start or no test ran: a bad revision, a failed install, a port in
141+
use. `scripts/.e2e/app.log` has the app's output.
142+
143+
Each side's result is also written as it ends, to `scripts/.e2e/prove/base.json` and
144+
`head.json`, so a run that is cut short keeps the side it finished.
145+
146+
It runs the scenario on one target: iOS when the file imports `t3-mobile.ts`, web otherwise.
147+
`--target web` or `--target ios` overrides that.
108148

109149
The scenario and e2e always come from your checkout. Only the app comes from the commit under
110-
test: a temporary worktree beside your checkout, installed with `vp i` and removed afterwards. A
150+
test: a temporary worktree beside your checkout, installed with `vp i` and removed afterwards (one
151+
left by a run that was killed is replaced). A
111152
commit that is your clean `HEAD` runs in place. So a new scenario can prove a fix against a base
112153
that predates it.
113154

‎scripts/e2e.config.ts‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,12 @@ const resolved = await Effect.gen(function* () {
3131
appDir: path.resolve(scripts, env.T3_SCENARIO_APP_DIR ?? ".."),
3232
home,
3333
startApp: path.join(scripts, "scenarios/start-app.ts"),
34+
// e2e gives the app only PATH, HOME and the temp directory. A scenario that needs more names
35+
// it here: exact names or prefixes ending in `*`, separated by commas.
36+
inheritEnv: (env.T3_SCENARIO_INHERIT_ENV ?? "")
37+
.split(",")
38+
.map((name) => name.trim())
39+
.filter((name) => name !== ""),
3440
mobile: {
3541
host: host || "127.0.0.1",
3642
// A mobile target has no `app.url`, so e2e cannot pick these ports.
@@ -143,6 +149,7 @@ export default {
143149
executable: resolved.node,
144150
args: [startApp, "web", T3_HOME, LOOPBACK, "{port}"],
145151
cwd: APP_DIR,
152+
inheritEnv: resolved.inheritEnv,
146153
startupTimeout: 300_000,
147154
log: ".e2e/app.log",
148155
},
@@ -170,6 +177,7 @@ export default {
170177
MOBILE.host,
171178
],
172179
cwd: APP_DIR,
180+
inheritEnv: resolved.inheritEnv,
173181
startupTimeout: 300_000,
174182
log: ".e2e/mobile-app.log",
175183
},

‎scripts/prove-scenario.test.ts‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,20 @@ describe("prove-scenario", () => {
9191
expect(proof.verdict).toMatch(row.verdict);
9292
});
9393

94+
it("tells a scenario that stopped from an app that did not start", () => {
95+
const side = (report: Parameters<typeof readSide>[2]) =>
96+
readSide("aaaaaaaaaaaa", "/out/base/results", report);
97+
const head = readSide("bbbbbbbbbbbb", "/out/head/results", passed);
98+
// A control the scenario did not find: the app ran, so the scenario is what needs fixing.
99+
const stopped = buildProof(side(brokenScenario), head, (path) => path);
100+
expect(stopped.couldNotRun).toBe("scenario");
101+
expect(stopped.verdict).toMatch(/fix the scenario/);
102+
const noApp = buildProof(side(undefined), head, (path) => path);
103+
expect(noApp.couldNotRun).toBe("app");
104+
expect(noApp.verdict).not.toMatch(/fix the scenario/);
105+
expect(buildProof(side(failed), head, (path) => path).couldNotRun).toBeNull();
106+
});
107+
94108
it("labels captures with their side and commit and keeps the pairing token out", () => {
95109
const proof = buildProof(
96110
readSide("aaaaaaaaaaaa", "/out/base/results", failed),

‎scripts/prove-scenario.ts‎

Lines changed: 74 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,11 @@ export interface SideResult {
5353
readonly status: "passed" | "failed" | "error";
5454
readonly title: string;
5555
readonly message?: string;
56+
/**
57+
* For `error`, what stopped: `app` when the app did not start or no test ran, `scenario` when
58+
* the app ran and the scenario stopped before its assertion, such as on a control it did not find.
59+
*/
60+
readonly stopped?: "app" | "scenario";
5661
readonly steps: ReadonlyArray<{ readonly step: string; readonly status: string }>;
5762
/** Capture paths under the run's results directory, in the order the scenario took them. */
5863
readonly captures: ReadonlyArray<string>;
@@ -75,7 +80,16 @@ export function readSide(
7580
executed.length > 1
7681
? `the scenario file ran ${executed.length} tests; a proof needs exactly one`
7782
: "e2e wrote no report: the app did not start, or no test was selected";
78-
return { sha, status: "error", title: "", message, steps: [], captures: [], videos: [] };
83+
return {
84+
sha,
85+
status: "error",
86+
stopped: "app",
87+
title: "",
88+
message,
89+
steps: [],
90+
captures: [],
91+
videos: [],
92+
};
7993
}
8094
const files = (kind: string) =>
8195
(attempt.artifacts ?? []).flatMap((artifact) =>
@@ -84,15 +98,18 @@ export function readSide(
8498
: [],
8599
);
86100
const error = attempt.error;
101+
// e2e exits 0 for a skipped test too, and a test that did not run proves nothing.
102+
const status =
103+
attempt.status === "passed"
104+
? "passed"
105+
: error?.code === "ASSERTION_FAILED"
106+
? "failed"
107+
: "error";
87108
return {
88109
sha,
89-
// e2e exits 0 for a skipped test too, and a test that did not run proves nothing.
90-
status:
91-
attempt.status === "passed"
92-
? "passed"
93-
: error?.code === "ASSERTION_FAILED"
94-
? "failed"
95-
: "error",
110+
status,
111+
// A test that ended with an error of its own got as far as the app.
112+
...(status === "error" ? { stopped: error === undefined ? "app" : "scenario" } : {}),
96113
title: result.titlePath.join(" > "),
97114
...(error === undefined
98115
? attempt.status === "passed"
@@ -123,7 +140,7 @@ export function buildProof(base: SideResult, head: SideResult, link: (path: stri
123140
const broken = base.status === "error" ? base : head.status === "error" ? head : undefined;
124141
const verdict =
125142
broken !== undefined
126-
? `Not proven: the scenario could not run on ${broken === base ? `base ${baseSha}` : `head ${headSha}`} (${broken.message ?? "unknown error"}).`
143+
? `Not proven: the scenario could not run on ${broken === base ? `base ${baseSha}` : `head ${headSha}`} (${broken.message ?? "unknown error"}).${broken.stopped === "scenario" ? " The app ran and the scenario stopped before its assertion: fix the scenario and run it again." : ""}`
127144
: base.status === "passed"
128145
? head.status === "passed"
129146
? `Not proven: the scenario passes on base ${baseSha} too, so it does not reproduce the bug.`
@@ -160,6 +177,8 @@ export function buildProof(base: SideResult, head: SideResult, link: (path: stri
160177
scenario: head.title || base.title,
161178
failsOnBase: base.status === "failed",
162179
passesOnHead: head.status === "passed",
180+
/** Why there is no verdict on the behavior: the app did not run, or the scenario stopped. */
181+
couldNotRun: broken?.stopped ?? null,
163182
baseSha: base.sha,
164183
headSha: head.sha,
165184
verdict,
@@ -192,10 +211,18 @@ const withCheckout = Effect.fn("prove.withCheckout")(function* <A, E, R>(
192211
if (clean && (yield* git(["rev-parse", "HEAD"])) === sha) return yield* use(repo);
193212
// Beside this checkout, not in the temp directory, so tooling resolves as it does here.
194213
const worktree = `${repo}-scenario-${short(sha)}`;
214+
// A run that was killed left its worktree behind, and git refuses to add over it.
215+
const discard = git(["worktree", "remove", "--force", worktree]).pipe(
216+
Effect.ignore,
217+
Effect.andThen(git(["worktree", "prune"])),
218+
Effect.ignore,
219+
);
195220
return yield* Effect.acquireUseRelease(
196-
git(["worktree", "add", "--quiet", "--detach", "--force", worktree, sha]),
221+
discard.pipe(
222+
Effect.andThen(git(["worktree", "add", "--quiet", "--detach", "--force", worktree, sha])),
223+
),
197224
() => run("vp", ["i"], { cwd: worktree, echo: true }).pipe(Effect.andThen(use(worktree))),
198-
() => git(["worktree", "remove", "--force", worktree]).pipe(Effect.ignore),
225+
() => discard,
199226
);
200227
});
201228

@@ -204,6 +231,7 @@ const runSide = Effect.fn("prove.runSide")(function* (
204231
side: "base" | "head",
205232
rev: string,
206233
scenarioPath: string,
234+
target: string,
207235
) {
208236
const fs = yield* FileSystem.FileSystem;
209237
const path = yield* Path.Path;
@@ -216,7 +244,7 @@ const runSide = Effect.fn("prove.runSide")(function* (
216244
// The scenario and e2e come from this checkout; only the app comes from `appDir`.
217245
run(
218246
path.join(scripts, "node_modules/.bin/e2e"),
219-
["run", scenarioPath, "--output", path.relative(scripts, outDir)],
247+
["run", scenarioPath, "--target", target, "--output", path.relative(scripts, outDir)],
220248
{
221249
cwd: scripts,
222250
env: { ...environment, T3_SCENARIO_APP_DIR: appDir, E2E_TELEMETRY_DISABLED: "1" },
@@ -230,7 +258,22 @@ const runSide = Effect.fn("prove.runSide")(function* (
230258
Effect.flatMap(decodeReport),
231259
Effect.orElseSucceed(() => undefined),
232260
);
233-
return readSide(sha, path.join(outDir, "results"), report);
261+
const result = readSide(sha, path.join(outDir, "results"), report);
262+
// Written as each side ends, so a proof whose second side is cut short still has its first.
263+
yield* fs
264+
.writeFileString(
265+
path.join(scripts, ".e2e/prove", `${side}.json`),
266+
`${yield* encodeProof(result)}\n`,
267+
)
268+
.pipe(Effect.ignore);
269+
return result;
270+
});
271+
272+
/** The one target a scenario runs on: an iOS scenario imports `t3-mobile.ts`, a web one does not. */
273+
const targetOf = Effect.fn("prove.targetOf")(function* (scenarioPath: string) {
274+
const fs = yield* FileSystem.FileSystem;
275+
const source = yield* fs.readFileString(scenarioPath).pipe(Effect.orElseSucceed(() => ""));
276+
return source.includes("t3-mobile") ? "ios" : "web";
234277
});
235278

236279
/** Uploads the captures and maps each to its URL; a failed upload leaves the map empty. */
@@ -262,6 +305,7 @@ const prove = Effect.fn("prove")(function* (input: {
262305
readonly base: string;
263306
readonly head: string;
264307
readonly uploadCommand: Option.Option<string>;
308+
readonly target: Option.Option<string>;
265309
}) {
266310
const fs = yield* FileSystem.FileSystem;
267311
const path = yield* Path.Path;
@@ -271,8 +315,10 @@ const prove = Effect.fn("prove")(function* (input: {
271315
// A proof left by an earlier run must not be read as this one's.
272316
yield* fs.remove(proveDir, { recursive: true, force: true });
273317
yield* fs.makeDirectory(proveDir, { recursive: true });
274-
const base = yield* runSide("base", input.base, scenarioPath);
275-
const head = yield* runSide("head", input.head, scenarioPath);
318+
// Only that target's app starts: a web scenario needs no Metro and no device.
319+
const target = Option.getOrElse(input.target, () => "") || (yield* targetOf(scenarioPath));
320+
const base = yield* runSide("base", input.base, scenarioPath, target);
321+
const head = yield* runSide("head", input.head, scenarioPath, target);
276322
const links = Option.isSome(input.uploadCommand)
277323
? yield* upload(input.uploadCommand.value, [...base.captures, ...head.captures])
278324
: new Map<string, string>();
@@ -285,8 +331,13 @@ const prove = Effect.fn("prove")(function* (input: {
285331
yield* fs.writeFileString(path.join(proveDir, "proof.md"), proof.markdown);
286332
yield* fs.writeFileString(path.join(proveDir, "proof.json"), `${json}\n`);
287333
yield* Console.log(json);
288-
const couldNotRun = base.status === "error" || head.status === "error";
289-
return proof.failsOnBase && proof.passesOnHead ? 0 : couldNotRun ? 2 : 1;
334+
return proof.failsOnBase && proof.passesOnHead
335+
? 0
336+
: proof.couldNotRun === "scenario"
337+
? 3
338+
: proof.couldNotRun === "app"
339+
? 2
340+
: 1;
290341
});
291342

292343
const command = Command.make(
@@ -304,6 +355,12 @@ const command = Command.make(
304355
Flag.withDescription("Command that takes capture paths and prints one URL per line."),
305356
Flag.optional,
306357
),
358+
target: Flag.String("target").pipe(
359+
Flag.withDescription(
360+
"Target to run on: web or ios. Default: ios when the scenario imports t3-mobile.ts, else web.",
361+
),
362+
Flag.optional,
363+
),
307364
},
308365
(input) =>
309366
prove(input).pipe(

‎scripts/scenarios/t3.ts‎

Lines changed: 33 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,13 @@
22
// app. `t3-mobile.ts` is the same for the iOS app.
33
import * as NodeServices from "@effect/platform-node/NodeServices";
44
import { test as base } from "@e2e-dev/web";
5+
import { expect } from "e2e";
56
import * as HostProcess from "@t3tools/shared/HostProcess";
67
import * as Effect from "effect/Effect";
78
import * as FileSystem from "effect/FileSystem";
89
import * as Path from "effect/Path";
910
import * as Schema from "effect/Schema";
11+
import { FetchHttpClient, HttpClient, HttpClientResponse } from "effect/http";
1012

1113
import { APP_DIR, T3_HOME } from "../e2e.config.ts";
1214
import { run } from "./shell.ts";
@@ -23,12 +25,19 @@ export interface T3State {
2325
readonly title: string;
2426
readonly messages: ReadonlyArray<string>;
2527
readonly model?: string;
26-
}) => Promise<void>;
28+
}) => Promise<T3Thread>;
29+
}
30+
31+
export interface T3Thread {
32+
readonly threadId: string;
2733
}
2834

2935
export interface T3 extends T3State {
30-
/** Opens the app, signed in, at `path`. */
31-
readonly open: (path?: string) => Promise<void>;
36+
/**
37+
* Opens the app, signed in, at a path or on a thread `t3.thread` returned. Opening the thread
38+
* by its link is the reliable way to reach it: a sidebar row covers its own title text.
39+
*/
40+
readonly open: (target?: string | T3Thread) => Promise<void>;
3241
}
3342

3443
/** e2e's fixtures are promise functions; this is the one place an Effect becomes a promise. */
@@ -99,6 +108,7 @@ export function makeState(home: string, env: NodeJS.ProcessEnv = {}) {
99108
const started = yield* turn(["start", input.project, first, "--title", input.title, ...replay]);
100109
const { threadId } = yield* decodeLaunched(started.stdout);
101110
for (const message of rest) yield* turn(["send", threadId, message]);
111+
return { threadId } satisfies T3Thread;
102112
});
103113

104114
return {
@@ -122,14 +132,31 @@ const readPairingPath = Effect.gen(function* () {
122132
return yield* fs.readFileString(path.join(T3_HOME, "pairing-path"));
123133
});
124134

135+
/** The id every thread link starts with, asked of the server the app talks to. */
136+
const environmentId = (baseUrl: string) =>
137+
HttpClient.get(`${baseUrl}/.well-known/t3/environment`).pipe(
138+
Effect.flatMap(HttpClientResponse.filterStatusOk),
139+
Effect.flatMap(
140+
HttpClientResponse.schemaBodyJson(Schema.Struct({ environmentId: Schema.String })),
141+
),
142+
Effect.map((environment) => environment.environmentId),
143+
Effect.provide(FetchHttpClient.layer),
144+
);
145+
125146
export const test = base.extend<{ t3: T3 }>({
126-
t3: async ({ app }, use) => {
147+
t3: async ({ app, browser }, use) => {
127148
const { state, cleanup } = makeState(T3_HOME);
128149
await use({
129150
...state,
130-
open: async (path = "/") => {
151+
open: async (target = "/") => {
131152
await app.open(await runPromise(readPairingPath));
132-
if (path !== "/") await app.open(path);
153+
// The pairing page signs in and then leaves itself. Going anywhere before it has done
154+
// so cancels the sign-in, and the next page is the pairing page again.
155+
await expect(browser).not.toHaveURL(/\/pair(?:[?#]|$)/, { timeout: 60_000 });
156+
if (typeof target !== "string") {
157+
const environment = await runPromise(environmentId(app.baseUrl ?? ""));
158+
await app.open(`/${environment}/${target.threadId}`);
159+
} else if (target !== "/") await app.open(target);
133160
},
134161
});
135162
await runPromise(cleanup);

0 commit comments

Comments
 (0)