Skip to content

Commit 4d064af

Browse files
feat(scripts): iOS scenarios on a pooled simulator
e2e's mobile engine leases an iPhone from the Simlock pool through agent-device's lease API. A mobile `t3` fixture seeds state like the web one, installs the dev client for the checkout's native code, loads it from Metro and pairs it. Scenarios name their platform, so a web-only run leases nothing. Co-Authored-By: Claude Fable 5.2 <[email protected]>
1 parent 9215dba commit 4d064af

12 files changed

Lines changed: 523 additions & 159 deletions

‎docs/operations/scenarios.md‎

Lines changed: 65 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,8 @@
11
# Scenarios
22

33
A scenario is an [e2e](https://e2e.com/docs) test that shows one behavior in the real app: a real
4-
server on its own T3 home, real HTTP and WebSocket, and a real browser on the real web client. The
4+
server on its own T3 home, real HTTP and WebSocket, and a real browser on the web client or a
5+
pooled iPhone simulator running the mobile app. The
56
dev replay agent stands in for the model, so a run is free and repeatable. Use one as the proof
67
for a bug fix: it fails on the base commit and passes on the fix.
78

@@ -13,27 +14,26 @@ runner; its docs ship in `scripts/node_modules/e2e/docs` and `e2e guide` prints
1314
```ts
1415
import { expect } from "e2e";
1516

16-
import { test } from "./t3.ts";
17-
18-
test("Tab moves from the add-project Git URL field to its submit button", async ({
19-
t3,
20-
app,
21-
screen,
22-
browser,
23-
}) => {
24-
await t3.project("demo");
25-
await t3.open();
26-
27-
await screen.getByRole("button", "Add project").tap();
28-
await screen.getByText("Git URL").tap();
29-
await screen.getByRole("combobox").fill("https://github.com/pingdotgg/t3code.git");
30-
const submit = screen.getByRole("button", "Continue (Enter)");
31-
await expect(submit).toBeEnabled();
32-
33-
await browser.keyboard.press("Tab");
34-
await app.screenshot("after-tab");
35-
await expect(submit).toBeFocused();
36-
});
17+
import { test, WEB } from "./t3.ts";
18+
19+
test(
20+
"Tab moves from the add-project Git URL field to its submit button",
21+
WEB,
22+
async ({ t3, app, screen, browser }) => {
23+
await t3.project("demo");
24+
await t3.open();
25+
26+
await screen.getByRole("button", "Add project").tap();
27+
await screen.getByText("Git URL").tap();
28+
await screen.getByRole("combobox").fill("https://github.com/pingdotgg/t3code.git");
29+
const submit = screen.getByRole("button", "Continue (Enter)");
30+
await expect(submit).toBeEnabled();
31+
32+
await browser.keyboard.press("Tab");
33+
await app.screenshot("after-tab");
34+
await expect(submit).toBeFocused();
35+
},
36+
);
3737
```
3838

3939
`screen`, `browser`, `app`, and `expect` are e2e's own. [`t3.ts`](../../scripts/scenarios/t3.ts)
@@ -94,14 +94,50 @@ test: a temporary worktree beside your checkout, installed with `vp i` and remov
9494
commit that is your clean `HEAD` runs in place. So a new scenario can prove a fix against a base
9595
that predates it.
9696

97-
## Other tiers
97+
## iOS
98+
99+
An iOS scenario imports from [`t3-mobile.ts`](../../scripts/scenarios/t3-mobile.ts) and passes
100+
`IOS` where a web one passes `WEB`:
101+
102+
```ts
103+
import { expect } from "e2e";
98104

99-
Only the web client with the replay agent runs today.
105+
import { IOS, test } from "./t3-mobile.ts";
106+
107+
test(
108+
"The iOS app lists a seeded thread and opens its history",
109+
IOS,
110+
async ({ t3, app, screen }) => {
111+
const project = await t3.project("demo");
112+
await t3.thread({ project, title: "Earlier work", messages: ["first", "second"] });
113+
await t3.open();
114+
115+
const thread = screen.getByText("Earlier work", { exact: false }).first();
116+
await expect(thread).toBeVisible({ timeout: 120_000 });
117+
await thread.tap();
118+
await expect(screen.getByText("second fixture turn complete", { exact: false })).toBeVisible();
119+
},
120+
);
121+
```
122+
123+
`t3.project` and `t3.thread` are the same. `t3.open()` installs the dev client for the checkout's
124+
native fingerprint, loads it from Metro, and pairs it with the scenario's server. `device` is
125+
e2e's [device fixture](https://e2e.com/docs/mobile#the-device-fixture) for system actions.
126+
127+
The run leases an iPhone 18 Pro from the Simlock pool, so it needs `SIMLOCK_AGENT_DEVICE_URL`
128+
and `AGENT_DEVICE_DAEMON_AUTH_TOKEN`, as `mobile:e2e` does, and a gateway on Simlock 0.7.0 or
129+
later. A web-only run leases nothing. The device reaches this machine over the tailnet on ports
130+
13990 (server) and 8190 (Metro); set `T3_SCENARIO_SERVER_PORT` and `T3_SCENARIO_METRO_PORT` when
131+
another run on the machine holds them. An iOS run takes two to three minutes, most of it the
132+
device and the first bundle. `e2e run --target web` or `--target ios` runs one side.
133+
134+
`scenarios:prove` works the same for an iOS scenario. Each commit gets the dev client built for
135+
its own native code.
136+
137+
## Other tiers
100138

101-
- **Mobile.** e2e's `@e2e-dev/mobile` engine drives iOS and Android through agent-device, which
102-
this repository already uses. A mobile target in
103-
[`e2e.config.ts`](../../scripts/e2e.config.ts) and a `t3` fixture that pairs the device would
104-
run scenarios written with `screen` locators unchanged. Tests that use `browser` are web-only.
139+
- **Android.** Add a target with `mobile({ platform: "android", device: simlock("Pixel 9") })`
140+
and open the dev client with the Android form of the Expo link. Not built.
105141
- **Live provider.** Point the default model in `start-app.ts` at a real provider's cheapest
106142
model. Assertions on exact reply text would not hold.
107143

@@ -114,7 +150,7 @@ result as a pull request comment.
114150

115151
## What a scenario cannot cover
116152

117-
- Desktop shell behavior (menus, windows, IPC), and the mobile app until it has a target.
153+
- Desktop shell behavior (menus, windows, IPC), and Android.
118154
- Provider-specific behavior. The replay agent speaks generic ACP.
119155
- State no command can create, such as a run stopped by a usage limit.
120156
- Timing. Replay streaming is simulated, so a scenario cannot show a performance regression.

‎knip.jsonc‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@
4141
"smoke-desktop-app.ts",
4242
"verify-windows-desktop-artifact.ts",
4343
],
44-
"ignoreDependencies": ["agent-device", "esbuild"],
44+
"ignoreDependencies": ["esbuild"],
4545
// The `t3` npm package's files: copied into the package, never imported.
4646
// knip-schemas.test.ts writes a throwaway project to .knip-test-*, which a
4747
// knip run beside it in check:changed must not report.

‎pnpm-lock.yaml‎

Lines changed: 25 additions & 5 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎scripts/e2e.config.ts‎

Lines changed: 115 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,37 +1,148 @@
1-
// @effect-diagnostics nodeBuiltinImport:off
1+
// @effect-diagnostics nodeBuiltinImport:off globalTimers:off
22
// Scenarios: e2e tests that run against the real dev stack. See docs/operations/scenarios.md.
3+
import * as NodeChildProcess from "node:child_process";
4+
import * as NodeCrypto from "node:crypto";
35
import * as NodePath from "node:path";
6+
import * as NodeTimersPromises from "node:timers/promises";
47

8+
import { mobile, type DeviceProvider } from "@e2e-dev/mobile";
59
import { web } from "@e2e-dev/web";
10+
import { createAgentDeviceClient } from "agent-device";
611
import type { E2EConfig } from "e2e";
712

813
/** The checkout whose app runs. `prove-scenario.ts` points it at a base or head worktree. */
914
export const APP_DIR = NodePath.resolve(process.env.T3_SCENARIO_APP_DIR ?? "..");
10-
/** A T3 home only scenarios use, recreated on every run. `.t3` is gitignored. */
15+
/** T3 homes only scenarios use, recreated on every run. `.t3` is gitignored. */
1116
export const T3_HOME = NodePath.resolve(import.meta.dirname, "../.t3/scenarios");
17+
export const MOBILE_T3_HOME = `${T3_HOME}-mobile`;
18+
19+
/**
20+
* A pooled device reaches this machine over the tailnet, not localhost. A mobile target has no
21+
* `app.url`, so e2e cannot pick these ports; override them when another run holds them.
22+
*/
23+
export const MOBILE = {
24+
host:
25+
NodeChildProcess.spawnSync("tailscale", ["ip", "-4"], { encoding: "utf8" })
26+
.stdout?.trim()
27+
.split("\n")[0] || "127.0.0.1",
28+
serverPort: Number(process.env.T3_SCENARIO_SERVER_PORT ?? 13_990),
29+
metroPort: Number(process.env.T3_SCENARIO_METRO_PORT ?? 8_190),
30+
};
31+
32+
const startApp = NodePath.join(import.meta.dirname, "scenarios/start-app.ts");
33+
34+
/**
35+
* Leases a device from the Simlock pool, whose gateway speaks agent-device's lease protocol.
36+
* `retainOnClose` keeps the device when the engine closes its session between attempts.
37+
*/
38+
const simlock = (device: string): DeviceProvider => {
39+
const connect = (env: Readonly<Record<string, string | undefined>>) => {
40+
const daemonBaseUrl = env.SIMLOCK_AGENT_DEVICE_URL;
41+
const daemonAuthToken = env.AGENT_DEVICE_DAEMON_AUTH_TOKEN;
42+
if (daemonBaseUrl === undefined || daemonAuthToken === undefined) {
43+
throw new Error("SIMLOCK_AGENT_DEVICE_URL and AGENT_DEVICE_DAEMON_AUTH_TOKEN are not set");
44+
}
45+
return {
46+
daemon: { baseUrl: daemonBaseUrl, authToken: daemonAuthToken },
47+
client: createAgentDeviceClient({ daemonBaseUrl, daemonAuthToken }),
48+
};
49+
};
50+
return {
51+
name: "simlock",
52+
acquire: async (request) => {
53+
const { daemon, client } = connect(request.env);
54+
const scope = { tenant: "t3-scenarios", runId: `${request.runId}-${request.slot}` };
55+
const allocate = () =>
56+
client.leases.allocate({
57+
...scope,
58+
platform: request.platform === "android" ? "android" : "ios",
59+
device,
60+
retainOnClose: true,
61+
// Simlock takes the lease id from the caller.
62+
leaseId: NodeCrypto.randomBytes(16).toString("hex"),
63+
});
64+
// A busy gateway answers its health probe late, which the client reports as unavailable.
65+
let lease = await allocate().catch(() => undefined);
66+
for (let attempt = 0; lease === undefined && attempt < 12; attempt++) {
67+
request.log(`the pool is busy; asking again in 10s`);
68+
await NodeTimersPromises.setTimeout(10_000, undefined, { signal: request.signal });
69+
lease = await (attempt === 11 ? allocate() : allocate().catch(() => undefined));
70+
}
71+
if (lease === undefined) throw new Error(`The pool leased no ${device}`);
72+
const leaseScope = { ...scope, leaseId: lease.leaseId, leaseBackend: lease.backend };
73+
// The pool answers `boot` with the leased device; a worker hosts several, so name ours.
74+
const platform = request.platform === "android" ? "android" : "ios";
75+
const leased = await createAgentDeviceClient({
76+
daemonBaseUrl: daemon.baseUrl,
77+
daemonAuthToken: daemon.authToken,
78+
...leaseScope,
79+
}).devices.boot({ platform });
80+
return {
81+
id: lease.leaseId,
82+
daemon,
83+
deviceId: leased.id,
84+
client: {
85+
...leaseScope,
86+
...(lease.leaseProvider === undefined ? {} : { leaseProvider: lease.leaseProvider }),
87+
},
88+
};
89+
},
90+
release: async (lease, context) => {
91+
await connect(context.env).client.leases.release({ leaseId: lease.id });
92+
},
93+
};
94+
};
1295

1396
export default {
1497
tests: "scenarios/**/*.e2e.ts",
1598
targets: [
1699
{
100+
name: "web",
17101
engine: web({ viewport: { width: 1280, height: 800 } }),
18102
app: {
19103
url: "http://127.0.0.1:0",
20104
// Vite answers before the server does; this path is proxied to the server.
21105
readyUrl: "http://127.0.0.1:{port}/.well-known/t3/environment",
22106
command: {
23107
executable: process.execPath,
24-
args: [NodePath.join(import.meta.dirname, "scenarios/start-app.ts"), T3_HOME, "{port}"],
108+
args: [startApp, "web", T3_HOME, "{port}"],
25109
cwd: APP_DIR,
26110
startupTimeout: 300_000,
27111
log: ".e2e/app.log",
28112
},
29113
},
30114
},
115+
{
116+
name: "ios",
117+
engine: mobile({
118+
platform: "ios",
119+
device: simlock("iPhone 18 Pro"),
120+
session: "t3-scenarios",
121+
}),
122+
app: {
123+
bundleId: "com.t3tools.t3code.dev",
124+
// Metro starts after the server listens, so it answering means both are up.
125+
readyUrl: `http://127.0.0.1:${MOBILE.metroPort}/status`,
126+
command: {
127+
executable: process.execPath,
128+
args: [
129+
startApp,
130+
"mobile",
131+
MOBILE_T3_HOME,
132+
String(MOBILE.serverPort),
133+
String(MOBILE.metroPort),
134+
MOBILE.host,
135+
],
136+
cwd: APP_DIR,
137+
startupTimeout: 300_000,
138+
log: ".e2e/mobile-app.log",
139+
},
140+
},
141+
},
31142
],
32143
// One server holds every test's state, and a dev stack is heavy.
33144
workers: 1,
34-
timeout: 120_000,
145+
timeout: 300_000,
35146
assertionTimeout: 15_000,
36147
video: "on",
37148
trace: "on",

‎scripts/package.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,14 +8,15 @@
88
"test": "vp test run --config ../vite.config.ts --dir ."
99
},
1010
"dependencies": {
11+
"@e2e-dev/mobile": "0.12.6",
1112
"@e2e-dev/web": "0.15.5",
1213
"@effect/platform-node": "catalog:",
1314
"@electron/asar": "^4.3.0",
1415
"@electron/osx-sign": "2.7.0",
1516
"@t3tools/contracts": "workspace:*",
1617
"@t3tools/shared": "workspace:*",
1718
"@t3tools/tailscale": "workspace:*",
18-
"agent-device": "0.21.32",
19+
"agent-device": "0.21.36",
1920
"e2e": "0.23.0",
2021
"effect": "catalog:",
2122
"playwright-core": "1.64.1",

0 commit comments

Comments
 (0)