Skip to content

Commit cfbbdb0

Browse files
committed
fix(#2682): make the activation lane manual and record why the CI simulator cannot run it
The lane was added to the iOS Smoke and replays-manual lists to satisfy "assert the field on a real device response", but on that host the runner reads its own target as `.runningForeground` while a foreign app is demonstrably on screen — the uploaded artifact pairs a rendered app screenshot with a Settings screenshot and 31 captures of the bound app in ~450 ms each, no repair. The runner is right to activate nothing there, so the assertion measures `XCUIApplication.state` on a headless simulator instead of this feature. Filed as #2696. The file stays, env-gated, with the reason at the lane and the command to run it, and the handoff goes back through Safari via `simctl openurl`: launching Settings directly changes the screen — the screenshot guard passes — yet the runner then reports its target as foreground for the whole polling window, which is #2696's divergence wearing a different hat. Verified against a freshly built daemon: first capture after the handoff carries `stale_target` / `runningBackground` / the foreign pid, the next one carries neither.
1 parent 1183b7c commit cfbbdb0

4 files changed

Lines changed: 49 additions & 24 deletions

File tree

‎.device-evidence/CHECKLIST.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,3 +76,18 @@ actually called. Assert the pair on the same device run:
7676
`press <x> <y>` and `press @ref` answered from a live ref frame consume no capture, so they carry no
7777
disclosure even though the runner may have re-activated the app to serve them. Treat a silent
7878
interaction as "unknown", never as "no repair". Follow-up: callstack/agent-device#2694.
79+
80+
## Why the live activation lane is manual
81+
82+
`test/integration/smoke-ios-target-activation.test.ts` is env-gated and deliberately absent from the
83+
`ios.yml` / `replays-manual.yml` test lists. On GitHub-hosted simulators the runner reads its own
84+
target as `.runningForeground` while a foreign app is provably on screen (uploaded artifact of
85+
https://github.com/callstack/agent-device/actions/runs/35460535722/job/105943468492: rendered app
86+
screenshot, Settings screenshot, 31 captures in ~450 ms each, no repair). The runner is then correct
87+
to activate nothing, so a CI assertion on the disclosure measures `XCUIApplication.state` on that
88+
host rather than this feature. Tracked as callstack/agent-device#2696.
89+
90+
Manual evidence stands in for it, and both are reproducible:
91+
- local simulator, first capture after the handoff: `{"reason":"stale_target","priorState":"runningBackground","otherActiveApplicationPid":33878}`
92+
with the shared sentence appended to `warnings`; the following capture reports neither.
93+
- the physical device sequence below, which the coordinator runs.

‎.github/workflows/ios.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -289,7 +289,7 @@ jobs:
289289
AGENT_DEVICE_IOS_UDID: ${{ steps.ios-simulator.outputs.simulator-udid }}
290290
run: |
291291
pnpm gate build
292-
node --experimental-strip-types scripts/node-test-tmpdir.ts --test --test-concurrency=1 test/integration/smoke-ios-simulator-coverage.test.ts test/integration/smoke-ios-simulator.test.ts test/integration/smoke-ios-target-activation.test.ts
292+
node --experimental-strip-types scripts/node-test-tmpdir.ts --test test/integration/smoke-ios-simulator-coverage.test.ts test/integration/smoke-ios-simulator.test.ts
293293
294294
# #1584: isolated, cheap automatic guard for the #1562 class of regression (iOS silently
295295
# ignoring a `gesture pan` duration). Split out of gesture-lab.ad, which stays full-tier

‎.github/workflows/replays-manual.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -170,7 +170,7 @@ jobs:
170170
AGENT_DEVICE_IOS_UDID: ${{ steps.ios-simulator.outputs.simulator-udid }}
171171
run: |
172172
pnpm gate build
173-
node --experimental-strip-types scripts/node-test-tmpdir.ts --test --test-concurrency=1 test/integration/smoke-ios-simulator-coverage.test.ts test/integration/smoke-ios-simulator.test.ts test/integration/smoke-ios-target-activation.test.ts
173+
node --experimental-strip-types scripts/node-test-tmpdir.ts --test test/integration/smoke-ios-simulator-coverage.test.ts test/integration/smoke-ios-simulator.test.ts
174174
175175
- name: Run iOS physical device replay suite
176176
if: env.IOS_UDID != ''

‎test/integration/smoke-ios-target-activation.test.ts‎

Lines changed: 32 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -18,18 +18,29 @@ import {
1818
* actually activated the bound app, so this drives the real runner on a real simulator — a mocked
1919
* runner response proves the decoder, not the fact.
2020
*
21-
* Runs on the same lanes as the iOS simulator fixture E2E (`smoke` and `full` tiers) and needs the
22-
* same environment: `AGENT_DEVICE_IOS_E2E=1`, `AGENT_DEVICE_IOS_E2E_TIER`, `AGENT_DEVICE_FIXTURE_APP_PATH`
23-
* and `AGENT_DEVICE_FIXTURE_APP_ID`, and `AGENT_DEVICE_IOS_UDID`. Those lanes fetch the fixture with
24-
* `install: 'false'`, so this lane installs it through the public CLI like the other fixture lanes do.
21+
* MANUAL, ENV-GATED LANE. It is deliberately absent from the `ios.yml` and `replays-manual.yml` test
22+
* lists: on the CI-hosted simulator the runner reads its own target as `.runningForeground` while a
23+
* foreign app is demonstrably on screen, so that host owes no repair and no disclosure
24+
* ([#2696](https://github.com/callstack/agent-device/issues/2696)). Running it there measures
25+
* `XCUIApplication.state` on a headless simulator, not this feature.
26+
*
27+
* Run it against a local simulator or a physical device:
28+
*
29+
* AGENT_DEVICE_IOS_E2E=1 AGENT_DEVICE_IOS_E2E_TIER=smoke \
30+
* AGENT_DEVICE_FIXTURE_APP_PATH=<fixture .app> AGENT_DEVICE_FIXTURE_APP_ID=com.callstack.agentdevicelab \
31+
* AGENT_DEVICE_IOS_UDID=<udid> node --experimental-strip-types scripts/node-test-tmpdir.ts --test \
32+
* test/integration/smoke-ios-target-activation.test.ts
33+
*
34+
* It installs the fixture through the public CLI, so it needs no pre-installed app.
2535
*/
2636

2737
const enabled = process.env.AGENT_DEVICE_IOS_E2E === '1';
28-
// Settings, not a web link: `simctl openurl https://…` hands off through Safari, and on a CI
29-
// simulator that handoff can complete without Safari ever taking the foreground, which leaves this
30-
// lane waiting for a repair that was never owed. `simctl launch com.apple.Preferences` is the
31-
// handoff the iOS replays already rely on for the same effect.
32-
const HANDOFF_BUNDLE_ID = 'com.apple.Preferences';
38+
// Safari, through LaunchServices, the way an in-app external link does it. Launching Settings
39+
// directly (`simctl launch com.apple.Preferences`) also changes the screen, and the screenshot guard
40+
// below would still pass, but on the local simulator the runner then reads its own target as
41+
// foreground for the whole polling window — the same state divergence recorded in #2696, which this
42+
// lane exists to avoid mistaking for a missing disclosure.
43+
const HANDOFF_URL = 'https://example.com';
3344
const HANDOFF_DEADLINE_MS = 90_000;
3445
const HANDOFF_POLL_MS = 2_000;
3546

@@ -45,11 +56,14 @@ test(
4556
// The handoff only means something once the session app is on screen. `open` returns while a
4657
// freshly installed app is still drawing — CI's before-handoff screenshot was a blank status
4758
// bar — and handing off from there asks the runner about an app that never reached foreground,
48-
// which is not the question this lane asks. The launch surface is the app's home screen, so the
49-
// gate is a card on it — `Automation lab` is a surface the fixture E2E has to navigate to first.
59+
// which is not the question this lane asks. The gate is a string the app draws on any of its
60+
// own surfaces — the release home screen and a development build's server menu both carry it —
61+
// because which screen the fixture draws is not what this lane is about, and a lane that only
62+
// passes on one build variant could not be run manually against a dev build at all.
5063
await runStep(context, 'wait for the fixture to render', [
5164
'wait',
52-
'label="Gesture lab"',
65+
'text',
66+
'Agent Device Tester',
5367
'30000',
5468
]);
5569

@@ -61,22 +75,18 @@ test(
6175

6276
const foreign = path.join(context.artifactDir, 'target-activation-foreign.png');
6377

64-
// Hand off to an app the session is not bound to. Settings comes forward and the fixture app
65-
// goes background, which is the state the next capture has to repair.
66-
const handoff = await runCmd('xcrun', ['simctl', 'launch', context.udid, HANDOFF_BUNDLE_ID]);
67-
assert.equal(
68-
handoff.exitCode,
69-
0,
70-
`simctl launch ${HANDOFF_BUNDLE_ID} failed: ${handoff.stderr}`,
71-
);
78+
// Hand off to Safari: the session stays bound to the fixture app while another app takes the
79+
// screen, which is the state the next capture has to repair.
80+
const handoff = await runCmd('xcrun', ['simctl', 'openurl', context.udid, HANDOFF_URL]);
81+
assert.equal(handoff.exitCode, 0, `simctl openurl ${HANDOFF_URL} failed: ${handoff.stderr}`);
7282
await runStep(context, 'screenshot after handoff', ['screenshot', '--out', foreign]);
7383
assert.ok(await exists(foreign), `post-handoff screenshot wrote no artifact: ${foreign}`);
7484
// A screenshot identical to the pre-handoff one means nothing moved to the foreground and the
7585
// lane would be waiting for a repair the device is not obliged to report.
7686
assert.notEqual(
7787
await checksum(before),
7888
await checksum(foreign),
79-
'the screen did not change after launching Settings: no handoff to repair',
89+
'the screen did not change after the handoff: nothing came forward to repair',
8090
);
8191

8292
// Poll instead of sleeping a fixed window: the disclosure must arrive on the command that
@@ -159,7 +169,7 @@ async function captureUntilDisclosed(context: LiveContext) {
159169
}
160170
assert.fail(
161171
`no foreground disclosure within ${HANDOFF_DEADLINE_MS}ms of the handoff to ` +
162-
`${HANDOFF_BUNDLE_ID}: ${JSON.stringify(last?.json ?? null)}`,
172+
`${HANDOFF_URL}: ${JSON.stringify(last?.json ?? null)}`,
163173
);
164174
}
165175

0 commit comments

Comments
 (0)