Repository navigation
scrollUntilVisible: agent-device fails to find an element real Maestro finds (layer-3 differential) #1299
Description
Found by the layer-3 conformance differential on its first working run (#1289) — real Maestro and agent-device executing the same flow on the same simulator, same app, same pinned Maestro 2.5.1.
The divergence
Flow: scripts/maestro-conformance/differential/flows/settle-after-tap.yaml, against the fixture app com.callstack.agentdevicelab:
- launchApp:
clearState: true
- assertVisible: Agent Device Tester
- scrollUntilVisible:
element:
id: home-open-form
- tapOn:
id: home-open-form| Engine | Outcome |
|---|---|
| Maestro 2.5.1 | pass (exit 0) — scrolls, finds home-open-form, taps it |
| agent-device | fail (exit 1) |
Replay failed at step 3 (scrollUntilVisible "home-open-form"):
Maestro scrollUntilVisible target did not become visible.
(settle-after-tap.yaml:line 11)
home-open-form is a real control on the fixture app's home screen (HomeScreen.tsx:89), inside a ScrollView, below the fold — the app's own helpers/open-checkout-form.yaml scrolls it into view the same way. Maestro reaches it; we time out.
This is engine correctness in an advertised command (scroll and scrollUntilVisible is in the supported subset), not a documented deviation.
Evidence
Run: https://github.com/callstack/agent-device/actions/runs/29504440599 (differential-report.json artifact attached to the run)
{
"id": "settle-after-tap",
"maestro": { "outcome": "pass", "exitCode": 0 },
"agentDevice": { "outcome": "fail", "exitCode": 1 },
"outcomeDiverged": true
}The other three scenarios in the same run rule out environment as the cause — percent-swipe and optional-warned-not-failed were pass/pass on both engines.
Leads (untested hypotheses, ranked)
visibilityPercentage === 100boundary — first bet. Our preset isscrollUntilVisiblePercentage: 100(compatibility-policy.ts). If upstream's visibility threshold accepts partial visibility at the default while we demand the element be fully visible, Maestro finds it at ~97% visible and we keep scrolling until the timeout — exactly the observed shape (a timeout, not a miss).- speed→duration mapping. Upstream
trunc(1000·(100−speed)/100)+1(mirrored inmaestroScrollDurationFromSpeed) vs our scroll-step geometry — a step size/direction mismatch could overscroll past the target. - Settle interplay after each scroll step — if we settle differently between steps, the visibility check may sample at the wrong moment.
Reproducing locally (do NOT iterate via CI)
The differential runner takes --only, so this loops in minutes on a local sim rather than 40-minute CI cycles:
pnpm test-app:install && pnpm --dir examples/test-app exec expo run:ios --configuration Release
pnpm maestro:conformance:differential -- --platform ios --only settle-after-tap --trace-root .agent-deviceAcceptance
The oracle itself is the acceptance test. The scenario is currently marked knownDivergence in differential/scenarios.ts with this issue as its tracking link, so the scheduled differential stays green while it is a known, declared gap. The fix PR removes the knownDivergence declaration — the differential then enforces that it stays fixed, and the runner fails if a declared divergence starts passing, so a stale declaration cannot linger.
Note this also currently blocks bug class 4's device detector: settle-after-tap is the settle-ordering scenario, and it cannot reach its tapOn step while scrollUntilVisible fails.
Activity
feat: declare layer-3 divergences and schedule the differential
fix: park the flaky retry scenario instead of declaring it a divergence
fix: make a knownDivergence waiver cover exactly one failure, not any…
thymikee commentedon Jul 16, 2026
Live evidence from the differential, and it redirects the investigation. My original ranked hypotheses were partly wrong — recording that before someone spends time on them.
Both engines, same flow, same simulator (iPhone 17 Pro, iOS 26.2)
Real Maestro completed the scroll with the same parameters we use:
agent-device, same flow:
This weakens hypothesis 1 (the
visibilityPercentage === 100boundary)I ranked that first, reasoning that Maestro might accept partial visibility where we demand 100%. Maestro's own log says it ran at
visibility percentage 100%, speed 40, timeout 20000 ms and still found the element. Same knobs, same values, different outcome — so the divergence is unlikely to be the threshold itself. Worth re-checking whether our 100% check means the same thing as upstream's (element bounds fully inside the viewport vs. inside the scroll container, say), but it is no longer the obvious first suspect.Stronger lead: which container are we scrolling?
From our failure diagnostics on the same screen:
There is a horizontal scroll bar and a tab bar alongside the vertical
[scroll-area]. If our scroll gesture resolves to the wrong scrollable (the horizontal one, or the tab bar's), we would swipe repeatedly, nothing would move vertically, and the target would never appear — which matches the observed shape exactly: a clean 20s timeout rather than an overshoot or a miss.Suggested first checks, in order:
[scroll-area]? Compare against the container Maestro drives.Note the app screen is unremarkable — a plain
ScrollView(HomeScreen.tsx:45) with a title and four buttons. If we cannot scroll this reliably, the bug is likely to affect ordinary RN apps, not an exotic layout.Artifacts on the run (
.agent-device/test-artifacts/…) include the full hierarchy capture andreplay-timing.ndjson.chore(ci): remove the temporary branch push trigger
test: replace the hand-typed Maestro fixture with a generated conform…
test: give the tap-retry differential a fixture control that forces t…
refactor(test-app): share one route-param parser and keep the heavy f…
feat(test-app): add a Bluesky-depth heavy feed screen (#3884)