Skip to content

Commit a84caa8

Browse files
authored
test: give the tap-retry differential a fixture control that forces the retry (#1327)
* test: give the tap-retry differential a fixture control that forces the retry tap-retry-if-no-change was parked in #1289 for being a coin flip: tapRetries measured 0 in run 29504440599 and 1 in 29510020718 with no change to the flow or commit. This re-adds it with a control that holds still, so the retry fires every run. The original diagnosis (a dynamic cart badge in the tapped title's subtree) had the right shape but the wrong scope. maestroSnapshotSignature hashes EVERY node on screen, not the tapped subtree, so no "static region" of the home screen could have worked. The actual coin flip is the gesture lab's remote image (reactnative.dev/img/logo-share.png): whether it lands before or after the tap decides whether the engine sees "changed" and skips the retry. So the fixture gets a dedicated inert surface — no state, effects, timers, images, or pressables — presented as a full-screen modal so iOS detaches the presenting screen and the tab bar's live badges leave the hierarchy too. On a real run it is 9 nodes against Settings' 55. Reached by a launcher on Settings via the Settings TAB, which is a deliberate choice twice over. A deep link would have kept the launcher out of every other screen's snapshot, but simctl openurl raises a SpringBoard "Open in app?" confirmation on iOS 26 even cold, and Maestro's openLink goes through the same path. And home's "Open settings" button sits below the fold, so reaching it needs scrollUntilVisible — the engine bug already waived under #1299. The flow carries no waitForAnimationToEnd: a navigating tap defers a stability requirement that the next tap settles before resolving its target, so the baseline signature is already captured on a settled screen. Adding one fails the flow outright, which is a real engine divergence filed as #1326. Verified on device (iPhone 17 Pro Max, iOS 26.2, Maestro 2.5.1): 10/10 consecutive differential runs ok, tapRetries [0,0,1] every run — the two navigating taps correctly do not retry, the inert tap retries exactly once. A single green run proves nothing here, which is the trap #1300 fell into. The parking guard in invariants.test.ts is replaced by three guards: the scenario stays active, carries its tapRetries invariant, and is never waived by a knownDivergence — a flaky scenario must be fixed, not declared. Fixes #1300 * chore: gitignore expo prebuild output in the test app Building the fixture app locally (what .github/actions/setup-fixture-app does in CI) runs expo prebuild and generates examples/test-app/ios/. It is generated and untracked but not ignored, so it shows up in git status and a `git commit -a` would sweep the whole native project in. Same for android/ when building there. Scoped to examples/test-app, so the repo-root android/ — which holds real tracked sources like android/ime-helper — is unaffected. Nothing is tracked under either path today, and the CI fixture-app cache key hashes src/**, app/**, modules/** and the config/lockfiles, so it does not reference these and is unaffected.
1 parent b2b7ddd commit a84caa8

9 files changed

Lines changed: 188 additions & 35 deletions

File tree

‎examples/test-app/.gitignore‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,8 @@
11
.expo/
22
node_modules/
3+
4+
# expo prebuild output, regenerated by `expo run:ios` / `expo run:android`
5+
# (what .github/actions/setup-fixture-app does). Untracked and generated, so
6+
# leaving them visible lets a `git commit -a` sweep the whole native project in.
7+
android/
8+
ios/

‎examples/test-app/app/(tabs)/settings.tsx‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ export default function SettingsRoute() {
1616
diagnosticsState={state.diagnosticsState}
1717
notificationsEnabled={state.notificationsEnabled}
1818
onOpenAccessorySetup={() => router.push('/accessory-setup')}
19+
onOpenInertSurface={() => router.push('/inert')}
1920
onConfirmReset={state.resetLabState}
2021
onLoadDiagnostics={state.loadDiagnostics}
2122
onRetryDiagnostics={state.retryDiagnostics}

‎examples/test-app/app/_layout.tsx‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ function RootLayoutContent() {
2424
>
2525
<Stack.Screen name="(tabs)" />
2626
<Stack.Screen name="accessory-setup" />
27+
<Stack.Screen name="inert" options={{ presentation: 'fullScreenModal' }} />
2728
<Stack.Screen name="product/[productId]" />
2829
</Stack>
2930
{toastMessage ? <ToastViewport message={toastMessage} /> : null}

‎examples/test-app/app/inert.tsx‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
import { AppFrame } from '../src/components';
2+
import { InertScreen } from '../src/screens/InertScreen';
3+
4+
// Presented as a fullScreenModal (see app/_layout.tsx) rather than pushed: iOS
5+
// detaches the presenting screen once a full-screen present settles, so Settings
6+
// and the tab bar — which carries live badges — leave the hierarchy entirely.
7+
// The retry signature hashes every node on screen, so that matters.
8+
export default function InertRoute() {
9+
return (
10+
<AppFrame>
11+
<InertScreen />
12+
</AppFrame>
13+
);
14+
}
Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
import { StyleSheet, Text, View } from 'react-native';
2+
3+
import { useAppColors, type AppColors } from '../theme';
4+
5+
/**
6+
* A surface on which a tap provably changes nothing.
7+
*
8+
* `retryTapIfNoChange` only re-taps when the post-tap snapshot signature equals
9+
* the pre-tap one, and that signature hashes EVERY node on screen — not just the
10+
* tapped subtree. So any live content anywhere on the screen suppresses the
11+
* retry, which is what made the layer-3 tap-retry scenario a coin flip on the
12+
* home screen (its gesture lab loads a remote image whose arrival lands on either
13+
* side of the tap). See https://github.com/callstack/agent-device/issues/1300.
14+
*
15+
* Everything here is therefore deliberately constrained, and each rule is load-
16+
* bearing for that scenario rather than stylistic:
17+
*
18+
* - no state, effects, timers, or async work — nothing to arrive after capture
19+
* - no images (a remote one loads whenever the network says so)
20+
* - no ScrollView — scroll offset moves node bounds, which are part of the hash
21+
* - nothing pressable, so a tap cannot mutate the screen even by accident
22+
*
23+
* Adding any of those back re-opens #1300. Put dynamic fixtures on another
24+
* screen; this one exists to hold still.
25+
*/
26+
export function InertScreen() {
27+
const colors = useAppColors();
28+
const styles = createStyles(colors);
29+
30+
return (
31+
<View style={styles.content}>
32+
<Text style={styles.title} testID="inert-title">
33+
Inert surface
34+
</Text>
35+
<Text style={styles.body} testID="inert-target">
36+
Tapping this text changes nothing on screen.
37+
</Text>
38+
<Text style={styles.footnote}>
39+
Nothing here reacts to touch, loads, or animates, so a tap leaves the
40+
hierarchy byte-identical and retryTapIfNoChange must re-tap.
41+
</Text>
42+
</View>
43+
);
44+
}
45+
46+
function createStyles(colors: AppColors) {
47+
return StyleSheet.create({
48+
body: {
49+
color: colors.text,
50+
fontSize: 17,
51+
fontWeight: '600',
52+
},
53+
content: {
54+
gap: 16,
55+
},
56+
footnote: {
57+
color: colors.textSoft,
58+
fontSize: 14,
59+
lineHeight: 21,
60+
},
61+
title: {
62+
color: colors.text,
63+
fontSize: 24,
64+
fontWeight: '700',
65+
},
66+
});
67+
}

‎examples/test-app/src/screens/SettingsScreen.tsx‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ export interface SettingsScreenProps {
1818
notificationsEnabled: boolean;
1919
reducedMotionEnabled: boolean;
2020
onOpenAccessorySetup: () => void;
21+
onOpenInertSurface: () => void;
2122
onLoadDiagnostics: () => void;
2223
onRetryDiagnostics: () => void;
2324
onSetNotificationsEnabled: (value: boolean) => void;
@@ -71,6 +72,17 @@ export function SettingsScreen(props: SettingsScreenProps) {
7172
/>
7273
</SectionCard>
7374

75+
<SectionCard
76+
subtitle="Open a surface where a tap provably changes nothing on screen."
77+
title="Inert surface"
78+
>
79+
<ActionButton
80+
label="Open inert surface"
81+
onPress={props.onOpenInertSurface}
82+
testID="open-inert-surface"
83+
/>
84+
</SectionCard>
85+
7486
<SectionCard subtitle="Simple switch rows for durable selectors." title="Preferences">
7587
<ToggleRow
7688
description="Disabled notifications should remain visible in plain snapshots."
Lines changed: 32 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,38 @@
1-
# Layer-3 device flow. Taps the app's static title, which is NOT interactive, so
2-
# the screen cannot change and the retryIfNoChange path is forced to run. A tap on
3-
# a navigating control (e.g. home-open-form) succeeds on the first attempt and
4-
# never exercises retry at all — the scenario would pass while proving nothing.
5-
# The scenario asserts tapRetries >= 1 from the trace, so a retry regression is
6-
# caught rather than assumed.
1+
# Layer-3 device flow for retryTapIfNoChange.
2+
#
3+
# The engine re-taps only when the post-tap snapshot signature equals the pre-tap
4+
# one, and that signature hashes EVERY node on screen — not just the tapped
5+
# subtree. So a non-interactive target is not enough on its own: any live content
6+
# anywhere on screen makes the engine see "changed" and skip the retry. Tapping
7+
# the home screen's title used to look sufficient and was actually a coin flip
8+
# (#1300) — its gesture lab loads a remote image whose arrival lands on either
9+
# side of the tap depending on the network.
10+
#
11+
# So this drives the fixture's inert surface (examples/test-app/app/inert.tsx),
12+
# a screen with nothing that loads, animates, or reacts to touch, presented as a
13+
# full-screen modal so the tab bar's live badges leave the hierarchy too. On a
14+
# real run the inert screen is 9 nodes against Settings' 55.
15+
#
16+
# Reaching it via the Settings TAB is deliberate: home's "Open settings" button
17+
# sits below the fold, and scrolling to it needs scrollUntilVisible — the engine
18+
# bug tracked in #1299. The tab is always on screen.
19+
#
20+
# There is deliberately no waitForAnimationToEnd before the tap. It is not needed
21+
# — a navigating tap defers a stability requirement that the next tap settles
22+
# before resolving its target, so the baseline signature is already captured on a
23+
# settled screen — and adding one currently FAILS the flow outright (#1326).
724
appId: com.callstack.agentdevicelab
825
---
926
- launchApp:
1027
clearState: true
11-
- assertVisible: Agent Device Tester
1228
- tapOn:
13-
text: Agent Device Tester
29+
text: Settings
30+
- tapOn:
31+
id: open-inert-surface
32+
- assertVisible:
33+
id: inert-title
34+
- tapOn:
35+
id: inert-target
1436
retryTapIfNoChange: true
15-
- assertVisible: Agent Device Tester
37+
- assertVisible:
38+
id: inert-title

‎scripts/maestro-conformance/differential/invariants.test.ts‎

Lines changed: 31 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -124,22 +124,40 @@ test('a trace whose taps record no metric reports no-data rather than passing',
124124
assert.equal(result.status, 'no-data');
125125
});
126126

127-
// tap-retry-if-no-change is parked (#1300): tapRetries measured 0 then 1 across
128-
// identical runs, so it is flaky rather than divergent. A knownDivergence
129-
// declaration assumes the failure reproduces, so declaring a flaky scenario
130-
// would make the schedule flip red at random. Assert it stays out of the active
131-
// set until it has an inert control — re-adding it without one silently returns
132-
// the coin-flip.
133-
test('the flaky retry scenario is parked, not declared as a divergence', () => {
127+
// tap-retry-if-no-change was parked while it was a coin flip (#1300): tapRetries
128+
// measured 0 then 1 across identical runs. It is active again now that it taps
129+
// the fixture's inert surface, and it carries its own detector — without the
130+
// tapRetries invariant the scenario is back to proving nothing, because outcome
131+
// parity passes whether or not the retry ever fires.
132+
test('the retry scenario is active and carries the invariant that makes it mean something', () => {
134133
const retry = DIFFERENTIAL_SCENARIOS.find((s) => s.id === 'tap-retry-if-no-change');
135-
assert.equal(retry, undefined, 'tap-retry-if-no-change must stay parked until #1300 gives it an inert control');
134+
assert.ok(retry, 'tap-retry-if-no-change must stay in the active differential set');
135+
const [invariant, ...rest] = retry?.engineInvariants ?? [];
136+
assert.equal(rest.length, 0);
137+
assert.equal(invariant?.kind, 'metricAtLeast');
138+
assert.equal(invariant?.command, 'tapOn');
139+
assert.equal(invariant?.kind === 'metricAtLeast' && invariant.metric, 'tapRetries');
140+
assert.equal(invariant?.kind === 'metricAtLeast' && invariant.min, 1);
141+
});
142+
143+
// A flaky scenario must be fixed, never waived: knownDivergence assumes the
144+
// failure reproduces, so declaring this one would make the schedule flip between
145+
// green and red at random — the trap #1300 was filed to escape.
146+
test('the retry scenario is not waived by a divergence declaration', () => {
147+
const retry = DIFFERENTIAL_SCENARIOS.find((s) => s.id === 'tap-retry-if-no-change');
148+
assert.equal(retry?.knownDivergence, undefined);
136149
});
137150

138-
// The invariant itself stays implemented and proven, so the fix PR only has to
139-
// re-add the scenario rather than rebuild the detector.
140-
test('the tapRetries invariant remains implemented for when the scenario returns', () => {
141-
assert.equal(evaluateInvariant([stopWithMetrics('tapOn', { tapRetries: 1 })], RETRY_INVARIANT).status, 'held');
142-
assert.equal(evaluateInvariant([stopWithMetrics('tapOn', { tapRetries: 0 })], RETRY_INVARIANT).status, 'violated');
151+
// The flow only forces the retry if it drives the inert surface. Tapping any
152+
// live screen is what made this flaky, and that regression is invisible on the
153+
// unit side — the invariant only goes red on a device.
154+
test('the retry flow drives the inert surface, not a live screen', () => {
155+
const flow = fs.readFileSync(
156+
path.join(import.meta.dirname, 'flows/tap-retry-if-no-change.yaml'),
157+
'utf8',
158+
);
159+
assert.match(flow, /id: open-inert-surface/);
160+
assert.match(flow, /id: inert-target/);
143161
});
144162

145163
// Truncation-vs-rounding is at most 1px and cannot be observed on a device, so

‎scripts/maestro-conformance/differential/scenarios.ts‎

Lines changed: 24 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -148,19 +148,30 @@ export const DIFFERENTIAL_SCENARIOS: DifferentialScenario[] = [
148148
],
149149
divergenceMeans: 'The swipe behaves differently enough on one engine to fail the flow.',
150150
},
151-
// PARKED: tap-retry-if-no-change (flow kept at differential/flows/, invariant
152-
// kept implemented and unit-tested) — https://github.com/callstack/agent-device/issues/1300
153-
//
154-
// It is NOT declared as a knownDivergence, deliberately. tapRetries measured 0
155-
// in run 29504440599 and 1 in run 29510020718 with no change to the flow or
156-
// commit: the tap sometimes holds the hierarchy signature still and sometimes
157-
// does not, because the fixture home screen has live content. A declaration
158-
// assumes the divergence REPRODUCES; a flaky one would flip between
159-
// known-divergence (green) and stale-declaration (red) at random, which is
160-
// worse than no scenario — a coin-flip job teaches people to ignore the
161-
// differential. So retryIfNoChange has no device coverage until the fixture
162-
// offers an inert control, and that absence is tracked rather than disguised
163-
// as a green run.
151+
{
152+
id: 'tap-retry-if-no-change',
153+
flow: 'differential/flows/tap-retry-if-no-change.yaml',
154+
comparesAcrossEngines: 'A tap on an inert target completes on both engines.',
155+
expect: 'pass',
156+
// The whole point of the scenario. Outcome parity cannot see a retry: a tap
157+
// that never re-taps passes just the same, which is how this scenario spent
158+
// its first two runs proving nothing (#1300). The flow taps a surface that
159+
// provably cannot change, so the retry MUST fire; if it does not, either the
160+
// retry path regressed or the fixture stopped being inert, and both are
161+
// things we want to hear about.
162+
engineInvariants: [
163+
{
164+
kind: 'metricAtLeast',
165+
command: 'tapOn',
166+
metric: 'tapRetries',
167+
min: 1,
168+
because:
169+
'a tap on an unchanging screen must re-tap; zero retries means retryIfNoChange never ran and the scenario proved nothing',
170+
},
171+
],
172+
divergenceMeans:
173+
'agent-device stopped re-tapping when the screen holds still (a retryIfNoChange regression), or the fixture surface it taps is no longer inert.',
174+
},
164175
{
165176
id: 'optional-warned-not-failed',
166177
flow: 'differential/flows/optional-warned-not-failed.yaml',

0 commit comments

Comments
 (0)