Skip to content

Commit 69032ca

Browse files
strzhaoclaude
andcommitted
fix(interaction): register --expect-screenshot-diff in the guarantee matrix, settle the after-frame, and document the out-of-tree readback
Review fixes for the press/click --expect-screenshot-diff PR: - The guarantee matrix (ADR 0011) gains the screenshotDiffEvidence row: runtime cells on runtime-selector/runtime-ref/coordinate scoped to press/click, a delegated cell on native-ref, and inapplicable cells on the paths the flag cannot reach. The three waivers that named only --verify/--settle as the opt-in post-action evidence routes now name this flag too. Each enforced cell carries a contract scenario, and the shared harness replays a solid-color frame plan. - The after-frame waits 150ms past the dispatch reply before it is captured: the reaction the flag asks about can render frames after the transport round trip returns, and an immediate frame would read an effective tap as unchanged. The flag help text, field descriptions, and schema prose state the timing instead of leaving it blank. - commands.md names the flag where it teaches that a successful press is not proof of effect: a PIN keypad target is the case --verify cannot answer, since it compares empty trees. - Gate drift the PR introduced: the new capture-kit export joins the layering manifest baseline, and the flag descriptor gate moves to test/repo-guards/ (a repo-level guard on command descriptors, not a module's unit test — R84 test-colocation). Co-Authored-By: Claude Code <[email protected]>
1 parent 2ce0e86 commit 69032ca

18 files changed

Lines changed: 227 additions & 27 deletions

‎packages/command-registry/src/flag-definitions-action.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -212,7 +212,7 @@ export const ACTION_FLAG_DEFINITIONS: readonly FlagDefinition[] = [
212212
type: 'boolean',
213213
usageLabel: '--expect-screenshot-diff',
214214
usageDescription:
215-
'Press/click: compare lightweight screenshots taken right before and after the dispatch and report screenshotDiff in the response; use it when the target lives outside the accessibility tree (a PIN keypad) and --verify would compare empty trees. A matching pair adds a warning, because the dispatch may have had no visible effect',
215+
'Press/click: compare lightweight screenshots taken right before the dispatch and after it settles briefly, and report screenshotDiff in the response; use it when the target lives outside the accessibility tree (a PIN keypad) and --verify would compare empty trees. A matching pair adds a warning, because the dispatch may have had no visible effect',
216216
projectConfig: true,
217217
recorded: false,
218218
},

‎packages/command-runtime/src/commands/interaction/runtime/screenshot-diff-evidence.ts‎

Lines changed: 22 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ import { promises as fs } from 'node:fs';
22
import type { TemporaryFile } from '@agent-device/contracts/artifact-adapter';
33
import type { ScreenshotDiffEvidence } from '@agent-device/contracts/interaction';
44
import type { AgentDeviceRuntime, CommandContext } from '@agent-device/contracts/runtime-contract';
5-
import { toBackendContext } from '../../runtime-common.ts';
5+
import { sleep, toBackendContext } from '../../runtime-common.ts';
66
import { createCommandTempFile } from '../../io-policy.ts';
77

88
/**
@@ -22,6 +22,17 @@ import { createCommandTempFile } from '../../io-policy.ts';
2222
* frame (the eager-closure budget).
2323
*/
2424

25+
/**
26+
* The after-frame waits this long past the dispatch reply before it is
27+
* captured. The reaction the caller asked about (a key highlight, a PIN dot)
28+
* can reach the screen several frames after the transport round trip returns —
29+
* exactly on the low-end or loaded devices this flag exists for — and a frame
30+
* that raced the rendering would read an effective tap as unchanged, which is
31+
* the one direction this primitive cannot cheaply retry. Long enough to cover
32+
* that render delay; short enough to keep the flag's added latency negligible.
33+
*/
34+
const AFTER_FRAME_SETTLE_MS = 150;
35+
2536
export type ScreenshotPairOutcome = {
2637
evidence?: ScreenshotDiffEvidence;
2738
/** Present when the caller expected a diff and the frames matched. */
@@ -31,9 +42,10 @@ export type ScreenshotPairOutcome = {
3142
/**
3243
* Runs the dispatch inside the screenshot bracket. Unrequested, this is the
3344
* dispatch alone. Requested, the before-frame lands after every pre-dispatch
34-
* guard and immediately before the dispatch, so the pair brackets the tap and
35-
* nothing else; a dispatch failure cleans the frame up and rethrows, because a
36-
* failed action owes no post-action comparison.
45+
* guard and immediately before the dispatch, and the after-frame lands once
46+
* the dispatch has settled briefly (AFTER_FRAME_SETTLE_MS), so the pair
47+
* brackets the tap and its rendering; a dispatch failure cleans the frame up
48+
* and rethrows, because a failed action owes no post-action comparison.
3749
*/
3850
export async function dispatchWithScreenshotPair<T>(
3951
runtime: AgentDeviceRuntime,
@@ -122,17 +134,19 @@ async function capturePairScreenshotFrame(
122134
}
123135

124136
/**
125-
* Closes the pair: one after-frame, then the pixel comparison. A frame that
126-
* changed shape (rotation, split-screen) changed the screen by construction —
127-
* the ratio is omitted because the two frames describe different viewports.
128-
* Any decode or capture failure drops the evidence rather than guessing.
137+
* Closes the pair: one after-frame after a brief settle, then the pixel
138+
* comparison. A frame that changed shape (rotation, split-screen) changed the
139+
* screen by construction — the ratio is omitted because the two frames
140+
* describe different viewports. Any decode or capture failure drops the
141+
* evidence rather than guessing.
129142
*/
130143
async function compareScreenshotPair(
131144
runtime: AgentDeviceRuntime,
132145
options: CommandContext,
133146
before: TemporaryFile,
134147
): Promise<ScreenshotPairOutcome> {
135148
try {
149+
await sleep(runtime, AFTER_FRAME_SETTLE_MS);
136150
const after = await capturePairScreenshotFrame(runtime, options, 'after');
137151
if (!after) return {};
138152
try {

‎packages/command-surface/src/interaction/index.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -87,14 +87,14 @@ const interactionEvidenceSchema: JsonSchema = objectSchema(
8787
const screenshotDiffSchema: JsonSchema = objectSchema(
8888
{
8989
changedFromBefore: booleanSchema(
90-
'Whether any pixel moved between the screenshots taken right before and after the dispatch. false means the frames match, which reads as "the dispatch may have had no visible effect" next to the response warning.',
90+
'Whether any pixel moved between the screenshots taken right before the dispatch and after it settles briefly. false means the frames match, which reads as "the dispatch may have had no visible effect" next to the response warning.',
9191
),
9292
changedPixelRatio: numberSchema(
9393
'Share of pixels whose color moved between the two frames. Absent when the frames could not be compared, for example a shape change.',
9494
),
9595
},
9696
['changedFromBefore'],
97-
'Pixel comparison between the screenshots taken right before and after the dispatch. Absent when the pair was not taken, which claims nothing about the screen.',
97+
'Pixel comparison between the screenshots taken right before the dispatch and after it settles briefly. Absent when the pair was not taken, which claims nothing about the screen.',
9898
);
9999

100100
const responseCostSchema: JsonSchema = objectSchema(

‎packages/command-surface/src/interaction/metadata.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -87,7 +87,7 @@ type InteractionCommandName = keyof typeof interactionCommandDescriptions;
8787

8888
const expectScreenshotDiffField = () =>
8989
booleanField(
90-
'Compare lightweight screenshots taken right before and after the dispatch, and report the comparison as screenshotDiff in the response. Use it when the target lives outside the accessibility tree, such as a PIN keypad, where verify would compare empty trees. A matching pair adds a warning, because the dispatch may have had no visible effect.',
90+
'Compare lightweight screenshots taken right before the dispatch and after it settles briefly, and report the comparison as screenshotDiff in the response. Use it when the target lives outside the accessibility tree, such as a PIN keypad, where verify would compare empty trees. A matching pair adds a warning, because the dispatch may have had no visible effect.',
9191
);
9292

9393
const clickFields = {

‎packages/contracts/src/interaction-guarantees.ts‎

Lines changed: 46 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,13 @@ export const INTERACTION_GUARANTEES = [
6363
// settled tree was stored) in the same response. Best-effort: never fails
6464
// the action.
6565
'settleObservation',
66+
// --expect-screenshot-diff brackets the dispatch with two lightweight
67+
// screenshots and compares pixels: the out-of-tree readback for surfaces the
68+
// accessibility tree cannot describe (a PIN keypad), where --verify's node
69+
// digest stays flat no matter what the tap did. Advisory by the same contract
70+
// as settleObservation — the comparison never fails the action, and a
71+
// matching pair adds a warning instead.
72+
'screenshotDiffEvidence',
6673
// Failures use the shared codes/messages/hints (no-match diagnostics,
6774
// ambiguous shape, offscreen reasons). NOTE: expected to split into
6875
// errorCodes (stable codes / fallback classification) vs errorDiagnostics
@@ -186,7 +193,7 @@ const SHARED_RESPONSE_CONSTRUCTION: GuaranteeEnforcement = {
186193
const TAP_OUTCOME_NOT_OBSERVED_GAP: GuaranteeEnforcement = {
187194
kind: 'waived',
188195
reason:
189-
'gap: the response reports the dispatch only; only opt-in --verify/--settle capture post-action evidence into it. The deferred marks set after dispatch (Android snapshot freshness after press/click, post-gesture stabilization when the request sets postGestureStabilization) are judged by the next capture, never in this response. An iOS Simulator open that did not read the launched app (any postOpenObservation other than observable) marks the next capture in that session to wait for a quiet surface (markPostOpenStabilization); that steadies where a target is resolved, not whether the tap landed. The iOS ambiguous-failure corroboration reconsiders only a thrown runner error.',
196+
'gap: the response reports the dispatch only; only the opt-in --verify, --settle, and --expect-screenshot-diff features capture post-action evidence into it. The deferred marks set after dispatch (Android snapshot freshness after press/click, post-gesture stabilization when the request sets postGestureStabilization) are judged by the next capture, never in this response. An iOS Simulator open that did not read the launched app (any postOpenObservation other than observable) marks the next capture in that session to wait for a quiet surface (markPostOpenStabilization); that steadies where a target is resolved, not whether the tap landed. The iOS ambiguous-failure corroboration reconsiders only a thrown runner error.',
190197
trackingIssue: TAP_OUTCOME_NOT_OBSERVED_GAP_ISSUE,
191198
};
192199

@@ -199,7 +206,7 @@ const TAP_OUTCOME_NOT_OBSERVED_GAP: GuaranteeEnforcement = {
199206
const DIRECT_IOS_OUTCOME_NOT_OBSERVED_GAP: GuaranteeEnforcement = {
200207
kind: 'waived',
201208
reason:
202-
"gap: this replay-only route observes no outcome beyond the runner's own report; --verify/--settle do not apply here (see the inapplicable cells on this row), and the shared ambiguous-failure corroboration only reconsiders a thrown error, never a successful dispatch.",
209+
"gap: this replay-only route observes no outcome beyond the runner's own report; --verify/--settle/--expect-screenshot-diff do not apply here (see the inapplicable cells on this row), and the shared ambiguous-failure corroboration only reconsiders a thrown error, never a successful dispatch.",
203210
trackingIssue: TAP_OUTCOME_NOT_OBSERVED_GAP_ISSUE,
204211
};
205212

@@ -280,6 +287,11 @@ const RUNTIME_TREE_SHARED_GUARANTEES = {
280287
kind: 'runtime',
281288
via: 'packages/command-runtime/src/commands/interaction/runtime/settle.ts#settleAfterInteraction',
282289
},
290+
screenshotDiffEvidence: {
291+
kind: 'runtime',
292+
via: 'packages/command-runtime/src/commands/interaction/runtime/screenshot-diff-evidence.ts#dispatchWithScreenshotPair',
293+
appliesTo: ['press', 'click'],
294+
},
283295
targetStability: RUNTIME_TREE_TARGET_STABILITY,
284296
} satisfies Partial<Record<InteractionGuarantee, GuaranteeEnforcement>>;
285297

@@ -379,6 +391,11 @@ export const INTERACTION_DISPATCH_PATHS: Record<InteractionPathId, InteractionPa
379391
reason:
380392
'Target-authored drag does not expose --settle; callers can wait on the destination state.',
381393
},
394+
screenshotDiffEvidence: {
395+
kind: 'inapplicable',
396+
reason:
397+
'Target-authored drag does not expose --expect-screenshot-diff; the pixel pair brackets a tap dispatch, which this path does not run.',
398+
},
382399
errorTaxonomy: {
383400
kind: 'runtime',
384401
via: 'packages/selectors/src/internal/resolve.ts#formatSelectorFailure',
@@ -463,6 +480,12 @@ export const INTERACTION_DISPATCH_PATHS: Record<InteractionPathId, InteractionPa
463480
to: 'runtime-ref',
464481
via: '--settle disables the native ref fast path when the descriptor post-action observation trait supports settle observation — settling needs the tree-based baseline and captures',
465482
},
483+
screenshotDiffEvidence: {
484+
kind: 'delegated',
485+
to: 'runtime-ref',
486+
via: '--expect-screenshot-diff disables the native ref fast path, because the fast path never reaches the screenshot bracket that only the runtime dispatch runs under',
487+
appliesTo: ['click'],
488+
},
466489
errorTaxonomy: {
467490
kind: 'runtime',
468491
via: 'packages/selectors/src/internal/resolve.ts#STALE_REF_HINT',
@@ -475,7 +498,7 @@ export const INTERACTION_DISPATCH_PATHS: Record<InteractionPathId, InteractionPa
475498
outcomeObservation: {
476499
kind: 'waived',
477500
reason:
478-
'gap: the fast path observes no post-action outcome by default; --settle/--verify disable it and delegate to runtime-ref instead of running here.',
501+
'gap: the fast path observes no post-action outcome by default; --settle/--verify/--expect-screenshot-diff disable it and delegate to runtime-ref instead of running here.',
479502
trackingIssue: GAPS_UMBRELLA_ISSUE,
480503
},
481504
targetStability: {
@@ -532,6 +555,11 @@ export const INTERACTION_DISPATCH_PATHS: Record<InteractionPathId, InteractionPa
532555
kind: 'runtime',
533556
via: 'packages/command-runtime/src/commands/interaction/runtime/settle.ts#settleAfterInteraction',
534557
},
558+
screenshotDiffEvidence: {
559+
kind: 'runtime',
560+
via: 'packages/command-runtime/src/commands/interaction/runtime/screenshot-diff-evidence.ts#dispatchWithScreenshotPair',
561+
appliesTo: ['press', 'click'],
562+
},
535563
errorTaxonomy: {
536564
kind: 'runtime',
537565
via: 'packages/kernel/src/errors.ts#normalizeError',
@@ -614,6 +642,11 @@ export const INTERACTION_DISPATCH_PATHS: Record<InteractionPathId, InteractionPa
614642
reason:
615643
'find exposes no --settle flag for the same reason; settling is built on the tree-based runtime baseline and the captures the acting leaf owns.',
616644
},
645+
screenshotDiffEvidence: {
646+
kind: 'inapplicable',
647+
reason:
648+
'find exposes no --expect-screenshot-diff flag; the screenshot bracket lives in the acting leaf tap dispatch, which this seam never enters.',
649+
},
617650
// Locator misses and candidate sets use find's shared diagnostics, and a pre-action guard
618651
// refusal travels as the AppError its owner built (`coveredInteractionDetails`,
619652
// TAP_KEYBOARD_OCCLUDES_TARGET_DETAILS) through `normalizeError`, so code, hint, and reason
@@ -691,6 +724,11 @@ export const INTERACTION_DISPATCH_PATHS: Record<InteractionPathId, InteractionPa
691724
kind: 'inapplicable',
692725
reason: 'The eligibility gate excludes --settle from this replay-only route.',
693726
},
727+
screenshotDiffEvidence: {
728+
kind: 'inapplicable',
729+
reason:
730+
'The eligibility gate excludes --expect-screenshot-diff from this replay-only route: the fused runner request never reaches the screenshot bracket, so the flag routes the dispatch back to the runtime path.',
731+
},
694732
errorTaxonomy: {
695733
kind: 'waived',
696734
reason: 'gap: Maestro preserves the runner-native direct-selector error shapes.',
@@ -753,6 +791,11 @@ export const INTERACTION_DISPATCH_PATHS: Record<InteractionPathId, InteractionPa
753791
kind: 'inapplicable',
754792
reason: 'Replay-only path; --settle is not part of replay semantics.',
755793
},
794+
screenshotDiffEvidence: {
795+
kind: 'inapplicable',
796+
reason:
797+
'The eligibility gate excludes --expect-screenshot-diff from the replay-only routes, so a flagged dispatch re-routes to the runtime path before this fallback can be selected.',
798+
},
756799
errorTaxonomy: {
757800
kind: 'waived',
758801
reason: 'gap: shares the direct path error shapes, including their missing hints.',

‎packages/contracts/src/interaction.ts‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -253,10 +253,10 @@ export type InteractionEvidence = {
253253
/**
254254
* Opt-in (`--expect-screenshot-diff`) out-of-tree post-action readback for
255255
* press/click: the pixel comparison between the screenshots taken immediately
256-
* before and after the dispatch. Answers "did the screen react at all" for
257-
* surfaces the accessibility tree cannot describe — a PIN keypad whose keys
258-
* never reach the tree leaves `--verify`'s digest flat no matter what the tap
259-
* did — so the comparison runs on pixels instead of nodes.
256+
* before the dispatch and after it settles briefly. Answers "did the screen
257+
* react at all" for surfaces the accessibility tree cannot describe — a PIN
258+
* keypad whose keys never reach the tree leaves `--verify`'s digest flat no
259+
* matter what the tap did — so the comparison runs on pixels instead of nodes.
260260
*
261261
* Advisory by contract: the comparison never fails the action. Absent means the
262262
* pair was not taken (capture failed or the platform cannot screenshot), which

‎scripts/layering/package-boundaries.test.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -504,6 +504,7 @@ test('the real tree parses, declares, and passes R11', () => {
504504
'@agent-device/capture-kit/perf-capture-session-resource',
505505
'@agent-device/capture-kit/perf-runtime-plan',
506506
'@agent-device/capture-kit/png',
507+
'@agent-device/capture-kit/png-changed-pixel-ratio',
507508
'@agent-device/capture-kit/png-crop',
508509
'@agent-device/capture-kit/png-resize',
509510
'@agent-device/capture-kit/png-rgb-difference',

‎test/integration/interaction-contract/coordinate.contract.test.ts‎

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ import { assertRpcError, assertRpcOk } from '../provider-scenarios/assertions.ts
66
import { scenarioName } from './coverage-manifest.ts';
77
import { COORDINATE_COVERAGE } from './coordinate.coverage.ts';
88
import { settledWelcomeSnapshot, viewportOnlySnapshot } from './fixtures.ts';
9-
import { createContractDevice } from './runtime-harness.ts';
9+
import { createContractDevice, screenshotFramePlanBackend } from './runtime-harness.ts';
1010
import { runnerTapEntry, runnerTapErrorEntry, withIosContractDaemon } from './daemon-harness.ts';
1111

1212
// ADR 0011 Layer 3, coordinate path: raw x/y tap, intentionally minimal
@@ -77,6 +77,27 @@ test(scenario('settleObservation'), async () => {
7777
assert.deepEqual(settle.diff?.summary, { additions: 1, removals: 1, unchanged: 0 });
7878
});
7979

80+
test(scenario('screenshotDiffEvidence'), async () => {
81+
const device = createContractDevice(viewportOnlySnapshot(), {
82+
...screenshotFramePlanBackend([{ color: [0, 0, 0] }, { color: [255, 255, 255] }]),
83+
tap: async () => ({ ok: true }),
84+
});
85+
86+
const result = await device.interactions.press(
87+
{ kind: 'point', x: 10, y: 20 },
88+
{
89+
session: 'default',
90+
expectScreenshotDiff: true,
91+
},
92+
);
93+
94+
// The escape hatch gets the same opt-in readback: pixels answer where the
95+
// tree-less point target gives --verify nothing to digest.
96+
assert.equal(result.kind, 'point');
97+
assert.ok(result.screenshotDiff);
98+
assert.equal(result.screenshotDiff?.changedFromBefore, true);
99+
});
100+
80101
test(scenario('errorTaxonomy'), async () => {
81102
await withIosContractDaemon(
82103
[runnerTapErrorEntry(new Error('runner tap crashed'))],

‎test/integration/interaction-contract/coordinate.coverage.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,5 +8,7 @@ export const COORDINATE_COVERAGE = definePathCoverage('coordinate', {
88
'coordinate verifyEvidence: point click --verify returns a digest with change detection',
99
settleObservation:
1010
'coordinate settleObservation: point press --settle captures a baseline and returns the settled diff',
11+
screenshotDiffEvidence:
12+
'coordinate screenshotDiffEvidence: point press --expect-screenshot-diff returns the pixel comparison',
1113
errorTaxonomy: 'coordinate errorTaxonomy: backend failure surfaces as a normalized daemon error',
1214
});

‎test/integration/interaction-contract/native-ref.contract.test.ts‎

Lines changed: 27 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
import { ref, createContractDevice } from './runtime-harness.ts';
1+
import { ref, createContractDevice, screenshotFramePlanBackend } from './runtime-harness.ts';
22
import assert from 'node:assert/strict';
33
import { test } from 'vitest';
44
import type { InteractionGuarantee } from '@agent-device/contracts/interaction-guarantees';
@@ -162,6 +162,32 @@ test(scenario('settleObservation'), async () => {
162162
assert.deepEqual(settle.diff?.summary, { additions: 1, removals: 1, unchanged: 0 });
163163
});
164164

165+
test(scenario('screenshotDiffEvidence'), async () => {
166+
const calls: string[] = [];
167+
const device = createContractDevice(continueButtonSnapshot(), {
168+
platform: 'web',
169+
...screenshotFramePlanBackend([{ color: [0, 0, 0] }, { color: [255, 255, 255] }]),
170+
captureSnapshot: async () => ({ snapshot: continueButtonSnapshot() }),
171+
tap: async () => ({ ok: true }),
172+
tapTarget: async (_context, target) => {
173+
calls.push(target.ref);
174+
return {};
175+
},
176+
});
177+
178+
const result = await device.interactions.click(ref('@e1'), {
179+
session: 'default',
180+
expectScreenshotDiff: true,
181+
});
182+
183+
// --expect-screenshot-diff delegates to the runtime-ref path: the fast path
184+
// is skipped so the pixel pair brackets the dispatch it names.
185+
assert.deepEqual(calls, []);
186+
assert.equal(result.kind, 'ref');
187+
assert.ok(result.screenshotDiff);
188+
assert.equal(result.screenshotDiff?.changedFromBefore, true);
189+
});
190+
165191
test(scenario('responseIdentity'), async () => {
166192
const calls: string[] = [];
167193
const device = createNativeRefDevice(continueButtonSnapshot(), calls);

0 commit comments

Comments
 (0)