Skip to content

Commit ffff797

Browse files
strzhaoclaude
andcommitted
fix(interaction): keep the screenshot readback from failing a delivered tap
Nothing after a resolved dispatch throws out of the evidence lane anymore: a decode, read, import or cleanup failure yields no evidence, because a throwing readback reported a delivered tap as failed and a PIN caller would re-enter the digit. The after-frame settle honors the request signal. Both frames decode in the codec worker, the guard audits a missing descriptor instead of skipping it, the docs say the readback is best-effort, and the delivered-tap tests record the tap. Co-Authored-By: Claude Code <[email protected]>
1 parent 7a61db9 commit ffff797

4 files changed

Lines changed: 67 additions & 14 deletions

File tree

‎packages/command-runtime/src/commands/interaction/runtime/screenshot-diff-evidence.test.ts‎

Lines changed: 31 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -177,13 +177,16 @@ test('press with expectScreenshotDiff reports a shape change without a ratio', a
177177

178178
test('press with expectScreenshotDiff drops the evidence when a capture fails', async () => {
179179
let captureCount = 0;
180+
const taps: Array<{ x: number; y: number }> = [];
180181
const device = createInteractionDevice(selectorSnapshot(), {
181182
captureScreenshot: async () => {
182183
captureCount += 1;
183184
if (captureCount === 1) throw new Error('screen capture unavailable');
184185
return { path: '/unused/after.png' };
185186
},
186-
tap: async () => ({ ok: true }),
187+
tap: async (_context, point) => {
188+
taps.push(point);
189+
},
187190
});
188191

189192
const result = await device.interactions.press(
@@ -194,10 +197,11 @@ test('press with expectScreenshotDiff drops the evidence when a capture fails',
194197
},
195198
);
196199

197-
// No pair, no claim — and the dispatch itself still succeeds.
200+
// No pair, no claim — the tap still crossed, and the readback says nothing.
198201
assert.equal('screenshotDiff' in result, false);
199202
assert.equal('warning' in result ? result.warning : undefined, undefined);
200203
assert.deepEqual(captureCount, 1);
204+
assert.deepEqual(taps, [{ x: 10, y: 20 }]);
201205
});
202206

203207
test('press without a screenshot backend drops the evidence and still taps', async () => {
@@ -247,3 +251,28 @@ test('press with expectScreenshotDiff routes a ref target through the runtime pa
247251
assert.ok(result.screenshotDiff);
248252
assert.equal(result.screenshotDiff?.changedFromBefore, true);
249253
});
254+
255+
test('a corrupted after-frame yields no evidence instead of failing a delivered tap', async () => {
256+
const corrupted = Buffer.from('not a png at all');
257+
const backend = framePlanBackend([
258+
{ bytes: solidPngBuffer(8, 8, { r: 0, g: 0, b: 0 }) },
259+
{ bytes: corrupted },
260+
]);
261+
const taps: Array<{ x: number; y: number }> = [];
262+
const device = createInteractionDevice(selectorSnapshot(), {
263+
...backend,
264+
tap: async (_context, point) => {
265+
taps.push(point);
266+
},
267+
});
268+
269+
const result = await device.interactions.press(
270+
{ kind: 'point', x: 10, y: 20 },
271+
{ session: 'default', expectScreenshotDiff: true },
272+
);
273+
274+
// The tap crossed the device; the unreadable readback claims nothing.
275+
assert.equal(taps.length, 1);
276+
assert.equal(result.screenshotDiff, undefined);
277+
assert.equal(result.warning, undefined);
278+
});

‎packages/command-runtime/src/commands/interaction/runtime/screenshot-diff-evidence.ts‎

Lines changed: 32 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,9 @@ export async function dispatchWithScreenshotPair<T>(
6060
try {
6161
result = await dispatch();
6262
} catch (error) {
63-
if (before) await before.cleanup();
63+
// The evidence lane owns nothing after a resolved dispatch, and a failed
64+
// dispatch rethrows its own error even if the frame cleanup fails.
65+
if (before) await before.cleanup().catch(() => {});
6466
throw error;
6567
}
6668
return {
@@ -145,23 +147,43 @@ async function compareScreenshotPair(
145147
options: CommandContext,
146148
before: TemporaryFile,
147149
): Promise<ScreenshotPairOutcome> {
150+
// Nothing after a resolved dispatch may throw out of this lane: a decode,
151+
// read, import or cleanup failure yields no evidence, because a throwing
152+
// readback would report a delivered tap as failed and a PIN caller would
153+
// re-enter the digit.
148154
try {
149-
await sleep(runtime, AFTER_FRAME_SETTLE_MS);
155+
return await compareScreenshotPairInner(runtime, options, before);
156+
} catch {
157+
return {};
158+
}
159+
}
160+
161+
async function compareScreenshotPairInner(
162+
runtime: AgentDeviceRuntime,
163+
options: CommandContext,
164+
before: TemporaryFile,
165+
): Promise<ScreenshotPairOutcome> {
166+
try {
167+
// A request canceled after the dispatch stops waiting for the after-frame.
168+
await sleep(runtime, AFTER_FRAME_SETTLE_MS, options.signal);
150169
const after = await capturePairScreenshotFrame(runtime, options, 'after');
151170
if (!after) return {};
152171
try {
153-
const [{ computePngChangedPixelRatio }, { decodePng }] = await Promise.all([
172+
const [{ computePngChangedPixelRatio }, { decodePngAsync }] = await Promise.all([
154173
import('@agent-device/capture-kit/png-changed-pixel-ratio'),
155-
import('@agent-device/capture-kit/png'),
174+
import('@agent-device/capture-kit/png-worker-client'),
156175
]);
157176
const [beforeBytes, afterBytes] = await Promise.all([
158177
fs.readFile(before.path),
159178
fs.readFile(after.path),
160179
]);
161-
const ratio = computePngChangedPixelRatio(
162-
decodePng(beforeBytes, 'the press before-screenshot'),
163-
decodePng(afterBytes, 'the press after-screenshot'),
164-
);
180+
// The decodes run in the codec worker: full-resolution frames never sit
181+
// on the event loop.
182+
const [beforePng, afterPng] = await Promise.all([
183+
decodePngAsync(beforeBytes, 'the press before-screenshot'),
184+
decodePngAsync(afterBytes, 'the press after-screenshot'),
185+
]);
186+
const ratio = computePngChangedPixelRatio(beforePng, afterPng);
165187
if (ratio.status === 'dimension_mismatch') {
166188
return { evidence: { changedFromBefore: true } };
167189
}
@@ -179,9 +201,9 @@ async function compareScreenshotPair(
179201
}),
180202
};
181203
} finally {
182-
await after.cleanup();
204+
await after.cleanup().catch(() => {});
183205
}
184206
} finally {
185-
await before.cleanup();
207+
await before.cleanup().catch(() => {});
186208
}
187209
}

‎test/repo-guards/command-descriptor-expect-screenshot-diff.test.ts‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -75,7 +75,9 @@ test('expectScreenshotDiff CLI flags are scoped to press and click', () => {
7575
test('expectScreenshotDiff metadata fields are scoped to press and click', () => {
7676
for (const command of [...SCREENSHOT_DIFF_COMMANDS, ...OUT_OF_SCOPE_COMMANDS]) {
7777
const metadata = findCommandMetadata(command);
78-
if (!metadata) continue;
78+
if (!metadata) {
79+
throw new Error(`listed command ${command} carries no metadata; the guard cannot audit it`);
80+
}
7981
const properties = metadata.inputSchema.properties ?? {};
8082
assert.equal(
8183
Object.hasOwn(properties, 'expectScreenshotDiff'),

‎website/docs/docs/commands.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -499,7 +499,7 @@ agent-device gesture tap 80 300 200 450 --count 20 --pause-ms 100 # repeated two
499499
500500
If rotation stops repeated iOS presses, inspect the screen before continuing with the remaining presses.
501501
502-
A successful `click` or `press` means the action was sent to where the command found the target. It does not confirm that the element received it or that the screen changed. When the next step depends on it, add `--verify` (reports `changedFromBefore`), add `--settle` (returns the diff once the UI goes quiet), or follow with `wait <selector>`. When the target lives outside the accessibility tree (a PIN keypad), `--verify` compares empty trees, so add `--expect-screenshot-diff` on `press`/`click` instead: the command compares screenshots taken right before the dispatch and after it settles briefly, and warns when they match. Before retrying a failed interaction, see [Retry after a failed command](#retry-after-a-failed-command).
502+
A successful `click` or `press` means the action was sent to where the command found the target. It does not confirm that the element received it or that the screen changed. When the next step depends on it, add `--verify` (reports `changedFromBefore`), add `--settle` (returns the diff once the UI goes quiet), or follow with `wait <selector>`. When the target lives outside the accessibility tree (a PIN keypad), `--verify` compares empty trees, so add `--expect-screenshot-diff` on `press`/`click` instead: the command compares screenshots taken right before the dispatch and after it settles briefly, and warns when they match. The readback is best-effort: on a backend without screenshot capture, or when a capture or comparison fails, the response carries neither `screenshotDiff` nor a warning and claims nothing about the screen. Before retrying a failed interaction, see [Retry after a failed command](#retry-after-a-failed-command).
503503
504504
A `click` or `press` whose target the daemon resolves to a point (`@ref` or selector) re-checks that target's position before dispatching when the session's previous action animated the screen, and dispatches where it came to rest. When that check cannot confirm a stable position — the target was still moving, or a re-check could not measure it — the command refuses with `target_still_moving` and `dispatched: no`: nothing was dispatched, the point the command resolved is in `error.details.rect`, and where the target was last seen is in `error.details.movedToRect` when that last look measured one. Run `wait <selector>` on the target and retry the same command, or add `--settle` so the action waits for the UI to go quiet. A refused `@ref` press needs a fresh `snapshot -i` and a selector, because the wait retires the ref. Two dispatch paths hand off before the check: a press at raw coordinates has no resolved target to re-check, and a dispatch the backend performs on the element itself — `click @ref` on a backend with element dispatch, or a Maestro non-hittable coordinate fallback — goes to that backend rather than through a resolved point. The re-check stops when the request is canceled or reaches its deadline, and nothing is dispatched (`dispatched: no`): a canceled request fails with `details.reason: request_canceled`, and a request out of time with `details.reason: daemon_request_timeout`.
505505

0 commit comments

Comments
 (0)