Skip to content

Commit 7471601

Browse files
strzhaoclaude
andcommitted
fix(interaction): honor --no-record in the plan recording refusal, refuse a malformed generation-shaped second key, and neutralize the plan wording
The recording refusal advised --no-record but never checked it; a second positional like @E14~s bypassed the space-form guard and became a fallback label that dispatched only the first key; press's plan help listed --button, which press does not take; and the click plan text described modality-specific pressing on a shared contract. Co-Authored-By: Claude Code <[email protected]>
1 parent 6327e1b commit 7471601

6 files changed

Lines changed: 15 additions & 6 deletions

File tree

‎packages/command-surface/src/interaction/index.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -240,7 +240,7 @@ const tapInteractionResponseDataSchema = interactionResponseDataSchema({
240240
},
241241
},
242242
['pressed', 'refs', 'points'],
243-
'Present when the command ran a multi-ref plan (press @e13,@e14,...): one press per key, in order.',
243+
'Present when the command ran a multi-ref plan (press @e13,@e14,...): one action per key, in order.',
244244
),
245245
},
246246
});
@@ -494,7 +494,7 @@ const pressCommandFacet = defineCommandFacet({
494494
text: {
495495
summary: 'Short-press a UI target',
496496
cliDetail:
497-
'The hold duration is positional on longpress, not press --hold-ms. Enter a fixed key plan in one command with a comma-separated ref sequence: press @e13,@e14,@e15,@e16 presses each key in order (2 to 16 refs; a repeated ref presses the same key twice, for a PIN digit). The whole plan resolves and admits against the current frame before the first press, so a member that cannot resolve refuses before anything dispatches. The plan is one logical command: key 1 expires the frame, later keys tap where the snapshot showed them, and the motion screen judges the plan once — take a fresh snapshot and rebuild the plan when the screen can move between keys. A later key that refuses carries details.dispatchedSteps counting the keys already sent, details.refPlanKey naming the key that refused, and details.refPlanPending listing the keys that did not run. --interval-ms sleeps between keys; --count, --hold-ms, --jitter-px, --double-tap, and --button apply to every key. The response ref is the first key; sequence.points carries where every key landed. Script recording refuses the sequence form: record one press per key.',
497+
'The hold duration is positional on longpress, not press --hold-ms. Enter a fixed key plan in one command with a comma-separated ref sequence: press @e13,@e14,@e15,@e16 presses each key in order (2 to 16 refs; a repeated ref presses the same key twice, for a PIN digit). The whole plan resolves and admits against the current frame before the first press, so a member that cannot resolve refuses before anything dispatches. The plan is one logical command: key 1 expires the frame, later keys tap where the snapshot showed them, and the motion screen judges the plan once — take a fresh snapshot and rebuild the plan when the screen can move between keys. A later key that refuses carries details.dispatchedSteps counting the keys already sent, details.refPlanKey naming the key that refused, and details.refPlanPending listing the keys that did not run. --interval-ms sleeps between keys; --count, --hold-ms, --jitter-px, and --double-tap apply to every key. The response ref is the first key; sequence.points carries where every key landed. Script recording refuses the sequence form: record one press per key.',
498498
},
499499
metadata: metadata('press'),
500500
run: (client, input) => client.interactions.press(toPressOptions(input)),

‎packages/command-surface/src/interaction/metadata.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,7 @@ const FIND_ACTION_VALUES = [
6565

6666
const interactionCommandDescriptions = {
6767
click:
68-
"Activate a UI target by snapshot ref, selector, or coordinates. Prefer a ref or selector after a snapshot; use coordinates only when semantic targeting is unavailable. This can change app state. Success means the click was dispatched, not that it landed: use settle or verify to confirm the result without a follow-up snapshot, or wait for the expected screen. A ref or selector target is re-checked for motion before dispatch when the session's previous action just animated the screen, and is dispatched at its resting position; one that has not stopped refuses with target_still_moving rather than acting where it was. A comma-separated ref list (target `@e13,@e14,@e15,@e16`) clicks each key once in order as one command, with the admission, rest, and refusal contract press documents for its sequence.",
68+
"Activate a UI target by snapshot ref, selector, or coordinates. Prefer a ref or selector after a snapshot; use coordinates only when semantic targeting is unavailable. This can change app state. Success means the click was dispatched, not that it landed: use settle or verify to confirm the result without a follow-up snapshot, or wait for the expected screen. A ref or selector target is re-checked for motion before dispatch when the session's previous action just animated the screen, and is dispatched at its resting position; one that has not stopped refuses with target_still_moving rather than acting where it was. A comma-separated ref list (target `@e13,@e14,@e15,@e16`) dispatches one action per key in order as one command, with the admission, rest, and refusal contract press documents for its sequence.",
6969
press:
7070
"Short-press a UI target by snapshot ref, selector, or coordinates. Use longpress instead when the target requires a context-menu or hold gesture. Success means the press was dispatched, not that it landed: use settle or verify to confirm the result, or wait for the expected screen. A ref or selector target is re-checked for motion before dispatch when the session's previous action just animated the screen, and is dispatched at its resting position; one that has not stopped refuses with target_still_moving rather than acting where it was. A comma-separated ref list (target `@e13,@e14,@e15,@e16`) presses each key once in order as one command; pass refs as a list of 2 to 16 snapshot refs (a repeated ref presses the same key twice), and the response carries a sequence summary with the refs, the keys pressed, and where each landed. The plan resolves and admits against the current frame before the first press; key 1 expires the frame, later keys tap where the snapshot showed them, and a later key that refuses carries the keys already sent in details.dispatchedSteps.",
7171
fill: 'Replace text in a UI input selected by snapshot ref, selector, or coordinates. Pass an empty text to clear the field. Inspect unconfirmed evidence and assert the expected value or resulting screen before continuing. Prefer refs or selectors after snapshot; use recordAs to keep sensitive text out of a recorded replay while sending it to the live app.',

‎packages/daemon/src/execution/interaction/internal/__tests__/interaction-touch-targets.test.ts‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -292,3 +292,12 @@ test('parseTouchRefSequence refuses a trailing positional after the plan', () =>
292292
expect(parsed.response.error.code).toBe('INVALID_ARGS');
293293
expect(parsed.response.error.message).toContain('takes no trailing label');
294294
});
295+
296+
test('parseTouchTarget refuses a malformed generation suffix as a second key', () => {
297+
// `@e14~s` is generation-shaped enough that treating it as a fallback label
298+
// would dispatch only the first key.
299+
const parsed = parseTouchTarget(['@e13', '@e14~s'], 'press');
300+
expect(parsed.ok).toBe(false);
301+
if (parsed.ok || parsed.response.ok) return;
302+
expect(parsed.response.error.message).toContain('Separate the keys');
303+
});

‎packages/daemon/src/execution/interaction/internal/interaction-touch-press-admission.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -206,7 +206,7 @@ function sequenceRecordingGuardResponse(
206206
parsedTarget: ParsedTargetedTouch,
207207
): DaemonResponse | undefined {
208208
if (!parsedTarget.refs) return undefined;
209-
if (isSessionRecording(session)) {
209+
if (isSessionRecording(session) && !req.flags?.noRecord) {
210210
return errorResponse(
211211
'INVALID_ARGS',
212212
'A multi-ref press sequence cannot be recorded: the recorder binds one target per step, so the recorded step would press only the first key. Record one press per key, or pass --no-record on this step.',

‎packages/daemon/src/execution/interaction/internal/interaction-touch-targets.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,7 @@ export function parseTouchTarget(positionals: string[], commandLabel: string): P
5656
if (
5757
SEQUENCE_TOUCH_COMMANDS.has(commandLabel) &&
5858
spaceSeparatedRef !== undefined &&
59-
/^@e\d+(~s\d+)?$/.test(spaceSeparatedRef.trim())
59+
/^@e\d+(~s?\d*)?$/.test(spaceSeparatedRef.trim())
6060
) {
6161
// `press @e13 @e14` is a comma-separated plan typed with spaces. On the
6262
// single-ref path the extra positional would become a fallback label and

‎src/commands/schema/cli-help-gestures.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ Shapes:
2323
agent-device gesture tap 80 300 200 450 310 600 --count 20 --pause-ms 100
2424
scroll <dir> --until <selector> repeats scroll-and-check passes until that element is on screen, then stops: one request instead of a scroll-then-snapshot loop, and it stops on the target rather than overshooting it. It reports the passes it spent; vertical searches compare repeated captures of descendants in the selected scroll container and fail when those captures show no progress. If no container is found, they compare the visible tree. It is refused on top/bottom, which already stop themselves. The default limit is ${DEFAULT_SCROLL_UNTIL_PASSES} gestures. For a longer list, add --max-passes <n> (1–${SCROLL_UNTIL_MAX_PASSES}); it requires --until. Large pass budgets can hit the 90-second request timeout; inspect the current screen and continue with another search if that happens.
2525
longpress accepts coordinates, @refs, or selectors; prefer @ref/selector, coordinates only as a fallback. Duration and gesture scale/center are positional. gesture pan is one finger by default; add --pointer-count 2 for a parallel two-finger pan. Keep count/pause/pattern on one swipe: --count (cap 200), --pause-ms (cap 10000ms), --pattern ping-pong; the combined swipe/pause schedule is capped at 60000ms.
26-
For repeated iOS smoke checks: press <x> <y> --count <n> --jitter-px <n> for tap series, swipe <x1> <y1> <x2> <y2> --count <n> for drag series. gesture tap touches 2 to 5 distinct points and holds them for --hold-ms (default 50, min 16); --count repeats the same contacts (cap 200) with --pause-ms between, the series must fit 60000ms counting about 500ms per repetition, and a repetition costs about 450ms on a physical iPhone. Use press for one finger. For repeated keys (PIN pad), enter the plan in one command: press @e13,@e14,@e15,@e16 (2 to 16 refs, one press per key, a repeated ref presses the same key twice, click takes the same form); the whole plan resolves and admits against the current frame before the first press, key 1 expires the frame, later keys tap where the snapshot showed them, so re-snapshot and rebuild the plan when the screen can move between keys; a later key that refuses carries details.dispatchedSteps counting the keys already sent, --interval-ms sets the key delay, and --count/--hold-ms/--jitter-px/--double-tap/--button apply to every key.
26+
For repeated iOS smoke checks: press <x> <y> --count <n> --jitter-px <n> for tap series, swipe <x1> <y1> <x2> <y2> --count <n> for drag series. gesture tap touches 2 to 5 distinct points and holds them for --hold-ms (default 50, min 16); --count repeats the same contacts (cap 200) with --pause-ms between, the series must fit 60000ms counting about 500ms per repetition, and a repetition costs about 450ms on a physical iPhone. Use press for one finger. For repeated keys (PIN pad), enter the plan in one command: press @e13,@e14,@e15,@e16 (2 to 16 refs, one press per key, a repeated ref presses the same key twice, click takes the same form); the whole plan resolves and admits against the current frame before the first press, key 1 expires the frame, later keys tap where the snapshot showed them, so re-snapshot and rebuild the plan when the screen can move between keys; a later key that refuses carries details.dispatchedSteps counting the keys already sent, --interval-ms sets the key delay, and --count/--hold-ms/--jitter-px/--double-tap apply to every key.
2727
2828
Platform quirks:
2929
If rotation stops repeated iOS presses, inspect the screen before continuing with the remaining presses.

0 commit comments

Comments
 (0)