Skip to content

Commit 14d1006

Browse files
thymikeeclaude
andcommitted
fix(macos-helper): stop the helper with SIGTERM before SIGKILL so a held mouse button is released
The helper's release handler traps SIGTERM, SIGINT and SIGHUP, but the host stopped the helper only through `killProcessTree`, which sends SIGKILL on both the deadline and a cancelled request. Now that the request signal reaches the surface press, a cancelled `press --hold-ms 10000` or a dropped client killed the helper between the mouse-down and its mouse-up and the button stayed down. `ExecOptions` gains a `kill` policy: the first signal to send and a grace after which SIGKILL follows. `killProcessTree` sends the policy's signal, arms an unref'd escalation timer, and clears it on the child's exit; the group path and the reaped-pid guard are shared by both signals. `runMacOsHelper` asks for SIGTERM with a one-second grace on every helper run, so the release handler runs on every stop the host applies. The helper also posts the mouse-up before it clears the held-button record, so a signal landing between the two can no longer exit with the button down. Two exec tests pin the route with a trap script standing in for the helper: a cancelled command with a kill policy sees the signal and records its release, and a child that ignores the first signal is still ended once the grace passes. The helper test asserts the policy the press hands the provider. The docs note named the `desktop` surface among those that post through the helper; that surface inspects only, and the note now says so. Co-Authored-By: Claude Fable 5.1 <[email protected]>
1 parent 0c6a78f commit 14d1006

7 files changed

Lines changed: 130 additions & 9 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,14 @@
22

33
## Unreleased
44

5+
- Fixed (macos): `press` and `click` on the `frontmost-app`, `desktop`, and `menubar` surfaces post
6+
clicks the way a trackpad does. The helper posted the release in the same tick as the press, so
7+
AppKit dropped it and controls never activated; each press is now held (`--hold-ms`, 60 ms by
8+
default, 40 ms at least). `--count` is that many independent clicks and `--double-tap` posts each
9+
as a double-click pair, the helper's deadline follows the click schedule instead of a fixed 30 s,
10+
and every stop the host applies mid-hold — a cancelled request, a dropped client, the deadline —
11+
reaches the helper as SIGTERM first so it releases the button before it exits; SIGKILL only
12+
follows a helper that has not exited a second later.
513
- Added (diff): `diff screenshot` accepts a JPEG baseline or current image. Both inputs had to be PNG,
614
so a capture exported by another tool had to be converted first and a HarmonyOS capture — which the
715
platform serves as JPEG under whatever name the command was given — could never be compared. Each

‎apple/macos-helper/Sources/AgentDeviceMacOSInput/MouseClickDelivery.swift‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,9 @@ public enum MouseClickDeliveryError: Error, Equatable {
3535

3636
/// The button the helper currently holds down, kept where a signal handler can reach it.
3737
/// A helper killed between a mouse-down and its mouse-up would otherwise leave the system's
38-
/// primary button stuck down for whatever the user touches next.
38+
/// primary button stuck down for whatever the user touches next. The host stops the helper
39+
/// with SIGTERM before SIGKILL (`runMacOsHelper` in `helper.ts`) so that this handler runs
40+
/// on a deadline and on a cancelled request, not only on a signal sent by hand.
3941
nonisolated(unsafe) private var heldMouseButton: (point: CGPoint, clickState: Int)?
4042

4143
private func releaseHeldMouseButton() {
@@ -105,7 +107,10 @@ public func postMouseClick(_ request: MouseClickRequest) throws {
105107
heldMouseButton = (point, press.clickState)
106108
down.post(tap: .cghidEventTap)
107109
usleep(UInt32(hold) * 1000)
108-
heldMouseButton = nil
110+
// The record clears only after the up is posted: a signal that lands between the two
111+
// would otherwise find nothing to release and exit with the button still down. A signal
112+
// that lands after the post releases a button already up, which is harmless.
109113
up.post(tap: .cghidEventTap)
114+
heldMouseButton = nil
110115
}
111116
}

‎packages/host-kit/src/internal/exec-kill-settle.test.ts‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -335,6 +335,67 @@ test.runIf(process.platform !== 'win32')(
335335
5_000,
336336
);
337337

338+
// A child that holds something the kill would strand — the macOS helper with the mouse button
339+
// down — asks for a signal it can handle before SIGKILL. The trap script below stands in for
340+
// that helper: it records the release the signal handler performs, and its `sleep` is the
341+
// hold the kill interrupts.
342+
343+
function releaseMarkerPath(label: string): string {
344+
return path.join(mkdtempForTestSync(`agent-device-exec-${label}-`), 'released');
345+
}
346+
347+
function holdUntilSignalledShellScript(markerPath: string): string {
348+
return `trap 'printf released > ${shellQuote(markerPath)}; exit 143' TERM; sleep ${HOLDER_LIFETIME_SECONDS} & wait`;
349+
}
350+
351+
test.runIf(process.platform !== 'win32')(
352+
'a cancelled command with a kill policy is signalled so it can release what it holds',
353+
async () => {
354+
const markerPath = releaseMarkerPath('graceful-abort');
355+
const controller = new AbortController();
356+
const held = runCmd('/bin/sh', ['-c', holdUntilSignalledShellScript(markerPath)], {
357+
signal: controller.signal,
358+
kill: { signal: 'SIGTERM', graceMs: 2_000 },
359+
});
360+
const rejection = settledRejection(held);
361+
await sleep(100);
362+
363+
controller.abort();
364+
const outcome = await rejection;
365+
366+
assert.ok(outcome, 'a cancelled command must not resolve');
367+
const details = (outcome.error as { details?: Record<string, unknown> }).details;
368+
assert.equal(details?.reason, 'request_canceled');
369+
assert.equal(
370+
fs.readFileSync(markerPath, 'utf8'),
371+
'released',
372+
'the child never saw the signal its handler releases on: it was killed outright',
373+
);
374+
},
375+
);
376+
377+
test.runIf(process.platform !== 'win32')(
378+
'a deadline with a kill policy still ends a child that ignores the first signal',
379+
async () => {
380+
const startedAt = Date.now();
381+
await assert.rejects(
382+
() =>
383+
runCmd('/bin/sh', ['-c', `trap '' TERM; sleep ${HOLDER_LIFETIME_SECONDS} & wait`], {
384+
timeoutMs: DEADLINE_MS,
385+
kill: { signal: 'SIGTERM', graceMs: 200 },
386+
}),
387+
(error: unknown) => {
388+
assert.equal(isCommandTimeoutError(error), true);
389+
return true;
390+
},
391+
);
392+
const elapsedMs = Date.now() - startedAt;
393+
assert.ok(elapsedMs >= DEADLINE_MS + 200, `escalated before the grace passed: ${elapsedMs}ms`);
394+
assert.ok(elapsedMs < 2_000, `the grace became a way to outlive the deadline: ${elapsedMs}ms`);
395+
},
396+
10_000,
397+
);
398+
338399
test.runIf(process.platform !== 'win32')(
339400
'runCmd that was never killed still drains output a descendant writes after its parent exited',
340401
async () => {

‎packages/host-kit/src/internal/exec.ts‎

Lines changed: 38 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,13 @@ export type ExecOptions = {
2727
signal?: AbortSignal;
2828
/** Max stdout/stderr bytes for synchronous runs (default Node ~1MB). */
2929
maxBuffer?: number;
30+
/**
31+
* How a deadline or a cancelled request stops the child. Without this it is SIGKILL at
32+
* once. A child that has to undo something before it dies — the macOS helper releasing a
33+
* mouse button it is holding — is sent `signal` first, and SIGKILL only once `graceMs`
34+
* pass without it exiting.
35+
*/
36+
kill?: { readonly signal: NodeJS.Signals; readonly graceMs: number };
3037
};
3138

3239
export type ExecStreamOptions = ExecOptions & {
@@ -206,7 +213,7 @@ function runSpawnedCommand(
206213
// group kill that would have ended the pipe holder can no longer run through a child
207214
// Node already reaped.
208215
const settlement = createCommandKillSettlement({
209-
killProcessTree: () => killProcessTree(child, options.detached),
216+
killProcessTree: () => killProcessTree(child, options),
210217
settle,
211218
});
212219
const abort = watchCommandAbort(options, settlement.requestKill);
@@ -224,7 +231,7 @@ function runSpawnedCommand(
224231
if (abort.didAbort || didTimeout) return;
225232
if (isEpipeError(error)) return;
226233
fail(createStdinError(executable, cmd, args, error));
227-
killProcessTree(child, options.detached);
234+
killProcessTree(child, options);
228235
});
229236

230237
child.stdout.on('data', (chunk) => {
@@ -470,7 +477,7 @@ export function runCmdBackground(
470477
resolve({ stdout, stderr, exitCode: finalExitCode });
471478
}
472479
const settlement = createCommandKillSettlement({
473-
killProcessTree: () => killProcessTree(child, options.detached),
480+
killProcessTree: () => killProcessTree(child, options),
474481
settle,
475482
});
476483
const abort = watchCommandAbort(options, settlement.requestKill);
@@ -910,17 +917,42 @@ export function signalProcessGroupBestEffort(pid: number, signal: NodeJS.Signals
910917
* waiting on. The one group-signal seam reports whether anything was reached rather than
911918
* throwing, and a group that is gone or not ours to signal is the case it reports false.
912919
*/
913-
function killProcessTree(child: ChildProcess, detached: boolean | undefined): void {
920+
function killProcessTree(
921+
child: ChildProcess,
922+
options: Pick<ExecOptions, 'detached' | 'kill'>,
923+
): void {
924+
if (!options.kill) {
925+
signalProcessTree(child, options.detached, 'SIGKILL');
926+
return;
927+
}
928+
// The child is given its chance to clean up, and the escalation is what keeps that
929+
// chance from becoming a way to outlive the deadline. The timer holds nothing open:
930+
// a child that exits on the first signal clears it, and a worker shutting down owes
931+
// a child that ignored the signal nothing further.
932+
signalProcessTree(child, options.detached, options.kill.signal);
933+
const escalation = setTimeout(
934+
() => signalProcessTree(child, options.detached, 'SIGKILL'),
935+
options.kill.graceMs,
936+
);
937+
escalation.unref();
938+
child.once('exit', () => clearTimeout(escalation));
939+
}
940+
941+
function signalProcessTree(
942+
child: ChildProcess,
943+
detached: boolean | undefined,
944+
signal: NodeJS.Signals,
945+
): void {
914946
if (detached && child.pid && process.platform !== 'win32') {
915-
signalProcessGroupBestEffort(child.pid, 'SIGKILL');
947+
signalProcessGroupBestEffort(child.pid, signal);
916948
return;
917949
}
918950
// A non-detached child leaves its pid free for the kernel to hand to an unrelated
919951
// process once Node has reaped it, so a late signal from a stale deadline could
920952
// strike a stranger. Nothing waits for a kill of a child that is already gone:
921953
// settlement happens on `exit`.
922954
if (child.exitCode !== null || child.signalCode !== null) return;
923-
child.kill('SIGKILL');
955+
child.kill(signal);
924956
}
925957

926958
/**

‎packages/platform-apple/src/os/macos/helper.test.ts‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -141,12 +141,14 @@ test('macOS helper press keeps repeats independent and names a double-click expl
141141
test('macOS helper press outlives its own click schedule and forwards cancellation', async () => {
142142
let receivedTimeoutMs: number | undefined;
143143
let receivedSignal: AbortSignal | undefined;
144+
let receivedKill: { signal: string; graceMs: number } | undefined;
144145
const controller = new AbortController();
145146
const provider = createLocalAppleToolProvider({
146147
macosHelper: {
147148
run: async (_args, options) => {
148149
receivedTimeoutMs = options?.timeoutMs;
149150
receivedSignal = options?.signal;
151+
receivedKill = options?.kill;
150152
return helperReturn({ x: 1, y: 2, holdMs: 10_000, clicks: 4 });
151153
},
152154
},
@@ -170,6 +172,10 @@ test('macOS helper press outlives its own click schedule and forwards cancellati
170172
assert.equal(scheduleMs, 40_360);
171173
assert.equal(receivedTimeoutMs, scheduleMs + 30_000);
172174
assert.equal(receivedSignal, controller.signal);
175+
// The host stops the helper with SIGKILL on both routes, and a helper killed between a
176+
// mouse-down and its mouse-up leaves the button stuck. The helper's release handler only
177+
// runs if the stop reaches it as a catchable signal first.
178+
assert.deepEqual(receivedKill, { signal: 'SIGTERM', graceMs: 1_000 });
173179
});
174180

175181
test('macOS click schedule mirrors the helper floors for the timeout it derives', () => {

‎packages/platform-apple/src/os/macos/helper.ts‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -268,6 +268,14 @@ export async function startMacOsAudioProbeProcess(options: {
268268
}
269269

270270
const MACOS_HELPER_TIMEOUT_MS = 30_000;
271+
/**
272+
* Every stop the host applies to the helper — a deadline, a cancelled request, a client that
273+
* dropped mid-command — reaches it as SIGTERM first. A helper posting a press may be holding
274+
* the mouse button down at that moment, and its SIGTERM handler releases the button before it
275+
* exits; SIGKILL would end it between the down and the up and leave the button stuck for
276+
* whatever the user touches next. A helper that has not exited a second later is killed.
277+
*/
278+
const MACOS_HELPER_KILL_GRACE_MS = 1_000;
271279

272280
async function runMacOsHelper<T extends Record<string, unknown>>(
273281
args: string[],
@@ -277,6 +285,7 @@ async function runMacOsHelper<T extends Record<string, unknown>>(
277285
allowFailure: true,
278286
timeoutMs: options.timeoutMs ?? MACOS_HELPER_TIMEOUT_MS,
279287
signal: options.signal,
288+
kill: { signal: 'SIGTERM' as const, graceMs: MACOS_HELPER_KILL_GRACE_MS },
280289
};
281290
const helperProvider = resolveAppleToolProvider().macosHelper;
282291
const helperPath = helperProvider

‎website/docs/docs/commands.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -324,7 +324,7 @@ agent-device snapshot -i --platform apple --target desktop
324324
- In macOS app sessions, `screenshot` captures the target app window bounds rather than the full desktop.
325325
- Prefer selector or `@ref`-driven interactions on macOS. Window position can shift between runs, so raw x/y point commands are less stable than snapshot-derived targets.
326326
- Use `click --button secondary` for context menus on macOS, then run `snapshot -i` again.
327-
- On `frontmost-app`, `desktop`, and `menubar` surfaces, `press` and `click` post synthetic mouse events through the macOS helper: `--hold-ms` is how long the button stays down (at least 40 ms, 60 ms by default, because AppKit drops a release posted in the same tick as its press), `--count` is that many independent clicks, and `--double-tap` posts each click as a double-click pair. A long schedule such as `--hold-ms 10000 --count 4` is given the time it needs, and a helper interrupted mid-hold releases the button before it exits. `--jitter-px` is not applied on these surfaces.
327+
- On `frontmost-app` and `menubar` surfaces, `press` and `click` post synthetic mouse events through the macOS helper (the `desktop` surface inspects only): `--hold-ms` is how long the button stays down (at least 40 ms, 60 ms by default, because AppKit drops a release posted in the same tick as its press), `--count` is that many independent clicks, and `--double-tap` posts each click as a double-click pair. A long schedule such as `--hold-ms 10000 --count 4` is given the time it needs, and a helper stopped mid-hold — by a cancelled request, a dropped client, or its deadline — releases the button before it exits. `--jitter-px` is not applied on these surfaces.
328328
- Mobile-only helpers remain unsupported on macOS: `boot`, `shutdown`, `home`, `orientation`, `app-switcher`, `action-button`, `install`, `reinstall`, `install-from-source`, and `push`.
329329
330330
Recommended loops:

0 commit comments

Comments
 (0)