Skip to content

Commit b9b2d06

Browse files
committed
Merge remote-tracking branch 'origin/main' into fix/3061-ime-rebind
* origin/main: (77 commits) fix(apple-runner): fence prep spawns behind a start-owned admission (callstack#3239) 0.21.22 test(apple): own the simctl settings plan tests in simctl-settings.test.ts (callstack#3244) fix(limrun): report the session device id in iOS settings refusals (callstack#3243) 0.21.21 feat(remote): add a host-allocated macos-app lease backend (callstack#3236) test(android): bound the screenshot write wait by wall time, not event-loop turns (callstack#3250) feat(recording): cap the touch overlay frame rate at the caller's --fps (callstack#3241) fix(ad-script): let .ad scripts carry scroll --until and wait capture flags (callstack#3197) (callstack#3234) feat(provider-webdriver): keyboard enter, dismiss, and status over WebDriver (callstack#3233) feat(selectors): match role= against snapshot kind with a node-scoped alias window (callstack#3232) fix(provider-webdriver): read field values, placeholders, secure fields, and checked state from page source (callstack#3231) feat(replay): accept --test-ime on test and replay so flow-owned Android opens opt into the test IME (callstack#3235) refactor(daemon): route daemon-level diagnostics through one scope helper (callstack#3242) docs(adr): correct ADR 0031 pointer event delivery evidence (callstack#3245) fix(ios): stop a tap's post-gesture lookup from recording an XCTest failure (callstack#3060) (callstack#3237) fix(recording): render the touch overlay at most 30 fps and inside the record request (callstack#3219) fix(daemon): keep an idle daemon alive only for retained leases (callstack#3227) fix(provider-webdriver): send an empty JSON object on bodyless POSTs (callstack#3230) Feat/maestro repeat while (callstack#3214) ...
2 parents 63f0ced + 1163b7a commit b9b2d06

908 files changed

Lines changed: 40481 additions & 8210 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.fallowrc.json‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@
99
"src/sdk/metro.ts",
1010
"src/sdk/remote-config.ts",
1111
"src/sdk/install-source.ts",
12+
"src/sdk/plugins.ts",
1213
"src/sdk/android-adb.ts",
1314
"src/sdk/contracts.ts",
1415
"src/sdk/selectors.ts",
@@ -217,9 +218,10 @@
217218
},
218219
{
219220
"comment": "Dedicated CLI command handlers are reached only through the dynamic `import()` table `dedicatedCliCommandHandlerLoaders` in src/cli/commands/router.ts, which --production analysis cannot follow to a consumer. Same shape as the daemon route-handler entry above; that table is what enumerates this list, so add/remove here whenever a loader is added/removed.",
220-
"file": "src/cli/commands/{auth,connection,daemon,device,proxy,recording,replay,screenshot,takeover}.ts",
221+
"file": "src/cli/commands/{auth,connection,daemon,device,plugins,proxy,recording,replay,screenshot,takeover}.ts",
221222
"exports": [
222223
"authCommand",
224+
"pluginsCommand",
223225
"connectCommand",
224226
"disconnectCommand",
225227
"connectionCommand",

‎AGENTS.md‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -56,9 +56,11 @@ Read the declaration rather than maintaining a prose copy:
5656
- common command input fields, and which surface may write an input key (model, operator, retired):
5757
`src/commands/common-input-fields.ts` and `src/commands/input-audience.ts`
5858

59-
Shared selector parsing and matching belongs in `@agent-device/selectors`; request cancellation
60-
and progress in `@agent-device/host-kit/request`; cross-layer contracts in `packages/contracts/src`;
61-
CLI flags in `src/commands/cli-grammar`; cross-surface schema composition in `src/commands/schema`.
59+
Shared selector parsing and matching belongs in `@agent-device/selectors`; daemon-side request
60+
cancellation and progress in `@agent-device/host-kit/request`, and the caller's per-call abort guard
61+
beside the transports it serves in `src/daemon-client/daemon-client-transport.ts`; cross-layer
62+
contracts in `packages/contracts/src`; CLI flags in `src/commands/cli-grammar`; cross-surface schema
63+
composition in `src/commands/schema`.
6264

6365
Resolve registry completeness failures at the missing declaration. Diagnose other gate failures
6466
at their reported invariant; do not suppress them or add an allowlist to get a pass. Build interaction

‎CONTRIBUTING.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,11 @@ pnpm test-app:typecheck
3737
`pnpm build` compiles the TypeScript CLI and library. If a running development daemon must pick up
3838
that build, use `pnpm rebuild:cli`; it builds and then stops the worktree-scoped daemon.
3939

40+
`pnpm clean:daemon` retains state when it cannot confirm the recorded daemon exited. Restore
41+
process inspection or stop the verified owner before retrying. Its `--prune-dev` option considers
42+
dev state directories whose newest observed modification is at least 14 days old. It retires
43+
confirmed abandoned registrations and keeps session artifacts and state directories for inspection.
44+
4045
Build only the Apple runner target you changed:
4146

4247
```bash
Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
package com.callstack.agentdevice.snapshothelper;
2+
3+
import android.content.Context;
4+
import android.graphics.Rect;
5+
import android.os.Build;
6+
import android.util.DisplayMetrics;
7+
import android.view.Display;
8+
import android.view.WindowManager;
9+
10+
/**
11+
* The physical display the captured bounds were measured on, in physical pixels, in the current
12+
* rotation (#3182).
13+
*
14+
* Node bounds reach the host through {@code AccessibilityNodeInfo.getBoundsInScreen()}, which is
15+
* absolute screen space across every interactive window, SystemUI's navigation bar included. The
16+
* extent therefore has to be the real display extent rather than {@code Resources.getSystem()}'s
17+
* app display size: the latter excludes a persistent navigation bar, so a nav-bar node's rect would
18+
* land outside the published box. On API 30+ that extent is
19+
* {@code WindowManager.getMaximumWindowMetrics().getBounds()}; below it, {@code
20+
* Display.getRealMetrics}, which likewise carries the full panel and not the app-usable inset.
21+
*
22+
* The caller reads this twice around the tree dump and publishes only what {@link #whenBothAgree}
23+
* returns. A rotation landing inside the dump changes the extent between the reads, and the capture
24+
* then says nothing about its box rather than pairing one rotation's bounds with the other's
25+
* dimensions. A display this process cannot address answers with null. Either way the host publishes
26+
* the absence it reads as unknown, and the host's own guard is what refuses a zero extent.
27+
*/
28+
final class DisplayExtent {
29+
private DisplayExtent() {}
30+
31+
/** The extent one read of the display answered with, in the current rotation. */
32+
static final class Extent {
33+
final int width;
34+
final int height;
35+
36+
Extent(int width, int height) {
37+
this.width = width;
38+
this.height = height;
39+
}
40+
}
41+
42+
/**
43+
* The extent of the display {@code context} is attached to, or null when this process could not
44+
* address it. A zero extent is passed through as read: refusing an unusable box is the host's job,
45+
* and {@code snapshotViewportSizeFrom} already refuses one, so a second refusal here would only
46+
* be a second place that decides what a usable display is.
47+
*/
48+
static Extent read(Context context) {
49+
try {
50+
WindowManager windowManager =
51+
(WindowManager) context.getSystemService(Context.WINDOW_SERVICE);
52+
if (windowManager == null) {
53+
return null;
54+
}
55+
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.R) {
56+
Rect bounds = windowManager.getMaximumWindowMetrics().getBounds();
57+
return new Extent(bounds.width(), bounds.height());
58+
}
59+
return legacyExtent(windowManager);
60+
} catch (Throwable error) {
61+
return null;
62+
}
63+
}
64+
65+
@SuppressWarnings("deprecation")
66+
private static Extent legacyExtent(WindowManager windowManager) {
67+
Display display = windowManager.getDefaultDisplay();
68+
DisplayMetrics metrics = new DisplayMetrics();
69+
display.getRealMetrics(metrics);
70+
return new Extent(metrics.widthPixels, metrics.heightPixels);
71+
}
72+
73+
/**
74+
* The extent both reads of one capture agreed on, or null when either read failed or the display
75+
* changed underneath the tree dump. Omitting the viewport on disagreement is what keeps the
76+
* published box a fact about the capture rather than about two different rotations.
77+
*/
78+
static Extent whenBothAgree(Extent before, Extent after) {
79+
if (before == null || after == null) {
80+
return null;
81+
}
82+
if (before.width != after.width || before.height != after.height) {
83+
return null;
84+
}
85+
return before;
86+
}
87+
}

‎android/snapshot-helper/src/main/java/com/callstack/agentdevice/snapshothelper/SnapshotInstrumentation.java‎

Lines changed: 24 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
import android.content.res.Resources;
66
import android.os.Bundle;
77
import android.util.Base64;
8+
import android.util.DisplayMetrics;
89
import java.io.BufferedReader;
910
import java.io.File;
1011
import java.io.FileOutputStream;
@@ -77,14 +78,17 @@ public void onStart() {
7778
return;
7879
}
7980
long startedAtMs = System.currentTimeMillis();
81+
DisplayExtent.Extent beforeDump = DisplayExtent.read(getContext());
8082
AccessibilityTreeCapture.Result capture =
8183
captureXml(waitForIdleQuietMs, waitForIdleTimeoutMs, timeoutMs, maxDepth, maxNodes);
84+
DisplayExtent.Extent display =
85+
DisplayExtent.whenBothAgree(beforeDump, DisplayExtent.read(getContext()));
8286
writeOutputFile(outputPath, capture.xml);
8387
if (emitChunks) {
8488
emitChunks(capture.xml);
8589
}
8690
result.putString("ok", "true");
87-
putCaptureMetadata(result, capture, System.currentTimeMillis() - startedAtMs);
91+
putCaptureMetadata(result, capture, display, System.currentTimeMillis() - startedAtMs);
8892
finishSafely(0, result);
8993
} catch (Throwable error) {
9094
result.putString("ok", "false");
@@ -114,7 +118,10 @@ private static void putBaseMetadata(
114118
}
115119

116120
private static void putCaptureMetadata(
117-
Bundle result, AccessibilityTreeCapture.Result capture, long elapsedMs) {
121+
Bundle result,
122+
AccessibilityTreeCapture.Result capture,
123+
DisplayExtent.Extent display,
124+
long elapsedMs) {
118125
result.putString("rootPresent", Boolean.toString(capture.rootPresent));
119126
result.putString("captureMode", capture.captureMode);
120127
result.putString("windowCount", Integer.toString(capture.windowCount));
@@ -124,8 +131,17 @@ private static void putCaptureMetadata(
124131
// Physical pixels per dp of the display the bounds above were measured on, from the same
125132
// configuration the framework lays out with (a `wm density` override included), so a host that
126133
// works in dp has the factor beside the pixels instead of a second adb round trip.
127-
result.putString(
128-
"pixelDensity", Float.toString(Resources.getSystem().getDisplayMetrics().density));
134+
DisplayMetrics metrics = Resources.getSystem().getDisplayMetrics();
135+
result.putString("pixelDensity", Float.toString(metrics.density));
136+
// The real extent of the display those bounds are measured in, read around the tree dump and
137+
// published only when both reads agreed, so the host can publish the box the bounds live in
138+
// without reading the tree for it. Published by absence: a display this helper could not
139+
// address, or one that rotated mid-dump, adds no keys, which the host reads as unknown rather
140+
// than as a screen of no size or of another rotation's size (#3182).
141+
if (display != null) {
142+
result.putString("displayWidth", Integer.toString(display.width));
143+
result.putString("displayHeight", Integer.toString(display.height));
144+
}
129145
}
130146

131147
private void runOneShotViewport(Bundle result) {
@@ -232,10 +248,13 @@ private void writeSessionSnapshot(
232248
result.putString("requestId", requestId);
233249
try {
234250
long startedAtMs = System.currentTimeMillis();
251+
DisplayExtent.Extent beforeDump = DisplayExtent.read(getContext());
235252
AccessibilityTreeCapture.Result capture =
236253
captureXml(waitForIdleQuietMs, waitForIdleTimeoutMs, timeoutMs, maxDepth, maxNodes);
254+
DisplayExtent.Extent display =
255+
DisplayExtent.whenBothAgree(beforeDump, DisplayExtent.read(getContext()));
237256
result.putString("ok", "true");
238-
putCaptureMetadata(result, capture, System.currentTimeMillis() - startedAtMs);
257+
putCaptureMetadata(result, capture, display, System.currentTimeMillis() - startedAtMs);
239258
result.putString("byteLength", Integer.toString(capture.xml.getBytes(StandardCharsets.UTF_8).length));
240259
SessionResponseWriter.writeSessionResponse(output, result, capture.xml);
241260
} catch (Throwable error) {
Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
package com.callstack.agentdevice.snapshothelper;
2+
3+
public final class DisplayExtentTest {
4+
private DisplayExtentTest() {}
5+
6+
private static final DisplayExtent.Extent PORTRAIT = new DisplayExtent.Extent(1080, 2400);
7+
private static final DisplayExtent.Extent LANDSCAPE = new DisplayExtent.Extent(2400, 1080);
8+
9+
static void run() {
10+
assertTwoReadsOfTheSameDisplayPublishItsExtent();
11+
assertADisplayThatMovedUnderTheDumpPublishesNothing();
12+
assertAReadThatFailedPublishesNothing();
13+
}
14+
15+
private static void assertTwoReadsOfTheSameDisplayPublishItsExtent() {
16+
// The rotation matters: a landscape capture publishes the box its bounds are actually measured
17+
// in, which is why the extent is read per capture rather than kept across a helper session.
18+
assertExtent(
19+
DisplayExtent.whenBothAgree(PORTRAIT, new DisplayExtent.Extent(1080, 2400)),
20+
1080,
21+
2400,
22+
"both reads of one portrait display");
23+
assertExtent(
24+
DisplayExtent.whenBothAgree(LANDSCAPE, LANDSCAPE),
25+
2400,
26+
1080,
27+
"both reads of one landscape display");
28+
}
29+
30+
private static void assertADisplayThatMovedUnderTheDumpPublishesNothing() {
31+
// A rotation landing inside the tree dump would pair that rotation's node bounds with the other
32+
// rotation's dimensions. The capture then names no box at all rather than a wrong one (#3182).
33+
assertNull(
34+
DisplayExtent.whenBothAgree(PORTRAIT, LANDSCAPE), "display rotated mid-dump");
35+
assertNull(
36+
DisplayExtent.whenBothAgree(
37+
PORTRAIT, new DisplayExtent.Extent(1076, 2400)),
38+
"display resized mid-dump");
39+
}
40+
41+
private static void assertAReadThatFailedPublishesNothing() {
42+
// A read that failed on either side of the dump leaves the box unknown: the surviving read says
43+
// nothing about what the tree between the two was measured against.
44+
assertNull(DisplayExtent.whenBothAgree(null, PORTRAIT), "read before the dump failed");
45+
assertNull(DisplayExtent.whenBothAgree(PORTRAIT, null), "read after the dump failed");
46+
assertNull(DisplayExtent.whenBothAgree(null, null), "both reads failed");
47+
}
48+
49+
private static void assertExtent(
50+
DisplayExtent.Extent extent, int width, int height, String label) {
51+
if (extent == null) {
52+
throw new AssertionError("expected an extent for " + label);
53+
}
54+
if (extent.width != width || extent.height != height) {
55+
throw new AssertionError(
56+
"wrong extent for " + label + ": got " + extent.width + "x" + extent.height);
57+
}
58+
}
59+
60+
private static void assertNull(DisplayExtent.Extent extent, String label) {
61+
if (extent != null) {
62+
throw new AssertionError(
63+
"expected no extent for " + label + " but got " + extent.width + "x" + extent.height);
64+
}
65+
}
66+
}

‎android/snapshot-helper/src/test/java/com/callstack/agentdevice/snapshothelper/SnapshotHelperTestSuite.java‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,5 +8,6 @@ public static void main(String[] args) throws Exception {
88
AccessibilityCaptureStabilizerTest.run();
99
BoundedUiAutomationConnectionTest.run();
1010
GestureViewportReaderTest.run();
11+
DisplayExtentTest.run();
1112
}
1213
}
Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
import AppKit
2+
import CoreGraphics
3+
import Foundation
4+
import ScreenCaptureKit
5+
6+
/// Captures the session app's front window by itself: windows above it, the ghost cursor, and
7+
/// the rest of the desktop are not in the image, so the app may sit behind the user's work.
8+
func captureAppWindowScreenshot(app: NSRunningApplication, outPath: String) throws {
9+
guard #available(macOS 14.0, *) else {
10+
throw HelperError.commandFailed("app window screenshots require macOS 14 or newer")
11+
}
12+
let pid = app.processIdentifier
13+
let frontWindowID = frontWindowNumber(pid: pid)
14+
let content = try awaitShareableContent()
15+
let candidates = content.windows.filter {
16+
$0.owningApplication?.processID == pid && $0.windowLayer == 0 && $0.frame.width > 0
17+
&& $0.frame.height > 0
18+
}
19+
guard
20+
let window = candidates.first(where: { Int($0.windowID) == frontWindowID })
21+
?? candidates.max(by: { $0.frame.width * $0.frame.height < $1.frame.width * $1.frame.height })
22+
else {
23+
throw HelperError.commandFailed(
24+
"screenshot could not find a window for the app",
25+
details: ["reason": "window-not-found", "bundleId": app.bundleIdentifier ?? ""]
26+
)
27+
}
28+
29+
let filter = SCContentFilter(desktopIndependentWindow: window)
30+
let configuration = SCStreamConfiguration()
31+
let scale = CGFloat(filter.pointPixelScale)
32+
configuration.width = Int(filter.contentRect.width * scale)
33+
configuration.height = Int(filter.contentRect.height * scale)
34+
configuration.showsCursor = false
35+
configuration.ignoreShadowsSingleWindow = true
36+
37+
let semaphore = DispatchSemaphore(value: 0)
38+
var capturedImage: CGImage?
39+
var capturedError: Error?
40+
SCScreenshotManager.captureImage(contentFilter: filter, configuration: configuration) { image, error in
41+
capturedImage = image
42+
capturedError = error
43+
semaphore.signal()
44+
}
45+
semaphore.wait()
46+
if let error = capturedError as NSError? {
47+
throw screenshotFailure(error, surface: "app")
48+
}
49+
guard let capturedImage else {
50+
throw HelperError.commandFailed("screenshot failed")
51+
}
52+
try writePNG(capturedImage, to: outPath)
53+
}
54+
55+
@available(macOS 14.0, *)
56+
private func awaitShareableContent() throws -> SCShareableContent {
57+
let semaphore = DispatchSemaphore(value: 0)
58+
var shareable: SCShareableContent?
59+
var failure: Error?
60+
SCShareableContent.getExcludingDesktopWindows(true, onScreenWindowsOnly: true) { content, error in
61+
shareable = content
62+
failure = error
63+
semaphore.signal()
64+
}
65+
semaphore.wait()
66+
if let error = failure as NSError? {
67+
throw screenshotFailure(error, surface: "app")
68+
}
69+
guard let shareable else {
70+
throw HelperError.commandFailed("screenshot could not list shareable windows")
71+
}
72+
return shareable
73+
}

0 commit comments

Comments
 (0)