Skip to content

Commit 507ef23

Browse files
authored
perf(android): bound target readiness and confirm moved clears (#3815)
* perf(android): measure target readiness and reuse helper app control * fix(android): retain lifecycle fences and verified launch status * fix(android): preserve launch output across helper transport * fix(android): keep reconciled targets in their application * fix(android): preserve launcher intent semantics * refactor(android): isolate launch and observation reconciliation * test(android): scroll checkout replay to its submitted summary * docs(evidence): record reliable Android readiness gains * docs(evidence): record Android startup safety experiments * docs(evidence): clarify checksummed archive file count * fix(android): confirm moved empty fields by captured identity * fix(android): require complete fresh captures for empty-target recovery * fix(android): anchor clear verification to unique pre-action identity * docs(evidence): record edited-field experiments and guarded clear verification * refactor(android): separate empty-target corroboration from point scanning * docs(evidence): pin post-review verifier validation * test(android): cover helper app admission and launch protocol * fix(android): normalize package before helper launch admission * test(android): cover moved-clear proof through native fill * test(android): isolate moved-clear helper installation probe * fix(android): keep helper admission within native package grammar
1 parent 1c3edff commit 507ef23

49 files changed

Lines changed: 2423 additions & 73 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎android/snapshot-helper/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -162,3 +162,7 @@ The release manifest is a stable provider contract for the current helper protoc
162162
resolve the APK from `apkUrl`, verify `sha256`, install it with `adb install -r <apk>`, and run
163163
`instrumentationRunner`. Manifests up to 0.21.3 also carried an `installArgs` array; it only ever
164164
spelled `install -r` (plus `-t` while the helper was `testOnly`), so consumers can ignore it.
165+
166+
Capture metadata reports `cacheInvalidated` when the capture cleared the platform node cache through `UiAutomation.clearCache()`. A complete capture with that evidence need not change routes after a control press. Incomplete captures and captures without that evidence retain freshness recovery.
167+
168+
Warm application open commands use the same automation session. Close retains the existing foreground and process-absence checks. A failed application command is never resent.

‎android/snapshot-helper/src/main/java/com/callstack/agentdevice/snapshothelper/AccessibilityTreeCapture.java‎

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -36,13 +36,13 @@ static void prepare(UiAutomation automation, long quietMs, long timeoutMs)
3636
static Result capture(
3737
UiAutomation automation, int maxDepth, int maxNodes, long stabilizationTimeoutMs)
3838
throws InterruptedException, AccessibilityCaptureStabilizer.IncompleteCaptureException {
39-
clearAccessibilityCache(automation);
4039
return AccessibilityCaptureStabilizer.capture(
4140
() -> captureOnce(automation, maxDepth, maxNodes), stabilizationTimeoutMs);
4241
}
4342

4443
@SuppressWarnings("deprecation")
4544
private static Result captureOnce(UiAutomation automation, int maxDepth, int maxNodes) {
45+
boolean cacheInvalidated = clearAccessibilityCache(automation);
4646
AccessibilityTreeXml.Stats stats = new AccessibilityTreeXml.Stats();
4747
StringBuilder xml = new StringBuilder();
4848
xml.append("<?xml version='1.0' encoding='UTF-8' standalone='yes' ?>");
@@ -87,27 +87,28 @@ private static Result captureOnce(UiAutomation automation, int maxDepth, int max
8787
windowCount,
8888
stats.nodeCount,
8989
stats.truncated,
90-
missingRootWindowTypes);
90+
missingRootWindowTypes,
91+
cacheInvalidated);
9192
}
9293

93-
static void clearAccessibilityCache(UiAutomation automation) {
94+
static boolean clearAccessibilityCache(UiAutomation automation) {
9495
// Navigation 3 and any in-place content swap that keeps the same window/Activity render every
9596
// destination inside one AndroidComposeView. A persistent helper session reuses a single
9697
// UiAutomation connection across captures, and its per-connection accessibility node cache is
9798
// not always invalidated by such a swap. Drop the cache before each traversal.
9899
//
99100
// Declared residue (agent-device #1832): the reset mechanism differs by API level — the public
100101
// clearCache() on 34+, a setServiceInfo() re-apply below — and the two are not known to be
101-
// equivalent. This is an acquisition-freshness difference the payload does not disclose.
102+
// equivalent. Only the public cache reset establishes cacheInvalidated evidence.
102103
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.UPSIDE_DOWN_CAKE) {
103104
try {
104-
automation.clearCache();
105-
return;
105+
return automation.clearCache();
106106
} catch (RuntimeException ignored) {
107107
// Fall back to the setServiceInfo() flush below if the platform rejects the public API.
108108
}
109109
}
110110
clearAccessibilityCacheViaServiceInfo(automation);
111+
return false;
111112
}
112113

113114
private static void clearAccessibilityCacheViaServiceInfo(UiAutomation automation) {
@@ -269,6 +270,7 @@ static final class Result implements AccessibilityCaptureStabilizer.Capture {
269270
final int windowCount;
270271
final int nodeCount;
271272
final boolean truncated;
273+
final boolean cacheInvalidated;
272274
// AccessibilityWindowInfo types of listed windows whose root read null or threw, so the host
273275
// can tell a window it could not read (an input method, say) from one that was not on screen.
274276
final List<Integer> missingRootWindowTypes;
@@ -281,14 +283,16 @@ static final class Result implements AccessibilityCaptureStabilizer.Capture {
281283
int windowCount,
282284
int nodeCount,
283285
boolean truncated,
284-
List<Integer> missingRootWindowTypes) {
286+
List<Integer> missingRootWindowTypes,
287+
boolean cacheInvalidated) {
285288
this.xml = xml;
286289
this.rootPresent = rootPresent;
287290
this.foregroundWindowRootsPresent = foregroundWindowRootsPresent;
288291
this.captureMode = captureMode;
289292
this.windowCount = windowCount;
290293
this.nodeCount = nodeCount;
291294
this.truncated = truncated;
295+
this.cacheInvalidated = cacheInvalidated;
292296
this.missingRootWindowTypes = missingRootWindowTypes;
293297
}
294298

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
package com.callstack.agentdevice.snapshothelper;
2+
3+
import android.app.UiAutomation;
4+
import android.os.Bundle;
5+
import android.util.Base64;
6+
import android.os.ParcelFileDescriptor;
7+
import java.io.ByteArrayOutputStream;
8+
import java.io.InputStream;
9+
import java.io.OutputStream;
10+
import java.nio.charset.StandardCharsets;
11+
import org.json.JSONObject;
12+
13+
final class ApplicationCommandHandler {
14+
private ApplicationCommandHandler() {}
15+
16+
static void execute(UiAutomation automation, JSONObject input, Bundle result) throws Exception {
17+
String packageName = input.getString("packageName");
18+
if (!packageName.matches("[A-Za-z][A-Za-z0-9_]*(\\.[A-Za-z][A-Za-z0-9_]*)+")) {
19+
throw new IllegalArgumentException("Invalid application package");
20+
}
21+
switch (input.getString("kind")) {
22+
case "open":
23+
open(automation, packageName, result);
24+
return;
25+
default:
26+
throw new IllegalArgumentException("Unknown application operation");
27+
}
28+
}
29+
30+
private static void open(UiAutomation automation, String packageName, Bundle result) throws Exception {
31+
String resolved = shell(automation, "cmd package resolve-activity --brief -a android.intent.action.MAIN -c android.intent.category.LAUNCHER " + packageName).trim();
32+
String[] lines = resolved.split("\\n");
33+
String component = lines[lines.length - 1].trim();
34+
if (!component.matches(java.util.regex.Pattern.quote(packageName) + "/[A-Za-z_.$][A-Za-z0-9_.$]*")) {
35+
result.putString("componentUnavailable", "true");
36+
return;
37+
}
38+
String output = shell(automation, "am start -W -a android.intent.action.MAIN -c android.intent.category.DEFAULT -c android.intent.category.LAUNCHER -n '" + component + "'");
39+
result.putString("launchOutputBase64", Base64.encodeToString(output.getBytes(StandardCharsets.UTF_8), Base64.NO_WRAP));
40+
}
41+
42+
private static String shell(UiAutomation automation, String command) throws Exception {
43+
ParcelFileDescriptor[] pipes = automation.executeShellCommandRw("sh");
44+
if (pipes == null || pipes.length != 2) throw new IllegalStateException("Application shell command was unavailable");
45+
try (InputStream stream = new ParcelFileDescriptor.AutoCloseInputStream(pipes[0]);
46+
OutputStream input = new ParcelFileDescriptor.AutoCloseOutputStream(pipes[1]);
47+
ByteArrayOutputStream bytes = new ByteArrayOutputStream()) {
48+
input.write(("exec 2>&1\n" + command + "\n").getBytes(StandardCharsets.UTF_8));
49+
input.close();
50+
byte[] buffer = new byte[4096];
51+
int count;
52+
while ((count = stream.read(buffer)) != -1) {
53+
if (bytes.size() + count > 65536) throw new IllegalStateException("Application command output exceeded its limit");
54+
bytes.write(buffer, 0, count);
55+
}
56+
return bytes.toString(StandardCharsets.UTF_8.name());
57+
}
58+
}
59+
}

‎android/snapshot-helper/src/main/java/com/callstack/agentdevice/snapshothelper/AutomationCommandHandler.java‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,9 @@ static void writeSessionResponse(
2828
Bundle result = SessionResponseWriter.sessionResponseBundle(requestId);
2929
InputDispatch dispatch = new InputDispatch();
3030
try {
31-
if ("screenshot".equals(action)) {
31+
if ("app".equals(action)) {
32+
ApplicationCommandHandler.execute(automation, new JSONObject(new String(Base64.decode(payload, Base64.DEFAULT), StandardCharsets.UTF_8)), result);
33+
} else if ("screenshot".equals(action)) {
3234
screenshot(result, automation);
3335
} else if ("focus".equals(action)) {
3436
focus(result, automation);

‎android/snapshot-helper/src/main/java/com/callstack/agentdevice/snapshothelper/SnapshotInstrumentation.java‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -122,6 +122,7 @@ private static void putCaptureMetadata(
122122
DisplayExtent.Extent display,
123123
long elapsedMs) {
124124
result.putString("rootPresent", Boolean.toString(capture.rootPresent));
125+
result.putString("cacheInvalidated", Boolean.toString(capture.cacheInvalidated));
125126
result.putString("captureMode", capture.captureMode);
126127
result.putString("windowCount", Integer.toString(capture.windowCount));
127128
result.putString(
@@ -180,7 +181,8 @@ private void runSnapshotSession(
180181
putBaseMetadata(
181182
ready, waitForIdleTimeoutMs, waitForIdleQuietMs, timeoutMs, maxDepth, maxNodes);
182183
ready.putString("sessionReady", "true");
183-
ready.putString("sessionCommands", "input,screenshot,focus,snapshot-immediate");
184+
ready.putString("sessionCommands", "input,screenshot,focus,snapshot-immediate"
185+
+ (android.os.Build.VERSION.SDK_INT >= 31 ? ",app" : ""));
184186
ready.putString("sessionPort", Integer.toString(sessionPort));
185187
sendStatus(2, ready);
186188

@@ -202,7 +204,7 @@ private void runSnapshotSession(
202204
SessionResponseWriter.writeSessionOk(socket.getOutputStream(), requestId);
203205
return;
204206
}
205-
if ("input".equals(action) || "screenshot".equals(action) || "focus".equals(action)) {
207+
if ("input".equals(action) || "screenshot".equals(action) || "focus".equals(action) || "app".equals(action)) {
206208
AutomationCommandHandler.writeSessionResponse(
207209
socket.getOutputStream(), requestId, action,
208210
parts.length > 2 ? parts[2] : "", getConnectedUiAutomationUnchecked());

‎docs/adr/0011-interaction-guarantee-contract.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ Completed process history is in git history.
1717
- The Apple runner does not resend a mutating command whose first send may have run.
1818
- Multiple mutating selector matches collapse only if they form one ancestor–descendant chain and every member resolves to the same actionable node. Otherwise, the mutation fails with `AMBIGUOUS_MATCH`. Maestro's explicit expected-point/non-hittable compatibility path stays separate.
1919
- Keep synthesized iOS gesture policy outside the element dispatch-path matrix.
20+
- On Android, the press/click moving-target guard samples fresh geometry until the target remains on one surface at one position for 100 ms, within a 1,500 ms transition window. A target that cannot establish that evidence is refused. Other platforms retain their existing guard schedule.
2021

2122
## Context
2223

‎docs/adr/0014-session-ref-frame-lifetime.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -194,6 +194,8 @@ A ref-targeting mutation is admitted only when all of these conditions hold:
194194

195195
The target is resolved against the authorized frame, never against a newer operational observation by positional coincidence.
196196
Android freshness captures may verify current identity or provide comparison evidence.
197+
On Android, an identifier that names exactly one same-local-identity node in both the authorized frame and the observation can carry fresh geometry across a positional reindex when both nodes name the same application package.
198+
This does not change the frame namespace or authorize a different identity.
197199
But they must not redefine which node an already-admitted `@eN` authorizes.
198200

199201
Read-only ref consumers remain fail-open with a structured staleness warning only when the retained frame contains the evidence necessary to resolve that ref.

0 commit comments

Comments
 (0)