Skip to content

Commit ab1f758

Browse files
committed
fix(settle): let the app-accessory exemption outrank the owned-role chrome walk
An app's `UITextField.inputAssistantItem` bar buttons render under the system assistant view, above the keys; the window pass exempted them and the owned-role subtree walk re-claimed them, hiding real app controls from the settle diff. An exempted node now re-joins chrome only when it declares a keyboard-owned role itself, so a system prediction cell still collapses while an app button under the same host survives. Also from review: hoist the chrome-free replacement projection behind a baseline/candidate-change guard so the 300ms poll of an unchanged screen pays no second tree pass, and count node 31 in the fixture breakdown.
1 parent e2ad899 commit ab1f758

4 files changed

Lines changed: 173 additions & 33 deletions

File tree

‎packages/capture-kit/src/snapshot-chrome.fixtures.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ import type { RawSnapshotNode } from '@agent-device/kernel/snapshot';
1010
* enabled) is verbatim.
1111
*
1212
* The keyboard surfaces arrive flattened beside app content: app nodes 0-30,
13+
* the floating home-indicator toolbar the window capture also reports (31),
1314
* the system `inputView` host frame (32), the QuickType assistant bar (33-39),
1415
* the `[Keyboard]` plane (40) with its layout wrapper (41) and keys (42-79),
1516
* and the dock's "Next keyboard" / "Dictate" buttons (80-82). This is exactly

‎packages/capture-kit/src/snapshot-chrome.test.ts‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -325,3 +325,82 @@ test('editable text hosted inside a keyboard-owned subtree survives classificati
325325
assert.equal(chromeRefs.has(refForIdentifier(nodes, 'SystemInputAssistantView')), true);
326326
assert.equal(chromeRefs.has(refFor(nodes, 'Compose')), false);
327327
});
328+
329+
// The window pass exempts app content above the keys, and the keyboard-owned subtree walk must
330+
// not outrank that exemption: an app's UITextField.inputAssistantItem bar button renders UNDER
331+
// the system assistant view and declares no keyboard-owned role, so it stays an app control —
332+
// while a system prediction cell beside it, which owns its chrome by role wherever it sits,
333+
// still collapses.
334+
test('app inputAssistantItem buttons under the assistant view keep the accessory exemption', () => {
335+
const nodes = attachRefs([
336+
{
337+
index: 0,
338+
depth: 0,
339+
type: 'Application',
340+
label: 'Example',
341+
rect: { x: 0, y: 0, width: 402, height: 874 },
342+
},
343+
{
344+
index: 1,
345+
depth: 1,
346+
parentIndex: 0,
347+
type: 'Window',
348+
label: 'Keyboard Window',
349+
rect: { x: 0, y: 300, width: 402, height: 574 },
350+
},
351+
{
352+
index: 2,
353+
depth: 2,
354+
parentIndex: 1,
355+
type: 'Other',
356+
role: 'TUISystemInputAssistantView',
357+
label: 'Assistant Bar',
358+
rect: { x: 0, y: 300, width: 402, height: 44 },
359+
},
360+
{
361+
index: 3,
362+
depth: 3,
363+
parentIndex: 2,
364+
type: 'Button',
365+
label: 'Paste',
366+
rect: { x: 8, y: 306, width: 60, height: 32 },
367+
hittable: true,
368+
},
369+
{
370+
index: 4,
371+
depth: 3,
372+
parentIndex: 2,
373+
type: 'Button',
374+
role: 'TUIPredictionViewCell',
375+
label: 'keep',
376+
rect: { x: 200, y: 306, width: 60, height: 32 },
377+
hittable: true,
378+
},
379+
{
380+
index: 5,
381+
depth: 2,
382+
parentIndex: 1,
383+
type: 'Keyboard',
384+
label: 'Keyboard',
385+
rect: { x: 0, y: 360, width: 402, height: 514 },
386+
},
387+
{
388+
index: 6,
389+
depth: 3,
390+
parentIndex: 5,
391+
type: 'Key',
392+
label: 'q',
393+
rect: { x: 0, y: 380, width: 39, height: 54 },
394+
},
395+
]);
396+
const chromeRefs = collectSettleChromeRefs(nodes, undefined);
397+
398+
assert.equal(
399+
chromeRefs.has(refFor(nodes, 'Paste')),
400+
false,
401+
'the app-provided assistant-item button is exempt, not re-claimed through the assistant subtree',
402+
);
403+
assert.equal(chromeRefs.has(refFor(nodes, 'keep')), true);
404+
assert.equal(chromeRefs.has(refFor(nodes, 'Assistant Bar')), true);
405+
assert.equal(chromeRefs.has(refFor(nodes, 'q')), true);
406+
});

‎packages/capture-kit/src/snapshot-chrome.ts‎

Lines changed: 63 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -143,24 +143,31 @@ function collectKeyboardChrome(nodes: SnapshotNode[]): SettleChrome {
143143
}
144144
// Preserve app-owned accessory controls hosted above the keyboard: they were
145145
// swept in by the whole-window pass but are not keyboard chrome.
146-
for (const index of collectKeyboardAccessoryIndexes(
146+
const accessoryExemptIndexes = collectKeyboardAccessoryIndexes(
147147
nodes,
148148
byIndex,
149149
windowIndexes,
150150
containerIndexes,
151151
containerSubtreeIndexes,
152-
)) {
152+
);
153+
for (const index of accessoryExemptIndexes) {
153154
chromeIndexes.delete(index);
154155
}
155156
// Keyboard-owned classes the projection flattened out of every reachable subtree are chrome
156-
// by their own declaration, so the app-accessory guard above never exempts them. An editable
157-
// text node inside one of them is the same inputAccessoryView guard in disguise: an app that
158-
// hosts a text field in its own input view keeps it, as it does under the window pass.
157+
// by their own declaration — but the walk may NOT outrank the app-accessory exemption above:
158+
// an app's `UITextField.inputAssistantItem` bar buttons render UNDER the system assistant
159+
// view, above the keys, and the window pass just exempted them; re-claiming them through the
160+
// assistant's subtree would hide real app controls from the settle diff. An exempted node
161+
// only re-joins chrome when IT declares a keyboard-owned role (a prediction cell is
162+
// system-owned wherever it sits); an app node under an owned host stays exempt. In the
163+
// flattened interactive shape there is no window anchor, the exemption is empty, and the
164+
// walk claims the key plane exactly as the container-descendant walk always did.
159165
for (const index of collectOwnedKeyboardChromeIndexes(
160166
nodes,
161167
byIndex,
162168
ownedSubtreeRootIndexes,
163169
containerSubtreeIndexes,
170+
accessoryExemptIndexes,
164171
)) {
165172
chromeIndexes.add(index);
166173
}
@@ -179,39 +186,69 @@ function collectKeyboardChrome(nodes: SnapshotNode[]): SettleChrome {
179186
}
180187

181188
/**
182-
* The subtree under each keyboard-owned-class root, minus editable text the app hosts inside it
183-
* outside the keyboard container. A keyboard's own `Key` nodes never trip that guard — the walk
184-
* claims them with their own subtree roots even when a stripped field re-parented them — and an
185-
* app that hosts its whole composer inside its input view still loses every key, exactly as the
186-
* container-descendant walk always did.
189+
* The subtree under each keyboard-owned-class root, minus two app-content guards: indexes the
190+
* app-accessory exemption already granted (an app's `inputAssistantItem` bar buttons ride the
191+
* system assistant view above the keys), and editable text the app hosts inside an owned root
192+
* outside the keyboard container, with its subtree. A keyboard's own `Key` nodes trip neither —
193+
* they sit inside the container subtree, which the accessory pass never exempts — and the
194+
* flattened interactive shape carries no window anchor, so the exemption is empty there and the
195+
* walk claims the whole key plane exactly as the container-descendant walk always did.
187196
*/
188197
function collectOwnedKeyboardChromeIndexes(
189198
nodes: SnapshotNode[],
190199
byIndex: Map<number, SnapshotNode>,
191200
ownedSubtreeRootIndexes: ReadonlySet<number>,
192201
containerSubtreeIndexes: ReadonlySet<number>,
202+
accessoryExemptIndexes: ReadonlySet<number>,
193203
): Set<number> {
194204
const chromeIndexes = collectSubtreeIndexes(nodes, byIndex, ownedSubtreeRootIndexes);
195-
for (const node of nodes) {
196-
if (
197-
chromeIndexes.has(node.index) &&
198-
EDITABLE_TEXT_TYPES.has(normalizeType(node.type ?? '')) &&
199-
!containerSubtreeIndexes.has(node.index)
200-
) {
201-
chromeIndexes.delete(node.index);
202-
for (const descendant of nodes) {
203-
if (
204-
chromeIndexes.has(descendant.index) &&
205-
hasAncestorIn(descendant, byIndex, new Set([node.index]))
206-
) {
207-
chromeIndexes.delete(descendant.index);
208-
}
209-
}
210-
}
205+
// An exempted node re-joins chrome only when IT declares a keyboard-owned role: a system
206+
// prediction cell is chrome wherever it sits, while an app's `inputAssistantItem` bar button
207+
// under the assistant view declares no owned role and keeps its exemption.
208+
for (const index of accessoryExemptIndexes) {
209+
if (!ownedSubtreeRootIndexes.has(index)) chromeIndexes.delete(index);
210+
}
211+
for (const index of collectHostedEditableTextIndexes(
212+
nodes,
213+
byIndex,
214+
chromeIndexes,
215+
containerSubtreeIndexes,
216+
)) {
217+
chromeIndexes.delete(index);
211218
}
212219
return chromeIndexes;
213220
}
214221

222+
/**
223+
* Editable text the app hosts under an owned root outside the keyboard container, and the
224+
* chrome claimed beneath it — the inputAccessoryView guard in disguise: an app that rides its
225+
* own composer on the assistant host keeps the field it is typing into.
226+
*/
227+
function collectHostedEditableTextIndexes(
228+
nodes: SnapshotNode[],
229+
byIndex: Map<number, SnapshotNode>,
230+
chromeIndexes: ReadonlySet<number>,
231+
containerSubtreeIndexes: ReadonlySet<number>,
232+
): Set<number> {
233+
const dropped = new Set<number>();
234+
const hostedTextIndexes = new Set(
235+
nodes
236+
.filter(
237+
(node) =>
238+
chromeIndexes.has(node.index) &&
239+
EDITABLE_TEXT_TYPES.has(normalizeType(node.type ?? '')) &&
240+
!containerSubtreeIndexes.has(node.index),
241+
)
242+
.map((node) => node.index),
243+
);
244+
for (const node of nodes) {
245+
if (!chromeIndexes.has(node.index)) continue;
246+
if (hasAncestorIn(node, byIndex, hostedTextIndexes)) dropped.add(node.index);
247+
}
248+
for (const index of hostedTextIndexes) dropped.add(index);
249+
return dropped;
250+
}
251+
215252
/**
216253
* Indexes of app-owned accessory content in a keyboard window: non-keyboard
217254
* nodes whose center sits ABOVE the keyboard container's top edge (an

‎src/commands/interaction/runtime/stable-capture.ts‎

Lines changed: 30 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -92,6 +92,35 @@ export async function runStableCaptureLoop(
9292
let lastCapture: CapturedSnapshot | undefined;
9393
let quietSinceMs = start;
9494
let requiredQuietMs = quietMs;
95+
let lastQuietWindowBaseline: StableCaptureSignal | undefined;
96+
let lastQuietWindowSignal: StableCaptureSignal | undefined;
97+
// The broad-replacement verdict is a pure function of the armed baseline, the requested
98+
// window, and this candidate signal, and the window it sets only latches upward — so the
99+
// verdict cannot change while both inputs hold. Rebuilding the chrome-free projection (a
100+
// full withoutSettleChrome pass plus a second signal) is owed on baseline arming (it can
101+
// land after the first capture) and on candidate change, not on every 300ms poll of a
102+
// quiet screen.
103+
const nextQuietWindow = (
104+
snapshot: Parameters<typeof stableCaptureSignal>[0],
105+
baseline: StableCaptureSignal | undefined,
106+
signal: StableCaptureSignal,
107+
): number => {
108+
if (
109+
baseline === lastQuietWindowBaseline &&
110+
stableCaptureSignalsEqual(lastQuietWindowSignal, signal)
111+
) {
112+
return requiredQuietMs;
113+
}
114+
lastQuietWindowBaseline = baseline;
115+
lastQuietWindowSignal = signal;
116+
return stableCaptureNextQuietWindowMs({
117+
baseline,
118+
snapshot,
119+
appBundleId: session?.appBundleId,
120+
requestedQuietMs: quietMs,
121+
currentQuietMs: requiredQuietMs,
122+
});
123+
};
95124
while (now(runtime) < deadlineMs) {
96125
const capture = await captureStableSignalWithinDeadline(
97126
runtime,
@@ -118,13 +147,7 @@ export async function runStableCaptureLoop(
118147
session?.appBundleId,
119148
);
120149
const signal = stableCaptureSignal(capture.snapshot);
121-
requiredQuietMs = stableCaptureNextQuietWindowMs({
122-
baseline: transitionBaseline,
123-
snapshot: capture.snapshot,
124-
appBundleId: session?.appBundleId,
125-
requestedQuietMs: quietMs,
126-
currentQuietMs: requiredQuietMs,
127-
});
150+
requiredQuietMs = nextQuietWindow(capture.snapshot, transitionBaseline, signal);
128151
preferredBackend ??= preferredSnapshotBackendForVerdict(capture.snapshot.snapshotQuality);
129152
const nowMs = now(runtime);
130153
const recoveredDeadlineMs = extendedDeadlineAfterPrivateAxRecovery({

0 commit comments

Comments
 (0)