Skip to content

Commit 0b5e645

Browse files
committed
fix(snapshot): preserve keyboard evidence and ownership boundaries
1 parent 07ad69b commit 0b5e645

11 files changed

Lines changed: 354 additions & 60 deletions

File tree

‎contracts/fixtures/tap-keyboard-occlusion-policy.json‎

Lines changed: 243 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{
2-
"description": "Golden table for keyboard occlusion on interaction tap points (#2589), consumed by packages/contracts/src/tap-keyboard-occlusion.ts. A visible software keyboard is its own system surface: it is never a covering sibling of app content, so the same-window occlusion classifier cannot see it, and it leaves the app window rect intact, so the viewport rule passes a covered tab bar. The band runs from the topmost keyboard node down to the bottom of the viewport, across the keyboard's own columns; the tap point is the point the interaction dispatches: the rect center the platform aims at for a ref handed over by name, or the touch point a parent with interactive descendants resolves instead. `nodes` are captured snapshot nodes as they appear in the projection the interaction resolved against — note that regular and interactive projections prune the keyboard's own container, leaving Key nodes beside app chrome. Three rules decide whether reported geometry may be measured at all. Docking: the band is derived only where the keyboard's own reported geometry reaches the viewport's bottom edge within 80 units — measured on iPhone 17 Pro (26.2) the reported key plane stops 58 pt short of an 874 pt portrait viewport and lands exactly on a 402 pt landscape one, and on iPad Pro 11-inch (M4) 3 pt short of a 1210 pt viewport — so an iPad floating keyboard, an app-drawn keypad, or an IME that stops above its own navigation bar stays unmeasured and fails open. The budget is counted in the same units as the rects it guards, which is what a measured Android IME asks for rather than what a dp table would give: with three-button navigation on Pixel 7 (Android 16, 1080x2400 at 420 dpi), Gboard's reported region runs to the physical display bottom at 2400 while the 126 px navigation bar occupies 2274..2400 inside it, so no Android keyboard measured so far needs slack for the bar. Width: geometry that comes back taller than it is wide did not arrive in the app's orientation space, i.e. it came from a capture that normalized no space — the producers that can name the app's interface orientation either turn it back (the runner's tree tiers) or refuse the capture (the Simulator AX bridge) — and is refused measurement for the same reason. Tiling: the keys themselves must form one unbroken run of columns, because a split iPad keyboard docks both clusters at the bottom edge while its surface container still spans the whole screen, and the app content between the clusters is visible and tappable. Reported seams are zero on every measured tree — iPhone 17 Pro's reported row adjoins across 395 pt, iPad Pro 11-inch's across 743.5 pt, and Gboard's across 1070 px — so the allowance covers rounding rather than layout, and a gapped band is refused measurement like the two above. That rule is deliberately blunt in both directions, and the last two cases name the trade: a sparse layout such as the iOS emoji panel reports corner keys over a wide container, does not tile, and reads `undetermined` — the guard abstains there rather than inventing which gaps are app content; and a projection that reports the container with no keys at all has nothing left to check, so the band is the container's own claim. Every captured iOS and Android keyboard measured so far reports its keys, so the second shape is the harness's, not a captured one. `target` names either a resolved node index or a bare coordinate point; both paths read one decision and choose their own consequence. A case that reaches a band declares its whole shape — the frame the refusal and the disclosure both quote, and the control rects that excuse a bare coordinate, which is where plane exclusion shows up. Provider facts distinguish measured visible frames, derived docked-tree bands, visibility-only probes, explicit absence, and unknown. Missing keyboard nodes are undetermined, never absence. A carried derived band was established before projection and stays usable after projection removes keyboard nodes; it obeys the same docking/orientation/tiling rules at acquisition. A measured floating frame needs no bottom extension.",
2+
"description": "Golden table for keyboard occlusion on interaction tap points (#2589), consumed by packages/contracts/src/tap-keyboard-occlusion.ts. A visible software keyboard is its own system surface: it is never a covering sibling of app content, so the same-window occlusion classifier cannot see it, and it leaves the app window rect intact, so the viewport rule passes a covered tab bar. The band runs from the topmost keyboard node down to the bottom of the viewport, across the keyboard's own columns; the tap point is the point the interaction dispatches: the rect center the platform aims at for a ref handed over by name, or the touch point a parent with interactive descendants resolves instead. `nodes` are captured snapshot nodes as they appear in the projection the interaction resolved against \u2014 note that regular and interactive projections prune the keyboard's own container, leaving Key nodes beside app chrome. Three rules decide whether reported geometry may be measured at all. Docking: the band is derived only where the keyboard's own reported geometry reaches the viewport's bottom edge within 80 units \u2014 measured on iPhone 17 Pro (26.2) the reported key plane stops 58 pt short of an 874 pt portrait viewport and lands exactly on a 402 pt landscape one, and on iPad Pro 11-inch (M4) 3 pt short of a 1210 pt viewport \u2014 so an iPad floating keyboard, an app-drawn keypad, or an IME that stops above its own navigation bar stays unmeasured and fails open. The budget is counted in the same units as the rects it guards, which is what a measured Android IME asks for rather than what a dp table would give: with three-button navigation on Pixel 7 (Android 16, 1080x2400 at 420 dpi), Gboard's reported region runs to the physical display bottom at 2400 while the 126 px navigation bar occupies 2274..2400 inside it, so no Android keyboard measured so far needs slack for the bar. Width: geometry that comes back taller than it is wide did not arrive in the app's orientation space, i.e. it came from a capture that normalized no space \u2014 the producers that can name the app's interface orientation either turn it back (the runner's tree tiers) or refuse the capture (the Simulator AX bridge) \u2014 and is refused measurement for the same reason. Tiling: the keys themselves must form one unbroken run of columns, because a split iPad keyboard docks both clusters at the bottom edge while its surface container still spans the whole screen, and the app content between the clusters is visible and tappable. Reported seams are zero on every measured tree \u2014 iPhone 17 Pro's reported row adjoins across 395 pt, iPad Pro 11-inch's across 743.5 pt, and Gboard's across 1070 px \u2014 so the allowance covers rounding rather than layout, and a gapped band is refused measurement like the two above. That rule is deliberately blunt in both directions, and the last two cases name the trade: a sparse layout such as the iOS emoji panel reports corner keys over a wide container, does not tile, and reads `undetermined` \u2014 the guard abstains there rather than inventing which gaps are app content; and a projection that reports the container with no keys at all has nothing left to check, so the band is the container's own claim. Every captured iOS and Android keyboard measured so far reports its keys, so the second shape is the harness's, not a captured one. `target` names either a resolved node index or a bare coordinate point; both paths read one decision and choose their own consequence. A case that reaches a band declares its whole shape \u2014 the frame the refusal and the disclosure both quote, and the control rects that excuse a bare coordinate, which is where plane exclusion shows up. Provider facts distinguish measured visible frames, derived docked-tree bands, visibility-only probes, explicit absence, and unknown. Missing keyboard nodes are undetermined, never absence. A carried derived band was established before projection and stays usable after projection removes keyboard nodes; it obeys the same docking/orientation/tiling rules at acquisition. A measured floating frame needs no bottom extension.",
33
"constants": {
44
"occlusionReason": "tap_keyboard_occludes_target"
55
},
@@ -4095,7 +4095,8 @@
40954095
},
40964096
"expected": {
40974097
"kind": "no-keyboard"
4098-
}
4098+
},
4099+
"observationVisibility": false
40994100
},
41004101
{
41014102
"name": "an unknown provider observation with no keyboard nodes never establishes absence",
@@ -4358,6 +4359,246 @@
43584359
"expected": {
43594360
"kind": "undetermined"
43604361
}
4362+
},
4363+
{
4364+
"name": "a docked provider tree outweighs a disagreeing negative visibility probe",
4365+
"viewport": {
4366+
"x": 0,
4367+
"y": 0,
4368+
"width": 390,
4369+
"height": 844
4370+
},
4371+
"nodes": [
4372+
{
4373+
"index": 0,
4374+
"type": "Keyboard",
4375+
"rect": {
4376+
"x": 0,
4377+
"y": 580,
4378+
"width": 390,
4379+
"height": 264
4380+
}
4381+
},
4382+
{
4383+
"index": 1,
4384+
"type": "Button",
4385+
"label": "Save draft",
4386+
"rect": {
4387+
"x": 200,
4388+
"y": 600,
4389+
"width": 100,
4390+
"height": 40
4391+
}
4392+
}
4393+
],
4394+
"target": {
4395+
"index": 1
4396+
},
4397+
"expected": {
4398+
"kind": "occluded",
4399+
"frame": {
4400+
"x": 0,
4401+
"y": 580,
4402+
"width": 390,
4403+
"height": 264
4404+
},
4405+
"controlRects": [
4406+
{
4407+
"x": 0,
4408+
"y": 580,
4409+
"width": 390,
4410+
"height": 264
4411+
}
4412+
]
4413+
},
4414+
"observationVisibility": false,
4415+
"keyboard": {
4416+
"kind": "derived",
4417+
"derivation": "docked-tree",
4418+
"frame": {
4419+
"x": 0,
4420+
"y": 580,
4421+
"width": 390,
4422+
"height": 264
4423+
}
4424+
}
4425+
},
4426+
{
4427+
"name": "app content under _UIRemoteKeyboardPlaceholderView inside a Keyboard is refused",
4428+
"viewport": {
4429+
"x": 0,
4430+
"y": 0,
4431+
"width": 390,
4432+
"height": 844
4433+
},
4434+
"nodes": [
4435+
{
4436+
"index": 0,
4437+
"type": "Keyboard",
4438+
"rect": {
4439+
"x": 0,
4440+
"y": 580,
4441+
"width": 390,
4442+
"height": 264
4443+
}
4444+
},
4445+
{
4446+
"index": 1,
4447+
"type": "Button",
4448+
"label": "Save draft",
4449+
"rect": {
4450+
"x": 200,
4451+
"y": 600,
4452+
"width": 100,
4453+
"height": 40
4454+
},
4455+
"parentIndex": 2
4456+
},
4457+
{
4458+
"index": 2,
4459+
"type": "Other",
4460+
"role": "_UIRemoteKeyboardPlaceholderView",
4461+
"parentIndex": 0
4462+
}
4463+
],
4464+
"target": {
4465+
"index": 1
4466+
},
4467+
"expected": {
4468+
"kind": "occluded",
4469+
"frame": {
4470+
"x": 0,
4471+
"y": 580,
4472+
"width": 390,
4473+
"height": 264
4474+
},
4475+
"controlRects": [
4476+
{
4477+
"x": 0,
4478+
"y": 580,
4479+
"width": 390,
4480+
"height": 264
4481+
}
4482+
]
4483+
}
4484+
},
4485+
{
4486+
"name": "app content under TUISystemInputAssistantView inside a Keyboard is refused",
4487+
"viewport": {
4488+
"x": 0,
4489+
"y": 0,
4490+
"width": 390,
4491+
"height": 844
4492+
},
4493+
"nodes": [
4494+
{
4495+
"index": 0,
4496+
"type": "Keyboard",
4497+
"rect": {
4498+
"x": 0,
4499+
"y": 580,
4500+
"width": 390,
4501+
"height": 264
4502+
}
4503+
},
4504+
{
4505+
"index": 1,
4506+
"type": "Button",
4507+
"label": "Save draft",
4508+
"rect": {
4509+
"x": 200,
4510+
"y": 600,
4511+
"width": 100,
4512+
"height": 40
4513+
},
4514+
"parentIndex": 2
4515+
},
4516+
{
4517+
"index": 2,
4518+
"type": "Other",
4519+
"role": "TUISystemInputAssistantView",
4520+
"parentIndex": 0
4521+
}
4522+
],
4523+
"target": {
4524+
"index": 1
4525+
},
4526+
"expected": {
4527+
"kind": "occluded",
4528+
"frame": {
4529+
"x": 0,
4530+
"y": 580,
4531+
"width": 390,
4532+
"height": 264
4533+
},
4534+
"controlRects": [
4535+
{
4536+
"x": 0,
4537+
"y": 580,
4538+
"width": 390,
4539+
"height": 264
4540+
}
4541+
]
4542+
}
4543+
},
4544+
{
4545+
"name": "app content under TUISystemInputAssistantPageView inside a Keyboard is refused",
4546+
"viewport": {
4547+
"x": 0,
4548+
"y": 0,
4549+
"width": 390,
4550+
"height": 844
4551+
},
4552+
"nodes": [
4553+
{
4554+
"index": 0,
4555+
"type": "Keyboard",
4556+
"rect": {
4557+
"x": 0,
4558+
"y": 580,
4559+
"width": 390,
4560+
"height": 264
4561+
}
4562+
},
4563+
{
4564+
"index": 1,
4565+
"type": "Button",
4566+
"label": "Save draft",
4567+
"rect": {
4568+
"x": 200,
4569+
"y": 600,
4570+
"width": 100,
4571+
"height": 40
4572+
},
4573+
"parentIndex": 2
4574+
},
4575+
{
4576+
"index": 2,
4577+
"type": "Other",
4578+
"role": "TUISystemInputAssistantPageView",
4579+
"parentIndex": 0
4580+
}
4581+
],
4582+
"target": {
4583+
"index": 1
4584+
},
4585+
"expected": {
4586+
"kind": "occluded",
4587+
"frame": {
4588+
"x": 0,
4589+
"y": 580,
4590+
"width": 390,
4591+
"height": 264
4592+
},
4593+
"controlRects": [
4594+
{
4595+
"x": 0,
4596+
"y": 580,
4597+
"width": 390,
4598+
"height": 264
4599+
}
4600+
]
4601+
}
43614602
}
43624603
]
43634604
}

‎packages/contracts/src/interactor-types.ts‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -266,9 +266,12 @@ export type SnapshotResult = Omit<BackendSnapshotResult, 'backend' | 'nodes'> &
266266
*/
267267
systemSurface?: IosSystemSurfaceProvenance;
268268
/**
269-
* The keyboard band the producer measured while capturing, when it can measure one (#2660). A
270-
* producer that publishes nothing measured nothing, so the tap-path guard keeps deriving the band
271-
* from `nodes`; see {@link SnapshotKeyboardBandFact}.
269+
* Keyboard evidence acquired with this tree: measured `visible`, policy-derived `derived`,
270+
* probe-only `visibility-only`, proven `absent`, or unknown `unmeasurable` facts. Measured and
271+
* derived frames survive projection and take precedence over tree reconstruction. Visibility-only
272+
* facts carry no band; rejected acquisition geometry stays rejected. Other failed measurements or
273+
* an omitted fact may use tree geometry, but missing nodes never prove absence.
274+
* See {@link SnapshotKeyboardBandFact}.
272275
*/
273276
keyboard?: SnapshotKeyboardBandFact;
274277
/**

‎packages/contracts/src/tap-keyboard-occlusion.test.ts‎

Lines changed: 18 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ import type {
88
SnapshotKeyboardBandFact,
99
} from '@agent-device/kernel/snapshot';
1010
import {
11+
observeProviderKeyboard,
1112
resolveKeyboardTapOcclusion,
1213
TAP_KEYBOARD_OCCLUDES_TARGET_DETAILS,
1314
TAP_KEYBOARD_OCCLUDES_TARGET_REASON,
@@ -32,6 +33,7 @@ type FixtureCase = {
3233
nodes: FixtureNode[];
3334
target: { index: number } | { point: { x: number; y: number } };
3435
keyboard?: SnapshotKeyboardBandFact;
36+
observationVisibility?: boolean;
3537
expected: { kind: string; frame?: Rect; controlRects?: readonly Rect[] };
3638
};
3739

@@ -71,6 +73,17 @@ test('keyboard tap occlusion agrees with every golden parity table case', () =>
7173
'point' in fixture.target
7274
? fixture.target.point
7375
: resolveCenter(byIndex.get(fixture.target.index));
76+
if (fixture.observationVisibility !== undefined) {
77+
assert.deepEqual(
78+
observeProviderKeyboard({
79+
nodes,
80+
viewport: fixture.viewport,
81+
visibility: fixture.observationVisibility,
82+
}),
83+
fixture.keyboard,
84+
`${fixture.name}: acquisition observation`,
85+
);
86+
}
7487
const occlusion = resolveKeyboardTapOcclusion({
7588
nodes,
7689
viewport: fixture.viewport,
@@ -128,7 +141,7 @@ function tableCase(name: string): FixtureCase {
128141

129142
function resolveWithFact(
130143
fixture: FixtureCase,
131-
keyboard: SnapshotKeyboardBandFact,
144+
keyboard: SnapshotKeyboardBandFact | undefined,
132145
): ReturnType<typeof resolveKeyboardTapOcclusion> {
133146
const nodes = fixture.nodes as RawSnapshotNode[];
134147
const byIndex = new Map(nodes.map((node) => [node.index, node]));
@@ -221,8 +234,10 @@ test('an unmeasurable fact leaves every golden-table decision exactly as the tre
221234
// either side of the refusal, or the table would stop describing this guard.
222235
for (const fixture of loadTable().cases) {
223236
const withFact = resolveWithFact(fixture, {
224-
...(fixture.keyboard ?? { kind: 'unmeasurable', reason: 'keyboard-frame-query-timeout' }),
237+
kind: 'unmeasurable',
238+
reason: 'keyboard-frame-query-timeout',
225239
});
226-
assert.equal(withFact.kind, fixture.expected.kind, `${fixture.name}: kind`);
240+
const treeOnly = resolveWithFact(fixture, undefined);
241+
assert.deepEqual(withFact, treeOnly, fixture.name);
227242
}
228243
});

0 commit comments

Comments
 (0)