Skip to content

Commit 7997dc2

Browse files
strzhaoclaude
andauthored
feat(harmonyos): report the keyboard band in snapshot responses (#4285)
* feat(harmonyos): measure the software keyboard band into snapshot's keyboard fact The ArkUI layout dump never shows the input method window, so a harmony snapshot left the keyboard band unmeasured: the tap guard's docked-tree derivation had no anchor to read, and a tap behind the keyboard failed open while the touch landed on a key. The window manager's public dump (hidumper -s WindowManagerService -a -a) lists the SCBKeyboardPanel window with its rect, so the producer now reads the band there and publishes it beside the tree. A positive rect with a non-negative z-order is the measured band; a zero rect is the hidden keyboard, published as visibility-only. A probe that fails, a table without the panel, or an unreadable rect publishes unmeasurable, which leaves the guard's tree derivation and its fail-open verdict exactly as they were. Gesture viewport reads skip the probe so gesture planning keeps its old cost. The keyboard status command stays unavailable; this fact rides snapshot only, where the capture that produced the tree measured the band too. Co-Authored-By: Claude Code <[email protected]> * docs(evidence): record the keyboard band through a raise and dismiss Co-Authored-By: Claude Code <[email protected]> * fix(harmonyos): keep the keyboard band off a display-spanning panel rect The WindowManagerService table reports SCBKeyboardPanel (and the IME's own softKeyboard1 window) with the full display rect whether the keyboard shows or hides; only the z-order column moves. Publishing that rect as a visible band made the tap guard read every on-screen point as keyboard-covered, refusing ref and selector presses above the keyboard. A panel rect the table cannot prove smaller than the display now publishes visibility-only, which leaves the guard on its tree derivation and fail-open; a rect the table proves to be a partial strip still publishes the frame. Fixtures carry the real JDY-AL50 rows captured with the IME raised and dismissed. Co-Authored-By: Claude Code <[email protected]> * docs(evidence): re-run the keyboard band at the rebased head with the real rows Replaces the invalidated run: the window table's real rows quoted verbatim (up and hidden), the visibility-only contract live through a raise and a dismiss, the guard regressions retired (a ref above the band succeeds, an in-band coordinate press dispatches), probe cost measured, the model string corrected to JUY-AL50, and the landing-order section the review asked for. The keys-only screenshot crop keeps the app's search page out of the repo. Co-Authored-By: Claude Code <[email protected]> * docs(evidence): state the 0.9 display-area share threshold for band publication cubic review P2: the evidence said 'rect smaller than the display' but the producer keys on DISPLAY_SPANNING_RECT_SHARE = 0.9 — a band publishes only under that share; at or above it the panel is the full-surface container. Say the threshold so the document matches the shipped contract. * refactor(harmonyos): publish keyboard visibility only, never a band frame The window table proves no band: on the measured build the panel row reports the whole display whether the keyboard shows or hides, so the rect branch of the producer was dead code on every build it could observe — and a future build with a partial container would have published that container as the key plane, which the tap guard trusts, reviving the false refusals #4285 set out to remove. The producer now reads the panel row's z-order alone: visible when any panel carries a non-negative z-order, hidden when none does. A readable-z-only guard answers unmeasurable (the rect column no longer backs any verdict), and panel rows that disagree across displays answer unmeasurable rather than pick a winner, since the probe has no display scope. The 0.9 display-share threshold, the largest-rect display derivation, and the multi-display scan are deleted — the guard cannot be handed a frame this table cannot prove. * docs(evidence): re-run the keyboard band at the z-order-only head Same device, same choreography, same facts: visibility-only false on the settings home, visibility-only true with the IME raised, a bare in-band coordinate press dispatching under the fail-open guard, and one Back returning the fact to hidden. The rect-share rule sections are rewritten for the z-order-only producer; the landscape note no longer derives from rect geometry. --------- Co-authored-by: Claude Code <[email protected]>
1 parent 5ec56d0 commit 7997dc2

6 files changed

Lines changed: 492 additions & 3 deletions

File tree

Lines changed: 141 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
# HarmonyOS keyboard band fact, live, 2026-11-08 (fixed run 2026-11-09)
2+
3+
Evidence for #4285 (report the keyboard band in snapshot responses), re-run at the
4+
z-order-only head after review asked for the frame branch to go. The window-manager
5+
table on this build never carries the key plane: the panel row reports the whole
6+
display whether the keyboard shows or hides. The producer therefore reads only the
7+
panel row's z-order and publishes visibility — it never publishes a frame, so the
8+
0.9-share threshold (`DISPLAY_SPANNING_RECT_SHARE`) and the largest-rect display
9+
derivation are gone: what they guarded against (a display-sized frame marking every
10+
on-screen point keyboard-covered) cannot be constructed any more. This run
11+
re-verifies the same facts at the new head. First fix: `2878b86e4`; simplification
12+
follows the review question of 2026-11-10.
13+
14+
## Device
15+
16+
- HUAWEI 畅享 90 Plus, model `JUY-AL50` (`param get const.product.model`;
17+
the earlier revision of this file wrote `JDY-AL50`, which is a typo),
18+
HarmonyOS 7.0.0.111 (SP10C00E111R4P3), API 26
19+
- Serial `77TYD26826028730`, USB
20+
- Surface: system Settings → search field on the Settings home page, then the
21+
search page. One character typed by the in-band press, gone with the page
22+
(search-page state is transient). No user data read or written.
23+
24+
## What the window table actually holds
25+
26+
`hidumper -s WindowManagerService -a -a`, keyboard raised (Settings search field
27+
focused). Both keyboard windows hold the full display rect; only the z-order
28+
column moves:
29+
30+
```
31+
softKeyboard1 0 6100 16 2105 102 0 103 0 [ 0 0 1080 2406 ] [ 0 0 ] [ 1 1 0.5 0.5 ]
32+
SCBKeyboardPanel 0 5714 17 2134 1 0 102 0 [ 0 0 1080 2406 ] [ 0 0 ] [ 1 1 0.5 0.5 ]
33+
```
34+
35+
After one Back dismisses the IME, the same rows flip only in z-order:
36+
37+
```
38+
softKeyboard1 0 6100 16 2105 102 0 -1 0 [ 0 0 1080 2406 ] [ 0 0 ] [ 1 1 0.5 0.5 ]
39+
SCBKeyboardPanel 0 5714 17 2134 1 0 -1 0 [ 0 0 1080 2406 ] [ 0 0 ] [ 1 1 0.5 0.5 ]
40+
```
41+
42+
The key plane on this build sits at roughly y 1630–2325 in display pixels (the
43+
keys-only crop below starts at y 1600); the IME window surface covers the display.
44+
No public source reports that inner band: the window table carries window-surface
45+
rects, and `hidumper -s InputMethodService -a '-a'` lists installed IMEs
46+
(`小艺输入法`) with no geometry. A rect the table cannot prove smaller than the
47+
display is the container, not a band — publishing one as `visible` made the tap
48+
guard read every on-screen point as keyboard-covered (the `@e20` refusal in an
49+
earlier revision of this file was that false refusal, presented as a feature).
50+
The producer no longer reads the rect column at all: z-order decides visibility,
51+
and the fact is always `visibility-only`.
52+
53+
## The fact through a raise and a dismiss (branch CLI, z-order-only head)
54+
55+
Settings home, keyboard hidden:
56+
57+
```json
58+
"keyboard": {"kind": "visibility-only", "visible": false}
59+
```
60+
61+
`press 540 381` on the search field raises the IME. The fact states visibility
62+
and publishes no frame:
63+
64+
```json
65+
"keyboard": {"kind": "visibility-only", "visible": true}
66+
```
67+
68+
Keyboard raised, portrait (keys-only crop; the strip above it holds the app's
69+
search page and is left out of the repo):
70+
71+
![keyboard keys with the IME raised](harmonyos-keyboard-band-keys-2026-11-09.jpg)
72+
73+
One Back hides the IME; the fact returns to `{"kind": "visibility-only",
74+
"visible": false}`. A second Back then leaves the search page — the dismissal
75+
choreography is one Back for the keyboard, one for the page, and the previous
76+
revision's "two Back keys" line described the same two steps as one event.
77+
78+
## The guard at the new head
79+
80+
A ref captured with the keyboard up, pressed above the band, succeeds — this is
81+
the class the display-spanning frame refused (`Ref @e20 is behind the visible
82+
keyboard`):
83+
84+
```
85+
$ agent-device press @e2 --platform harmonyos --serial 77TYD26826028730
86+
Tapped @e2 (291, 192)
87+
```
88+
89+
A bare coordinate press inside the band dispatches (the guard stays fail-open on
90+
`visibility-only`; the press landed on a key and typed one character into the
91+
transient search field):
92+
93+
```
94+
$ agent-device press 540 1700 --platform harmonyos --serial 77TYD26826028730
95+
Tapped (540, 1700)
96+
```
97+
98+
## Probe cost
99+
100+
The probe is one `hidumper` round trip, measured 0.129 s alone on this device.
101+
`snapshot --json` wall clock, three runs per CLI on the same page: with the probe
102+
(built from #4285) 1.593 / 1.527 / 1.578 s; without it (a build from upstream
103+
main) 1.976 / 1.437 / 1.495 s. Medians differ by 0.083 s, inside the run-to-run
104+
spread; the first no-probe run shows the cold-start tail. The probe runs on every
105+
HarmonyOS capture by design — every capture's keyboard feeds the snapshot state
106+
later taps read.
107+
108+
## Landscape
109+
110+
Not run. The device exposes no public rotation channel — `uitest` has no rotate
111+
subcommand, and agent-device's own `setHarmonyOrientation` answers
112+
UNSUPPORTED_OPERATION on HarmonyOS — so the landscape run needs a physical
113+
rotation this session cannot perform. The producer is orientation-independent by
114+
construction: z-order does not depend on rotation, nothing derives from rect
115+
geometry, and the real-row fixtures pin both states.
116+
117+
## Landing order
118+
119+
Proposed order in #4318 stands: #4292, #4290, #4293, #4287, then this PR, then
120+
#4239. This PR lands fifth, rebased onto main (`b24ecf209`) with the snapshot
121+
help conflict resolved.
122+
123+
## Reproduce
124+
125+
```sh
126+
# z-order-only head, built in the PR worktree
127+
node bin/agent-device.mjs open com.huawei.hmos.settings --activity com.huawei.hmos.settings.MainAbility \
128+
--platform harmonyos --serial 77TYD26826028730
129+
node bin/agent-device.mjs snapshot --json --platform harmonyos --serial 77TYD26826028730 # visibility-only false
130+
node bin/agent-device.mjs press 540 381 --platform harmonyos --serial 77TYD26826028730 # search field
131+
node bin/agent-device.mjs snapshot --json --platform harmonyos --serial 77TYD26826028730 # visibility-only true, no frame
132+
node bin/agent-device.mjs snapshot -i --json --platform harmonyos --serial 77TYD26826028730 # refs
133+
node bin/agent-device.mjs press @e2 --platform harmonyos --serial 77TYD26826028730 # above band: succeeds
134+
node bin/agent-device.mjs press 540 1700 --platform harmonyos --serial 77TYD26826028730 # in band: dispatches
135+
node bin/agent-device.mjs back --platform harmonyos --serial 77TYD26826028730 # IME hides
136+
node bin/agent-device.mjs snapshot --json --platform harmonyos --serial 77TYD26826028730 # visibility-only false
137+
hidumper -s WindowManagerService -a -a # on the device: the raw rows quoted above
138+
```
139+
140+
CI cannot exercise a HarmonyOS device; this file is the device-side proof of the
141+
z-order parse, the visibility-only fact, and the guard behavior.
65.7 KB
Loading

‎packages/command-surface/src/capture/snapshot.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -75,7 +75,7 @@ export const snapshotCommandFacet = defineCommandFacet({
7575
name: SNAPSHOT_COMMAND_NAME,
7676
text: {
7777
summary: 'Capture or diff the accessibility tree',
78-
cliDetail: `Scoped output (--scope) renders every returned ref with its ref generation pin (@eN~s<refsGeneration>) and one Refs generation line, so a pasted ref targets the ref frame that issued it; --json keeps plain refs plus the response-level refsGeneration, and --raw keeps byte-for-byte output. Regular iOS snapshots clip overflowing elements to their visible area and omit fully outside elements. --json reports clipping counts in snapshotQualityNotes with code regular-geometry-clipped; --raw retains reported frames. --observe-only is local iOS snapshot-only: an already-ready compatible runner reads the session app without activation or foreground repair, refuses unavailable observation or a readable system alert over the app, and reports XCUIApplication state rather than verified foreground ownership (#2696). Regular targetActivation disclosure stays unchanged (#2682, #2694). Repeated equivalent unfiltered Android snapshots return a compact unchanged acknowledgement. Use --force-full to re-emit the tree; --json and --raw retain full output. Android Appium JSON snapshots publish a display viewport only when measurements before and after the tree agree, independent of tree extent or keyboard content resizing. On Android, JSON value carries field content rather than static labels; saved selectors and replay scripts using value= for static labels must use label= instead; a reported showing hint means value is empty, and text equal to the placeholder is preserved when hint state is unknown. For iOS raw-coordinate fallback after a no-op ref press, inspect rects with snapshot -i --json, press the rect center, then verify with diff snapshot -i or snapshot --diff. iOS backend capability contract: ${snapshotBackendCapabilityHelp}.`,
78+
cliDetail: `Scoped output (--scope) renders every returned ref with its ref generation pin (@eN~s<refsGeneration>) and one Refs generation line, so a pasted ref targets the ref frame that issued it; --json keeps plain refs plus the response-level refsGeneration, and --raw keeps byte-for-byte output. Regular iOS snapshots clip overflowing elements to their visible area and omit fully outside elements. --json reports clipping counts in snapshotQualityNotes with code regular-geometry-clipped; --raw retains reported frames. --observe-only is local iOS snapshot-only: an already-ready compatible runner reads the session app without activation or foreground repair, refuses unavailable observation or a readable system alert over the app, and reports XCUIApplication state rather than verified foreground ownership (#2696). Regular targetActivation disclosure stays unchanged (#2682, #2694). Repeated equivalent unfiltered Android snapshots return a compact unchanged acknowledgement. Use --force-full to re-emit the tree; --json and --raw retain full output. Android Appium JSON snapshots publish a display viewport only when measurements before and after the tree agree, independent of tree extent or keyboard content resizing. On Android, JSON value carries field content rather than static labels; saved selectors and replay scripts using value= for static labels must use label= instead; a reported showing hint means value is empty, and text equal to the placeholder is preserved when hint state is unknown. HarmonyOS reads the on-screen keyboard's own system window and publishes the keyboard field of --json output: the field carries a band rect only when the window table proves that rect smaller than the display, current builds state visibility only, and a failed window probe publishes unmeasurable, never a guessed band. For iOS raw-coordinate fallback after a no-op ref press, inspect rects with snapshot -i --json, press the rect center, then verify with diff snapshot -i or snapshot --diff. iOS backend capability contract: ${snapshotBackendCapabilityHelp}.`,
7979
},
8080
metadata: snapshotCommandMetadata,
8181
run: (client, input) => client.capture.snapshot(input),

0 commit comments

Comments
 (0)