Skip to content

Commit 2878b86

Browse files
strzhaoclaude
andcommitted
fix(harmonyos): keep the keyboard band off a display-spanning panel rect
The WindowManagerService table reports SCBKeyboardPanel (and the IME's own softKeyboard1 window) with the full display rect whether the keyboard shows or hides; only the z-order column moves. Publishing that rect as a visible band made the tap guard read every on-screen point as keyboard-covered, refusing ref and selector presses above the keyboard. A panel rect the table cannot prove smaller than the display now publishes visibility-only, which leaves the guard on its tree derivation and fail-open; a rect the table proves to be a partial strip still publishes the frame. Fixtures carry the real JDY-AL50 rows captured with the IME raised and dismissed. Co-Authored-By: Claude Code <[email protected]>
1 parent 38e4060 commit 2878b86

3 files changed

Lines changed: 114 additions & 14 deletions

File tree

‎packages/command-surface/src/capture/snapshot.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -75,7 +75,7 @@ export const snapshotCommandFacet = defineCommandFacet({
7575
name: SNAPSHOT_COMMAND_NAME,
7676
text: {
7777
summary: 'Capture or diff the accessibility tree',
78-
cliDetail: `Scoped output (--scope) renders every returned ref with its ref generation pin (@eN~s<refsGeneration>) and one Refs generation line, so a pasted ref targets the ref frame that issued it; --json keeps plain refs plus the response-level refsGeneration, and --raw keeps byte-for-byte output. Regular iOS snapshots clip overflowing elements to their visible area and omit fully outside elements. --json reports clipping counts in snapshotQualityNotes with code regular-geometry-clipped; --raw retains reported frames. --observe-only is local iOS snapshot-only: an already-ready compatible runner reads the session app without activation or foreground repair, refuses unavailable observation or a readable system alert over the app, and reports XCUIApplication state rather than verified foreground ownership (#2696). Regular targetActivation disclosure stays unchanged (#2682, #2694). Repeated equivalent unfiltered Android snapshots return a compact unchanged acknowledgement. Use --force-full to re-emit the tree; --json and --raw retain full output. Android Appium JSON snapshots publish a display viewport only when measurements before and after the tree agree, independent of tree extent or keyboard content resizing. On Android, JSON value carries field content rather than static labels; saved selectors and replay scripts using value= for static labels must use label= instead; a reported showing hint means value is empty, and text equal to the placeholder is preserved when hint state is unknown. HarmonyOS measures the on-screen keyboard's own system window and publishes its rect in the keyboard field of --json output; a hidden keyboard publishes visibility-only there, and a failed window probe publishes unmeasurable, never a guessed band. For iOS raw-coordinate fallback after a no-op ref press, inspect rects with snapshot -i --json, press the rect center, then verify with diff snapshot -i or snapshot --diff. iOS backend capability contract: ${snapshotBackendCapabilityHelp}.`,
78+
cliDetail: `Scoped output (--scope) renders every returned ref with its ref generation pin (@eN~s<refsGeneration>) and one Refs generation line, so a pasted ref targets the ref frame that issued it; --json keeps plain refs plus the response-level refsGeneration, and --raw keeps byte-for-byte output. Regular iOS snapshots clip overflowing elements to their visible area and omit fully outside elements. --json reports clipping counts in snapshotQualityNotes with code regular-geometry-clipped; --raw retains reported frames. --observe-only is local iOS snapshot-only: an already-ready compatible runner reads the session app without activation or foreground repair, refuses unavailable observation or a readable system alert over the app, and reports XCUIApplication state rather than verified foreground ownership (#2696). Regular targetActivation disclosure stays unchanged (#2682, #2694). Repeated equivalent unfiltered Android snapshots return a compact unchanged acknowledgement. Use --force-full to re-emit the tree; --json and --raw retain full output. Android Appium JSON snapshots publish a display viewport only when measurements before and after the tree agree, independent of tree extent or keyboard content resizing. On Android, JSON value carries field content rather than static labels; saved selectors and replay scripts using value= for static labels must use label= instead; a reported showing hint means value is empty, and text equal to the placeholder is preserved when hint state is unknown. HarmonyOS reads the on-screen keyboard's own system window and publishes the keyboard field of --json output: the field carries a band rect only when the window table proves that rect smaller than the display, current builds state visibility only, and a failed window probe publishes unmeasurable, never a guessed band. For iOS raw-coordinate fallback after a no-op ref press, inspect rects with snapshot -i --json, press the rect center, then verify with diff snapshot -i or snapshot --diff. iOS backend capability contract: ${snapshotBackendCapabilityHelp}.`,
7979
},
8080
metadata: snapshotCommandMetadata,
8181
run: (client, input) => client.capture.snapshot(input),

‎packages/platform-harmonyos/src/__tests__/keyboard-band.test.ts‎

Lines changed: 67 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -38,9 +38,25 @@ const HIDDEN_KEYBOARD_TABLE = [
3838
'SCBKeyboardPanel 0 5714 17 2134 1 0 -1 0 [ 0 0 0 0 ] [ 0 0 ] [ 1 1 0 0 ]',
3939
].join('\n');
4040

41-
/** The visible panel row follows the same table schema; only the rect and z-order change. */
42-
const VISIBLE_KEYBOARD_ROW =
43-
'SCBKeyboardPanel 0 5714 17 2134 1 0 4400 0 [ 0 1546 1080 860 ] [ 0 0 ] [ 1 1 0.5 0.5 ]';
41+
/**
42+
* The real JUY-AL50 7.0.0.111 rows, captured verbatim from
43+
* `hidumper -s WindowManagerService -a -a` with the Settings search field focused and the IME
44+
* raised, then after a Back dismissal. The panel and the IME's own `softKeyboard1` window both
45+
* hold the full display rect in both states; only the z-order column moves (102/103 up,
46+
* both -1 hidden). No column in the public table carries the key plane.
47+
*/
48+
const REAL_KEYBOARD_UP_ROWS = [
49+
'softKeyboard1 0 6100 16 2105 102 0 103 0 [ 0 0 1080 2406 ] [ 0 0 ] [ 1 1 0.5 0.5 ]',
50+
'SCBKeyboardPanel 0 5714 17 2134 1 0 102 0 [ 0 0 1080 2406 ] [ 0 0 ] [ 1 1 0.5 0.5 ]',
51+
].join('\n');
52+
const REAL_KEYBOARD_HIDDEN_ROWS = [
53+
'softKeyboard1 0 6100 16 2105 102 0 -1 0 [ 0 0 1080 2406 ] [ 0 0 ] [ 1 1 0.5 0.5 ]',
54+
'SCBKeyboardPanel 0 5714 17 2134 1 0 -1 0 [ 0 0 1080 2406 ] [ 0 0 ] [ 1 1 0.5 0.5 ]',
55+
].join('\n');
56+
57+
/** A panel rect materially smaller than the display: the shape a true key-plane report would take. */
58+
const TRUE_BAND_KEYBOARD_ROW =
59+
'SCBKeyboardPanel 0 5714 17 2134 1 0 102 0 [ 0 1546 1080 860 ] [ 0 0 ] [ 1 1 0.5 0.5 ]';
4460

4561
beforeEach(() => {
4662
runHarmonyHdc.mockReset();
@@ -55,14 +71,53 @@ test('a zero-rect keyboard panel reads as hidden, not as a band', () => {
5571
});
5672
});
5773

58-
test('a positive panel rect is the measured band', () => {
59-
const dump = `${WINDOW_TABLE_HEADER}\n${VISIBLE_KEYBOARD_ROW}`;
74+
test('a zero rect on a positively ordered window still reads as hidden', () => {
75+
const zeroRectShownRow =
76+
'SCBKeyboardPanel 0 5714 17 2134 1 0 4400 0 [ 0 0 0 0 ] [ 0 0 ] [ 1 1 0 0 ]';
77+
const dump = `${WINDOW_TABLE_HEADER}\n${zeroRectShownRow}`;
78+
assert.deepEqual(harmonyKeyboardBandFromWindowDump(dump), {
79+
kind: 'visibility-only',
80+
visible: false,
81+
});
82+
});
83+
84+
test('the real display-spanning up-state rows publish visibility-only, not a full-screen band', () => {
85+
const dump = `${WINDOW_TABLE_HEADER}\n${REAL_KEYBOARD_UP_ROWS}`;
86+
assert.deepEqual(harmonyKeyboardBandFromWindowDump(dump), {
87+
kind: 'visibility-only',
88+
visible: true,
89+
});
90+
});
91+
92+
test('the real hidden-state rows publish visibility-only false', () => {
93+
const dump = `${WINDOW_TABLE_HEADER}\n${REAL_KEYBOARD_HIDDEN_ROWS}`;
94+
assert.deepEqual(harmonyKeyboardBandFromWindowDump(dump), {
95+
kind: 'visibility-only',
96+
visible: false,
97+
});
98+
});
99+
100+
test('a panel rect materially smaller than the display publishes the band frame', () => {
101+
const dump = `${WINDOW_TABLE_HEADER}\n${TRUE_BAND_KEYBOARD_ROW}`;
60102
assert.deepEqual(harmonyKeyboardBandFromWindowDump(dump), {
61103
kind: 'visible',
62104
frame: { x: 0, y: 1546, width: 1080, height: 860 },
63105
});
64106
});
65107

108+
test('a keyboard-only table with no display-sized row stays visibility-only', () => {
109+
// The panel's own rect is the only positive rect, so the display cannot be derived and a
110+
// frame must not be published from it: the band would be an unproven guess.
111+
const dump = [
112+
'WindowName DisplayId Pid WinId Type Mode Flag ZOrd Orientation [ x y w h ] [ OffsetX OffsetY ] [ ScaleX ScaleY PivotX PivotY ]',
113+
TRUE_BAND_KEYBOARD_ROW,
114+
].join('\n');
115+
assert.deepEqual(harmonyKeyboardBandFromWindowDump(dump), {
116+
kind: 'visibility-only',
117+
visible: true,
118+
});
119+
});
120+
66121
test('a positive rect on a negatively ordered window stays hidden', () => {
67122
const staleRectRow =
68123
'SCBKeyboardPanel 0 5714 17 2134 1 0 -1 0 [ 0 1546 1080 860 ] [ 0 0 ] [ 1 1 0 0 ]';
@@ -122,7 +177,11 @@ test('readHarmonyKeyboardBand dumps the window manager table', async () => {
122177
test('snapshotHarmony carries the keyboard band beside its tree', async () => {
123178
runHarmonyShell.mockImplementation(async (_device: unknown, words: string[]) => {
124179
if (words[0] === 'hidumper') {
125-
return { exitCode: 0, stdout: `${WINDOW_TABLE_HEADER}\n${VISIBLE_KEYBOARD_ROW}`, stderr: '' };
180+
return {
181+
exitCode: 0,
182+
stdout: `${WINDOW_TABLE_HEADER}\n${REAL_KEYBOARD_UP_ROWS}`,
183+
stderr: '',
184+
};
126185
}
127186
return { exitCode: 0, stdout: '', stderr: '' };
128187
});
@@ -143,8 +202,8 @@ test('snapshotHarmony carries the keyboard band beside its tree', async () => {
143202
const snapshot = await snapshotHarmony(DEVICE);
144203

145204
assert.deepEqual(snapshot.keyboard, {
146-
kind: 'visible',
147-
frame: { x: 0, y: 1546, width: 1080, height: 860 },
205+
kind: 'visibility-only',
206+
visible: true,
148207
});
149208
assert.deepEqual(
150209
snapshot.nodes.map((node) => node.value ?? node.type),

‎packages/platform-harmonyos/src/keyboard-band.ts‎

Lines changed: 46 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,9 @@ import { runHarmonyShell } from './hdc.ts';
44

55
/**
66
* The system window that hosts the on-screen keyboard. HarmonyOS keeps this SceneBoard panel
7-
* alive across captures, so presence alone says nothing: a positive rect is the visible band,
8-
* and a zero rect is the hidden keyboard.
7+
* alive across captures, so presence alone says nothing: the z-order column decides visibility
8+
* (negative while hidden), and the rect is a band only when the table proves it smaller than
9+
* the display — current builds report the full display rect in both states.
910
*/
1011
const KEYBOARD_PANEL_WINDOW_PREFIX = 'SCBKeyboardPanel';
1112

@@ -41,10 +42,25 @@ function keyboardWindowProbeFailed(): SnapshotKeyboardBandFact {
4142
return { kind: 'unmeasurable', reason: 'keyboard-window-probe-failed' };
4243
}
4344

45+
/**
46+
* A rect at or above this share of the display's area is the window manager's full-surface
47+
* container, not the key plane. On JUY-AL50 7.0.0.111 the panel row reports the whole display
48+
* whether the keyboard shows or hides, and the key plane lives inside the window's own surface
49+
* where no public dump reads it — `softKeyboard1` and `SCBKeyboardPanel` both answer
50+
* `[0 0 1080 2406]`. A frame this table cannot prove to be a band must not become one: the
51+
* tap guard measures a published frame directly, so a display-spanning frame would mark every
52+
* on-screen point keyboard-covered.
53+
*/
54+
const DISPLAY_SPANNING_RECT_SHARE = 0.9;
55+
4456
/**
4557
* Reads the keyboard band out of a WindowManagerService window table. The table lists the
4658
* panel on every device that shows a software keyboard; a device whose table omits it, or
47-
* whose panel row carries no readable rect, answers `unmeasurable` rather than a guess.
59+
* whose panel row carries no readable rect, answers `unmeasurable` rather than a guess. The
60+
* z-order column is the visibility signal (negative while hidden); the rect is trusted as a
61+
* band only when the table proves it smaller than the display, which current builds do not —
62+
* a display-spanning rect publishes `visibility-only` and leaves the band to the guard's
63+
* tree derivation.
4864
*/
4965
export function harmonyKeyboardBandFromWindowDump(raw: string): SnapshotKeyboardBandFact {
5066
const rows = parseKeyboardPanelRows(raw);
@@ -56,8 +72,33 @@ export function harmonyKeyboardBandFromWindowDump(raw: string): SnapshotKeyboard
5672
return { kind: 'unmeasurable', reason: 'keyboard-window-rect-unparseable' };
5773
}
5874
const visible = measured.find(isVisibleKeyboardPanelRow);
59-
if (visible?.rect) return { kind: 'visible', frame: visible.rect };
60-
return { kind: 'visibility-only', visible: false };
75+
if (!visible?.rect) return { kind: 'visibility-only', visible: false };
76+
const display = displayRectFromWindowDump(raw);
77+
if (!display || rectAreaShare(visible.rect, display) >= DISPLAY_SPANNING_RECT_SHARE) {
78+
return { kind: 'visibility-only', visible: true };
79+
}
80+
return { kind: 'visible', frame: visible.rect };
81+
}
82+
83+
/** The largest positive rect in the table, which the display-sized system windows hold. */
84+
function displayRectFromWindowDump(raw: string): Rect | undefined {
85+
let display: Rect | undefined;
86+
let largestArea = 0;
87+
for (const line of raw.split('\n')) {
88+
const rect = rectFromWindowRow(line);
89+
if (!rect || rect.width <= 0 || rect.height <= 0) continue;
90+
const area = rect.width * rect.height;
91+
if (area > largestArea) {
92+
display = rect;
93+
largestArea = area;
94+
}
95+
}
96+
return display;
97+
}
98+
99+
function rectAreaShare(rect: Rect, display: Rect): number {
100+
if (display.width <= 0 || display.height <= 0) return 0;
101+
return (rect.width * rect.height) / (display.width * display.height);
61102
}
62103

63104
type KeyboardPanelRow = {

0 commit comments

Comments
 (0)