Skip to content

Commit 6f91f64

Browse files
airoweclaude
andauthored
fix(scroll): take edge hints from a same-rect scrollable ancestor (callstack#3784)
* fix(scroll): take edge hints from a same-rect scrollable ancestor On Android Settings, the RecyclerView nested in main_content_scrollable_container offers ACTION_SCROLL_BACKWARD even at the top, so `scroll top` kept swiping a list that could not move and failed with scroll_edge_no_progress. When a scroller has the same rect as its nearest scrollable ancestor, the edge verdict now reads the ancestor's hints. The snapshot's own per-node hints are unchanged. Closes callstack#3737 Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(scroll): scope same-rect ancestor hints to the edge verdict Container selection reads each node's own hints again, as on main; only the final canScroll verdict consults the same-rect scrollable ancestor. The ancestor walk tracks visited indexes so a cyclic parentIndex chain in a malformed tree cannot hang edge analysis. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(scroll): select the same-rect ancestor so scoped recaptures keep it Scoping the ancestor's hints to the verdict alone let selection pick the nested RecyclerView at the top; the loop then scoped its recapture to that subtree, lost the ancestor, and fell back to the stale hint (scroll_edge_no_progress on a second `scroll top`). Scroll edge selection now stands a same-rect scrollable ancestor in for the scroller nested in it, so the chosen container, its scope and its verdict all come from the ancestor. The visited-index guard stays. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(scroll): guard the targeted scrollable-ancestor walk against cycles findNearestScrollableAncestor followed parentIndex without a visited set, so a targeted scroll on a malformed tree whose parent chain loops with no scrollable ancestor could hang edge analysis. It now stops on a revisited index. Co-Authored-By: Claude Opus 5.5 <[email protected]> * refactor(scroll): name the same-rect heuristic and share the scoped-subtree fixture The doc comment now states that the same-rect check is geometric and ignores scroll axis. subtreeFor moves to the shared scroll-edge-state fixtures. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(scroll): require a same-rect ancestor to report a vertical edge Geometry carries no scroll axis, so a horizontal pager framing the same pixels as a vertical list on its page could stand in for that list and hide its edge hints. The snapshot emits vertical edge hints only for vertical scrollers, so an ancestor now stands in only when it reports one. subtreeFor matches scopes and reindexes nodes the way an Android scoped snapshot does. Co-Authored-By: Claude Opus 5.5 <[email protected]> --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent 91d1380 commit 6f91f64

4 files changed

Lines changed: 275 additions & 4 deletions

File tree

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
import { formatRole } from '@agent-device/kernel/snapshot';
2+
import type { Rect, SnapshotNode } from '@agent-device/kernel/snapshot';
3+
4+
/**
5+
* The scroll-container chain of the Android 16 Settings home screen (Pixel 7, API 36), trimmed from
6+
* live `snapshot --json` captures for #3737. `main_content_scrollable_container` and the
7+
* `recycler_view` nested inside it share one rect. The RecyclerView offers ACTION_SCROLL_BACKWARD
8+
* even at the top, so it reports `hiddenContentAbove` in both states; only the outer ScrollView's
9+
* hint tracks where the content actually is.
10+
*/
11+
export type SettingsHomeState = 'at-top' | 'scrolled-down';
12+
13+
const SCREEN: Rect = { x: 0, y: 0, width: 1440, height: 3120 };
14+
const LIST_FRAME: Rect = { x: 0, y: 480, width: 1440, height: 2556 };
15+
const ROW_TOPS: Record<SettingsHomeState, readonly number[]> = {
16+
'at-top': [480, 807, 1078, 1405],
17+
'scrolled-down': [493, 764, 1035, 1362],
18+
};
19+
const ROW_LABELS: Record<SettingsHomeState, readonly string[]> = {
20+
'at-top': ['Google', 'Network & internet', 'Connected devices', 'Apps'],
21+
'scrolled-down': ['Network & internet', 'Connected devices', 'Apps', 'Notifications'],
22+
};
23+
const RECYCLER_VIEW_INDEX = 23;
24+
25+
function node(
26+
index: number,
27+
type: string,
28+
rect: Rect,
29+
overrides: Partial<SnapshotNode> = {},
30+
): SnapshotNode {
31+
return { ref: `e${index + 1}`, index, type, kind: formatRole(type), rect, ...overrides };
32+
}
33+
34+
export function settingsHomeTree(state: SettingsHomeState): SnapshotNode[] {
35+
const rows = ROW_TOPS[state].map((y, row) =>
36+
node(
37+
RECYCLER_VIEW_INDEX + 1 + row,
38+
'android.widget.LinearLayout',
39+
{ x: 0, y, width: 1440, height: 271 },
40+
{ parentIndex: RECYCLER_VIEW_INDEX, label: ROW_LABELS[state][row] },
41+
),
42+
);
43+
return [
44+
node(8, 'android.widget.FrameLayout', SCREEN),
45+
node(10, 'android.widget.FrameLayout', SCREEN, {
46+
parentIndex: 8,
47+
identifier: 'android:id/content',
48+
}),
49+
node(
50+
11,
51+
'android.widget.ScrollView',
52+
{ x: 0, y: 0, width: 1440, height: 3036 },
53+
{
54+
parentIndex: 10,
55+
identifier: 'com.android.settings:id/settings_homepage_container',
56+
},
57+
),
58+
node(
59+
12,
60+
'android.widget.TextView',
61+
{ x: 252, y: 251, width: 518, height: 93 },
62+
{
63+
parentIndex: 11,
64+
identifier: 'com.android.settings:id/search_bar_title',
65+
label: 'Search Settings',
66+
},
67+
),
68+
node(18, 'android.widget.ScrollView', LIST_FRAME, {
69+
parentIndex: 11,
70+
identifier: 'com.android.settings:id/main_content_scrollable_container',
71+
hiddenContentBelow: true,
72+
...(state === 'scrolled-down' ? { hiddenContentAbove: true } : {}),
73+
}),
74+
node(22, 'android.widget.FrameLayout', LIST_FRAME, {
75+
parentIndex: 18,
76+
identifier: 'android:id/list_container',
77+
}),
78+
node(RECYCLER_VIEW_INDEX, 'androidx.recyclerview.widget.RecyclerView', LIST_FRAME, {
79+
parentIndex: 22,
80+
identifier: 'com.android.settings:id/recycler_view',
81+
hiddenContentAbove: true,
82+
}),
83+
...rows,
84+
];
85+
}

‎packages/capture-kit/src/snapshot/__tests__/scroll-edge-state-fixtures.ts‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ import {
44
type ScrollEdgeTarget,
55
} from '../scroll-edge-state.ts';
66
import { AppError } from '@agent-device/kernel/errors';
7+
import { matchesSnapshotScope, reindexSnapshotNodes } from '@agent-device/contracts/snapshot';
78
import { formatRole } from '@agent-device/kernel/snapshot';
89
import type { RawSnapshotNode, SnapshotNode } from '@agent-device/kernel/snapshot';
910

@@ -116,3 +117,22 @@ export async function captureThrows(scope: string | undefined): Promise<AppError
116117
throw error;
117118
}
118119
}
120+
121+
/**
122+
* The scope root's subtree, reindexed, as an Android scoped snapshot returns it: the first node in
123+
* document order whose label, value or identifier matches, plus its descendants. Unscoped returns
124+
* the whole tree.
125+
*/
126+
export function subtreeFor(
127+
nodes: readonly SnapshotNode[],
128+
scope: string | undefined,
129+
): SnapshotNode[] {
130+
if (scope === undefined) return [...nodes];
131+
const root = nodes.find((node) => matchesSnapshotScope(node, scope));
132+
if (!root) return [];
133+
const inScope = new Set([root.index]);
134+
for (const node of nodes) {
135+
if (node.parentIndex !== undefined && inScope.has(node.parentIndex)) inScope.add(node.index);
136+
}
137+
return reindexSnapshotNodes(nodes.filter((node) => inScope.has(node.index)));
138+
}
Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
import { test } from 'vitest';
2+
import assert from 'node:assert/strict';
3+
import { runScrollEdgePasses } from '../scroll-edge-state.ts';
4+
import { capture, scrollNode, subtreeFor, windowRoot } from './scroll-edge-state-fixtures.ts';
5+
import { settingsHomeTree } from './scroll-edge-state-android-settings-fixture.ts';
6+
7+
// ---------------------------------------------------------------------------
8+
// A scroller nested in a scrollable ancestor with the same rect takes its edge hints from that
9+
// ancestor (#3737): the inner RecyclerView on Android Settings claims hidden content above even at
10+
// the top, while the outer ScrollView framing the same pixels reports where the content is.
11+
// ---------------------------------------------------------------------------
12+
13+
test('scroll top edge: the Android Settings list at the top reports no hidden content above', async () => {
14+
const state = await capture(settingsHomeTree('at-top'), 'top');
15+
assert.equal(state.canScroll, false);
16+
});
17+
18+
test('scroll top on Android Settings already at the top costs no pass under scoped recapture', async () => {
19+
// Scoped captures return only the scope root's subtree, as an Android scoped snapshot does.
20+
const tree = settingsHomeTree('at-top');
21+
const outcome = await runScrollEdgePasses({
22+
edge: 'top',
23+
captureState: async (scope) => await capture(subtreeFor(tree, scope), 'top'),
24+
scroll: async () => {
25+
throw new Error('scroll top dispatched a gesture at the top');
26+
},
27+
});
28+
assert.equal(outcome.passes, 0);
29+
});
30+
31+
test('scroll top edge: a cyclic chain of same-rect scrollables ends the ancestor walk', async () => {
32+
const rect = { x: 0, y: 100, width: 400, height: 600 };
33+
const nodes = [
34+
windowRoot(),
35+
scrollNode(1, { parentIndex: 2, identifier: 'first', hiddenContentAbove: true, rect }),
36+
scrollNode(2, { parentIndex: 1, identifier: 'second', rect }),
37+
];
38+
const state = await capture(nodes, 'top');
39+
assert.equal(state.scope, 'first');
40+
assert.equal(state.canScroll, true);
41+
});
42+
43+
test('scroll top edge: the Android Settings list scrolled down still reports hidden content above', async () => {
44+
const state = await capture(settingsHomeTree('scrolled-down'), 'top');
45+
assert.equal(state.canScroll, true);
46+
});
47+
48+
test('scroll bottom edge: the Android Settings list at the top still reports hidden content below', async () => {
49+
const state = await capture(settingsHomeTree('at-top'), 'bottom');
50+
assert.equal(state.canScroll, true);
51+
});
52+
53+
test('scroll top on Android Settings stops at the top instead of reporting no progress', async () => {
54+
const frames = [settingsHomeTree('scrolled-down'), settingsHomeTree('at-top')];
55+
let scrolls = 0;
56+
const outcome = await runScrollEdgePasses({
57+
edge: 'top',
58+
captureState: async () => await capture(frames[Math.min(scrolls, frames.length - 1)]!, 'top'),
59+
scroll: async () => {
60+
scrolls += 1;
61+
},
62+
});
63+
assert.equal(outcome.passes, 1);
64+
});
65+
66+
test('scroll top edge: a nested scroller whose rect differs from its scrollable ancestor keeps its own hint', async () => {
67+
const nodes = [
68+
windowRoot(),
69+
scrollNode(1, { identifier: 'outer', rect: { x: 0, y: 0, width: 400, height: 800 } }),
70+
scrollNode(2, {
71+
parentIndex: 1,
72+
identifier: 'inner',
73+
hiddenContentAbove: true,
74+
rect: { x: 0, y: 100, width: 400, height: 600 },
75+
}),
76+
];
77+
const state = await capture(nodes, 'top');
78+
assert.equal(state.scope, 'inner');
79+
assert.equal(state.canScroll, true);
80+
});
81+
82+
test('scroll top edge: a targeted node in a cyclic chain with no scrollable ancestor ends the walk', async () => {
83+
const nodes = [
84+
windowRoot(),
85+
scrollNode(1, { identifier: 'list', hiddenContentAbove: true }),
86+
{ ...scrollNode(2, { parentIndex: 3 }), type: 'Button', kind: 'button' },
87+
{ ...scrollNode(3, { parentIndex: 2 }), type: 'Group', kind: 'group' },
88+
];
89+
const state = await capture(nodes, 'top', { nodeIndex: 2 });
90+
assert.equal(state.scope, 'list');
91+
assert.equal(state.canScroll, true);
92+
});
93+
94+
test('scroll bottom edge: a same-rect horizontal pager does not stand in for the vertical list on its page', async () => {
95+
// A horizontal pager reports no vertical edge hints; its page's vertical list carries them.
96+
const rect = { x: 0, y: 100, width: 400, height: 600 };
97+
const nodes = [
98+
windowRoot(),
99+
scrollNode(1, { identifier: 'pager', rect }),
100+
scrollNode(2, { parentIndex: 1, identifier: 'feed', hiddenContentBelow: true, rect }),
101+
];
102+
const state = await capture(nodes, 'bottom');
103+
assert.equal(state.scope, 'feed');
104+
assert.equal(state.canScroll, true);
105+
});

‎packages/capture-kit/src/snapshot/scroll-edge-state/selection.ts‎

Lines changed: 65 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -64,12 +64,23 @@ function selectScrollContainer(
6464
target: ScrollEdgeTarget,
6565
): SnapshotNode | null {
6666
const visibility = createSnapshotVisibility(nodes);
67-
const scrollables = nodes.filter((node) => isScrollableNodeLike(node) && isUsableRect(node.rect));
67+
const reportsVerticalEdge = (node: SnapshotNode): boolean =>
68+
hasHiddenContentAtEdge(node, hiddenHints.get(node.index), 'top') ||
69+
hasHiddenContentAtEdge(node, hiddenHints.get(node.index), 'bottom');
70+
const edgeContainer = (node: SnapshotNode): SnapshotNode =>
71+
sameRectScrollableAncestor(node, visibility.nodeByIndex, reportsVerticalEdge) ?? node;
72+
const scrollables = [
73+
...new Set(
74+
nodes
75+
.filter((node) => isScrollableNodeLike(node) && isUsableRect(node.rect))
76+
.map(edgeContainer),
77+
),
78+
];
6879
if (scrollables.length === 0) return null;
6980

7081
const targetAncestor = findNearestScrollableAncestor(target.nodeIndex, visibility.nodeByIndex);
7182
if (targetAncestor) {
72-
return targetAncestor;
83+
return edgeContainer(targetAncestor);
7384
}
7485

7586
const targetPoint = target.point;
@@ -136,12 +147,14 @@ function findNearestScrollableAncestor(
136147
byIndex: ReadonlyMap<number, SnapshotNode>,
137148
): SnapshotNode | null {
138149
if (nodeIndex === undefined) return null;
150+
const visited = new Set<number>();
139151
let node = byIndex.get(nodeIndex);
140-
while (node) {
152+
while (node && !visited.has(node.index)) {
153+
visited.add(node.index);
141154
if (isScrollableNodeLike(node) && isUsableRect(node.rect)) {
142155
return node;
143156
}
144-
node = node.parentIndex === undefined ? undefined : byIndex.get(node.parentIndex);
157+
node = parentOf(node, byIndex);
145158
}
146159
return null;
147160
}
@@ -157,6 +170,54 @@ function hasHiddenContentAtEdge(
157170
return node.hiddenContentAbove === true || hint?.hiddenContentAbove === true;
158171
}
159172

173+
/**
174+
* The outermost scrollable in an unbroken chain of same-rect, vertically scrolling ancestors, or
175+
* undefined when the nearest scrollable ancestor frames different pixels or reports no vertical edge.
176+
* Scroll edge selection stands that ancestor in for the nested scroller: Android's Settings
177+
* RecyclerView offers scroll-backward even at the top, while the ScrollView framing the same pixels
178+
* reports the real position (#3737). Choosing the ancestor also keeps the nested scroller inside the
179+
* scoped recapture. Visited indexes stop a cyclic `parentIndex` chain in a malformed tree.
180+
*
181+
* An identical rect is a geometric heuristic for "these two scrollers move the same content", and
182+
* geometry carries no scroll axis. Requiring the ancestor to report a vertical edge hint keeps a
183+
* horizontal pager whose page holds a vertical list from standing in for that list; the snapshot
184+
* emits vertical edge hints only for vertical scrollers. A vertical ancestor whose content fits in
185+
* both directions reports no edge either, and leaves the nested scroller to answer for itself.
186+
*/
187+
function sameRectScrollableAncestor(
188+
node: SnapshotNode,
189+
byIndex: ReadonlyMap<number, SnapshotNode>,
190+
reportsVerticalEdge: (candidate: SnapshotNode) => boolean,
191+
): SnapshotNode | undefined {
192+
const visited = new Set<number>([node.index]);
193+
let owner: SnapshotNode | undefined;
194+
let current = node;
195+
let parent = parentOf(current, byIndex);
196+
while (parent && !visited.has(parent.index)) {
197+
visited.add(parent.index);
198+
if (isScrollableNodeLike(parent) && isUsableRect(parent.rect)) {
199+
if (!sameRect(parent.rect, current.rect) || !reportsVerticalEdge(parent)) return owner;
200+
owner = parent;
201+
current = parent;
202+
}
203+
parent = parentOf(parent, byIndex);
204+
}
205+
return owner;
206+
}
207+
208+
function parentOf(
209+
node: SnapshotNode,
210+
byIndex: ReadonlyMap<number, SnapshotNode>,
211+
): SnapshotNode | undefined {
212+
return node.parentIndex === undefined ? undefined : byIndex.get(node.parentIndex);
213+
}
214+
215+
function sameRect(a: SnapshotNode['rect'], b: SnapshotNode['rect']): boolean {
216+
return Boolean(
217+
a && b && a.x === b.x && a.y === b.y && a.width === b.width && a.height === b.height,
218+
);
219+
}
220+
160221
function buildScrollContainerScope(
161222
node: SnapshotNode,
162223
nodes: readonly SnapshotNode[],

0 commit comments

Comments
 (0)