Skip to content

Commit b68ed4b

Browse files
strzhaoclaudethymikee
authored
refactor(gestures): one owner for the densified gesture path; drop the unproven sendevent transport (#4239)
* feat(harmonyos): inject gesture pan as a densified touch-panel trajectory Gesture pan on HarmonyOS densifies the plan's timed path on the shared 16 ms sampling grid and writes the samples to the touchscreen evdev node. The lowering maps display pixels into the panel's own axis range, which it reads from the MMI service dump because commercial builds deny the input device listing, and refuses the transport for rotated displays. The rendered script paces each sleep before its sample, joins the chain with && under a success marker the host verifies, and releases the contact when the chain fails. Any unavailable transport falls back to the uiInput endpoint swipe and reports the reason under trajectoryFallback. Co-Authored-By: Claude Code <[email protected]> * fix(harmonyos): align transform test fixtures with the resolver's shape contract Co-Authored-By: Claude Code <[email protected]> * refactor(harmonyos): keep the injection timeout budget constants module-private Co-Authored-By: Claude Code <[email protected]> * docs(help): fold the HarmonyOS pan transport line into the existing quirk to hold the first screen Co-Authored-By: Claude Code <[email protected]> * fix(harmonyos): pace the injection per spawned process and re-probe the panel after a failed injection Co-Authored-By: Claude Code <[email protected]> * docs(evidence): record a densified pan dragging a real slider to a 3px pass Co-Authored-By: Claude Code <[email protected]> * refactor(contracts): own the densified gesture path next to its sampler densifyGesturePath takes the merge-and-interpolate loop every transport lowering kept as its own copy: profile grid and canonical offsets merge, each merged offset interpolates inside its canonical segment, endpoints stay the plan's. A progress hook reshapes within-segment pacing (Android's controlled scroll) without forking the segment walk. Co-Authored-By: Claude Code <[email protected]> * refactor(harmonyos): drop the sendevent pan transport until a readable input listing exists The only reachable HarmonyOS device denies /proc/bus/input/devices and /dev/input at shell uid, so the trajectory chain never executed on hardware and its uinput sendevent contract stays unverified. The pan keeps the endpoint swipe as its one transport; --json names harmonyos-hdc-uiinput and carries no fallback field. Co-Authored-By: Claude Code <[email protected]> * docs(evidence): correct the slider pass to the endpoint swipe it took Co-Authored-By: Claude Code <[email protected]> * fix(contracts): keep both path endpoints clear of the progress hook The start sample sat on the sampling grid, so the hook saw progress 0 and a distorting hook could move the plan's start point; the end sample already kept the canonical point. The start endpoint now does too, and the duration must equal the last canonical offset so the path cannot extrapolate past its final sample. The route evidence file is withdrawn: it described machinery that never shipped, cited an amended-away commit, and misnamed the contracts path. The device denial and the route-identified run move into the PR description. Co-Authored-By: Claude Code <[email protected]> * refactor(contracts): take the gesture duration from its last canonical offset The separate durationMs argument could disagree with the samples and the guard guarding that could not fire from the only caller. The last offset IS the duration: the parameter, its guard, its reason and its test are gone, and a path of fewer than two timed samples refuses with GESTURE_PATH_TOO_SHORT. Evidence adds the live Android pan and scroll through the densified helper path. Co-Authored-By: Claude Code <[email protected]> * chore: retrigger CI after the live-emulator back-navigation flake Co-Authored-By: Claude Code <[email protected]> * test(gestures): pin typed short-path refusal and qualify live evidence * test(sdk): load plist module before timed host-scope assertions * docs(gestures): limit live scroll evidence to reported success * fix(gestures): enforce canonical timelines and qualify Android evidence * chore: retrigger CI after the live-emulator transition flake Co-Authored-By: Claude Code <[email protected]> --------- Co-authored-by: Claude Code <[email protected]> Co-authored-by: Michał Pierzchała <[email protected]>
1 parent a247d80 commit b68ed4b

5 files changed

Lines changed: 234 additions & 36 deletions

File tree

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
# Android densified lowering through the shared densifyGesturePath, 2026-11-13
2+
3+
Contributor-reported evidence for #4239's Android side records one pan and one
4+
scroll on hardware. The retained pan notes name the Android helper. The scroll
5+
notes report success without identifying a backend. Raw CLI responses were not
6+
preserved, so the excerpts below are not verbatim response artifacts.
7+
8+
## Device and build
9+
10+
- Redmi K60 (Android), adb serial `5p4hqglvpzxshiuw`, Settings app foreground
11+
- Contributor-reported tested commit: `181935c951`, built in the contribution
12+
worktree, daemon restarted clean; the Android automation helper (0.32.0 build, packaged under
13+
this tree's version name) was installed with `adb install -t` before the run
14+
15+
## Retained contributor notes
16+
17+
The original pan notes included `executionProfile: timed-pan`. Android's command
18+
does not emit that field. It is omitted below; it describes the plan selected by
19+
the code, not a field observed in the response.
20+
21+
```json
22+
{
23+
"kind": "pan",
24+
"backend": "android-helper",
25+
"message": "Panned (500, 1400) by (0, -600)"
26+
}
27+
```
28+
29+
```json
30+
{
31+
"message": "Scrolled down",
32+
"success": true
33+
}
34+
```
35+
36+
The pan notes name `backend: android-helper`. For a pan plan, the shared planner
37+
selects `timed-pan` and Android lowers its samples before helper dispatch; this is
38+
an inference from the code. The scroll notes do not identify a backend or sampled
39+
route. The contributor reported visible scrolling in both runs.
40+
41+
## What this evidences
42+
43+
The contributor reports Android helper pan dispatch and a successful scroll.
44+
The retained notes cannot establish a response-reported execution profile or
45+
which backend or lowering executed the scroll. Neither result shows the cadence
46+
of individual timed samples. Cadence equivalence rests on the pure-function and
47+
Android-lowering tests rather than these reported runs.
48+
49+
The retained notes name `gesture pan --json` and `scroll down`, but do not
50+
preserve the complete argument lists. The pan response records the start point
51+
`(500, 1400)` and displacement `(0, -600)`. These are response facts, not a
52+
reconstruction of the missing invocation.

‎packages/contracts/src/gesture-plan.test.ts‎

Lines changed: 96 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,10 @@
11
import assert from 'node:assert/strict';
22
import { test } from 'vitest';
3-
import { buildDragGesturePlan, singlePointerPlanEndpoints } from './gesture-plan.ts';
3+
import {
4+
buildDragGesturePlan,
5+
densifyGesturePath,
6+
singlePointerPlanEndpoints,
7+
} from './gesture-plan.ts';
48

59
const DRAG_PROPERTY_RUNS = 100;
610

@@ -77,5 +81,96 @@ test('every drag plan preserves ordered samples and exact hold/move endpoints',
7781
for (let index = 1; index < samples.length; index += 1) {
7882
assert.ok(samples[index]!.offsetMs >= samples[index - 1]!.offsetMs);
7983
}
84+
const dense = densifyGesturePath(samples, 'android');
85+
assert.deepEqual(dense[0]?.point, from);
86+
assert.deepEqual(dense.at(-1)?.point, to);
8087
}
8188
});
89+
90+
test('densifyGesturePath merges the profile grid into the canonical path and keeps its endpoints', () => {
91+
const canonical = [
92+
{ offsetMs: 0, point: { x: 0, y: 0 } },
93+
{ offsetMs: 100, point: { x: 100, y: 0 } },
94+
{ offsetMs: 200, point: { x: 100, y: 100 } },
95+
];
96+
const dense = densifyGesturePath(canonical, 'default');
97+
98+
// The 16 ms grid and the canonical offsets merge into one ascending timeline.
99+
const offsets = dense.map(({ offsetMs }) => offsetMs);
100+
assert.equal(offsets[0], 0);
101+
assert.equal(offsets.at(-1), 200);
102+
assert.ok([...offsets].sort((left, right) => left - right).every((o, i) => o === offsets[i]));
103+
assert.deepEqual(dense[0]?.point, { x: 0, y: 0 });
104+
assert.deepEqual(dense.at(-1)?.point, { x: 100, y: 100 });
105+
// A merged grid offset inside the first segment interpolates on that segment's line.
106+
const at50 = dense.find(({ offsetMs }) => offsetMs === 50);
107+
assert.deepEqual(at50?.point, { x: 50, y: 0 });
108+
const at150 = dense.find(({ offsetMs }) => offsetMs === 150);
109+
assert.deepEqual(at150?.point, { x: 100, y: 50 });
110+
// Canonical offsets survive verbatim.
111+
const at100 = dense.find(({ offsetMs }) => offsetMs === 100);
112+
assert.deepEqual(at100?.point, { x: 100, y: 0 });
113+
});
114+
115+
test('densifyGesturePath hands within-segment pacing to the progress hook and keeps its endpoint rule', () => {
116+
const canonical = [
117+
{ offsetMs: 0, point: { x: 0, y: 0 } },
118+
{ offsetMs: 100, point: { x: 100, y: 0 } },
119+
];
120+
const seen: Array<[number, number]> = [];
121+
const dense = densifyGesturePath(canonical, 'default', (progress, segmentDurationMs) => {
122+
seen.push([progress, segmentDurationMs]);
123+
return progress * progress;
124+
});
125+
126+
const at50 = dense.find(({ offsetMs }) => offsetMs === 50);
127+
assert.deepEqual(at50?.point, { x: 25, y: 0 });
128+
assert.ok(
129+
seen.some(([progress, segmentDurationMs]) => progress === 0.5 && segmentDurationMs === 100),
130+
);
131+
// The hook only shapes interior offsets: every progress it saw lies strictly
132+
// between 0 and 1, so the endpoints never passed through it.
133+
assert.ok(seen.length >= 1);
134+
for (const [progress] of seen) {
135+
assert.ok(progress > 0 && progress < 1, `hook saw progress ${progress}`);
136+
}
137+
assert.deepEqual(dense[0]?.point, { x: 0, y: 0 });
138+
assert.deepEqual(dense.at(-1)?.point, { x: 100, y: 0 });
139+
});
140+
141+
test('densifyGesturePath refuses a path of fewer than two timed samples', () => {
142+
assert.throws(() => densifyGesturePath([{ offsetMs: 0, point: { x: 0, y: 0 } }], 'default'), {
143+
code: 'INVALID_ARGS',
144+
details: { reason: 'GESTURE_PATH_TOO_SHORT' },
145+
});
146+
assert.throws(
147+
() =>
148+
densifyGesturePath(
149+
[
150+
{ offsetMs: 0, point: { x: 0, y: 0 } },
151+
{ offsetMs: 0, point: { x: 1, y: 0 } },
152+
],
153+
'default',
154+
),
155+
{ code: 'INVALID_ARGS', details: { reason: 'GESTURE_PATH_TOO_SHORT' } },
156+
);
157+
});
158+
159+
for (const [name, offsets] of [
160+
['a nonzero first offset', [5, 100]],
161+
['tied first offsets', [0, 0, 100]],
162+
['tied last offsets', [0, 100, 100]],
163+
['reversed offsets', [0, 75, 50, 100]],
164+
['a non-finite interior offset', [0, Number.NaN, 100]],
165+
] as const) {
166+
test(`densifyGesturePath refuses ${name}`, () => {
167+
const canonical = offsets.map((offsetMs, index) => ({
168+
offsetMs,
169+
point: { x: index * 10, y: 0 },
170+
}));
171+
assert.throws(() => densifyGesturePath(canonical, 'default'), {
172+
code: 'INVALID_ARGS',
173+
details: { reason: 'GESTURE_PATH_INVALID_TIMELINE' },
174+
});
175+
});
176+
}

‎packages/contracts/src/gesture-plan.ts‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ import type {
2222
GestureExecutionProfile,
2323
GestureSemanticInput,
2424
PointerTrajectory,
25+
PointerTrajectorySample,
2526
SinglePointerGesturePlan,
2627
SinglePointerTrajectory,
2728
TapGesturePlan,
@@ -467,6 +468,80 @@ export function sampleGestureOffsets(
467468
);
468469
}
469470

471+
/**
472+
* Densifies a plan's own timed path onto one merged sampling grid: the profile grid and the
473+
* canonical offsets merge, every merged offset interpolates inside its canonical segment, and the
474+
* endpoints and duration stay the plan's — the duration IS the last canonical offset, so a
475+
* disagreeing duration cannot be written. This is the one owner of the sampled-path rule; every
476+
* transport lowering calls it instead of keeping its own merge-and-interpolate loop. The optional
477+
* progress hook lets a transport reshape within-segment pacing (Android's controlled scroll) while
478+
* the segment walk and the endpoint rule stay shared.
479+
*/
480+
export function densifyGesturePath(
481+
canonical: readonly PointerTrajectorySample[],
482+
profile: GestureSamplingProfile,
483+
progress: (baseProgress: number, segmentDurationMs: number) => number = (p) => p,
484+
): PointerTrajectorySample[] {
485+
// The walk reads each segment's end, so a path of fewer than two timed samples has no
486+
// interpolable segment; the last offset is the duration the sampling grid builds on.
487+
const durationMs = canonical.at(-1)?.offsetMs ?? 0;
488+
if (canonical.length < 2 || !Number.isFinite(durationMs) || durationMs <= 0) {
489+
throw new AppError('INVALID_ARGS', 'gesture path needs at least two timed samples', {
490+
reason: 'GESTURE_PATH_TOO_SHORT',
491+
});
492+
}
493+
if (
494+
canonical[0]!.offsetMs !== 0 ||
495+
canonical.some(
496+
({ offsetMs }, index) =>
497+
!Number.isFinite(offsetMs) || (index > 0 && offsetMs <= canonical[index - 1]!.offsetMs),
498+
)
499+
) {
500+
throw new AppError(
501+
'INVALID_ARGS',
502+
'gesture path needs a strictly increasing timeline from zero',
503+
{
504+
reason: 'GESTURE_PATH_INVALID_TIMELINE',
505+
},
506+
);
507+
}
508+
const offsets = [
509+
...new Set([
510+
...sampleGestureOffsets(durationMs, profile),
511+
...canonical.map(({ offsetMs }) => offsetMs),
512+
]),
513+
].sort((left, right) => left - right);
514+
let segmentIndex = 0;
515+
return offsets.map((offsetMs) => {
516+
while (
517+
segmentIndex < canonical.length - 2 &&
518+
offsetMs > canonical[segmentIndex + 1]!.offsetMs
519+
) {
520+
segmentIndex += 1;
521+
}
522+
const start = canonical[segmentIndex]!;
523+
const end = canonical[segmentIndex + 1]!;
524+
const segmentDurationMs = end.offsetMs - start.offsetMs;
525+
const baseProgress =
526+
segmentDurationMs === 0 ? 0 : (offsetMs - start.offsetMs) / segmentDurationMs;
527+
return {
528+
offsetMs,
529+
point:
530+
// Both endpoints and every zero-length segment keep the canonical
531+
// point, so the hook only ever reshapes strictly interior offsets.
532+
offsetMs === end.offsetMs || segmentDurationMs === 0
533+
? end.point
534+
: baseProgress === 0
535+
? start.point
536+
: interpolateGesturePoint(
537+
start.point,
538+
end.point,
539+
progress(baseProgress, segmentDurationMs),
540+
),
541+
};
542+
});
543+
}
544+
470545
function assertSamplesInViewport(
471546
samples: readonly { offsetMs: number; point: Point }[],
472547
viewport: Rect,

‎packages/platform-android/src/touch-plan-lowering.ts‎

Lines changed: 6 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,7 @@
11
import type { ScrollReleaseBehavior } from '@agent-device/contracts/scroll-command';
22
import {
33
GESTURE_SAMPLE_INTERVAL_MS,
4-
interpolateGesturePoint,
5-
sampleGestureOffsets,
4+
densifyGesturePath,
65
} from '@agent-device/contracts/gesture-plan';
76
import type {
87
GesturePlan,
@@ -63,38 +62,11 @@ export function lowerAndroidTouchPlan(plan: AndroidTouchPlan): AndroidLoweredTou
6362

6463
const { releaseBehavior, ...gesturePlan } = plan;
6564
const [{ pointerId, samples: canonicalSamples }] = gesturePlan.pointers;
66-
const offsets = [
67-
...new Set([
68-
...sampleGestureOffsets(plan.durationMs, 'android'),
69-
...canonicalSamples.map(({ offsetMs }) => offsetMs),
70-
]),
71-
].sort((left, right) => left - right);
72-
let segmentIndex = 0;
73-
const dense = offsets.map((offsetMs) => {
74-
while (
75-
segmentIndex < canonicalSamples.length - 2 &&
76-
offsetMs > canonicalSamples[segmentIndex + 1]!.offsetMs
77-
) {
78-
segmentIndex += 1;
79-
}
80-
const start = canonicalSamples[segmentIndex]!;
81-
const end = canonicalSamples[segmentIndex + 1]!;
82-
const segmentDurationMs = end.offsetMs - start.offsetMs;
83-
const progress = (offsetMs - start.offsetMs) / segmentDurationMs;
84-
return {
85-
offsetMs,
86-
point:
87-
offsetMs === end.offsetMs || segmentDurationMs === 0
88-
? end.point
89-
: interpolateGesturePoint(
90-
start.point,
91-
end.point,
92-
releaseBehavior === 'controlled'
93-
? controlledScrollProgress(progress, segmentDurationMs)
94-
: progress,
95-
),
96-
};
97-
});
65+
const dense = densifyGesturePath(canonicalSamples, 'android', (progress, segmentDurationMs) =>
66+
releaseBehavior === 'controlled'
67+
? controlledScrollProgress(progress, segmentDurationMs)
68+
: progress,
69+
);
9870
// The sampler floors the frame count at three, so `dense` always holds at least four samples.
9971
const samples: AndroidTransportSamples = [dense[0]!, dense[1]!, dense[2]!, ...dense.slice(3)];
10072

‎src/sdk/limrun-host-scope.test.ts‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,15 @@
11
import assert from 'node:assert/strict';
22
import { chmodSync, mkdirSync, writeFileSync } from 'node:fs';
33
import path from 'node:path';
4-
import { test } from 'vitest';
4+
import { beforeAll, test } from 'vitest';
55
import { mkdtempForTestSync } from '@agent-device/test-kit';
66
import { withCommandHost } from '@agent-device/host-kit/command';
77
import { createLimrunRuntimeDependencies } from './limrun-runtime-dependencies.ts';
88

9+
beforeAll(async () => {
10+
if (process.platform !== 'win32') await import('@agent-device/platform-apple/install-artifact');
11+
});
12+
913
test.skipIf(process.platform === 'win32')(
1014
'Limrun ADB callbacks retain their owning host outside and across concurrent foreign scopes',
1115
async () => {

0 commit comments

Comments
 (0)