Skip to content

Commit 989f53a

Browse files
committed
fix(fold): report screen dimensions in their native-panel coordinate space
The fold response's `screen.widthPt`/`heightPt` are the lit panel's native points (pixels divided by its point scale, never rotated), not the next snapshot's viewport. Add the required `screen.coordinateSpace: "native-panel"` discriminator at the construction site, export it from the owning contract so the Apple owner and the MCP schema share one value, and correct every doc and message that implied these numbers could place a tap. A fresh snapshot stays the only source of app-viewport coordinates: on an open Duo the 669x951 inner panel hosts a 951x669 app window. Closes #2729
1 parent 9b68ab5 commit 989f53a

11 files changed

Lines changed: 145 additions & 18 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,13 +16,18 @@
1616
the pose control in the Xcode Device Hub window through macOS accessibility, and it is confirmed
1717
the way ADR 0025 asked for, by reading the hinge angle back from CoreDevice
1818
(`devicectl device motion hinge-angle`) until it agrees with the request. The response reports
19-
the verified pose, the hinge angle, and the panel the device now lights with its point size, so an
20-
agent can see that its refs and coordinates are stale without another capture. The macOS helper
19+
the verified pose, the hinge angle, and the panel the device now lights with its native panel point
20+
size, so an agent can see the lit panel changed without another capture. The macOS helper
2121
gained a `device-hub pose` subcommand that finds Device Hub in the process table (LaunchServices
2222
registers the trampolined app with no pid), reopens its window when it shows none, and selects the
2323
simulator through the sidebar row keyed by its UDID so two simulators sharing a name cannot be
2424
confused. Simulator-only; a single-panel simulator refuses with `UNSUPPORTED_OPERATION`, and every
2525
other platform states its own refusal cell.
26+
- Fixed (ios): the `fold` response's `screen` dimensions now match their coordinate-space contract. They
27+
are the lit panel's own native points (pixels divided by its point scale, never rotated), so the
28+
response carries the `coordinateSpace: "native-panel"` discriminator and the docs no longer claim
29+
they are the next snapshot's coordinates. They cannot place a tap; take a fresh snapshot for the app
30+
viewport.
2631
- Added (diff): `diff screenshot` accepts a JPEG baseline or current image. Both inputs had to be PNG,
2732
so a capture exported by another tool had to be converted first and a HarmonyOS capture — which the
2833
platform serves as JPEG under whatever name the command was given — could never be compared. Each

‎packages/contracts/src/fold-runtime.ts‎

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,12 +9,25 @@ import type { RuntimeOperationFact } from './platform-runtime.ts';
99
export type SetFoldPoseInput = Readonly<{ pose: FoldPose }>;
1010

1111
/**
12-
* The lit panel after the pose settled, in the points the next snapshot will use. Reported so an
13-
* agent can see that the coordinate space changed without a second capture.
12+
* The one coordinate space a `FoldScreenReport` is ever measured in: the lit panel's own native
13+
* points. It is the single source of truth for the discriminator the Apple owner sets and the MCP
14+
* schema advertises, so the wire value and the type cannot drift apart.
15+
*/
16+
export const FOLD_SCREEN_COORDINATE_SPACE = 'native-panel' as const;
17+
18+
/**
19+
* The panel the device lights after the pose settled, in that panel's own native points: its pixel
20+
* size divided by its point scale, never rotated. `coordinateSpace` is always
21+
* {@link FOLD_SCREEN_COORDINATE_SPACE}, and these numbers are NOT snapshot coordinates — the active
22+
* app window can differ from the panel (iPhone Duo: a 669x951 inner panel hosts a 951x669 app
23+
* window), so they cannot place a tap. A caller that needs the app viewport must take a fresh
24+
* snapshot.
1425
*/
1526
export type FoldScreenReport = Readonly<{
1627
/** The CoreDevice display name of the panel the device now lights. */
1728
display: string;
29+
/** Marks these dimensions as the panel's native points, never a snapshot's app viewport. */
30+
coordinateSpace: typeof FOLD_SCREEN_COORDINATE_SPACE;
1831
widthPt: number;
1932
heightPt: number;
2033
}>;

‎packages/contracts/src/navigation.ts‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -53,8 +53,9 @@ export type OrientationCommandResult = {
5353
*
5454
* Unlike `orientation`, there is no unconfirmed variant: the Apple owner reads the hinge angle
5555
* back from CoreDevice after pressing the Device Hub pose control, and reports a pose only when
56-
* that reading agrees with the request. `screen` names the panel the device lights afterwards, in
57-
* points, because a pose change moves the app to a different coordinate space (ADR 0025).
56+
* that reading agrees with the request. `screen` names the panel the device lights afterwards and
57+
* that panel's native point size (ADR 0025); it is the panel's geometry, not the app viewport, so a
58+
* caller must take a fresh snapshot before placing a tap.
5859
*/
5960
export type FoldCommandResult = {
6061
action: 'fold';

‎packages/platform-apple/src/foldable/pose.test.ts‎

Lines changed: 58 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -99,7 +99,7 @@ test('presses the Device Hub control for the pose and reports the pose CoreDevic
9999
await expect(setAppleFoldPose(duo, 'open')).resolves.toEqual({
100100
pose: 'open',
101101
hingeAngleDegrees: 180,
102-
screen: { display: 'LCD-1', widthPt: 669, heightPt: 951 },
102+
screen: { display: 'LCD-1', coordinateSpace: 'native-panel', widthPt: 669, heightPt: 951 },
103103
});
104104

105105
expect(mockPress).toHaveBeenCalledWith({
@@ -111,6 +111,62 @@ test('presses the Device Hub control for the pose and reports the pose CoreDevic
111111
expect(mockHinge).toHaveBeenCalledTimes(2);
112112
});
113113

114+
test('reports the closed outer panel in native points, not rotated to a snapshot viewport', async () => {
115+
// The outer panel is 1398x2034 px at scale 3, i.e. 466x678 native points; the pose must not
116+
// rotate that into the app window's shape, because a caller cannot place a tap from it.
117+
mockInventory
118+
.mockResolvedValueOnce(duoInventory('outer'))
119+
.mockResolvedValueOnce(duoInventory('outer'));
120+
mockHinge.mockResolvedValue(0);
121+
122+
await expect(setAppleFoldPose(duo, 'closed')).resolves.toEqual({
123+
pose: 'closed',
124+
hingeAngleDegrees: 0,
125+
screen: { display: 'LCD', coordinateSpace: 'native-panel', widthPt: 466, heightPt: 678 },
126+
});
127+
});
128+
129+
test('reports native panel points regardless of the display own currentOrientation', async () => {
130+
// The same inner panel (2007x2853 px at scale 3) with a portrait orientation tag still measures
131+
// 669x951: the report divides by point scale only and never swaps on `currentOrientation`.
132+
const rot0Inner = buildInventory([
133+
panel({ power: 'dark' }),
134+
panel({
135+
name: 'LCD-1',
136+
displayId: 3,
137+
primary: false,
138+
power: 'lit',
139+
widthPx: 2007,
140+
heightPx: 2853,
141+
currentOrientation: 'rot0',
142+
}),
143+
]);
144+
mockInventory.mockResolvedValueOnce(duoInventory('inner')).mockResolvedValueOnce(rot0Inner);
145+
mockHinge.mockResolvedValue(180);
146+
147+
const result = await setAppleFoldPose(duo, 'open');
148+
expect(result.screen).toEqual({
149+
display: 'LCD-1',
150+
coordinateSpace: 'native-panel',
151+
widthPt: 669,
152+
heightPt: 951,
153+
});
154+
});
155+
156+
test('omits the screen report when panel selection is ambiguous, never inventing a viewport', async () => {
157+
// Two lit panels make the inventory ambiguous, so `readLitPanel` refuses to name one; the pose is
158+
// still verified but carries no `screen` rather than guessing the app viewport.
159+
const bothLit = buildInventory([panel({}), panel({ name: 'LCD-1', displayId: 3 })]);
160+
expect(bothLit.ambiguous).toBe(true);
161+
mockInventory.mockResolvedValueOnce(bothLit).mockResolvedValueOnce(bothLit);
162+
mockHinge.mockResolvedValue(180);
163+
164+
await expect(setAppleFoldPose(duo, 'open')).resolves.toEqual({
165+
pose: 'open',
166+
hingeAngleDegrees: 180,
167+
});
168+
});
169+
114170
test('maps half-open onto the Book preset and reports it only once the hinge has stopped', async () => {
115171
mockInventory
116172
.mockResolvedValueOnce(duoInventory('inner'))
@@ -122,6 +178,7 @@ test('maps half-open onto the Book preset and reports it only once the hinge has
122178
await expect(setAppleFoldPose(duo, 'half-open')).resolves.toMatchObject({
123179
pose: 'half-open',
124180
hingeAngleDegrees: 130,
181+
screen: { display: 'LCD-1', coordinateSpace: 'native-panel', widthPt: 669, heightPt: 951 },
125182
});
126183
expect(mockPress).toHaveBeenCalledWith(expect.objectContaining({ pose: 'book' }));
127184
expect(mockHinge).toHaveBeenCalledTimes(3);

‎packages/platform-apple/src/foldable/pose.ts‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
11
import { foldPoseForHingeAngle, type FoldPose } from '@agent-device/contracts/device';
2-
import type { FoldScreenReport, SetFoldPoseResult } from '@agent-device/contracts/fold-runtime';
2+
import {
3+
FOLD_SCREEN_COORDINATE_SPACE,
4+
type FoldScreenReport,
5+
type SetFoldPoseResult,
6+
} from '@agent-device/contracts/fold-runtime';
37
import type { DeviceInfo } from '@agent-device/kernel/device';
48
import { AppError } from '@agent-device/kernel/errors';
59
import { emitDiagnostic } from '@agent-device/host-kit/diagnostics';
@@ -151,9 +155,11 @@ async function readLitPanel(
151155
return inventory.activeDisplay;
152156
}
153157

158+
/** The panel's native points: pixels divided by its own point scale, never rotated by orientation. */
154159
function screenReport(display: AppleDeviceDisplay): FoldScreenReport {
155160
return {
156161
display: display.name,
162+
coordinateSpace: FOLD_SCREEN_COORDINATE_SPACE,
157163
widthPt: Math.round(display.widthPx / display.pointScale),
158164
heightPt: Math.round(display.heightPx / display.pointScale),
159165
};

‎src/commands/schema/cli-help.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -742,12 +742,12 @@ A foldable Apple device (iPhone Duo) carries two integrated panels, which Apple
742742
743743
Screens are handled for you:
744744
Each iOS simulator capture resolves the CoreDevice display table, captures the lit panel explicitly, and normalizes density with that panel's own point scale. Do not add a screen flag to the normal loop; there is none, because the lit panel is always the only capturable one: the dark panel yields an all-black PNG.
745-
The two panels are different sizes and different coordinate spaces (iPhone Duo: 466x678 points closed on the outer panel, 669x951 open on the inner). A pose change therefore invalidates every ref and coordinate. Re-snapshot after any pose change and never carry coordinates or refs across one.
745+
The two panels are different sizes (iPhone Duo: 466x678 points closed on the outer panel, 669x951 open on the inner). A pose change therefore invalidates every ref and coordinate. Re-snapshot after any pose change and never carry coordinates or refs across one.
746746
Check which panel is lit before trusting a geometry claim: agent-device screenshot reports its point size, and 466x678 versus 669x951 says which panel you captured.
747747
748748
Changing the pose:
749749
agent-device fold closed | half-open | open
750-
fold presses the pose control in the Xcode Device Hub window for this simulator (Closed, Book, Open) and then reads the hinge angle back from CoreDevice until it agrees: closed is 0 degrees, open is 180, and half-open is any angle between them (Device Hub's Book preset, 130 degrees on iOS 27.1), reported once the hinge stops moving or when the read budget ends while it still reads half-open. The response reports the verified pose, the hinge angle, and the panel the device now lights with its point size. A refusal never names the pose that was asked for: only a hinge whose last reading is some other pose fails, with COMMAND_FAILED and reason fold-pose-unverified. A single-panel simulator fails with UNSUPPORTED_OPERATION.
750+
fold presses the pose control in the Xcode Device Hub window for this simulator (Closed, Book, Open) and then reads the hinge angle back from CoreDevice until it agrees: closed is 0 degrees, open is 180, and half-open is any angle between them (Device Hub's Book preset, 130 degrees on iOS 27.1), reported once the hinge stops moving or when the read budget ends while it still reads half-open. The response reports the verified pose, the hinge angle, and the panel the device now lights with its native panel point size, marked coordinateSpace "native-panel". That point size is the panel's own geometry, not the next snapshot's viewport, so it cannot place a tap: the active app window can differ (a 669x951 inner panel hosts a 951x669 window). A refusal never names the pose that was asked for: only a hinge whose last reading is some other pose fails, with COMMAND_FAILED and reason fold-pose-unverified. A single-panel simulator fails with UNSUPPORTED_OPERATION.
751751
Expect a fold to take 10-16 seconds: each hinge read is a five-second devicectl stream, and half-open waits for the hinge to stop moving. Re-snapshot after every fold; refs and coordinates from before it are stale, and the command's message says so.
752752
Requirements: an iOS simulator session on a foldable device, Xcode 27.1 or newer with Device Hub, and Accessibility permission for the host (agent-device settings permission grant accessibility --platform macos). The command launches Device Hub if needed, reopens its window when it shows none, and selects the simulator through its sidebar by UDID, so no operator step is needed. No official host API sets the pose; the app under test still reads it as UIHinge.status.
753753
If a task asserts behavior for more than one pose, fold to each pose and re-snapshot, and report which poses the run covered.`,

‎src/daemon/__tests__/fold-runtime.test.ts‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -88,7 +88,12 @@ test('resolves one admitted binding and reports the pose the owner read back', a
8888
const setFoldPose = vi.fn(async () => ({
8989
pose: 'open' as const,
9090
hingeAngleDegrees: 180,
91-
screen: { display: 'LCD-1', widthPt: 669, heightPt: 951 },
91+
screen: {
92+
display: 'LCD-1',
93+
coordinateSpace: 'native-panel' as const,
94+
widthPt: 669,
95+
heightPt: 951,
96+
},
9297
}));
9398
const harness = runtimeHarness(
9499
foldRuntimeOperationFacts({ fold: available }).setFoldPose,
@@ -109,9 +114,9 @@ test('resolves one admitted binding and reports the pose the owner read back', a
109114
action: 'fold',
110115
pose: 'open',
111116
hingeAngleDegrees: 180,
112-
screen: { display: 'LCD-1', widthPt: 669, heightPt: 951 },
117+
screen: { display: 'LCD-1', coordinateSpace: 'native-panel', widthPt: 669, heightPt: 951 },
113118
message:
114-
'Folded to open (hinge 180°, LCD-1 lit at 669x951pt); refs from before the pose change are stale',
119+
'Folded to open (hinge 180°, LCD-1 native panel 669x951pt, not snapshot coordinates); refs from before the pose change are stale',
115120
});
116121
expect(setFoldPose).toHaveBeenCalledWith({ pose: 'open' });
117122
});

‎src/daemon/fold-runtime.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,7 @@ async function executeSetFoldPose(
5555
...(screen ? { screen } : {}),
5656
...successText(
5757
screen
58-
? `Folded to ${result.pose} (hinge ${result.hingeAngleDegrees}°, ${screen.display} lit at ${screen.widthPt}x${screen.heightPt}pt); refs from before the pose change are stale`
58+
? `Folded to ${result.pose} (hinge ${result.hingeAngleDegrees}°, ${screen.display} native panel ${screen.widthPt}x${screen.heightPt}pt, not snapshot coordinates); refs from before the pose change are stale`
5959
: `Folded to ${result.pose} (hinge ${result.hingeAngleDegrees}°); refs from before the pose change are stale`,
6060
),
6161
};

‎src/mcp/__tests__/command-tools-parity.test.ts‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -166,6 +166,40 @@ test('MCP fill projects target-bound unconfirmed verification through its advert
166166
assert.notDeepEqual(validateAgainstSchema(missingTarget, fillTool.outputSchema), []);
167167
});
168168

169+
test('MCP fold advertises the native-panel discriminator and rejects a screen missing it', async () => {
170+
const foldResult = {
171+
action: 'fold',
172+
pose: 'open',
173+
hingeAngleDegrees: 180,
174+
screen: { display: 'LCD-1', coordinateSpace: 'native-panel', widthPt: 669, heightPt: 951 },
175+
message:
176+
'Folded to open (hinge 180°, LCD-1 native panel 669x951pt, not snapshot coordinates); refs from before the pose change are stale',
177+
} satisfies CommandExecutionResult<'fold'>;
178+
const executor = createCommandToolExecutor({
179+
createClient: () => ({}) as AgentDeviceClient,
180+
runCommand: async () => foldResult,
181+
});
182+
183+
const foldTool = listCommandTools().find((tool) => tool.name === 'fold');
184+
assert.ok(foldTool?.outputSchema);
185+
assert.equal(foldTool.outputSchema, COMMAND_OUTPUT_SCHEMAS.fold);
186+
187+
const result = await executor.execute('fold', { pose: 'open' });
188+
assert.deepEqual(result.structuredContent, foldResult);
189+
assert.deepEqual(validateAgainstSchema(result.structuredContent, foldTool.outputSchema), []);
190+
191+
// A panel report that drops the discriminator no longer validates: the coordinate-space contract
192+
// is required, not inferable from the numbers alone.
193+
const { coordinateSpace: _coordinateSpace, ...screenWithoutDiscriminator } = foldResult.screen;
194+
assert.notDeepEqual(
195+
validateAgainstSchema(
196+
{ ...foldResult, screen: screenWithoutDiscriminator },
197+
foldTool.outputSchema,
198+
),
199+
[],
200+
);
201+
});
202+
169203
test('MCP applies config-backed command defaults; explicit operator input is refused', async () => {
170204
const home = mkdtempForTestSync('agent-device-mcp-config-');
171205
temporaryDirectory = home;

‎src/mcp/command-output-schemas.ts‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ import { BACK_MODES } from '@agent-device/contracts/back-mode';
66
import { NATIVE_PATH_DISPOSITION_VALUES } from '@agent-device/contracts/recording-native-path';
77
import { RECORDER_OBSERVATION_VALUES } from '@agent-device/contracts/recording-stop-observation';
88
import { DEVICE_ROTATIONS, FOLD_POSES } from '@agent-device/contracts/device';
9+
import { FOLD_SCREEN_COORDINATE_SPACE } from '@agent-device/contracts/fold-runtime';
910
import { SESSION_SURFACES } from '@agent-device/contracts/session';
1011
import { TV_REMOTE_BUTTONS } from '@agent-device/contracts/tv-remote';
1112
import { DEVICE_TARGETS, PUBLIC_PLATFORMS } from '@agent-device/kernel/device';
@@ -525,10 +526,15 @@ const BASE_COMMAND_OUTPUT_SCHEMAS = {
525526
screen: objectSchema(
526527
{
527528
display: stringSchema('CoreDevice name of the panel the device now lights.'),
528-
widthPt: numberSchema(),
529-
heightPt: numberSchema(),
529+
coordinateSpace: constSchema(FOLD_SCREEN_COORDINATE_SPACE),
530+
widthPt: numberSchema(
531+
'Panel width in native panel points (pixels divided by point scale), NOT snapshot coordinates; take a fresh snapshot to place a tap.',
532+
),
533+
heightPt: numberSchema(
534+
'Panel height in native panel points (pixels divided by point scale), NOT snapshot coordinates; take a fresh snapshot to place a tap.',
535+
),
530536
},
531-
['display', 'widthPt', 'heightPt'],
537+
['display', 'coordinateSpace', 'widthPt', 'heightPt'],
532538
),
533539
message: stringSchema(),
534540
},

0 commit comments

Comments
 (0)