Skip to content

Commit d783deb

Browse files
thymikeethymikee
committed
fix(fold): report screen dimensions in their native-panel coordinate space
The fold response's `screen.widthPt`/`heightPt` are the lit panel's native points (pixels divided by its point scale, never rotated), not the next snapshot's viewport. Add the required `screen.coordinateSpace: "native-panel"` discriminator at the construction site, export it from the owning contract so the Apple owner and the MCP schema share one value, and correct every doc and message that implied these numbers could place a tap. A fresh snapshot stays the only source of app-viewport coordinates: on an open Duo the 669x951 inner panel hosts a 951x669 app window. Closes #2729
1 parent 08d977f commit d783deb

11 files changed

Lines changed: 141 additions & 18 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -34,13 +34,18 @@
3434
the pose control in the Xcode Device Hub window through macOS accessibility, and it is confirmed
3535
the way ADR 0025 asked for, by reading the hinge angle back from CoreDevice
3636
(`devicectl device motion hinge-angle`) until it agrees with the request. The response reports
37-
the verified pose, the hinge angle, and the panel the device now lights with its point size, so an
38-
agent can see that its refs and coordinates are stale without another capture. The macOS helper
37+
the verified pose, the hinge angle, and the panel the device now lights with its native panel point
38+
size, so an agent can see the lit panel changed without another capture. The macOS helper
3939
gained a `device-hub pose` subcommand that finds Device Hub in the process table (LaunchServices
4040
registers the trampolined app with no pid), reopens its window when it shows none, and selects the
4141
simulator through the sidebar row keyed by its UDID so two simulators sharing a name cannot be
4242
confused. Simulator-only; a single-panel simulator refuses with `UNSUPPORTED_OPERATION`, and every
4343
other platform states its own refusal cell.
44+
- Fixed (ios): the `fold` response's `screen` dimensions now match their coordinate-space contract. They
45+
are the lit panel's own native points (pixels divided by its point scale, never rotated), so the
46+
response carries the `coordinateSpace: "native-panel"` discriminator and the docs no longer claim
47+
they are the next snapshot's coordinates. They cannot place a tap; take a fresh snapshot for the app
48+
viewport.
4449
- Added (diff): `diff screenshot` accepts a JPEG baseline or current image. Both inputs had to be PNG,
4550
so a capture exported by another tool had to be converted first and a HarmonyOS capture — which the
4651
platform serves as JPEG under whatever name the command was given — could never be compared. Each

‎packages/contracts/src/fold-runtime.ts‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,13 +8,22 @@ import type { RuntimeOperationFact } from './platform-runtime.ts';
88
*/
99
export type SetFoldPoseInput = Readonly<{ pose: FoldPose }>;
1010

11+
/** Single source of truth for the discriminator the Apple owner sets and the MCP schema advertises. */
12+
export const FOLD_SCREEN_COORDINATE_SPACE = 'native-panel' as const;
13+
1114
/**
12-
* The lit panel after the pose settled, in the points the next snapshot will use. Reported so an
13-
* agent can see that the coordinate space changed without a second capture.
15+
* The panel the device lights after the pose settled, in that panel's own native points: its pixel
16+
* size divided by its point scale, never rotated. `coordinateSpace` is always
17+
* {@link FOLD_SCREEN_COORDINATE_SPACE}, and these numbers are NOT snapshot coordinates — the active
18+
* app window can differ from the panel (iPhone Duo: a 669x951 inner panel hosts a 951x669 app
19+
* window), so they cannot place a tap. A caller that needs the app viewport must take a fresh
20+
* snapshot.
1421
*/
1522
export type FoldScreenReport = Readonly<{
1623
/** The CoreDevice display name of the panel the device now lights. */
1724
display: string;
25+
/** Marks these dimensions as the panel's native points, never a snapshot's app viewport. */
26+
coordinateSpace: typeof FOLD_SCREEN_COORDINATE_SPACE;
1827
widthPt: number;
1928
heightPt: number;
2029
}>;

‎packages/contracts/src/navigation.ts‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -53,8 +53,9 @@ export type OrientationCommandResult = {
5353
*
5454
* Unlike `orientation`, there is no unconfirmed variant: the Apple owner reads the hinge angle
5555
* back from CoreDevice after pressing the Device Hub pose control, and reports a pose only when
56-
* that reading agrees with the request. `screen` names the panel the device lights afterwards, in
57-
* points, because a pose change moves the app to a different coordinate space (ADR 0025).
56+
* that reading agrees with the request. `screen` names the panel the device lights afterwards and
57+
* that panel's native point size (ADR 0025); it is the panel's geometry, not the app viewport, so a
58+
* caller must take a fresh snapshot before placing a tap.
5859
*/
5960
export type FoldCommandResult = {
6061
action: 'fold';

‎packages/platform-apple/src/foldable/pose.test.ts‎

Lines changed: 58 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -99,7 +99,7 @@ test('presses the Device Hub control for the pose and reports the pose CoreDevic
9999
await expect(setAppleFoldPose(duo, 'open')).resolves.toEqual({
100100
pose: 'open',
101101
hingeAngleDegrees: 180,
102-
screen: { display: 'LCD-1', widthPt: 669, heightPt: 951 },
102+
screen: { display: 'LCD-1', coordinateSpace: 'native-panel', widthPt: 669, heightPt: 951 },
103103
});
104104

105105
expect(mockPress).toHaveBeenCalledWith({
@@ -111,6 +111,62 @@ test('presses the Device Hub control for the pose and reports the pose CoreDevic
111111
expect(mockHinge).toHaveBeenCalledTimes(2);
112112
});
113113

114+
test('reports the closed outer panel in native points, not rotated to a snapshot viewport', async () => {
115+
// The outer panel is 1398x2034 px at scale 3, i.e. 466x678 native points; the pose must not
116+
// rotate that into the app window's shape, because a caller cannot place a tap from it.
117+
mockInventory
118+
.mockResolvedValueOnce(duoInventory('outer'))
119+
.mockResolvedValueOnce(duoInventory('outer'));
120+
mockHinge.mockResolvedValue(0);
121+
122+
await expect(setAppleFoldPose(duo, 'closed')).resolves.toEqual({
123+
pose: 'closed',
124+
hingeAngleDegrees: 0,
125+
screen: { display: 'LCD', coordinateSpace: 'native-panel', widthPt: 466, heightPt: 678 },
126+
});
127+
});
128+
129+
test("reports native panel points regardless of the display's own currentOrientation", async () => {
130+
// The same inner panel (2007x2853 px at scale 3) with a portrait orientation tag still measures
131+
// 669x951: the report divides by point scale only and never swaps on `currentOrientation`.
132+
const rot0Inner = buildInventory([
133+
panel({ power: 'dark' }),
134+
panel({
135+
name: 'LCD-1',
136+
displayId: 3,
137+
primary: false,
138+
power: 'lit',
139+
widthPx: 2007,
140+
heightPx: 2853,
141+
currentOrientation: 'rot0',
142+
}),
143+
]);
144+
mockInventory.mockResolvedValueOnce(duoInventory('inner')).mockResolvedValueOnce(rot0Inner);
145+
mockHinge.mockResolvedValue(180);
146+
147+
const result = await setAppleFoldPose(duo, 'open');
148+
expect(result.screen).toEqual({
149+
display: 'LCD-1',
150+
coordinateSpace: 'native-panel',
151+
widthPt: 669,
152+
heightPt: 951,
153+
});
154+
});
155+
156+
test('omits the screen report when panel selection is ambiguous, never inventing a viewport', async () => {
157+
// Two lit panels make the inventory ambiguous, so `readLitPanel` refuses to name one; the pose is
158+
// still verified but carries no `screen` rather than guessing the app viewport.
159+
const bothLit = buildInventory([panel({}), panel({ name: 'LCD-1', displayId: 3 })]);
160+
expect(bothLit.ambiguous).toBe(true);
161+
mockInventory.mockResolvedValueOnce(bothLit).mockResolvedValueOnce(bothLit);
162+
mockHinge.mockResolvedValue(180);
163+
164+
await expect(setAppleFoldPose(duo, 'open')).resolves.toEqual({
165+
pose: 'open',
166+
hingeAngleDegrees: 180,
167+
});
168+
});
169+
114170
test('maps half-open onto the Book preset and reports it only once the hinge has stopped', async () => {
115171
mockInventory
116172
.mockResolvedValueOnce(duoInventory('inner'))
@@ -122,6 +178,7 @@ test('maps half-open onto the Book preset and reports it only once the hinge has
122178
await expect(setAppleFoldPose(duo, 'half-open')).resolves.toMatchObject({
123179
pose: 'half-open',
124180
hingeAngleDegrees: 130,
181+
screen: { display: 'LCD-1', coordinateSpace: 'native-panel', widthPt: 669, heightPt: 951 },
125182
});
126183
expect(mockPress).toHaveBeenCalledWith(expect.objectContaining({ pose: 'book' }));
127184
expect(mockHinge).toHaveBeenCalledTimes(3);

‎packages/platform-apple/src/foldable/pose.ts‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
11
import { foldPoseForHingeAngle, type FoldPose } from '@agent-device/contracts/device';
2-
import type { FoldScreenReport, SetFoldPoseResult } from '@agent-device/contracts/fold-runtime';
2+
import {
3+
FOLD_SCREEN_COORDINATE_SPACE,
4+
type FoldScreenReport,
5+
type SetFoldPoseResult,
6+
} from '@agent-device/contracts/fold-runtime';
37
import type { DeviceInfo } from '@agent-device/kernel/device';
48
import { AppError } from '@agent-device/kernel/errors';
59
import { emitDiagnostic } from '@agent-device/host-kit/diagnostics';
@@ -196,9 +200,11 @@ async function readLitPanel(
196200
return inventory.activeDisplay;
197201
}
198202

203+
/** The panel's native points: pixels divided by its own point scale, never rotated by orientation. */
199204
function screenReport(display: AppleDeviceDisplay): FoldScreenReport {
200205
return {
201206
display: display.name,
207+
coordinateSpace: FOLD_SCREEN_COORDINATE_SPACE,
202208
widthPt: Math.round(display.widthPx / display.pointScale),
203209
heightPt: Math.round(display.heightPx / display.pointScale),
204210
};

‎src/commands/schema/cli-help.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -742,12 +742,12 @@ A foldable Apple device (iPhone Duo) carries two integrated panels, which Apple
742742
743743
Screens are handled for you:
744744
Each iOS simulator capture resolves the CoreDevice display table, captures the lit panel explicitly, and normalizes density with that panel's own point scale. Do not add a screen flag to the normal loop; there is none, because the lit panel is always the only capturable one: the dark panel yields an all-black PNG.
745-
The two panels are different sizes and different coordinate spaces (iPhone Duo: 466x678 points closed on the outer panel, 669x951 open on the inner). A pose change therefore invalidates every ref and coordinate. Re-snapshot after any pose change and never carry coordinates or refs across one.
745+
The two panels are different sizes (iPhone Duo: 466x678 points closed on the outer panel, 669x951 open on the inner). A pose change therefore invalidates every ref and coordinate. Re-snapshot after any pose change and never carry coordinates or refs across one.
746746
Check which panel is lit before trusting a geometry claim: agent-device screenshot reports its point size, and 466x678 versus 669x951 says which panel you captured.
747747
748748
Changing the pose:
749749
agent-device fold closed | half-open | open
750-
fold presses the pose control in the Xcode Device Hub window for this simulator (Closed, Book, Open) and then reads the hinge angle back from CoreDevice until it agrees: closed is 0 degrees, open is 180, and half-open is any angle between them (Device Hub's Book preset, 130 degrees on iOS 27.1). An angle in that interval only proves the category, so half-open is reported once two consecutive readings both fall inside it and agree within 0.5 degrees. The response reports the verified pose, the hinge angle, and the panel the device now lights with its point size. A hinge whose last reading is some other pose fails with COMMAND_FAILED and reason fold-pose-unverified, naming the angle CoreDevice still reports; a hinge seen half-open but never at rest fails with reason fold-pose-unsettled, naming the observed and previous angles. A single-panel simulator fails with UNSUPPORTED_OPERATION.
750+
fold presses the pose control in the Xcode Device Hub window for this simulator (Closed, Book, Open) and then reads the hinge angle back from CoreDevice until it agrees: closed is 0 degrees, open is 180, and half-open is any angle between them (Device Hub's Book preset, 130 degrees on iOS 27.1). An angle in that interval only proves the category, so half-open is reported once two consecutive readings both fall inside it and agree within 0.5 degrees. The response reports the verified pose, the hinge angle, and the panel the device now lights with its native panel point size, marked coordinateSpace "native-panel". That point size is the panel's own geometry, not the next snapshot's viewport, so it cannot place a tap: the active app window can differ (a 669x951 inner panel hosts a 951x669 window). A hinge whose last reading is some other pose fails with COMMAND_FAILED and reason fold-pose-unverified, naming the angle CoreDevice still reports; a hinge seen half-open but never at rest fails with reason fold-pose-unsettled, naming the observed and previous angles. A single-panel simulator fails with UNSUPPORTED_OPERATION.
751751
Expect a fold to take 10-16 seconds: each hinge read is a five-second devicectl stream, and half-open waits for the hinge to stop moving. Re-snapshot after every fold; refs and coordinates from before it are stale, and the command's message says so.
752752
Requirements: an iOS simulator session on a foldable device, Xcode 27.1 or newer with Device Hub, and Accessibility permission for the host (agent-device settings permission grant accessibility --platform macos). The command launches Device Hub if needed, reopens its window when it shows none, and selects the simulator through its sidebar by UDID, so no operator step is needed. No official host API sets the pose; the app under test still reads it as UIHinge.status.
753753
If a task asserts behavior for more than one pose, fold to each pose and re-snapshot, and report which poses the run covered.`,

‎src/daemon/__tests__/fold-runtime.test.ts‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -88,7 +88,12 @@ test('resolves one admitted binding and reports the pose the owner read back', a
8888
const setFoldPose = vi.fn(async () => ({
8989
pose: 'open' as const,
9090
hingeAngleDegrees: 180,
91-
screen: { display: 'LCD-1', widthPt: 669, heightPt: 951 },
91+
screen: {
92+
display: 'LCD-1',
93+
coordinateSpace: 'native-panel' as const,
94+
widthPt: 669,
95+
heightPt: 951,
96+
},
9297
}));
9398
const harness = runtimeHarness(
9499
foldRuntimeOperationFacts({ fold: available }).setFoldPose,
@@ -109,9 +114,9 @@ test('resolves one admitted binding and reports the pose the owner read back', a
109114
action: 'fold',
110115
pose: 'open',
111116
hingeAngleDegrees: 180,
112-
screen: { display: 'LCD-1', widthPt: 669, heightPt: 951 },
117+
screen: { display: 'LCD-1', coordinateSpace: 'native-panel', widthPt: 669, heightPt: 951 },
113118
message:
114-
'Folded to open (hinge 180°, LCD-1 lit at 669x951pt); refs from before the pose change are stale',
119+
'Folded to open (hinge 180°, LCD-1 native panel 669x951pt, not snapshot coordinates); refs from before the pose change are stale',
115120
});
116121
expect(setFoldPose).toHaveBeenCalledWith({ pose: 'open' });
117122
});

‎src/daemon/fold-runtime.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,7 @@ async function executeSetFoldPose(
5555
...(screen ? { screen } : {}),
5656
...successText(
5757
screen
58-
? `Folded to ${result.pose} (hinge ${result.hingeAngleDegrees}°, ${screen.display} lit at ${screen.widthPt}x${screen.heightPt}pt); refs from before the pose change are stale`
58+
? `Folded to ${result.pose} (hinge ${result.hingeAngleDegrees}°, ${screen.display} native panel ${screen.widthPt}x${screen.heightPt}pt, not snapshot coordinates); refs from before the pose change are stale`
5959
: `Folded to ${result.pose} (hinge ${result.hingeAngleDegrees}°); refs from before the pose change are stale`,
6060
),
6161
};

‎src/mcp/__tests__/command-tools-parity.test.ts‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -166,6 +166,40 @@ test('MCP fill projects target-bound unconfirmed verification through its advert
166166
assert.notDeepEqual(validateAgainstSchema(missingTarget, fillTool.outputSchema), []);
167167
});
168168

169+
test('MCP fold advertises the native-panel discriminator and rejects a screen missing it', async () => {
170+
const foldResult = {
171+
action: 'fold',
172+
pose: 'open',
173+
hingeAngleDegrees: 180,
174+
screen: { display: 'LCD-1', coordinateSpace: 'native-panel', widthPt: 669, heightPt: 951 },
175+
message:
176+
'Folded to open (hinge 180°, LCD-1 native panel 669x951pt, not snapshot coordinates); refs from before the pose change are stale',
177+
} satisfies CommandExecutionResult<'fold'>;
178+
const executor = createCommandToolExecutor({
179+
createClient: () => ({}) as AgentDeviceClient,
180+
runCommand: async () => foldResult,
181+
});
182+
183+
const foldTool = listCommandTools().find((tool) => tool.name === 'fold');
184+
assert.ok(foldTool?.outputSchema);
185+
assert.equal(foldTool.outputSchema, COMMAND_OUTPUT_SCHEMAS.fold);
186+
187+
const result = await executor.execute('fold', { pose: 'open' });
188+
assert.deepEqual(result.structuredContent, foldResult);
189+
assert.deepEqual(validateAgainstSchema(result.structuredContent, foldTool.outputSchema), []);
190+
191+
// A panel report that drops the discriminator no longer validates: the coordinate-space contract
192+
// is required, not inferable from the numbers alone.
193+
const { coordinateSpace: _coordinateSpace, ...screenWithoutDiscriminator } = foldResult.screen;
194+
assert.notDeepEqual(
195+
validateAgainstSchema(
196+
{ ...foldResult, screen: screenWithoutDiscriminator },
197+
foldTool.outputSchema,
198+
),
199+
[],
200+
);
201+
});
202+
169203
test('MCP applies config-backed command defaults; explicit operator input is refused', async () => {
170204
const home = mkdtempForTestSync('agent-device-mcp-config-');
171205
temporaryDirectory = home;

‎src/mcp/command-output-schemas.ts‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ import { BACK_MODES } from '@agent-device/contracts/back-mode';
66
import { NATIVE_PATH_DISPOSITION_VALUES } from '@agent-device/contracts/recording-native-path';
77
import { RECORDER_OBSERVATION_VALUES } from '@agent-device/contracts/recording-stop-observation';
88
import { DEVICE_ROTATIONS, FOLD_POSES } from '@agent-device/contracts/device';
9+
import { FOLD_SCREEN_COORDINATE_SPACE } from '@agent-device/contracts/fold-runtime';
910
import { SESSION_SURFACES } from '@agent-device/contracts/session';
1011
import { TV_REMOTE_BUTTONS } from '@agent-device/contracts/tv-remote';
1112
import { DEVICE_TARGETS, PUBLIC_PLATFORMS } from '@agent-device/kernel/device';
@@ -525,10 +526,15 @@ const BASE_COMMAND_OUTPUT_SCHEMAS = {
525526
screen: objectSchema(
526527
{
527528
display: stringSchema('CoreDevice name of the panel the device now lights.'),
528-
widthPt: numberSchema(),
529-
heightPt: numberSchema(),
529+
coordinateSpace: constSchema(FOLD_SCREEN_COORDINATE_SPACE),
530+
widthPt: numberSchema(
531+
'Panel width in native panel points (pixels divided by point scale), NOT snapshot coordinates; take a fresh snapshot to place a tap.',
532+
),
533+
heightPt: numberSchema(
534+
'Panel height in native panel points (pixels divided by point scale), NOT snapshot coordinates; take a fresh snapshot to place a tap.',
535+
),
530536
},
531-
['display', 'widthPt', 'heightPt'],
537+
['display', 'coordinateSpace', 'widthPt', 'heightPt'],
532538
),
533539
message: stringSchema(),
534540
},

0 commit comments

Comments
 (0)