Skip to content

Commit cfd8dc1

Browse files
committed
docs: bless the works-today iOS SpringBoard/widget workflow (#1296 PR A)
Live probe on iOS 26.2/Xcode 26.2 proved open com.apple.springboard already binds a driveable SpringBoard session with zero code changes: the full widget add/edit/remove flow is selector-driven from a fresh snapshot, aside from two documented coordinate fallbacks. Add a help ios-system-ui topic (and cross-links from physical-device/workflow, the skill router, and docs/commands.md) so agents can use it today, ahead of the --ui-target contract and gallery-capture fix landing.
1 parent 5ef52ad commit cfd8dc1

4 files changed

Lines changed: 65 additions & 2 deletions

File tree

‎skills/agent-device/SKILL.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ agent-device help remote
3535
agent-device help macos
3636
agent-device help dogfood
3737
agent-device help tv
38+
agent-device help ios-system-ui # iOS SpringBoard, widgets, and system-UI surfaces
3839
```
3940

4041
Default loop: `open -> snapshot/-i -> get/is/find or press/fill/scroll/wait -> verify -> close`. When target-specific help says capture or selectors are unsupported, use its control-only loop and the device display as visual truth.

‎src/cli/parser/__tests__/cli-help-topics.test.ts‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -183,6 +183,10 @@ test('usage includes agent workflows, config, environment, and examples footers'
183183
usageText,
184184
/agent-device help physical-device\s+Use when using a connected phone\/tablet or iOS signing setup/,
185185
);
186+
assert.match(
187+
usageText,
188+
/agent-device help ios-system-ui\s+Use when driving iOS SpringBoard, widgets, or other system-UI surfaces/,
189+
);
186190
assert.match(
187191
usageText,
188192
/agent-device help react-native\s+Use when the target app is React Native, Expo, or a dev client/,
@@ -502,6 +506,17 @@ test('usageForCommand resolves physical-device help topic', async () => {
502506
);
503507
});
504508

509+
test('usageForCommand resolves ios-system-ui help topic', async () => {
510+
const help = await usageForCommand('ios-system-ui');
511+
if (help === null) throw new Error('Expected ios-system-ui help text');
512+
assert.match(help, /agent-device help ios-system-ui/);
513+
assert.match(help, /agent-device open com\.apple\.springboard --platform ios/);
514+
assert.match(help, /longpress <x> <y> on an empty area of the home screen/);
515+
assert.match(help, /discover them from the current snapshot/);
516+
assert.match(help, /Do not hard-code Edit\/Done\/Add Widget or other SpringBoard label text/);
517+
assert.match(help, /Reopen the app bundle under test/);
518+
});
519+
505520
test('usageForCommand resolves manual QA help topic', async () => {
506521
const help = await usageForCommand('manual-qa');
507522
if (help === null) throw new Error('Expected manual QA help text');

‎src/cli/parser/cli-help.ts‎

Lines changed: 39 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,10 @@ const AGENT_WORKFLOWS = [
5757
label: 'agent-device help physical-device',
5858
description: 'Use when using a connected phone/tablet or iOS signing setup',
5959
},
60+
{
61+
label: 'agent-device help ios-system-ui',
62+
description: 'Use when driving iOS SpringBoard, widgets, or other system-UI surfaces',
63+
},
6064
{
6165
label: 'agent-device help remote',
6266
description: 'Use when working through cloud config, tenants, leases, or local tunnels',
@@ -438,7 +442,8 @@ Escalate:
438442
help react-devtools React Native performance, profiling, props/state/hooks, slow renders, rerenders
439443
help react-native React Native app automation hazards, overlays, Metro/Re.Pack, and routing
440444
help remote remote/cloud config, tenant, lease, local service tunnels
441-
help macos desktop, frontmost-app, menu bar surfaces`,
445+
help macos desktop, frontmost-app, menu bar surfaces
446+
help ios-system-ui iOS SpringBoard, widget add/edit/remove, system-UI surfaces`,
442447
},
443448
tv: {
444449
summary: 'Android TV, tvOS, and Vega VVD focus-first remote navigation',
@@ -842,7 +847,39 @@ iOS physical-device prerequisites:
842847
843848
Android physical-device prerequisites:
844849
Enable USB debugging and confirm the device appears in agent-device devices --platform android.
845-
Android does not need the iOS runner signing setup. For React Native/Expo Metro reachability, read help react-native.`,
850+
Android does not need the iOS runner signing setup. For React Native/Expo Metro reachability, read help react-native.
851+
852+
For iOS SpringBoard, widget, or other system-UI surfaces, read agent-device help ios-system-ui.`,
853+
},
854+
'ios-system-ui': {
855+
summary: 'iOS SpringBoard, widget, and system-surface workflow',
856+
body: `agent-device help ios-system-ui
857+
858+
Use this when a task needs iOS SpringBoard (home screen), widget add/edit/remove, or other system-UI surfaces instead of the app under test.
859+
860+
This works today by opening SpringBoard as the session app; there is no separate widget/system command. System labels vary by iOS version and locale, so discover them from the current snapshot instead of relying on the literal strings shown below.
861+
862+
Core loop:
863+
1. Reach the app state you want to prepare (for example, arrange the widget/Live Activity data the app should show) with normal app automation, then agent-device open com.apple.springboard --platform ios. From an existing app session, agent-device home first also lands on the home screen, but open com.apple.springboard is what actually binds the session to SpringBoard for selector-driven commands.
864+
2. agent-device snapshot -i to read the current localized SpringBoard controls.
865+
3. agent-device longpress <x> <y> on an empty area of the home screen to enter edit mode. This is the one deliberate coordinate step; there is no reliable non-coordinate way to trigger it.
866+
4. Re-snapshot and use selectors from the fresh tree to drive the Edit menu -> widget gallery -> search -> size picker -> Add Widget, for example:
867+
agent-device snapshot -i
868+
agent-device press 'label="Add Widget"'
869+
agent-device fill 'label="Search"' "Calendar"
870+
5. The widget-gallery search-result rows currently fall back to unlabeled nodes (a known capture gap), so tap the result by coordinates read from a screenshot until that is fixed:
871+
agent-device screenshot
872+
agent-device press <x> <y>
873+
6. Continue with the semantic size picker and agent-device press 'label="Add Widget"' to place it.
874+
7. To edit or remove an installed widget, longpress it, then re-snapshot and use the fresh context-menu selectors (Edit Widget / Remove Widget).
875+
8. Use screenshots for visual assertions, and as the fallback wherever a system surface exposes sparse accessibility, not only in the gallery step.
876+
9. Reopen the app bundle under test (agent-device open <app-id> --platform ios) to return to normal app automation; leaving SpringBoard bound does not resume the app session on its own.
877+
878+
Rules:
879+
Do not hard-code Edit/Done/Add Widget or other SpringBoard label text into a plan as a fixed assumption; take them from the latest snapshot -i so the plan survives iOS version/locale differences.
880+
A real system permission alert can appear mid-flow; it composes with this workflow normally, so handle it with alert wait/accept/dismiss or by pressing the visible label like any other step.
881+
Prefer refs/selectors from the fresh snapshot for every step except the two documented coordinate fallbacks (empty-space long-press to enter edit mode, and the gallery search-result tap).
882+
This topic covers what already works by opening SpringBoard as the session app. It does not yet cover keeping an app session open while alternating individual commands against SpringBoard, or Live Activity/Dynamic Island semantics; those land separately.`,
846883
},
847884
remote: {
848885
summary: 'Direct proxy, cloud profiles, and remote config',

‎website/docs/docs/commands.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ agent-device help remote
2020
agent-device help web
2121
agent-device help macos
2222
agent-device help dogfood
23+
agent-device help ios-system-ui
2324
```
2425

2526
Skills are recommended for auto-routing when your agent runtime supports them, but they are not required. The CLI help topics are the version-matched operating contract.
@@ -1037,3 +1038,12 @@ For CLI-discoverable setup guidance, run `agent-device help physical-device`.
10371038
- If you override the iOS runner derived-data path and also force cleanup, keep `AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH` under the project `.tmp/` directory. Other cleanup override paths are rejected with a recovery hint.
10381039
- For daemon startup troubleshooting:
10391040
- follow stale metadata hints for `<state-dir>/daemon.json` and `<state-dir>/daemon.lock` (`state-dir` defaults to `~/.agent-device` for packaged installs, or a worktree-scoped dir under `~/.agent-device/dev/` from source)
1041+
1042+
## iOS SpringBoard, widgets, and system-UI surfaces
1043+
1044+
For CLI-discoverable workflow guidance, run `agent-device help ios-system-ui`.
1045+
1046+
- `agent-device open com.apple.springboard --platform ios` binds the session to SpringBoard on a simulator or physical device today; there is no separate widget/system command.
1047+
- The full widget add/edit/remove flow is selector-driven from a fresh `snapshot -i`, except two coordinate-based steps: the empty-space long-press that enters edit mode, and (until fixed) the widget-gallery search-result rows, which currently return unlabeled accessibility nodes.
1048+
- SpringBoard labels vary by iOS version and locale; discover them from the current snapshot rather than hard-coding strings like `Edit` or `Add Widget`.
1049+
- Reopen the app bundle under test to return to normal app automation after a SpringBoard step.

0 commit comments

Comments
 (0)