You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit cfd8dc1
Browse filesBrowse the repository at this point in the historyBrowse files
docs: bless the works-today iOS SpringBoard/widget workflow (#1296 PR A)
Live probe on iOS 26.2/Xcode 26.2 proved open com.apple.springboard
already binds a driveable SpringBoard session with zero code changes:
the full widget add/edit/remove flow is selector-driven from a fresh
snapshot, aside from two documented coordinate fallbacks. Add a
help ios-system-ui topic (and cross-links from physical-device/workflow,
the skill router, and docs/commands.md) so agents can use it today,
ahead of the --ui-target contract and gallery-capture fix landing.
Copy file name to clipboardExpand all lines: skills/agent-device/SKILL.md
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -35,6 +35,7 @@ agent-device help remote
35
35
agent-device help macos
36
36
agent-device help dogfood
37
37
agent-device help tv
38
+
agent-device help ios-system-ui # iOS SpringBoard, widgets, and system-UI surfaces
38
39
```
39
40
40
41
Default loop: `open -> snapshot/-i -> get/is/find or press/fill/scroll/wait -> verify -> close`. When target-specific help says capture or selectors are unsupported, use its control-only loop and the device display as visual truth.
Enable USB debugging and confirm the device appears in agent-device devices --platform android.
845
-
Android does not need the iOS runner signing setup. For React Native/Expo Metro reachability, read help react-native.`,
850
+
Android does not need the iOS runner signing setup. For React Native/Expo Metro reachability, read help react-native.
851
+
852
+
For iOS SpringBoard, widget, or other system-UI surfaces, read agent-device help ios-system-ui.`,
853
+
},
854
+
'ios-system-ui': {
855
+
summary: 'iOS SpringBoard, widget, and system-surface workflow',
856
+
body: `agent-device help ios-system-ui
857
+
858
+
Use this when a task needs iOS SpringBoard (home screen), widget add/edit/remove, or other system-UI surfaces instead of the app under test.
859
+
860
+
This works today by opening SpringBoard as the session app; there is no separate widget/system command. System labels vary by iOS version and locale, so discover them from the current snapshot instead of relying on the literal strings shown below.
861
+
862
+
Core loop:
863
+
1. Reach the app state you want to prepare (for example, arrange the widget/Live Activity data the app should show) with normal app automation, then agent-device open com.apple.springboard --platform ios. From an existing app session, agent-device home first also lands on the home screen, but open com.apple.springboard is what actually binds the session to SpringBoard for selector-driven commands.
864
+
2. agent-device snapshot -i to read the current localized SpringBoard controls.
865
+
3. agent-device longpress <x> <y> on an empty area of the home screen to enter edit mode. This is the one deliberate coordinate step; there is no reliable non-coordinate way to trigger it.
866
+
4. Re-snapshot and use selectors from the fresh tree to drive the Edit menu -> widget gallery -> search -> size picker -> Add Widget, for example:
867
+
agent-device snapshot -i
868
+
agent-device press 'label="Add Widget"'
869
+
agent-device fill 'label="Search"' "Calendar"
870
+
5. The widget-gallery search-result rows currently fall back to unlabeled nodes (a known capture gap), so tap the result by coordinates read from a screenshot until that is fixed:
871
+
agent-device screenshot
872
+
agent-device press <x> <y>
873
+
6. Continue with the semantic size picker and agent-device press 'label="Add Widget"' to place it.
874
+
7. To edit or remove an installed widget, longpress it, then re-snapshot and use the fresh context-menu selectors (Edit Widget / Remove Widget).
875
+
8. Use screenshots for visual assertions, and as the fallback wherever a system surface exposes sparse accessibility, not only in the gallery step.
876
+
9. Reopen the app bundle under test (agent-device open <app-id> --platform ios) to return to normal app automation; leaving SpringBoard bound does not resume the app session on its own.
877
+
878
+
Rules:
879
+
Do not hard-code Edit/Done/Add Widget or other SpringBoard label text into a plan as a fixed assumption; take them from the latest snapshot -i so the plan survives iOS version/locale differences.
880
+
A real system permission alert can appear mid-flow; it composes with this workflow normally, so handle it with alert wait/accept/dismiss or by pressing the visible label like any other step.
881
+
Prefer refs/selectors from the fresh snapshot for every step except the two documented coordinate fallbacks (empty-space long-press to enter edit mode, and the gallery search-result tap).
882
+
This topic covers what already works by opening SpringBoard as the session app. It does not yet cover keeping an app session open while alternating individual commands against SpringBoard, or Live Activity/Dynamic Island semantics; those land separately.`,
846
883
},
847
884
remote: {
848
885
summary: 'Direct proxy, cloud profiles, and remote config',
Copy file name to clipboardExpand all lines: website/docs/docs/commands.md
+10Lines changed: 10 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -20,6 +20,7 @@ agent-device help remote
20
20
agent-device help web
21
21
agent-device help macos
22
22
agent-device help dogfood
23
+
agent-device help ios-system-ui
23
24
```
24
25
25
26
Skills are recommended for auto-routing when your agent runtime supports them, but they are not required. The CLI help topics are the version-matched operating contract.
@@ -1037,3 +1038,12 @@ For CLI-discoverable setup guidance, run `agent-device help physical-device`.
1037
1038
- If you override the iOS runner derived-data path and also force cleanup, keep `AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH` under the project `.tmp/` directory. Other cleanup override paths are rejected with a recovery hint.
1038
1039
- For daemon startup troubleshooting:
1039
1040
- follow stale metadata hints for`<state-dir>/daemon.json` and `<state-dir>/daemon.lock` (`state-dir` defaults to `~/.agent-device`for packaged installs, or a worktree-scoped dir under `~/.agent-device/dev/` from source)
1041
+
1042
+
## iOS SpringBoard, widgets, and system-UI surfaces
1043
+
1044
+
For CLI-discoverable workflow guidance, run `agent-device help ios-system-ui`.
1045
+
1046
+
- `agent-device open com.apple.springboard --platform ios` binds the session to SpringBoard on a simulator or physical device today; there is no separate widget/system command.
1047
+
- The full widget add/edit/remove flow is selector-driven from a fresh `snapshot -i`, except two coordinate-based steps: the empty-space long-press that enters edit mode, and (until fixed) the widget-gallery search-result rows, which currently return unlabeled accessibility nodes.
1048
+
- SpringBoard labels vary by iOS version and locale; discover them from the current snapshot rather than hard-coding strings like `Edit` or `Add Widget`.
1049
+
- Reopen the app bundle under test to return to normal app automation after a SpringBoard step.
0 commit comments