Repository navigation
Let agents drive hosted macOS apps through a host-owned driver #2477
Description
Track E of #2403. Contracts: contracts comment, sections 1 (agent route), 5 and 6. Shared types and the hosting.agentDriver setting: #2472.
Problem
A coding agent on the client Mac cannot drive a hosted macOS app (snapshot, click, type, screenshot) the way it drives local simulators with agent-device. The maintainer decided:
- the driver is a small adapter behind
hosting.agentDriver(defaultnone, opt-inagent-device); - stim-server on the host owns one shared driver daemon, reference-counted on running hosted apps;
- each client gets a credential scoped to its own apps;
- traffic goes only over the tailnet through the existing approved connection.
Evidence
Local experiments with isolated state directories and a fixture app:
- Local
agent-device(0.21.12) open, snapshot -i, click and screenshot work on astim macosapp. The screenshot is the window only. - Through
agent-device proxyplusconnect proxy,devices --platform macoslists the host, butopenrefuses: remote leases exist only forios-simulator,ios-instance,android-instanceandharmonyos-instance. This holds in 0.21.12 and 0.21.20 and on upstream main, whereLEASE_BACKEND_BY_PLATFORMmaps the macOS desktop host to no backend. fix(remote): speak one platform axis between a device and its bound connection (#2962) callstack/agent-device#2989 (the platform-axis fix, 0.21.16) does not change this. - The remote-config schema accepts
daemonBaseUrl,daemonAuthToken,leaseId,leaseBackendandplatform. - The mini's stim-server LaunchAgent
PATHis only system directories. agent-device lives at~/.local/bin/agent-device(0.21.20 on the mini).
Fix idea / scope
- Upstream first: propose a macOS app lease backend to callstack/agent-device. Its device key is one bundle id, optionally a pid. Admission would refuse other bundles, the
desktop/frontmost-app/menubarsurfaces, install and launch of other apps, and non-window screenshots. A host administrator allocates the lease. Search upstream first, and link the result here. - Host adapter interface (contract section 6) plus the agent-device adapter:
- the daemon is held under an ownership claim (child = daemon), reference-counted, and stopped on the last app, on revocation and on close, and restarted on a crash;
- the binary is resolved explicitly;
/device-host/agent/<session>/*is forwarded with the session token, only from the client's node.
- Client adapter: write the 0600
--remote-configand fillHostedAgentAccess(handled with Run macOS apps on a hosting Mac with stim macos --host #2475). doctorandguidenamehosting.agentDriverwhen a hosted macOS app runs with no driver.
Until the upstream lease exists, issue refuses, and deliveries carry { driver: 'none' } with a notice. No unscoped desktop access is ever handed out.
Acceptance
- Unit tests for the reference-counted lifecycle, the claim, token scoping and the forward refusal for another session or node.
- A real two-Mac run: the agent on the MacBook opens, snapshots, clicks and screenshots the hosted fixture on the mini, and cannot reach another session's app.
Out of scope
Other drivers (one adapter each, later).
The upstream proposal and the adapter skeleton can start now. End-to-end use depends on #2473 and #2475.
Activity
janicduplessis commentedon Oct 5, 2026
Claimed: feat/2477-hosted-agent-driver
janicduplessis commentedon Oct 5, 2026
Merged #2479: the host driver boundary (
HostedAgentHost, refcount, restart, grant scoping), the agent-device adapter (explicit binary path, claim with the daemon as child, verified stop) and the/device-host/agent/<session>/route. The adapter stays gated: agent-device has no remote lease limited to one macOS app, sostart()andissue()refuse and hosted apps get{ driver: 'none' }with a notice. Lease binding inforward()must be added when that lease exists. Still to do after #2473 merges: callappRunning/appStoppedfromDeviceHost(replacing the fixednonein #2473), and add the doctor line for a hosted macOS app running with no driver. The upstream proposal is drafted and has not been filed.janicduplessis commentedon Oct 5, 2026
Claimed: feat/2477-agent-driver-wiring (wiring of appRunning/appStopped and the no-driver doctor/guide line; the upstream-lease part stays open)
janicduplessis commentedon Oct 5, 2026
Upstream proposal filed: callstack/agent-device#3229 (a
macos-applease backend scoped to one app). The driver stays gated off ({driver:'none'}) until it lands.janicduplessis commentedon Oct 5, 2026
Merged #2484 (55b77d7):
DeviceHostnow callsappRunning/appStoppedon hosted macOS install, stop, revocation and close, andapp.attach/app.launchreturn the manager's grant ornonewith its notice. The install worker returns the app pid.stim doctoron the hosting Mac notes a ready hosted macOS app whilehosting.agentDriverisnone; the guide, website and server README say so. Still open here: the upstream macOS app lease (proposal drafted, not filed), lease binding inforward()and theagent-deviceadapter'sissue(), the client's 0600 remote-config (#2475), and the real two-Mac run. Known gap: grants are in memory, so an app still running after a stim-server restart reports `none"; nothing re-attaches it today.janicduplessis commentedon Oct 5, 2026
Claimed: feat/2477-macos-app-lease (implement the macos-app lease in agent-device per callstack/agent-device#3229, use it from Stim's agent-device adapter behind feature detection, and write the client remote-config lease fields)
feat: let agents drive hosted macOS apps through agent-device macos-a…
janicduplessis commentedon Oct 5, 2026
Upstream draft PR for the macos-app lease: callstack/agent-device#3236 (implements #3229). Stim side: #2516 (driver enables only when the daemon's /health lists macos-app; setting default stays none). The two-Mac run is pending: redeploying the mini's stim-server with STIM_AGENT_DEVICE_BIN was blocked by the session's permission classifier.
feat: let agents drive hosted macOS apps through agent-device macos-a…
feat: let agents drive hosted macOS apps through agent-device macos-a…
janicduplessis commentedon Oct 5, 2026
End-to-end result for the host-owned agent driver on the real tailnet path (MacBook client, Mac mini host, current main 787f051, signed Stim Host 0.1.0, agent-device from callstack/agent-device#3236 draft at 5c958e109,
AGENT_DEVICE_MACOS_APP_BACKEND=native).Setup on the host:
hosting.agentDriver = agent-deviceon the mini; the LaunchAgent carriesSTIM_AGENT_DEVICE_BINandAGENT_DEVICE_MACOS_HELPER_BINand runs under Stim Host (Screen & System Audio Recording and Device Control and Data Access allowed). The setup stays in place for dogfooding.Results
stim macos --host janics-mac-miniwith a fixture SwiftPM app: built here, delivered, launched as<id>.hosted1;status --jsonreportedagent.driver = "agent-device"with the remote config.open,snapshot -i,click,fill,type,find,get text,screenshot(1800x900, only the app window),closeall worked over https 7443.openof another app (UNAUTHORIZED, "only opens "),open --surface desktop,screenshot --fullscreen,apps/devices("Unsupported request"),--platform ioson the leased session (CONNECTION_PLATFORM_CONFLICT),clickoutside any accessibility element.apps/desktopfrom a fresh worktree (hosted Stim Desktop, 200 files delivered, ~63 s build): snapshot of 49 nodes, clickingGet Started, dismissing the onboarding sheet, opening Machines, window-only screenshot./healthlistsmacos-appinleaseBackends; it exits afterstim stop.stim stopremoved the hosted apps and the remote config.Observations
stim stop, the client's local agent-device session stays bound to the old remote config, so the next workspace's--remote-configfails with "A different remote connection is already active" untilagent-device disconnectruns.stim guide macoscould say to runagent-device closethenagent-device disconnectbeforestim stop.screenshotafter aclickreturned a 132x40 image (a transient window-sharing indicator,WindowSharingSessionButton, appeared in the snapshot). It did not recur in the next 7 captures.click @reffailed several times with helper reasonno-accessible-target(point-based hit test), then worked after the sheet changed; typing Escape dismissed the sheet.~/.stim/server/agent-device/sessions/stim.<session>_defaultdirectories remain on the host after each hosted session.janicduplessis commentedon Oct 7, 2026
Backlog triage: Stim's driver lifecycle/scoping, wiring and macos-app lease adapter landed in #2479, #2484 and 38da60b (#2516). The recorded two-Mac run satisfies the driver acceptance, including other-app and broad-surface refusals plus daemon teardown: #2477 (comment) . The connection/session cleanup observations were fixed by acbf044 (#2527).
On 43395b6,
pnpm test packages/stim-cli/src/__tests__/hosted-agent-driver.test.ts packages/stim-cli/src/__tests__/hosted-macos-client.test.tspassed 50 tests. Closing completed Stim implementation/acceptance. The runtime evidence used the upstream callstack/agent-device#3236 draft build; an installed agent-device must advertisemacos-appor Stim still refuses the grant as designed. This closure does not claim that the upstream capability has shipped.