Skip to content

Commit 741b207

Browse files
committed
feat(daemon): opt-in inactivity expiry for idle local device claims
#2833: on a host shared by several agents, a session that never runs `close` keeps its host-global device claim until the daemon stops, and every other agent reads DEVICE_IN_USE with no way to tell active from abandoned. Set AGENT_DEVICE_SESSION_IDLE_TIMEOUT_MS to expire a claim-holding session with no remote lease and no capture running for it, once it has taken no attached commands for that long. The daemon that owns the session settles it under that session's own execution lock pair, releases the claim, and leaves a bounded marker so the next command answers SESSION_NOT_FOUND with details.reason SESSION_IDLE_EXPIRED naming the window and the released device. Off by default; a positive window below one millisecond is the shortest window, not off. A settle that cannot confirm its claim gone holds the session record back and retries a window later: this daemon stays alive, so forgetting the record would strand a claim owned by a process that no longer knows what it holds, which `device release --stale` cannot reclaim. `clearDeviceClaim` therefore gains `unattributable` for a record that yields no attributable owner, because that says nothing about a successor having taken the device. The lock pair belongs to the release, not to the sweep's wait for it. A teardown budget bounds only that wait, so one stuck recorder cannot hang a sweep while a budget releasing the locks would let a retried `close` join a teardown in progress, or a late release delete a session a retried `open` just created and mark it expired. A sweep that finds a release in flight defers that address rather than re-arming at a deadline already past, and a release that lands after its budget still reports the session it freed.
1 parent 34f2eb1 commit 741b207

22 files changed

Lines changed: 2703 additions & 45 deletions

‎src/commands/schema/cli-help-topics.test.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -517,6 +517,7 @@ test('usageForCommand resolves physical-device help topic', async () => {
517517
);
518518
assert.match(help, /AGENT_DEVICE_IOS_RUNNER_IDLE_STOP_MS/);
519519
assert.match(help, /AGENT_DEVICE_DAEMON_IDLE_TIMEOUT_MS/);
520+
assert.match(help, /AGENT_DEVICE_SESSION_IDLE_TIMEOUT_MS/);
520521
assert.match(
521522
help,
522523
/a stale iOS runner lease — its owner process dead, or its AGENT_DEVICE_STATE_DIR deleted — is reclaimed automatically/i,

‎src/commands/schema/cli-help.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -702,6 +702,7 @@ Runner and daemon lifecycle (applies to simulators too):
702702
No runner read launches a session app that is not running: snapshot, wait, is, get, a reading find, and an interaction's leading reads (a gesture's viewport read, the capture that resolves a selector click/fill) answer the retriable APP_NOT_RUNNING instead of bare-launching over a launch SpringBoard still holds behind its deep-link confirmation. Only open, activate, and a command that mutates without a leading read bring a stopped app up.
703703
close keeps a healthy iOS simulator XCTest runner warm by default so the next open on that simulator (same udid in the same simulator set) skips the runner build, unless --shutdown was requested, the session was recording, or the session held a device lease. A retained runner auto-stops after an idle window (default 5 minutes); set AGENT_DEVICE_IOS_RUNNER_IDLE_STOP_MS to override, or 0 to disable idle stop and retain until daemon exit.
704704
Each AGENT_DEVICE_STATE_DIR runs its own daemon. It self-exits after an idle window (default 5 minutes, matching the runner idle-stop default) once it has no open sessions, no in-flight requests, and no active recording; set AGENT_DEVICE_DAEMON_IDLE_TIMEOUT_MS to override, or 0 to disable idle reap.
705+
On a machine shared by several agents, set AGENT_DEVICE_SESSION_IDLE_TIMEOUT_MS to also expire an individual session that has taken a host-global device claim and then received no commands for that long; the claim is released and the next command on that session answers SESSION_NOT_FOUND with details.reason SESSION_IDLE_EXPIRED naming the window and the device. Off by default, because on a shared host "idle" and "thinking" are indistinguishable: only set it where an abandoned claim blocking every other agent is the worse failure. Sessions holding a remote lease or any active capture (recording, logs, audio, performance, or trace) are never expired this way.
705706
A stale iOS runner lease — its owner process dead, or its AGENT_DEVICE_STATE_DIR deleted — is reclaimed automatically instead of failing with "is already owned by another agent-device daemon". A live owner's runner is also reclaimed when the requesting daemon holds the host-global device claim for that device: claims are exclusive, so holding one proves the runner's owner released the device and merely kept the runner warm. The error remains only for owners outside claim arbitration (a pre-claims build, or daemons pointed at different claim stores).
706707
707708
For iOS SpringBoard, widget, or other system-UI surfaces, read agent-device help ios-system-ui.`,

‎src/daemon/__tests__/application-lifecycle-recovery.test.ts‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -188,6 +188,30 @@ test('daemon lifecycle finalization admits facts once, binds once, and disposes
188188
expect(dispose).toHaveBeenCalledOnce();
189189
});
190190

191+
// #2833: an idle-session expiry is the one daemon-owned teardown with no successor to hand a healthy
192+
// execution host to. Deferring runner termination to the gateway's shutdown phase would park the
193+
// runner — and the device behind it — until process exit, which is the opposite of why the expiry ran.
194+
test('a daemon that stays alive finalizes without the shutdown deferral', async () => {
195+
const finalize = vi.fn(async () => {});
196+
const runtime = gateway({ finalize });
197+
198+
await finalizeDaemonSessionApplicationLifecycle({
199+
gateway: runtime.gateway,
200+
scope: scope(),
201+
session,
202+
stateDir: '/state',
203+
runtimeHints: {},
204+
daemonLeaving: false,
205+
});
206+
207+
expect(finalize).toHaveBeenCalledWith({
208+
appBundleId: undefined,
209+
surface: 'app',
210+
retainRunner: false,
211+
stateDir: '/state',
212+
});
213+
});
214+
191215
test.each([
192216
{
193217
name: 'Android provider device',
Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
import { createTestDeviceInventoryGateways } from '../../__tests__/test-utils/device-inventory-gateways.ts';
2+
/**
3+
* #2833: when a command finds no session because this daemon already expired it for idleness, the
4+
* answer stays `SESSION_NOT_FOUND` but carries the typed reason, the window it missed, and the device
5+
* it released — never a bare "Run open first" with nothing to reason about. The repair tombstone is
6+
* consulted first, so an abandoned repair transaction keeps its own, more specific guidance.
7+
*/
8+
import { test, expect } from 'vitest';
9+
import path from 'node:path';
10+
import fs from 'node:fs';
11+
12+
import { createRequestHandler } from './test-device-runtime-gateway.ts';
13+
import { buildIdleExpiryTombstone } from '../session-idle-expiry.ts';
14+
import type { DaemonRequest } from '../daemon-request.ts';
15+
import { LeaseRegistry } from '../lease-registry.ts';
16+
import { makeSessionStore } from '../../__tests__/test-utils/store-factory.ts';
17+
import { mkdtempForTestSync } from '../../__tests__/test-utils/tmp-dir.ts';
18+
19+
function makeHandler(prefix: string) {
20+
const sessionStore = makeSessionStore(prefix);
21+
const handler = createRequestHandler({
22+
logPath: path.join(mkdtempForTestSync('daemon'), 'daemon.log'),
23+
token: 'test-token',
24+
sessionStore,
25+
leaseRegistry: new LeaseRegistry(),
26+
deviceInventoryGateways: createTestDeviceInventoryGateways(),
27+
trackDownloadableArtifact: () => 'artifact-id',
28+
});
29+
return { sessionStore, handler };
30+
}
31+
32+
function closeRequest(session: string, tenant?: string): DaemonRequest {
33+
return {
34+
token: 'test-token',
35+
session,
36+
command: 'close',
37+
positionals: [],
38+
flags: {},
39+
...(tenant ? { meta: { tenantId: tenant, sessionIsolation: 'tenant' as const } } : {}),
40+
};
41+
}
42+
43+
function writeIdleMarker(sessionStore: ReturnType<typeof makeSessionStore>, session: string) {
44+
sessionStore.writeIdleExpiryTombstone(
45+
session,
46+
buildIdleExpiryTombstone(session, {
47+
expiredAtMs: Date.now(),
48+
idleExpiryMs: 1_500_000,
49+
deviceKey: 'ios:sim-1',
50+
}),
51+
);
52+
}
53+
54+
test('a command on an idle-expired session names the reason, window, and released device', async () => {
55+
const { sessionStore, handler } = makeHandler('agent-device-router-idle-expired-');
56+
writeIdleMarker(sessionStore, 'idle-x');
57+
58+
const response = await handler(closeRequest('idle-x'));
59+
60+
expect(response.ok).toBe(false);
61+
if (response.ok) return;
62+
// Still SESSION_NOT_FOUND: the session genuinely is gone. What changes is what it says why.
63+
expect(response.error.code).toBe('SESSION_NOT_FOUND');
64+
expect(response.error.details?.reason).toBe('SESSION_IDLE_EXPIRED');
65+
expect(response.error.details?.idleExpiryMs).toBe(1_500_000);
66+
expect(response.error.details?.deviceKey).toBe('ios:sim-1');
67+
expect(response.error.hint).toMatch(/ios:sim-1/);
68+
});
69+
70+
test('an expired session with no marker keeps the plain SESSION_NOT_FOUND', async () => {
71+
const { handler } = makeHandler('agent-device-router-idle-no-marker-');
72+
const response = await handler(closeRequest('never-existed'));
73+
74+
expect(response.ok).toBe(false);
75+
if (response.ok) return;
76+
expect(response.error.details?.reason).toBeUndefined();
77+
});
78+
79+
test('an expired marker that has aged out stops explaining the absence', async () => {
80+
const { sessionStore, handler } = makeHandler('agent-device-router-idle-stale-');
81+
writeIdleMarker(sessionStore, 'idle-stale');
82+
const markerPath = path.join(sessionStore.resolveSessionDir('idle-stale'), 'idle-expiry.json');
83+
const marker = JSON.parse(fs.readFileSync(markerPath, 'utf8')) as { expiresAt: number };
84+
marker.expiresAt = Date.now() - 1;
85+
fs.writeFileSync(markerPath, `${JSON.stringify(marker)}\n`);
86+
87+
const response = await handler(closeRequest('idle-stale'));
88+
89+
expect(response.ok).toBe(false);
90+
if (response.ok) return;
91+
expect(response.error.details?.reason).toBeUndefined();
92+
});
93+
94+
test('an abandoned repair transaction outranks the idle-expiry marker', async () => {
95+
const { sessionStore, handler } = makeHandler('agent-device-router-idle-vs-repair-');
96+
writeIdleMarker(sessionStore, 'repair-x');
97+
sessionStore.writeRepairTombstone({
98+
name: 'repair-x',
99+
device: { platform: 'apple', id: 'sim-1', name: 'iPhone', kind: 'simulator', booted: true },
100+
createdAt: Date.now(),
101+
actions: [],
102+
scriptPublication: {
103+
kind: 'repair',
104+
status: 'armed',
105+
target: { kind: 'default', force: false },
106+
boundary: 0,
107+
sourcePath: '/flows/login.ad',
108+
},
109+
});
110+
111+
const response = await handler(closeRequest('repair-x'));
112+
113+
expect(response.ok).toBe(false);
114+
if (response.ok) return;
115+
expect(response.error.code).toBe('REPAIR_SESSION_EXPIRED');
116+
});
117+
118+
test('a tenant-isolated request reads the marker for its own scoped session', async () => {
119+
const { sessionStore, handler } = makeHandler('agent-device-router-idle-tenant-');
120+
// The request names `idle-x` and the daemon owns it as `<tenant>:idle-x`. Reading the raw name
121+
// would miss the marker entirely and return a bare "Run open first".
122+
writeIdleMarker(sessionStore, 'tenant-a:idle-x');
123+
124+
const response = await handler(closeRequest('idle-x', 'tenant-a'));
125+
126+
expect(response.ok).toBe(false);
127+
if (response.ok) return;
128+
expect(response.error.details?.reason).toBe('SESSION_IDLE_EXPIRED');
129+
expect(response.error.details?.deviceKey).toBe('ios:sim-1');
130+
});
131+
132+
test('another tenant marker never explains this tenant absent session', async () => {
133+
const { sessionStore, handler } = makeHandler('agent-device-router-idle-tenant-leak-');
134+
writeIdleMarker(sessionStore, 'tenant-b:idle-x');
135+
136+
const response = await handler(closeRequest('idle-x', 'tenant-a'));
137+
138+
expect(response.ok).toBe(false);
139+
if (response.ok) return;
140+
// Reading the unscoped key would have reported tenant-b's device as the one tenant-a just lost.
141+
expect(response.error.details?.reason).toBeUndefined();
142+
});
Lines changed: 183 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,183 @@
1+
/**
2+
* #2833: the request path reports session activity by finishing UNDER the session's execution lock —
3+
* the same lock an expiry must take before settling. That is what makes the stamp and the verdict
4+
* one clock: a session in use is never caught mid-command, and the deadline an expiry reads is the
5+
* one the last finished command set. Commands that never take that lock (the registry's
6+
* lock-exempt inventory surface) report nothing, so a bystander polling `devices` on a shared host
7+
* cannot keep another agent's abandoned claim alive; and a request whose client hung up preserves
8+
* nothing, exactly as a canceled request renews no lease (ADR 0007).
9+
*/
10+
import { afterAll, test, expect, vi } from 'vitest';
11+
import fs from 'node:fs';
12+
import path from 'node:path';
13+
import { withDiagnosticsScope } from '@agent-device/host-kit/diagnostics';
14+
import { clearRequestCanceled, markRequestCanceled } from '@agent-device/host-kit/request';
15+
import { makeIosSession } from '../../__tests__/test-utils/session-factories.ts';
16+
import { makeSessionStore } from '../../__tests__/test-utils/store-factory.ts';
17+
import { LeaseRegistry } from '../lease-registry.ts';
18+
import {
19+
existingSessionExecutionLockKeys,
20+
resolveRequestExecutionLockPlan,
21+
} from '../request-binding.ts';
22+
import {
23+
createRequestExecutionScope,
24+
getLeaseRegistryExecutionLocks,
25+
} from '../request-execution-scope.ts';
26+
import { withKeyedLock } from '@agent-device/kernel/keyed-lock';
27+
import type { DaemonRequest } from '../daemon-request.ts';
28+
import { mkdtempForTestSync } from '../../__tests__/test-utils/tmp-dir.ts';
29+
30+
const TEST_ROOT = mkdtempForTestSync('agent-device-session-idle-activity-');
31+
const LOG_PATH = path.join(TEST_ROOT, 'diagnostics.log');
32+
33+
afterAll(() => {
34+
fs.rmSync(TEST_ROOT, { recursive: true, force: true });
35+
});
36+
37+
function makeRequest(overrides: Partial<DaemonRequest> = {}): DaemonRequest {
38+
return {
39+
token: 'test-token',
40+
session: 'default',
41+
command: 'snapshot',
42+
positionals: [],
43+
...overrides,
44+
};
45+
}
46+
47+
function storeWithSession() {
48+
const sessionStore = makeSessionStore('agent-device-idle-activity-');
49+
sessionStore.set('default', makeIosSession('default'));
50+
return sessionStore;
51+
}
52+
53+
test('a command that completes under the session lock stamps the session it ran on', async () => {
54+
const sessionStore = storeWithSession();
55+
const scope = await withDiagnosticsScope({ command: 'snapshot', logPath: LOG_PATH }, () =>
56+
createRequestExecutionScope({
57+
req: makeRequest({ command: 'snapshot' }),
58+
sessionStore,
59+
leaseRegistry: new LeaseRegistry(),
60+
}),
61+
);
62+
63+
await scope.runLocked(async () => 'ran');
64+
65+
expect(sessionStore.get('default')?.lastActivityAtMs).toBeGreaterThan(0);
66+
});
67+
68+
test('a lock-exempt inventory command reports no activity, even against a claim-holding session', async () => {
69+
const sessionStore = storeWithSession();
70+
const scope = await withDiagnosticsScope({ command: 'devices', logPath: LOG_PATH }, () =>
71+
createRequestExecutionScope({
72+
req: makeRequest({ command: 'devices' }),
73+
sessionStore,
74+
leaseRegistry: new LeaseRegistry(),
75+
}),
76+
);
77+
78+
await scope.runLocked(async () => 'ran');
79+
80+
// `devices` resolves a session address only to locate its own artifacts. On the shared host this
81+
// feature exists for, stamping there would let one agent's polling extend another's deadline.
82+
expect(sessionStore.get('default')?.lastActivityAtMs).toBeUndefined();
83+
});
84+
85+
test('a command whose client hung up mid-flight preserves no activity', async () => {
86+
const sessionStore = storeWithSession();
87+
const requestId = 'canceled-mid-command';
88+
const scope = await withDiagnosticsScope(
89+
{ command: 'snapshot', requestId, logPath: LOG_PATH },
90+
() =>
91+
createRequestExecutionScope({
92+
req: makeRequest({ command: 'snapshot', meta: { requestId } }),
93+
sessionStore,
94+
leaseRegistry: new LeaseRegistry(),
95+
}),
96+
);
97+
98+
// The request was admitted, then the client hung up while the handler worked. A handler that
99+
// ignores its cancellation still reaches the stamp site — and must not renew anything there, the
100+
// same rule that stops a canceled request renewing a remote lease (ADR 0007). An agent that
101+
// timed out and left a hung handler is precisely what this deadline exists to catch.
102+
try {
103+
await scope.runLocked(async () => {
104+
markRequestCanceled(requestId);
105+
return 'ran to completion despite the disconnect';
106+
});
107+
expect(sessionStore.get('default')?.lastActivityAtMs).toBeUndefined();
108+
} finally {
109+
clearRequestCanceled(requestId);
110+
}
111+
});
112+
113+
test('a command still running when the expiry arrives blocks it until the stamp lands', async () => {
114+
const sessionStore = storeWithSession();
115+
const leaseRegistry = new LeaseRegistry();
116+
const locks = getLeaseRegistryExecutionLocks(leaseRegistry);
117+
const scope = await withDiagnosticsScope({ command: 'snapshot', logPath: LOG_PATH }, () =>
118+
createRequestExecutionScope({
119+
req: makeRequest({ command: 'snapshot' }),
120+
sessionStore,
121+
leaseRegistry,
122+
}),
123+
);
124+
125+
let release!: () => void;
126+
const running = new Promise<void>((resolve) => {
127+
release = resolve;
128+
});
129+
const command = scope.runLocked(async () => {
130+
await running;
131+
return 'ran';
132+
});
133+
134+
// An expiry that starts while the command holds the lock cannot settle — and once the command
135+
// finishes, its stamp is already visible to whoever finally holds the lock.
136+
let settledFirst = true;
137+
const expiry = withKeyedLock(locks, 'session:default', async () => {
138+
settledFirst = sessionStore.get('default')?.lastActivityAtMs === undefined;
139+
});
140+
141+
await new Promise((resolve) => setTimeout(resolve, 5));
142+
release();
143+
await Promise.all([command, expiry]);
144+
145+
expect(settledFirst).toBe(false);
146+
});
147+
148+
test('the request plan for an existing session keeps the canonical session-first lock pair', async () => {
149+
const sessionStore = storeWithSession();
150+
const session = sessionStore.get('default')!;
151+
const requestPlan = await resolveRequestExecutionLockPlan({
152+
req: makeRequest({ command: 'snapshot' }),
153+
sessionName: 'default',
154+
sessionStore,
155+
});
156+
157+
// What this pins is the PLANNER: an existing session's request takes `existingSessionExecutionLockKeys`
158+
// with the session first, which is the order a reaper must match to avoid deadlocking against it. The
159+
// reaper's own pair is exercised against a live request in server/daemon-session-idle-expiry.test.ts;
160+
// comparing this function with itself would prove nothing about that side.
161+
expect(requestPlan.keys).toEqual(existingSessionExecutionLockKeys('default', session.device.id));
162+
});
163+
164+
test('the reaper never observes a mid-command session through the shared map', async () => {
165+
const sessionStore = storeWithSession();
166+
const leaseRegistry = new LeaseRegistry();
167+
const scope = await withDiagnosticsScope({ command: 'snapshot', logPath: LOG_PATH }, () =>
168+
createRequestExecutionScope({
169+
req: makeRequest({ command: 'snapshot' }),
170+
sessionStore,
171+
leaseRegistry,
172+
}),
173+
);
174+
175+
const inside = vi.fn();
176+
await scope.runLocked(async () => {
177+
inside(getLeaseRegistryExecutionLocks(leaseRegistry).has('session:default'));
178+
return 'ran';
179+
});
180+
181+
expect(inside).toHaveBeenCalledWith(true);
182+
expect(getLeaseRegistryExecutionLocks(leaseRegistry).has('session:default')).toBe(false);
183+
});

0 commit comments

Comments
 (0)