Skip to content

Commit b90f3a9

Browse files
committed
fix: reap idle daemons and take over stale runner leases
Each AGENT_DEVICE_STATE_DIR spawns its own daemon that never exits on its own; deleted codex/claude sandboxes leave orphaned daemons accumulating (10+ observed). A stale-but-still-running orphan also keeps holding its iOS runner lease, so a fresh daemon for the same device fails with COMMAND_FAILED "already owned by another agent-device daemon". - Daemon self-reaps after an idle window (default 5 minutes, matching the iOS runner idle-stop default) once it has no open sessions, no in-flight requests, and no active recording. AGENT_DEVICE_DAEMON_IDLE_TIMEOUT_MS overrides the window; 0 disables it. - A runner lease whose owner PID is dead, or whose owner AGENT_DEVICE_STATE_DIR no longer exists, is now reclaimed automatically instead of erroring; a genuinely live owner with an existing state dir still gets the existing rejection + hint.
1 parent ff7fc5b commit b90f3a9

8 files changed

Lines changed: 613 additions & 59 deletions

File tree

‎src/__tests__/cli-help.test.ts‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -150,6 +150,22 @@ test('help workflow documents open/close/relaunch runner guarantees (#1051)', as
150150
assert.match(result.stdout, /default 5 minutes/);
151151
});
152152

153+
test('help workflow documents daemon idle reap and stale lease takeover guarantees', async () => {
154+
const result = await runCliCapture(['help', 'workflow']);
155+
assert.equal(result.code, 0);
156+
assert.equal(result.calls.length, 0);
157+
assert.match(
158+
result.stdout,
159+
/self-exits after an idle window \(default 5 minutes, matching the runner idle-stop default\)/,
160+
);
161+
assert.match(result.stdout, /AGENT_DEVICE_DAEMON_IDLE_TIMEOUT_MS/);
162+
assert.match(
163+
result.stdout,
164+
/a stale iOS runner lease — its owner process dead, or its AGENT_DEVICE_STATE_DIR deleted — is reclaimed automatically/i,
165+
);
166+
assert.match(result.stdout, /genuinely live owner whose state dir still exists still rejects/);
167+
});
168+
153169
test('help workflow documents ref lifetime, snapshot diff, and wait guarantees (#1051)', async () => {
154170
const result = await runCliCapture(['help', 'workflow']);
155171
assert.equal(result.code, 0);

‎src/cli/parser/cli-help.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -390,6 +390,7 @@ Guarantees:
390390
Hittability: iOS AX hittable:false on a resolved node does not block resolution or fail the command; non-hittable resolution is allowed by design because iOS AX hittable flags are unreliable on deep React Native trees. press/fill/click success responses carry targetHittable: false plus a hint when the resolved ref or selector target reports hittable: false, so treat that as a signal to verify with a snapshot or re-target, not as a failure.
391391
Open: on iOS, open <app> without --relaunch dispatches a plain simctl launch, which is idempotent-foreground for an already-running app (it brings the process forward; it does not restart it). open --relaunch restarts the app; on iOS simulators (not real devices or macOS) this collapses to one simctl launch --terminate-running-process call instead of a separate terminate-then-launch, so relaunch is a single step there.
392392
Close and runner retention: close keeps a healthy iOS simulator XCTest runner warm by default so the next open on that device skips the runner build, unless --shutdown was requested, the session was recording, the session held a device lease, or the device used a scoped (non-default) simulator set — any of those tear the runner down on close. A retained runner auto-stops after an idle window (default 5 minutes) to release the device's runner lease for other daemons; set AGENT_DEVICE_IOS_RUNNER_IDLE_STOP_MS to override the window, or 0 to disable idle stop and retain until daemon exit.
393+
Daemon lifetime and stale lease takeover: each AGENT_DEVICE_STATE_DIR runs its own daemon. It self-exits after an idle window (default 5 minutes, matching the runner idle-stop default) once it has no open sessions, no in-flight requests, and no active recording; an open session always blocks this even if quiet for minutes between commands. Set AGENT_DEVICE_DAEMON_IDLE_TIMEOUT_MS to override the window, or 0 to disable idle reap and run until killed. A stale iOS runner lease — its owner process dead, or its AGENT_DEVICE_STATE_DIR deleted — is reclaimed automatically instead of failing with "is already owned by another agent-device daemon"; a genuinely live owner whose state dir still exists still rejects with that error.
393394
Ref lifetime: refs from a snapshot/snapshot -i are only valid until the next state-changing command. open and open --relaunch clear the session's stored snapshot outright, so every ref from before an open/relaunch is invalid; press/fill/click/scroll/back and similar mutations invalidate refs from prior turns even though the session snapshot itself is refreshed internally by those commands.
394395
Snapshot diff: diff snapshot compares the current capture against the session's last stored snapshot (from any prior snapshot, snapshot -i, or diff snapshot call), not a fixed baseline from session start. If no prior snapshot exists yet, diff snapshot initializes the baseline and reports zero additions/removals instead of failing.
395396
Wait: wait text|selector|@ref polls on a fixed interval (300ms) up to a timeout (10s default, override with the trailing timeoutMs positional) by re-capturing state each poll; it does not push/subscribe. Timing out raises a command failure rather than returning a not-found result.
Lines changed: 220 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,220 @@
1+
import assert from 'node:assert/strict';
2+
import fs from 'node:fs';
3+
import os from 'node:os';
4+
import path from 'node:path';
5+
import { afterEach, beforeEach, test } from 'vitest';
6+
import { IOS_SIMULATOR } from '../../__tests__/test-utils/index.ts';
7+
import { SessionStore } from '../session-store.ts';
8+
import type { SessionState } from '../types.ts';
9+
import {
10+
createDaemonIdleReap,
11+
hasActiveRecording,
12+
hasOpenSessions,
13+
isDaemonIdle,
14+
resolveDaemonIdleReapMs,
15+
} from './daemon-idle-reap.ts';
16+
17+
let stateDir: string;
18+
let sessionStore: SessionStore;
19+
20+
beforeEach(() => {
21+
stateDir = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-daemon-idle-reap-'));
22+
sessionStore = new SessionStore(path.join(stateDir, 'sessions'));
23+
});
24+
25+
afterEach(() => {
26+
fs.rmSync(stateDir, { recursive: true, force: true });
27+
});
28+
29+
function makeSession(overrides: Partial<SessionState> = {}): SessionState {
30+
return {
31+
name: 'default',
32+
device: IOS_SIMULATOR,
33+
createdAt: Date.now(),
34+
actions: [],
35+
...overrides,
36+
};
37+
}
38+
39+
test('resolveDaemonIdleReapMs falls back to the 5 minute default', () => {
40+
assert.equal(resolveDaemonIdleReapMs({}), 5 * 60_000);
41+
});
42+
43+
test('resolveDaemonIdleReapMs honors AGENT_DEVICE_DAEMON_IDLE_TIMEOUT_MS', () => {
44+
assert.equal(
45+
resolveDaemonIdleReapMs({ AGENT_DEVICE_DAEMON_IDLE_TIMEOUT_MS: '1200' }),
46+
1_200,
47+
);
48+
});
49+
50+
test('resolveDaemonIdleReapMs treats 0 as an explicit value (disabled)', () => {
51+
assert.equal(resolveDaemonIdleReapMs({ AGENT_DEVICE_DAEMON_IDLE_TIMEOUT_MS: '0' }), 0);
52+
});
53+
54+
test('resolveDaemonIdleReapMs ignores invalid overrides', () => {
55+
assert.equal(
56+
resolveDaemonIdleReapMs({ AGENT_DEVICE_DAEMON_IDLE_TIMEOUT_MS: 'not-a-number' }),
57+
5 * 60_000,
58+
);
59+
assert.equal(resolveDaemonIdleReapMs({ AGENT_DEVICE_DAEMON_IDLE_TIMEOUT_MS: '-5' }), 5 * 60_000);
60+
});
61+
62+
test('hasOpenSessions reflects the session store', () => {
63+
assert.equal(hasOpenSessions(sessionStore), false);
64+
sessionStore.set('default', makeSession());
65+
assert.equal(hasOpenSessions(sessionStore), true);
66+
});
67+
68+
test('hasActiveRecording is true only when a stored session carries a recording', () => {
69+
sessionStore.set('default', makeSession());
70+
assert.equal(hasActiveRecording(sessionStore), false);
71+
72+
sessionStore.set(
73+
'default',
74+
makeSession({
75+
recording: {
76+
platform: 'ios',
77+
outPath: '/tmp/demo.mp4',
78+
startedAt: Date.now(),
79+
showTouches: false,
80+
gestureEvents: [],
81+
child: { kill: () => true },
82+
wait: Promise.resolve({ stdout: '', stderr: '', exitCode: 0 }),
83+
},
84+
}),
85+
);
86+
assert.equal(hasActiveRecording(sessionStore), true);
87+
});
88+
89+
test('isDaemonIdle requires no in-flight requests, no sessions, and no recording', () => {
90+
assert.equal(isDaemonIdle({ sessionStore, inFlightRequestCount: 0 }), true);
91+
assert.equal(isDaemonIdle({ sessionStore, inFlightRequestCount: 1 }), false);
92+
93+
sessionStore.set('default', makeSession());
94+
assert.equal(isDaemonIdle({ sessionStore, inFlightRequestCount: 0 }), false);
95+
});
96+
97+
test('idle reap fires after the idle window when nothing is using the daemon', async () => {
98+
let reaped = 0;
99+
const idleReap = createDaemonIdleReap({
100+
sessionStore,
101+
getInFlightRequestCount: () => 0,
102+
onIdleReap: () => {
103+
reaped++;
104+
},
105+
env: { AGENT_DEVICE_DAEMON_IDLE_TIMEOUT_MS: '40' },
106+
});
107+
108+
idleReap.noteActivity();
109+
await new Promise((resolve) => setTimeout(resolve, 150));
110+
111+
assert.equal(reaped, 1);
112+
});
113+
114+
test('idle reap does not fire while a session is open', async () => {
115+
let reaped = 0;
116+
sessionStore.set('default', makeSession());
117+
const idleReap = createDaemonIdleReap({
118+
sessionStore,
119+
getInFlightRequestCount: () => 0,
120+
onIdleReap: () => {
121+
reaped++;
122+
},
123+
env: { AGENT_DEVICE_DAEMON_IDLE_TIMEOUT_MS: '40' },
124+
});
125+
126+
// Simulates a quiet mid-session pause: activity settles, but the session
127+
// itself stays open the whole time.
128+
idleReap.noteActivity();
129+
await new Promise((resolve) => setTimeout(resolve, 150));
130+
131+
assert.equal(reaped, 0);
132+
});
133+
134+
test('idle reap does not fire while a recording is active', async () => {
135+
let reaped = 0;
136+
sessionStore.set(
137+
'default',
138+
makeSession({
139+
recording: {
140+
platform: 'ios',
141+
outPath: '/tmp/demo.mp4',
142+
startedAt: Date.now(),
143+
showTouches: false,
144+
gestureEvents: [],
145+
child: { kill: () => true },
146+
wait: Promise.resolve({ stdout: '', stderr: '', exitCode: 0 }),
147+
},
148+
}),
149+
);
150+
const idleReap = createDaemonIdleReap({
151+
sessionStore,
152+
getInFlightRequestCount: () => 0,
153+
onIdleReap: () => {
154+
reaped++;
155+
},
156+
env: { AGENT_DEVICE_DAEMON_IDLE_TIMEOUT_MS: '40' },
157+
});
158+
159+
idleReap.noteActivity();
160+
await new Promise((resolve) => setTimeout(resolve, 150));
161+
162+
assert.equal(reaped, 0);
163+
});
164+
165+
test('idle reap does not fire while a request is in flight', async () => {
166+
let reaped = 0;
167+
let inFlightRequestCount = 1;
168+
const idleReap = createDaemonIdleReap({
169+
sessionStore,
170+
getInFlightRequestCount: () => inFlightRequestCount,
171+
onIdleReap: () => {
172+
reaped++;
173+
},
174+
env: { AGENT_DEVICE_DAEMON_IDLE_TIMEOUT_MS: '40' },
175+
});
176+
177+
idleReap.noteActivity();
178+
await new Promise((resolve) => setTimeout(resolve, 150));
179+
assert.equal(reaped, 0);
180+
181+
inFlightRequestCount = 0;
182+
idleReap.noteActivity();
183+
await new Promise((resolve) => setTimeout(resolve, 150));
184+
assert.equal(reaped, 1);
185+
});
186+
187+
test('idle reap is disabled when the window is zero', async () => {
188+
let reaped = 0;
189+
const idleReap = createDaemonIdleReap({
190+
sessionStore,
191+
getInFlightRequestCount: () => 0,
192+
onIdleReap: () => {
193+
reaped++;
194+
},
195+
env: { AGENT_DEVICE_DAEMON_IDLE_TIMEOUT_MS: '0' },
196+
});
197+
198+
idleReap.noteActivity();
199+
await new Promise((resolve) => setTimeout(resolve, 100));
200+
201+
assert.equal(reaped, 0);
202+
});
203+
204+
test('cancel prevents a scheduled reap from firing', async () => {
205+
let reaped = 0;
206+
const idleReap = createDaemonIdleReap({
207+
sessionStore,
208+
getInFlightRequestCount: () => 0,
209+
onIdleReap: () => {
210+
reaped++;
211+
},
212+
env: { AGENT_DEVICE_DAEMON_IDLE_TIMEOUT_MS: '40' },
213+
});
214+
215+
idleReap.noteActivity();
216+
idleReap.cancel();
217+
await new Promise((resolve) => setTimeout(resolve, 150));
218+
219+
assert.equal(reaped, 0);
220+
});
Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
import { emitDiagnostic } from '../../utils/diagnostics.ts';
2+
import type { SessionStore } from '../session-store.ts';
3+
4+
// Bounds the daemon's own lifetime when nothing is using it. Each
5+
// AGENT_DEVICE_STATE_DIR spawns a dedicated daemon that otherwise never exits
6+
// on its own, so short-lived sandboxes (deleted codex/claude worktrees, one-off
7+
// CLI invocations) leave orphaned daemon processes behind indefinitely -
8+
// observed accumulating (10+) in production. A stale daemon left running this
9+
// way also keeps holding its iOS runner lease, which blocks a fresh daemon for
10+
// the same device (see runner-lease.ts stale-lease takeover).
11+
//
12+
// The window mirrors the iOS runner idle-stop default
13+
// (AGENT_DEVICE_IOS_RUNNER_IDLE_STOP_MS, 5 minutes, see runner-session.ts):
14+
// graceful daemon shutdown already hands off a healthy retained simulator
15+
// runner for the next daemon to adopt (detachIosSimulatorRunnerSessionsForShutdown),
16+
// so reaping the daemon process itself on the same timescale does not force a
17+
// runner rebuild in the common case - it only pays the cheap daemon
18+
// bootstrap (socket/HTTP listen) on the next command.
19+
const DAEMON_IDLE_REAP_DEFAULT_MS = 5 * 60_000;
20+
21+
// AGENT_DEVICE_DAEMON_IDLE_TIMEOUT_MS overrides the window; 0 disables idle
22+
// reap and restores the pre-existing "runs until killed" behavior.
23+
export function resolveDaemonIdleReapMs(env: NodeJS.ProcessEnv = process.env): number {
24+
const raw = env.AGENT_DEVICE_DAEMON_IDLE_TIMEOUT_MS?.trim();
25+
if (raw) {
26+
const parsed = Number(raw);
27+
if (Number.isFinite(parsed) && parsed >= 0) return Math.floor(parsed);
28+
}
29+
return DAEMON_IDLE_REAP_DEFAULT_MS;
30+
}
31+
32+
export function hasOpenSessions(sessionStore: SessionStore): boolean {
33+
return sessionStore.toArray().length > 0;
34+
}
35+
36+
// Recording lifecycle is session-scoped (session.recording), so this is
37+
// currently implied by hasOpenSessions. Kept as an explicit, independently
38+
// testable guard so a future recording path that outlives its session cannot
39+
// silently lose this protection.
40+
export function hasActiveRecording(sessionStore: SessionStore): boolean {
41+
return sessionStore.toArray().some((session) => Boolean(session.recording));
42+
}
43+
44+
export function isDaemonIdle(params: {
45+
sessionStore: SessionStore;
46+
inFlightRequestCount: number;
47+
}): boolean {
48+
if (params.inFlightRequestCount > 0) return false;
49+
if (hasOpenSessions(params.sessionStore)) return false;
50+
return !hasActiveRecording(params.sessionStore);
51+
}
52+
53+
export type DaemonIdleReapController = {
54+
/**
55+
* Call whenever daemon activity changes (request start/end, session
56+
* open/close). Reschedules the pending reap when the daemon is currently
57+
* idle, or cancels it when it is not.
58+
*/
59+
noteActivity: () => void;
60+
cancel: () => void;
61+
readonly idleMs: number;
62+
};
63+
64+
export function createDaemonIdleReap(params: {
65+
sessionStore: SessionStore;
66+
getInFlightRequestCount: () => number;
67+
onIdleReap: () => void;
68+
env?: NodeJS.ProcessEnv;
69+
}): DaemonIdleReapController {
70+
const idleMs = resolveDaemonIdleReapMs(params.env);
71+
let timer: NodeJS.Timeout | undefined;
72+
73+
const isIdleNow = (): boolean =>
74+
isDaemonIdle({
75+
sessionStore: params.sessionStore,
76+
inFlightRequestCount: params.getInFlightRequestCount(),
77+
});
78+
79+
const cancel = (): void => {
80+
if (!timer) return;
81+
clearTimeout(timer);
82+
timer = undefined;
83+
};
84+
85+
const schedule = (): void => {
86+
cancel();
87+
if (idleMs <= 0) return;
88+
timer = setTimeout(() => {
89+
timer = undefined;
90+
// Re-check at fire time instead of trusting the scheduling snapshot: a
91+
// session open or new request must never lose a race against a
92+
// previously scheduled reap.
93+
if (!isIdleNow()) return;
94+
emitDiagnostic({ level: 'info', phase: 'daemon_idle_reap', data: { idleMs } });
95+
params.onIdleReap();
96+
}, idleMs);
97+
timer.unref?.();
98+
};
99+
100+
const noteActivity = (): void => {
101+
if (isIdleNow()) schedule();
102+
else cancel();
103+
};
104+
105+
return { noteActivity, cancel, idleMs };
106+
}

0 commit comments

Comments
 (0)