Skip to content

Commit 203aebb

Browse files
committed
fix(daemon): wait for a slow daemon startup and fail with typed reasons
A client killed a healthy daemon that was still starting: the local startup used one 15 s wall-clock deadline, and a timeout never retried. On an overloaded host that is a failed run before any work starts (#3782, failure class #3342). - startLocalDaemon now waits while the spawned daemon stays alive, up to a 120 s ceiling; AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS overrides it for the CLI, MCP, and Node client alike. - The daemon records startup phase markers (modules-loaded, lock-acquired, listening, registered) with timestamps in daemon-startup-phases.jsonl beside its registration. A live daemon that publishes no marker for 60 s is retired as stalled; one that keeps progressing is waited for to the ceiling. - Failure details carry reason (startup-exited, startup-stalled, startup-ceiling), the observed phase timings, and the effective startupTimeoutMs. Each reason has its own hint. - help commands documents the variable and the three reasons.
1 parent 0fb25c4 commit 203aebb

8 files changed

Lines changed: 630 additions & 26 deletions

File tree

‎packages/daemon-contracts/src/daemon-registration-owner.ts‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,85 @@ import {
4646
} from './daemon-shutdown-report.ts';
4747

4848
export const DAEMON_STARTUP_EXIT_CODES = Object.freeze({ busy: 75, unproven: 78 });
49+
50+
/**
51+
* Startup phase markers (#3782): the record a daemon leaves beside its registration of how far its
52+
* startup got. The client waiting for registration reads this record to tell "slow but progressing"
53+
* from "stuck". Each marker is one JSONL line appended to a dedicated state-directory file, so no
54+
* `daemon.log` reader has to learn a new line shape and `daemon.log` keeps its existing ownership.
55+
* The record is append-only across startups: a waiting client scopes it to its own launch by the
56+
* marker timestamps, never by deleting what an earlier startup left.
57+
*/
58+
const DAEMON_STARTUP_PHASES = Object.freeze([
59+
'modules-loaded',
60+
'lock-acquired',
61+
'listening',
62+
'registered',
63+
] as const);
64+
65+
export type DaemonStartupPhase = (typeof DAEMON_STARTUP_PHASES)[number];
66+
67+
const DAEMON_STARTUP_PHASE_SET: ReadonlySet<string> = new Set<string>(DAEMON_STARTUP_PHASES);
68+
69+
const DAEMON_STARTUP_PHASES_FILE = 'daemon-startup-phases.jsonl';
70+
71+
type DaemonStartupMarkerRecord = {
72+
data?: { startupPhase?: unknown; timestampMs?: unknown };
73+
};
74+
75+
export function resolveDaemonStartupPhasesPath(stateDir: string): string {
76+
return path.join(stateDir, DAEMON_STARTUP_PHASES_FILE);
77+
}
78+
79+
export function recordDaemonStartupMarker(
80+
startupPhasesPath: string,
81+
phase: DaemonStartupPhase,
82+
): void {
83+
try {
84+
const timestampMs = Date.now();
85+
fs.mkdirSync(path.dirname(startupPhasesPath), { recursive: true });
86+
fs.appendFileSync(
87+
startupPhasesPath,
88+
`${JSON.stringify({ data: { startupPhase: phase, timestampMs } })}\n`,
89+
);
90+
} catch {
91+
// A marker is progress evidence, never a startup requirement: a host that refuses the append
92+
// must not lose a daemon that can still register.
93+
}
94+
}
95+
96+
/**
97+
* The phases observed in a startup marker tail, keyed by phase, holding each phase's earliest
98+
* occurrence. Unparsable and unknown lines are ignored: the protocol is additive, and a client must
99+
* never fail a startup over a line shape it does not know.
100+
*/
101+
export function readDaemonStartupPhaseMarkers(
102+
markerTail: string,
103+
): Partial<Record<DaemonStartupPhase, number>> {
104+
const phases: Partial<Record<DaemonStartupPhase, number>> = {};
105+
for (const line of markerTail.split('\n')) {
106+
if (!line.includes('"startupPhase"')) continue;
107+
let record: DaemonStartupMarkerRecord;
108+
try {
109+
record = JSON.parse(line) as DaemonStartupMarkerRecord;
110+
} catch {
111+
continue;
112+
}
113+
const phase = record.data?.startupPhase;
114+
const timestampMs = record.data?.timestampMs;
115+
if (
116+
typeof phase !== 'string' ||
117+
!DAEMON_STARTUP_PHASE_SET.has(phase) ||
118+
typeof timestampMs !== 'number' ||
119+
!Number.isFinite(timestampMs)
120+
) {
121+
continue;
122+
}
123+
const seen = phases[phase as DaemonStartupPhase];
124+
if (seen === undefined || timestampMs < seen) phases[phase as DaemonStartupPhase] = timestampMs;
125+
}
126+
return phases;
127+
}
49128
export type DaemonRegistrationFields = Readonly<{
50129
socketPort?: number;
51130
httpPort?: number;

‎packages/daemon/src/server/http/daemon-runtime.ts‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,8 @@ import {
8888
import {
8989
tryAcquireDaemonRegistration,
9090
DAEMON_STARTUP_EXIT_CODES,
91+
recordDaemonStartupMarker,
92+
resolveDaemonStartupPhasesPath,
9193
type DaemonRegistrationOwner,
9294
} from '@agent-device/daemon-contracts/daemon-registration-owner';
9395
import { watchDaemonMetadataLoss } from './daemon-metadata-loss.ts';
@@ -660,6 +662,10 @@ async function startDaemonRuntimeOnHost(
660662
rollback.onRollback(async () => {
661663
await finishDaemonRegistration({ registration, infoPath, logPath });
662664
});
665+
// #3782: the client waiting on this state dir reads these markers to tell a slow daemon from a
666+
// stuck one.
667+
recordDaemonStartupMarker(resolveDaemonStartupPhasesPath(baseDir), 'lock-acquired');
668+
663669
rollback.onRollback(() => {
664670
stopMetadataLossWatch();
665671
});
@@ -752,7 +758,11 @@ async function startDaemonRuntimeOnHost(
752758
},
753759
rollback,
754760
);
761+
// #3782: the last two progress markers. A client that polls the registration and the markers
762+
// together can tell "listening but slow to answer" from "never progressed".
763+
recordDaemonStartupMarker(resolveDaemonStartupPhasesPath(baseDir), 'listening');
755764
publishDaemonInfo(socketPort, httpPort);
765+
recordDaemonStartupMarker(resolveDaemonStartupPhasesPath(baseDir), 'registered');
756766
stopMetadataLossWatch = armDaemonMetadataLossWatch(baseDir, infoPath, logPath, daemonIdentity);
757767
await flushDaemonStartupDiagnostics(logPath, startupDiagnostics);
758768
for (const plugin of deps.pluginInventory.filter(({ state }) => state === 'failed')) {

‎src/commands/schema/cli-help.ts‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,10 @@ const ENVIRONMENT_LINES = [
4141
label: 'AGENT_DEVICE_DAEMON_AUTH_TOKEN',
4242
description: 'Remote daemon service/API token',
4343
},
44+
{
45+
label: 'AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS',
46+
description: 'Local daemon startup ceiling (default 120000)',
47+
},
4448
{
4549
label: 'AGENT_DEVICE_CLOUD_BASE_URL',
4650
description: 'Bridge/control-plane API origin for cloud auth and /api-keys',
@@ -508,6 +512,7 @@ Runner and daemon lifecycle (applies to simulators too):
508512
No runner read launches a session app that is not running: snapshot, wait, is, get, a reading find, and an interaction's leading reads (a gesture's viewport read, the capture that resolves a selector click/fill) answer the retriable APP_NOT_RUNNING instead of bare-launching over a launch SpringBoard still holds behind its deep-link confirmation. Only open, activate, and a command that mutates without a leading read bring a stopped app up.
509513
close keeps a healthy iOS simulator XCTest runner warm by default so the next open on that simulator (same udid in the same simulator set) skips the runner build, unless --shutdown was requested, the session was recording, or the session held a device lease. A retained runner auto-stops after an idle window (default 5 minutes); set AGENT_DEVICE_IOS_RUNNER_IDLE_STOP_MS to override, or 0 to disable idle stop and retain until daemon exit. Only a ready runner with an observed listener is retained; close retires unfinished prewarm recovery. Losing the retained listener stops that generation, and daemon shutdown stops idle retained runners rather than handing them off without supervision.
510514
Each AGENT_DEVICE_STATE_DIR runs its own daemon. It self-exits after an idle window (default 5 minutes, matching the runner idle-stop default) once it has no open sessions, no in-flight requests, and no active recording; set AGENT_DEVICE_DAEMON_IDLE_TIMEOUT_MS to override, or 0 to disable idle reap.
515+
A client starting a local daemon waits while the daemon process stays alive and keeps publishing startup progress, so an overloaded host makes startup slower, not failed. It stops at a ceiling (default 120 s; set AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS to override) and names the outcome in error details.reason: startup-exited (the process ended before registering), startup-stalled (alive but no startup progress for 60 s), or startup-ceiling (progressing, not registered before the ceiling). details.phases carries the startup phase timings it observed and details.startupTimeoutMs the effective ceiling. Branch on details.reason, never on the message.
511516
A command that needs the shared local daemon starts it for its state dir when none runs (device status and device release --stale never contact one). When the running daemon comes from another version or build, the command replaces it only if the daemon reports no work: no in-flight request, session, repair session, or live lease (a streaming resumable upload and a host human-control hold that names no lease are not counted); otherwise it fails with details.reason daemon-busy and leaves the daemon alone. A daemon that agreed to leave is waited for up to 60 s and never signaled while it drains; one still running then fails with details.reason daemon-drain-timeout unless it answers nothing and is provably a daemon process, which is force-retired. A daemon from a release that predates this check is never replaced and fails with daemon-incompatible. End a busy or older daemon deliberately with agent-device daemon stop --state-dir <path>. Node clients pick daemonLifecycle 'attach' (never starts or replaces a daemon; fails with daemon-not-running or daemon-incompatible) or 'managed' (the default until 0.22), and ensureDaemon() starts or replaces one explicitly.
512517
On a machine shared by several agents, set AGENT_DEVICE_SESSION_IDLE_TIMEOUT_MS to also expire an individual session that has taken a host-global device claim and then received no commands for that long; the claim is released and the next command on that session answers SESSION_NOT_FOUND with details.reason SESSION_IDLE_EXPIRED naming the window and the device. Off by default, because on a shared host "idle" and "thinking" are indistinguishable: only set it where an abandoned claim blocking every other agent is the worse failure. Sessions holding a remote lease or any active capture (recording, logs, audio, performance, or trace) are never expired this way.
513518
A stale iOS runner lease — its owner process dead, or its AGENT_DEVICE_STATE_DIR deleted — is reclaimed automatically instead of failing with "is already owned by another agent-device daemon". A live owner's runner is also reclaimed when the requesting daemon holds the host-global device claim for that device: claims are exclusive, so holding one proves the runner's owner released the device and merely kept the runner warm. The error remains only for owners outside claim arbitration (a pre-claims build, or daemons pointed at different claim stores).

0 commit comments

Comments
 (0)