Skip to content

Commit 72e7965

Browse files
thymikeeclaude
andauthored
fix(daemon): replace fixed 15s startup kill with progress-aware startup budget (#3791)
* fix(daemon): wait for a slow daemon startup and fail with typed reasons A client killed a healthy daemon that was still starting: the local startup used one 15 s wall-clock deadline, and a timeout never retried. On an overloaded host that is a failed run before any work starts (#3782, failure class #3342). - startLocalDaemon now waits while the spawned daemon stays alive, up to a 120 s ceiling; AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS overrides it for the CLI, MCP, and Node client alike. - The daemon records startup phase markers (modules-loaded, lock-acquired, listening, registered) with timestamps in daemon-startup-phases.jsonl beside its registration. A live daemon that publishes no marker for 60 s is retired as stalled; one that keeps progressing is waited for to the ceiling. - Failure details carry reason (startup-exited, startup-stalled, startup-ceiling), the observed phase timings, and the effective startupTimeoutMs. Each reason has its own hint. - help commands documents the variable and the three reasons. * fix(daemon): give each startup its own marker file and retire it at its owner The integration leak oracle failed because every daemon appended its startup phase markers to one shared `daemon-startup-phases.jsonl` beside the registration and nothing retired that file, so any completed launch left state-dir residue the strict oracle names as a stray entry. Ownership now follows the file name: each launch writes only to `daemon-startup-phases-<pid>.jsonl` through `resolveDaemonStartupPhasesPath(stateDir, pid)`. The writing daemon removes its file once it has registered; the launching client removes it once that process's exit is joined (in `retireStartupAttempt`, after the exit is observed, never before); and each new registration-lock owner sweeps the files of provably dead pids next to the shutdown-report removal. A live pid's file is always kept, so a concurrent or replaced launch can neither add to, hide behind, nor delete another launch's evidence, and the client no longer needs `spawnedAtMs` to attribute markers to its own launch. Along the protocol boundary: - `recordDaemonStartupMarker` takes an optional `reportFailure` callback so producers surface a refused append on stderr (the daemon log tail the waiting client already ships as failure evidence) instead of swallowing it; the append stays non-fatal and a crash-truncated final line is separated so it cannot glue two records together. - The phase list gains real completed boundaries, `composition-loaded` and `recovery-finished`, and no heartbeats, so silence still means a stalled startup and progress extends the stall window exactly as far as the work took. - The launch-exit join drops its redundant `.catch(() => {})`: the `runCmdDetachedMonitored` owner resolves the promise on error and exit, so it never rejects. - Startup code splits out of `daemon-client-lifecycle.ts` into `daemon-client-startup.ts` (budget, wait loop, marker refresh, hints), keeping both modules under the 1000-line tripwire; importers attach and ensure point at the owning module directly. Regression coverage: the budget suite now pins per-launch attribution, including a fresh daemon producing its own markers while an earlier launch's leftover file is swept at the new registration, and a new protocol suite pins the per-pid paths, the dead-pid sweep, removal idempotence, and the write-failure seam. Refs #3782, #3791 * fix(help): state the startup failure reasons the code actually produces The documented reasons overclaimed. `startup-ceiling` said "progressing, not registered before the ceiling", but the ceiling is reached by a daemon that is merely alive: a progressing one is only ever stopped by a ceiling above the stall window, so the promise was unverifiable from the client's evidence. `startup-stalled` named a 60 s window without saying that `AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS` does not change it, so raising the ceiling looked like a fix for silence. Both the CLI help and the configuration table now describe the two reasons as what they are, and the env var is specified as a positive integer number of milliseconds. Refs #3791 * docs(daemon): remove narrating module headers The lifecycle and startup module headers only narrate each file's role, which AGENTS.md reserves for public APIs, tool directives, and constraints code cannot show. The split is owned by the imports and the file names. Fixes review thread PRRT_kwDORFGHaM6tkpef on #3791. * test(daemon): scope fixture marker emission to the child that asked for it The shared emit-startup-markers flag file applied to every later fixture in the same state dir, so the stale-history cleaner inherited the request, wrote its own live marker, and the sweep assertion raced it. The request is now a typed per-child option bound into that child's entry script, and readiness waits use a per-pid file so a stale registration-held from an earlier child cannot shortcut the await. Fixes review thread PRRT_kwDORFGHaM6tkpeV on #3791. * fix(daemon): sweep an exited contender's own record on every terminal exit A launch that exited busy, unproven, or lost the registration to a contending winner returned its retry, error, or winner handoff without joining its exact exited child, so the child's pid-named phase file sat in the state dir until some later registration sweep. Every terminal child outcome now joins the child first and removes only that child's own file, the same ownership rule a retired startup follows; the current winner's registration and record are untouched. Regression tests write markers at the real pre-acquisition producer seam (new emitStartupMarkerBeforeAcquisition fixture option) and pin the residue rule on the retry, winner-attach, and unproven-exit paths: red at the prior head, green with the fix. Fixes review thread PRRT_kwDORFGHaM6tkpec on #3791. * fix(daemon): let each launch claim its pid's marker record The per-pid file names its writer, not its lifetime: if client and daemon of an old launch both died before either cleanup ran and the host reused the pid, the old record survived under the new writer's name, the sweep skipped it as live, and the earliest-per-phase reader mixed two lifetimes into one observation. The producer now claims its own record (claimDaemonStartupPhasesFile) before its first marker, so stale same-pid content can neither mask nor extend the fresh startup's progress. A refused claim follows the marker contract: reported, never fatal. The test child repeats the real producer sequence against its own pid, which no outside test can arrange. Addresses review finding R2-1 on #3791. * refactor(daemon): move startup diagnostics and registration completion out of the runtime Code ownership allows no exception: behavior was added to a module over 1000 lines, so the module is split at its existing boundary first. flushDaemonStartupDiagnostics, emitDaemonDiagnostic, finishDaemonRegistration, and armDaemonMetadataLossWatch form the daemon-log diagnostic and registration-completion edge the runtime already treats as one unit; they move with their callers updated, no behavior change. daemon-runtime.ts is back under the limit. * refactor(daemon): drop the phase-timings return cast describeStartupPhaseTimings builds a Record<string, number> that always starts at spawned: 0; the intersection cast asserted what the literal already showed. No caller branched on it. Addresses review finding R2-2 on #3791. * ci(ios): capture the cold daemon-startup timer in the preflight step The #3782 startup-budget evidence needs one measured successful cold startup: after clean:daemon, the prepare command's daemon_startup diagnostic now wraps a fresh managed startup and lands in the step log. Behavior and JSON output are unchanged. Records the evidence obligation for #3791. * test(daemon): give every fixture launch its own entry script registeredDaemonFixtureArgs rewrote one shared dist/src/internal/ daemon.js per construction, so a second build could replace the script before an earlier launched child loaded it, silently applying the wrong per-child marker switches and registration fields. Each construction now writes a unique fixture-entries/<uuid>/dist/src/internal/daemon.js: the per-launch directory keeps the daemon entry path shape the retirement identity checks require, and cleanup still removes the state dir. A regression test builds both arg sets before launching either child and proves child 1 executes its own switches while child 2 exits busy: red on the shared path, green per-launch. Fixes review thread PRRT_kwDORFGHaM6tk7ed on #3791. * perf(daemon): keep the startup-diagnostics edge out of the eager closure The split moved the helpers next to the runtime but left a static edge, so both daemon entries evaluated one more module and 12-15 more top-level statements than the merge-base (Coverage lane red: 3 eager-closure-budgets failures at OUTPUT_ECONOMY_BASE=4a8abed7b9). startDaemonRuntime now loads them through a function-scoped await import at its first use point, the loading shape the gate prescribes; composition, publication, flush, and shutdown order are unchanged and the source-order guards still read them from this module. Red at 62a09bd (3 failures), green here: 846/846. Addresses the eager-closure failure reported at 62a09bd on #3791. * ci(ios): measure the cold daemon start at the public SDK ensure seam The prepare step's daemon_startup diagnostic is not a cold-startup measurement: the CLI's ensured transport finishes ensureDaemon before sendToDaemon's timer wraps the attach connect, so the timer only sees an attach. Time the public ensureDaemon() directly after clean:daemon and record phase, durationMs, started, and pid, so the record counts as cold only when started is true. The prepare command keeps its original semantics and attaches to the same daemon. Corrects the measurement assumption behind 62a09bd on #3791; the cold-startup CI evidence for #3782 is still missing until a run reports this record with started:true. * fix(daemon): report startup-ceiling when any live startup is pending at the deadline When the launched contender exited busy while another live daemon held the registration and deferred publication, the wait loop ended in an unlabeled 'timeout' result that mapped to startup-exited with the generic hint. Any deadline with a live startup still pending, this client's launch or a verified live lock owner, now reports startup-ceiling with the effective ceiling and the overload hint; startup-exited stays reserved for a real exit or no live startup. Remove the 'timeout' wait kind so the unlabeled state cannot exist and add the sendToDaemon-boundary regressions for both outcomes. Also compress this PR's narration and duplicate test setup to keep the gross diff within one coherent change, and add the Unreleased changelog entry for the startup budget, the stall window, and the three reasons. * fix(daemon): classify the expired startup wait by its observed exit and pid identity At an expired deadline, a delivered child exit is authoritative for the lifetime it watched, so it outranks an unobserved liveness probe of a pid that may have been reused. Unobserved, the launch's recorded birth time must prove the pid changed hands before it counts as a live startup; an unavailable identity stays fail-closed. Extracted classifyExpiredDaemonStartup keeps the wait loop within its complexity thresholds. The budget test pins liveness of its fabricated pids at the process seam instead of trusting the host, and the winner-sweep race coverage is deduplicated into the joined contender test. * docs(daemon): scope the startup ceiling env to daemon starts and restore the marker sweep owner Both facts were stated incompletely against the code: - resolveDaemonStartupTimeoutMs runs only inside startLocalDaemon, so AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS is validated only when a client must start a daemon. A client that attaches to or reuses a running daemon never reads the variable. CHANGELOG, cli-help, and the configuration guide now say so; the guide already named INVALID_ARGS. - The startup-marker ownership comment dropped the registration owner's sweep of provably dead pids, which sweepDeadDaemonStartupPhasesFiles still performs at each new registration lock acquisition. Addresses review threads r4305110030 and r4305299946 on #3791. Co-authored-by: Claude Opus 4.8 <[email protected]> * test(daemon): point the host test's observe mock at the startup module The startup split moved observeLocalDaemon out of daemon-client-lifecycle.ts, so the host test's lifecycle-module mock no longer stopped the ensure path and the CLI-run case fell through to the real observation. Mock the seam the subject imports it from now. Drop two narration comments the assertions already state. Co-authored-by: Claude Opus 4.8 <[email protected]> * refactor(daemon): give startup phase records their own contracts module The per-pid startup phase file has its own record format, writer, reader and dead-pid sweep, so it no longer belongs inside the registration owner the review pointed at (#3791). The declarations move verbatim into daemon-contracts/daemon-startup-phases.ts beside daemon-shutdown-report and the suffix-named markers test follows them as daemon-startup-phases.test.ts. Registration acquisition imports only sweepDeadDaemonStartupPhasesFiles; the runtime, the daemon entry, the client wait and the fixture import the record owner directly, with no re-export shim (unreleased surface). The daemon entry must write its earliest modules-loaded marker at module load, so the eager-closure gate gets one APPROVED_CLOSURE_ADDITIONS row naming the new module; the entry pays for the record it writes first. Addresses review 6248815887 item 1 on #3791. Refs #3782, #3791 * refactor(daemon): point every startup-marker consumer at the new owner Importers attach to daemon-contracts/daemon-startup-phases directly; daemon-registration-owner keeps only the sweep call at acquisition. src/daemon.ts captures its host environment once and resolves the phases path from the same env and cwd startDaemonRuntime resolves its state dir from, so a relative AGENT_DEVICE_STATE_DIR cannot land the earliest markers beside a different directory than the registration (the module top-level second resolveDaemonPaths(process.env) is gone), and the modules-loaded marker stays the first thing recorded. Addresses review 6248815887 items 1 and 2 on #3791. Refs #3782, #3791 * test(daemon): prove the progress-aware budget on successive phases The headline slow-start test rewrote lock-acquired repeatedly; the reader keeps each phase's earliest timestamp, so those writes never advanced progress and the run spanned only 20 s against the 60 s stall window. It now schedules four distinct phases (modules-loaded, composition-loaded, lock-acquired, recovery-finished) across 70 s of virtual time under a 150 s ceiling and asserts the wait survived on observed successive progress. With the marker read removed, the same scenario fails as startup-stalled at the 60 s window (verified with a throwaway stub, not committed), so the test is credibly red without the progress rule. Addresses review 6248815887 item 3 on #3791. Refs #3782, #3791 * refactor(daemon): converge the marker helpers and shared fixture waits * fix(daemon): scope the startup-ceiling claim to the owned launch and prove the default ceiling The help and configuration docs claimed only a progressing daemon reaches a ceiling above the stall window. On the contended path this client's launch exits busy and the wait runs on the exit branch, where the stall check never runs, so a live startup owned by another client reaches the ceiling with no observed progress. Qualify the clause to this client's own launch, matching the CHANGELOG definition. The no-marker budget test stubbed 120000 instead of unsetting the variable, so it never proved the default. Unset it (stubEnv with undefined) so the resolved ceiling is the unoverridden default even on a host that exports the variable. Also drop a stray blank line in the Unreleased list. --------- Co-authored-by: Claude Opus 4.8 <[email protected]>
1 parent 63f762e commit 72e7965

22 files changed

Lines changed: 1827 additions & 673 deletions

‎.github/workflows/ios.yml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -169,6 +169,14 @@ jobs:
169169
- name: Preflight iOS runner through public CLI
170170
run: |
171171
pnpm clean:daemon
172+
# The public-SDK record is the cold-startup evidence; the CLI timer only wraps attach.
173+
node --input-type=module -e '
174+
const { performance } = await import("node:perf_hooks");
175+
const { ensureDaemon } = await import("./dist/src/index.js");
176+
const t0 = performance.now();
177+
const c = await ensureDaemon({ stateDir: process.env.AGENT_DEVICE_STATE_DIR });
178+
console.log(JSON.stringify({ phase: "daemon_cold_startup", started: c.started, pid: c.daemon.pid, durationMs: Math.round(performance.now() - t0) }));
179+
'
172180
node bin/agent-device.mjs prepare ios-runner --platform ios --udid "${{ steps.ios-simulator.outputs.simulator-udid }}" --timeout "$AGENT_DEVICE_IOS_PREPARE_TIMEOUT_MS" --json
173181
174182
# The replay reuses the prepared runner in the same daemon. --debug keeps every step's request

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44

55
- The PNG worker no longer calls `process.emitWarning` when it is unavailable, and it starts with an empty environment (`env: {}`). The daemon registers a sink when it pre-warms the worker, so every transition to unavailable (spawn failure, missing entry, load error, exit) leaves exactly one `png_worker_unavailable` event, with a typed `reason` (`entry-not-found`, `spawn-failed`, `worker-error`, `worker-exited`) and its `detail`, in `daemon.log`, with or without `--debug`. Requests then continue on the in-process fallback without repeating the event.
66
- Behavior change: `resolveHubAppReference`, shipped through `agent-device/plugins/webdriver`, now needs `cwd` to resolve a relative app path. Without it the call fails with `INVALID_ARGS` and `details.reason: provider_app_cwd_missing`; it used to resolve against the process's working directory. Absolute paths, URLs and provider references are unaffected. The daemon fills it from the client's working directory (or its host's) for every provider lease request, so `connect` and `lease_allocate` are unchanged.
7+
- A client starting a local daemon now waits while the daemon process stays alive and keeps publishing startup progress, so an overloaded host makes startup slower instead of failing it. The wait stops at a ceiling of 120 s by default; set `AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS` to a positive integer number of milliseconds to override it for the CLI, MCP, and Node clients alike. The variable is read only when the client must start a daemon; a value that is not a positive integer fails with `INVALID_ARGS` before the daemon spawns, and a client that only attaches to or reuses a running daemon never reads it. A failed startup names the outcome in `error.details.reason`, and you should branch on it, never on the message: `startup-exited` (the process ended before registering), `startup-stalled` (alive but no startup progress for a fixed 60 s window, which the ceiling variable does not change; the daemon is stopped), or `startup-ceiling` (a startup was still live — this client's daemon or another client's — when the ceiling passed; the client stops its own launch and never signals another client's live daemon). `details.phases` carries the startup phase timings the client observed and `details.startupTimeoutMs` the effective ceiling.
78
- The `daemon-not-running` hint now always names `--state-dir <dir>`, and a daemon force-retired for outlasting its drain is reported as `Replaced unreachable daemon` instead of `Replaced idle daemon`. `daemon-drain-timeout` and the post-drain force retire are documented in the client API and `help`.
89
- Fix `ensureDaemon()`, the CLI and MCP killing a daemon they had just asked to leave. After the daemon answers the quiesce request with `draining`, the client now waits up to 60 s for it to exit by itself and clean up its registration, so plugin disposal and lease finalization finish (on Windows a signal ends the process outright, and on POSIX a drain longer than 3 s was killed). A daemon still running at the deadline that fails its liveness probes is force-retired; one that still answers is left running and the call fails with `details.reason: 'daemon-drain-timeout'`, with a hint naming `agent-device daemon stop --state-dir <path>`.
910
- `@agent-device/testmu` is now a version 2 provider plugin. `import testmu from '@agent-device/testmu'` gives a factory, `testmu({ credentials: { username, accessKey } })`, that reads those credentials instead of `LT_USERNAME` and `LT_ACCESS_KEY`; without options it reads the variables as before. The package still declares the version 1 plugin entry, so `agent-device plugins add @agent-device/testmu` keeps working on hosts that predate the version 2 contract.

‎packages/daemon-contracts/package.json‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,10 @@
4444
"types": "./src/daemon-shutdown-report.ts",
4545
"default": "./src/daemon-shutdown-report.ts"
4646
},
47+
"./daemon-startup-phases": {
48+
"types": "./src/daemon-startup-phases.ts",
49+
"default": "./src/daemon-startup-phases.ts"
50+
},
4751
"./daemon-stop": {
4852
"types": "./src/daemon-stop.ts",
4953
"default": "./src/daemon-stop.ts"

‎packages/daemon-contracts/src/daemon-registration-owner.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,7 @@ import {
4444
resolveDaemonShutdownReportPath,
4545
type DaemonShutdownOutcome,
4646
} from './daemon-shutdown-report.ts';
47+
import { sweepDeadDaemonStartupPhasesFiles } from './daemon-startup-phases.ts';
4748

4849
export const DAEMON_STARTUP_EXIT_CODES = Object.freeze({ busy: 75, unproven: 78 });
4950
export type DaemonRegistrationFields = Readonly<{
@@ -85,6 +86,7 @@ export async function tryAcquireDaemonRegistration(
8586
try {
8687
acquisition.assertHeld();
8788
fs.rmSync(resolveDaemonShutdownReportPath(boundPaths.baseDir), { force: true });
89+
sweepDeadDaemonStartupPhasesFiles(boundPaths.baseDir);
8890
} catch (error) {
8991
await releaseRegistrationAfterFailure(acquisition, error, boundPaths.logPath);
9092
}
Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
import { appendFile, readFile, writeFile } from 'node:fs/promises';
2+
import { spawnSync } from 'node:child_process';
3+
import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs';
4+
import path from 'node:path';
5+
import { expect, test } from 'vitest';
6+
import { tryAcquireDaemonRegistration } from '@agent-device/daemon-contracts/daemon-registration-owner';
7+
import {
8+
readDaemonStartupPhaseMarkers,
9+
recordDaemonStartupMarker,
10+
removeDaemonStartupPhasesFile,
11+
resolveDaemonStartupPhasesPath,
12+
} from '@agent-device/daemon-contracts/daemon-startup-phases';
13+
import { resolveDaemonPaths } from '@agent-device/daemon-contracts/daemon-resolution';
14+
import { mkdtempForTest } from '@agent-device/test-kit';
15+
16+
test('per-launch marker files: a fresh launch never reads an older launch phases', async () => {
17+
const stateDir = await mkdtempForTest('agent-device-markers-');
18+
const first = resolveDaemonStartupPhasesPath(stateDir, 1111);
19+
const second = resolveDaemonStartupPhasesPath(stateDir, 2222);
20+
recordDaemonStartupMarker(first, 'lock-acquired');
21+
expect(second).not.toBe(first);
22+
expect(existsSync(second)).toBe(false);
23+
recordDaemonStartupMarker(second, 'modules-loaded');
24+
const firstPhases = readDaemonStartupPhaseMarkers(readFileSync(first, 'utf8'));
25+
const secondPhases = readDaemonStartupPhaseMarkers(readFileSync(second, 'utf8'));
26+
expect(Object.keys(firstPhases)).toEqual(['lock-acquired']);
27+
expect(Object.keys(secondPhases)).toEqual(['modules-loaded']);
28+
});
29+
30+
test('marker reading keeps the earliest record when a phase replays after a later one', () => {
31+
const text = [
32+
'{"data":{"startupPhase":"listening","timestampMs":500}}',
33+
'{"data":{"startupPhase":"lock-acquired","timestampMs":20}}',
34+
'{"data":{"startupPhase":"lock-acquired","timestampMs":35}}',
35+
].join('\n');
36+
const phases = readDaemonStartupPhaseMarkers(text);
37+
expect(phases['lock-acquired']).toBe(20);
38+
expect(phases.listening).toBe(500);
39+
});
40+
41+
test('marker reading ignores unknown phases and unparsable lines', () => {
42+
const text = [
43+
'{"data":{"startupPhase":"heartbeat","timestampMs":10}}',
44+
'{"data":{"startupPhase":"registered","timestampMs":"7"}}',
45+
'not json',
46+
'{"data":{"startupPhase":"listening","timestampMs":30}}',
47+
].join('\n');
48+
expect(readDaemonStartupPhaseMarkers(text)).toEqual({ listening: 30 });
49+
});
50+
51+
test('the producer lifetime claim empties a same-pid leftover', async () => {
52+
// Pid reuse after both the old client and daemon died before either cleanup: the entry removes
53+
// its record before its first marker; this child repeats that producer sequence itself.
54+
const stateDir = await mkdtempForTest('agent-device-markers-');
55+
writeFileSync(path.join(stateDir, 'package.json'), '{"type":"module"}');
56+
const entry = path.join(stateDir, 'claim-producer.js');
57+
const registrationUrl = new URL('./daemon-startup-phases.ts', import.meta.url).href;
58+
writeFileSync(
59+
entry,
60+
`import { removeDaemonStartupPhasesFile, recordDaemonStartupMarker, resolveDaemonStartupPhasesPath } from ${JSON.stringify(registrationUrl)};
61+
const file = resolveDaemonStartupPhasesPath(${JSON.stringify(stateDir)}, process.pid);
62+
recordDaemonStartupMarker(file, 'listening');
63+
recordDaemonStartupMarker(file, 'registered');
64+
removeDaemonStartupPhasesFile(file);
65+
recordDaemonStartupMarker(file, 'modules-loaded');
66+
`,
67+
);
68+
const child = spawnSync(process.execPath, ['--experimental-strip-types', entry], {
69+
encoding: 'utf8',
70+
});
71+
if (child.status !== 0) throw new Error(child.stderr);
72+
const phases = readDaemonStartupPhaseMarkers(
73+
readFileSync(resolveDaemonStartupPhasesPath(stateDir, child.pid!), 'utf8'),
74+
);
75+
expect(phases).toEqual({ 'modules-loaded': expect.any(Number) });
76+
});
77+
78+
test.each([
79+
['remove', (p: string, cb: (e: unknown) => void) => removeDaemonStartupPhasesFile(p, cb)],
80+
[
81+
'record',
82+
(p: string, cb: (e: unknown) => void) => recordDaemonStartupMarker(p, 'modules-loaded', cb),
83+
],
84+
] as const)('a refused %s reports through the callback instead of throwing', async (_name, run) => {
85+
const stateDir = await mkdtempForTest('agent-device-markers-');
86+
const markerPath = resolveDaemonStartupPhasesPath(stateDir, 777);
87+
mkdirSync(markerPath, { recursive: true });
88+
const reported: unknown[] = [];
89+
expect(() =>
90+
run(markerPath, (error) => {
91+
reported.push(error);
92+
}),
93+
).not.toThrow();
94+
expect(reported).toHaveLength(1);
95+
expect(reported[0]).toHaveProperty('message');
96+
});
97+
98+
test('acquiring the registration retires files of dead or unowned pids only', async () => {
99+
const paths = resolveDaemonPaths(await mkdtempForTest('agent-device-markers-'));
100+
const dead = resolveDaemonStartupPhasesPath(paths.baseDir, spawnDeadPid());
101+
const own = resolveDaemonStartupPhasesPath(paths.baseDir, process.pid);
102+
const unparseable = path.join(paths.baseDir, 'daemon-startup-phases-.jsonl');
103+
const legacy = path.join(paths.baseDir, 'daemon-startup-phases.jsonl');
104+
await Promise.all([
105+
writeFile(own, 'x\n', 'utf8'),
106+
writeFile(dead, 'x\n', 'utf8'),
107+
writeFile(unparseable, 'x\n', 'utf8'),
108+
writeFile(legacy, 'x\n', 'utf8'),
109+
]);
110+
const attempt = await tryAcquireDaemonRegistration(paths);
111+
expect(attempt.status).toBe('acquired');
112+
if (attempt.status !== 'acquired') throw new Error('registration refused');
113+
expect(existsSync(dead)).toBe(false);
114+
expect(existsSync(unparseable)).toBe(false);
115+
expect(existsSync(own)).toBe(true);
116+
expect(existsSync(legacy)).toBe(true);
117+
await attempt.owner.finish();
118+
});
119+
120+
test('removeDaemonStartupPhasesFile removes a file and tolerates one already gone', async () => {
121+
const stateDir = await mkdtempForTest('agent-device-markers-');
122+
const file = resolveDaemonStartupPhasesPath(stateDir, 4242);
123+
await writeFile(file, 'x\n', 'utf8');
124+
removeDaemonStartupPhasesFile(file);
125+
expect(existsSync(file)).toBe(false);
126+
expect(() => removeDaemonStartupPhasesFile(file)).not.toThrow();
127+
});
128+
129+
test('a crash-replay append keeps the file valid JSONL', async () => {
130+
const stateDir = await mkdtempForTest('agent-device-markers-');
131+
const markerPath = resolveDaemonStartupPhasesPath(stateDir, 999);
132+
await appendFile(markerPath, '{"data":{"startupPhase":"modules-lo', 'utf8');
133+
recordDaemonStartupMarker(markerPath, 'composition-loaded');
134+
const tail = await readFile(markerPath, 'utf8');
135+
expect(Object.keys(readDaemonStartupPhaseMarkers(tail))).toEqual(['composition-loaded']);
136+
});
137+
138+
function spawnDeadPid(): number {
139+
const child = spawnSync(process.execPath, ['-e', ''], { encoding: 'utf8' });
140+
if (child.error) throw child.error;
141+
const pid = child.pid;
142+
if (pid === undefined) throw new Error('spawnSync returned no pid');
143+
return pid;
144+
}
Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
import fs from 'node:fs';
2+
import path from 'node:path';
3+
import { normalizeError, type NormalizedError } from '@agent-device/kernel/errors';
4+
import { isProcessAlive } from '@agent-device/host-kit/process';
5+
6+
/**
7+
* Startup phase markers: a per-launch record, named for the writing pid, of how far that startup
8+
* got, so a waiting client tells "slow but progressing" from "stuck" without a heartbeat. The
9+
* writer removes it on completion, the client after a joined exit, and each new registration
10+
* owner sweeps the files of provably dead pids.
11+
*/
12+
const DAEMON_STARTUP_PHASES = Object.freeze([
13+
'modules-loaded',
14+
'composition-loaded',
15+
'lock-acquired',
16+
'recovery-finished',
17+
'listening',
18+
'registered',
19+
] as const);
20+
21+
export type DaemonStartupPhase = (typeof DAEMON_STARTUP_PHASES)[number];
22+
23+
const DAEMON_STARTUP_PHASE_SET: ReadonlySet<string> = new Set<string>(DAEMON_STARTUP_PHASES);
24+
25+
const DAEMON_STARTUP_PHASES_ENTRY = /^daemon-startup-phases-(\d*)\.jsonl$/;
26+
27+
/** The marker file the daemon process `pid` writes in `stateDir`; an absent file is no evidence. */
28+
export function resolveDaemonStartupPhasesPath(stateDir: string, pid: number): string {
29+
return path.join(stateDir, `daemon-startup-phases-${pid}.jsonl`);
30+
}
31+
32+
/** Appends one phase marker; a refused append goes to `reportFailure`, never fatal to a startup. */
33+
export function recordDaemonStartupMarker(
34+
startupPhasesPath: string,
35+
phase: DaemonStartupPhase,
36+
reportFailure?: (error: NormalizedError) => void,
37+
): void {
38+
try {
39+
const timestampMs = Date.now();
40+
fs.mkdirSync(path.dirname(startupPhasesPath), { recursive: true });
41+
// A crash mid-append can leave no trailing newline; glue two appends and both are lost.
42+
const separator = endsWithoutNewline(startupPhasesPath) ? '\n' : '';
43+
fs.appendFileSync(
44+
startupPhasesPath,
45+
`${separator}${JSON.stringify({ data: { startupPhase: phase, timestampMs } })}\n`,
46+
);
47+
} catch (error) {
48+
reportFailure?.(normalizeError(error));
49+
}
50+
}
51+
52+
function endsWithoutNewline(file: string): boolean {
53+
let fd: number | undefined;
54+
try {
55+
const stats = fs.statSync(file);
56+
if (stats.size === 0) return false;
57+
const buffer = Buffer.alloc(1);
58+
fd = fs.openSync(file, 'r');
59+
if (fs.readSync(fd, buffer, 0, 1, stats.size - 1) !== 1) return false;
60+
return buffer[0] !== 0x0a;
61+
} catch {
62+
return false;
63+
} finally {
64+
if (fd !== undefined) fs.closeSync(fd);
65+
}
66+
}
67+
68+
/**
69+
* Removes this launch's record: before its first marker (a per-pid name identifies a writer, not
70+
* a lifetime, so the entry claims its file first), and after a finished startup or joined exit.
71+
* A refused removal only reports through `reportFailure`; it never stops a startup.
72+
*/
73+
export function removeDaemonStartupPhasesFile(
74+
startupPhasesPath: string,
75+
reportFailure?: (error: NormalizedError) => void,
76+
): void {
77+
try {
78+
fs.rmSync(startupPhasesPath, { force: true });
79+
} catch (error) {
80+
reportFailure?.(normalizeError(error));
81+
}
82+
}
83+
84+
/** Sweeps marker files of provably dead (or unnameable) pids; a live pid's file is kept. */
85+
export function sweepDeadDaemonStartupPhasesFiles(stateDir: string): void {
86+
let entries: string[];
87+
try {
88+
entries = fs.readdirSync(stateDir);
89+
} catch {
90+
return;
91+
}
92+
for (const entry of entries) {
93+
const match = DAEMON_STARTUP_PHASES_ENTRY.exec(entry);
94+
if (match === null) continue;
95+
const deadPid = Number(match[1]);
96+
if (deadPid === process.pid) continue;
97+
if (!Number.isSafeInteger(deadPid) || deadPid <= 0 || !isProcessAlive(deadPid)) {
98+
removeDaemonStartupPhasesFile(path.join(stateDir, entry));
99+
}
100+
}
101+
}
102+
103+
/**
104+
* The phases observed in a marker tail, keeping each phase's earliest occurrence. Unknown and
105+
* unparsable lines are ignored: a client never fails a startup over a line it does not know.
106+
*/
107+
export function readDaemonStartupPhaseMarkers(
108+
markerTail: string,
109+
): Partial<Record<DaemonStartupPhase, number>> {
110+
const phases: Partial<Record<DaemonStartupPhase, number>> = {};
111+
for (const line of markerTail.split('\n')) {
112+
if (!line.includes('"startupPhase"')) continue;
113+
let record: DaemonStartupMarkerRecord;
114+
try {
115+
record = JSON.parse(line) as DaemonStartupMarkerRecord;
116+
} catch {
117+
continue;
118+
}
119+
const phase = record.data?.startupPhase;
120+
const timestampMs = record.data?.timestampMs;
121+
if (
122+
typeof phase !== 'string' ||
123+
!DAEMON_STARTUP_PHASE_SET.has(phase) ||
124+
typeof timestampMs !== 'number' ||
125+
!Number.isFinite(timestampMs)
126+
) {
127+
continue;
128+
}
129+
const seen = phases[phase as DaemonStartupPhase];
130+
if (seen === undefined || timestampMs < seen) phases[phase as DaemonStartupPhase] = timestampMs;
131+
}
132+
return phases;
133+
}
134+
135+
type DaemonStartupMarkerRecord = {
136+
data?: { startupPhase?: unknown; timestampMs?: unknown };
137+
};

‎packages/daemon/src/server/__tests__/daemon-runtime-app-log.test.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ import type { ResourceDiagnostic } from '@agent-device/host-kit/diagnostics';
1010
import { createDurableResourceEnvelope } from '@agent-device/capture-kit';
1111
import { createTestAppLogLiveHandle, mkdtempForTestSync } from '@agent-device/test-kit';
1212
import { makeSessionStore } from '../../__tests__/support/store-factory.ts';
13-
import { flushDaemonStartupDiagnostics } from '../http/daemon-runtime.ts';
13+
import { flushDaemonStartupDiagnostics } from '../http/daemon-startup-diagnostics.ts';
1414
import type { SessionState } from '../../core/session-state.ts';
1515
import { unavailableDeviceRuntimeGateway } from '../../__tests__/support/unavailable-device-runtime-gateway.ts';
1616

0 commit comments

Comments
 (0)