Repository navigation
Commit 72e7965
fix(daemon): replace fixed 15s startup kill with progress-aware startup budget (#3791)
* fix(daemon): wait for a slow daemon startup and fail with typed reasons
A client killed a healthy daemon that was still starting: the local
startup used one 15 s wall-clock deadline, and a timeout never retried.
On an overloaded host that is a failed run before any work starts
(#3782, failure class #3342).
- startLocalDaemon now waits while the spawned daemon stays alive, up to
a 120 s ceiling; AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS overrides it
for the CLI, MCP, and Node client alike.
- The daemon records startup phase markers (modules-loaded,
lock-acquired, listening, registered) with timestamps in
daemon-startup-phases.jsonl beside its registration. A live daemon
that publishes no marker for 60 s is retired as stalled; one that
keeps progressing is waited for to the ceiling.
- Failure details carry reason (startup-exited, startup-stalled,
startup-ceiling), the observed phase timings, and the effective
startupTimeoutMs. Each reason has its own hint.
- help commands documents the variable and the three reasons.
* fix(daemon): give each startup its own marker file and retire it at its owner
The integration leak oracle failed because every daemon appended its
startup phase markers to one shared `daemon-startup-phases.jsonl` beside
the registration and nothing retired that file, so any completed launch
left state-dir residue the strict oracle names as a stray entry.
Ownership now follows the file name: each launch writes only to
`daemon-startup-phases-<pid>.jsonl` through
`resolveDaemonStartupPhasesPath(stateDir, pid)`. The writing daemon
removes its file once it has registered; the launching client removes it
once that process's exit is joined (in `retireStartupAttempt`, after the
exit is observed, never before); and each new registration-lock owner
sweeps the files of provably dead pids next to the shutdown-report
removal. A live pid's file is always kept, so a concurrent or replaced
launch can neither add to, hide behind, nor delete another launch's
evidence, and the client no longer needs `spawnedAtMs` to attribute
markers to its own launch.
Along the protocol boundary:
- `recordDaemonStartupMarker` takes an optional `reportFailure` callback
so producers surface a refused append on stderr (the daemon log tail
the waiting client already ships as failure evidence) instead of
swallowing it; the append stays non-fatal and a crash-truncated final
line is separated so it cannot glue two records together.
- The phase list gains real completed boundaries, `composition-loaded`
and `recovery-finished`, and no heartbeats, so silence still means a
stalled startup and progress extends the stall window exactly as far
as the work took.
- The launch-exit join drops its redundant `.catch(() => {})`: the
`runCmdDetachedMonitored` owner resolves the promise on error and
exit, so it never rejects.
- Startup code splits out of `daemon-client-lifecycle.ts` into
`daemon-client-startup.ts` (budget, wait loop, marker refresh, hints),
keeping both modules under the 1000-line tripwire; importers attach
and ensure point at the owning module directly.
Regression coverage: the budget suite now pins per-launch attribution,
including a fresh daemon producing its own markers while an earlier
launch's leftover file is swept at the new registration, and a new
protocol suite pins the per-pid paths, the dead-pid sweep, removal
idempotence, and the write-failure seam.
Refs #3782, #3791
* fix(help): state the startup failure reasons the code actually produces
The documented reasons overclaimed. `startup-ceiling` said "progressing,
not registered before the ceiling", but the ceiling is reached by a
daemon that is merely alive: a progressing one is only ever stopped by a
ceiling above the stall window, so the promise was unverifiable from the
client's evidence. `startup-stalled` named a 60 s window without saying
that `AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS` does not change it, so
raising the ceiling looked like a fix for silence. Both the CLI help and
the configuration table now describe the two reasons as what they are,
and the env var is specified as a positive integer number of
milliseconds.
Refs #3791
* docs(daemon): remove narrating module headers
The lifecycle and startup module headers only narrate each file's role,
which AGENTS.md reserves for public APIs, tool directives, and
constraints code cannot show. The split is owned by the imports and the
file names.
Fixes review thread PRRT_kwDORFGHaM6tkpef on #3791.
* test(daemon): scope fixture marker emission to the child that asked for it
The shared emit-startup-markers flag file applied to every later fixture
in the same state dir, so the stale-history cleaner inherited the
request, wrote its own live marker, and the sweep assertion raced it.
The request is now a typed per-child option bound into that child's
entry script, and readiness waits use a per-pid file so a stale
registration-held from an earlier child cannot shortcut the await.
Fixes review thread PRRT_kwDORFGHaM6tkpeV on #3791.
* fix(daemon): sweep an exited contender's own record on every terminal exit
A launch that exited busy, unproven, or lost the registration to a
contending winner returned its retry, error, or winner handoff without
joining its exact exited child, so the child's pid-named phase file sat
in the state dir until some later registration sweep. Every terminal
child outcome now joins the child first and removes only that child's
own file, the same ownership rule a retired startup follows; the
current winner's registration and record are untouched.
Regression tests write markers at the real pre-acquisition producer
seam (new emitStartupMarkerBeforeAcquisition fixture option) and pin
the residue rule on the retry, winner-attach, and unproven-exit paths:
red at the prior head, green with the fix.
Fixes review thread PRRT_kwDORFGHaM6tkpec on #3791.
* fix(daemon): let each launch claim its pid's marker record
The per-pid file names its writer, not its lifetime: if client and
daemon of an old launch both died before either cleanup ran and the
host reused the pid, the old record survived under the new writer's
name, the sweep skipped it as live, and the earliest-per-phase reader
mixed two lifetimes into one observation. The producer now claims its
own record (claimDaemonStartupPhasesFile) before its first marker, so
stale same-pid content can neither mask nor extend the fresh startup's
progress. A refused claim follows the marker contract: reported, never
fatal. The test child repeats the real producer sequence against its
own pid, which no outside test can arrange.
Addresses review finding R2-1 on #3791.
* refactor(daemon): move startup diagnostics and registration completion out of the runtime
Code ownership allows no exception: behavior was added to a module
over 1000 lines, so the module is split at its existing boundary first.
flushDaemonStartupDiagnostics, emitDaemonDiagnostic,
finishDaemonRegistration, and armDaemonMetadataLossWatch form the
daemon-log diagnostic and registration-completion edge the runtime
already treats as one unit; they move with their callers updated, no
behavior change. daemon-runtime.ts is back under the limit.
* refactor(daemon): drop the phase-timings return cast
describeStartupPhaseTimings builds a Record<string, number> that always
starts at spawned: 0; the intersection cast asserted what the literal
already showed. No caller branched on it.
Addresses review finding R2-2 on #3791.
* ci(ios): capture the cold daemon-startup timer in the preflight step
The #3782 startup-budget evidence needs one measured successful cold
startup: after clean:daemon, the prepare command's daemon_startup
diagnostic now wraps a fresh managed startup and lands in the step log.
Behavior and JSON output are unchanged.
Records the evidence obligation for #3791.
* test(daemon): give every fixture launch its own entry script
registeredDaemonFixtureArgs rewrote one shared dist/src/internal/
daemon.js per construction, so a second build could replace the script
before an earlier launched child loaded it, silently applying the wrong
per-child marker switches and registration fields. Each construction
now writes a unique fixture-entries/<uuid>/dist/src/internal/daemon.js:
the per-launch directory keeps the daemon entry path shape the
retirement identity checks require, and cleanup still removes the state
dir. A regression test builds both arg sets before launching either
child and proves child 1 executes its own switches while child 2 exits
busy: red on the shared path, green per-launch.
Fixes review thread PRRT_kwDORFGHaM6tk7ed on #3791.
* perf(daemon): keep the startup-diagnostics edge out of the eager closure
The split moved the helpers next to the runtime but left a static edge,
so both daemon entries evaluated one more module and 12-15 more
top-level statements than the merge-base (Coverage lane red: 3
eager-closure-budgets failures at OUTPUT_ECONOMY_BASE=4a8abed7b9).
startDaemonRuntime now loads them through a function-scoped await
import at its first use point, the loading shape the gate prescribes;
composition, publication, flush, and shutdown order are unchanged and
the source-order guards still read them from this module.
Red at 62a09bd (3 failures), green here: 846/846.
Addresses the eager-closure failure reported at 62a09bd on #3791.
* ci(ios): measure the cold daemon start at the public SDK ensure seam
The prepare step's daemon_startup diagnostic is not a cold-startup
measurement: the CLI's ensured transport finishes ensureDaemon before
sendToDaemon's timer wraps the attach connect, so the timer only sees
an attach. Time the public ensureDaemon() directly after clean:daemon
and record phase, durationMs, started, and pid, so the record counts
as cold only when started is true. The prepare command keeps its
original semantics and attaches to the same daemon.
Corrects the measurement assumption behind 62a09bd on #3791; the
cold-startup CI evidence for #3782 is still missing until a run
reports this record with started:true.
* fix(daemon): report startup-ceiling when any live startup is pending at the deadline
When the launched contender exited busy while another live daemon held
the registration and deferred publication, the wait loop ended in an
unlabeled 'timeout' result that mapped to startup-exited with the
generic hint. Any deadline with a live startup still pending, this
client's launch or a verified live lock owner, now reports
startup-ceiling with the effective ceiling and the overload hint;
startup-exited stays reserved for a real exit or no live startup.
Remove the 'timeout' wait kind so the unlabeled state cannot exist and
add the sendToDaemon-boundary regressions for both outcomes.
Also compress this PR's narration and duplicate test setup to keep the
gross diff within one coherent change, and add the Unreleased changelog
entry for the startup budget, the stall window, and the three reasons.
* fix(daemon): classify the expired startup wait by its observed exit and pid identity
At an expired deadline, a delivered child exit is authoritative for the
lifetime it watched, so it outranks an unobserved liveness probe of a pid
that may have been reused. Unobserved, the launch's recorded birth time
must prove the pid changed hands before it counts as a live startup; an
unavailable identity stays fail-closed. Extracted classifyExpiredDaemonStartup
keeps the wait loop within its complexity thresholds. The budget test pins
liveness of its fabricated pids at the process seam instead of trusting the
host, and the winner-sweep race coverage is deduplicated into the joined
contender test.
* docs(daemon): scope the startup ceiling env to daemon starts and restore the marker sweep owner
Both facts were stated incompletely against the code:
- resolveDaemonStartupTimeoutMs runs only inside startLocalDaemon, so
AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS is validated only when a client
must start a daemon. A client that attaches to or reuses a running
daemon never reads the variable. CHANGELOG, cli-help, and the
configuration guide now say so; the guide already named INVALID_ARGS.
- The startup-marker ownership comment dropped the registration owner's
sweep of provably dead pids, which sweepDeadDaemonStartupPhasesFiles
still performs at each new registration lock acquisition.
Addresses review threads r4305110030 and r4305299946 on #3791.
Co-authored-by: Claude Opus 4.8 <[email protected]>
* test(daemon): point the host test's observe mock at the startup module
The startup split moved observeLocalDaemon out of daemon-client-lifecycle.ts,
so the host test's lifecycle-module mock no longer stopped the ensure path
and the CLI-run case fell through to the real observation. Mock the seam the
subject imports it from now. Drop two narration comments the assertions
already state.
Co-authored-by: Claude Opus 4.8 <[email protected]>
* refactor(daemon): give startup phase records their own contracts module
The per-pid startup phase file has its own record format, writer, reader
and dead-pid sweep, so it no longer belongs inside the registration
owner the review pointed at (#3791). The declarations move verbatim into
daemon-contracts/daemon-startup-phases.ts beside daemon-shutdown-report
and the suffix-named markers test follows them as
daemon-startup-phases.test.ts. Registration acquisition imports only
sweepDeadDaemonStartupPhasesFiles; the runtime, the daemon entry, the
client wait and the fixture import the record owner directly, with no
re-export shim (unreleased surface).
The daemon entry must write its earliest modules-loaded marker at module
load, so the eager-closure gate gets one APPROVED_CLOSURE_ADDITIONS row
naming the new module; the entry pays for the record it writes first.
Addresses review 6248815887 item 1 on #3791.
Refs #3782, #3791
* refactor(daemon): point every startup-marker consumer at the new owner
Importers attach to daemon-contracts/daemon-startup-phases directly;
daemon-registration-owner keeps only the sweep call at acquisition.
src/daemon.ts captures its host environment once and resolves the phases
path from the same env and cwd startDaemonRuntime resolves its state dir
from, so a relative AGENT_DEVICE_STATE_DIR cannot land the earliest
markers beside a different directory than the registration (the module
top-level second resolveDaemonPaths(process.env) is gone), and the
modules-loaded marker stays the first thing recorded.
Addresses review 6248815887 items 1 and 2 on #3791.
Refs #3782, #3791
* test(daemon): prove the progress-aware budget on successive phases
The headline slow-start test rewrote lock-acquired repeatedly; the
reader keeps each phase's earliest timestamp, so those writes never
advanced progress and the run spanned only 20 s against the 60 s stall
window. It now schedules four distinct phases (modules-loaded,
composition-loaded, lock-acquired, recovery-finished) across 70 s of
virtual time under a 150 s ceiling and asserts the wait survived on
observed successive progress. With the marker read removed, the same
scenario fails as startup-stalled at the 60 s window (verified with a
throwaway stub, not committed), so the test is credibly red without the
progress rule.
Addresses review 6248815887 item 3 on #3791.
Refs #3782, #3791
* refactor(daemon): converge the marker helpers and shared fixture waits
* fix(daemon): scope the startup-ceiling claim to the owned launch and prove the default ceiling
The help and configuration docs claimed only a progressing daemon reaches a
ceiling above the stall window. On the contended path this client's launch
exits busy and the wait runs on the exit branch, where the stall check never
runs, so a live startup owned by another client reaches the ceiling with no
observed progress. Qualify the clause to this client's own launch, matching
the CHANGELOG definition.
The no-marker budget test stubbed 120000 instead of unsetting the variable,
so it never proved the default. Unset it (stubEnv with undefined) so the
resolved ceiling is the unoverridden default even on a host that exports the
variable. Also drop a stray blank line in the Unreleased list.
---------
Co-authored-by: Claude Opus 4.8 <[email protected]>1 parent 63f762e commit 72e7965
22 files changed
Lines changed: 1827 additions & 673 deletions
File tree
- .github/workflows
- packages
- daemon-contracts
- src
- daemon/src/server
- __tests__
- http
- scripts/__tests__
- src
- __tests__/test-utils
- commands/schema
- daemon-client
- __tests__
- website/docs/docs
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
169 | 169 | | |
170 | 170 | | |
171 | 171 | | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
172 | 180 | | |
173 | 181 | | |
174 | 182 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| 7 | + | |
7 | 8 | | |
8 | 9 | | |
9 | 10 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
44 | 44 | | |
45 | 45 | | |
46 | 46 | | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
47 | 51 | | |
48 | 52 | | |
49 | 53 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
44 | 44 | | |
45 | 45 | | |
46 | 46 | | |
| 47 | + | |
47 | 48 | | |
48 | 49 | | |
49 | 50 | | |
| |||
85 | 86 | | |
86 | 87 | | |
87 | 88 | | |
| 89 | + | |
88 | 90 | | |
89 | 91 | | |
90 | 92 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
13 | | - | |
| 13 | + | |
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
| |||
0 commit comments