Skip to content

Commit 00b82dc

Browse files
thymikeeclaude
andcommitted
fix(daemon): let a retained close answer every later stop of its instance
A close that keeps its registration left the state dir claimed with no record of why, and the record it did write stopped answering after 60 s. A stop that arrived later found a claimed dir, a closed listener, and no evidence: it sat out its whole budget and failed with an unconfirmed shutdown at a runtime that had already stopped serving. - Every close that leaves the registration in place publishes its retention record, whether or not a stop asked for it. The ledger answers a request; this is a fact about the state dir, owed to whoever reads it next. `finish()` states it for the case where the removal itself failed. - A `retained` record answers by lifetime, not by age: the state it reports holds until a new acquisition clears the file, so the age bound added nothing beyond the pid and start-time fence. A `released` record keeps its window, because its job is to answer the stop that asked. - The wait proves the instance is still the process the record names before trusting an aged record. A dead host's leftover file is not a registration someone holds, and its honest answer is that nothing runs. - Only a runtime a host owns publishes these records. A detached daemon's stop signals its process and reads the shutdown report, so a completion record there would state a fact nobody reads. Co-Authored-By: Claude <[email protected]>
1 parent 1c2e917 commit 00b82dc

9 files changed

Lines changed: 291 additions & 35 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
## Unreleased
44

5-
- A daemon a host process starts for itself now records that owner, and a stop honors it. `daemon status` names such a record `host-owned`, or `owner-not-recognized` for one whose owner this version cannot read. `daemon stop --state-dir <path>` reaches such a runtime over its registered local transport with an authenticated shutdown request and waits for it to report its own close, so nothing outside the host signals it; what that close could not finish is reported as warnings. A record whose owner this version cannot recognize gets the same request, because the running process answers for itself. Stop fails with `details.reason: 'daemon-embedded-stop-unreachable'` when a host-owned record's request cannot be delivered, and `'daemon-shutdown-unconfirmed'` when the runtime agreed and never reported its close. A `daemon.json` that exists but cannot be read fails the stop with `'daemon-record-unreadable'` instead of answering that nothing runs: the record may describe a daemon that is serving, so nothing is signaled and nothing is deleted. `--clean` is refused with `'daemon-clean-not-detached'` for any record that does not prove the process is a daemon of its own, because that cleanup walks the owner's whole process tree; stopping without `--clean` still works. A `--clean` of a detached daemon now follows the lifetime that same read proved.
5+
- A daemon a host process starts for itself now records that owner, and a stop honors it. `daemon status` names such a record `host-owned`, or `owner-not-recognized` for one whose owner this version cannot read. `daemon stop --state-dir <path>` reaches such a runtime over its registered local transport with an authenticated shutdown request and waits for it to report its own close, so nothing outside the host signals it; what that close could not finish is reported as warnings. A record whose owner this version cannot recognize gets the same request, because the running process answers for itself. Stop fails with `details.reason: 'daemon-embedded-stop-unreachable'` when a host-owned record's request cannot be delivered, and `'daemon-shutdown-unconfirmed'` when the runtime agreed and never reported its close. A runtime that closed while it still held work reports what it kept, and a later stop reads that answer whenever the same host still holds the registration, however long ago the close happened. A `daemon.json` that exists but cannot be read fails the stop with `'daemon-record-unreadable'` instead of answering that nothing runs: the record may describe a daemon that is serving, so nothing is signaled and nothing is deleted. `--clean` is refused with `'daemon-clean-not-detached'` for any record that does not prove the process is a daemon of its own, because that cleanup walks the owner's whole process tree; stopping without `--clean` still works. A `--clean` of a detached daemon now follows the lifetime that same read proved.
66
- The PNG worker no longer calls `process.emitWarning` when it is unavailable, and it starts with an empty environment (`env: {}`). The daemon registers a sink when it pre-warms the worker, so every transition to unavailable (spawn failure, missing entry, load error, exit) leaves exactly one `png_worker_unavailable` event, with a typed `reason` (`entry-not-found`, `spawn-failed`, `worker-error`, `worker-exited`) and its `detail`, in `daemon.log`, with or without `--debug`. Requests then continue on the in-process fallback without repeating the event.
77
- Behavior change: `resolveHubAppReference`, shipped through `agent-device/plugins/webdriver`, now needs `cwd` to resolve a relative app path. Without it the call fails with `INVALID_ARGS` and `details.reason: provider_app_cwd_missing`; it used to resolve against the process's working directory. Absolute paths, URLs and provider references are unaffected. The daemon fills it from the client's working directory (or its host's) for every provider lease request, so `connect` and `lease_allocate` are unchanged.
88
- The `daemon-not-running` hint now always names `--state-dir <dir>`, and a daemon force-retired for outlasting its drain is reported as `Replaced unreachable daemon` instead of `Replaced idle daemon`. `daemon-drain-timeout` and the post-drain force retire are documented in the client API and `help`.

‎packages/daemon-contracts/src/daemon-registration-owner.ts‎

Lines changed: 21 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -84,10 +84,15 @@ export type DaemonRegistrationOwner = Readonly<{
8484
* it to leave — all under the claim this instance holds, and before it is released. The answer's
8585
* disposition is what this call proves happened, never what the caller expected: a registration the
8686
* unlink could not remove, or a step that threw before it, says `retained`.
87+
*
88+
* `unfinished` states what this close left outstanding. It matters when `completion` is absent, which
89+
* is a close no stop asked for: the removal can still fail, and a state dir that stays claimed gets
90+
* its retention record whatever the ledger says, because that record is what the next stop reads.
8791
*/
8892
finish(
8993
outcome?: DaemonShutdownOutcome,
9094
completion?: DaemonShutdownStopAnswer,
95+
unfinished?: readonly string[],
9196
): Promise<DaemonRegistrationRemoval>;
9297
}>;
9398

@@ -150,33 +155,42 @@ export async function tryAcquireDaemonRegistration(
150155
acquisition,
151156
);
152157
},
153-
async finish(outcome?: DaemonShutdownOutcome, completion?: DaemonShutdownStopAnswer) {
158+
async finish(
159+
outcome?: DaemonShutdownOutcome,
160+
completion?: DaemonShutdownStopAnswer,
161+
unfinished?: readonly string[],
162+
) {
163+
// What this close reports as its work in hand. A call that is not a close — startup rollback —
164+
// states nothing and so publishes no record: it never had a runtime that could keep serving.
165+
const closeReport = completion ? completion.unfinished : unfinished;
154166
let removal: DaemonRegistrationRemoval;
155167
try {
156168
if (outcome) writeShutdownReport(boundPaths.baseDir, outcome, acquisition);
157169
removal = removeRegistrationUnderLock(boundPaths.infoPath, identity, acquisition);
158170
} catch (error) {
159171
// A release that never ran kept the registration, which is the disposition a stop is owed
160-
// before this claim goes away and this process can say nothing more.
161-
if (completion) {
172+
// before this claim goes away and this process can say nothing more. It is owed to whoever
173+
// reads that state dir next, not only to a stop that asked.
174+
if (closeReport) {
162175
writeShutdownCompletion(
163176
boundPaths.baseDir,
164177
identity,
165178
'retained',
166-
completion.unfinished,
179+
closeReport,
167180
acquisition,
168181
);
169182
}
170183
return await releaseRegistrationAfterFailure(acquisition, error, boundPaths.logPath);
171184
}
172185
// The disposition comes from the removal this call just proved, not from what the caller
173186
// hoped: a registration an unlink could not remove is `retained` whatever the close planned.
174-
if (completion) {
187+
const released = removal.state === 'removed' || removal.state === 'absent';
188+
if (closeReport && (completion || !released)) {
175189
writeShutdownCompletion(
176190
boundPaths.baseDir,
177191
identity,
178-
removal.state === 'removed' || removal.state === 'absent' ? 'released' : 'retained',
179-
completion.unfinished,
192+
released ? 'released' : 'retained',
193+
closeReport,
180194
acquisition,
181195
);
182196
}

‎packages/daemon-contracts/src/daemon-shutdown-report-completion.test.ts‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,10 @@ import {
55
DAEMON_SHUTDOWN_COMPLETION_MAX_AGE_MS,
66
buildDaemonShutdownCompletion,
77
daemonShutdownCompletionAnswers,
8+
daemonShutdownCompletionStillRetained,
89
readDaemonShutdownCompletion,
910
resolveDaemonShutdownCompletionPath,
11+
type DaemonShutdownCompletion,
1012
} from './daemon-shutdown-report.ts';
1113

1214
/** The instance a record in these rows describes: some other runtime, never this test process. */
@@ -140,3 +142,40 @@ describe('daemonShutdownCompletionAnswers', () => {
140142
}
141143
});
142144
});
145+
146+
// A retained close keeps its registration, so its record reports a state that holds rather than an answer
147+
// to a request that is being answered now. These rows say which records hold and which expire.
148+
describe('daemonShutdownCompletionStillRetained', () => {
149+
const aged = (record: DaemonShutdownCompletion) => ({
150+
...record,
151+
writtenAtMs: Date.now() - DAEMON_SHUTDOWN_COMPLETION_MAX_AGE_MS - 1000,
152+
});
153+
154+
test('a retained record older than the window still describes the state directory', () => {
155+
// The stop that arrives a day later reads the same truth the first one did: this instance closed and
156+
// kept the registration. Nothing clears the file but a new acquisition, which is its own event.
157+
const retained = aged(
158+
buildDaemonShutdownCompletion(WRITER, 'retained', { unfinished: ['1 lease'] }),
159+
);
160+
expect(daemonShutdownCompletionAnswers(retained, WRITER)).toBe(false);
161+
expect(daemonShutdownCompletionStillRetained(retained, WRITER)).toBe(true);
162+
});
163+
164+
test('a released record older than the window describes no state that holds', () => {
165+
// A close that released its registration left nothing behind to report on, so only the stop that
166+
// asked reads that record, and only while its answer is current.
167+
const released = aged(buildDaemonShutdownCompletion(WRITER, 'released', { unfinished: [] }));
168+
expect(daemonShutdownCompletionStillRetained(released, WRITER)).toBe(false);
169+
});
170+
171+
test("a retained record does not describe another instance's state directory", () => {
172+
const retained = buildDaemonShutdownCompletion(WRITER, 'retained', { unfinished: [] });
173+
expect(daemonShutdownCompletionStillRetained(retained, { ...WRITER, pid: 9090 })).toBe(false);
174+
expect(
175+
daemonShutdownCompletionStillRetained(retained, {
176+
...WRITER,
177+
processStartTime: 'Sun Oct 18 10:00:00 2026',
178+
}),
179+
).toBe(false);
180+
});
181+
});

‎packages/daemon-contracts/src/daemon-shutdown-report.ts‎

Lines changed: 35 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -219,8 +219,10 @@ export type DaemonShutdownAnswerLedger = {
219219
};
220220

221221
/**
222-
* Keeps the promise a runtime makes to the stop it accepted. A close that was never asked to leave
223-
* answers nothing: its owner watches the process and needs no record.
222+
* Keeps the promise a runtime makes to the stop it accepted: a close that was never asked to leave owes
223+
* no answer to anybody, because its owner watches the process. A close that keeps the state directory
224+
* is different, and is not the ledger's business — it publishes its record whether or not a stop asked,
225+
* because that directory is what the next reader will find.
224226
*/
225227
export function createDaemonShutdownAnswerLedger(): DaemonShutdownAnswerLedger {
226228
let accepted = false;
@@ -272,6 +274,9 @@ export function readDaemonShutdownCompletion(stateDir: string): DaemonShutdownCo
272274
* written by another instance is not evidence, so a waiter keeps waiting rather than reporting
273275
* somebody else's shutdown as this one. A record stamped ahead of now is not evidence either: a writer
274276
* whose clock runs ahead must not get to make an old answer look fresh.
277+
*
278+
* This bounds a close that was asked for and released, which is what the stop is waiting for. A close
279+
* that kept its registration is not time-bounded — see {@link daemonShutdownCompletionStillRetained}.
275280
*/
276281
export function daemonShutdownCompletionAnswers(
277282
completion: DaemonShutdownCompletion,
@@ -283,13 +288,39 @@ export function daemonShutdownCompletionAnswers(
283288
// stale answer look fresh.
284289
const ageMs = nowMs - completion.writtenAtMs;
285290
return (
286-
completion.pid === query.pid &&
287-
completion.processStartTime === query.processStartTime &&
291+
daemonShutdownCompletionNamesInstance(completion, query) &&
288292
ageMs >= 0 &&
289293
ageMs <= DAEMON_SHUTDOWN_COMPLETION_MAX_AGE_MS
290294
);
291295
}
292296

297+
/**
298+
* Whether a `retained` record still reports `query`'s instance, whatever its age.
299+
*
300+
* A retained close keeps its registration on purpose, so its record describes a state that holds until
301+
* someone ends it: a new daemon clears the file when it acquires the state dir, and a close that does
302+
* not keep the registration answers with `released`. So the lifetime fence decides this and the age adds
303+
* nothing, unlike a `released` record, whose job is to answer the stop that asked for it.
304+
*
305+
* The caller must prove the instance is still the process the record names. A dead host's leftover
306+
* record is a stale file, not a runtime that agreed to leave.
307+
*/
308+
export function daemonShutdownCompletionStillRetained(
309+
completion: DaemonShutdownCompletion,
310+
query: DaemonShutdownCompletionQuery,
311+
): boolean {
312+
return (
313+
completion.outcome === 'retained' && daemonShutdownCompletionNamesInstance(completion, query)
314+
);
315+
}
316+
317+
function daemonShutdownCompletionNamesInstance(
318+
completion: DaemonShutdownCompletion,
319+
query: DaemonShutdownCompletionQuery,
320+
): boolean {
321+
return completion.pid === query.pid && completion.processStartTime === query.processStartTime;
322+
}
323+
293324
function isDaemonShutdownCompletion(value: unknown): value is DaemonShutdownCompletion {
294325
if (typeof value !== 'object' || value === null) return false;
295326
const record = value as Partial<DaemonShutdownCompletion>;

‎packages/daemon/src/server/http/daemon-runtime-lifecycle.test.ts‎

Lines changed: 30 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -397,7 +397,7 @@ test('a shutdown carrying no request id is accepted and still writes a record a
397397
).toBe(true);
398398
});
399399

400-
test('a host that closes its own runtime leaves no stop to answer', async () => {
400+
test('a host that closes its own runtime with nothing in hand leaves no stop to answer', async () => {
401401
const daemon = await bootDaemon({ ownership: 'embedded' });
402402
await daemon.runtime.close();
403403
// The close's own `finally` offers a failure line for the case where a step rejected. A close that
@@ -406,6 +406,35 @@ test('a host that closes its own runtime leaves no stop to answer', async () =>
406406
expect(readDaemonShutdownCompletion(resolveDaemonPaths(daemon.stateDir).baseDir)).toBe(null);
407407
});
408408

409+
test('a host that closes its own runtime with work in hand still records what it kept', async () => {
410+
// Nobody asked this runtime to leave, so the ledger owes no answer. The state dir is the point: the
411+
// close keeps its registration, and a later `daemon stop` will read that record and ask. Without this
412+
// record it waits on a listener that is already closed and reports an unconfirmed failure at a runtime
413+
// that stopped serving.
414+
const daemon = await bootDaemon({ ownership: 'embedded', requestDrainMs: 50 });
415+
const paths = resolveDaemonPaths(daemon.stateDir);
416+
let release: () => void = () => {};
417+
const entered = new Promise<void>((resolve) => {
418+
probe.entered = resolve;
419+
});
420+
probe.held = new Promise<void>((resolve) => {
421+
release = resolve;
422+
});
423+
daemon.sessionList().catch(() => undefined);
424+
await entered;
425+
426+
try {
427+
await daemon.runtime.close();
428+
429+
const completion = readDaemonShutdownCompletion(paths.baseDir);
430+
expect(completion).toMatchObject({ pid: process.pid, outcome: 'retained' });
431+
expect(completion?.unfinished.join('\n')).toContain('ignored cancellation');
432+
expect(fs.existsSync(paths.infoPath)).toBe(true);
433+
} finally {
434+
release();
435+
}
436+
});
437+
409438
test('a close whose step rejects still answers the stop that asked for it', async () => {
410439
const stateDir = mkdtempForTestSync('agent-device-daemon-lifecycle-close-failure-');
411440
const paths = resolveDaemonPaths(stateDir);

‎packages/daemon/src/server/http/daemon-runtime.ts‎

Lines changed: 26 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -292,9 +292,14 @@ async function finishDaemonRegistration(params: {
292292
logPath: string;
293293
outcome?: Parameters<DaemonRegistrationOwner['finish']>[0];
294294
completion?: Parameters<DaemonRegistrationOwner['finish']>[1];
295+
unfinished?: Parameters<DaemonRegistrationOwner['finish']>[2];
295296
}): Promise<void> {
296297
try {
297-
const removal = await params.registration.finish(params.outcome, params.completion);
298+
const removal = await params.registration.finish(
299+
params.outcome,
300+
params.completion,
301+
params.unfinished,
302+
);
298303
if (removal.state !== 'removed' && removal.state !== 'absent') {
299304
await emitDaemonDiagnostic(params.logPath, 'daemon_info_removal_declined', {
300305
infoPath: params.infoPath,
@@ -316,13 +321,19 @@ async function finishDaemonRegistration(params: {
316321
* the removal that call proves, and nothing that reports the close can move past it.
317322
*
318323
* Work that ignored cancellation keeps both holds: releasing them would let a second daemon drive the
319-
* same devices. That stop is answered with a retention record instead.
324+
* same devices. Such a close publishes a retention record, and does it once — a close reports its work in
325+
* hand the first time, which is the specific truth, rather than once per call.
326+
*
327+
* Only a runtime a host owns publishes these records. Stopping a detached daemon signals its process and
328+
* reads the shutdown report that close leaves, so a completion record would state a fact nobody reads;
329+
* `daemon-shutdown-completion.json` means "a host-owned runtime's close", and this is where that holds.
320330
*/
321331
function createDaemonShutdownRelease(params: {
322332
registration: DaemonRegistrationOwner;
323333
infoPath: string;
324334
logPath: string;
325335
answers: DaemonShutdownAnswerLedger;
336+
ownership: DaemonRuntimeOwnership;
326337
clearLockConfiguration: () => Promise<void>;
327338
}): {
328339
release(params: {
@@ -331,15 +342,24 @@ function createDaemonShutdownRelease(params: {
331342
outcome?: Parameters<DaemonRegistrationOwner['finish']>[0];
332343
}): Promise<void>;
333344
} {
345+
// A stop of this runtime reads its state dir, so a close that keeps that dir claimed owes it a record.
346+
const recordsCloses = params.ownership !== 'detached';
334347
let registrationFinished = false;
335348
let lockConfigurationCleared = false;
349+
let workInHandReported = false;
336350
return {
337351
release: async (releaseParams) => {
338-
const answer = params.answers.take(releaseParams.unfinished);
339352
if (releaseParams.holdsWorkInHand) {
340-
if (answer) params.registration.recordShutdownRetention(answer.unfinished);
353+
// A close that keeps the registration leaves the state dir claimed, which is exactly what the
354+
// next stop will find and ask about. So it publishes the record whether or not a stop asked for
355+
// one: the ledger answers a request, and this is a fact about the state dir. Without it, a stop
356+
// of a runtime its host already closed finds no answer and waits on a listener that is gone.
357+
if (workInHandReported) return;
358+
workInHandReported = true;
359+
if (recordsCloses) params.registration.recordShutdownRetention(releaseParams.unfinished);
341360
return;
342361
}
362+
const answer = params.answers.take(releaseParams.unfinished);
343363
if (!registrationFinished) {
344364
registrationFinished = true;
345365
await finishDaemonRegistration({
@@ -348,6 +368,7 @@ function createDaemonShutdownRelease(params: {
348368
logPath: params.logPath,
349369
outcome: releaseParams.outcome,
350370
completion: answer,
371+
unfinished: recordsCloses ? releaseParams.unfinished : undefined,
351372
});
352373
}
353374
if (lockConfigurationCleared) return;
@@ -871,6 +892,7 @@ async function startDaemonRuntimeOnHost(
871892
infoPath,
872893
logPath,
873894
answers: shutdownAnswers,
895+
ownership,
874896
clearLockConfiguration: () => deps.ownerLifecycle.clearDaemonLockConfiguration(),
875897
});
876898
const drainInFlightRequests = async (): Promise<ShutdownReport['requests']> => {

0 commit comments

Comments
 (0)