Skip to content

Commit 6fce5ec

Browse files
committed
fix(daemon): let a retained close answer every later stop of its instance
A close that keeps its registration left the state dir claimed with no record of why, and the record it did write stopped answering after 60 s. A stop that arrived later found a claimed dir, a closed listener, and no evidence: it sat out its whole budget and failed with an unconfirmed shutdown at a runtime that had already stopped serving. - Every close that leaves the registration in place publishes its retention record, whether or not a stop asked for it. The ledger answers a request; this is a fact about the state dir, owed to whoever reads it next. `finish()` states it for the case where the removal itself failed. - A `retained` record answers by lifetime, not by age: the state it reports holds until a new acquisition clears the file, so the age bound added nothing beyond the pid and start-time fence. A `released` record keeps its window, because its job is to answer the stop that asked. - The wait proves the instance is still the process the record names before trusting an aged record. A dead host's leftover file is not a registration someone holds, and its honest answer is that nothing runs. - Only a runtime a host owns publishes these records. A detached daemon's stop signals its process and reads the shutdown report, so a completion record there would state a fact nobody reads. Co-Authored-By: Claude <[email protected]>
1 parent 2ddf3d2 commit 6fce5ec

9 files changed

Lines changed: 291 additions & 34 deletions

File tree

‎packages/daemon-contracts/src/daemon-registration-owner.ts‎

Lines changed: 21 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -85,10 +85,15 @@ export type DaemonRegistrationOwner = Readonly<{
8585
* it to leave — all under the claim this instance holds, and before it is released. The answer's
8686
* disposition is what this call proves happened, never what the caller expected: a registration the
8787
* unlink could not remove, or a step that threw before it, says `retained`.
88+
*
89+
* `unfinished` states what this close left outstanding. It matters when `completion` is absent, which
90+
* is a close no stop asked for: the removal can still fail, and a state dir that stays claimed gets
91+
* its retention record whatever the ledger says, because that record is what the next stop reads.
8892
*/
8993
finish(
9094
outcome?: DaemonShutdownOutcome,
9195
completion?: DaemonShutdownStopAnswer,
96+
unfinished?: readonly string[],
9297
): Promise<DaemonRegistrationRemoval>;
9398
}>;
9499

@@ -152,33 +157,42 @@ export async function tryAcquireDaemonRegistration(
152157
acquisition,
153158
);
154159
},
155-
async finish(outcome?: DaemonShutdownOutcome, completion?: DaemonShutdownStopAnswer) {
160+
async finish(
161+
outcome?: DaemonShutdownOutcome,
162+
completion?: DaemonShutdownStopAnswer,
163+
unfinished?: readonly string[],
164+
) {
165+
// What this close reports as its work in hand. A call that is not a close — startup rollback —
166+
// states nothing and so publishes no record: it never had a runtime that could keep serving.
167+
const closeReport = completion ? completion.unfinished : unfinished;
156168
let removal: DaemonRegistrationRemoval;
157169
try {
158170
if (outcome) writeShutdownReport(boundPaths.baseDir, outcome, acquisition);
159171
removal = removeRegistrationUnderLock(boundPaths.infoPath, identity, acquisition);
160172
} catch (error) {
161173
// A release that never ran kept the registration, which is the disposition a stop is owed
162-
// before this claim goes away and this process can say nothing more.
163-
if (completion) {
174+
// before this claim goes away and this process can say nothing more. It is owed to whoever
175+
// reads that state dir next, not only to a stop that asked.
176+
if (closeReport) {
164177
writeShutdownCompletion(
165178
boundPaths.baseDir,
166179
identity,
167180
'retained',
168-
completion.unfinished,
181+
closeReport,
169182
acquisition,
170183
);
171184
}
172185
return await releaseRegistrationAfterFailure(acquisition, error, boundPaths.logPath);
173186
}
174187
// The disposition comes from the removal this call just proved, not from what the caller
175188
// hoped: a registration an unlink could not remove is `retained` whatever the close planned.
176-
if (completion) {
189+
const released = removal.state === 'removed' || removal.state === 'absent';
190+
if (closeReport && (completion || !released)) {
177191
writeShutdownCompletion(
178192
boundPaths.baseDir,
179193
identity,
180-
removal.state === 'removed' || removal.state === 'absent' ? 'released' : 'retained',
181-
completion.unfinished,
194+
released ? 'released' : 'retained',
195+
closeReport,
182196
acquisition,
183197
);
184198
}

‎packages/daemon-contracts/src/daemon-shutdown-report-completion.test.ts‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,10 @@ import {
55
DAEMON_SHUTDOWN_COMPLETION_MAX_AGE_MS,
66
buildDaemonShutdownCompletion,
77
daemonShutdownCompletionAnswers,
8+
daemonShutdownCompletionStillRetained,
89
readDaemonShutdownCompletion,
910
resolveDaemonShutdownCompletionPath,
11+
type DaemonShutdownCompletion,
1012
} from './daemon-shutdown-report.ts';
1113

1214
/** The instance a record in these rows describes: some other runtime, never this test process. */
@@ -140,3 +142,40 @@ describe('daemonShutdownCompletionAnswers', () => {
140142
}
141143
});
142144
});
145+
146+
// A retained close keeps its registration, so its record reports a state that holds rather than an answer
147+
// to a request that is being answered now. These rows say which records hold and which expire.
148+
describe('daemonShutdownCompletionStillRetained', () => {
149+
const aged = (record: DaemonShutdownCompletion) => ({
150+
...record,
151+
writtenAtMs: Date.now() - DAEMON_SHUTDOWN_COMPLETION_MAX_AGE_MS - 1000,
152+
});
153+
154+
test('a retained record older than the window still describes the state directory', () => {
155+
// The stop that arrives a day later reads the same truth the first one did: this instance closed and
156+
// kept the registration. Nothing clears the file but a new acquisition, which is its own event.
157+
const retained = aged(
158+
buildDaemonShutdownCompletion(WRITER, 'retained', { unfinished: ['1 lease'] }),
159+
);
160+
expect(daemonShutdownCompletionAnswers(retained, WRITER)).toBe(false);
161+
expect(daemonShutdownCompletionStillRetained(retained, WRITER)).toBe(true);
162+
});
163+
164+
test('a released record older than the window describes no state that holds', () => {
165+
// A close that released its registration left nothing behind to report on, so only the stop that
166+
// asked reads that record, and only while its answer is current.
167+
const released = aged(buildDaemonShutdownCompletion(WRITER, 'released', { unfinished: [] }));
168+
expect(daemonShutdownCompletionStillRetained(released, WRITER)).toBe(false);
169+
});
170+
171+
test("a retained record does not describe another instance's state directory", () => {
172+
const retained = buildDaemonShutdownCompletion(WRITER, 'retained', { unfinished: [] });
173+
expect(daemonShutdownCompletionStillRetained(retained, { ...WRITER, pid: 9090 })).toBe(false);
174+
expect(
175+
daemonShutdownCompletionStillRetained(retained, {
176+
...WRITER,
177+
processStartTime: 'Sun Oct 18 10:00:00 2026',
178+
}),
179+
).toBe(false);
180+
});
181+
});

‎packages/daemon-contracts/src/daemon-shutdown-report.ts‎

Lines changed: 35 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -219,8 +219,10 @@ export type DaemonShutdownAnswerLedger = {
219219
};
220220

221221
/**
222-
* Keeps the promise a runtime makes to the stop it accepted. A close that was never asked to leave
223-
* answers nothing: its owner watches the process and needs no record.
222+
* Keeps the promise a runtime makes to the stop it accepted: a close that was never asked to leave owes
223+
* no answer to anybody, because its owner watches the process. A close that keeps the state directory
224+
* is different, and is not the ledger's business — it publishes its record whether or not a stop asked,
225+
* because that directory is what the next reader will find.
224226
*/
225227
export function createDaemonShutdownAnswerLedger(): DaemonShutdownAnswerLedger {
226228
let accepted = false;
@@ -272,6 +274,9 @@ export function readDaemonShutdownCompletion(stateDir: string): DaemonShutdownCo
272274
* written by another instance is not evidence, so a waiter keeps waiting rather than reporting
273275
* somebody else's shutdown as this one. A record stamped ahead of now is not evidence either: a writer
274276
* whose clock runs ahead must not get to make an old answer look fresh.
277+
*
278+
* This bounds a close that was asked for and released, which is what the stop is waiting for. A close
279+
* that kept its registration is not time-bounded — see {@link daemonShutdownCompletionStillRetained}.
275280
*/
276281
export function daemonShutdownCompletionAnswers(
277282
completion: DaemonShutdownCompletion,
@@ -283,13 +288,39 @@ export function daemonShutdownCompletionAnswers(
283288
// stale answer look fresh.
284289
const ageMs = nowMs - completion.writtenAtMs;
285290
return (
286-
completion.pid === query.pid &&
287-
completion.processStartTime === query.processStartTime &&
291+
daemonShutdownCompletionNamesInstance(completion, query) &&
288292
ageMs >= 0 &&
289293
ageMs <= DAEMON_SHUTDOWN_COMPLETION_MAX_AGE_MS
290294
);
291295
}
292296

297+
/**
298+
* Whether a `retained` record still reports `query`'s instance, whatever its age.
299+
*
300+
* A retained close keeps its registration on purpose, so its record describes a state that holds until
301+
* someone ends it: a new daemon clears the file when it acquires the state dir, and a close that does
302+
* not keep the registration answers with `released`. So the lifetime fence decides this and the age adds
303+
* nothing, unlike a `released` record, whose job is to answer the stop that asked for it.
304+
*
305+
* The caller must prove the instance is still the process the record names. A dead host's leftover
306+
* record is a stale file, not a runtime that agreed to leave.
307+
*/
308+
export function daemonShutdownCompletionStillRetained(
309+
completion: DaemonShutdownCompletion,
310+
query: DaemonShutdownCompletionQuery,
311+
): boolean {
312+
return (
313+
completion.outcome === 'retained' && daemonShutdownCompletionNamesInstance(completion, query)
314+
);
315+
}
316+
317+
function daemonShutdownCompletionNamesInstance(
318+
completion: DaemonShutdownCompletion,
319+
query: DaemonShutdownCompletionQuery,
320+
): boolean {
321+
return completion.pid === query.pid && completion.processStartTime === query.processStartTime;
322+
}
323+
293324
function isDaemonShutdownCompletion(value: unknown): value is DaemonShutdownCompletion {
294325
if (typeof value !== 'object' || value === null) return false;
295326
const record = value as Partial<DaemonShutdownCompletion>;

‎packages/daemon/src/server/http/daemon-runtime-lifecycle.test.ts‎

Lines changed: 30 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -397,7 +397,7 @@ test('a shutdown carrying no request id is accepted and still writes a record a
397397
).toBe(true);
398398
});
399399

400-
test('a host that closes its own runtime leaves no stop to answer', async () => {
400+
test('a host that closes its own runtime with nothing in hand leaves no stop to answer', async () => {
401401
const daemon = await bootDaemon({ ownership: 'embedded' });
402402
await daemon.runtime.close();
403403
// The close's own `finally` offers a failure line for the case where a step rejected. A close that
@@ -406,6 +406,35 @@ test('a host that closes its own runtime leaves no stop to answer', async () =>
406406
expect(readDaemonShutdownCompletion(resolveDaemonPaths(daemon.stateDir).baseDir)).toBe(null);
407407
});
408408

409+
test('a host that closes its own runtime with work in hand still records what it kept', async () => {
410+
// Nobody asked this runtime to leave, so the ledger owes no answer. The state dir is the point: the
411+
// close keeps its registration, and a later `daemon stop` will read that record and ask. Without this
412+
// record it waits on a listener that is already closed and reports an unconfirmed failure at a runtime
413+
// that stopped serving.
414+
const daemon = await bootDaemon({ ownership: 'embedded', requestDrainMs: 50 });
415+
const paths = resolveDaemonPaths(daemon.stateDir);
416+
let release: () => void = () => {};
417+
const entered = new Promise<void>((resolve) => {
418+
probe.entered = resolve;
419+
});
420+
probe.held = new Promise<void>((resolve) => {
421+
release = resolve;
422+
});
423+
daemon.sessionList().catch(() => undefined);
424+
await entered;
425+
426+
try {
427+
await daemon.runtime.close();
428+
429+
const completion = readDaemonShutdownCompletion(paths.baseDir);
430+
expect(completion).toMatchObject({ pid: process.pid, outcome: 'retained' });
431+
expect(completion?.unfinished.join('\n')).toContain('ignored cancellation');
432+
expect(fs.existsSync(paths.infoPath)).toBe(true);
433+
} finally {
434+
release();
435+
}
436+
});
437+
409438
test('a close whose step rejects still answers the stop that asked for it', async () => {
410439
const stateDir = mkdtempForTestSync('agent-device-daemon-lifecycle-close-failure-');
411440
const paths = resolveDaemonPaths(stateDir);

‎packages/daemon/src/server/http/daemon-runtime.ts‎

Lines changed: 21 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -268,7 +268,12 @@ export function daemonStartupExitCode(error: unknown): number {
268268
* nothing that module does not already own.
269269
*
270270
* Work that ignored cancellation keeps both holds: releasing them would let a second daemon drive the
271-
* same devices. That stop is answered with a retention record instead.
271+
* same devices. Such a close publishes a retention record, and does it once — a close reports its work in
272+
* hand the first time, which is the specific truth, rather than once per call.
273+
*
274+
* Only a runtime a host owns publishes these records. Stopping a detached daemon signals its process and
275+
* reads the shutdown report that close leaves, so a completion record would state a fact nobody reads;
276+
* `daemon-shutdown-completion.json` means "a host-owned runtime's close", and this is where that holds.
272277
*/
273278
function createDaemonShutdownRelease(params: {
274279
registration: DaemonRegistrationOwner;
@@ -281,7 +286,9 @@ function createDaemonShutdownRelease(params: {
281286
logPath: string;
282287
outcome?: Parameters<DaemonRegistrationOwner['finish']>[0];
283288
completion?: Parameters<DaemonRegistrationOwner['finish']>[1];
289+
unfinished?: Parameters<DaemonRegistrationOwner['finish']>[2];
284290
}) => Promise<void>;
291+
ownership: DaemonRuntimeOwnership;
285292
clearLockConfiguration: () => Promise<void>;
286293
}): {
287294
release(params: {
@@ -290,15 +297,24 @@ function createDaemonShutdownRelease(params: {
290297
outcome?: Parameters<DaemonRegistrationOwner['finish']>[0];
291298
}): Promise<void>;
292299
} {
300+
// A stop of this runtime reads its state dir, so a close that keeps that dir claimed owes it a record.
301+
const recordsCloses = params.ownership !== 'detached';
293302
let registrationFinished = false;
294303
let lockConfigurationCleared = false;
304+
let workInHandReported = false;
295305
return {
296306
release: async (releaseParams) => {
297-
const answer = params.answers.take(releaseParams.unfinished);
298307
if (releaseParams.holdsWorkInHand) {
299-
if (answer) params.registration.recordShutdownRetention(answer.unfinished);
308+
// A close that keeps the registration leaves the state dir claimed, which is exactly what the
309+
// next stop will find and ask about. So it publishes the record whether or not a stop asked for
310+
// one: the ledger answers a request, and this is a fact about the state dir. Without it, a stop
311+
// of a runtime its host already closed finds no answer and waits on a listener that is gone.
312+
if (workInHandReported) return;
313+
workInHandReported = true;
314+
if (recordsCloses) params.registration.recordShutdownRetention(releaseParams.unfinished);
300315
return;
301316
}
317+
const answer = params.answers.take(releaseParams.unfinished);
302318
if (!registrationFinished) {
303319
registrationFinished = true;
304320
await params.finishRegistration({
@@ -307,6 +323,7 @@ function createDaemonShutdownRelease(params: {
307323
logPath: params.logPath,
308324
outcome: releaseParams.outcome,
309325
completion: answer,
326+
unfinished: recordsCloses ? releaseParams.unfinished : undefined,
310327
});
311328
}
312329
if (lockConfigurationCleared) return;
@@ -826,6 +843,7 @@ async function startDaemonRuntimeOnHost(
826843
logPath,
827844
answers: shutdownAnswers,
828845
finishRegistration: finishDaemonRegistration,
846+
ownership,
829847
clearLockConfiguration: () => deps.ownerLifecycle.clearDaemonLockConfiguration(),
830848
});
831849
const drainInFlightRequests = async (): Promise<ShutdownReport['requests']> => {

‎packages/daemon/src/server/http/daemon-startup-diagnostics.ts‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,9 +38,14 @@ export async function finishDaemonRegistration(params: {
3838
logPath: string;
3939
outcome?: Parameters<DaemonRegistrationOwner['finish']>[0];
4040
completion?: Parameters<DaemonRegistrationOwner['finish']>[1];
41+
unfinished?: Parameters<DaemonRegistrationOwner['finish']>[2];
4142
}): Promise<void> {
4243
try {
43-
const removal = await params.registration.finish(params.outcome, params.completion);
44+
const removal = await params.registration.finish(
45+
params.outcome,
46+
params.completion,
47+
params.unfinished,
48+
);
4449
if (removal.state !== 'removed' && removal.state !== 'absent') {
4550
await emitDaemonDiagnostic(params.logPath, 'daemon_info_removal_declined', {
4651
infoPath: params.infoPath,

0 commit comments

Comments
 (0)