Skip to content

Commit cc5e356

Browse files
authored
fix(daemon): fence daemon.json removal to its owning process (#3102)
* fix(daemon): fence daemon.json removal to its owning process A daemon leaving the pool unlinked `daemon.json` whatever it found there, so an idle reap or a shutdown racing a successor deleted the metadata of the daemon then serving clients: sessions became unreachable and `daemon stop` reported a daemon that was never started. Removal now re-reads the record and proceeds only while it names this process, matching the rule the lock beside it already applied, and the identity compared is pid plus process start time because a recycled pid names an unrelated process. A live takeover is recorded rather than silently tolerated: the runtime watches its own registration after publication and reports a replacement or removal once to `daemon.log`, distinguishing it from a record that is merely unreadable or unparseable, which is still on disk. `daemon.json` is published through an atomic rename so a client reading during startup never decodes a half-written record that names no owner. Refs #3087 * fix(daemon): take the owner identity from host-kit and refuse unproven records Address review on #3102. The daemon asserted its own ownership identity with a second copy of the host-kit constructor, so the daemon.json fence and the device-claim fence could drift apart; it now calls readCurrentOwnerIdentity() like every other ownership fence does. That exposed a weaker rule than the issue needs: two records that both failed to read a start time agreed on nothing, and agreeing on nothing was reported as a match — the pid-only fallback the fence exists to refuse. Ownership now returns a third verdict, unproven, for a record that matches on pid and can be proved neither way. removeInfoOwnedBy declines it, because leaving a record a dead pid left behind costs a client one liveness probe while deleting a successor's metadata is unrecoverable; the loss watch declines it too, since nothing proves a takeover happened. Reporting it as `replaced` would announce an event with no evidence of one. Also from review: the startup-failure exit now tears the loss watch down before touching daemon.json, so a failed runtime cannot report a loss for a record it just removed itself; the watch stops when the state dir is pruned rather than polling a directory that no longer exists; the state `decodable` is renamed `ownerless`, because it names the record that has no owner rather than the one that fails to decode; and the dead readProcessStartTime re-export is gone. * docs(daemon): name the lock premise the ownership re-read rests on
1 parent bf6abb2 commit cc5e356

10 files changed

Lines changed: 1044 additions & 28 deletions

File tree

‎.github/workflows/ios.yml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -160,6 +160,13 @@ jobs:
160160
}
161161
'
162162
163+
# Preflight and every step below deliberately share one AGENT_DEVICE_STATE_DIR: the prepared
164+
# runner lives in the daemon this step leaves running, and a per-step state dir would rebuild it
165+
# per step. That also means this daemon outlives the step and can be superseded while it lives,
166+
# which used to make its 5-minute idle reap delete the metadata of the daemon then serving
167+
# clients (#3087). It now removes daemon.json only when the file names its own pid and process
168+
# start time, so a recycled pid cannot be mistaken for the owner, and the reap costs nothing
169+
# but a daemon.log line.
163170
- name: Preflight iOS runner through public CLI
164171
run: |
165172
pnpm clean:daemon

‎src/daemon/__tests__/daemon-registration.test.ts‎

Lines changed: 73 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,11 @@ import assert from 'node:assert/strict';
22
import fs from 'node:fs';
33
import path from 'node:path';
44
import { afterEach, test } from 'vitest';
5-
import { isSupersededDaemonOwner, readRegisteredDaemonIdentity } from '../daemon-registration.ts';
5+
import {
6+
isSupersededDaemonOwner,
7+
readRegisteredDaemonIdentity,
8+
readRegisteredDaemonOwnership,
9+
} from '../daemon-registration.ts';
610
import { writeInfo } from '../server/server-lifecycle.ts';
711
import { publishDaemonRegistration } from '../../__tests__/test-utils/device-claim-store.ts';
812
import { mkdtempForTestSync } from '../../__tests__/test-utils/tmp-dir.ts';
@@ -85,3 +89,71 @@ test('the reading process is never superseded by a registration naming someone e
8589

8690
assert.equal(isSupersededDaemonOwner({ stateDir, pid: process.pid, startTime: 'ours' }), false);
8791
});
92+
93+
test('ownership is decided by the identity in the record, not by its pid alone', () => {
94+
const stateDir = useStateDir();
95+
const infoPath = infoPathOf(stateDir);
96+
const owner = { pid: process.pid, startTime: 'ours' };
97+
98+
publishDaemonRegistration(stateDir, owner);
99+
assert.deepEqual(readRegisteredDaemonOwnership(infoPath, owner), { state: 'match' });
100+
101+
// Two unreadable start times agree, and agreeing on nothing is not proof of ownership. Nor is this
102+
// a proved takeover, so it lands between the two rather than defaulting either way.
103+
publishDaemonRegistration(stateDir, { pid: process.pid, startTime: null });
104+
assert.deepEqual(readRegisteredDaemonOwnership(infoPath, { pid: process.pid, startTime: null }), {
105+
state: 'unproven',
106+
});
107+
108+
// Same pid with a readable start time on the record but not on this side is still unproven: the
109+
// proof of a recycle has to run on two readable birth times, and `ownerIdentityDiffers` is one-way
110+
// by design so no caller mistakes one owner for two.
111+
publishDaemonRegistration(stateDir, { pid: process.pid, startTime: 'recycled' });
112+
assert.deepEqual(readRegisteredDaemonOwnership(infoPath, { pid: process.pid, startTime: null }), {
113+
state: 'unproven',
114+
});
115+
116+
publishDaemonRegistration(stateDir, { pid: 4242, startTime: 'successor-start' });
117+
assert.deepEqual(readRegisteredDaemonOwnership(infoPath, owner), {
118+
state: 'replaced',
119+
identity: { pid: 4242, startTime: 'successor-start' },
120+
});
121+
});
122+
123+
test('ownership names each way a record yields no owner', () => {
124+
const stateDir = useStateDir();
125+
const infoPath = infoPathOf(stateDir);
126+
const owner = { pid: process.pid, startTime: 'ours' };
127+
128+
assert.deepEqual(readRegisteredDaemonOwnership(infoPath, owner), { state: 'absent' });
129+
130+
fs.writeFileSync(infoPath, '{not json');
131+
assert.deepEqual(readRegisteredDaemonOwnership(infoPath, owner), { state: 'ownerless' });
132+
133+
fs.writeFileSync(infoPath, JSON.stringify({ pid: 4242 }));
134+
assert.deepEqual(readRegisteredDaemonOwnership(infoPath, owner), {
135+
state: 'replaced',
136+
identity: { pid: 4242, startTime: null },
137+
});
138+
});
139+
140+
test.skipIf(process.getuid?.() === 0)(
141+
'a registration this process cannot read is unreadable, not absent',
142+
() => {
143+
// EACCES leaves the file on disk: only a root-owned CI host reads mode 000 anyway, and reading it
144+
// there — or reading the failure as absence — would repeat #3087 from the other side.
145+
const infoPath = infoPathOf(useStateDir());
146+
fs.writeFileSync(infoPath, JSON.stringify({ pid: process.pid, processStartTime: 'ours' }));
147+
fs.chmodSync(infoPath, 0o000);
148+
try {
149+
assert.deepEqual(
150+
readRegisteredDaemonOwnership(infoPath, { pid: process.pid, startTime: 'ours' }),
151+
{
152+
state: 'unreadable',
153+
},
154+
);
155+
} finally {
156+
fs.chmodSync(infoPath, 0o600);
157+
}
158+
},
159+
);

‎src/daemon/daemon-registration.ts‎

Lines changed: 82 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
11
import fs from 'node:fs';
2-
import { ownerIdentityDiffers, type OwnerIdentity } from '@agent-device/host-kit/process';
2+
import {
3+
ownerIdentityDiffers,
4+
ownerIdentityMatches,
5+
type OwnerIdentity,
6+
} from '@agent-device/host-kit/process';
37
import { resolveDaemonPaths } from '../daemon-resolution.ts';
48

59
/**
@@ -9,16 +13,86 @@ import { resolveDaemonPaths } from '../daemon-resolution.ts';
913
* however alive it is.
1014
*/
1115
export function readRegisteredDaemonIdentity(infoPath: string): OwnerIdentity | null {
16+
const record = readRegistration(infoPath);
17+
return record.status === 'registered' ? record.identity : null;
18+
}
19+
20+
/** The raw record's identity. A pid of `null` is a file that names no owner, not owner zero. */
21+
type ParsedRegistration = { pid: number | null; startTime: string | null };
22+
23+
function readRegistration(
24+
infoPath: string,
25+
):
26+
| Readonly<{ status: 'registered'; identity: OwnerIdentity }>
27+
| Readonly<{ status: 'absent' | 'unreadable' | 'ownerless' }> {
28+
let raw: string;
29+
try {
30+
raw = fs.readFileSync(infoPath, 'utf8');
31+
} catch (error) {
32+
// Only a path the host reports as gone proves there is no record. A permission error, an I/O
33+
// fault, or a read this process lost leaves a file present whose contents are unknown, which no
34+
// caller may read as "someone removed it".
35+
const code = (error as NodeJS.ErrnoException | null)?.code;
36+
return code === 'ENOENT' || code === 'ENOTDIR'
37+
? { status: 'absent' }
38+
: { status: 'unreadable' };
39+
}
40+
let parsed: ParsedRegistration;
1241
try {
13-
const { pid, processStartTime } = JSON.parse(fs.readFileSync(infoPath, 'utf8')) as {
14-
pid?: unknown;
15-
processStartTime?: unknown;
16-
};
17-
if (typeof pid !== 'number' || !Number.isInteger(pid) || pid <= 0) return null;
18-
return { pid, startTime: readableStartTime(processStartTime) };
42+
parsed = parseRegistration(JSON.parse(raw) as { pid?: unknown; processStartTime?: unknown });
1943
} catch {
20-
return null;
44+
return { status: 'ownerless' };
45+
}
46+
return parsed.pid === null
47+
? { status: 'ownerless' }
48+
: { status: 'registered', identity: { pid: parsed.pid, startTime: parsed.startTime } };
49+
}
50+
51+
function parseRegistration(parsed: {
52+
pid?: unknown;
53+
processStartTime?: unknown;
54+
}): ParsedRegistration {
55+
const pid = parsed.pid;
56+
if (typeof pid !== 'number' || !Number.isInteger(pid) || pid <= 0) {
57+
return { pid: null, startTime: null };
58+
}
59+
return { pid, startTime: readableStartTime(parsed.processStartTime) };
60+
}
61+
62+
/**
63+
* Whether `daemon.json` still names `owner` as the daemon serving its state dir, in one read.
64+
*
65+
* Three verdicts, because ownership here is proved rather than assumed, and a pid alone proves
66+
* nothing: it cannot tell the owner from the process that recycled its number. `match` needs a start
67+
* time readable on both sides to agree, since every caller acts on it by deleting the record.
68+
* `replaced` is the opposite proof, from {@link ownerIdentityDiffers}. `unproven` is neither, and is
69+
* kept distinct from `replaced` because it describes a record this daemon cannot place — ours by pid
70+
* and unreadable by birth — so a caller that reports a takeover would be reporting an event that did
71+
* not happen.
72+
*
73+
* `absent`, `unreadable`, and `ownerless` stay separate too: only `absent` describes a record somebody
74+
* removed, while a file that is present but unreadable or names no owner is still on disk.
75+
*/
76+
export type RegisteredDaemonOwnership =
77+
| Readonly<{ state: 'match' }>
78+
| Readonly<{ state: 'replaced'; identity: OwnerIdentity }>
79+
| Readonly<{ state: 'unproven' | 'absent' | 'unreadable' | 'ownerless' }>;
80+
81+
export function readRegisteredDaemonOwnership(
82+
infoPath: string,
83+
owner: OwnerIdentity,
84+
): RegisteredDaemonOwnership {
85+
const record = readRegistration(infoPath);
86+
if (record.status !== 'registered') return { state: record.status };
87+
if (ownerIdentityDiffers(record.identity, owner)) {
88+
return { state: 'replaced', identity: record.identity };
89+
}
90+
// Two missing start times agree, and agreeing on nothing is not ownership: this is the pid-only
91+
// fallback the fence exists to refuse, so it falls to `unproven` rather than to `match`.
92+
if (ownerIdentityMatches(record.identity, owner) && record.identity.startTime !== null) {
93+
return { state: 'match' };
2194
}
95+
return { state: 'unproven' };
2296
}
2397

2498
function readableStartTime(value: unknown): string | null {
Lines changed: 183 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,183 @@
1+
import assert from 'node:assert/strict';
2+
import fs from 'node:fs';
3+
import path from 'node:path';
4+
import { describe, test, vi } from 'vitest';
5+
import { mkdtempForTestSync } from '../../__tests__/test-utils/tmp-dir.ts';
6+
import { publishDaemonRegistration } from '../../__tests__/test-utils/device-claim-store.ts';
7+
import {
8+
readDaemonMetadataLoss,
9+
watchDaemonMetadataLoss,
10+
type DaemonMetadataLoss,
11+
} from './daemon-metadata-loss.ts';
12+
13+
const OWN_PID = process.pid;
14+
const FOREIGN_PID = 999_999_999;
15+
const OWN = { pid: OWN_PID, startTime: 'own-start' } as const;
16+
17+
function scratch(): { stateDir: string; infoPath: string } {
18+
const stateDir = mkdtempForTestSync('agent-device-daemon-metadata-loss-');
19+
return { stateDir, infoPath: path.join(stateDir, 'daemon.json') };
20+
}
21+
22+
function publish(stateDir: string, pid: number, startTime: string | null): void {
23+
publishDaemonRegistration(stateDir, { pid, startTime });
24+
}
25+
26+
describe('readDaemonMetadataLoss', () => {
27+
test('a registration naming this daemon is not a loss', () => {
28+
const { stateDir, infoPath } = scratch();
29+
publish(stateDir, OWN_PID, 'own-start');
30+
31+
assert.equal(readDaemonMetadataLoss({ infoPath, stateDir, owner: OWN }), undefined);
32+
});
33+
34+
test('a successor that published over this daemon is a loss naming it', () => {
35+
const { stateDir, infoPath } = scratch();
36+
publish(stateDir, FOREIGN_PID, 'successor-start');
37+
38+
assert.deepEqual(readDaemonMetadataLoss({ infoPath, stateDir, owner: OWN }), {
39+
state: 'replaced',
40+
registeredPid: FOREIGN_PID,
41+
});
42+
});
43+
44+
test('a record recycling this pid under a different start time is a loss', () => {
45+
const { stateDir, infoPath } = scratch();
46+
publish(stateDir, OWN_PID, 'recycled-start');
47+
48+
assert.deepEqual(readDaemonMetadataLoss({ infoPath, stateDir, owner: OWN }), {
49+
state: 'replaced',
50+
registeredPid: OWN_PID,
51+
});
52+
});
53+
54+
test('a registration deleted while the state dir stands is an absent loss', () => {
55+
const { stateDir, infoPath } = scratch();
56+
publish(stateDir, OWN_PID, 'own-start');
57+
fs.rmSync(infoPath);
58+
59+
assert.deepEqual(readDaemonMetadataLoss({ infoPath, stateDir, owner: OWN }), {
60+
state: 'absent',
61+
});
62+
});
63+
64+
test('a state dir that was pruned away entirely is not a loss this daemon should report', () => {
65+
// Reporting one would write a `daemon.log` back into the directory the operator just removed.
66+
const { stateDir, infoPath } = scratch();
67+
publish(stateDir, OWN_PID, 'own-start');
68+
fs.rmSync(stateDir, { recursive: true, force: true });
69+
70+
assert.equal(readDaemonMetadataLoss({ infoPath, stateDir, owner: OWN }), undefined);
71+
});
72+
73+
test.skipIf(process.getuid?.() === 0)(
74+
'a record this process cannot read is not evidence that it was taken',
75+
() => {
76+
// A root-owned CI host reads mode 000 anyway, so the case only binds as a non-root process.
77+
const { stateDir, infoPath } = scratch();
78+
publish(stateDir, OWN_PID, 'own-start');
79+
fs.chmodSync(infoPath, 0o000);
80+
try {
81+
assert.equal(readDaemonMetadataLoss({ infoPath, stateDir, owner: OWN }), undefined);
82+
} finally {
83+
fs.chmodSync(infoPath, 0o600);
84+
}
85+
},
86+
);
87+
88+
test('a record that agrees on pid alone is ours by neither proof, so it is not reported lost', () => {
89+
// The shape a host whose start-time probe failed publishes. Reporting it would announce a takeover
90+
// with no evidence of one, while `removeInfoOwnedBy` refuses the same record: the two callers read
91+
// one verdict, and neither gets to default in its own direction.
92+
const { stateDir, infoPath } = scratch();
93+
publish(stateDir, OWN_PID, null);
94+
95+
assert.equal(readDaemonMetadataLoss({ infoPath, stateDir, owner: OWN }), undefined);
96+
});
97+
98+
test('a corrupt record is not evidence that the registration was taken', () => {
99+
const { stateDir, infoPath } = scratch();
100+
fs.writeFileSync(infoPath, '{not json');
101+
102+
assert.equal(readDaemonMetadataLoss({ infoPath, stateDir, owner: OWN }), undefined);
103+
});
104+
});
105+
106+
describe('watchDaemonMetadataLoss', () => {
107+
function startWatch(
108+
stateDir: string,
109+
infoPath: string,
110+
): { losses: DaemonMetadataLoss[]; advance: () => void; cancel: () => void } {
111+
const losses: DaemonMetadataLoss[] = [];
112+
const cancel = watchDaemonMetadataLoss({
113+
infoPath,
114+
stateDir,
115+
owner: OWN,
116+
onLoss: (loss) => losses.push(loss),
117+
});
118+
// A window generous against several poll cycles rather than a pin on the interval constant: what
119+
// these cases own is that a loss is reported once, not how often the poll runs.
120+
return { losses, advance: () => vi.advanceTimersByTime(60_000), cancel };
121+
}
122+
123+
test('reports a takeover once and stops watching', () => {
124+
vi.useFakeTimers();
125+
try {
126+
const { stateDir, infoPath } = scratch();
127+
publish(stateDir, OWN_PID, 'own-start');
128+
const watch = startWatch(stateDir, infoPath);
129+
try {
130+
watch.advance();
131+
assert.deepEqual(watch.losses, [], 'owning the record is not a loss');
132+
133+
publish(stateDir, FOREIGN_PID, 'successor-start');
134+
watch.advance();
135+
watch.advance();
136+
assert.deepEqual(watch.losses, [{ state: 'replaced', registeredPid: FOREIGN_PID }]);
137+
} finally {
138+
watch.cancel();
139+
}
140+
} finally {
141+
vi.useRealTimers();
142+
}
143+
});
144+
145+
test('a pruned state dir ends the watch instead of polling a directory that no longer exists', () => {
146+
vi.useFakeTimers();
147+
try {
148+
const { stateDir, infoPath } = scratch();
149+
publish(stateDir, OWN_PID, 'own-start');
150+
const watch = startWatch(stateDir, infoPath);
151+
152+
fs.rmSync(stateDir, { recursive: true, force: true });
153+
watch.advance();
154+
assert.deepEqual(watch.losses, [], 'a pruned dir is not a loss to report');
155+
156+
// The dir is gone, so a later record under that path cannot be this daemon's metadata to lose.
157+
fs.mkdirSync(stateDir, { recursive: true });
158+
publish(stateDir, FOREIGN_PID, 'successor-start');
159+
watch.advance();
160+
assert.deepEqual(watch.losses, [], 'the watch stopped at the prune');
161+
fs.rmSync(stateDir, { recursive: true, force: true });
162+
} finally {
163+
vi.useRealTimers();
164+
}
165+
});
166+
167+
test('a watch cancelled at shutdown stops reporting', () => {
168+
vi.useFakeTimers();
169+
try {
170+
const { stateDir, infoPath } = scratch();
171+
publish(stateDir, OWN_PID, 'own-start');
172+
const watch = startWatch(stateDir, infoPath);
173+
watch.cancel();
174+
watch.cancel();
175+
176+
fs.rmSync(infoPath);
177+
watch.advance();
178+
assert.deepEqual(watch.losses, []);
179+
} finally {
180+
vi.useRealTimers();
181+
}
182+
});
183+
});

0 commit comments

Comments
 (0)