Skip to content

Commit a1ae1cf

Browse files
committed
refactor(daemon): ride the owned health read for the liveness probe's HTTP leg
Review at 7d78066 (non-blocking): `probeHttpHealth` repeated the /health GET that `readDaemonHttpHealth` already owns. The health read now takes the caller's abort signal and an opt-out from its reachability timeout caps, and the probe's HTTP leg is a call to it: the socket leg and the concurrent race stay, because a TCP connect alone cannot show a blocked event loop answers. One verdict nuance the reuse makes visible: the reachability reader reports a 5xx as unreachable, while the probe asks only whether the endpoint answered, so the leg counts any status code. Reading `reachable` there would make a daemon answering 500 look silent and authorize the kill #3177 is about; a test pins the difference. Also folds the reviewer's comment trims (probe header, loopback teardown, timeout reset block) and refreshes the wire ledger's compatible-change ack for the client-local signature change: the request line, method, headers and accepted payload fields a released peer sees are unchanged.
1 parent d89f898 commit a1ae1cf

6 files changed

Lines changed: 68 additions & 54 deletions

File tree

‎src/__tests__/test-utils/loopback.ts‎

Lines changed: 2 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -45,11 +45,8 @@ export async function skipWhenLoopbackUnavailable(
4545
}
4646

4747
// Connections the test server accepted after it started listening. A `net.Server` refuses to close
48-
// while a connection is live, and a connection whose handler never consumed the bytes the client
49-
// wrote stays open even after the client sends a RST, so a stand-in daemon that hangs one request
50-
// (the whole point of a timeout test) would otherwise hang the teardown with it. The real daemon
51-
// answers this with `DaemonServer.destroyConnections` (`src/daemon/server/transport.ts`); this is
52-
// the same teardown for a stand-in, armed at listen time so no connection can be missed.
48+
// while a connection is live, and a hung-request stand-in keeps one open past the client's RST, so
49+
// teardown destroys them — the job `DaemonServer.destroyConnections` does for the real daemon.
5350
const acceptedConnections = new WeakMap<LoopbackServer, Set<net.Socket>>();
5451

5552
export async function listenOnLoopback(server: LoopbackServer): Promise<number> {

‎src/daemon-client/__tests__/daemon-client-liveness-probe.test.ts‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,20 @@ test('a daemon answering /health is responsive', async (t) => {
7676
});
7777
});
7878

79+
test('a 5xx answer is still an answer: the finding is liveness, not reachability policy', async (t) => {
80+
if (await skipWhenLoopbackUnavailable(t)) return;
81+
// The HTTP leg rides the reachability reader (`readDaemonHttpHealth`), which reports a 5xx as
82+
// UNREACHABLE for its own policy. Reading that flag here would make a daemon that answered 500
83+
// look silent and authorize killing a live shared daemon — the bug #3177 is about.
84+
const server = http.createServer((_req, res) => {
85+
res.statusCode = 503;
86+
res.end('overloaded');
87+
});
88+
await withLoopback(server, async (port) => {
89+
assert.equal(await probeDaemonResponsive(daemonInfo({ httpPort: port })), true);
90+
});
91+
});
92+
7993
test('a refused endpoint is negative, not a hang: the verdict arrives well inside the budget', async (t) => {
8094
if (await skipWhenLoopbackUnavailable(t)) return;
8195
// The wedged-daemon shape: connections are accepted and destroyed without an answer. The

‎src/daemon-client/daemon-client-liveness-probe.ts‎

Lines changed: 29 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,15 @@
11
import net from 'node:net';
22
import { INTERNAL_COMMANDS } from '@agent-device/command-registry/catalog';
33
import { createRequestId } from '@agent-device/host-kit/diagnostics';
4-
import { loadNodeHttpRequester, consumeTextLines } from '@agent-device/host-kit/transport';
4+
import { consumeTextLines } from '@agent-device/host-kit/transport';
5+
import { readDaemonHttpHealth } from './daemon-client-transport.ts';
56
import type { DaemonRequest } from '../daemon/daemon-request.ts';
67
import type { DaemonInfo } from './daemon-client-metadata.ts';
78

8-
// The post-timeout liveness probe (#3177): the question a daemon reset may no longer assume.
9-
// The daemon owns request-scoped recovery, so a reset is only justified for a daemon that answers
10-
// nothing — and the question must be asked on FRESH connections, because the timed-out connection
11-
// is the one the transport tears down to cancel that request.
9+
// The post-timeout liveness probe (#3177): the question a daemon reset may no longer assume. The
10+
// daemon owns request-scoped recovery, so a reset is only justified for a daemon that answers
11+
// nothing, and it must be asked on FRESH connections — the timed-out connection is the one the
12+
// transport already tore down to cancel that request.
1213

1314
// The extra latency a caller of a reset-eligible timed-out request can pay before its error lands.
1415
// An order of magnitude under the narrowest reset-eligible envelope (pinned in
@@ -17,24 +18,24 @@ import type { DaemonInfo } from './daemon-client-metadata.ts';
1718
export const LIVENESS_PROBE_BUDGET_MS = 1_000;
1819

1920
/**
20-
* Whether this local daemon answers anything within the probe budget: its HTTP health route, or
21-
* one sessionless RPC on its socket. An answer on either leg proves the daemon's event loop serves
22-
* fresh requests; a refused or silent leg answers "not this one", not "not the daemon".
21+
* Whether this local daemon answers anything within the probe budget: its HTTP health route, or one
22+
* sessionless RPC on its socket. An answer on either leg proves the event loop serves fresh
23+
* requests; a refused or silent leg answers "not this one", not "not the daemon".
2324
*
2425
* Both legs run CONCURRENTLY against one shared deadline: the finding is per-daemon, not per-leg,
25-
* and a leg that hangs must not spend the window the other leg needs. The first AFFIRMATIVE is
26-
* unrevocable and settles at once. The deadline is ABSOLUTE — the transports' own idle timeouts
27-
* never fire while an endpoint trickles bytes — so the negative verdict always lands inside the
28-
* window. A leg never rejects: a probe that cannot ask is an endpoint that did not answer.
29-
* `session` rides along so an isolation-scoped daemon routes the probe like the request it follows.
26+
* and a leg that hangs must not spend the window the other needs. The first AFFIRMATIVE is
27+
* unrevocable and settles at once, retiring the other leg. The deadline is ABSOLUTE, so a leg whose
28+
* endpoint trickles bytes still answers inside the window. A leg never rejects: a probe that cannot
29+
* ask is an endpoint that did not answer. `session` rides along so an isolation-scoped daemon routes
30+
* the probe like the request it follows.
3031
*/
3132
export async function probeDaemonResponsive(
3233
info: DaemonInfo,
3334
params: Readonly<{ session?: string }> = {},
3435
): Promise<boolean> {
3536
const deadlineAtMs = Date.now() + LIVENESS_PROBE_BUDGET_MS;
3637
const legs: ProbeLeg[] = [];
37-
if (info.httpPort) legs.push(probeHttpHealth(info.httpPort, deadlineAtMs));
38+
if (info.httpPort) legs.push(probeHttpHealth(info, deadlineAtMs));
3839
if (info.port) legs.push(probeSocketRpc(info.port, info.token, params.session, deadlineAtMs));
3940
if (legs.length === 0) return false;
4041
return await new Promise<boolean>((resolve) => {
@@ -84,31 +85,27 @@ function createProbeLeg(deadlineAtMs: number): {
8485
};
8586
}
8687

87-
function probeHttpHealth(httpPort: number, deadlineAtMs: number): ProbeLeg {
88+
function probeHttpHealth(info: DaemonInfo, deadlineAtMs: number): ProbeLeg {
8889
const probe = createProbeLeg(deadlineAtMs);
89-
// `readDaemonInfo` accepts any positive integer port, and `transport.request` throws on one out
90-
// of range; this leg builds detached, so the throw would reach the caller as an unhandled
91-
// rejection.
90+
const abort = new AbortController();
91+
// `readDaemonHttpHealth` owns the /health request; the probe supplies its own absolute budget and
92+
// a way to retire the request once the other leg decides the finding. A malformed record throws
93+
// inside that function — an endpoint that did not answer, never a rejection from the recovery path.
9294
void (async () => {
9395
try {
94-
const transport = await loadNodeHttpRequester('http:');
95-
const request = transport.request(
96-
{ host: '127.0.0.1', port: String(httpPort), path: '/health', method: 'GET' },
97-
(res) => {
98-
// Any status is an answer: the health route is served before any request handling, so
99-
// reaching it proves the event loop serves fresh requests. Drain so the response cannot
100-
// hold the socket open past the finding.
101-
res.resume();
102-
probe.settle(true);
103-
},
104-
);
105-
request.on('error', () => probe.settle(false));
106-
probe.attach(() => request.destroy());
107-
request.end();
96+
const health = await readDaemonHttpHealth(info, Math.max(1, deadlineAtMs - Date.now()), {
97+
signal: abort.signal,
98+
budgetOverridesHealthCheckCap: true,
99+
});
100+
// ANY status is an answer (the health route is served ahead of request handling, so reaching
101+
// it proves the event loop serves fresh requests). Reading `reachable` instead would let a
102+
// 5xx read as silence and authorize killing a daemon that was answering.
103+
probe.settle(health.statusCode !== undefined);
108104
} catch {
109105
probe.settle(false);
110106
}
111107
})();
108+
probe.attach(() => abort.abort());
112109
return probe.leg;
113110
}
114111

‎src/daemon-client/daemon-client-timeout.ts‎

Lines changed: 4 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -171,15 +171,11 @@ type DaemonReset = Readonly<{
171171
// proves is ours, then clear the registration the dead daemon can no longer release. Identity is
172172
// re-verified immediately before the signal so a recycled pid is never signaled, and the
173173
// registration is re-read afterward under the same proof (#3125): the probe window is exactly when
174-
// a replacement daemon can publish, and a reset that deleted on faith would orphan the
175-
// replacement's record. A `replaced` verdict leaves that record alone; a kill proves the OLD
176-
// daemon unconditionally, so the signal itself needed no such proof.
174+
// a replacement daemon can publish, so only a `match` verdict may delete the record.
177175
//
178-
// The startup lock is deliberately untouched. ADR 0030 gives reclaim to the acquirer under its
179-
// mutation guard, and `daemon.lock` is now that protocol's directory — a client deleting it
180-
// out-of-band is the legacy-reclaimer pattern the ADR refuses. A reset therefore ends the process
181-
// and its own registration only; the next acquirer reclaims the dead holder's lock through the
182-
// protocol.
176+
// The startup lock is deliberately untouched: ADR 0030 gives reclaim to the acquirer under its
177+
// mutation guard, and a client deleting `daemon.lock` out-of-band is the legacy-reclaimer pattern
178+
// that ADR refuses.
183179
async function resetDaemonAfterTimeout(info: DaemonInfo, paths: DaemonPaths): Promise<DaemonReset> {
184180
let forcedKill = false;
185181
try {

‎src/daemon-client/daemon-client-transport.ts‎

Lines changed: 16 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -141,9 +141,16 @@ export async function readRemoteDaemonHealth(
141141
return health;
142142
}
143143

144-
async function readDaemonHttpHealth(
144+
export async function readDaemonHttpHealth(
145145
info: DaemonInfo,
146146
probeTimeoutMs?: number,
147+
/**
148+
* `signal` lets a caller that races this health read against another probe retire the request
149+
* once the race is decided. `budgetOverridesHealthCheckCap` opts out of the health-check timeouts,
150+
* which are a policy for reachability probes: the post-timeout liveness probe (#3177) carries its
151+
* own absolute budget, and truncating it to 500ms would make a slow-to-answer daemon read as dead.
152+
*/
153+
options: Readonly<{ signal?: AbortSignal; budgetOverridesHealthCheckCap?: boolean }> = {},
147154
): Promise<RemoteDaemonHealth> {
148155
const endpoint = info.baseUrl
149156
? buildDaemonHttpUrl(info.baseUrl, 'health')
@@ -153,12 +160,15 @@ async function readDaemonHttpHealth(
153160
if (!endpoint) return { reachable: false };
154161
const url = new URL(endpoint);
155162
const transport = await loadNodeHttpRequester(url.protocol);
156-
const timeoutMs = Math.min(
157-
info.baseUrl ? REMOTE_DAEMON_HEALTHCHECK_TIMEOUT_MS : LOCAL_DAEMON_HEALTHCHECK_TIMEOUT_MS,
158-
probeTimeoutMs ?? Number.POSITIVE_INFINITY,
159-
);
163+
const healthCheckCapMs = info.baseUrl
164+
? REMOTE_DAEMON_HEALTHCHECK_TIMEOUT_MS
165+
: LOCAL_DAEMON_HEALTHCHECK_TIMEOUT_MS;
166+
const timeoutMs =
167+
options.budgetOverridesHealthCheckCap && probeTimeoutMs !== undefined
168+
? probeTimeoutMs
169+
: Math.min(healthCheckCapMs, probeTimeoutMs ?? Number.POSITIVE_INFINITY);
160170
if (timeoutMs <= 0) return { reachable: false, timedOut: true };
161-
const signal = AbortSignal.timeout(Math.ceil(timeoutMs));
171+
const signal = options.signal ?? AbortSignal.timeout(Math.ceil(timeoutMs));
162172
return await new Promise((resolve) => {
163173
const headers = info.baseUrl ? buildDaemonHttpAuthHeaders(info.token) : {};
164174
const unreachable = (): RemoteDaemonHealth =>

‎test/wire-compat/ledger.json‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,7 @@
6767
"src/daemon-client/daemon-client-rpc.ts#toDaemonHttpRpcError": "sha256:888246763c48670e7da893054d025744654f8715c3b4906312617a2b5028316b",
6868
"src/daemon-client/daemon-client-transport.ts#RemoteDaemonHealth": "sha256:3bac36fa97090b273afe1128103e1bf476d05441fd9de41317f1b78775b88304",
6969
"src/daemon-client/daemon-client-transport.ts#RemoteDaemonHealthLink": "sha256:7702598468b4c82b4ffa93064cd6bdfec938c1c527f7e6007c0584f3884a6eea",
70-
"src/daemon-client/daemon-client-transport.ts#readDaemonHttpHealth": "sha256:eef0e153eb6f0bc02175e464dd6d98559b500b65ea8c74ba2203cfef09ec09a0",
70+
"src/daemon-client/daemon-client-transport.ts#readDaemonHttpHealth": "sha256:334280feb8adb487ddbd53c9b79bd837fd8fa7155fd124c5528192455d49fac1",
7171
"src/daemon-client/daemon-client-transport.ts#readHealthLink": "sha256:f56404b94d73da57de7248719c9136b760d2be8b4a53cde5315a2311b088425b",
7272
"src/daemon-client/daemon-client-transport.ts#readHealthPayload": "sha256:4e85ffc3e35e02379c393e9312344757e003cf1f0ad9eb8d1f77d90c81c861f1",
7373
"src/daemon-client/daemon-client-transport.ts#readRemoteDaemonHealth": "sha256:bcefa89fbb7fcbd6fee1b5ecb217955b9eee8d6fd2ad175fb653011edbd199d9",
@@ -431,8 +431,8 @@
431431
},
432432
{
433433
"declaration": "src/daemon-client/daemon-client-transport.ts#readDaemonHttpHealth",
434-
"digest": "sha256:eef0e153eb6f0bc02175e464dd6d98559b500b65ea8c74ba2203cfef09ec09a0",
435-
"rationale": "A restart retry must tell a probe that ran out of time from one that failed, so the client can report the RPC deadline instead of 'Remote daemon is unavailable'. `timedOut` is client-local: the prober sets it on its own timeout and abort paths and never reads it from a /health payload. The health request and the accepted payload fields are unchanged, so a released daemon or proxy is probed and parsed exactly as before."
434+
"digest": "sha256:334280feb8adb487ddbd53c9b79bd837fd8fa7155fd124c5528192455d49fac1",
435+
"rationale": "The post-timeout liveness probe (#3177) reuses this health read as its HTTP leg, so the budget and abort signal became caller-supplied options instead of an internal AbortSignal.timeout. Both are client-local: the /health request line, method, headers and accepted payload fields a released daemon or proxy sees are unchanged, and the health-check timeout caps still apply to every reachability probe that does not opt out."
436436
}
437437
]
438438
}

0 commit comments

Comments
 (0)