Skip to content

Commit 5030987

Browse files
committed
refactor(daemon): ride the owned health read for the liveness probe's HTTP leg
Review at 7d78066 (non-blocking): `probeHttpHealth` repeated the /health GET that `readDaemonHttpHealth` already owns. The health read now takes the caller's abort signal and an opt-out from its reachability timeout caps, and the probe's HTTP leg is a call to it: the socket leg and the concurrent race stay, because a TCP connect alone cannot show a blocked event loop answers. One verdict nuance the reuse makes visible: the reachability reader reports a 5xx as unreachable, while the probe asks only whether the endpoint answered, so the leg counts any status code. Reading `reachable` there would make a daemon answering 500 look silent and authorize the kill #3177 is about; a test pins the difference. Also folds the reviewer's comment trims (probe header, loopback teardown, timeout reset block) and refreshes the wire ledger's compatible-change ack for the client-local signature change: the request line, method, headers and accepted payload fields a released peer sees are unchanged.
1 parent de801e6 commit 5030987

6 files changed

Lines changed: 69 additions & 56 deletions

File tree

‎src/__tests__/test-utils/loopback.ts‎

Lines changed: 2 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -45,11 +45,8 @@ export async function skipWhenLoopbackUnavailable(
4545
}
4646

4747
// Connections the test server accepted after it started listening. A `net.Server` refuses to close
48-
// while a connection is live, and a connection whose handler never consumed the bytes the client
49-
// wrote stays open even after the client sends a RST, so a stand-in daemon that hangs one request
50-
// (the whole point of a timeout test) would otherwise hang the teardown with it. The real daemon
51-
// answers this with `DaemonServer.destroyConnections` (`src/daemon/server/transport.ts`); this is
52-
// the same teardown for a stand-in, armed at listen time so no connection can be missed.
48+
// while a connection is live, and a hung-request stand-in keeps one open past the client's RST, so
49+
// teardown destroys them — the job `DaemonServer.destroyConnections` does for the real daemon.
5350
const acceptedConnections = new WeakMap<LoopbackServer, Set<net.Socket>>();
5451

5552
export async function listenOnLoopback(server: LoopbackServer): Promise<number> {

‎src/daemon-client/__tests__/daemon-client-liveness-probe.test.ts‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,20 @@ test('a daemon answering /health is responsive', async (t) => {
7676
});
7777
});
7878

79+
test('a 5xx answer is still an answer: the finding is liveness, not reachability policy', async (t) => {
80+
if (await skipWhenLoopbackUnavailable(t)) return;
81+
// The HTTP leg rides the reachability reader (`readDaemonHttpHealth`), which reports a 5xx as
82+
// UNREACHABLE for its own policy. Reading that flag here would make a daemon that answered 500
83+
// look silent and authorize killing a live shared daemon — the bug #3177 is about.
84+
const server = http.createServer((_req, res) => {
85+
res.statusCode = 503;
86+
res.end('overloaded');
87+
});
88+
await withLoopback(server, async (port) => {
89+
assert.equal(await probeDaemonResponsive(daemonInfo({ httpPort: port })), true);
90+
});
91+
});
92+
7993
test('a refused endpoint is negative, not a hang: the verdict arrives well inside the budget', async (t) => {
8094
if (await skipWhenLoopbackUnavailable(t)) return;
8195
// The wedged-daemon shape: connections are accepted and destroyed without an answer. The

‎src/daemon-client/daemon-client-liveness-probe.ts‎

Lines changed: 29 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,15 @@
11
import net from 'node:net';
22
import { INTERNAL_COMMANDS } from '@agent-device/command-registry/catalog';
33
import { createRequestId } from '@agent-device/host-kit/diagnostics';
4-
import { loadNodeHttpRequester, consumeTextLines } from '@agent-device/host-kit/transport';
4+
import { consumeTextLines } from '@agent-device/host-kit/transport';
5+
import { readDaemonHttpHealth } from './daemon-client-transport.ts';
56
import type { DaemonRequest } from '../daemon/daemon-request.ts';
67
import type { DaemonInfo } from './daemon-client-metadata.ts';
78

8-
// The post-timeout liveness probe (#3177): the question a daemon reset may no longer assume.
9-
// The daemon owns request-scoped recovery, so a reset is only justified for a daemon that answers
10-
// nothing — and the question must be asked on FRESH connections, because the timed-out connection
11-
// is the one the transport tears down to cancel that request.
9+
// The post-timeout liveness probe (#3177): the question a daemon reset may no longer assume. The
10+
// daemon owns request-scoped recovery, so a reset is only justified for a daemon that answers
11+
// nothing, and it must be asked on FRESH connections — the timed-out connection is the one the
12+
// transport already tore down to cancel that request.
1213

1314
// The extra latency a caller of a reset-eligible timed-out request can pay before its error lands.
1415
// An order of magnitude under the narrowest reset-eligible envelope (pinned in
@@ -17,24 +18,24 @@ import type { DaemonInfo } from './daemon-client-metadata.ts';
1718
export const LIVENESS_PROBE_BUDGET_MS = 1_000;
1819

1920
/**
20-
* Whether this local daemon answers anything within the probe budget: its HTTP health route, or
21-
* one sessionless RPC on its socket. An answer on either leg proves the daemon's event loop serves
22-
* fresh requests; a refused or silent leg answers "not this one", not "not the daemon".
21+
* Whether this local daemon answers anything within the probe budget: its HTTP health route, or one
22+
* sessionless RPC on its socket. An answer on either leg proves the event loop serves fresh
23+
* requests; a refused or silent leg answers "not this one", not "not the daemon".
2324
*
2425
* Both legs run CONCURRENTLY against one shared deadline: the finding is per-daemon, not per-leg,
25-
* and a leg that hangs must not spend the window the other leg needs. The first AFFIRMATIVE is
26-
* unrevocable and settles at once. The deadline is ABSOLUTE — the transports' own idle timeouts
27-
* never fire while an endpoint trickles bytes — so the negative verdict always lands inside the
28-
* window. A leg never rejects: a probe that cannot ask is an endpoint that did not answer.
29-
* `session` rides along so an isolation-scoped daemon routes the probe like the request it follows.
26+
* and a leg that hangs must not spend the window the other needs. The first AFFIRMATIVE is
27+
* unrevocable and settles at once, retiring the other leg. The deadline is ABSOLUTE, so a leg whose
28+
* endpoint trickles bytes still answers inside the window. A leg never rejects: a probe that cannot
29+
* ask is an endpoint that did not answer. `session` rides along so an isolation-scoped daemon routes
30+
* the probe like the request it follows.
3031
*/
3132
export async function probeDaemonResponsive(
3233
info: DaemonInfo,
3334
params: Readonly<{ session?: string }> = {},
3435
): Promise<boolean> {
3536
const deadlineAtMs = Date.now() + LIVENESS_PROBE_BUDGET_MS;
3637
const legs: ProbeLeg[] = [];
37-
if (info.httpPort) legs.push(probeHttpHealth(info.httpPort, deadlineAtMs));
38+
if (info.httpPort) legs.push(probeHttpHealth(info, deadlineAtMs));
3839
if (info.port) legs.push(probeSocketRpc(info.port, info.token, params.session, deadlineAtMs));
3940
if (legs.length === 0) return false;
4041
return await new Promise<boolean>((resolve) => {
@@ -84,31 +85,27 @@ function createProbeLeg(deadlineAtMs: number): {
8485
};
8586
}
8687

87-
function probeHttpHealth(httpPort: number, deadlineAtMs: number): ProbeLeg {
88+
function probeHttpHealth(info: DaemonInfo, deadlineAtMs: number): ProbeLeg {
8889
const probe = createProbeLeg(deadlineAtMs);
89-
// `readDaemonInfo` accepts any positive integer port, and `transport.request` throws on one out
90-
// of range; this leg builds detached, so the throw would reach the caller as an unhandled
91-
// rejection.
90+
const abort = new AbortController();
91+
// `readDaemonHttpHealth` owns the /health request; the probe supplies its own absolute budget and
92+
// a way to retire the request once the other leg decides the finding. A malformed record throws
93+
// inside that function — an endpoint that did not answer, never a rejection from the recovery path.
9294
void (async () => {
9395
try {
94-
const transport = await loadNodeHttpRequester('http:');
95-
const request = transport.request(
96-
{ host: '127.0.0.1', port: String(httpPort), path: '/health', method: 'GET' },
97-
(res) => {
98-
// Any status is an answer: the health route is served before any request handling, so
99-
// reaching it proves the event loop serves fresh requests. Drain so the response cannot
100-
// hold the socket open past the finding.
101-
res.resume();
102-
probe.settle(true);
103-
},
104-
);
105-
request.on('error', () => probe.settle(false));
106-
probe.attach(() => request.destroy());
107-
request.end();
96+
const health = await readDaemonHttpHealth(info, Math.max(1, deadlineAtMs - Date.now()), {
97+
signal: abort.signal,
98+
budgetOverridesHealthCheckCap: true,
99+
});
100+
// ANY status is an answer (the health route is served ahead of request handling, so reaching
101+
// it proves the event loop serves fresh requests). Reading `reachable` instead would let a
102+
// 5xx read as silence and authorize killing a daemon that was answering.
103+
probe.settle(health.statusCode !== undefined);
108104
} catch {
109105
probe.settle(false);
110106
}
111107
})();
108+
probe.attach(() => abort.abort());
112109
return probe.leg;
113110
}
114111

‎src/daemon-client/daemon-client-timeout.ts‎

Lines changed: 4 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -171,15 +171,11 @@ type DaemonReset = Readonly<{
171171
// proves is ours, then clear the registration the dead daemon can no longer release. Identity is
172172
// re-verified immediately before the signal so a recycled pid is never signaled, and the
173173
// registration is re-read afterward under the same proof (#3125): the probe window is exactly when
174-
// a replacement daemon can publish, and a reset that deleted on faith would orphan the
175-
// replacement's record. A `replaced` verdict leaves that record alone; a kill proves the OLD
176-
// daemon unconditionally, so the signal itself needed no such proof.
174+
// a replacement daemon can publish, so only a `match` verdict may delete the record.
177175
//
178-
// The startup lock is deliberately untouched. ADR 0030 gives reclaim to the acquirer under its
179-
// mutation guard, and `daemon.lock` is now that protocol's directory — a client deleting it
180-
// out-of-band is the legacy-reclaimer pattern the ADR refuses. A reset therefore ends the process
181-
// and its own registration only; the next acquirer reclaims the dead holder's lock through the
182-
// protocol.
176+
// The startup lock is deliberately untouched: ADR 0030 gives reclaim to the acquirer under its
177+
// mutation guard, and a client deleting `daemon.lock` out-of-band is the legacy-reclaimer pattern
178+
// that ADR refuses.
183179
async function resetDaemonAfterTimeout(info: DaemonInfo, paths: DaemonPaths): Promise<DaemonReset> {
184180
let forcedKill = false;
185181
try {

‎src/daemon-client/daemon-client-transport.ts‎

Lines changed: 17 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -213,7 +213,7 @@ export async function readRemoteDaemonHealth(
213213
probeTimeoutMs?: number,
214214
callerSignal?: AbortSignal,
215215
): Promise<RemoteDaemonHealth> {
216-
const health = await readDaemonHttpHealth(info, probeTimeoutMs, callerSignal);
216+
const health = await readDaemonHttpHealth(info, probeTimeoutMs, { signal: callerSignal });
217217
if (!info.baseUrl || !health.reachable) return health;
218218
// Every link a command RPC crosses has to speak the client's protocol: a proxy that reports a
219219
// skewed daemon behind it fails here, before the RPC, exactly like a skewed proxy does.
@@ -236,10 +236,16 @@ export async function readRemoteDaemonHealth(
236236
return health;
237237
}
238238

239-
async function readDaemonHttpHealth(
239+
export async function readDaemonHttpHealth(
240240
info: DaemonInfo,
241241
probeTimeoutMs?: number,
242-
callerSignal?: AbortSignal,
242+
/**
243+
* `signal` lets a caller that races this health read against another probe retire the request
244+
* once the race is decided. `budgetOverridesHealthCheckCap` opts out of the health-check timeouts,
245+
* which are a policy for reachability probes: the post-timeout liveness probe (#3177) carries its
246+
* own absolute budget, and truncating it to 500ms would make a slow-to-answer daemon read as dead.
247+
*/
248+
options: Readonly<{ signal?: AbortSignal; budgetOverridesHealthCheckCap?: boolean }> = {},
243249
): Promise<RemoteDaemonHealth> {
244250
const endpoint = info.baseUrl
245251
? buildDaemonHttpUrl(info.baseUrl, 'health')
@@ -249,15 +255,18 @@ async function readDaemonHttpHealth(
249255
if (!endpoint) return { reachable: false };
250256
const url = new URL(endpoint);
251257
const transport = await loadNodeHttpRequester(url.protocol);
252-
const timeoutMs = Math.min(
253-
info.baseUrl ? REMOTE_DAEMON_HEALTHCHECK_TIMEOUT_MS : LOCAL_DAEMON_HEALTHCHECK_TIMEOUT_MS,
254-
probeTimeoutMs ?? Number.POSITIVE_INFINITY,
255-
);
258+
const healthCheckCapMs = info.baseUrl
259+
? REMOTE_DAEMON_HEALTHCHECK_TIMEOUT_MS
260+
: LOCAL_DAEMON_HEALTHCHECK_TIMEOUT_MS;
261+
const timeoutMs =
262+
options.budgetOverridesHealthCheckCap && probeTimeoutMs !== undefined
263+
? probeTimeoutMs
264+
: Math.min(healthCheckCapMs, probeTimeoutMs ?? Number.POSITIVE_INFINITY);
256265
if (timeoutMs <= 0) return { reachable: false, timedOut: true };
257266
// `timedOut` is keyed on the probe's own budget alone: a caller abort must never read as a
258267
// timeout, because a timed-out probe on an RPC-capped budget is answered as the RPC timing out.
259268
const timeoutSignal = AbortSignal.timeout(Math.ceil(timeoutMs));
260-
const signal = callerSignal ? AbortSignal.any([timeoutSignal, callerSignal]) : timeoutSignal;
269+
const signal = options.signal ? AbortSignal.any([timeoutSignal, options.signal]) : timeoutSignal;
261270
return await new Promise((resolve) => {
262271
const headers = info.baseUrl ? buildDaemonHttpAuthHeaders(info.token) : {};
263272
const unreachable = (): RemoteDaemonHealth =>

‎test/wire-compat/ledger.json‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,7 @@
6767
"src/daemon-client/daemon-client-rpc.ts#toDaemonHttpRpcError": "sha256:888246763c48670e7da893054d025744654f8715c3b4906312617a2b5028316b",
6868
"src/daemon-client/daemon-client-transport.ts#RemoteDaemonHealth": "sha256:3bac36fa97090b273afe1128103e1bf476d05441fd9de41317f1b78775b88304",
6969
"src/daemon-client/daemon-client-transport.ts#RemoteDaemonHealthLink": "sha256:7702598468b4c82b4ffa93064cd6bdfec938c1c527f7e6007c0584f3884a6eea",
70-
"src/daemon-client/daemon-client-transport.ts#readDaemonHttpHealth": "sha256:a483fbdd845db39ffea596061ff04e9db4acdafd20b593dfed74e7c82a93e522",
70+
"src/daemon-client/daemon-client-transport.ts#readDaemonHttpHealth": "sha256:334280feb8adb487ddbd53c9b79bd837fd8fa7155fd124c5528192455d49fac1",
7171
"src/daemon-client/daemon-client-transport.ts#readHealthLink": "sha256:f56404b94d73da57de7248719c9136b760d2be8b4a53cde5315a2311b088425b",
7272
"src/daemon-client/daemon-client-transport.ts#readHealthPayload": "sha256:4e85ffc3e35e02379c393e9312344757e003cf1f0ad9eb8d1f77d90c81c861f1",
7373
"src/daemon-client/daemon-client-transport.ts#readRemoteDaemonHealth": "sha256:df174d1ccf73851ca58bd75533fb4660424003b98bd9e41fe942cf3ab3698e5f",
@@ -431,8 +431,8 @@
431431
},
432432
{
433433
"declaration": "src/daemon-client/daemon-client-transport.ts#readDaemonHttpHealth",
434-
"digest": "sha256:a483fbdd845db39ffea596061ff04e9db4acdafd20b593dfed74e7c82a93e522",
435-
"rationale": "A restart retry must tell a probe that ran out of time from one that failed, so the client can report the RPC deadline instead of 'Remote daemon is unavailable'. `timedOut` is client-local: the prober sets it on its own timeout and abort paths and never reads it from a /health payload. The #3178 caller signal joins the probe's own timeout in-process (AbortSignal.any) and never enters the /health request or the accepted payload fields, so a released daemon or proxy is probed and parsed exactly as before."
434+
"digest": "sha256:334280feb8adb487ddbd53c9b79bd837fd8fa7155fd124c5528192455d49fac1",
435+
"rationale": "The post-timeout liveness probe (#3177) reuses this health read as its HTTP leg, so the budget and abort signal became caller-supplied options instead of an internal AbortSignal.timeout. Both are client-local: the /health request line, method, headers and accepted payload fields a released daemon or proxy sees are unchanged, and the health-check timeout caps still apply to every reachability probe that does not opt out."
436436
}
437437
]
438438
}

0 commit comments

Comments
 (0)