Skip to content

Commit f7d4ed6

Browse files
committed
fix(daemon): make the liveness probe's health answer a fresh connection's status line
Two review findings about the same question: what may the HTTP leg count as an answer that forbids a reset? - Keep-alive reuse (valid). `http.globalAgent` has had `keepAlive: true` since Node 19, so the leg inherited a pool it was never meant to have: a second command's probe rides the idle socket an earlier /health read left behind, and the module header's claim that the probe asks on FRESH connections — the timed-out connection being the one the transport already tore down — was silently false. Worse than a stale reading: a pooled socket the daemon had half torn down answers nothing, the verdict goes negative, and a live shared daemon is SIGKILLed. That is #3177 reintroduced by the recovery path, so `agent: false` is load-bearing and said so at the request. - Body stall (valid against c71b306, already reverted). A body-reading health helper treats a daemon that sends headers and stalls mid-response as silent. The reverted reuse restored this leg to its own request, whose response callback fires on headers, so production was not affected — but nothing pinned WHY the leg settles there, and the same suggestion will come back. Each finding now has a regression test whose assertion fails under the mutation it forbids: waiting for the response's `end` (verified: the stall test fails), and dropping `agent: false` (verified: the reuse test fails). The stall test needs `flushHeaders()` to mean anything: Node buffers headers with the first body chunk, so a handler that only writes a status line never delivers one and the leg would have nothing to settle on. The reuse test primes the pool through the very module object the probe loads — `loadNodeHttpRequester('http:')` resolves real `node:http` — so the pool holds a candidate, and discriminates on the server seeing the same socket object twice.
1 parent eb10231 commit f7d4ed6

2 files changed

Lines changed: 81 additions & 1 deletion

File tree

‎src/daemon-client/__tests__/daemon-client-liveness-probe.test.ts‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ import { test } from 'vitest';
1212

1313
import { PUBLIC_COMMANDS } from '@agent-device/command-registry/catalog';
1414
import { resolveCommandTimeoutPolicy } from '@agent-device/command-registry/registry';
15+
import { loadNodeHttpRequester } from '@agent-device/host-kit/transport';
1516
import {
1617
LIVENESS_PROBE_BUDGET_MS,
1718
probeDaemonResponsive,
@@ -91,6 +92,81 @@ test('a 5xx answer is still an answer: the finding is liveness, not reachability
9192
});
9293
});
9394

95+
test('headers alone answer: a daemon that stalls mid-body is still served', async (t) => {
96+
if (await skipWhenLoopbackUnavailable(t)) return;
97+
// The finding is "the event loop served a fresh request", and that is already proven by a status
98+
// line. A reader that waits for the whole BODY would call a daemon which answers and then stalls
99+
// mid-response silent — the negative verdict would reset a live shared daemon, which is the bug
100+
// #3177 is about. (A body-reading health helper does exactly that for its own purposes; the probe
101+
// must not borrow it.) So the affirmative must arrive by the deadline's FIRST fraction, not at it.
102+
const stallingBodyServer = http.createServer((_req, res) => {
103+
res.writeHead(200, { 'content-length': '1000' });
104+
// Node buffers headers with the first chunk, so this is what makes the status line actually
105+
// reach the client. The declared body is never sent: headers delivered, body pending forever.
106+
res.flushHeaders();
107+
res.on('error', () => {});
108+
});
109+
await withLoopback(stallingBodyServer, async (port) => {
110+
const startedAt = Date.now();
111+
assert.equal(
112+
await probeDaemonResponsive(daemonInfo({ httpPort: port })),
113+
true,
114+
'a stalled body must not be read as an unresponsive daemon',
115+
);
116+
assert.ok(
117+
Date.now() - startedAt < LIVENESS_PROBE_BUDGET_MS / 2,
118+
'the answer is headers-arrival, not the deadline running out',
119+
);
120+
});
121+
});
122+
123+
test('the health leg rides a fresh connection, never the keep-alive pool', async (t) => {
124+
if (await skipWhenLoopbackUnavailable(t)) return;
125+
// Since Node 19 `http.globalAgent` has `keepAlive: true`, so a default request can REUSE an idle
126+
// socket from an earlier command's health read. Reuse would make "did this probe connect?" a
127+
// question about a socket this probe never opened — and a pooled socket the daemon had half torn
128+
// down would answer silent, resetting a live daemon. Discriminator: on a reused connection the
129+
// server sees the SAME socket object for both requests, so `clientSockets.size` is 1 for reuse
130+
// and 2 for a fresh connection. The prime must use the SAME module object the probe loads
131+
// (`loadNodeHttpRequester('http:')` resolves the real `node:http`) or the pool holds no candidate
132+
// and the test proves nothing.
133+
// `keepAlive` is set at runtime (Node >=19) but not on the base `Agent` type.
134+
if (!(http.globalAgent as { keepAlive?: boolean }).keepAlive) return; // pool cannot prime without keep-alive
135+
const clientSockets = new Set<net.Socket>();
136+
let probeRequests = 0;
137+
let primed = false;
138+
const server = http.createServer((req, res) => {
139+
clientSockets.add(req.socket);
140+
req.socket.on('error', () => {});
141+
if (primed) probeRequests += 1;
142+
else primed = true;
143+
res.end('{}');
144+
});
145+
const httpRequester = await loadNodeHttpRequester('http:');
146+
await withLoopback(server, async (port) => {
147+
// Prime: a full keep-alive request/response whose socket returns to the global pool.
148+
await new Promise<void>((resolve, reject) => {
149+
const request = httpRequester.request(
150+
{ host: '127.0.0.1', port, path: '/health', method: 'GET' },
151+
(res) => {
152+
res.resume();
153+
res.on('end', resolve);
154+
},
155+
);
156+
request.on('error', reject);
157+
request.end();
158+
});
159+
assert.equal(clientSockets.size, 1, 'the prime must have connected');
160+
assert.equal(await probeDaemonResponsive(daemonInfo({ httpPort: port })), true);
161+
assert.equal(probeRequests, 1, 'the health leg must actually have been asked');
162+
assert.equal(
163+
clientSockets.size,
164+
2,
165+
'the probe rode the primed keep-alive socket instead of opening a fresh connection',
166+
);
167+
});
168+
});
169+
94170
test('a refused endpoint is negative, not a hang: the verdict arrives well inside the budget', async (t) => {
95171
if (await skipWhenLoopbackUnavailable(t)) return;
96172
// The wedged-daemon shape: connections are accepted and destroyed without an answer. The

‎src/daemon-client/daemon-client-liveness-probe.ts‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -93,7 +93,11 @@ function probeHttpHealth(httpPort: number, deadlineAtMs: number): ProbeLeg {
9393
try {
9494
const transport = await loadNodeHttpRequester('http:');
9595
const request = transport.request(
96-
{ host: '127.0.0.1', port: String(httpPort), path: '/health', method: 'GET' },
96+
// `agent: false` is load-bearing, not hygiene: since Node 19 the global agent keeps sockets
97+
// alive, so a pooled request would ride an idle socket from an earlier command's health read
98+
// instead of the FRESH connection this probe exists to make — and a half-torn-down pooled
99+
// socket would answer "silent", resetting a live daemon.
100+
{ host: '127.0.0.1', port: String(httpPort), path: '/health', method: 'GET', agent: false },
97101
(res) => {
98102
// Any status is an answer: the health route is served before any request handling, so
99103
// reaching it proves the event loop serves fresh requests. Drain so the response cannot

0 commit comments

Comments
 (0)