Skip to content

Commit 284f114

Browse files
committed
fix(remote): beat to the lease's own window and stop the upload with it
The upload beat waited a fixed twenty seconds for its first renewal, so a lease admitted with a shorter window lapsed while a perfectly valid upload ran — the #2946 symptom one seam earlier. A beat answers with the window it just renewed, so the phase now beats immediately and then a third of that window after each beat lands: a beat slower than the cadence delays its successor instead of replacing the schedule or silencing every beat behind it, and a floor keeps a pathological window from becoming a request loop. A beat refused for a missing or mismatched owner scope says this request can never renew the lease, so retrying it only spent the upload against a lease that had stopped renewing. Those reasons now end the phase beside the lost-lease ones, and the phase hands its task an abort signal the upload chain honours: preflight, finalize, the direct PUT, and the legacy stream all run under the caller's cancellation combined with their own timeout, and a canceled upload stops asking the daemon for a fresh ticket. node:http's own request signal is what stops bytes already piped at a device nobody holds.
1 parent 3b243c5 commit 284f114

10 files changed

Lines changed: 608 additions & 118 deletions

‎docs/adr/0007-remote-device-leases.md‎

Lines changed: 15 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -77,13 +77,24 @@ long as that phase runs, naming the lease scope exactly as the command named it
7777
own. An install names no window, so its beats renew for the window the lease already carries; a
7878
caller that did name one keeps renewing on it. Resolving an absent window to the registry default
7979
instead — which is what a heartbeat used to do — quietly shortened every lease allocated above that
80-
default, which is the other half of why the upload could not survive.
80+
default, which is the other half of why the upload could not survive. Request admission had its own
81+
copy of that mistake: it named a proxy-specific default for every admitted request, so a lease
82+
allocated longer than the default lost its window on the next command. Admission renews on the lease's
83+
window too; the window a lease carries is the one its client named when it allocated.
84+
85+
The first beat is fired when the phase starts, not one cadence in, because a beat is what proves the
86+
lease the upload is spending time on is still alive — a lease shorter than any fixed cadence would
87+
otherwise lapse before anything renewed it, and a lease already gone is worth learning that before any
88+
bytes move. Each beat answers with the window it just renewed, and the next one is armed a third of
89+
that window after the beat lands: beats never overlap, and one slower than the cadence delays its
90+
successor instead of replacing the schedule or suppressing every beat behind it.
8191

8292
A beat is a fresh request each time, never the protected request rewritten: a request identity is
8393
what a timed-out beat is canceled under, and beats must not inherit each other's cancellation. A beat
84-
that finds the lease gone ends the phase with that lease error rather than finishing the upload
85-
against a device nobody owns. A beat that fails for any other reason is reported and survived, because
86-
a later beat covers one lost request.
94+
that finds the lease gone, or finds that this request can never renew it because its scope is missing
95+
or belongs to another lease, ends the phase with that error and cancels the upload rather than
96+
finishing bytes against a device nobody owns or a lease that will stop renewing. A beat that fails for
97+
any other reason is reported and survived, because a later beat covers one lost request.
8798

8899
## Human control
89100

Lines changed: 194 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,194 @@
1+
// Ending an artifact upload when the work it serves is over (#2946).
2+
//
3+
// A beat that finds the device's lease gone aborts the upload protecting that lease. An upload is a
4+
// piped `node:http` request, so the only way to stop bytes already in flight is the request's own
5+
// abort signal — a rejection the caller swallows would keep streaming a full app bundle to a device
6+
// nobody owns. Kept out of `upload-client.test.ts`, which is already over the test-file size
7+
// tripwire and may not grow (docs/agents/testing.md).
8+
9+
import { afterEach, test } from 'vitest';
10+
import assert from 'node:assert/strict';
11+
import { createHash } from 'node:crypto';
12+
import fs from 'node:fs';
13+
import http, { type IncomingMessage, type ServerResponse } from 'node:http';
14+
import path from 'node:path';
15+
import { once } from 'node:events';
16+
import { uploadArtifact } from '../remote/upload-client.ts';
17+
import { mkdtempForTestSync } from './test-utils/tmp-dir.ts';
18+
19+
const TEST_TOKEN = 'agent-device-upload-cancel-token';
20+
const tempDirs: string[] = [];
21+
22+
afterEach(async () => {
23+
for (const dir of tempDirs) {
24+
await fs.promises.rm(dir, { recursive: true, force: true }).catch(() => {});
25+
}
26+
tempDirs.length = 0;
27+
});
28+
29+
test('an aborted signal ends a legacy upload mid-stream instead of finishing the bytes', async () => {
30+
const content = Buffer.alloc(8 * 1024 * 1024, 'x');
31+
const artifactPath = createTempFile('app.apk', content);
32+
const control = new AbortController();
33+
let sawBytes = 0;
34+
let connectionEnded: () => void;
35+
const ended = new Promise<void>((resolve) => {
36+
connectionEnded = resolve;
37+
});
38+
39+
// Preflight reports itself unsupported so the upload takes the legacy stream, the one path that
40+
// pipes a file at the daemon and keeps going for as long as the daemon drains it.
41+
const server = await startServer(async (req, res) => {
42+
if (req.method === 'POST' && req.url === '/upload/preflight') {
43+
await readRequestBody(req);
44+
res.statusCode = 404;
45+
res.end('not found');
46+
return;
47+
}
48+
if (req.method === 'POST' && req.url === '/upload') {
49+
req.on('data', (chunk: Buffer) => {
50+
sawBytes += chunk.length;
51+
// The lease dies once bytes are genuinely in flight — not before the request starts.
52+
// Stopping the read applies backpressure so the rest of the file cannot race through
53+
// loopback and make "stopped early" a matter of timing.
54+
req.pause();
55+
if (!control.signal.aborted) control.abort();
56+
});
57+
req.on('close', connectionEnded);
58+
return;
59+
}
60+
res.statusCode = 404;
61+
res.end('not found');
62+
});
63+
64+
try {
65+
await assert.rejects(
66+
async () =>
67+
await uploadArtifact({
68+
localPath: artifactPath,
69+
baseUrl: server.baseUrl,
70+
token: TEST_TOKEN,
71+
signal: control.signal,
72+
}),
73+
);
74+
assert.ok(sawBytes > 0, 'the upload had started streaming before the abort');
75+
assert.ok(sawBytes < content.length, `the stream stopped early, at ${sawBytes} bytes`);
76+
await ended;
77+
} finally {
78+
await server.close();
79+
}
80+
});
81+
82+
test('an aborted signal before preflight refuses to ask the daemon for a ticket', async () => {
83+
const artifactPath = createTempFile('app.apk', 'payload');
84+
const control = new AbortController();
85+
control.abort();
86+
const requests: string[] = [];
87+
88+
const server = await startServer(async (req, res) => {
89+
requests.push(`${req.method} ${req.url}`);
90+
res.statusCode = 404;
91+
res.end('not found');
92+
});
93+
94+
try {
95+
await assert.rejects(
96+
async () =>
97+
await uploadArtifact({
98+
localPath: artifactPath,
99+
baseUrl: server.baseUrl,
100+
token: TEST_TOKEN,
101+
signal: control.signal,
102+
}),
103+
);
104+
assert.deepEqual(requests, [], 'an upload nobody waits for never reaches the daemon');
105+
} finally {
106+
await server.close();
107+
}
108+
});
109+
110+
test('an upload with no signal behaves exactly as before', async () => {
111+
const content = 'unprotected-payload';
112+
const artifactPath = createTempFile('app.apk', content);
113+
const expectedHash = createHash('sha256').update(content).digest('hex');
114+
115+
const server = await startServer(async (req, res) => {
116+
if (req.method === 'POST' && req.url === '/upload/preflight') {
117+
await readRequestBody(req);
118+
res.statusCode = 404;
119+
res.end('not found');
120+
return;
121+
}
122+
if (req.method === 'POST' && req.url === '/upload') {
123+
assert.equal(req.headers['x-artifact-hash'], expectedHash);
124+
await readRequestBody(req);
125+
sendJson(res, { ok: true, uploadId: 'upload-uncancelled' });
126+
return;
127+
}
128+
res.statusCode = 404;
129+
res.end('not found');
130+
});
131+
132+
try {
133+
const uploadId = await uploadArtifact({
134+
localPath: artifactPath,
135+
baseUrl: server.baseUrl,
136+
token: TEST_TOKEN,
137+
});
138+
assert.equal(uploadId, 'upload-uncancelled');
139+
} finally {
140+
await server.close();
141+
}
142+
});
143+
144+
function createTempFile(filename: string, content: string | Buffer): string {
145+
const dir = mkdtempForTestSync('agent-device-upload-cancel-');
146+
tempDirs.push(dir);
147+
const filePath = path.join(dir, filename);
148+
fs.writeFileSync(filePath, content);
149+
return filePath;
150+
}
151+
152+
async function startServer(
153+
handler: (req: IncomingMessage, res: ServerResponse) => Promise<void>,
154+
): Promise<{ baseUrl: string; close: () => Promise<void> }> {
155+
const server = http.createServer((req, res) => {
156+
void handler(req, res).catch(() => {
157+
res.statusCode = 500;
158+
res.end();
159+
});
160+
});
161+
// A canceled upload leaves its socket in a state `closeAllConnections` can race; without a bound
162+
// idle keep-alive, `close` would then wait out Node's five-second default.
163+
server.keepAliveTimeout = 100;
164+
server.listen(0, '127.0.0.1');
165+
server.unref();
166+
await once(server, 'listening');
167+
const address = server.address();
168+
assert.ok(address && typeof address === 'object');
169+
return {
170+
baseUrl: `http://127.0.0.1:${address.port}`,
171+
close: async () => {
172+
// A canceled upload leaves its socket half-open and a pooled keep-alive one behind; without
173+
// dropping them, `close` would wait out the server's five-second keep-alive timeout.
174+
server.closeAllConnections();
175+
await new Promise<void>((resolve, reject) => {
176+
server.close((error) => (error ? reject(error) : resolve()));
177+
});
178+
},
179+
};
180+
}
181+
182+
async function readRequestBody(req: IncomingMessage): Promise<Buffer> {
183+
const chunks: Buffer[] = [];
184+
for await (const chunk of req) {
185+
chunks.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk));
186+
}
187+
return Buffer.concat(chunks);
188+
}
189+
190+
function sendJson(res: ServerResponse, body: unknown): void {
191+
res.statusCode = 200;
192+
res.setHeader('content-type', 'application/json');
193+
res.end(JSON.stringify(body));
194+
}

0 commit comments

Comments
 (0)