Skip to content

Commit 915ae73

Browse files
authored
fix(remote): renew the lease while an artifact upload runs (#2946) (#2990)
* fix(lease): renew a heartbeat with no ttl for the window the lease already carries `heartbeatLease` resolved an absent `ttlMs` through the registry's default resolver, so a caller that heartbeats without repeating its allocation TTL silently shortened the lease to the daemon default. Every renewal that is not asked to change the window now renews for the window the lease is living on, which is the rule `refreshProtectedLease` already applied to protected work; that arithmetic moves to `leaseOwnTtlMs` beside the other lease-scope rules so both callers read one definition. The caller that heartbeats without a TTL is not asking for the default, it is asking for the same lease to keep going. An admitted request does exactly that, and this is half of why a long upload expired the lease paying for its own device (#2946). * fix(remote): renew the lease while an artifact upload runs on the caller's side A remote install uploaded the artifact from the caller before the install request was admitted, so nothing renewed the lease while the bytes moved: the daemon protects a lease while admitted work runs on it (#2509, ADR 0007), and an upload is the mirror image of that. A 449 MB APK that took 1m47s to upload against the one-minute default TTL expired the lease paying for the device it was uploading to, and the install then failed `Lease is not active` (#2946). `sendToDaemon` now brackets the upload phase with beats over the same transport the command uses. A beat names the command's lease scope, its own request id, and nothing else: the scope is what the daemon needs, and reusing the install request would send the upload's own payload once per beat. Each beat gets a fresh id because a beat that times out is canceled under its own. Only a remote daemon uploads and only a remote daemon holds a billed device, so a local command and a command that names no lease get no timer at all, and the interval starts rather than fires immediately, so an install that beats never sends no extra request. A beat that finds the lease gone ends the upload with that lease error rather than finishing bytes to a device nobody owns; a beat that fails for any other reason is reported through diagnostics and survived, since a later beat covers one lost request. ADR 0007 gains the rule this closes: what protects work that happens before admission. * fix(lease): renew an admitted request for the window its lease carries Admission named its own proxy default on every request, so a client that rented a device for longer than that default lost the window on the next command — the lease-side twin of the heartbeat bug in #2946, which the client fix could not reach because the shortening happened at admission. The window a lease carries is the one its client named when it allocated; only a request naming its own window changes it now, which retires DEFAULT_PROXY_LEASE_TTL_MS with no producer left. * fix(remote): beat to the lease's own window and stop the upload with it The upload beat waited a fixed twenty seconds for its first renewal, so a lease admitted with a shorter window lapsed while a perfectly valid upload ran — the #2946 symptom one seam earlier. A beat answers with the window it just renewed, so the phase now beats immediately and then a third of that window after each beat lands: a beat slower than the cadence delays its successor instead of replacing the schedule or silencing every beat behind it, and a floor keeps a pathological window from becoming a request loop. A beat refused for a missing or mismatched owner scope says this request can never renew the lease, so retrying it only spent the upload against a lease that had stopped renewing. Those reasons now end the phase beside the lost-lease ones, and the phase hands its task an abort signal the upload chain honours: preflight, finalize, the direct PUT, and the legacy stream all run under the caller's cancellation combined with their own timeout, and a canceled upload stops asking the daemon for a fresh ticket. node:http's own request signal is what stops bytes already piped at a device nobody holds. * chore(gates): ack the upload wire digests the abort signal moved Every moved declaration is a client-side optional parameter — the request bytes an older daemon parses are unchanged, and an aborted request is the client disconnect it already handles — so each gets its own digest-keyed compatibleChanges entry rather than a protocol bump. AbortSignal joins TS_GLOBALS: a lib global with no declaration site to digest, like URL. * fix(remote): treat a beat the daemon rejects on its own terms as terminal The beat's scope and window are fixed where it is built, so a daemon that refuses them — a ttl outside [minLeaseTtlMs, maxLeaseTtlMs], an unusable lease id — refuses every successor identically. Such a refusal carries no reason to key on, so the code is the signal: end the phase the way a lost lease does instead of spending the upload against a lease that stopped renewing on the first beat. * test(remote): drop the connection-close wait that hung the cancellation test The byte counts are what prove the stream stopped early; waiting for the server to see the socket close added a second promise that only loopback timing could settle, and under a loaded coverage lane it never did inside the test timeout. A smaller payload keeps the same pause-and-abort scenario off the CPU. * fix(remote): reject an aborted upload with the reason it was aborted The contract on the upload signal is that an aborted upload rejects with the signal's own reason. The preflight leg honoured it and the stream leg did not: node:http reports an aborted request as a transport error, and the shared handler wrapped that into COMMAND_FAILED. That is not just the wrong message — a wrapped cancellation is indistinguishable from a broken transport, so the direct-upload retry policy is one refactor away from re-preflighting for a fresh ticket after the caller asked for none. Finalize wrapped the same way. Both tests now pin the rejection instead of accepting anything, and the admission test pins heartbeatAt so the window assertion cannot pass on a lease that was never renewed. * chore(gates): ack the two upload digests the abort-reason fix moved finalizeDirectUpload and streamFileToHttpRequestAttempt changed shape; both edits are client-side only — the bytes a protocol-2 daemon parses for an uncancelled request are unchanged, and a canceled request is the disconnect it already handles. * fix(remote): give each lease beat a budget and prove the beat on the real route A beat was sent under the lease_heartbeat command policy's 90s timeout and its successor was armed only after it settled, so one stalled round trip stopped the renewals for up to 90s while the lease's 60s window lapsed underneath the upload and the install RPC waited behind it. Each beat now carries a budget no longer than the cadence it started on and arms its successor at beat start, so a beat that never settles is abandoned on schedule; nothing awaits an outstanding beat when the phase settles. The beat's cadence is gone with it: the loop beats at the registry's floor until a beat names the window, and an answer that names no window keeps the cadence rather than slowing down on the absence of evidence. The lease-lost reasons move to contracts as one taxonomy shared with the connection runtime, and LEASE_SESSION_MISMATCH leaves the terminal set: only request admission raises it and lease_heartbeat is admission-exempt. Two suites back it. The beat loop is pinned against fake timers for the stalled beat, its budget, and an abandoned beat that answers too late to matter. remote-upload-lease-beat.test.ts drives an install through sendToDaemon against a fake remote daemon, so the wiring between the loop, the beat it is handed, and the upload is covered, not just each of them alone: the first holds the upload until a second beat names the lease and asserts a renewed artifact lands on a live lease before the install RPC; the second answers a beat with LEASE_NOT_FOUND and asserts the artifact is destroyed mid-flight, is not left to drain once the daemon stops applying backpressure, and no install RPC goes out. Killing either half of that wiring fails them: dropping the heartbeat shows no beat, and dropping the signal hands the daemon a drained artifact. * refactor(remote): move the lease beat to its own module runProtectedLeaseWork, the beat that keeps a remote lease alive across a client-side phase, and the request it sends were three of daemon-client.ts's four exports and existed there only so their tests could reach them. They move to daemon-client-lease-beat.ts, which leaves the transport entry module 287 lines and knowing nothing about cadence, budgets, or which lease reasons are terminal. The test moves with them and is renamed to match. The beat family is a pure move: no behaviour change, and the beat's transport is still the send it is handed, so the module stays free of transport decisions. * chore(gates): record the lease-beat module's type-only read of the wire request R78 names measured edges, never a directory, so the extracted module owes its own entry: a type-only import of the same DaemonRequest vocabulary the four sibling client modules already record. * docs(leases): state what the first beat actually buys, and what an absent ttlMs keeps ADR 0007 and the loop's JSDoc both claimed a missing lease is caught before any bytes move. Hashing, the preflight, and the start of the stream can all run while the opening beat is still outstanding, so the claim was stronger than the behaviour: the first beat buys learning the loss during the upload, not before it. The same section described the successor as armed a third of a window after the beat lands, which the budget change just superseded; it now says the successor is armed at beat start and the budget is capped at the cadence, and the survived- failure paragraph says that promise covers a beat abandoned at its budget, not only one that fails fast — the distinction the review found in the old shape. remote-proxy.md promises five minutes of proxy lease inactivity for CLI connect. That holds because `open` names the window; a lease allocated over the RPC with no ttlMs keeps the daemon's one-minute default, so the sentence now says which one a reader gets. * test(remote): let the caller release the stalled upload, so the outcome is causal The lost-lease case waited on whichever of two events reached the fake daemon first, with a grace window deciding an inconclusive answer. A client blocked in a kernel write cannot observe its own cancellation until the peer drains, so that window had to be longer than the drain — timing the test had to win rather than a fact it could state. The fake daemon now holds the artifact until the test releases it, and only releases it after sendToDaemon has already rejected. An upload nobody canceled has nothing left that could stop it by then and drains; one that was destroyed cannot. The success case reads the delivered byte count instead of racing the same oracle. Killing either half of the wiring still fails: dropping the heartbeat leaves no beat and both cases time out, dropping the signal hands the daemon a drained artifact. * refactor(remote): fold the beat's thin wrappers into the one factory that needs them collapse createLeaseRenewalBeat and leaseScopeForHeartbeat into buildUploadLeaseHeartbeat, which was their only caller, and let the beat tests read the scope through the contracts helper the factory itself uses. the per-beat request id is now pinned over the real transport, where a shared id is observable, instead of against an injected send. * fix(remote): budget each beat from the window it protects, not the cadence Budgeting a beat at the cadence it started on made one number do two jobs. Until a beat answers, the cadence sits at the 1s floor, so `sendRequest` got a 1s budget, destroyed the request, and emitted a `daemon_request_timeout`. That answer is non-terminal, so the cadence never left the floor: on any link where a round trip takes over a second — a tunneled proxy, a fresh TLS connect per beat because every timeout tore the socket down, an uplink the upload was saturating — no beat ever got a window in which to answer, and the lease could lapse on exactly the slow uploads #2946 exists to protect, with a diagnostic pair every second. Cadence and budget are two numbers now. A beat is armed every third of the window and allowed the whole window to answer in, so a slow first answer sets the cadence instead of being cut off before it can say anything. The window the loop assumes before the first answer is the registry's five-second minimum, which is now shared with the client as `MIN_LEASE_WINDOW_MS` rather than assumed twice: planning against a longer window than the daemon would ever grant is the same mistake pointed the other way. The slow-link case is covered through `sendToDaemon`, which no loopback unit test travels: the fake daemon answers a beat after 3s — longer than the assumed cadence, shorter than the assumed window, so a beat budgeted at the cadence is cut off and one budgeted at the window is not. It asserts the beat stops arriving once the window is known and counts no `daemon_request_timeout` inside the upload, in a diagnostics scope because outside one every count reads zero for the wrong reason. Verified against the previous beat loop: five arrivals there, none of them answered in time. * fix(remote): load the lease beat only for the request that can use it The extraction that moved the beat out of daemon-client.ts put it in cli.ts's eager closure, which every command pays for, and took it from 295 modules to 296 (#2990's coverage failure). The beat is only ever useful to one kind of request: one heading to a remote daemon that names a lease. Both guards are cheap and local — the lease scope comes off the request the caller already built — so the module now loads behind them and the closure is back to 295. Also folds the beat's thin wrappers away from the entry surface, which is what the four test-only exports in daemon-client.ts had become: createLeaseRenewalBeat and leaseScopeForHeartbeat existed only for tests, the scope precedence they pinned is already covered where it lives in contracts, and the one invariant that wasn't covered anywhere — a beat per request id, so a timed-out beat cannot cancel its successor — is now pinned over the real transport, where a shared id is actually observable. * test(remote): give the slow-beat scenario the lane's parallel-scenario timeout The provider-integration project sets no `testTimeout`, so the scenario ran under Vitest's 5s default and the Coverage job's full-suite run — heavier than a lane-local one — tripped it: the slow-link case spends 4.5s watching a real lease window, and that body has to sit past the assumed cadence so the opening beat's late answer can move the loop off it. Under the documented parallel import tail the worst case approached the default, and CI drew first. Uses the project's own PARALLEL_PROVIDER_SCENARIO_TIMEOUT_MS, which exists for scenarios that build a real in-process daemon path and get timed out by setup contention rather than by their own work. The observation window is untouched: shortening it would cut the margin between the cadence-budgeted beat (three arrivals, each timed out) and the window-budgeted one (the answer lands), which is the only thing this test reads. * test(remote): center the slow-beat number and un-detach the window doc Two non-blocking notes from the 4c74a43 pass. SLOW_BEAT_MS only discriminates the window-budgeted beat from the cadence-budgeted one inside a narrow band: slower than the pre-window cadence (1666ms) so a beat held to that cadence is cut off before the answer lands, and fast enough to answer before that cadence arms a third beat (3332ms). At 3000ms the late answer had ~332ms of margin to the buggy third arrival — enough to flake on a loaded runner. 2500ms is the center of the band; verified by mutation, budgeting the beat at its cadence now trips "beats seen: 3". Moved the leaseWindowFromHeartbeatResponse doc off leaseBeatIntervalMs and onto the function it describes, and reordered the renewed-window guard to match the undefined guard above it. No behavior change. * test(remote): state the beat's transport cap, and fail the slow case on its cause The ADR told slow links they get the whole window as a beat's answer deadline. The implementation takes the shorter of the window and the heartbeat's request policy, so a ten-minute lease caps at ninety seconds and the paragraph promised a budget the code never grants. Documented rather than removed: the daemon renews when it handles the request, before it answers, so a beat abandoned at the cap still extended the lease when the request landed, and when it didn't a saturated link is not served by a socket held for the rest of a multi-minute window. Shorter windows are unchanged. The slow-link case released its withheld artifact through a module-level global assigned inside the current test's upload handler. A client regression that stops the artifact from ever arriving left the previous test's closure in place, so the call was a no-op and the test hung on `await running` until the lane timeout instead of naming the cause. It now calls the fake daemon's own resumeUpload, which throws when no artifact reached that daemon. Verified by pointing the install at a path that is never uploaded: all three cases fail on that message rather than timing out. * test(remote): settle the slow-beat count on arrivals instead of a wall clock The slow-link case sampled `beatArrivals.length` at a fixed 4.5s and asserted 2. Both directions of that number depended on event-loop timing: a stall over the margin between the first answer landing and the assumed cadence arming a third beat read as three beats, and a stall long enough to keep the second beat out of the sample read as one. Neither says anything about the loop under test. The count now waits for the two beats it expects, then keeps watching through a quiet window of two assumed cadences and returns what arrived. A loop that budgets a beat at its cadence is still arriving throughout that window and is counted, not awaited; a loop that moved to the window it just renewed is silent until ten seconds in. A runner stall delays the read instead of moving a beat across it. Verified by mutation both ways: budgeting the beat at its interval reports 4 beats, and arming the opening beat late fails with "only 0 of 2 beats arrived" rather than hanging to the lane timeout. The assumed cadence is derived from MIN_LEASE_WINDOW_MS instead of restated, so it cannot drift from what the loop actually assumes.
1 parent 99af4e8 commit 915ae73

29 files changed

Lines changed: 2124 additions & 84 deletions

‎docs/adr/0007-remote-device-leases.md‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,56 @@ one minute, while a cloud WebDriver connection profile asks for ten. A single co
6767
longer than its own lease is therefore ordinary on the default and only reachable through a profile
6868
on the longer one.
6969

70+
## Client-side work that precedes admission
71+
72+
Protecting admitted work covers nothing that happens before a request is admitted. Installing an
73+
artifact uploads it from the caller while the request that will consume it has not been admitted yet,
74+
so an upload slower than the lease's inactivity window expired the lease paying for the device the
75+
bytes were going to (#2946). The caller therefore beats the lease over the ordinary transport for as
76+
long as that phase runs, naming the lease scope exactly as the command named it and no payload of its
77+
own. An install names no window, so its beats renew for the window the lease already carries; a
78+
caller that did name one keeps renewing on it. Resolving an absent window to the registry default
79+
instead — which is what a heartbeat used to do — quietly shortened every lease allocated above that
80+
default, which is the other half of why the upload could not survive. Request admission had its own
81+
copy of that mistake: it named a proxy-specific default for every admitted request, so a lease
82+
allocated longer than the default lost its window on the next command. Admission renews on the lease's
83+
window too; the window a lease carries is the one its client named when it allocated.
84+
85+
The first beat is fired when the phase starts, not one cadence in, because a beat is what proves the
86+
lease the upload is spending time on is still alive — a lease shorter than any fixed cadence would
87+
otherwise lapse before anything renewed it. A lease already gone is caught early, but not before the
88+
phase begins: hashing, the preflight, and the start of the stream can all run while the first beat is
89+
still outstanding, so what the first beat buys is that the loss is learned during the upload rather
90+
than after it. Each beat answers with the window it just renewed, and the loop arms its successor when
91+
the beat starts rather than when it settles, so a beat that never answers is abandoned on schedule
92+
instead of taking the schedule with it. A phase that settles does not wait on a beat still in flight.
93+
94+
Cadence and budget are deliberately two numbers. A beat is armed every third of the window, but it is
95+
allowed the whole window to answer in. Budgeting a beat at its cadence instead would give a heartbeat
96+
that needs more than a third of a window for its round trip — a tunneled proxy, an uplink the upload
97+
itself is saturating — no way to answer at all, and every beat would re-time-out on exactly the slow
98+
links the beat exists to protect. The window, not the cadence, is the interval in which being alive
99+
still matters, so it is also the longest a beat's answer is worth waiting for.
100+
101+
That ceiling is the shorter of the window and the heartbeat's own request policy, currently ninety
102+
seconds. A ten-minute lease therefore gives a beat ninety seconds rather than the full window, which
103+
costs nothing the beat exists to buy: the daemon renews the lease when it handles the request, before
104+
it answers, so a round trip abandoned at the cap has still extended the lease if the request ever
105+
landed — and if it never landed, a transport that has stopped carrying traffic is not served by holding
106+
the socket open for the rest of a window measured in minutes. What the cap refuses is exactly that
107+
idle hold. It binds only on windows longer than the policy, and on those the beat already has the
108+
longest answer deadline any lease gets; shorter windows are bounded by their own window, as above.
109+
110+
A beat is a fresh request each time, never the protected request rewritten: a request identity is
111+
what a timed-out beat is canceled under, and beats must not inherit each other's cancellation. A beat
112+
that finds the lease gone, or finds that this request can never renew it — its scope is missing or
113+
belongs to another lease, or the daemon rejects the scope and window the beat itself was built with,
114+
which no successor will ask differently — ends the phase with that error and cancels the upload rather
115+
than finishing bytes against a device nobody owns or a lease that will stop renewing. A beat that fails
116+
for any other reason is reported and survived, because a later beat covers one lost request — including
117+
one that was abandoned at its budget rather than answered, which is what makes that promise hold for a
118+
stalled round trip and not only for one that fails fast.
119+
70120
## Human control
71121

72122
Human-control holds coexist with an open remote session. They belong to `LeaseRegistry` and use

‎packages/contracts/src/__tests__/lease-scope.test.ts‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ import assert from 'node:assert/strict';
22
import { test } from 'vitest';
33
import {
44
findMissingProxyLeaseFields,
5+
isInactiveLeaseError,
56
leaseScopeFromOptions,
67
leaseScopeFromRequest,
78
leaseScopeToCommandFlags,
@@ -215,3 +216,29 @@ test('readLeaseAllocateProviderFlags carries the provider-allocation flags and d
215216
);
216217
assert.deepEqual(readLeaseAllocateProviderFlags(undefined), {});
217218
});
219+
220+
test('isInactiveLeaseError recognizes the lease being gone, and refuses a request mismatch', () => {
221+
for (const reason of ['LEASE_NOT_FOUND', 'LEASE_EXPIRED', 'LEASE_REVOKED']) {
222+
assert.equal(
223+
isInactiveLeaseError({ code: 'UNAUTHORIZED', details: { reason } }),
224+
true,
225+
`${reason} says the lease is no longer usable`,
226+
);
227+
}
228+
// A mismatch is about the request that asked, not the lease: the lease may be perfectly alive and
229+
// held by this same client under another request, so a beat must keep renewing on this answer.
230+
assert.equal(
231+
isInactiveLeaseError({ code: 'UNAUTHORIZED', details: { reason: 'LEASE_SCOPE_MISMATCH' } }),
232+
false,
233+
'a scope mismatch is not the lease being gone',
234+
);
235+
// Both halves are required: the code alone is any unauthorized call, and the reason alone could
236+
// ride on an error the client has no business acting on.
237+
assert.equal(isInactiveLeaseError({ code: 'UNAUTHORIZED', details: {} }), false);
238+
assert.equal(
239+
isInactiveLeaseError({ code: 'COMMAND_FAILED', details: { reason: 'LEASE_NOT_FOUND' } }),
240+
false,
241+
);
242+
assert.equal(isInactiveLeaseError(new Error('LEASE_NOT_FOUND')), false);
243+
assert.equal(isInactiveLeaseError(undefined), false);
244+
});

‎packages/contracts/src/lease-scope.ts‎

Lines changed: 35 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,6 @@ import type { CloudProviderProfileFields } from './remote-config-fields.ts';
44
import type { CommandFlags } from './command-flags.ts';
55

66
const PROXY_LEASE_PROVIDER = 'proxy';
7-
export const DEFAULT_PROXY_LEASE_TTL_MS = 300_000;
87

98
const REQUIRED_PROXY_LEASE_FIELDS = [
109
'leaseId',
@@ -296,6 +295,41 @@ export function findMissingProxyLeaseFields(scope: LeaseScope): string[] {
296295
return REQUIRED_PROXY_LEASE_FIELDS.filter((field) => !scope[field]);
297296
}
298297

298+
/**
299+
* The shortest inactivity window a lease can hold by default.
300+
*
301+
* The registry clamps every ttl to this floor, so it is also the worst case a client plans against
302+
* before it has seen a lease answer: work that has to finish inside a lease window it does not know
303+
* yet must assume this one, or it can be planning for a window the daemon would never grant.
304+
*/
305+
export const MIN_LEASE_WINDOW_MS = 5_000;
306+
307+
/**
308+
* Why a lease stopped being ours: it is gone, spent, or taken back.
309+
*
310+
* This is the whole taxonomy of "the lease is no longer usable" as the daemon reports it, and both
311+
* readers ask the same question — a client deciding whether a connection still owns a device, and a
312+
* lease beat deciding whether an upload is still worth finishing. A reason naming a mismatch between
313+
* a request and a lease is not in here: that says something about the request, not the lease.
314+
*/
315+
const INACTIVE_LEASE_REASONS: ReadonlySet<unknown> = new Set([
316+
'LEASE_NOT_FOUND',
317+
'LEASE_EXPIRED',
318+
'LEASE_REVOKED',
319+
]);
320+
321+
/** Whether `error` says the lease is gone rather than merely unavailable to this request. */
322+
export function isInactiveLeaseError(error: unknown): boolean {
323+
return (
324+
typeof error === 'object' &&
325+
error !== null &&
326+
(error as Readonly<{ code?: unknown }>).code === 'UNAUTHORIZED' &&
327+
INACTIVE_LEASE_REASONS.has(
328+
(error as Readonly<{ details?: Readonly<{ reason?: unknown }> }>).details?.reason,
329+
)
330+
);
331+
}
332+
299333
function leaseScopeToScopedRequest(scope: LeaseScope): LeaseScopedRequestScope {
300334
return stripUndefined({
301335
leaseId: scope.leaseId ?? '',

‎scripts/layering/daemon-client-entry.ts‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,11 @@ export const DAEMON_CLIENT_ENTRY_EDGES: readonly DaemonClientEntryEdge[] = [
4343
target: 'src/daemon/daemon-request.ts',
4444
rationale: REQUEST_RATIONALE,
4545
},
46+
{
47+
file: 'src/daemon-client/daemon-client-lease-beat.ts',
48+
target: 'src/daemon/daemon-request.ts',
49+
rationale: REQUEST_RATIONALE,
50+
},
4651
{
4752
file: 'src/daemon-client/daemon-client-progress.ts',
4853
target: 'src/daemon/daemon-request.ts',
Lines changed: 199 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,199 @@
1+
// Ending an artifact upload when the work it serves is over (#2946).
2+
//
3+
// A beat that finds the device's lease gone aborts the upload protecting that lease. An upload is a
4+
// piped `node:http` request, so the only way to stop bytes already in flight is the request's own
5+
// abort signal — a rejection the caller swallows would keep streaming a full app bundle to a device
6+
// nobody owns. Kept out of `upload-client.test.ts`, which is already over the test-file size
7+
// tripwire and may not grow (docs/agents/testing.md).
8+
9+
import { afterEach, test } from 'vitest';
10+
import assert from 'node:assert/strict';
11+
import { createHash } from 'node:crypto';
12+
import fs from 'node:fs';
13+
import http, { type IncomingMessage, type ServerResponse } from 'node:http';
14+
import path from 'node:path';
15+
import { once } from 'node:events';
16+
import { uploadArtifact } from '../remote/upload-client.ts';
17+
import { mkdtempForTestSync } from './test-utils/tmp-dir.ts';
18+
19+
const TEST_TOKEN = 'agent-device-upload-cancel-token';
20+
const tempDirs: string[] = [];
21+
22+
afterEach(async () => {
23+
for (const dir of tempDirs) {
24+
await fs.promises.rm(dir, { recursive: true, force: true }).catch(() => {});
25+
}
26+
tempDirs.length = 0;
27+
});
28+
29+
test('an aborted signal ends a legacy upload mid-stream instead of finishing the bytes', async () => {
30+
// Two megabytes is well past what a paused read lets through: the assertions below are about the
31+
// stream stopping, and a smaller payload keeps that off the CPU in a loaded lane.
32+
const content = Buffer.alloc(2 * 1024 * 1024, 'x');
33+
const artifactPath = createTempFile('app.apk', content);
34+
const control = new AbortController();
35+
let sawBytes = 0;
36+
37+
// Preflight reports itself unsupported so the upload takes the legacy stream, the one path that
38+
// pipes a file at the daemon and keeps going for as long as the daemon drains it.
39+
const server = await startServer(async (req, res) => {
40+
if (req.method === 'POST' && req.url === '/upload/preflight') {
41+
await readRequestBody(req);
42+
res.statusCode = 404;
43+
res.end('not found');
44+
return;
45+
}
46+
if (req.method === 'POST' && req.url === '/upload') {
47+
req.on('data', (chunk: Buffer) => {
48+
sawBytes += chunk.length;
49+
// The lease dies once bytes are genuinely in flight — not before the request starts.
50+
// Stopping the read applies backpressure so the rest of the file cannot race through
51+
// loopback and make "stopped early" a matter of timing.
52+
req.pause();
53+
if (!control.signal.aborted) control.abort();
54+
});
55+
return;
56+
}
57+
res.statusCode = 404;
58+
res.end('not found');
59+
});
60+
61+
try {
62+
await assert.rejects(
63+
async () =>
64+
await uploadArtifact({
65+
localPath: artifactPath,
66+
baseUrl: server.baseUrl,
67+
token: TEST_TOKEN,
68+
signal: control.signal,
69+
}),
70+
isAbortError,
71+
);
72+
assert.ok(sawBytes > 0, 'the upload had started streaming before the abort');
73+
assert.ok(sawBytes < content.length, `the stream stopped early, at ${sawBytes} bytes`);
74+
} finally {
75+
await server.close();
76+
}
77+
});
78+
79+
test('an aborted signal before preflight refuses to ask the daemon for a ticket', async () => {
80+
const artifactPath = createTempFile('app.apk', 'payload');
81+
const control = new AbortController();
82+
control.abort();
83+
const requests: string[] = [];
84+
85+
const server = await startServer(async (req, res) => {
86+
requests.push(`${req.method} ${req.url}`);
87+
res.statusCode = 404;
88+
res.end('not found');
89+
});
90+
91+
try {
92+
await assert.rejects(
93+
async () =>
94+
await uploadArtifact({
95+
localPath: artifactPath,
96+
baseUrl: server.baseUrl,
97+
token: TEST_TOKEN,
98+
signal: control.signal,
99+
}),
100+
isAbortError,
101+
);
102+
assert.deepEqual(requests, [], 'an upload nobody waits for never reaches the daemon');
103+
} finally {
104+
await server.close();
105+
}
106+
});
107+
108+
test('an upload with no signal behaves exactly as before', async () => {
109+
const content = 'unprotected-payload';
110+
const artifactPath = createTempFile('app.apk', content);
111+
const expectedHash = createHash('sha256').update(content).digest('hex');
112+
113+
const server = await startServer(async (req, res) => {
114+
if (req.method === 'POST' && req.url === '/upload/preflight') {
115+
await readRequestBody(req);
116+
res.statusCode = 404;
117+
res.end('not found');
118+
return;
119+
}
120+
if (req.method === 'POST' && req.url === '/upload') {
121+
assert.equal(req.headers['x-artifact-hash'], expectedHash);
122+
await readRequestBody(req);
123+
sendJson(res, { ok: true, uploadId: 'upload-uncancelled' });
124+
return;
125+
}
126+
res.statusCode = 404;
127+
res.end('not found');
128+
});
129+
130+
try {
131+
const uploadId = await uploadArtifact({
132+
localPath: artifactPath,
133+
baseUrl: server.baseUrl,
134+
token: TEST_TOKEN,
135+
});
136+
assert.equal(uploadId, 'upload-uncancelled');
137+
} finally {
138+
await server.close();
139+
}
140+
});
141+
142+
function isAbortError(error: unknown): boolean {
143+
// The contract on `signal` is that an aborted upload rejects with the signal's own reason. Any
144+
// other rejection — a transport failure, the server-error fallback — means the upload stopped for
145+
// a reason this test is not about and would let the abort path regress while staying green.
146+
return error instanceof DOMException && error.name === 'AbortError';
147+
}
148+
149+
function createTempFile(filename: string, content: string | Buffer): string {
150+
const dir = mkdtempForTestSync('agent-device-upload-cancel-');
151+
tempDirs.push(dir);
152+
const filePath = path.join(dir, filename);
153+
fs.writeFileSync(filePath, content);
154+
return filePath;
155+
}
156+
157+
async function startServer(
158+
handler: (req: IncomingMessage, res: ServerResponse) => Promise<void>,
159+
): Promise<{ baseUrl: string; close: () => Promise<void> }> {
160+
const server = http.createServer((req, res) => {
161+
void handler(req, res).catch(() => {
162+
res.statusCode = 500;
163+
res.end();
164+
});
165+
});
166+
// A canceled upload leaves its socket in a state `closeAllConnections` can race; without a bound
167+
// idle keep-alive, `close` would then wait out Node's five-second default.
168+
server.keepAliveTimeout = 100;
169+
server.listen(0, '127.0.0.1');
170+
server.unref();
171+
await once(server, 'listening');
172+
const address = server.address();
173+
assert.ok(address && typeof address === 'object');
174+
return {
175+
baseUrl: `http://127.0.0.1:${address.port}`,
176+
close: async () => {
177+
// A canceled upload leaves its socket half-open and a pooled keep-alive one behind; without
178+
// dropping them, `close` would wait out the server's five-second keep-alive timeout.
179+
server.closeAllConnections();
180+
await new Promise<void>((resolve, reject) => {
181+
server.close((error) => (error ? reject(error) : resolve()));
182+
});
183+
},
184+
};
185+
}
186+
187+
async function readRequestBody(req: IncomingMessage): Promise<Buffer> {
188+
const chunks: Buffer[] = [];
189+
for await (const chunk of req) {
190+
chunks.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk));
191+
}
192+
return Buffer.concat(chunks);
193+
}
194+
195+
function sendJson(res: ServerResponse, body: unknown): void {
196+
res.statusCode = 200;
197+
res.setHeader('content-type', 'application/json');
198+
res.end(JSON.stringify(body));
199+
}

‎src/cli/commands/connection-runtime.ts‎

Lines changed: 1 addition & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ import {
1313
type DeviceInfo,
1414
} from '@agent-device/kernel/device';
1515
import { shouldAgentCdpUseRemoteBridgeUrl } from './agent-cdp.ts';
16+
import { isInactiveLeaseError } from '@agent-device/contracts/lease-scope';
1617
import {
1718
narrowConnectionPlatform,
1819
buildConnectionDeviceKey,
@@ -1040,12 +1041,3 @@ async function heartbeatOrAllocateLease(
10401041
throw error;
10411042
}
10421043
}
1043-
1044-
function isInactiveLeaseError(error: unknown): boolean {
1045-
if (!(error instanceof AppError) || error.code !== 'UNAUTHORIZED') return false;
1046-
return (
1047-
error.details?.reason === 'LEASE_NOT_FOUND' ||
1048-
error.details?.reason === 'LEASE_EXPIRED' ||
1049-
error.details?.reason === 'LEASE_REVOKED'
1050-
);
1051-
}

0 commit comments

Comments
 (0)