Skip to content

Commit bad5525

Browse files
committed
fix(daemon): let admitted work keep the lease it is working on
A lease renewed only at admission, so a command slower than its own inactivity TTL expired the lease that was paying for the device it was using. Expiry then tore the provider session down underneath a client still waiting for that same command's result, and every later command on the session reported a lease that was no longer active. The session's own work was the thing that killed it. Admitted work now preserves its lease the way a human-control hold does: while the request is still wanted it defers expiry, and finishing while still wanted renews the lease for its existing TTL from the moment the work ended. Work whose client hung up preserves nothing — it neither defers expiry past that cancellation nor renews the lease when it finally lands — so a handler that ignores its cancellation cannot hold a rented device open. Found while investigating #2509. Not its reported mechanism: a cloud WebDriver connection profile asks for a ten-minute lease, so a one-minute hang cannot starve it. This reaches leases on the daemon's one-minute default.
1 parent 513df98 commit bad5525

9 files changed

Lines changed: 432 additions & 12 deletions

‎docs/adr/0007-remote-device-leases.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,22 @@ owning lease expiry.
5151
Backend-only leases remain valid for older remote clients, while provider-aware
5252
clients get device-level contention and clearer recovery.
5353

54+
## Admitted request work
55+
56+
A lease renews when a request is admitted and never again while that request runs, so an admitted
57+
command slower than the inactivity TTL used to expire the lease that was paying for its own device
58+
and tear the session down underneath the client still waiting for its result. Admitted work
59+
therefore preserves its lease the way a human-control hold does: while the request is still wanted
60+
it defers expiry, and finishing while still wanted renews the lease for its existing inactivity TTL
61+
from the moment the work ended. A request whose client hung up preserves nothing — it neither defers
62+
expiry past that cancellation nor renews the lease when it finally lands — so a handler that ignores
63+
its cancellation cannot hold a rented device open.
64+
65+
Which leases this reaches depends on the inactivity TTL the client asked for: the daemon default is
66+
one minute, while a cloud WebDriver connection profile asks for ten. A single command that runs
67+
longer than its own lease is therefore ordinary on the default and only reachable through a profile
68+
on the longer one.
69+
5470
## Human control
5571

5672
Human-control holds coexist with an open remote session. They belong to `LeaseRegistry` and use
Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
import { test, expect } from 'vitest';
2+
import { LeaseInFlightWorkRegistry } from '../lease-in-flight-work.ts';
3+
4+
test('a pass defers its lease until it is released', () => {
5+
const work = new LeaseInFlightWorkRegistry();
6+
const pass = work.retain('lease-a', () => true);
7+
8+
expect(work.isDeferred('lease-a')).toBe(true);
9+
expect(pass.release()).toBe(true);
10+
expect(work.isDeferred('lease-a')).toBe(false);
11+
});
12+
13+
// The deferral is a claim that somebody is still waiting. The client hanging up
14+
// ends the claim immediately, without waiting for the abandoned work to unwind.
15+
test('a pass whose request was cancelled stops deferring unreleased', () => {
16+
const work = new LeaseInFlightWorkRegistry();
17+
let wanted = true;
18+
const pass = work.retain('lease-a', () => wanted);
19+
wanted = false;
20+
21+
expect(work.isDeferred('lease-a')).toBe(false);
22+
expect(pass.release()).toBe(false);
23+
});
24+
25+
// Two requests can work one leased device. Only work still wanted defers, and one
26+
// release must not disturb a pass that outlives it.
27+
test('one wanted pass keeps deferral while an abandoned sibling releases', () => {
28+
const work = new LeaseInFlightWorkRegistry();
29+
let abandonedWanted = true;
30+
const abandoned = work.retain('lease-a', () => abandonedWanted);
31+
const wanted = work.retain('lease-a', () => true);
32+
abandonedWanted = false;
33+
34+
expect(abandoned.release()).toBe(false);
35+
expect(work.isDeferred('lease-a')).toBe(true);
36+
expect(wanted.release()).toBe(true);
37+
expect(work.isDeferred('lease-a')).toBe(false);
38+
});
39+
40+
test('releasing a pass twice renews nothing twice', () => {
41+
const work = new LeaseInFlightWorkRegistry();
42+
const pass = work.retain('lease-a', () => true);
43+
44+
expect(pass.release()).toBe(true);
45+
expect(pass.release()).toBe(false);
46+
});
47+
48+
test('passes on different leases defer independently', () => {
49+
const work = new LeaseInFlightWorkRegistry();
50+
const other = work.retain('lease-b', () => true);
51+
52+
expect(work.isDeferred('lease-a')).toBe(false);
53+
expect(work.isDeferred('lease-b')).toBe(true);
54+
other.release();
55+
});

‎src/daemon/__tests__/lease-registry.test.ts‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -564,3 +564,70 @@ test('canceling a superseded activation cannot remove its successor or another h
564564
);
565565
assert.equal(registry.listHumanControlHolds(authority)[0]?.reason, 'successor');
566566
});
567+
568+
// #2509: nothing heartbeats a lease while its request works, so an admitted
569+
// capture that legitimately outruns the lease TTL expired its own lease and lost
570+
// the device it was still holding. In-flight work defers expiry and renews on
571+
// completion, exactly as a human-control hold does.
572+
test('in-flight request work defers expiry and renews the lease when it finishes', async () => {
573+
let now = 0;
574+
const registry = new LeaseRegistry({ now: () => now, defaultLeaseTtlMs: 5_000 });
575+
const lease = registry.allocateLease(HUMAN_CONTROL_LEASE_REQUEST);
576+
const pass = registry.retainLeaseWork(lease, () => true);
577+
578+
now = 9_000;
579+
assert.deepEqual(registry.consumeExpiredLeases(), []);
580+
assert.equal(registry.consumeExpiredLease(lease.leaseId), undefined);
581+
assert.equal(registry.listActiveLeases()[0]?.leaseId, lease.leaseId);
582+
583+
pass.release();
584+
assert.equal(registry.listActiveLeases()[0]?.expiresAt, 14_000);
585+
now = 14_000;
586+
assert.equal(registry.consumeExpiredLeases()[0]?.leaseId, lease.leaseId);
587+
});
588+
589+
// The deferral belongs to work somebody is still waiting for. A request whose
590+
// client hung up stops deferring the moment it is cancelled, and re-earns nothing
591+
// when it finally lands, so a handler that ignores cancellation cannot pin a
592+
// rented device open forever.
593+
test('a cancelled request stops deferring expiry and cannot revive its lease', async () => {
594+
let now = 0;
595+
const registry = new LeaseRegistry({ now: () => now, defaultLeaseTtlMs: 5_000 });
596+
const lease = registry.allocateLease(HUMAN_CONTROL_LEASE_REQUEST);
597+
let wanted = true;
598+
const pass = registry.retainLeaseWork(lease, () => wanted);
599+
600+
wanted = false;
601+
now = 9_000;
602+
assert.deepEqual(
603+
registry.consumeExpiredLeases().map((entry) => entry.leaseId),
604+
[lease.leaseId],
605+
);
606+
pass.release();
607+
assert.deepEqual(registry.listActiveLeases(), []);
608+
});
609+
610+
// The closest negative: two requests on one device, one abandoned. Releasing the
611+
// abandoned work must renew nothing while the wanted work still defers expiry.
612+
test('abandoned work renews nothing while work still wanted holds the lease', async () => {
613+
let now = 0;
614+
const registry = new LeaseRegistry({ now: () => now, defaultLeaseTtlMs: 5_000 });
615+
const lease = registry.allocateLease(HUMAN_CONTROL_LEASE_REQUEST);
616+
let abandonedWanted = true;
617+
const abandoned = registry.retainLeaseWork(lease, () => abandonedWanted);
618+
const wanted = registry.retainLeaseWork(lease, () => true);
619+
620+
abandonedWanted = false;
621+
now = 9_000;
622+
assert.equal(registry.listActiveLeases()[0]?.leaseId, lease.leaseId);
623+
624+
abandoned.release();
625+
assert.equal(
626+
registry.listActiveLeases()[0]?.expiresAt,
627+
5_000,
628+
'abandoned work must not renew the lease it was sitting on',
629+
);
630+
631+
wanted.release();
632+
assert.equal(registry.listActiveLeases()[0]?.expiresAt, 14_000);
633+
});

‎src/daemon/__tests__/request-execution-scope.test.ts‎

Lines changed: 58 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -509,8 +509,53 @@ test('expired leases remove owned sessions before the next command and free capa
509509
expect(nextLease.tenantId).toBe('tenant-b');
510510
});
511511

512+
// #2509: the slowest command in a cloud session killed every command after it.
513+
// Nothing heartbeats a lease while its request works, so one capture that ran
514+
// past the lease TTL expired its own lease, which tore the provider session down
515+
// under the client that was still waiting for it.
516+
test('an admitted request that outlives the lease TTL keeps its lease and session', async () => {
517+
let now = 1_000;
518+
const sessionStore = makeSessionStore('agent-device-request-scope-');
519+
const leaseRegistry = new LeaseRegistry({
520+
defaultLeaseTtlMs: 10,
521+
minLeaseTtlMs: 1,
522+
now: () => now,
523+
});
524+
const lease = leaseRegistry.allocateLease({ tenantId: 'tenant-a', runId: 'run-1' });
525+
sessionStore.set(
526+
'default',
527+
makeIosSession('default', {
528+
lease: {
529+
leaseId: lease.leaseId,
530+
tenantId: lease.tenantId,
531+
runId: lease.runId,
532+
leaseBackend: lease.backend,
533+
expiresAt: lease.expiresAt,
534+
},
535+
}),
536+
);
537+
538+
const slow = await createRequestExecutionScope({
539+
req: makeRequest({ command: 'snapshot' }),
540+
sessionStore,
541+
leaseRegistry,
542+
});
543+
// The capture is still being waited on when it crosses the TTL, as a cloud
544+
// page-source read does on a screen that never goes idle.
545+
expect(await slow.runLocked(async () => (now = 1_011))).toBe(1_011);
546+
547+
const next = await createRequestExecutionScope({
548+
req: makeRequest({ command: 'screenshot' }),
549+
sessionStore,
550+
leaseRegistry,
551+
});
552+
expect(await next.runLocked(async () => 'ran')).toBe('ran');
553+
expect(sessionStore.get('default')).toBeDefined();
554+
});
555+
512556
test('expired leased session cleanup waits for the request execution lock', async () => {
513557
let now = 1_000;
558+
const requestId = 'request-scope-holds-execution-lock';
514559
const sessionStore = makeSessionStore('agent-device-request-scope-');
515560
const leaseRegistry = new LeaseRegistry({
516561
defaultLeaseTtlMs: 10,
@@ -531,7 +576,7 @@ test('expired leased session cleanup waits for the request execution lock', asyn
531576
}),
532577
);
533578
const first = await createRequestExecutionScope({
534-
req: makeRequest({ command: 'click' }),
579+
req: makeRequest({ command: 'click', meta: { requestId } }),
535580
sessionStore,
536581
leaseRegistry,
537582
});
@@ -554,16 +599,24 @@ test('expired leased session cleanup waits for the request execution lock', asyn
554599
}),
555600
);
556601
await firstEnteredPromise;
602+
// The client walked away from this request. Abandoned work no longer defers the
603+
// expiry it is sitting on, which is what makes this case about the lock and not
604+
// about in-flight lease liveness.
605+
markRequestCanceled(requestId);
557606

558607
now = 1_011;
559608
const secondRun = second.runLocked(async () => 'second');
560609
await new Promise((resolve) => setTimeout(resolve, 20));
561610
expect(sessionStore.get('default')).toBeDefined();
562611

563-
releaseFirst();
564-
await firstRun;
565-
await expect(secondRun).resolves.toBe('second');
566-
expect(sessionStore.get('default')).toBeUndefined();
612+
try {
613+
releaseFirst();
614+
await firstRun;
615+
await expect(secondRun).resolves.toBe('second');
616+
expect(sessionStore.get('default')).toBeUndefined();
617+
} finally {
618+
clearRequestCanceled(requestId);
619+
}
567620
});
568621

569622
test('tenant lease rejection flushes diagnostics into the effective session request log', async () => {
Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
import { test, expect } from 'vitest';
2+
import {
3+
clearRequestAbortRegistration,
4+
markRequestCanceled,
5+
registerRequestAbort,
6+
} from '@agent-device/host-kit/request';
7+
import type { DaemonRequest } from '../daemon-request.ts';
8+
import { LeaseRegistry } from '../lease-registry.ts';
9+
import { runAdmittedLeaseWork } from '../request-lease-work.ts';
10+
import { HUMAN_CONTROL_LEASE_REQUEST } from './human-control-fixtures.ts';
11+
12+
function admittedRequest(
13+
leaseRegistry: LeaseRegistry,
14+
overrides: Partial<DaemonRequest> = {},
15+
): DaemonRequest {
16+
return {
17+
token: 'test-token',
18+
session: 'default',
19+
command: 'snapshot',
20+
positionals: [],
21+
internal: { admittedLease: leaseRegistry.allocateLease(HUMAN_CONTROL_LEASE_REQUEST) },
22+
...overrides,
23+
};
24+
}
25+
26+
// #2509: a capture that ran past the lease TTL expired the lease that was paying
27+
// for the device, so the session died underneath the client still waiting for it.
28+
test('admitted work that outlives the lease TTL keeps the lease it is working on', async () => {
29+
let now = 0;
30+
const leaseRegistry = new LeaseRegistry({ now: () => now, defaultLeaseTtlMs: 5_000 });
31+
const req = admittedRequest(leaseRegistry);
32+
33+
const worked = await runAdmittedLeaseWork({
34+
leaseRegistry,
35+
req,
36+
task: async () => {
37+
now = 9_000;
38+
expect(leaseRegistry.listActiveLeases()).toHaveLength(1);
39+
return 'captured';
40+
},
41+
});
42+
43+
expect(worked).toBe('captured');
44+
expect(leaseRegistry.listActiveLeases()[0]?.expiresAt).toBe(14_000);
45+
});
46+
47+
// The pass protects work somebody is still waiting for, and nothing else. Once the
48+
// client hangs up, work that lands later re-earns no lease: a handler that ignores
49+
// its cancellation cannot pin a rented device open.
50+
test('work nobody waited for renews nothing once its lease fell due', async () => {
51+
let now = 0;
52+
const leaseRegistry = new LeaseRegistry({ now: () => now, defaultLeaseTtlMs: 5_000 });
53+
const requestId = 'request-2509-abandoned';
54+
const registration = registerRequestAbort(requestId);
55+
try {
56+
const req = admittedRequest(leaseRegistry, { meta: { requestId } });
57+
const worked = await runAdmittedLeaseWork({
58+
leaseRegistry,
59+
req,
60+
task: async () => {
61+
now = 9_000;
62+
markRequestCanceled(requestId);
63+
return 'late';
64+
},
65+
});
66+
67+
expect(worked).toBe('late');
68+
expect(leaseRegistry.listActiveLeases()).toEqual([]);
69+
} finally {
70+
clearRequestAbortRegistration(registration);
71+
}
72+
});
73+
74+
test('a request admitted without a lease runs its work untouched', async () => {
75+
const leaseRegistry = new LeaseRegistry();
76+
const result = await runAdmittedLeaseWork({
77+
leaseRegistry,
78+
req: { token: 'test-token', session: 'default', command: 'status', positionals: [] },
79+
task: async () => 'ran',
80+
});
81+
82+
expect(result).toBe('ran');
83+
expect(leaseRegistry.listActiveLeases()).toEqual([]);
84+
});

‎src/daemon/lease-in-flight-work.ts‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
/**
2+
* Which leased devices have admitted request work running on them right now.
3+
*
4+
* A remote lease renews when a request is admitted and never again while that
5+
* request works, so work that legitimately outlives the lease TTL — a cloud
6+
* page-source read on a screen that never goes idle (#2509) — expired the very
7+
* lease paying for the device, and the session with it.
8+
*/
9+
10+
/** Whether the client behind one request is still waiting for its result. */
11+
export type LeaseWorkWanted = () => boolean;
12+
13+
/** A claim on one leased device for the duration of one admitted request's work. */
14+
export type LeaseWorkPass = Readonly<{
15+
leaseId: string;
16+
/**
17+
* Ends the pass. Reports whether the work was still wanted when it ended, which
18+
* is what entitles it to renew the lease it just worked on.
19+
*/
20+
release(): boolean;
21+
}>;
22+
23+
type LeaseWorkEntry = {
24+
readonly wanted: LeaseWorkWanted;
25+
released: boolean;
26+
};
27+
28+
/**
29+
* A pass defers its lease's expiry for as long as the request that opened it is
30+
* still wanted. The moment that client hangs up the pass defers nothing, so a
31+
* handler that ignores its cancellation cannot hold a rented device open.
32+
*/
33+
export class LeaseInFlightWorkRegistry {
34+
private readonly entriesByLeaseId = new Map<string, Set<LeaseWorkEntry>>();
35+
36+
retain(leaseId: string, wanted: LeaseWorkWanted): LeaseWorkPass {
37+
const entry: LeaseWorkEntry = { wanted, released: false };
38+
const entries = this.entriesByLeaseId.get(leaseId) ?? new Set<LeaseWorkEntry>();
39+
entries.add(entry);
40+
this.entriesByLeaseId.set(leaseId, entries);
41+
return { leaseId, release: () => this.releasePass(entries, entry) };
42+
}
43+
44+
/** True while any pass on this lease is still wanted. Unwanted passes defer nothing. */
45+
isDeferred(leaseId: string): boolean {
46+
const entries = this.entriesByLeaseId.get(leaseId);
47+
if (!entries) return false;
48+
for (const entry of entries) {
49+
if (entry.wanted()) continue;
50+
entries.delete(entry);
51+
}
52+
if (entries.size > 0) return true;
53+
this.entriesByLeaseId.delete(leaseId);
54+
return false;
55+
}
56+
57+
/** Idempotent: only the first release of a pass can report its work as wanted. */
58+
private releasePass(entries: Set<LeaseWorkEntry>, entry: LeaseWorkEntry): boolean {
59+
if (entry.released) return false;
60+
entry.released = true;
61+
entries.delete(entry);
62+
return entry.wanted();
63+
}
64+
}

0 commit comments

Comments
 (0)