Skip to content

Commit b6aa27b

Browse files
thymikeeclaude
andcommitted
fix(providers): stop a released Revyl start without waiting out readiness
A release, recovery or shutdown that took over a lease mid-start waited for the start to finish readiness before it could stop the device, up to Revyl's whole allocation timeout. The table now aborts the start's own signal as soon as a teardown takes the lease, since no caller can receive that device any more, and reports the lease's end, not the aborted start, to callers still waiting. Revyl publishes its session through `control.rented` the moment `start_device` answers with it, the first moment Revyl holds it, and a release of a session that never became ready still reports its run id. New test: a release during a stalled readiness stops the device at once. The two table tests the seeded sweep fully covers are folded into it to keep the change within its budget, and Revyl's lease tests share one refusal helper to stay within the file's line cap. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 831f01b commit b6aa27b

6 files changed

Lines changed: 45 additions & 56 deletions

File tree

‎docs/adr/0007-remote-device-leases.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -167,7 +167,7 @@ The provider supplies its start timeout, an optional stop timeout, start, stop,
167167
The table owns the rules that decide whether a billed device is still rented:
168168

169169
- One lease id has at most one start. Overlapping allocations join it, and a ready session is reused.
170-
- Each caller waits until the earlier of its own deadline and the start timeout. A caller whose budget is already spent sends no start. A caller that gives up stops nothing. The start runs while any caller still waits, for at most the start timeout: no single caller's signal or deadline reaches it, and the table tells the provider to abandon it once no caller waits. A ready device goes to the callers still waiting within their budgets, and it is stopped when no caller waits. Each caller's budget is checked when the session is handed over, on reuse as well.
170+
- Each caller waits until the earlier of its own deadline and the start timeout. A caller whose budget is already spent sends no start. A caller that gives up stops nothing. The start runs while any caller still waits, for at most the start timeout: no single caller's signal or deadline reaches it, and the table tells the provider to abandon it once no caller waits or the lease starts ending. A ready device goes to the callers still waiting within their budgets, and it is stopped when no caller waits. Each caller's budget is checked when the session is handed over, on reuse as well.
171171
- A start can publish its rental before it answers. A teardown waits for the answer for at most the start timeout, then stops the published rental and keeps the lease pending until the answer arrives.
172172
- Release, recovery, a failed start, and shutdown each end a lease through one teardown that all of them join. The teardown fences the lease in the same step: from then on, an allocation of the lease is refused, including an allocation that was already waiting, and the device inventory and interactor no longer serve the lease. Revyl reports a refusal for a lease that a release, a recovery, or a failed start is ending as `provider-lease-releasing`, and a refusal after shutdown began as `runtime-shutting-down`.
173173
- A teardown forgets a lease only after the provider confirmed the stop of every rental that it knows. It stops them one at a time, in the order they were named, and a rental named while a teardown settles gets a teardown of its own. A failed or late stop keeps the lease for the next teardown, which joins the stop still in flight. Shutdown is a fence: from then on, an allocation, or a recovery of a lease that has no entry, is refused with `runtime-shutting-down`, and a release only joins an entry that already exists, so one drain pass ends every entry. It reports what stays rented with `provider-cleanup-pending`.

‎packages/provider-revyl/src/revyl-session.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,8 @@ export type RevylStartRequestInput = Readonly<{
6363
signal?: AbortSignal;
6464
/** Epoch-ms deadline the caller has for the whole allocation; it outranks the runtime's own. */
6565
deadline?: number;
66+
/** Told the session once `start_device` answered with it, the first moment Revyl holds it. */
67+
onRented?: (rented: RevylRentedSession) => void;
6668
}>;
6769

6870
/**
@@ -158,6 +160,7 @@ export async function startRevylSession(
158160
),
159161
};
160162
}
163+
input.onRented?.({ runId, sessionId });
161164
// Revyl may have answered a run for a start whose caller has already hung up. The device is rented
162165
// either way, so the answer still reports it as what is owed an end.
163166
if (input.signal?.aborted) {

‎packages/provider-revyl/src/runtime.test.ts‎

Lines changed: 31 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,17 @@ afterEach(async () => {
2020
await Promise.all(openBackends.splice(0).map((fixture) => fixture.close()));
2121
});
2222

23+
/** Asserts an allocation failed with this typed reason. */
24+
async function refusedWith(
25+
allocation: Promise<unknown> | undefined,
26+
reason: string,
27+
): Promise<void> {
28+
await assert.rejects(
29+
async () => await allocation,
30+
(error: unknown) => error instanceof AppError && error.details?.reason === reason,
31+
);
32+
}
33+
2334
/** Stops the fixture recorded, which is what every lifecycle assertion reads. */
2435
function stops(fixture: RevylFixtureBackend): string[] {
2536
return fixture.paths().filter((path) => path.includes('/stop'));
@@ -192,22 +203,17 @@ test('a canceled joiner reports its own cancellation while the shared start runs
192203
});
193204

194205
test('a joiner keeps the device when the caller that started it hangs up mid-readiness', async () => {
195-
// The start belongs to every caller that joined it, so the first caller's cancellation answers only
196-
// that caller: readiness goes on, and the caller still waiting gets the device.
206+
// The first caller's cancellation answers only that caller: readiness goes on for the one still waiting.
197207
const fixture = await backend({ healthDelayMs: 200 });
198208
const runtime = fixture.createRuntime();
199209
const lease = revylLease('ios');
200210
const controller = new AbortController();
201211
const first = runtime.leaseLifecycle!.allocate!(lease, { signal: controller.signal });
202-
const firstCanceled = assert.rejects(
203-
async () => await first,
204-
(error: unknown) => error instanceof AppError && error.details?.reason === 'request_canceled',
205-
);
212+
const firstCanceled = refusedWith(first, 'request_canceled');
206213
const joining = runtime.leaseLifecycle!.allocate!(lease);
207214
await waitFor(() => fixture.paths().some((path) => path.endsWith('/health')));
208215
controller.abort();
209216
await firstCanceled;
210-
211217
assert.equal((await joining)?.providerSessionId, revylSessionIdForLease('lease-1'));
212218
assert.deepEqual(stops(fixture), []);
213219
await runtime.shutdown();
@@ -224,11 +230,7 @@ test('a release that lands while the shared start is still running stops the dev
224230
const first = runtime.leaseLifecycle?.allocate?.(lease);
225231
// The release owns the device once it starts, so the caller still waiting is refused rather than handed
226232
// a session that is being stopped.
227-
const firstRefused = assert.rejects(
228-
async () => await first,
229-
(error: unknown) =>
230-
error instanceof AppError && error.details?.reason === 'provider-lease-releasing',
231-
);
233+
const firstRefused = refusedWith(first, 'provider-lease-releasing');
232234
await waitFor(() => fixture.paths().some((path) => path.endsWith('/start_device')));
233235

234236
const controller = new AbortController();
@@ -245,7 +247,6 @@ test('a release that lands while the shared start is still running stops the dev
245247
`POST /api/v1/execution/device/sessions/${revylSessionIdForLease('lease-1')}/stop`,
246248
]);
247249

248-
// The runtime holds no session a later shutdown would stop twice.
249250
await firstRefused;
250251
await runtime.shutdown();
251252
assert.equal(stops(fixture).length, 1);
@@ -420,6 +421,22 @@ test('shutdown waits for a start already in flight and releases the device it re
420421
);
421422
});
422423

424+
test('a release during a stalled readiness stops the device at once', async () => {
425+
// `start_device` answered, so the session exists and is published; readiness then stalls.
426+
const fixture = await backend({ healthDelayMs: 10_000 });
427+
const runtime = fixture.createRuntime();
428+
const lease = revylLease('ios');
429+
const allocating = runtime.leaseLifecycle!.allocate!(lease);
430+
const refused = refusedWith(allocating, 'provider-lease-releasing');
431+
await waitFor(() => fixture.paths().some((path) => path.endsWith('/health')));
432+
const releasedAt = Date.now();
433+
const released = await runtime.leaseLifecycle!.release!(lease);
434+
assert.ok(Date.now() - releasedAt < 2_000, 'the release does not wait out readiness');
435+
assert.equal(released?.providerSessionId, revylSessionIdForLease('lease-1'));
436+
await refused;
437+
assert.deepEqual(stops(fixture), [stopPathFor('lease-1')]);
438+
});
439+
423440
test('a release that lands mid-start stops the session only once Revyl has made it', async () => {
424441
// Revyl creates the session when it answers `start_device`, not when it is asked. A `/stop` sent inside
425442
// that gap names a session Revyl never had, and Revyl's 404 is exactly what a stop reads as "already
@@ -430,18 +447,13 @@ test('a release that lands mid-start stops the session only once Revyl has made
430447
const lease = revylLease('ios');
431448

432449
const allocating = runtime.leaseLifecycle?.allocate?.(lease);
433-
const refused = assert.rejects(
434-
async () => await allocating,
435-
(error: unknown) =>
436-
error instanceof AppError && error.details?.reason === 'provider-lease-releasing',
437-
);
450+
const refused = refusedWith(allocating, 'provider-lease-releasing');
438451
await waitFor(() => fixture.paths().some((path) => path.endsWith('/start_device')));
439452

440453
// The release lands while the start is in flight. It waits for the answer rather than stopping the id it
441454
// only has from the lease, and that is what makes the stop below land after the session exists.
442455
const released = await runtime.leaseLifecycle!.release!(lease);
443456
assert.equal(released?.providerSessionId, revylSessionIdForLease('lease-1'));
444-
// The release reports the device the start rented, not only the session id.
445457
assert.equal(released?.revylRunId, 'run-1');
446458
await refused;
447459

‎packages/provider-revyl/src/runtime.ts‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -198,7 +198,7 @@ class RevylRuntimeImplementation implements ProviderDeviceRuntime {
198198
if (released === undefined) return undefined;
199199
return released.session
200200
? allocationResult(released.session)
201-
: { providerSessionId: released.handle.sessionId };
201+
: { revylRunId: released.handle.runId, providerSessionId: released.handle.sessionId };
202202
}
203203

204204
private sessionForDevice(device: DeviceInfo): RevylSession | undefined {
@@ -226,9 +226,10 @@ function createRevylLeaseDriver(
226226
appUrl: request.appUrl,
227227
deviceModel: request.deviceModel,
228228
osVersion: request.osVersion,
229-
// The table aborts this only once no caller waits; Revyl bounds `start_device` and readiness with
230-
// its own budgets, which `REVYL_ALLOCATION_TIMEOUT_MS` covers.
229+
// The table aborts this once no caller waits or the lease is ending; Revyl bounds `start_device`
230+
// and readiness with its own budgets, which `REVYL_ALLOCATION_TIMEOUT_MS` covers.
231231
signal: control.signal,
232+
onRented: control.rented,
232233
});
233234
if (started.status === 'failed') return started;
234235
const { runId, sessionId } = started.session;

‎packages/provision-kit/src/provider-lease-table.test.ts‎

Lines changed: 1 addition & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -240,6 +240,7 @@ test('a release during a start refuses the waiting allocation and stops the devi
240240
const releasing = provider.table.release('lease-1');
241241
await settle();
242242
assert.deepEqual(provider.stopped, [], 'the stop waits for the answer that names the device');
243+
assert.ok(provider.controls[0]!.signal.aborted, 'the start is told the lease is ending');
243244
provider.answerStart(0, ready('session-1'));
244245
assert.deepEqual(await releasing, {
245246
handle: { id: 'session-1' },
@@ -652,19 +653,6 @@ test('shutdown fences recovery: a lease it holds nothing for is left to the next
652653
assert.deepEqual(provider.stopped, ['session-1']);
653654
});
654655

655-
test('shutdown reports what it could not stop and keeps it for a later release', async () => {
656-
const provider = fakeProvider();
657-
await allocated(provider);
658-
provider.queueStop(refusedStop());
659-
660-
await assert.rejects(provider.table.shutdown(), (error: unknown) => {
661-
assert.ok(reason('provider-cleanup-pending', 'shutdown')(error));
662-
assert.deepEqual((error as TableError).details.handles, ['session-1']);
663-
return true;
664-
});
665-
assert.equal((await provider.table.release('lease-1'))?.handle.id, 'session-1');
666-
});
667-
668656
test('recovery stops what the provider finds, and keeps that handle when the stop fails', async () => {
669657
const provider = fakeProvider();
670658
provider.queueStop(refusedStop());
@@ -710,21 +698,3 @@ test('recoveries, a release, and a shutdown of an entryless lease share one look
710698
assert.deepEqual(provider.found, ['lease-1']);
711699
assert.deepEqual(provider.stopped, ['found-lease-1']);
712700
});
713-
714-
test('a release, a recovery and a shutdown that overlap share one stop', async () => {
715-
const provider = fakeProvider();
716-
await allocated(provider);
717-
718-
const stop = deferred<void>();
719-
provider.queueStop(async () => await stop.promise);
720-
const ending = Promise.all([
721-
provider.table.release('lease-1'),
722-
provider.table.recover(lease()),
723-
provider.table.shutdown(),
724-
]);
725-
await settle();
726-
stop.resolve();
727-
await ending;
728-
assert.deepEqual(provider.stopped, ['session-1']);
729-
assert.deepEqual(provider.found, [], 'a lease this process holds is not searched for');
730-
});

‎packages/provision-kit/src/provider-lease-table.ts‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ export type ProviderLeaseAllocateOptions<Request> = ProviderLeaseBudget &
2323

2424
export type ProviderLeaseStartControl<Handle> = Readonly<{
2525
/**
26-
* Aborts when no caller still waits for the start before it answers, so the provider can abandon it.
26+
* Aborts when no caller still waits for the start before it answers, or the lease starts ending.
2727
* No single caller's signal or deadline reaches the start: it runs for every caller that joins it.
2828
*/
2929
signal: AbortSignal;
@@ -432,8 +432,9 @@ class LeaseTable<Session, Handle, Request> {
432432
caller: ProviderLeaseBudget,
433433
startedAt: number,
434434
): unknown {
435-
if (answer.status !== 'ready') return answer.error;
435+
// A lease that is ending is why the callers get nothing, whatever the aborted start answered.
436436
if (entry.phase === 'ending') return this.driver.releasing(entry.lease);
437+
if (answer.status !== 'ready') return answer.error;
437438
const budgetMs = Math.max(0, entry.answerBy - startedAt);
438439
return this.driver.gaveUp(entry.lease, caller, budgetMs, answer.handle);
439440
}
@@ -519,6 +520,8 @@ class LeaseTable<Session, Handle, Request> {
519520
*/
520521
private teardown(entry: LeaseEntry<Session, Handle, Request>): Promise<Ended<Session, Handle>> {
521522
entry.phase = 'ending';
523+
// No caller can take this start's device any more, so the provider may stop working on it.
524+
if (!entry.answered) entry.abandon.abort();
522525
if (entry.ending) return entry.ending;
523526
const ending = this.runTeardown(entry).finally(() => {
524527
if (entry.ending === ending) entry.ending = undefined;

0 commit comments

Comments
 (0)