Skip to content

Commit b391a4a

Browse files
committed
fix(remote): budget each beat from the window it protects, not the cadence
Budgeting a beat at the cadence it started on made one number do two jobs. Until a beat answers, the cadence sits at the 1s floor, so `sendRequest` got a 1s budget, destroyed the request, and emitted a `daemon_request_timeout`. That answer is non-terminal, so the cadence never left the floor: on any link where a round trip takes over a second — a tunneled proxy, a fresh TLS connect per beat because every timeout tore the socket down, an uplink the upload was saturating — no beat ever got a window in which to answer, and the lease could lapse on exactly the slow uploads #2946 exists to protect, with a diagnostic pair every second. Cadence and budget are two numbers now. A beat is armed every third of the window and allowed the whole window to answer in, so a slow first answer sets the cadence instead of being cut off before it can say anything. The window the loop assumes before the first answer is the registry's five-second minimum, which is now shared with the client as `MIN_LEASE_WINDOW_MS` rather than assumed twice: planning against a longer window than the daemon would ever grant is the same mistake pointed the other way. The slow-link case is covered through `sendToDaemon`, which no loopback unit test travels: the fake daemon answers a beat after 3s — longer than the assumed cadence, shorter than the assumed window, so a beat budgeted at the cadence is cut off and one budgeted at the window is not. It asserts the beat stops arriving once the window is known and counts no `daemon_request_timeout` inside the upload, in a diagnostics scope because outside one every count reads zero for the wrong reason. Verified against the previous beat loop: five arrivals there, none of them answered in time.
1 parent 8beb7fb commit b391a4a

6 files changed

Lines changed: 179 additions & 53 deletions

File tree

‎docs/adr/0007-remote-device-leases.md‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -89,9 +89,14 @@ phase begins: hashing, the preflight, and the start of the stream can all run wh
8989
still outstanding, so what the first beat buys is that the loss is learned during the upload rather
9090
than after it. Each beat answers with the window it just renewed, and the loop arms its successor when
9191
the beat starts rather than when it settles, so a beat that never answers is abandoned on schedule
92-
instead of taking the schedule with it; the beat's own budget is capped at the cadence it started on,
93-
which is what keeps one stalled round trip from outliving the window it exists to protect. A phase
94-
that settles does not wait on a beat still in flight.
92+
instead of taking the schedule with it. A phase that settles does not wait on a beat still in flight.
93+
94+
Cadence and budget are deliberately two numbers. A beat is armed every third of the window, but it is
95+
allowed the whole window to answer in. Budgeting a beat at its cadence instead would give a heartbeat
96+
that needs more than a third of a window for its round trip — a tunneled proxy, an uplink the upload
97+
itself is saturating — no way to answer at all, and every beat would re-time-out on exactly the slow
98+
links the beat exists to protect. The window, not the cadence, is the interval in which being alive
99+
still matters, so it is also the longest a beat's answer is worth waiting for.
95100

96101
A beat is a fresh request each time, never the protected request rewritten: a request identity is
97102
what a timed-out beat is canceled under, and beats must not inherit each other's cancellation. A beat

‎packages/contracts/src/lease-scope.ts‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -295,6 +295,15 @@ export function findMissingProxyLeaseFields(scope: LeaseScope): string[] {
295295
return REQUIRED_PROXY_LEASE_FIELDS.filter((field) => !scope[field]);
296296
}
297297

298+
/**
299+
* The shortest inactivity window a lease can hold by default.
300+
*
301+
* The registry clamps every ttl to this floor, so it is also the worst case a client plans against
302+
* before it has seen a lease answer: work that has to finish inside a lease window it does not know
303+
* yet must assume this one, or it can be planning for a window the daemon would never grant.
304+
*/
305+
export const MIN_LEASE_WINDOW_MS = 5_000;
306+
298307
/**
299308
* Why a lease stopped being ours: it is gone, spent, or taken back.
300309
*

‎src/daemon-client/__tests__/daemon-client-lease-beat.test.ts‎

Lines changed: 16 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -133,17 +133,23 @@ describe('runProtectedLeaseWork', () => {
133133
heartbeat,
134134
});
135135

136-
await vi.advanceTimersByTimeAsync(1_000);
136+
// No window named, so the loop stays on the cadence it assumed: a third of the shortest window
137+
// the daemon will accept.
138+
await vi.advanceTimersByTimeAsync(1_666);
137139
assert.equal(heartbeat.mock.calls.length, 2, 'immediate first, then the same cadence again');
138-
await vi.advanceTimersByTimeAsync(1_000);
140+
await vi.advanceTimersByTimeAsync(1_666);
139141
assert.equal(heartbeat.mock.calls.length, 3);
140142

141143
upload.resolve('installed');
142144
assert.equal(await running, 'installed');
143145
});
144146

145-
test('each beat is given a budget no longer than the cadence it starts on', async () => {
147+
test('each beat is given the window it protects as its budget, not the cadence', async () => {
146148
vi.useFakeTimers();
149+
// Cadence and budget are separate numbers. A beat is armed every third of the window, but it is
150+
// allowed the whole window to answer in: budgeting it at the cadence would leave a heartbeat
151+
// that needs more than a third of a window for its round trip with no chance to answer, on
152+
// exactly the slow links the beat exists to protect.
147153
const budgets: number[] = [];
148154
const heartbeat = vi.fn(async (budgetMs: number) => {
149155
budgets.push(budgetMs);
@@ -156,11 +162,9 @@ describe('runProtectedLeaseWork', () => {
156162
});
157163

158164
await vi.advanceTimersByTimeAsync(40_000);
159-
// Before the first answer the loop assumes the shortest window the daemon accepts, so the beat
160-
// that has to prove that lease is alive cannot itself take longer than a fifth of it. Once the
161-
// window is known the budget is the cadence: a stalled beat dies inside one, not the 90s the
162-
// command's own heartbeat policy would allow.
163-
assert.deepEqual(budgets, [1_000, 20_000, 20_000]);
165+
// Before the first answer the loop assumes the shortest window the daemon will accept; once the
166+
// lease names its own, the budget is that window, not the 90s the heartbeat policy would allow.
167+
assert.deepEqual(budgets, [5_000, 60_000, 60_000]);
164168

165169
upload.resolve('installed');
166170
assert.equal(await running, 'installed');
@@ -215,7 +219,7 @@ describe('runProtectedLeaseWork', () => {
215219
heartbeat,
216220
});
217221

218-
await vi.advanceTimersByTimeAsync(3_000);
222+
await vi.advanceTimersByTimeAsync(5_000);
219223
assert.equal(heartbeat.mock.calls.length, 4, 'the stalled beats were abandoned, not awaited');
220224

221225
upload.resolve('installed');
@@ -253,11 +257,12 @@ describe('runProtectedLeaseWork', () => {
253257
heartbeat,
254258
});
255259

256-
await vi.advanceTimersByTimeAsync(2_500);
260+
await vi.advanceTimersByTimeAsync(3_332);
257261
assert.equal(heartbeat.mock.calls.length, 3, 'one failed beat does not stop the others');
258262

259263
heartbeat.mockImplementation(async () => renewedLeaseResponse(30_000));
260-
await vi.advanceTimersByTimeAsync(1_000);
264+
await vi.advanceTimersByTimeAsync(10_000);
265+
assert.equal(heartbeat.mock.calls.length, 4, 'the window the beat reported sets the cadence');
261266
upload.resolve('installed');
262267
assert.equal(await running, 'installed');
263268
});

‎src/daemon-client/daemon-client-lease-beat.ts‎

Lines changed: 34 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ import { INTERNAL_COMMANDS } from '@agent-device/command-registry/catalog';
55
import { resolveCommandTimeoutPolicy } from '@agent-device/command-registry/registry';
66
import { resolveCommandRequestTimeoutMs } from '@agent-device/command-registry/timeout-policy';
77
import {
8+
MIN_LEASE_WINDOW_MS,
89
isInactiveLeaseError,
910
leaseScopeFromRequest,
1011
leaseScopeToRequestMeta,
@@ -24,14 +25,12 @@ import { sendRequest } from './daemon-client-transport.ts';
2425
// knows nothing about transports beyond the `send` it is handed.
2526

2627
/**
27-
* The fastest cadence a phase beats at, and the budget each beat gets until the window is known.
28+
* The fastest cadence a phase beats at.
2829
*
29-
* Before the first beat answers, the window is unknown and the worst legal case is the registry's
30-
* five-second minimum: a beat that stalls must be abandoned and retried inside that window, or the
31-
* thing the beat exists to prevent happens while it waits. One second is a fifth of that minimum,
32-
* and no window-derived cadence is ever allowed below it, so it is also the loop's floor: a
33-
* misreported or pathologically short window cannot turn the beat into a request loop faster than
34-
* this.
30+
* A cadence is a third of a window, and a window can be as short as the registry's five-second
31+
* minimum, so without a floor a short lease would turn the beat into a request loop. One second is
32+
* a fifth of that minimum: it keeps the shortest legal window beaten five times over, and no
33+
* window-derived cadence is ever allowed below it.
3534
*/
3635
const MIN_LEASE_BEAT_INTERVAL_MS = 1_000;
3736

@@ -82,12 +81,16 @@ function isTerminalLeaseBeatError(error: unknown): boolean {
8281
* and the first bytes can all happen while the opening beat is still outstanding. Each beat answers
8382
* with the window it just renewed, and the cadence becomes a third of that window.
8483
*
85-
* A beat's budget is the cadence it started on, and its successor is armed when the beat starts
86-
* rather than when it settles. A beat that never returns — a half-open connection, a daemon wedged
87-
* before it admits anything — is therefore abandoned on schedule instead of holding the schedule:
88-
* the lease is still beaten at window/3, and the abandoned round trip is cut off by its own budget
89-
* in the transport rather than by the command's 90-second heartbeat policy. An abandoned beat is
90-
* still listened to, because the answer it eventually gives can be a lost lease.
84+
* The cadence and the budget are two numbers, and only one of them is the cadence. A beat is armed
85+
* every third of the window, and its budget is the whole window it protects: a heartbeat that needs
86+
* more than a cadence to come back — a tunneled proxy, a slow uplink the upload itself is
87+
* saturating — still answers inside the lease it is renewing, and the cadence it proves becomes the
88+
* loop's. Budgeting a beat at the cadence instead would hand a slow link no window in which to
89+
* answer at all and re-time-out every beat forever. Successors are armed when a beat starts rather
90+
* than when it settles, so a beat that never returns at all is abandoned on schedule instead of
91+
* taking the schedule with it. A stalled beat is never awaited, so the outstanding ones are bounded
92+
* by the budget divided by the cadence rather than by the phase's length. An abandoned beat is still
93+
* listened to, because the answer it eventually gives can be a lost lease.
9194
*
9295
* A beat that fails for a reason that says nothing about this lease is reported and survived — one
9396
* lost request must not fail an upload that a later beat will cover. A beat that finds the lease
@@ -98,8 +101,10 @@ function isTerminalLeaseBeatError(error: unknown): boolean {
98101
export async function runProtectedLeaseWork<T>(
99102
options: Readonly<{
100103
/**
101-
* One renewal. `budgetMs` is how long this beat may take before the loop abandons it. Absent
102-
* when the request names no lease to renew, which is the ordinary unleased install.
104+
* One renewal. `budgetMs` is how long this beat may take before the loop abandons it: the window
105+
* the beat protects, so a slow round trip still gets a chance to answer inside the lease it is
106+
* renewing. Absent when the request names no lease to renew, which is the ordinary unleased
107+
* install.
103108
*/
104109
heartbeat?: ((budgetMs: number) => Promise<unknown>) | undefined;
105110
task: (signal: AbortSignal) => Promise<T>;
@@ -109,9 +114,10 @@ export async function runProtectedLeaseWork<T>(
109114
if (!heartbeat) return await options.task(new AbortController().signal);
110115

111116
const control = new AbortController();
112-
// Until a beat names the window, the loop assumes the shortest window the daemon will accept: the
113-
// budget of the beat that has to prove a short lease is alive cannot itself be longer than it.
114-
let intervalMs = MIN_LEASE_BEAT_INTERVAL_MS;
117+
// Until a beat names the window, the loop assumes the shortest window the daemon will accept: a
118+
// beat that budgets itself on a longer window than the lease actually has would outlive it.
119+
let windowMs = MIN_LEASE_WINDOW_MS;
120+
let intervalMs = leaseBeatIntervalMs(windowMs);
115121
let timer: ReturnType<typeof setTimeout> | undefined;
116122
let stopped = false;
117123
let terminalError: unknown;
@@ -130,14 +136,15 @@ export async function runProtectedLeaseWork<T>(
130136
};
131137
arm(intervalMs);
132138
const settle = (async () => {
133-
const budgetMs = intervalMs;
139+
const budgetMs = windowMs;
134140
try {
135141
const renewed = leaseWindowFromHeartbeatResponse(await heartbeat(budgetMs));
136142
// An answer that names no window keeps the cadence it was asked at: the loop only ever
137-
// slows down on evidence of how long the lease is good for, and never on the absence of it.
143+
// moves on evidence of how long the lease is good for, and never on the absence of it.
138144
if (renewed === undefined) return;
139-
const cadence = Math.max(MIN_LEASE_BEAT_INTERVAL_MS, Math.floor(renewed / 3));
140-
if (cadence === intervalMs) return;
145+
const cadence = leaseBeatIntervalMs(renewed);
146+
if (renewed === windowMs) return;
147+
windowMs = renewed;
141148
intervalMs = cadence;
142149
// The window just moved, so the next beat is due one cadence from this answer.
143150
arm(cadence);
@@ -201,6 +208,11 @@ export async function runProtectedLeaseWork<T>(
201208
* extended — the same pair `leaseOwnTtlMs` renews on. Anything unrecognizable leaves the caller on
202209
* the fallback cadence rather than guessing one.
203210
*/
211+
/** The cadence a window of `windowMs` beats at: a third of it, never faster than the floor. */
212+
function leaseBeatIntervalMs(windowMs: number): number {
213+
return Math.max(MIN_LEASE_BEAT_INTERVAL_MS, Math.floor(windowMs / 3));
214+
}
215+
204216
function leaseWindowFromHeartbeatResponse(response: unknown): number | undefined {
205217
const lease = (
206218
response as Readonly<{ data?: Readonly<{ lease?: Readonly<Record<string, unknown>> }> }>

‎src/daemon/lease-registry-scope.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import crypto from 'node:crypto';
22
import type { DeviceLease } from '@agent-device/contracts/device';
3+
import { MIN_LEASE_WINDOW_MS } from '@agent-device/contracts/lease-scope';
34
import type { LeaseBackend } from '@agent-device/kernel/contracts';
45
import { AppError } from '@agent-device/kernel/errors';
56
import { normalizeTenantId } from './config.ts';
@@ -84,7 +85,6 @@ export type NormalizedAllocateLeaseRequest = {
8485
};
8586

8687
const DEFAULT_LEASE_TTL_MS = 60_000;
87-
const MIN_LEASE_TTL_MS = 5_000;
8888
const MAX_LEASE_TTL_MS = 10 * 60_000;
8989
const DEFAULT_LEASE_PROVIDER = 'default';
9090

@@ -94,7 +94,7 @@ export function createLeaseTtlResolver(options: LeaseRegistryOptions) {
9494
: DEFAULT_LEASE_TTL_MS;
9595
const minTtl = Number.isInteger(options.minLeaseTtlMs)
9696
? Math.max(1, Number(options.minLeaseTtlMs))
97-
: MIN_LEASE_TTL_MS;
97+
: MIN_LEASE_WINDOW_MS;
9898
const maxTtl = Number.isInteger(options.maxLeaseTtlMs)
9999
? Math.max(minTtl, Number(options.maxLeaseTtlMs))
100100
: MAX_LEASE_TTL_MS;

0 commit comments

Comments
 (0)