Skip to content

Commit 79692a9

Browse files
thymikeeclaude
andcommitted
fix(daemon): read inherited lease records only after the registration is held
The releaser now touches no file in the state directory until the daemon calls recoverInherited, after its registration is acquired: that step loads the expired record, adopts the live journal and starts recovery. A daemon refused the registration leaves the live journal byte-identical. A durable expired record rewrites the journal, and ADR 0007 names both routes where the registry forgets a provider lease without a crash. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 665e27c commit 79692a9

4 files changed

Lines changed: 40 additions & 24 deletions

File tree

‎docs/adr/0007-remote-device-leases.md‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -88,11 +88,14 @@ Shutdown waits for any lease-end teardown that already started before it tears s
8888
A daemon that dies while it holds a provider lease leaves the rented device running on the provider side, and that device stays billed.
8989
Thus, the daemon journals the live leases of each provider that can recover an expired lease, in `live-provider-leases.json` in its state directory.
9090
A daemon that starts and finds this journal hands each lease in it to the expired-lease recovery of its provider, which ends the rented device.
91-
It does so only after it holds the daemon registration for the state directory, because until then the journal can belong to a daemon that is still running.
91+
It reads both lease records only after it holds the daemon registration for the state directory, because until then they can belong to a daemon that is still running.
9292
The lease itself is not restored, and its controller connects again.
9393
Restoring the live lease was rejected: the sessions and device claims of the dead daemon are gone, so a restored lease would name a device that no session can reach.
9494
A lease that leaves the registry for expiry stays in the journal until its expired-lease record is durable, and a journal write that fails is retried.
95-
One gap remains: the registry releases a canceled first allocation without a provider release, so that lease leaves the journal while its provider start may still run, and a crash in that interval loses it.
95+
Two routes still let the registry forget a provider lease while its device may be rented. The journal and expiry recovery both follow the registry, so neither covers it, and only the provider runtime still holding the device can end it:
96+
97+
- A failed first allocation releases the registry entry without a provider release, while its provider start may still run.
98+
- An allocation that finishes after its requester left releases the registry entry even when the provider release throws.
9699

97100
When a request closes a session, the daemon ends the lease of that session, unless the lease was allocated with `retainOnClose`.
98101
The reply to that request reports each lease that it ended once, at response level, as `endedLeases: [{ leaseId, outcome: 'released' }]`.

‎packages/daemon/src/server/__tests__/provider-lease-expiry.test.ts‎

Lines changed: 11 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -75,12 +75,12 @@ test('retries a persisted expired provider lease after daemon restart', async ()
7575
...options,
7676
recoverableProviderIds: [],
7777
});
78-
await daemonWithoutLimrun.retryPending();
78+
await daemonWithoutLimrun.recoverInherited();
7979
expect(release).toHaveBeenCalledTimes(1);
8080
daemonWithoutLimrun.shutdown();
8181

8282
const restartedDaemon = createExpiredProviderLeaseReleaser(options);
83-
await restartedDaemon.retryPending();
83+
await restartedDaemon.recoverInherited();
8484
expect(release).toHaveBeenCalledTimes(2);
8585
expect(release).toHaveBeenLastCalledWith(lease);
8686
restartedDaemon.shutdown();
@@ -182,16 +182,14 @@ test('a daemon killed while holding a provider lease recovers it on restart', as
182182
killedDaemon.shutdown();
183183

184184
const restartedDaemon = createExpiredProviderLeaseReleaser(options);
185-
restartedDaemon.adoptOrphanedLeases();
186-
await restartedDaemon.retryPending();
185+
await restartedDaemon.recoverInherited();
187186
expect(recoverExpiredLease.mock.calls.map(([lease]) => lease.leaseId)).toEqual([held.leaseId]);
188187
expect(fs.existsSync(`${stateDir}/live-provider-leases.json`)).toBe(false);
189188
restartedDaemon.shutdown();
190189

191190
// Recovered once means recovered: the next daemon inherits nothing.
192191
const thirdDaemon = createExpiredProviderLeaseReleaser(options);
193-
thirdDaemon.adoptOrphanedLeases();
194-
await thirdDaemon.retryPending();
192+
await thirdDaemon.recoverInherited();
195193
expect(recoverExpiredLease).toHaveBeenCalledTimes(1);
196194
thirdDaemon.shutdown();
197195
} finally {
@@ -213,8 +211,7 @@ test('a provider lease released before the daemon stops is not recovered again',
213211
daemon.shutdown();
214212

215213
const restartedDaemon = createExpiredProviderLeaseReleaser(options);
216-
restartedDaemon.adoptOrphanedLeases();
217-
await restartedDaemon.retryPending();
214+
await restartedDaemon.recoverInherited();
218215
expect(recoverExpiredLease).not.toHaveBeenCalled();
219216
restartedDaemon.shutdown();
220217
} finally {
@@ -337,8 +334,8 @@ test('adopted leases whose record is not durable stay journaled through later jo
337334
...options,
338335
recoverExpiredLease: () => new Promise<void>(() => {}),
339336
});
340-
restarted.adoptOrphanedLeases();
341337
try {
338+
void restarted.recoverInherited();
342339
// The new daemon's registry is empty, so this journal write has no live leases of its own.
343340
restarted.recordLiveLeases([]);
344341
const journal = JSON.parse(fs.readFileSync(`${stateDir}/live-provider-leases.json`, 'utf8'));
@@ -349,7 +346,7 @@ test('adopted leases whose record is not durable stay journaled through later jo
349346
}
350347
});
351348

352-
test('a releaser leaves the journal alone until its daemon holds the registration', async () => {
349+
test('a daemon refused the registration leaves the live journal byte-identical and recovers nothing', async () => {
353350
const stateDir = mkdtempForTestSync('agent-device-live-provider-lease-');
354351
const recoverExpiredLease = vi.fn(async (_lease: DeviceLease) => undefined);
355352
const options = { recoverExpiredLease, stateDir, recoverableProviderIds: ['revyl'] };
@@ -359,13 +356,15 @@ test('a releaser leaves the journal alone until its daemon holds the registratio
359356
});
360357
try {
361358
registry.allocateLease({ tenantId: 't', runId: 'run-1', leaseProvider: 'revyl' });
362-
// A second daemon on the same state directory is built before it learns the registration is taken.
359+
const journalBefore = fs.readFileSync(`${stateDir}/live-provider-leases.json`);
360+
// A second daemon on the same state directory is built, then refused the registration: it never
361+
// reaches `recoverInherited`, and nothing it does before that may touch the live daemon's records.
363362
const refusedDaemon = createExpiredProviderLeaseReleaser(options);
364363
await refusedDaemon.retryPending();
365364
refusedDaemon.shutdown();
366365

367366
expect(recoverExpiredLease).not.toHaveBeenCalled();
368-
expect(fs.existsSync(`${stateDir}/live-provider-leases.json`)).toBe(true);
367+
expect(fs.readFileSync(`${stateDir}/live-provider-leases.json`)).toEqual(journalBefore);
369368
expect(fs.existsSync(`${stateDir}/expired-provider-leases.json`)).toBe(false);
370369
} finally {
371370
runningDaemon.shutdown();

‎packages/daemon/src/server/http/daemon-runtime.ts‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -619,9 +619,8 @@ async function startDaemonRuntimeOnHost(
619619
});
620620
}
621621
const registration = acquisition.owner;
622-
// Only the registered daemon may treat the journal as orphaned and start recovering provider leases.
623-
expiredProviderLeaseReleaser.adoptOrphanedLeases();
624-
void expiredProviderLeaseReleaser.retryPending();
622+
// Only the registered daemon may read the state directory's lease records and act on them.
623+
void expiredProviderLeaseReleaser.recoverInherited();
625624
rollback.onRollback(async () => {
626625
await deps.ownerLifecycle.clearDaemonLockConfiguration();
627626
});

‎packages/daemon/src/server/provider-lease-expiry.ts‎

Lines changed: 22 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -41,11 +41,13 @@ export type ExpiredProviderLeaseReleaser = {
4141
/** Journals the registry's live provider leases, so a daemon killed while holding them recovers them. */
4242
recordLiveLeases: (leases: readonly DeviceLease[]) => void;
4343
/**
44-
* Moves the leases a dead daemon journaled into recovery. Call it only once this process holds the
45-
* daemon registration for the state directory: before that, the journal may belong to a daemon that is
46-
* still running, and recovering its leases would end devices it is using.
44+
* Reads the lease records an earlier daemon left in the state directory (the expired record and the
45+
* live journal), moves the journal's leases into recovery, and starts recovering. Call it only once
46+
* this process holds the daemon registration: before that, both records may belong to a daemon that is
47+
* still running, and acting on them would end devices it is using. Until then the releaser touches no
48+
* file in the state directory.
4749
*/
48-
adoptOrphanedLeases: () => void;
50+
recoverInherited: () => Promise<void>;
4951
retryPending: () => Promise<void>;
5052
/**
5153
* `pending` holds every lease still unreleased; `unknown` is the part of it whose release was
@@ -68,8 +70,9 @@ export function createExpiredProviderLeaseReleaser(options: {
6870
retryDelayMs?: number;
6971
}): ExpiredProviderLeaseReleaser {
7072
const pendingLiveLeases = new Map<string, DeviceLease>();
71-
const pendingRecoveryLeases = loadLeaseRecord(options.stateDir, PENDING_RELEASES_FILE);
72-
const persistedRecoveryLeaseIds = new Set(pendingRecoveryLeases.keys());
73+
const pendingRecoveryLeases = new Map<string, DeviceLease>();
74+
const persistedRecoveryLeaseIds = new Set<string>();
75+
let inherited = false;
7376
const retryDelayMs = options.retryDelayMs ?? DEFAULT_RETRY_DELAY_MS;
7477
let retryTimer: Timer | undefined;
7578
const activeReleaseAttempts = new Map<string, Promise<void>>();
@@ -96,6 +99,8 @@ export function createExpiredProviderLeaseReleaser(options: {
9699
for (const leaseId of pendingRecoveryLeases.keys()) {
97100
persistedRecoveryLeaseIds.add(leaseId);
98101
}
102+
// Leases now durable in the expired record no longer need the journal to hold them.
103+
writeJournal();
99104
return true;
100105
} catch (error) {
101106
emitPersistenceFailure(pendingRecoveryLeases.size, error);
@@ -155,6 +160,16 @@ export function createExpiredProviderLeaseReleaser(options: {
155160
journalRetryTimer.unref?.();
156161
}
157162
};
163+
const recoverInherited = async (): Promise<void> => {
164+
if (inherited) return;
165+
inherited = true;
166+
for (const [leaseId, lease] of loadLeaseRecord(options.stateDir, PENDING_RELEASES_FILE)) {
167+
pendingRecoveryLeases.set(leaseId, lease);
168+
persistedRecoveryLeaseIds.add(leaseId);
169+
}
170+
adoptOrphanedLeases();
171+
await retryPending();
172+
};
158173
const recordLiveLeases = (leases: readonly DeviceLease[]): void => {
159174
liveLeases = leases.filter((lease) =>
160175
isRecoverableProviderAvailable(lease, options.recoverableProviderIds),
@@ -270,7 +285,7 @@ export function createExpiredProviderLeaseReleaser(options: {
270285
}
271286
},
272287
recordLiveLeases,
273-
adoptOrphanedLeases,
288+
recoverInherited,
274289
retryPending,
275290
drain: async (timeoutMs) => {
276291
// Shutdown gets one final, bounded attempt at every pending release. A

0 commit comments

Comments
 (0)