Skip to content

Commit 0f37589

Browse files
committed
fix(host-kit): a reclaim re-decides in place, and a release never speaks over its caller
Rename-aside made a reclaim a two-step transaction: move the abandoned directory out from under the lock path, judge it there, and move it back if it turned out to belong to somebody else. Between those steps the lock path is simply absent, and a contender polling the path reads that as free: it claims the path and publishes, and the rename back returns `ENOTEMPTY` into a `catch {}`. The judged directory ends up nobody's. Removal happens in place now, behind `mkdir <lock>.reclaim`, and every branch decides again from what is on disk rather than from a `Stats` read before the mutex was taken: - a directory whose record named a dead claim goes only while that record is still there answering to the same claim token. A token is a random id no later acquisition repeats, so a contender that claimed the path in between walks away holding its lock; - a directory with no record has no claim to attribute its contents to, so `rmdir` is the only call made on it and its age is asked again. `ENOTEMPTY` is a publication saying so, and a directory dated a moment ago is an acquisition that has not published yet, not an abandoned one; - a path that is not a directory is unlinked, and `unlink` itself answers `EISDIR` for the one case this branch must not touch; - a path that has already gone is left alone, so the caller's next `mkdir` simply wins it. `releaseProcessLock` lost its recursive removal for the same reason: it unlinks the record it verified and removes the directory only while empty. Callers that were answering "which of two failures do I report?" by hand with `finally` now go through one shape: `withProcessLock({ acquire, task })` releases best-effort when the task failed and strictly when it did not. `runner-artifact.ts`, `runner-cache.ts`, `managed-allocation/src/store.ts` and `store-lock.ts` each had a release that could speak `ownerReleaseUnverified` over the task's own failure. `managed agent-browser setup gives the lock back on every path out` pins a second bug that shape was hiding: setup returned early when the package was already installed, skipping the only `await release()` and leaving the lock for the stale-clear path to notice five seconds later. The XCTest device-set redirect had that shape twice more, and neither site is a task: a launch that failed waits on the redirect in `runner-session.ts`, and a teardown in `runner-disposal.ts`. Both call `releaseXcodebuildSimulatorSetRedirectBestEffort` now, and the build path calls `withXcodebuildSimulatorSetRedirect`, so the `xcodebuild` failure is the error a caller reads. The parked directory is gone from the vocabulary as well: a stale reclaim holds `<lock>.reclaim.lock`, which is what the `managed-allocation` store comment and its lock-scan test now say, and the two-contender test asks with a second of grace rather than none, because a zero grace reads the winner's own mutex as abandoned.
1 parent b098cae commit 0f37589

20 files changed

Lines changed: 870 additions & 381 deletions

‎packages/host-kit/src/file.ts‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,4 +12,9 @@ export {
1212
openVerifiedFileForTruncate,
1313
} from './internal/verified-file.ts';
1414
export { expandUserHomePath, resolveUserPath } from './internal/path-resolution.ts';
15-
export { acquireProcessLock, type ProcessLockOwner } from './internal/process-lock.ts';
15+
export {
16+
acquireProcessLock,
17+
withProcessLock,
18+
type ProcessLockOwner,
19+
type ProcessLockRelease,
20+
} from './internal/process-lock.ts';

‎packages/host-kit/src/internal/process-lock.test.ts‎

Lines changed: 299 additions & 105 deletions
Large diffs are not rendered by default.

‎packages/host-kit/src/internal/process-lock.ts‎

Lines changed: 178 additions & 86 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ const DEFAULT_LOCK_TIMEOUT_MS = 30_000;
1111
const DEFAULT_LOCK_POLL_MS = 100;
1212
const DEFAULT_LOCK_OWNER_GRACE_MS = 5_000;
1313
const LOCK_DIRECTORY_SUFFIX = '.lock';
14+
const RECLAIM_MUTEX_SUFFIX = '.reclaim';
1415

1516
export type ProcessLockOwner = {
1617
pid: number;
@@ -27,6 +28,33 @@ export type ProcessLockOwnerRecord = ProcessLockOwner & {
2728
claimToken: string | null;
2829
};
2930

31+
/** Gives a lock back. Rejects when the lock is standing and this process cannot prove it owns it. */
32+
export type ProcessLockRelease = () => Promise<void>;
33+
34+
/**
35+
* Runs `task` while the lock that `acquire` returns is held, and settles the question every
36+
* caller otherwise answers by hand: which of two failures to report.
37+
*
38+
* A task that failed is the reportable fact, and an unverified release afterwards only says the
39+
* lock is still standing, which the stale-clear path resolves on its own. On the success path
40+
* the release is not best effort: a lock this process could not give back is not a completed
41+
* task, and swallowing it would report success while the next contender waits.
42+
*/
43+
export async function withProcessLock<Task>(params: {
44+
acquire: () => Promise<ProcessLockRelease>;
45+
task: () => Promise<Task>;
46+
}): Promise<Task> {
47+
const release = await params.acquire();
48+
try {
49+
const result = await params.task();
50+
await release();
51+
return result;
52+
} catch (error) {
53+
await release().catch(() => undefined);
54+
throw error;
55+
}
56+
}
57+
3058
type ProcessLockOwnerReading =
3159
| { kind: 'owner'; owner: ProcessLockOwnerRecord }
3260
| { kind: 'unwritten' }
@@ -39,7 +67,7 @@ export async function acquireProcessLock(params: {
3967
pollMs?: number;
4068
ownerGraceMs?: number;
4169
description?: string;
42-
}): Promise<() => Promise<void>> {
70+
}): Promise<ProcessLockRelease> {
4371
const { lockDirPath, owner } = params;
4472
const ownerFilePath = path.join(lockDirPath, OWNER_FILE_NAME);
4573
const deadline = Date.now() + (params.timeoutMs ?? DEFAULT_LOCK_TIMEOUT_MS);
@@ -115,11 +143,30 @@ function releaseProcessLock(
115143
if (reading.kind === 'unreadable') return 'unverified';
116144
if (reading.kind === 'unwritten' || !ownerIdentityMatches(reading.owner, claim))
117145
return 'not-owner';
118-
// The same process can hold this path twice in sequence, and a reclaim that moved our
119-
// directory aside leaves a record behind that names us as though nothing had happened.
146+
// The same process can hold this path twice in sequence, so the token is what tells this
147+
// acquisition's record from an earlier one that names the very same process.
120148
if (reading.owner.claimToken !== claim.claimToken) return 'not-owner';
121-
fs.rmSync(lockDirPath, { recursive: true, force: true });
122-
return 'removed';
149+
return clearLockDirectory(lockDirPath, ownerFilePath);
150+
}
151+
152+
/**
153+
* A lock directory is written by this module and holds one file: its record. So it is emptied
154+
* and removed rather than removed with everything inside, and a directory that turns out to
155+
* hold something else is left standing. That distinction is the difference between clearing a
156+
* lock and destroying whoever put their thing in that path.
157+
*/
158+
function clearLockDirectory(lockDirPath: string, ownerFilePath: string): 'removed' | 'unverified' {
159+
try {
160+
fs.unlinkSync(ownerFilePath);
161+
} catch (error) {
162+
if (errorCode(error) !== 'ENOENT') return 'unverified';
163+
}
164+
try {
165+
fs.rmdirSync(lockDirPath);
166+
return 'removed';
167+
} catch (error) {
168+
return errorCode(error) === 'ENOENT' ? 'removed' : 'unverified';
169+
}
123170
}
124171

125172
function clearStaleProcessLock(
@@ -137,137 +184,186 @@ function clearStaleProcessLock(
137184
// A lock path held by anything that is not a directory cannot carry a readable
138185
// owner record, so its age is the only evidence available about it.
139186
if (!lockStats.isDirectory()) {
140-
return reclaimWhenAbandoned(lockStats, ownerGraceMs)
141-
? reclaimProcessLockDirectory(lockDirPath, ownerFilePath, {
142-
stats: lockStats,
143-
claimToken: null,
144-
})
145-
: false;
187+
return (
188+
reclaimWhenAbandoned(lockStats, ownerGraceMs) &&
189+
reclaimLockUnderMutex(lockDirPath, ownerFilePath, ownerGraceMs, { kind: 'stray' })
190+
);
146191
}
147192

148193
const reading = readProcessLockOwner(ownerFilePath);
149194
if (reading.kind === 'owner') {
150-
if (isLiveProcessLockOwner(reading.owner)) {
151-
return false;
152-
}
153-
return reclaimProcessLockDirectory(lockDirPath, ownerFilePath, {
154-
stats: lockStats,
155-
claimToken: reading.owner.claimToken,
156-
});
195+
// A record identifies the acquisition that wrote it, so the directory around a claim judged
196+
// dead is that claim's property.
197+
return (
198+
!isLiveProcessLockOwner(reading.owner) &&
199+
reclaimLockUnderMutex(lockDirPath, ownerFilePath, ownerGraceMs, {
200+
kind: 'dead-claim',
201+
claimToken: reading.owner.claimToken,
202+
})
203+
);
157204
}
158205
// A record we cannot read leaves an owner whose identity is unknown, which is not
159206
// evidence of death. Only a record that is genuinely absent lets the directory's
160207
// own age speak for it.
161208
if (reading.kind === 'unreadable') {
162209
return false;
163210
}
164-
return reclaimWhenAbandoned(lockStats, ownerGraceMs)
165-
? reclaimProcessLockDirectory(lockDirPath, ownerFilePath, {
166-
stats: lockStats,
167-
claimToken: null,
168-
})
169-
: false;
211+
return (
212+
reclaimWhenAbandoned(lockStats, ownerGraceMs) &&
213+
reclaimLockUnderMutex(lockDirPath, ownerFilePath, ownerGraceMs, { kind: 'empty' })
214+
);
170215
}
171216

172217
function reclaimWhenAbandoned(lockStats: fs.Stats, ownerGraceMs: number): boolean {
173218
return Date.now() - lockStats.mtimeMs >= ownerGraceMs;
174219
}
175220

176221
/**
177-
* Moves the abandoned directory aside under a unique name before removing it, so
178-
* exactly one contender can reclaim one lock. Two plain removals look different from
179-
* the caller's side: the second succeeds silently on the path the first already
180-
* cleared, and both contenders continue as though they had freed the lock. `mkdir`'s
181-
* `EEXIST`, unchanged, stays the arbiter of the lock itself.
222+
* What the judgement outside the mutex found, in the one form the removal decision needs: a claim
223+
* whose owner is dead, a directory that has never held a record, or a path that is not one.
224+
*/
225+
type JudgedLock =
226+
| { kind: 'dead-claim'; claimToken: ProcessLockOwnerRecord['claimToken'] }
227+
| { kind: 'empty' }
228+
| { kind: 'stray' };
229+
230+
/**
231+
* An abandoned lock is the one directory this module destroys without having created it, and
232+
* two contenders that both remove it independently both walk away believing they freed the
233+
* path. So the decision is taken again inside a mutex of its own, aged by the same grace as the
234+
* lock it guards, and everyone who cannot hold it keeps polling.
235+
*
236+
* Nothing is moved out of the way first: an absent lock path is an invitation, and a lock parked
237+
* under another name would return to a path somebody else already wrote a record on. Each branch
238+
* re-decides from what is on disk now and removes in place, and a path that has already gone is
239+
* left untouched so the caller's next `mkdir` simply wins it.
182240
*/
183-
function reclaimProcessLockDirectory(
241+
function reclaimLockUnderMutex(
184242
lockDirPath: string,
185243
ownerFilePath: string,
244+
ownerGraceMs: number,
186245
judged: JudgedLock,
187246
): boolean {
188-
const asidePath = reclaimedLockPath(lockDirPath);
247+
if (!holdReclaimMutex(lockDirPath, ownerGraceMs)) return false;
189248
try {
190-
fs.renameSync(lockDirPath, asidePath);
191-
} catch (error) {
192-
return reclaimWithoutTheRename(lockDirPath, ownerFilePath, error);
193-
}
194-
if (!reclaimedLockIsTheOneJudged(asidePath, judged)) {
195-
restoreReclaimedLockDirectory(asidePath, lockDirPath);
196-
return false;
249+
switch (judged.kind) {
250+
case 'dead-claim':
251+
return removeDeadClaimLock(lockDirPath, ownerFilePath, judged.claimToken);
252+
case 'empty':
253+
return removeAbandonedEmptyLock(lockDirPath, ownerGraceMs);
254+
case 'stray':
255+
return removeStrayLockPath(lockDirPath);
256+
}
257+
} finally {
258+
releaseReclaimMutex(lockDirPath);
197259
}
198-
removeReclaimedLockDirectory(asidePath);
199-
return true;
200260
}
201261

202262
/**
203-
* A win32 directory with a handle open inside it refuses the rename and often the removal
204-
* too, which is why a forced removal remains as the fallback: stale-clear atomicity is
205-
* best effort there and exact elsewhere. The fallback re-reads the record first, because a
206-
* refused rename means time passed and a live contender may have claimed the path in it.
263+
* The mutex says no other contender is reclaiming. It says nothing about the lock's owner, who
264+
* may have released the path and handed it to someone new while this process took the mutex, so
265+
* the record decides: a claim token is a random id no later acquisition can repeat, and a
266+
* directory whose record carries any other token, or no record at all, belongs to somebody else.
207267
*/
208-
function reclaimWithoutTheRename(
268+
function removeDeadClaimLock(
209269
lockDirPath: string,
210270
ownerFilePath: string,
211-
renameError: unknown,
271+
claimToken: ProcessLockOwnerRecord['claimToken'],
212272
): boolean {
213-
const code = (renameError as NodeJS.ErrnoException | null)?.code;
214-
if (code === 'ENOENT') return true;
215-
if (code !== 'EPERM' && code !== 'EACCES' && code !== 'ENOTEMPTY') return false;
216273
const reading = readProcessLockOwner(ownerFilePath);
217-
if (reading.kind === 'owner' && isLiveProcessLockOwner(reading.owner)) return false;
218-
fs.rmSync(lockDirPath, { recursive: true, force: true });
219-
return true;
274+
if (reading.kind !== 'owner' || reading.owner.claimToken !== claimToken) return false;
275+
try {
276+
fs.rmSync(lockDirPath, { recursive: true, force: true });
277+
return true;
278+
} catch {
279+
return false;
280+
}
281+
}
282+
283+
/**
284+
* A directory that has never held a record has no claim to attribute its contents to, so `rmdir`
285+
* is the only call made on it: it cannot destroy what a new owner published between the judgement
286+
* and here, and `ENOTEMPTY` is that publication saying so. Age re-speaks for the same reason — a
287+
* directory created a moment ago is an acquisition that has not published yet, not an abandoned one.
288+
*/
289+
function removeAbandonedEmptyLock(lockDirPath: string, ownerGraceMs: number): boolean {
290+
let current: fs.Stats;
291+
try {
292+
current = fs.statSync(lockDirPath);
293+
} catch {
294+
return false;
295+
}
296+
if (!current.isDirectory() || !reclaimWhenAbandoned(current, ownerGraceMs)) return false;
297+
try {
298+
fs.rmdirSync(lockDirPath);
299+
return true;
300+
} catch {
301+
return false;
302+
}
220303
}
221304

222305
/**
223-
* A rename addresses whatever stands at the path now, not the directory whose record was
224-
* read. A contender that reclaimed first and claimed the path again has put a different
225-
* directory there, so what arrived is compared against what was judged: the same inode,
226-
* no live owner inside, and the claim token that was read before the rename.
306+
* `unlink` answers `EISDIR` for a directory, which is a claim this process never held, so the
307+
* system call itself refuses the one case this branch must not touch.
227308
*/
228-
function reclaimedLockIsTheOneJudged(asidePath: string, judged: JudgedLock): boolean {
229-
let moved: fs.Stats;
309+
function removeStrayLockPath(lockDirPath: string): boolean {
230310
try {
231-
moved = fs.statSync(asidePath);
311+
fs.unlinkSync(lockDirPath);
312+
return true;
232313
} catch {
233314
return false;
234315
}
235-
if (moved.ino !== judged.stats.ino || moved.dev !== judged.stats.dev) return false;
236-
const reading = readProcessLockOwner(path.join(asidePath, OWNER_FILE_NAME));
237-
if (reading.kind === 'owner' && isLiveProcessLockOwner(reading.owner)) return false;
238-
return readClaimToken(reading) === judged.claimToken;
239316
}
240317

241-
type JudgedLock = {
242-
stats: fs.Stats;
243-
claimToken: string | null;
244-
};
318+
/** Keeps the `.lock` suffix so a sibling scanner still reads the name as a lock. */
319+
function reclaimMutexPath(lockDirPath: string): string {
320+
const stem = lockDirPath.endsWith(LOCK_DIRECTORY_SUFFIX)
321+
? lockDirPath.slice(0, -LOCK_DIRECTORY_SUFFIX.length)
322+
: lockDirPath;
323+
return `${stem}${RECLAIM_MUTEX_SUFFIX}${LOCK_DIRECTORY_SUFFIX}`;
324+
}
325+
326+
function holdReclaimMutex(lockDirPath: string, abandonedAfterMs: number): boolean {
327+
const mutexPath = reclaimMutexPath(lockDirPath);
328+
for (let attempt = 0; attempt < 2; attempt += 1) {
329+
try {
330+
fs.mkdirSync(mutexPath);
331+
return true;
332+
} catch (error) {
333+
if (errorCode(error) !== 'EEXIST') throw error;
334+
if (!clearAbandonedReclaimMutex(mutexPath, abandonedAfterMs)) return false;
335+
}
336+
}
337+
return false;
338+
}
245339

246340
/**
247-
* Returns a directory that turned out to belong to someone else. Failure is not a
248-
* licence to delete it: its holder's own release reports an unreadable owner rather than
249-
* let this process clear a lock it does not own.
341+
* A mutex left behind by a process that died mid-reclaim is cleared by age, the same evidence
342+
* an abandoned lock is judged by. Two contenders may both decide to clear it; the `mkdir` that
343+
* follows still admits one of them.
250344
*/
251-
function restoreReclaimedLockDirectory(asidePath: string, lockDirPath: string): void {
345+
function clearAbandonedReclaimMutex(mutexPath: string, abandonedAfterMs: number): boolean {
346+
let stats: fs.Stats;
347+
try {
348+
stats = fs.statSync(mutexPath);
349+
} catch {
350+
return true;
351+
}
352+
if (Date.now() - stats.mtimeMs < abandonedAfterMs) return false;
252353
try {
253-
fs.renameSync(asidePath, lockDirPath);
354+
fs.rmdirSync(mutexPath);
254355
} catch {}
356+
return true;
255357
}
256358

257-
/** Janitorial work after the rename already achieved the exclusion. */
258-
function removeReclaimedLockDirectory(asidePath: string): void {
359+
function releaseReclaimMutex(lockDirPath: string): void {
259360
try {
260-
fs.rmSync(asidePath, { recursive: true, force: true });
361+
fs.rmdirSync(reclaimMutexPath(lockDirPath));
261362
} catch {}
262363
}
263364

264-
/** Keeps the `.lock` suffix so a sibling scanner still reads the name as a lock. */
265-
function reclaimedLockPath(lockDirPath: string): string {
266-
const token = `${process.pid}-${crypto.randomUUID()}`;
267-
const stem = lockDirPath.endsWith(LOCK_DIRECTORY_SUFFIX)
268-
? lockDirPath.slice(0, -LOCK_DIRECTORY_SUFFIX.length)
269-
: lockDirPath;
270-
return `${stem}.reclaimed-${token}${LOCK_DIRECTORY_SUFFIX}`;
365+
function errorCode(error: unknown): string | undefined {
366+
return (error as NodeJS.ErrnoException | null)?.code;
271367
}
272368

273369
/**
@@ -287,10 +383,6 @@ function readProcessLockOwner(ownerFilePath: string): ProcessLockOwnerReading {
287383
return owner ? { kind: 'owner', owner } : { kind: 'unreadable' };
288384
}
289385

290-
function readClaimToken(reading: ProcessLockOwnerReading): string | null {
291-
return reading.kind === 'owner' ? reading.owner.claimToken : null;
292-
}
293-
294386
function parseProcessLockOwner(contents: string): ProcessLockOwnerRecord | null {
295387
let parsed: unknown;
296388
try {

‎packages/managed-allocation/src/__tests__/store.test.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -126,13 +126,13 @@ test('retains corrupt, unsupported, unfenced, and path-ambiguous records as diag
126126
assert.equal(path.dirname(path.dirname(operationPath)), path.join(root, 'allocations'));
127127
});
128128

129-
test('listing a lane lock and the directory a stale reclaim renamed aside finds no extra record', () => {
129+
test('listing a lane lock and the mutex a stale reclaim holds finds no extra record', () => {
130130
const { root, store, record } = fixture();
131131
store.create(record);
132132
const listed = store.list();
133133
const allocationsDir = path.join(root, 'allocations');
134134
const lane = path.basename(path.dirname(store.resolvePath(record)));
135-
for (const lockName of [`${lane}.lane.lock`, `${lane}.reclaimed-4242-0.lock`]) {
135+
for (const lockName of [`${lane}.lane.lock`, `${lane}.lane.reclaim.lock`]) {
136136
const lockDir = path.join(allocationsDir, lockName);
137137
fs.mkdirSync(lockDir, { recursive: true });
138138
fs.writeFileSync(path.join(lockDir, 'owner.json'), JSON.stringify({ pid: process.pid }));

‎packages/managed-allocation/src/store-filesystem.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -114,8 +114,8 @@ function readDirectory(directory: string): DirectoryRead {
114114
}
115115

116116
function listLanePaths(allocationsDir: string, lane: fs.Dirent): AllocationOperationPath[] {
117-
// Lane and operation locks sit beside the records, and a stale reclaim renames one
118-
// aside under its own name; none of those directories holds an operation record.
117+
// Lane and operation locks sit beside the records, including the `.reclaim` mutex a stale
118+
// reclaim holds while it decides; none of those directories holds an operation record.
119119
if (lane.name.endsWith('.lock')) return [];
120120
const lanePath = path.join(allocationsDir, lane.name);
121121
if (!lane.isDirectory()) {

0 commit comments

Comments
 (0)