Skip to content

Commit 36ad681

Browse files
author
lane-u5a-2359
committed
test: release the lock and reap the writer when a merge row fails
The waiting rows in this block hold the lock in this process and launch a writer that is refused until they let go, and every release and every reap in them sits inline after an assertion, so an assertion that lands between the acquire and the release leaves this process holding the lock and the writer it launched still contending for it. That is the half of the reply on row 4544685126 that the longer retry window does not answer: more attempts change how long a writer is willing to wait, not what becomes of the lock and the child when a row stops early. Guard the failure path for the block those rows live in, in the shape the row named for letting go of both locks and the renewal timer already uses there, releasing only what has not been released yet and then awaiting the writer, which is admitted the moment the artefact is gone and exits on its own. Pin it with two rows, because a guard nobody has failed into is a shape and not a result. The first forces the failure these rows can hit, an assertion that lands while this process still holds the lock and the writer it launched is still refused, and records that at that moment the artefact is still there and the writer has no outcome line, which is the hole and not the fix. The second runs after the guard and asserts what the guard left, namely no artefact, a writer that was admitted, wrote, released and exited, and the same path acquirable again. Without the guard the second row fails and the first passes; with it both pass and the file suite carries 31 passed and 8 skipped of 39 where it carried 29 passed and 8 skipped of 37. No production file changed, so the priced unit delta measured against the unit base is unchanged, and the mutation leg for these commits is still unmeasured locally.
1 parent 9bedf7e commit 36ad681

1 file changed

Lines changed: 90 additions & 0 deletions

File tree

‎src/utils/__tests__/fileLock.test.ts‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1502,6 +1502,31 @@ describe("a writer that had to wait merges into what landed while it waited", ()
15021502
beforeEach(() => {
15031503
vi.mocked(lockfile.lock).mockImplementation((target, options) => actualLock(target, options))
15041504
})
1505+
let heldRelease: (() => Promise<void>) | null = null
1506+
let liveWaiter: HolderHandle | null = null
1507+
let forcedDirectory: string | null = null
1508+
let forcedTarget: string | null = null
1509+
let forcedTrace: string | null = null
1510+
1511+
// The hole the three rows above share: a row that gives up part way, at an assertion between its
1512+
// acquire and its release, leaves this process holding the lock and the writer it launched still
1513+
// contending for it, because every release and every reap in those rows sits inline and an
1514+
// assertion that lands skips the lines after it. This guard takes the shape the row named for
1515+
// letting go of both locks and the renewal timer already uses, releasing only what has not been
1516+
// released yet, and it sits on the block rather than inside one row because the same hole is in
1517+
// all three of them here.
1518+
afterEach(async () => {
1519+
if (heldRelease !== null) {
1520+
await heldRelease()
1521+
heldRelease = null
1522+
}
1523+
if (liveWaiter !== null) {
1524+
// Releasing first is what lets the writer finish on its own: it is refused while this
1525+
// process still holds, and it writes, releases and exits once the artefact is gone.
1526+
await liveWaiter.waitForExit()
1527+
liveWaiter = null
1528+
}
1529+
})
15051530

15061531
// The queue this module builds when the artefact belongs to someone else: for a plain call
15071532
// fileLock.ts:119-125 hands the library five retries after the first attempt, capped at 1000 ms
@@ -1541,6 +1566,7 @@ describe("a writer that had to wait merges into what landed while it waited", ()
15411566
writeKey: "child",
15421567
baseMode: "under",
15431568
})
1569+
liveWaiter = holder
15441570
const artefactLine = await readHolderTrace(holder.tracePath, "ARTEFACT", 15000, holder.childOutput)
15451571
const outcomeLine = await readHolderTrace(holder.tracePath, "OUTCOME", 15000, holder.childOutput)
15461572
expect(traceArtefact(artefactLine)).toBe(path.join(directory, "history.json.lock"))
@@ -1557,6 +1583,7 @@ describe("a writer that had to wait merges into what landed while it waited", ()
15571583
safeWriteJson(target, { writerB: 1 }, { merge: mergeRecords }),
15581584
])
15591585
expect(await holder.waitForExit()).toBe(0)
1586+
liveWaiter = null
15601587
expect(readJsonBody(target)).toEqual({ pre: 1, child: 1, writerA: 1, writerB: 1 })
15611588
expect(artefactsOf(directory)).toEqual([])
15621589
},
@@ -1568,13 +1595,15 @@ describe("a writer that had to wait merges into what landed while it waited", ()
15681595
async () => {
15691596
const { directory, target } = scenario("merge-waiter-under")
15701597
const release = await acquireFileLock(target, { immediate: true })
1598+
heldRelease = release
15711599
const waiter = startLockHolder(target, "hold", lockModule, directory, nextTraceName(), false, {
15721600
holdMs: 0,
15731601
writeKey: "child",
15741602
baseMode: "under",
15751603
retries: waiterAttemptsForOutcomeBudget,
15761604
retryMs: 250,
15771605
})
1606+
liveWaiter = waiter
15781607
await readHolderTrace(waiter.tracePath, "ARTEFACT", 15000, waiter.childOutput)
15791608
// Observed rather than assumed: the waiter attempted while this process still held the lock and
15801609
// was refused, so the row cannot pass with a writer that merely ran after the write below.
@@ -1586,11 +1615,13 @@ describe("a writer that had to wait merges into what landed while it waited", ()
15861615
expect(fs.readFileSync(waiter.tracePath, "utf8")).not.toContain("OUTCOME held")
15871616
writeMergedKey(target, "writerA")
15881617
await release()
1618+
heldRelease = null
15891619
const outcomeLine = await readHolderTrace(waiter.tracePath, "OUTCOME", 25000, waiter.childOutput)
15901620
expect(traceOutcome(outcomeLine)).toBe("held")
15911621
const baseLine = await readHolderTrace(waiter.tracePath, "BASE-UNDER-LOCK", 15000, waiter.childOutput)
15921622
expect(tracePayload(baseLine, "BASE-UNDER-LOCK")).toEqual({ pre: 1, writerA: 1 })
15931623
expect(await waiter.waitForExit()).toBe(0)
1624+
liveWaiter = null
15941625
expect(readJsonBody(target)).toEqual({ pre: 1, writerA: 1, child: 1 })
15951626
expect(artefactsOf(directory)).toEqual([])
15961627
},
@@ -1602,13 +1633,15 @@ describe("a writer that had to wait merges into what landed while it waited", ()
16021633
async () => {
16031634
const { directory, target } = scenario("merge-waiter-before")
16041635
const release = await acquireFileLock(target, { immediate: true })
1636+
heldRelease = release
16051637
const waiter = startLockHolder(target, "hold", lockModule, directory, nextTraceName(), false, {
16061638
holdMs: 0,
16071639
writeKey: "child",
16081640
baseMode: "before",
16091641
retries: waiterAttemptsForOutcomeBudget,
16101642
retryMs: 250,
16111643
})
1644+
liveWaiter = waiter
16121645
const beforeLine = await readHolderTrace(waiter.tracePath, "BASE-BEFORE", 15000, waiter.childOutput)
16131646
expect(tracePayload(beforeLine, "BASE-BEFORE")).toEqual({ pre: 1 })
16141647
// The base was read before the attempt, and the attempt then happened and was refused; both are
@@ -1618,17 +1651,74 @@ describe("a writer that had to wait merges into what landed while it waited", ()
16181651
expect(fs.readFileSync(waiter.tracePath, "utf8")).not.toContain("OUTCOME held")
16191652
writeMergedKey(target, "writerA")
16201653
await release()
1654+
heldRelease = null
16211655
const outcomeLine = await readHolderTrace(waiter.tracePath, "OUTCOME", 25000, waiter.childOutput)
16221656
expect(traceOutcome(outcomeLine)).toBe("held")
16231657
const usedLine = await readHolderTrace(waiter.tracePath, "BASE-USED", 15000, waiter.childOutput)
16241658
expect(tracePayload(usedLine, "BASE-USED")).toEqual({ pre: 1 })
16251659
expect(await waiter.waitForExit()).toBe(0)
1660+
liveWaiter = null
16261661
const content = readJsonBody(target)
16271662
expect(content).toEqual({ pre: 1, child: 1 })
16281663
expect(content).not.toHaveProperty("writerA")
16291664
expect(artefactsOf(directory)).toEqual([])
16301665
},
16311666
)
1667+
1668+
it(
1669+
"leaves the lock held and the waiting writer refused when a row fails between the two",
1670+
{ timeout: 60000 },
1671+
async () => {
1672+
const { directory, target } = scenario("merge-waiter-forced-failure")
1673+
forcedDirectory = directory
1674+
forcedTarget = target
1675+
heldRelease = await acquireFileLock(target, { immediate: true })
1676+
liveWaiter = startLockHolder(target, "hold", lockModule, directory, nextTraceName(), false, {
1677+
holdMs: 0,
1678+
writeKey: "child",
1679+
baseMode: "under",
1680+
retries: waiterAttemptsForOutcomeBudget,
1681+
retryMs: 250,
1682+
})
1683+
forcedTrace = liveWaiter.tracePath
1684+
await readHolderTrace(liveWaiter.tracePath, "REFUSED", 15000, liveWaiter.childOutput)
1685+
let forced = ""
1686+
try {
1687+
// The failure these rows can hit, forced on purpose so that the letting go is what is
1688+
// tested and not the merge: an assertion that lands while this process still holds the
1689+
// lock and while the writer it launched is still being refused.
1690+
expect(artefactsOf(directory)).toEqual([])
1691+
} catch (thrown) {
1692+
forced = String(thrown).slice(0, 40)
1693+
}
1694+
expect(forced).not.toBe("")
1695+
// What that failure leaves at the moment it lands, before this block guard runs: the
1696+
// artefact still under this process and the writer still short of its outcome line. The
1697+
// next row is the assertion about the guard, because a hook cannot run inside the row that
1698+
// failed.
1699+
expect(artefactsOf(directory)).toEqual(["history.json.lock"])
1700+
expect(fs.readFileSync(forcedTrace ?? "", "utf8")).not.toContain("OUTCOME")
1701+
},
1702+
)
1703+
1704+
it(
1705+
"gives back the lock and reaps the waiting writer that a failed row left behind",
1706+
{ timeout: 60000 },
1707+
async () => {
1708+
if (forcedDirectory === null || forcedTarget === null || forcedTrace === null) {
1709+
throw new Error("the forced failure row recorded no objects to check")
1710+
}
1711+
// Running here, after the guard of this block, is the point: what is asserted is what the
1712+
// guard left, namely no artefact under this process, the writer admitted once the lock was
1713+
// free having written, let go and exited, and the same path acquirable again.
1714+
expect(artefactsOf(forcedDirectory)).toEqual([])
1715+
expect(fs.readFileSync(forcedTrace, "utf8")).toContain("RELEASED")
1716+
heldRelease = await acquireFileLock(forcedTarget, { immediate: true })
1717+
await heldRelease()
1718+
heldRelease = null
1719+
expect(artefactsOf(forcedDirectory)).toEqual([])
1720+
},
1721+
)
16321722
})
16331723

16341724
/**

0 commit comments

Comments
 (0)