Skip to content

Commit d56d82d

Browse files
fix(cache): reclaim stale filesystem cache files reliably (#21539)
* fix: age unreferenced cache files by recorded time The cleanup grace period was measured from each file's modification time, so a cache restore that refreshes mtimes kept resetting it and orphans in the directory were never expired. Record when a file is first seen unreferenced and age it from that instead, which survives a restore. The index backup still ages by mtime because every store rewrites it. * fix: also age unreferenced cache files by recorded time Aging orphans only by modification time leaves them forever in caches that are restored with refreshed mtimes, which is how the directory kept growing across CI runs. Record when a file is first seen unreferenced and treat either signal as expired, so a restored cache still shrinks while a cache that kept its modification times is cleaned on the very first store as before. * fix: collect every fully expired pack in one store Garbage collection was limited to a single pack per store to avoid the cost of unpacking, so a cache holding many stale packs needed as many builds to shrink. Dropping a pack whose items all expired needs no unpacking at all, so collect those in one pass and keep the rate limit only for packs that must be rewritten. * fix: reword comment so the spellchecker accepts it * fix: do not trust a recorded time for a just written cache file A concurrent build sharing the cache directory can reuse a pack name with the same size, which keeps the recorded time of the previous file and could expire a file written seconds ago. Ignore the recorded time while a file is that new, and stop collecting the oldest content when expiry already emptied the items.
1 parent ae67e2a commit d56d82d

4 files changed

Lines changed: 383 additions & 175 deletions

File tree

‎.changeset/055-cache-cleanup-unused-files.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,4 @@
22
"webpack": patch
33
---
44

5-
Delete no longer referenced files from the filesystem cache directory after storing the cache.
5+
Delete no longer referenced files from the filesystem cache directory after storing the cache, age them by recorded time so restored caches are cleaned too, and collect every fully expired pack in one store instead of one per build.

‎lib/cache/PackFileCacheStrategy.js‎

Lines changed: 146 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -35,10 +35,17 @@ const {
3535
/** @typedef {Set<string>} Items */
3636
/** @typedef {Set<string>} BuildDependencies */
3737
/** @typedef {Map<string, PackItemInfo>} ItemInfo */
38+
/** @typedef {{ firstSeen: number, size: number }} UnreferencedFile */
39+
/** @typedef {Map<string, UnreferencedFile>} UnreferencedFiles */
3840

39-
// Unreferenced files younger than this are kept to not race concurrent builds sharing the cache directory.
40-
// Kept below typical CI build durations so restored caches with refreshed modification times still get cleaned.
41+
// Unreferenced files are kept for this long to not race concurrent builds sharing the cache directory.
4142
const CLEANUP_GRACE_PERIOD = 30 * 60 * 1000;
43+
// Records when each unreferenced file was first seen. Aging by recorded time keeps
44+
// orphans expiring across caches restored with refreshed modification times.
45+
const UNREFERENCED_FILE = "unreferenced.json";
46+
// A file written this recently may belong to a concurrent build that reused the name,
47+
// in which case the recorded time describes the previous file and must not be trusted.
48+
const CLEANUP_RECENT_WRITE_PERIOD = 60 * 1000;
4249

4350
class PackContainer {
4451
/**
@@ -574,6 +581,41 @@ class Pack {
574581
}
575582
}
576583

584+
/**
585+
* Drops every content whose items all expired. Unlike a partial collection this
586+
* never unpacks, so it is not limited to a single content per store and lets a
587+
* long unused cache shrink in one go instead of one pack per build.
588+
*/
589+
_gcExpiredContent() {
590+
const now = Date.now();
591+
let packCount = 0;
592+
let itemCount = 0;
593+
for (let loc = 0; loc < this.content.length; loc++) {
594+
const content = this.content[loc];
595+
if (!content) continue;
596+
let expired = true;
597+
for (const identifier of content.items) {
598+
const info = this.itemInfo.get(identifier);
599+
if (info !== undefined && now - info.lastAccess <= this.maxAge) {
600+
expired = false;
601+
break;
602+
}
603+
}
604+
if (!expired) continue;
605+
for (const identifier of content.items) this.itemInfo.delete(identifier);
606+
this.content[loc] = undefined;
607+
packCount++;
608+
itemCount += content.items.size;
609+
}
610+
if (packCount > 0) {
611+
this.logger.log(
612+
"Garbage Collected %d completely expired packs with %d items",
613+
packCount,
614+
itemCount
615+
);
616+
}
617+
}
618+
577619
/**
578620
* Find the content with the oldest item and run GC on that.
579621
* Only runs for one content to avoid large invalidation.
@@ -586,11 +628,10 @@ class Pack {
586628
oldest = info;
587629
}
588630
}
589-
if (
590-
Date.now() - /** @type {PackItemInfo} */ (oldest).lastAccess >
591-
this.maxAge
592-
) {
593-
const loc = /** @type {PackItemInfo} */ (oldest).location;
631+
// collecting expired content may have left no items at all
632+
if (oldest === undefined) return;
633+
if (Date.now() - oldest.lastAccess > this.maxAge) {
634+
const loc = oldest.location;
594635
if (loc < 0) return;
595636
const content = /** @type {PackContent} */ (this.content[loc]);
596637
const items = new Set(content.items);
@@ -626,6 +667,7 @@ class Pack {
626667
this._persistFreshContent();
627668
this._optimizeSmallContent();
628669
this._optimizeUnusedContent();
670+
this._gcExpiredContent();
629671
this._gcOldestContent();
630672
for (const identifier of this.itemInfo.keys()) {
631673
write(identifier);
@@ -1648,6 +1690,61 @@ class PackFileCacheStrategy {
16481690
}));
16491691
}
16501692

1693+
/**
1694+
* Reads when the currently unreferenced files were first seen. A missing or
1695+
* unreadable file just restarts the grace period for every orphan.
1696+
* @returns {Promise<UnreferencedFiles>} first seen time and size per file name
1697+
*/
1698+
_readUnreferencedFiles() {
1699+
return new Promise((resolve) => {
1700+
this.fs.readFile(
1701+
`${this.cacheLocation}/${UNREFERENCED_FILE}`,
1702+
(err, content) => {
1703+
/** @type {UnreferencedFiles} */
1704+
const result = new Map();
1705+
if (err) return resolve(result);
1706+
try {
1707+
const data = JSON.parse(
1708+
/** @type {Buffer} */ (content).toString("utf8")
1709+
);
1710+
for (const [file, entry] of Object.entries(data)) {
1711+
const { firstSeen, size } = /** @type {UnreferencedFile} */ (
1712+
entry
1713+
);
1714+
if (typeof firstSeen === "number" && typeof size === "number") {
1715+
result.set(file, { firstSeen, size });
1716+
}
1717+
}
1718+
} catch (_err) {
1719+
result.clear();
1720+
}
1721+
resolve(result);
1722+
}
1723+
);
1724+
});
1725+
}
1726+
1727+
/**
1728+
* Persists when the still unreferenced files were first seen. Failing to write
1729+
* only costs the orphans another grace period, so errors are ignored.
1730+
* @param {UnreferencedFiles} unreferenced first seen time and size per file name
1731+
* @param {boolean} hadEntries whether a previous state exists that must be replaced
1732+
* @returns {Promise<void>} promise
1733+
*/
1734+
_writeUnreferencedFiles(unreferenced, hadEntries) {
1735+
if (unreferenced.size === 0 && !hadEntries) return Promise.resolve();
1736+
/** @type {Record<string, UnreferencedFile>} */
1737+
const data = {};
1738+
for (const [file, entry] of unreferenced) data[file] = entry;
1739+
return new Promise((resolve) => {
1740+
this.fs.writeFile(
1741+
`${this.cacheLocation}/${UNREFERENCED_FILE}`,
1742+
JSON.stringify(data),
1743+
() => resolve()
1744+
);
1745+
});
1746+
}
1747+
16511748
/**
16521749
* Deletes files from the cache directory that are no longer referenced by the
16531750
* stored pack. Retained files are walked on disk since nested lazy segments
@@ -1666,7 +1763,7 @@ class PackFileCacheStrategy {
16661763
// rewritten files may reference different names now
16671764
for (const name of writtenNames) referencedFilesCache.delete(name);
16681765
/** @type {Set<string>} */
1669-
const liveFiles = new Set([`index${extension}`]);
1766+
const liveFiles = new Set([`index${extension}`, UNREFERENCED_FILE]);
16701767
for (const name of writtenNames) liveFiles.add(`${name}${extension}`);
16711768
/** @type {string[]} */
16721769
const queue = [];
@@ -1711,28 +1808,56 @@ class PackFileCacheStrategy {
17111808
resolve(/** @type {string[]} */ (files));
17121809
});
17131810
});
1714-
const expireTime = Date.now() - CLEANUP_GRACE_PERIOD;
1811+
const seenFiles = await this._readUnreferencedFiles();
1812+
const now = Date.now();
1813+
// every store rewrites the index backup, so a recorded time would age a file
1814+
// that is in fact new; renaming carries the previous index mtime onto it
1815+
const indexBackup = `index${extension}.old`;
1816+
/** @type {UnreferencedFiles} */
1817+
const stillUnreferenced = new Map();
17151818
let deletedCount = 0;
17161819
for (const file of files) {
17171820
if (typeof file !== "string" || liveFiles.has(file)) continue;
17181821
const path = `${cacheLocation}/${file}`;
1719-
const deleted = await new Promise((resolve) => {
1822+
const stats = await new Promise((resolve) => {
17201823
fs.stat(path, (err, stats) => {
1721-
if (
1722-
err ||
1723-
!(/** @type {import("../util/fs").IStats} */ (stats).isFile()) ||
1724-
/** @type {number} */ (
1725-
/** @type {import("../util/fs").IStats} */ (stats).mtimeMs
1726-
) > expireTime
1727-
) {
1728-
return resolve(false);
1729-
}
1824+
resolve(
1825+
err
1826+
? undefined
1827+
: /** @type {import("../util/fs").IStats} */ (stats)
1828+
);
1829+
});
1830+
});
1831+
if (stats === undefined || !stats.isFile()) continue;
1832+
const size = /** @type {number} */ (stats.size);
1833+
const seen = seenFiles.get(file);
1834+
// a differing size means the name was rewritten, so it is a new orphan
1835+
const firstSeen =
1836+
seen !== undefined && seen.size === size ? seen.firstSeen : now;
1837+
const expireTime = now - CLEANUP_GRACE_PERIOD;
1838+
const mtimeMs = /** @type {number} */ (stats.mtimeMs);
1839+
// either signal is enough: modification times are lost when a cache is
1840+
// restored, and recorded times are lost when the cache directory is new
1841+
const expired =
1842+
mtimeMs <= expireTime ||
1843+
(file !== indexBackup &&
1844+
firstSeen <= expireTime &&
1845+
mtimeMs <= now - CLEANUP_RECENT_WRITE_PERIOD);
1846+
if (expired) {
1847+
const deleted = await new Promise((resolve) => {
17301848
/** @type {NonNullable<IntermediateFileSystem["unlink"]>} */
17311849
(fs.unlink)(path, (err) => resolve(!err));
17321850
});
1733-
});
1734-
if (deleted) deletedCount++;
1851+
if (deleted) {
1852+
deletedCount++;
1853+
continue;
1854+
}
1855+
}
1856+
if (file !== indexBackup) {
1857+
stillUnreferenced.set(file, { firstSeen, size });
1858+
}
17351859
}
1860+
await this._writeUnreferencedFiles(stillUnreferenced, seenFiles.size > 0);
17361861
// drop entries of files that are no longer alive
17371862
for (const name of referencedFilesCache.keys()) {
17381863
if (!liveFiles.has(`${name}${extension}`)) {

0 commit comments

Comments
 (0)