Skip to content

Commit 228df5f

Browse files
authored
[backport] Retain fewer stale cache versions and use a TTL, plus the mtime fallback (#97304)
Backports the following to `next-16-3`, as a pair: - #96941 — "[turbopack] Retain fewer stale cache versions and use a TTL" (`ae1e53a`) - #97268 — "Add the mtime fallback back" (`52a3410`) The second fixes a bug in the first: #96941 changes the `CURRENT` file format from a bare big-endian `u32` to a JSON object, and an existing cache directory in the old format could no longer be aged. #97268 adds the mtime fallback so an unparsable `CURRENT` is aged by its mtime instead of failing. They only make sense together, hence one branch. Both cherry-picked with `-x`, no conflicts. Every touched source file is byte-identical to canary. ### Verification on this branch - `cargo check`, `cargo clippy --all-targets`, `cargo fmt --check` on `turbo-persistence` and `turbo-tasks-backend` — clean - `cargo test -p turbo-persistence -p turbo-tasks-backend` — all pass, 0 failures - includes the `db_versioning` tests, notably `test_unparsable_current_falls_back_to_mtime::case_1_old_u32_format` (the old-format compatibility case #97268 restores) - `cargo metadata --locked` — `Cargo.lock` consistent with the manifests, no lockfile update needed - `pnpm --filter=next types`, prettier, eslint — clean - `pnpm build-all` — 18/18 tasks - `test/production/next-server-nft` — 5/5 tests, 5/5 snapshots pass (covers the traced `turbopack-cache-seed.js`) ### End-to-end verification against a real old-format cache `[email protected]` + `next build` (with `turbopackFileSystemCacheForBuild`) on released `[email protected]` writes a `CURRENT` in the old pre-JSON format — a bare big-endian `u32` (`0000 0008`), 30M cache. This branch was then applied with `pnpm pack-next -p <app> -- --release` (release binary confirmed to contain `TURBO_ENGINE_VERSION_TTL_DAYS` and the new `CURRENT` field names), and two arms were run from an identical starting state — same cache, same code, only the env var differing: | Arm | Build | Cache dirs after | | --- | --- | --- | | `TURBO_ENGINE_VERSION_TTL_DAYS=0` | exit 0 | old dir **deleted**, 1 remains | | default TTL (3 days) | exit 0 | old dir **retained**, intact at 30M, 2 remain | The retention arm is the meaningful control: without #97268's fallback, `read_current_version` would error on the bare-`u32` file, the age would become `Duration::MAX`, and the directory would be evicted in *both* arms. Retention under the default TTL is direct evidence the mtime fallback works. Also confirmed: - new `CURRENT` is written as JSON: `{"max_sequence_number":8,"commit_time":"..."}` - retention cap holds: seeded 4 version dirs → build left exactly 2 (current + one most-recently-used other) - clean deletion, no `DELETION_PREFIX` leftovers in any run - `TURBO_ENGINE_VERSION_TTL_DAYS=notanumber` warns (`expected a whole number of days`) and falls back to the 3-day default rather than failing <!-- NEXT_JS_LLM -->
1 parent 16eab3a commit 228df5f

10 files changed

Lines changed: 431 additions & 114 deletions

File tree

‎Cargo.lock‎

Lines changed: 3 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎packages/next/src/lib/turbopack-cache-seed.ts‎

Lines changed: 18 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -76,10 +76,10 @@ function findSeedSource(
7676
const currentWorktree = path.resolve(worktreeInfo.worktreeRoot)
7777

7878
// We are going to find the best candidate worktree
79-
// based on the newest mtime of the CURRENT file in the cache directory.
79+
// based on the most recently written cache directory.
8080
// We only look at our version
8181

82-
let best: { versionDir: string; mtimeMs: number } | undefined
82+
let best: { versionDir: string; commitTimeMs: number } | undefined
8383
for (const root of [
8484
worktreeInfo.mainRepoRoot,
8585
...listLinkedWorktreeRoots(worktreeInfo.mainRepoRoot),
@@ -93,21 +93,31 @@ function findSeedSource(
9393
'turbopack',
9494
version
9595
)
96-
const mtimeMs = currentMtimeMs(versionDir)
97-
if (mtimeMs === undefined) continue
98-
if (!best || mtimeMs > best.mtimeMs) {
99-
best = { versionDir, mtimeMs }
96+
const commitTimeMs = currentCommitTimeMs(versionDir)
97+
if (commitTimeMs === undefined) continue
98+
if (!best || commitTimeMs > best.commitTimeMs) {
99+
best = { versionDir, commitTimeMs }
100100
}
101101
}
102102
return best?.versionDir
103103
}
104104

105-
function currentMtimeMs(versionDir: string): number | undefined {
105+
// When the cache in `versionDir` was last committed to, in epoch milliseconds, or undefined if
106+
// there is no usable cache there. Read from the `commit_time` the persistence layer records in
107+
// CURRENT.
108+
function currentCommitTimeMs(versionDir: string): number | undefined {
109+
let commitTime
106110
try {
107-
return fs.statSync(path.join(versionDir, 'CURRENT')).mtimeMs
111+
commitTime = JSON.parse(
112+
fs.readFileSync(path.join(versionDir, 'CURRENT'), 'utf8')
113+
).commit_time
108114
} catch {
115+
// missing, unreadable, or not valid JSON - treat it as not a seed candidate
109116
return undefined
110117
}
118+
if (typeof commitTime !== 'string') return undefined
119+
const parsed = Date.parse(commitTime)
120+
return Number.isNaN(parsed) ? undefined : parsed
111121
}
112122

113123
function dirHasEntries(dir: string): boolean {

‎turbopack/crates/turbo-persistence/Cargo.toml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ crc32fast = { workspace = true }
2020
dashmap = { workspace = true}
2121
either = { workspace = true }
2222
fs-err = { workspace = true }
23-
jiff = "0.2.10"
23+
jiff = { version = "0.2.10", features = ["serde"] }
2424
lzzzz = { workspace = true }
2525
memmap2 = "0.9.5"
2626
nohash-hasher = { workspace = true }
@@ -31,6 +31,8 @@ postcard = { workspace = true, features = ["alloc", "use-std"] }
3131
zerocopy = { version = "0.8", features = ["derive"] }
3232
quick_cache = { workspace = true }
3333
rustc-hash = { workspace = true }
34+
serde = { workspace = true }
35+
serde_json = { workspace = true }
3436
smallvec = { workspace = true }
3537
thread_local = { workspace = true }
3638
tracing = { workspace = true }

‎turbopack/crates/turbo-persistence/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ It supports having multiple key families, which are stored in separate files, bu
1212

1313
## On disk format
1414

15-
There is a single `CURRENT` file which stores the latest committed sequence number.
15+
There is a single `CURRENT` file, a small JSON object holding the latest committed sequence number (`max_sequence_number`) and when that commit happened (`commit_time`). The commit time is stored in the file rather than taken from its mtime so that it survives the directory being copied or restored. External tools read this file, so its field names are a stable contract.
1616

1717
All other files have a sequence number as file name, e. g. `0000123.sst`. All files are immutable once their sequence number is <= the committed sequence number. But they might be deleted when they are superseded by other committed files.
1818

‎turbopack/crates/turbo-persistence/src/bin/sst_inspect.rs‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ use turbo_persistence::{
2424
BLOCK_HEADER_SIZE, checksum_block,
2525
meta_file::MetaFile,
2626
mmap_helper::advise_mmap_for_persistence,
27+
read_current_version,
2728
sst_filter::SstFilter,
2829
static_sorted_file::{
2930
BLOCK_TYPE_FIXED_KEY_NO_HASH, BLOCK_TYPE_FIXED_KEY_WITH_HASH, BLOCK_TYPE_KEY_NO_HASH,
@@ -211,9 +212,9 @@ fn format_bytes(bytes: u64) -> String {
211212
/// and apply SstFilter to skip superseded entries.
212213
fn collect_sst_info(db_path: &Path) -> Result<BTreeMap<u32, Vec<SstInfo>>> {
213214
// Read the CURRENT sequence number — only files with seq <= current are valid.
214-
let current: u32 = File::open(db_path.join("CURRENT"))?
215-
.read_u32::<BE>()
216-
.context("Failed to read CURRENT file")?;
215+
let current = read_current_version(db_path)?
216+
.context("CURRENT file is missing")?
217+
.max_sequence_number;
217218

218219
// Read .del files to find sequences that were deleted but not yet cleaned up.
219220
let mut deleted_seqs: HashSet<u32> = HashSet::new();

‎turbopack/crates/turbo-persistence/src/db.rs‎

Lines changed: 69 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ use std::{
22
borrow::Cow,
33
collections::HashSet,
44
fmt::Display,
5-
io::{BufWriter, Write},
5+
io::{BufWriter, ErrorKind, Write},
66
mem::take,
77
ops::RangeInclusive,
88
path::{Path, PathBuf},
@@ -20,6 +20,7 @@ use jiff::Timestamp;
2020
use memmap2::Mmap;
2121
use nohash_hasher::BuildNoHashHasher;
2222
use parking_lot::{Mutex, RwLock};
23+
use serde::{Deserialize, Serialize};
2324
use smallvec::SmallVec;
2425
use tracing::span::EnteredSpan;
2526

@@ -159,24 +160,76 @@ impl WriteOperationGuard<'_> {
159160
}
160161
}
161162

163+
/// The contents of the `CURRENT` file: which sequence number is committed, and when that commit
164+
/// happened.
165+
///
166+
/// # Compatibility
167+
///
168+
/// Unlike other parts of the persistent database the `CURRENT` file is occasionally read by other
169+
/// versions of turbopack, so we should be careful when updating this struct
170+
///
171+
/// - Never rename a field. This will break readers from other versions
172+
/// - Never remove a field, unless it has always had `[serde(default)]`
173+
/// - Never change the type of a field
174+
/// - New fields should be `#[serde(default)]` and semantically optional to readers from other
175+
/// versions
176+
/// - Never add `#[serde(deny_unknown_fields)]`.
177+
///
178+
/// Field names are also parsed outside this crate (next.js reads `CURRENT` directly, in
179+
/// `turbopack-cache-seed.ts`), so a rename would have to move in lockstep there too.
180+
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
181+
pub struct CurrentDbVersion {
182+
/// The highest sequence number that is part of the committed database.
183+
pub max_sequence_number: u32,
184+
/// When this database was last committed to.
185+
pub commit_time: Timestamp,
186+
}
187+
188+
/// Reads the `CURRENT` file in the database directory `path`.
189+
///
190+
/// Returns `Ok(None)` if the file doesn't exist, which for a writable database means "not
191+
/// initialized yet".
192+
pub fn read_current_version(path: &Path) -> Result<Option<CurrentDbVersion>> {
193+
let current_path = path.join("CURRENT");
194+
let content = match fs::read(&current_path) {
195+
Ok(content) => content,
196+
Err(e) if e.kind() == ErrorKind::NotFound => return Ok(None),
197+
Err(e) => return Err(e).context("Failed to read CURRENT file"),
198+
};
199+
200+
serde_json::from_slice::<CurrentDbVersion>(&content)
201+
.with_context(|| {
202+
format!(
203+
"CURRENT file at {} is corrupt ({} bytes)",
204+
current_path.display(),
205+
content.len()
206+
)
207+
})
208+
.map(Some)
209+
}
210+
162211
/// Durably and atomically updates the `CURRENT` file in the database directory `path` to point at
163-
/// `seq`.
212+
/// `seq`, stamping it as used now.
164213
///
165-
/// The write is made atomic by writing `seq` to a temporary `CURRENT.next` file, flushing it, and
166-
/// then `rename`ing it over `CURRENT`. A `rename` within a directory is atomic on POSIX and
167-
/// replaces the destination on Windows, so a concurrent or crashing writer can never observe a
168-
/// torn `CURRENT` (in-place overwrites, by contrast, can leave a partially-written value on a
169-
/// crash mid-write). After the rename we fsync the directory so the new `CURRENT` → inode mapping
170-
/// survives a crash.
214+
/// The write is made atomic by writing to a temporary `CURRENT.next` file, flushing it, and then
215+
/// `rename`ing it over `CURRENT`. A `rename` within a directory is atomic on POSIX and replaces the
216+
/// destination on Windows, so a concurrent or crashing writer can never observe a torn `CURRENT`
217+
/// (in-place overwrites, by contrast, can leave a partially-written value on a crash mid-write).
218+
/// After the rename we fsync the directory so the new `CURRENT` → inode mapping survives a crash.
171219
fn commit_current(path: &Path, seq: u32) -> Result<()> {
220+
let version: &CurrentDbVersion = &CurrentDbVersion {
221+
max_sequence_number: seq,
222+
commit_time: Timestamp::now(),
223+
};
224+
let mut contents =
225+
serde_json::to_string(version).context("Failed to serialize the CURRENT file")?;
226+
contents.push('\n');
172227
let next_path = path.join("CURRENT.next");
173228
let mut next_file = File::create(&next_path)?;
174-
next_file.write_u32::<BE>(seq)?;
229+
next_file.write_all(contents.as_bytes())?;
175230
next_file.sync_data()?;
176231
drop(next_file);
177-
178232
fs::rename(&next_path, path.join("CURRENT"))?;
179-
180233
// Fsync the directory. This is the single durability barrier for a commit: by the time we get
181234
// here every file created earlier in the commit (SST/meta/blob and any `.del` file) already
182235
// exists, so this one fsync flushes *all* of their directory entries together with the CURRENT
@@ -459,7 +512,7 @@ impl<S: ParallelScheduler, const FAMILIES: usize> TurboPersistence<S, FAMILIES>
459512
Ok(())
460513
}
461514
Err(e) => {
462-
if !read_only && e.kind() == std::io::ErrorKind::NotFound {
515+
if !read_only && e.kind() == ErrorKind::NotFound {
463516
self.create_and_init_directory()
464517
.context("Creating and initializing persistence directory failed")?;
465518
Ok(())
@@ -479,18 +532,11 @@ impl<S: ParallelScheduler, const FAMILIES: usize> TurboPersistence<S, FAMILIES>
479532
/// Loads an existing database directory and performs cleanup if necessary.
480533
fn load_directory(&mut self, entries: ReadDir, read_only: bool) -> Result<bool> {
481534
let mut meta_files = Vec::new();
482-
let mut current_file = match File::open(self.path.join("CURRENT")) {
483-
Ok(file) => file,
484-
Err(e) => {
485-
if !read_only && e.kind() == std::io::ErrorKind::NotFound {
486-
return Ok(false);
487-
} else {
488-
return Err(e).context("Failed to open CURRENT file");
489-
}
490-
}
535+
let current = match read_current_version(&self.path)? {
536+
Some(version) => version.max_sequence_number,
537+
None if !read_only => return Ok(false),
538+
None => bail!("Failed to open database: CURRENT file is missing"),
491539
};
492-
let current = current_file.read_u32::<BE>()?;
493-
drop(current_file);
494540

495541
let mut deleted_files = HashSet::new();
496542
for entry in entries {

‎turbopack/crates/turbo-persistence/src/lib.rs‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,10 @@ mod tests;
3030

3131
pub use arc_bytes::ArcBytes;
3232
pub use compression::checksum_block;
33-
pub use db::{CommitStats, CompactConfig, MetaFileEntryInfo, MetaFileInfo, TurboPersistence};
33+
pub use db::{
34+
CommitStats, CompactConfig, CurrentDbVersion, MetaFileEntryInfo, MetaFileInfo,
35+
TurboPersistence, read_current_version,
36+
};
3437

3538
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
3639
pub enum FamilyKind {

‎turbopack/crates/turbo-persistence/src/tests.rs‎

Lines changed: 36 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ use rayon::iter::{IntoParallelIterator, ParallelIterator};
66
use crate::{
77
DbConfig, FamilyConfig, FamilyKind,
88
constants::{MAX_MEDIUM_VALUE_SIZE, MAX_SMALL_VALUE_SIZE},
9-
db::{CompactConfig, TurboPersistence},
9+
db::{CompactConfig, TurboPersistence, read_current_version},
1010
parallel_scheduler::ParallelScheduler,
1111
write_batch::WriteBatch,
1212
};
@@ -2229,3 +2229,38 @@ fn stale_current_next_is_recovered() -> Result<()> {
22292229

22302230
Ok(())
22312231
}
2232+
2233+
/// `CURRENT` round-trips through JSON, recording both the sequence number and the commit time.
2234+
#[test]
2235+
fn current_file_is_json_with_commit_time() -> Result<()> {
2236+
use crate::parallel_scheduler::SerialScheduler;
2237+
2238+
let tempdir = tempfile::tempdir()?;
2239+
let path = tempdir.path();
2240+
2241+
let before = jiff::Timestamp::now();
2242+
{
2243+
let db = TurboPersistence::<SerialScheduler, 1>::open(path.to_path_buf())?;
2244+
let batch = db.write_batch()?;
2245+
batch.put(0, vec![1u8], vec![42u8].into())?;
2246+
db.commit_write_batch(batch)?;
2247+
db.shutdown()?;
2248+
}
2249+
let after = jiff::Timestamp::now();
2250+
2251+
// next.js parses `CURRENT` without going through this crate, so these field names are a
2252+
// public contract.
2253+
let raw = fs::read_to_string(path.join("CURRENT"))?;
2254+
assert!(raw.contains("max_sequence_number"), "got: {raw}");
2255+
assert!(raw.contains("commit_time"), "got: {raw}");
2256+
2257+
let version = read_current_version(path)?.expect("CURRENT should exist");
2258+
assert!(version.max_sequence_number > 0);
2259+
assert!(
2260+
version.commit_time >= before && version.commit_time <= after,
2261+
"commit_time {} outside [{before}, {after}]",
2262+
version.commit_time
2263+
);
2264+
2265+
Ok(())
2266+
}

‎turbopack/crates/turbo-tasks-backend/Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@ dashmap = { workspace = true, features = ["raw-api"]}
4545
fs-err = { workspace = true }
4646
hashbrown = { workspace = true, features = ["raw"] }
4747
indexmap = { workspace = true }
48+
jiff = "0.2.10"
4849
lzzzz = { workspace = true, optional = true }
4950
parking_lot = { workspace = true }
5051
rand = { workspace = true }

0 commit comments

Comments
 (0)