Repository navigation
Remove unnecessary fsync during ResumeActor - #1883
Merged
Benjamin Elder (BenTheElder) merged 2 commits intoSep 26, 2026
Merged
Conversation
Stage local checkpoint files into the restore staging directory via os.Link (with EXDEV fallback to sparsefile.CopyFile) instead of copying the full working set, and guard MergeDeltaIntoBase (Nlink > 1) and rewriteSnapshotSocketPaths (temp file + rename) against mutating shared inodes.
Remove f.Sync() and dir.Sync() from writeFileAtomic and writeFileAtomicRoot while keeping CreateTemp + Rename for read atomicity, avoiding ext4 data=ordered writeback stalls on the ResumeActor critical path.
Collaborator
|
commit 2 looks good to me. Thanks Michelle. |
Benjamin Elder (BenTheElder)
approved these changes
Sep 25, 2026
Benjamin Elder (BenTheElder)
removed this pull request from the merge queue due to a manual request
Sep 25, 2026
Collaborator
|
BTW for the first commit ... For micro-VM restores from a local snapshot, the copy doesn't go away, it moves. It used to happen on resume; now the next suspend pays a full sparse copy of the working set, because the merge guard sees the shared inode. I'm not actually sure if that's the right tradeoff. let me look at the other PR. The second commit is an obvious win, if you want to just ship that one for now. |
Collaborator
|
eh, let's merge and iterate, this is a huge performance win. |
Bowei Du (bowei)
added
area/microVM
area/node
area/storage
kind/cleanup
2 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
While benchmarking pause actor I ran into multiple bottlenecks.
I'm currently importing #935 into my branch but #1876 is another alternative to solve the first bottleneck.
The 2nd bottleneck (primary target of this PR) is due to fsyncs being called in writeFileAtomic() for sandbox-assets.json and SystemInfoVolume files. Because these are new files that require ext4 filesystem metadata updates, and we have ext4 mounted in ordered mode, calling fsync on the file + directory causes all other pending writes (such as other checkpoints) to be flushed. So the file creation ends up being blocked behind other actor's checkpoint syncs.
These files do not actually need to be flushed to disk. SystemInfo is recreatable. sandbox-assets.json is written during Run/Resume(), and read during Checkpoint(). But once the checkpoint is finished, the manifest is saved with the snapshot. If the node crashes between Run/Resume and Checkpoint, the actor is crashed anyway.
Single-Node Pause/Resume Benchmark Summary
Configuration: 17 workers, 15 actors,
glutton(512 MiBRAM capacity,32 MiBchurn per cycle),120srun,1.0swait,100%pause (0%suspend)upstream/main)os.Linklocal checkpoint)fsyncon ephemeral volumes)120s)106.5(112pauses /101resumes)160.0(165pauses /155resumes)212.5(214pauses /211resumes)53.3 cycles/min(0.89/s)80.0 cycles/min(1.33/s, +50.2%)106.3 cycles/min(1.77/s, 2.00x)ResumeActorp506,000 ms4,100 ms210 ms(28.6x faster)ResumeActoravg7,023 ms4,867 ms246 ms(28.5x faster)ResumeActorp90 / p9915,000 ms/19,000 ms9,400 ms/20,000 ms350 ms/590 ms