Repository navigation
atelet: verify microvm SandboxConfig assets before writing sandbox-runtimes.json - #3143
Merged
Dmitry Berkovich (dberkov) merged 3 commits intoOct 24, 2026
Conversation
Shrutiya Mohan (shrutiyam-glitch)
marked this pull request as ready for review
October 23, 2026 00:44
Shrutiya Mohan (shrutiyam-glitch)
changed the title
…ntimes.json On microvm, workerSandboxSyncer.versionAttrs only probed /proc/cpuinfo and wrote sandbox-runtimes.json without checking whether a version's assets could be fetched and verified. A broken defaultVersion URL or SHA256 overwrote sandbox-runtimes.json right away, causing workers to advertise a defaultRuntime that failed at Run or Restore. Stage each microvm version's assets through ensureSandboxAssets into a scratch directory during versionAttrs, matching gvisor. Cached assets only link and unlink in scratch. A failed defaultVersion leaves the existing file in place, and a failed restorable version is omitted with errIncomplete while finish retries with backoff.
Remember verified microVM bundle hashes in workerSandboxSyncer.microvmAssets, matching cpuFeatures for gVisor. Without memoization, every sync re-stages each version into scratchDir, and once sandboxAssetCache evicts an unlinked restorable version a new worker pod has to download it again and a transient fetch failure on re-sync drops it from sandbox-runtimes.json. Update docs/sandbox-asset-cache.md and the prewarm and scratch-dir comments to match buildFile fetching each version's assets on first check.
Shrutiya Mohan (shrutiyam-glitch)
force-pushed
the
atelet-bug
branch
from
October 23, 2026 22:12
4f4d842 to
455d81e
Compare
Shrutiya Mohan (shrutiyam-glitch)
requested a review
from Michelle Au (msau42)
October 23, 2026 22:13
Memoizing every verified microVM bundle hash skips ensureMicroVMAssets when an evicted restorable version is promoted to defaultVersion. If the asset bucket is failing, syncPod writes the new default anyway and removes the previous default's directory. Only skip ensureMicroVMAssets on the in-memory memo for restorable versions, and stage the default version on every sync (a cache hit once prewarmed). Raise probeFetchTimeout to 10m to cover all assets of a cold microVM version under --sandbox-asset-fetch-timeout, and document the per-process memo behavior across restarts and eviction in docs/sandbox-asset-cache.md.
Shrutiya Mohan (shrutiyam-glitch)
requested review from
Dmitry Berkovich (dberkov) and
Michelle Au (msau42)
October 24, 2026 00:49
Dmitry Berkovich (dberkov)
approved these changes
Oct 24, 2026
2 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3139
On microVM,
workerSandboxSyncer.versionAttrsonly probed/proc/cpuinfoand wrotesandbox-runtimes.jsonwithout checking whether a version's assets could be fetched and verified. ASandboxConfigrollout with a broken asset URL or SHA256 ondefaultVersionoverwrotesandbox-runtimes.jsonright away, and workers advertised adefaultRuntimethat failed atRunorRestore.Stage each microVM version's assets through
ensureSandboxAssetsinto a scratch directory inbuildFile(withprobeFetchTimeoutraised to 10m) and record verified bundle hashes inworkerSandboxSyncer.microvmAssets. Always stage the default version (a cache hit once prewarmed) so promoting an evicted version todefaultVersionverifies it before unpinning the previous default. SkipensureMicroVMAssetsfor restorable versions already inmicrovmAssets; the memo is perateletprocess, so a restart re-checks each version (re-downloading any restorable version evicted before the restart).A failed
defaultVersionfetch or SHA256 check failsbuildFilebefores.writeruns, while a failed restorable version is left out witherrIncompleteand retried. As with gVisor, a SHA256 mismatch is not cached or memoized, so retries re-download until theSandboxConfigis fixed. Updatedocs/sandbox-asset-cache.mdand the prewarmer comments to match.AI-assisted, I reviewed and tested it.