Repository navigation
atelet: evict the sandbox asset cache to a budget, each worker linking its default runtime - #2756
Conversation
14b7ffb to
0e85c85
Compare
0e85c85 to
1a02a2a
Compare
5254ea1 to
61d16a4
Compare
61d16a4 to
1dd0aef
Compare
A hard link from the cache into a consumer's directory is how a consumer declares an entry in use. Eviction sorted such entries last but could still retire them once nothing else was left. Retiring one frees nothing while the link lasts, and the next caller for the same content downloads a second copy beside the first, so under sustained pressure a pass churned exactly the assets in use. A pass now skips every linked entry, counted as SkippedLinked, and chooses among the rest by age and last use. The listing runs lock-free, so retire re-checks the links under the hit lock as it re-checks the last-use clock, and a link gained since the listing is a veto. Every retired entry frees its whole size, so PendingBytes is gone from the stats. The contract becomes: eviction never takes what a caller holds; its only cost is a re-download by the next caller. This is what lets atelet protect a worker's default runtime by holding a link to it.
1dd0aef to
df71b67
Compare
…g its default runtime The sandbox asset cache only grew: the min-age flag was recorded but no pass read it. atelet now runs an eviction pass every --sandbox-asset-cache-gc-period that brings the cache from over --sandbox-asset-cache-max-bytes down to 90% of it, least recently used first, with --sandbox-asset-cache-gc-dry-run to log the decisions without deleting. A zero budget runs no pass, leaving the cache unbounded, and is warned about at startup. A runtime an actor runs is safe from the pass through the actor's own links. The default runtime of each local worker needs the same protection while no actor runs it, so the first Run on the node stays a cache hit. The prewarmer therefore links it into a directory under the worker's own, default-runtime/<sync>/<hash>, instead of a scratch directory it emptied again. The syncer numbers the <sync> directories in order, counting on from what is on disk so the order survives a restart, makes a fresh one on every sync of the worker and hands the prewarmer the sync's defaults, one <hash> subdirectory each, as one item. The prewarm stages them side by side; one that finds its <sync> directory gone, or a later one beside it, was superseded and does nothing; one that staged every default removes the earlier <sync> directories, and only those. The links go with the worker's directory when the pod does, so nothing counts references, nothing prunes, and a restart changes nothing: the directories of pods still on the node keep protecting, and the startup sweep removes those of pods that are gone. The pass reports ate.sandboxassetcache.size, ate.sandboxassetcache.evictions and ate.sandboxassetcache.gc.passes by outcome, so a node whose cache stays over budget shows whether the collector runs and what stops it: a shortfall means what remains is linked by actors or by workers for their default runtime.
df71b67 to
0c8f1bb
Compare
| syncDir, err := s.newDefaultRuntimeDir(string(pod.UID)) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| s.prewarm.enqueue(syncDir, defaults) | ||
| } | ||
| want, buildErr := s.buildFile(ctx, plan, hashed) |
There was a problem hiding this comment.
If buildFile fails on the new default runtime (for example, gvisorCPUFeatures runs runsc cpu-features and it fails or reports an unknown CPU feature, returning errPermanent), syncPod aborts before s.write(...). The worker's sandbox-runtimes.json stays on the previous default (H1), and finish forgets the item without retrying.
Meanwhile, prewarm already got a new <sync> directory for the rejected default (H2). Because stageSandboxAssets only checks that runsc is a regular file (without running runsc cpu-features), prewarm succeeds for H2 and calls removeEarlierSiblings, deleting the earlier <sync> directory that linked H1. That leaves all local workers of the pool advertising H1 while default-runtime/ only pins H2, so the next GC pass can evict H1.
Could we validate the default runtime's versionAttrs before creating syncDir and enqueuing prewarm (since planVersions puts the default first in hashed, prewarm still gets the head start while restorable versions hashed[1:] are probed), or only advance <sync> after buildFile succeeds?
(Minor: the comment on buildFile at line 265–266 still says the syncer prewarms the default runtime "once the file is written".)
There was a problem hiding this comment.
Right, and thanks for the case: it was a hole, not a corner. Fixed in d577ab9 by taking your second option further. The hand-over now happens after the write and the register, so a build that fails leaves the file, and with it the directories, exactly as they were. With the hand-over after the register the index is authoritative at that point, so the syncer owns the directories outright: it removes the ones for defaults the file no longer names, creates an empty one per default it does, and hands each to the prewarmer, which only fills what exists and creates nothing. That let the numbered sync directories, the sibling scan and the cleanup in the prewarmer go; the layout is default-runtime/<hash> under the worker's directory. The head start is not missed, since a gVisor default's release is fetched by the build's probe anyway and a micro-VM build takes milliseconds. The buildFile comment you mention is true again as written. The description's lifecycle section has the details.
| // change adds and which will read it, so the operator surface does not | ||
| // change when it lands. | ||
| sandboxAssetCacheMinAge = pflag.Duration("sandbox-asset-cache-min-age", time.Hour, "Cached sandbox assets younger than this are never evicted. No eviction pass runs on this cache yet, so the value is recorded for when one does. A running actor's assets are protected by its links regardless; this covers an asset prewarmed for a SandboxConfig that no actor on the node has used yet.") | ||
| sandboxAssetCacheDir = pflag.String("sandbox-asset-cache-dir", ateletpath.SandboxAssetCacheDir, "Directory for the node-local sandbox asset cache (gVisor releases, micro-VM kernels, VMMs and guest images). Must be on the same filesystem mount as the actor state dirs: an actor's assets are hard links from the cache into its own directory.") |
There was a problem hiding this comment.
With workers now linking their default runtime under nodepath.WorkersDir(), --sandbox-asset-cache-dir also needs to be on the same filesystem mount as the worker dirs, not just the actor state dirs.
openSandboxAssetCache below only checks canHardLink(dir, nodepath.ActorsDir) at startup, so if --sandbox-asset-cache-dir shares a mount with ActorsDir but not WorkersDir(), startup succeeds and every worker prewarm fails with EXDEV. Should we update the flag help and also check canHardLink against the workers dir?
There was a problem hiding this comment.
Done in e65b921: openSandboxAssetCache takes the directories links go into and probes each, main passes the actors dir and the workers dir, the flag help names both, and the other-mount test on Linux covers the workers dir as well.
0c8f1bb to
a41e01f
Compare
The syncer handed the default to the prewarmer before the build, in a numbered directory per sync, so the download could start while the probes ran. A build that then failed on the new default left the file on the old one while the prewarm of the new one still ran, staged, and removed the earlier sync's directory, so every worker of the pool advertised a runtime nothing linked and the next pass could evict it. Hand over after the write and the register instead. The syncer then owns the directories outright: once the file names its defaults it removes the directories of defaults it no longer names, creates an empty one per default it does, and hands each to the prewarmer, which fills what exists and creates nothing. A failed build touches no directory. The sync numbering, the sibling scan and the cleanup in the prewarmer go, the item is one hash and its directory again, and the layout is default-runtime/<hash> under the worker's directory. The head start is not missed: a gVisor default's release is fetched by the build's probe, and a micro-VM build takes milliseconds.
Startup probed that a file in the cache can be hard-linked into the actors dir, the only place links went. Each worker's default runtime is now linked under the worker's own dir too, so a cache on a mount shared with the actors dir but not the workers dir passed startup and failed every prewarm with EXDEV. Probe the workers dir as well, and say so in the flag help.
a41e01f to
e65b921
Compare
…t in the metrics suite (agent-substrate#2792) > agent-substrate#2756 has merged; this is one commit on `main`. The page describes agent-substrate#2756's final design: the default runtime is linked under the worker's own directory, in a directory per default the worker's file names, made and removed by the syncer and gone with the pod, so the on-disk table, syncer section, eviction rule, and runbook follow that layout and a subsection gives its lifecycle; and with no budget set atelet starts no eviction pass. Step 6 of agent-substrate#2675, in its small form: the e2e under eviction pressure was dropped as too much machinery for what it proves, and this is what remained worth doing. **Metrics suite.** `ate_sandboxassetcache_size` joins `PlatformMetricPrefixes`, so `TestPlatformMetricsEmitted` proves on every kind lane that atelet registers the instrument, its callback walks a real cache directory without error, and the OTLP path to the collector carries it. Only size qualifies: it is an observable instrument read on every collection, so it has a series from the first export on any install. The eviction and pass counters emit only once a pass had work to do, which never happens with the default unbounded budget. **Doc.** `docs/sandbox-asset-cache.md`, for the operator or engineer who meets the cache for the first time: what lives under `/var/lib/ate` and why it must share a mount; how an actor gets its assets, from the runtime the request names through the index, the cache and the hard links, and what Run, Restore and Checkpoint each do; how the worker sandbox syncer decides which runtimes a node needs, prewarms and pins the defaults, and prunes; the eviction rules and the six flags; the three metrics and what a shortfall means; a runbook for a node over budget; and what is deliberately left out. Linked from the README's doc list, the node supervisor section of the architecture page, and the metrics paragraph of the observability guide. The page describes the request as naming the runtime the worker registered, which is agent-substrate#2788's behavior rather than this stack's. It documents the state once both land rather than the interim. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
…etric family (agent-substrate#2889) Follow-up to agent-substrate#2756, so that agent-substrate#2796 can give the snapshot cache a budget without a third copy of the eviction loop. **What moves.** The sandbox asset cache's eviction loop, flags, instruments, target formula and shortfall backoff move from `cmd/atelet/sandbox_asset_gc.go` into a package, `cmd/atelet/internal/filecachegc`, written against the filecache store. The loop's behavior is unchanged: immediate first pass, serialized passes, panic recovery, dry run, budget with a 90% low water, shortfall warnings that back off to a periodic reminder. **How a cache uses it.** - `filecachegc.NewFlags("sandbox-asset-cache", "sandbox asset cache")` registers `--sandbox-asset-cache-gc-period`, `--sandbox-asset-cache-max-bytes` and `--sandbox-asset-cache-gc-dry-run`, so the asset cache's flags are the same as before. The snapshot cache will call it once with its own prefix. - `filecachegc.NewInstruments(meter)` creates the instruments once per process. - `filecachegc.Start(ctx, name, store, flags, instruments)` reports the cache's size, warns once when there is no budget, and runs the loop when there is a budget and a period. **One metric family.** `ate.filecache.size`, `ate.filecache.evictions` and `ate.filecache.gc.passes`, labeled by `ate.filecache.name` (today `sandbox_asset`) and, for passes, `ate.filecache.gc.outcome`, replace `ate.sandboxassetcache.size`, `.evictions` and `.gc.passes`. The caches run on one store and one loop, so one set of instruments describes them, and "which cache is over budget on this node" is one query grouped by name. The registry, `ateattr` and the observability guide are updated. The name is `filecache` rather than something broader because the family is about the filecache store; the image cache, which measures layers against volume watermarks, is not a member and keeps its loop. **Tests.** The asset GC's tests move into the package unchanged in what they prove, plus one that puts two stores on the same instruments and asserts two size series under their names. **Docs and e2e.** The sandbox asset cache page from agent-substrate#2792 and the e2e metrics suite follow the rename in the last commit: the suite asserts `ate_filecache_size`, and the page's metrics table and runbook use the new names and say where the label comes from. **Follow-up.** agent-substrate#2796 adopts the driver for the snapshot cache and adds the `snapshot` label value. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
…agent-substrate#2796) Replaces agent-substrate#1551, which was four commits from September that no longer applied: the restore source message, the download path and the module path all changed underneath it. Same design, ported onto today's main, after agent-substrate#2756 and agent-substrate#2889, in four commits, plus three from review: the sharing derived from the snapshot's owner, the gVisor worker evicting a shared file's pages before unlinking it, and the manifest read through the cache. Milestone M2 of the node-local artifact cache from agent-substrate#690: an actor restoring from a shared snapshot, the one a tag holds and every actor created from it starts on, gets its files from a node-level cache instead of its own download. **Sharing is the control plane's knowledge.** `ObjectStorage` gains `SnapshotSharing`, the property rather than the snapshot's kind, so new shared snapshot kinds need no atelet change. ate-api-server sets it on Restore from the snapshot's recorded owner: SHARED for a tag's snapshot, borrowed at creation and immutable, PRIVATE for the actor's own, which its next suspend replaces, and for an unrecorded owner; the local-copy fallback object carries it too. Neither side infers it from the URI. **The cache.** A `filecache.Store` under `/var/lib/ate/snapshot-cache`, behind `--snapshot-cache-dir` (empty disables), `--snapshot-cache-min-age` and `--snapshot-cache-fetch-timeout`, opened at startup with crash debris swept before atelet serves. The directory must be under the node state dir: the object-store plugin writes only there, so a cache elsewhere could never be filled, and startup refuses it. Every flag is validated whether or not the cache is enabled, and the help text says which are read only while it is. The image cache's base-path check moved to `nodepath.UnderBasePath`, which both caches use. **Eviction.** The cache runs on the eviction loop every file cache on the node shares, `cmd/atelet/internal/filecachegc` from agent-substrate#2889: `--snapshot-cache-gc-period`, `--snapshot-cache-max-bytes` and `--snapshot-cache-gc-dry-run`, a budget with a 90% low water, least recently used first, never an entry a restore still links. Its size, evictions and passes appear under the existing `ate.filecache.*` instruments as a second value of `ate.filecache.name`, `snapshot`; no new instrument and no new loop. Every flag is validated whether or not the cache is enabled, as the other flags in atelet are, and the help text says which ones are read only while it is; a disabled cache starts no loop and reports no size. **The restore path.** A restore whose durable snapshot is declared SHARED stages each file through the cache; concurrent restores of one snapshot share a single download, later ones read disk. How a hit is served depends on the sandbox class the request's runtime resolved to. ateworker-gvisor gets hard links. Once the restore's page loading is done the worker evicts the memory and filesystem images from the page cache and unlinks them from its restore dir, keeping only the small descriptors linked: a file with another link is treated like a preserved restore dir, since unlinking it would free nothing and would leave its pages charged to the actor's cgroup, the case agent-substrate#1917 fixed. So a link protects an image's entry through the restore and the min age, not for the actor's life; an entry evicted later is still read to the end through the descriptors runsc holds open, and the next restore reads the images from local disk. ateworker-microvm gets private hole-preserving copies, not because a link would corrupt the cache, it replaces `config.json` by rename and its delta merge copies when the inode is shared, but because a link would cost more than it saves: an on-demand restore keeps `memory-ranges` open for the VM's life, which would pin the entry that long and turn every suspend's in-place merge into a full copy, and the worker's page-cache eviction would drop the cached pages for every other restore; agent-substrate#2948 tracks what links would take. EXDEV on a link, from a cache bind-mounted apart from the restore dirs, degrades to a copy, warned once per process. The manifest of a shared snapshot is read through the cache too, so a restore that finds the snapshot on the node makes no request to object storage; a private snapshot's manifest is read fresh, as the commit marker it is. The cache key is the object's address without its query, so a signed URL's credential reaches neither the key, the cache's errors, `meta.json` nor the logs; and a restore that fails after staging removes its restore-state dir, so the links do not hold the cache's entries until the next lifecycle call. The fetch goes through the object-store plugin as the direct path does, one file into a scratch dir on the cache's filesystem; because the cache then writes the restore dir by path rather than through the plugin's `os.Root`, a symlink waiting there fails the restore instead of being followed. **Not here.** Private snapshots stay uncached by design. Links for ateworker-microvm are agent-substrate#2948, a worker change. The open-and-sweep code the two file caches share in shape is still duplicated, since the snapshot cache can be disabled and degrades to copies across mounts where the asset cache refuses to start. **Tests.** ate-api functional: a golden restore declares SHARED, the actor's own snapshot PRIVATE. ate-api unit: the detach-and-resume test from agent-substrate#2852 compares the whole Restore snapshot, so its fixture now puts the snapshot under the actor's own UID, and it expects PRIVATE for the downloaded copy and nothing for a local one. atelet: flag validation, enabled or not, including the eviction flags, the fetch timeout, and a directory outside the node state dir; two shared manifest reads download once and a private one reads fresh; open disabled, creating its root, sweeping debris; the sharing gate and the class modes; through the plugin-backed test atelet, two link restores share one download and one inode and the file is read-only, copy mode gives private inodes and survives a consumer's mutation, a nil cache downloads, a missing file's error reaches the caller with nothing left behind, a plugin error through the cached modes keeps its code and leaks no credential from a signed URI, a restore that fails at the worker after staging leaves no file in restore-state and no link on the cache's entry, four concurrent restores download once, a file in the checkpoint subdirectory lands there, and symlinks in place of the file or the subdirectory fail; on Linux, a restore dir on another mount gets a copy with no second download and one warning for two restores. ateworker-gvisor, on Linux: staged images hard-linked elsewhere are evicted after the restore wait and only then unlinked, with the other links intact. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Step 5 of #2675: the sandbox asset cache gets an eviction pass with a budget, and each worker's default runtime is kept in the cache by links under the worker's own directory.
Commit 1, filecache. A pass never evicts an entry a consumer still hard-links. A link is how a consumer declares an entry in use; retiring such an entry frees nothing while the link lasts and makes the next caller download a second copy beside it. Linked entries are skipped and counted as
SkippedLinked, and every retired entry frees its full size, so the stats need noPendingBytes. The listing runs lock-free, so the retire step re-checks the links under the hit lock as it re-checks the last-use clock, and a link gained since the listing vetoes the entry even when the hit's best-effort touch did not land. The contract is that eviction never takes what a caller holds and its only cost is a re-download.Commit 2, atelet.
Design. The cache has one protection rule, the link rule above, and both things that must survive a pass use it the same way. An actor protects its runtime through the links in its own directory for as long as it lives on the node. A worker protects its default runtime through links in its own directory for as long as the pod exists. There is no root set the pass consults, no reference count across pods, no prune, and nothing to reconcile after a restart: the directory of a pod still on the node keeps protecting, and the startup sweep that already removes the directories of pods that are gone releases their links with them. Several pods of one pool each hold their own links to the same entries; the filesystem counts them, and the entry becomes evictable when the last pod's directory goes.
Directory layout. Under each worker pod's directory,
<base path>/workers/<pod UID>/, which the syncer already owns for the pod's sandbox runtimes file:Lifecycle. The syncer owns the directories and the prewarmer only fills them. Once a pod's file is written and registered, the syncer makes
default-runtime/match what the file names as defaults: the directory of a default the file no longer names is removed, which unlinks that runtime, and an empty directory is created for each default that lacks one. Then it hands the prewarmer one item per default, the hash and its directory. The hand-over comes after the build on purpose: a build that fails, as a permanent runsc probe failure does, leaves the file and so the directories as they were, with the previous default still linked and still advertised. The prewarm starts late by only a little: for a gVisor default the build's probe has already fetched the release, so the prewarm is links and the pause image pull, and for a micro-VM default the build takes milliseconds. The prewarm job checks that its directory exists, stages into it with a variant of the staging call that creates no directories, pulls the pause image beside it, and on an error checks again: a directory gone, after a later sync or the pod's removal, means the item was superseded, and it ends without a retry and without recreating anything. A directory the syncer already filled costs a stat per asset, so a re-sync with an unchanged default downloads nothing and leaves the links alone. A failed download retries with the existing backoff and gives up after eight attempts; the directory stays, empty or partial, until the next sync, and a Run placed on that default meanwhile fetches on demand and shares the download. One queue and one worker drain the items in order, so defaults download one at a time across the node and never compete for its bandwidth.Why the old default is not kept during the download. By the time the directories change, the file names the new default and the worker has registered it, so the scheduler places nothing new on the old one. Actors already running on it hold their own links, and restores of snapshots taken on it go through the on-demand path like every non-default version. Keeping its links would only hold cache space.
GC loop.
sandbox_asset_gc.goadds--sandbox-asset-cache-gc-period(5m, 0 disables),--sandbox-asset-cache-max-bytes(0 means unbounded: no pass runs, warned at startup) and--sandbox-asset-cache-gc-dry-run. A pass that finds the cache over budget asks the store to free down to 90% of it, oldest first. Same shape as the image GC: immediate first pass, serialized passes, panic recovery, shortfall warnings that back off to a periodic reminder.Metrics.
ate.sandboxassetcache.size(observable up/down counter, bytes),ate.sandboxassetcache.evictionsandate.sandboxassetcache.gc.passesbyate.sandboxassetcache.gc.outcome(complete,shortfall,skipped); a pass within budget records nothing, and a dry run records its pass but no evictions. Registry group and entries added,hack/verify/metrics.shpasses, rows and a paragraph indocs/observability.md.What a shortfall means. Everything left is linked by an actor, linked by a worker as its default, younger than the min age, or busy. The size then measures what the node's actors and defaults need, not waste; the budget is too small or the node hosts too many runtimes.
Not in this PR. A shared GC driver for the image and asset caches, which would also give the image GC the metrics it lacks; filed as a follow-up once this lands. Step 6, the summary doc, which #2792 carries and which will be rewritten for this layout.
Tests. filecache: a linked file, a partly linked tree and a fully linked tree all survive a pass; unlinked trees still go oldest first; an entry linked between the listing and the retire, without a touch, is vetoed at retire time. atelet: a prewarm links the runtime into its directory and an eviction pass keeps the entry; a second prewarm of the same directory fetches nothing; removing the pod directory makes the entry evictable; a job whose directory is gone before it runs, or whose pod goes while it stages, fetches nothing, recreates nothing, and is not retried; the sync hands the default over only after the build, removes a retired default's directory, keeps the current one and its contents across a re-sync, hands nothing over and touches no directory when the build fails, and the pod's removal takes the directories; the end-to-end informer test asserts the link under the worker's directory; retry, pause-image and attempt-timeout cases as before. GC: target table, flag validation, quiet, skipped, complete and shortfall passes with the metrics checked through a manual reader, the size callback, immediate first pass, ticking, panic recovery.
🤖 Generated with Claude Code