Skip to content

atelet: evict the sandbox asset cache to a budget, each worker linking its default runtime - #2756

Merged
Michelle Au (msau42) merged 4 commits into
agent-substrate:mainfrom
dberkov:asset-cache-gc
Oct 19, 2026
Merged

Michelle Au (msau42) merged 4 commits into
agent-substrate:mainfrom
dberkov:asset-cache-gc

Conversation

@dberkov

@dberkov Dmitry Berkovich (dberkov) commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Step 5 of #2675: the sandbox asset cache gets an eviction pass with a budget, and each worker's default runtime is kept in the cache by links under the worker's own directory.

Commit 1, filecache. A pass never evicts an entry a consumer still hard-links. A link is how a consumer declares an entry in use; retiring such an entry frees nothing while the link lasts and makes the next caller download a second copy beside it. Linked entries are skipped and counted as SkippedLinked, and every retired entry frees its full size, so the stats need no PendingBytes. The listing runs lock-free, so the retire step re-checks the links under the hit lock as it re-checks the last-use clock, and a link gained since the listing vetoes the entry even when the hit's best-effort touch did not land. The contract is that eviction never takes what a caller holds and its only cost is a re-download.

Commit 2, atelet.

Design. The cache has one protection rule, the link rule above, and both things that must survive a pass use it the same way. An actor protects its runtime through the links in its own directory for as long as it lives on the node. A worker protects its default runtime through links in its own directory for as long as the pod exists. There is no root set the pass consults, no reference count across pods, no prune, and nothing to reconcile after a restart: the directory of a pod still on the node keeps protecting, and the startup sweep that already removes the directories of pods that are gone releases their links with them. Several pods of one pool each hold their own links to the same entries; the filesystem counts them, and the entry becomes evictable when the last pod's directory goes.

Directory layout. Under each worker pod's directory, <base path>/workers/<pod UID>/, which the syncer already owns for the pod's sandbox runtimes file:

workers/<pod UID>/
  sandbox-runtimes.json
  default-runtime/
    <hash>/              one per default runtime the file advertises, by bundle hash
      gvisor/runsc …     hard links into the cache, made by the staging call Run uses

Lifecycle. The syncer owns the directories and the prewarmer only fills them. Once a pod's file is written and registered, the syncer makes default-runtime/ match what the file names as defaults: the directory of a default the file no longer names is removed, which unlinks that runtime, and an empty directory is created for each default that lacks one. Then it hands the prewarmer one item per default, the hash and its directory. The hand-over comes after the build on purpose: a build that fails, as a permanent runsc probe failure does, leaves the file and so the directories as they were, with the previous default still linked and still advertised. The prewarm starts late by only a little: for a gVisor default the build's probe has already fetched the release, so the prewarm is links and the pause image pull, and for a micro-VM default the build takes milliseconds. The prewarm job checks that its directory exists, stages into it with a variant of the staging call that creates no directories, pulls the pause image beside it, and on an error checks again: a directory gone, after a later sync or the pod's removal, means the item was superseded, and it ends without a retry and without recreating anything. A directory the syncer already filled costs a stat per asset, so a re-sync with an unchanged default downloads nothing and leaves the links alone. A failed download retries with the existing backoff and gives up after eight attempts; the directory stays, empty or partial, until the next sync, and a Run placed on that default meanwhile fetches on demand and shares the download. One queue and one worker drain the items in order, so defaults download one at a time across the node and never compete for its bandwidth.

Why the old default is not kept during the download. By the time the directories change, the file names the new default and the worker has registered it, so the scheduler places nothing new on the old one. Actors already running on it hold their own links, and restores of snapshots taken on it go through the on-demand path like every non-default version. Keeping its links would only hold cache space.

GC loop. sandbox_asset_gc.go adds --sandbox-asset-cache-gc-period (5m, 0 disables), --sandbox-asset-cache-max-bytes (0 means unbounded: no pass runs, warned at startup) and --sandbox-asset-cache-gc-dry-run. A pass that finds the cache over budget asks the store to free down to 90% of it, oldest first. Same shape as the image GC: immediate first pass, serialized passes, panic recovery, shortfall warnings that back off to a periodic reminder.

Metrics. ate.sandboxassetcache.size (observable up/down counter, bytes), ate.sandboxassetcache.evictions and ate.sandboxassetcache.gc.passes by ate.sandboxassetcache.gc.outcome (complete, shortfall, skipped); a pass within budget records nothing, and a dry run records its pass but no evictions. Registry group and entries added, hack/verify/metrics.sh passes, rows and a paragraph in docs/observability.md.

What a shortfall means. Everything left is linked by an actor, linked by a worker as its default, younger than the min age, or busy. The size then measures what the node's actors and defaults need, not waste; the budget is too small or the node hosts too many runtimes.

Not in this PR. A shared GC driver for the image and asset caches, which would also give the image GC the metrics it lacks; filed as a follow-up once this lands. Step 6, the summary doc, which #2792 carries and which will be rewritten for this layout.

Tests. filecache: a linked file, a partly linked tree and a fully linked tree all survive a pass; unlinked trees still go oldest first; an entry linked between the listing and the retire, without a touch, is vetoed at retire time. atelet: a prewarm links the runtime into its directory and an eviction pass keeps the entry; a second prewarm of the same directory fetches nothing; removing the pod directory makes the entry evictable; a job whose directory is gone before it runs, or whose pod goes while it stages, fetches nothing, recreates nothing, and is not retried; the sync hands the default over only after the build, removes a retired default's directory, keeps the current one and its contents across a re-sync, hands nothing over and touches no directory when the build fails, and the pod's removal takes the directories; the end-to-end informer test asserts the link under the worker's directory; retry, pause-image and attempt-timeout cases as before. GC: target table, flag validation, quiet, skipped, complete and shortfall passes with the metrics checked through a manual reader, the size callback, immediate first pass, ticking, panic recovery.

🤖 Generated with Claude Code

A hard link from the cache into a consumer's directory is how a consumer
declares an entry in use. Eviction sorted such entries last but could
still retire them once nothing else was left. Retiring one frees nothing
while the link lasts, and the next caller for the same content downloads
a second copy beside the first, so under sustained pressure a pass
churned exactly the assets in use.

A pass now skips every linked entry, counted as SkippedLinked, and
chooses among the rest by age and last use. The listing runs lock-free,
so retire re-checks the links under the hit lock as it re-checks the
last-use clock, and a link gained since the listing is a veto. Every
retired entry frees its whole size, so PendingBytes is gone from the
stats. The contract
becomes: eviction never takes what a caller holds; its only cost is a
re-download by the next caller. This is what lets atelet protect a
worker's default runtime by holding a link to it.
…g its default runtime

The sandbox asset cache only grew: the min-age flag was recorded but no
pass read it. atelet now runs an eviction pass every
--sandbox-asset-cache-gc-period that brings the cache from over
--sandbox-asset-cache-max-bytes down to 90% of it, least recently used
first, with --sandbox-asset-cache-gc-dry-run to log the decisions
without deleting. A zero budget runs no pass, leaving the cache
unbounded, and is warned about at startup.

A runtime an actor runs is safe from the pass through the actor's own
links. The default runtime of each local worker needs the same
protection while no actor runs it, so the first Run on the node stays a
cache hit. The prewarmer therefore links it into a directory under the
worker's own, default-runtime/<sync>/<hash>, instead of a scratch
directory it emptied again. The syncer numbers the <sync> directories
in order, counting on from what is on disk so the order survives a
restart, makes a fresh one on every sync of the worker and hands the
prewarmer the sync's defaults, one <hash> subdirectory each, as one
item. The prewarm stages them side by side; one that finds its <sync>
directory gone, or a later one beside it, was superseded and does
nothing; one that staged every default removes the earlier <sync>
directories, and only those. The links go with the worker's directory
when the pod does, so nothing counts references, nothing prunes, and a
restart changes nothing: the directories of pods still on the node keep
protecting, and the startup sweep removes those of pods that are gone.

The pass reports ate.sandboxassetcache.size,
ate.sandboxassetcache.evictions and ate.sandboxassetcache.gc.passes by
outcome, so a node whose cache stays over budget shows whether the
collector runs and what stops it: a shortfall means what remains is
linked by actors or by workers for their default runtime.
@dberkov
Dmitry Berkovich (dberkov) marked this pull request as ready for review October 18, 2026 21:36
@dberkov Dmitry Berkovich (dberkov) changed the title atelet: evict the sandbox asset cache to a budget, pinning each worker's default runtime atelet: evict the sandbox asset cache to a budget, each worker linking its default runtime Oct 18, 2026
Comment thread cmd/atelet/worker_sandbox_versions.go Outdated
Comment on lines 511 to 517
syncDir, err := s.newDefaultRuntimeDir(string(pod.UID))
if err != nil {
return err
}
s.prewarm.enqueue(syncDir, defaults)
}
want, buildErr := s.buildFile(ctx, plan, hashed)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If buildFile fails on the new default runtime (for example, gvisorCPUFeatures runs runsc cpu-features and it fails or reports an unknown CPU feature, returning errPermanent), syncPod aborts before s.write(...). The worker's sandbox-runtimes.json stays on the previous default (H1), and finish forgets the item without retrying.

Meanwhile, prewarm already got a new <sync> directory for the rejected default (H2). Because stageSandboxAssets only checks that runsc is a regular file (without running runsc cpu-features), prewarm succeeds for H2 and calls removeEarlierSiblings, deleting the earlier <sync> directory that linked H1. That leaves all local workers of the pool advertising H1 while default-runtime/ only pins H2, so the next GC pass can evict H1.

Could we validate the default runtime's versionAttrs before creating syncDir and enqueuing prewarm (since planVersions puts the default first in hashed, prewarm still gets the head start while restorable versions hashed[1:] are probed), or only advance <sync> after buildFile succeeds?

(Minor: the comment on buildFile at line 265–266 still says the syncer prewarms the default runtime "once the file is written".)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, and thanks for the case: it was a hole, not a corner. Fixed in d577ab9 by taking your second option further. The hand-over now happens after the write and the register, so a build that fails leaves the file, and with it the directories, exactly as they were. With the hand-over after the register the index is authoritative at that point, so the syncer owns the directories outright: it removes the ones for defaults the file no longer names, creates an empty one per default it does, and hands each to the prewarmer, which only fills what exists and creates nothing. That let the numbered sync directories, the sibling scan and the cleanup in the prewarmer go; the layout is default-runtime/<hash> under the worker's directory. The head start is not missed, since a gVisor default's release is fetched by the build's probe anyway and a micro-VM build takes milliseconds. The buildFile comment you mention is true again as written. The description's lifecycle section has the details.

Comment thread cmd/atelet/sandbox_asset_cache.go Outdated
// change adds and which will read it, so the operator surface does not
// change when it lands.
sandboxAssetCacheMinAge = pflag.Duration("sandbox-asset-cache-min-age", time.Hour, "Cached sandbox assets younger than this are never evicted. No eviction pass runs on this cache yet, so the value is recorded for when one does. A running actor's assets are protected by its links regardless; this covers an asset prewarmed for a SandboxConfig that no actor on the node has used yet.")
sandboxAssetCacheDir = pflag.String("sandbox-asset-cache-dir", ateletpath.SandboxAssetCacheDir, "Directory for the node-local sandbox asset cache (gVisor releases, micro-VM kernels, VMMs and guest images). Must be on the same filesystem mount as the actor state dirs: an actor's assets are hard links from the cache into its own directory.")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With workers now linking their default runtime under nodepath.WorkersDir(), --sandbox-asset-cache-dir also needs to be on the same filesystem mount as the worker dirs, not just the actor state dirs.

openSandboxAssetCache below only checks canHardLink(dir, nodepath.ActorsDir) at startup, so if --sandbox-asset-cache-dir shares a mount with ActorsDir but not WorkersDir(), startup succeeds and every worker prewarm fails with EXDEV. Should we update the flag help and also check canHardLink against the workers dir?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in e65b921: openSandboxAssetCache takes the directories links go into and probes each, main passes the actors dir and the workers dir, the flag help names both, and the other-mount test on Linux covers the workers dir as well.

The syncer handed the default to the prewarmer before the build, in a
numbered directory per sync, so the download could start while the
probes ran. A build that then failed on the new default left the file
on the old one while the prewarm of the new one still ran, staged, and
removed the earlier sync's directory, so every worker of the pool
advertised a runtime nothing linked and the next pass could evict it.

Hand over after the write and the register instead. The syncer then
owns the directories outright: once the file names its defaults it
removes the directories of defaults it no longer names, creates an
empty one per default it does, and hands each to the prewarmer, which
fills what exists and creates nothing. A failed build touches no
directory. The sync numbering, the sibling scan and the cleanup in the
prewarmer go, the item is one hash and its directory again, and the
layout is default-runtime/<hash> under the worker's directory. The head
start is not missed: a gVisor default's release is fetched by the
build's probe, and a micro-VM build takes milliseconds.
Startup probed that a file in the cache can be hard-linked into the
actors dir, the only place links went. Each worker's default runtime is
now linked under the worker's own dir too, so a cache on a mount shared
with the actors dir but not the workers dir passed startup and failed
every prewarm with EXDEV. Probe the workers dir as well, and say so in
the flag help.
@msau42
Michelle Au (msau42) added this pull request to the merge queue Oct 19, 2026
Merged via the queue into agent-substrate:main with commit c6c890a Oct 19, 2026
15 checks passed
Shrutiya Mohan (shrutiyam-glitch) pushed a commit to shrutiyam-glitch/substrate that referenced this pull request Oct 19, 2026
…t in the metrics suite (agent-substrate#2792)

> agent-substrate#2756 has merged; this is one commit on `main`.

The page describes agent-substrate#2756's final design: the default runtime is linked
under the worker's own directory, in a directory per default the
worker's file names, made and removed by the syncer and gone with the
pod, so the on-disk table, syncer section, eviction rule, and runbook
follow that layout and a subsection gives its lifecycle; and with no
budget set atelet starts no eviction pass.

Step 6 of agent-substrate#2675, in its small form: the e2e under eviction pressure was
dropped as too much machinery for what it proves, and this is what
remained worth doing.

**Metrics suite.** `ate_sandboxassetcache_size` joins
`PlatformMetricPrefixes`, so `TestPlatformMetricsEmitted` proves on
every kind lane that atelet registers the instrument, its callback walks
a real cache directory without error, and the OTLP path to the collector
carries it. Only size qualifies: it is an observable instrument read on
every collection, so it has a series from the first export on any
install. The eviction and pass counters emit only once a pass had work
to do, which never happens with the default unbounded budget.

**Doc.** `docs/sandbox-asset-cache.md`, for the operator or engineer who
meets the cache for the first time: what lives under `/var/lib/ate` and
why it must share a mount; how an actor gets its assets, from the
runtime the request names through the index, the cache and the hard
links, and what Run, Restore and Checkpoint each do; how the worker
sandbox syncer decides which runtimes a node needs, prewarms and pins
the defaults, and prunes; the eviction rules and the six flags; the
three metrics and what a shortfall means; a runbook for a node over
budget; and what is deliberately left out. Linked from the README's doc
list, the node supervisor section of the architecture page, and the
metrics paragraph of the observability guide.

The page describes the request as naming the runtime the worker
registered, which is agent-substrate#2788's behavior rather than this stack's. It
documents the state once both land rather than the interim.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Max Thompson (thompsonmax) pushed a commit to thompsonmax/substrate that referenced this pull request Oct 19, 2026
…etric family (agent-substrate#2889)

Follow-up to agent-substrate#2756, so that agent-substrate#2796 can give the snapshot cache a budget
without a third copy of the eviction loop.

**What moves.** The sandbox asset cache's eviction loop, flags,
instruments, target formula and shortfall backoff move from
`cmd/atelet/sandbox_asset_gc.go` into a package,
`cmd/atelet/internal/filecachegc`, written against the filecache store.
The loop's behavior is unchanged: immediate first pass, serialized
passes, panic recovery, dry run, budget with a 90% low water, shortfall
warnings that back off to a periodic reminder.

**How a cache uses it.**
- `filecachegc.NewFlags("sandbox-asset-cache", "sandbox asset cache")`
registers `--sandbox-asset-cache-gc-period`,
`--sandbox-asset-cache-max-bytes` and
`--sandbox-asset-cache-gc-dry-run`, so the asset cache's flags are the
same as before. The snapshot cache will call it once with its own
prefix.
- `filecachegc.NewInstruments(meter)` creates the instruments once per
process.
- `filecachegc.Start(ctx, name, store, flags, instruments)` reports the
cache's size, warns once when there is no budget, and runs the loop when
there is a budget and a period.

**One metric family.** `ate.filecache.size`, `ate.filecache.evictions`
and `ate.filecache.gc.passes`, labeled by `ate.filecache.name` (today
`sandbox_asset`) and, for passes, `ate.filecache.gc.outcome`, replace
`ate.sandboxassetcache.size`, `.evictions` and `.gc.passes`. The caches
run on one store and one loop, so one set of instruments describes them,
and "which cache is over budget on this node" is one query grouped by
name. The registry, `ateattr` and the observability guide are updated.
The name is `filecache` rather than something broader because the family
is about the filecache store; the image cache, which measures layers
against volume watermarks, is not a member and keeps its loop.

**Tests.** The asset GC's tests move into the package unchanged in what
they prove, plus one that puts two stores on the same instruments and
asserts two size series under their names.

**Docs and e2e.** The sandbox asset cache page from agent-substrate#2792 and the e2e
metrics suite follow the rename in the last commit: the suite asserts
`ate_filecache_size`, and the page's metrics table and runbook use the
new names and say where the label comes from.

**Follow-up.** agent-substrate#2796 adopts the driver for the snapshot cache and adds
the `snapshot` label value.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Haowei Cai (Roy) (roycaihw) pushed a commit to roycaihw/substrate that referenced this pull request Oct 20, 2026
…agent-substrate#2796)

Replaces agent-substrate#1551, which was four commits from September that no longer
applied: the restore source message, the download path and the module
path all changed underneath it. Same design, ported onto today's main,
after agent-substrate#2756 and agent-substrate#2889, in four commits, plus three from review: the
sharing derived from the snapshot's owner, the gVisor worker evicting a
shared file's pages before unlinking it, and the manifest read through
the cache.

Milestone M2 of the node-local artifact cache from agent-substrate#690: an actor
restoring from a shared snapshot, the one a tag holds and every actor
created from it starts on, gets its files from a node-level cache
instead of its own download.

**Sharing is the control plane's knowledge.** `ObjectStorage` gains
`SnapshotSharing`, the property rather than the snapshot's kind, so new
shared snapshot kinds need no atelet change. ate-api-server sets it on
Restore from the snapshot's recorded owner: SHARED for a tag's snapshot,
borrowed at creation and immutable, PRIVATE for the actor's own, which
its next suspend replaces, and for an unrecorded owner; the local-copy
fallback object carries it too. Neither side infers it from the URI.

**The cache.** A `filecache.Store` under `/var/lib/ate/snapshot-cache`,
behind `--snapshot-cache-dir` (empty disables),
`--snapshot-cache-min-age` and `--snapshot-cache-fetch-timeout`, opened
at startup with crash debris swept before atelet serves. The directory
must be under the node state dir: the object-store plugin writes only
there, so a cache elsewhere could never be filled, and startup refuses
it. Every flag is validated whether or not the cache is enabled, and the
help text says which are read only while it is. The image cache's
base-path check moved to `nodepath.UnderBasePath`, which both caches
use.

**Eviction.** The cache runs on the eviction loop every file cache on
the node shares, `cmd/atelet/internal/filecachegc` from agent-substrate#2889:
`--snapshot-cache-gc-period`, `--snapshot-cache-max-bytes` and
`--snapshot-cache-gc-dry-run`, a budget with a 90% low water, least
recently used first, never an entry a restore still links. Its size,
evictions and passes appear under the existing `ate.filecache.*`
instruments as a second value of `ate.filecache.name`, `snapshot`; no
new instrument and no new loop. Every flag is validated whether or not
the cache is enabled, as the other flags in atelet are, and the help
text says which ones are read only while it is; a disabled cache starts
no loop and reports no size.

**The restore path.** A restore whose durable snapshot is declared
SHARED stages each file through the cache; concurrent restores of one
snapshot share a single download, later ones read disk. How a hit is
served depends on the sandbox class the request's runtime resolved to.
ateworker-gvisor gets hard links. Once the restore's page loading is
done the worker evicts the memory and filesystem images from the page
cache and unlinks them from its restore dir, keeping only the small
descriptors linked: a file with another link is treated like a preserved
restore dir, since unlinking it would free nothing and would leave its
pages charged to the actor's cgroup, the case agent-substrate#1917 fixed. So a link
protects an image's entry through the restore and the min age, not for
the actor's life; an entry evicted later is still read to the end
through the descriptors runsc holds open, and the next restore reads the
images from local disk. ateworker-microvm gets private hole-preserving
copies, not because a link would corrupt the cache, it replaces
`config.json` by rename and its delta merge copies when the inode is
shared, but because a link would cost more than it saves: an on-demand
restore keeps `memory-ranges` open for the VM's life, which would pin
the entry that long and turn every suspend's in-place merge into a full
copy, and the worker's page-cache eviction would drop the cached pages
for every other restore; agent-substrate#2948 tracks what links would take. EXDEV on a
link, from a cache bind-mounted apart from the restore dirs, degrades to
a copy, warned once per process. The manifest of a shared snapshot is
read through the cache too, so a restore that finds the snapshot on the
node makes no request to object storage; a private snapshot's manifest
is read fresh, as the commit marker it is. The cache key is the object's
address without its query, so a signed URL's credential reaches neither
the key, the cache's errors, `meta.json` nor the logs; and a restore
that fails after staging removes its restore-state dir, so the links do
not hold the cache's entries until the next lifecycle call. The fetch
goes through the object-store plugin as the direct path does, one file
into a scratch dir on the cache's filesystem; because the cache then
writes the restore dir by path rather than through the plugin's
`os.Root`, a symlink waiting there fails the restore instead of being
followed.

**Not here.** Private snapshots stay uncached by design. Links for
ateworker-microvm are agent-substrate#2948, a worker change. The open-and-sweep code
the two file caches share in shape is still duplicated, since the
snapshot cache can be disabled and degrades to copies across mounts
where the asset cache refuses to start.

**Tests.** ate-api functional: a golden restore declares SHARED, the
actor's own snapshot PRIVATE. ate-api unit: the detach-and-resume test
from agent-substrate#2852 compares the whole Restore snapshot, so its fixture now puts
the snapshot under the actor's own UID, and it expects PRIVATE for the
downloaded copy and nothing for a local one. atelet: flag validation,
enabled or not, including the eviction flags, the fetch timeout, and a
directory outside the node state dir; two shared manifest reads download
once and a private one reads fresh; open disabled, creating its root,
sweeping debris; the sharing gate and the class modes; through the
plugin-backed test atelet, two link restores share one download and one
inode and the file is read-only, copy mode gives private inodes and
survives a consumer's mutation, a nil cache downloads, a missing file's
error reaches the caller with nothing left behind, a plugin error
through the cached modes keeps its code and leaks no credential from a
signed URI, a restore that fails at the worker after staging leaves no
file in restore-state and no link on the cache's entry, four concurrent
restores download once, a file in the checkpoint subdirectory lands
there, and symlinks in place of the file or the subdirectory fail; on
Linux, a restore dir on another mount gets a copy with no second
download and one warning for two restores. ateworker-gvisor, on Linux:
staged images hard-linked elsewhere are evicted after the restore wait
and only then unlinked, with the other links intact.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants