docs: describe the sandbox asset cache, and assert its size instrument in the metrics suite - #2792

…g its default runtime (agent-substrate#2756)

Step 5 of agent-substrate#2675: the sandbox asset cache gets an eviction pass with a
budget, and each worker's default runtime is kept in the cache by links
under the worker's own directory.

**Commit 1, filecache.** A pass never evicts an entry a consumer still
hard-links. A link is how a consumer declares an entry in use; retiring
such an entry frees nothing while the link lasts and makes the next
caller download a second copy beside it. Linked entries are skipped and
counted as `SkippedLinked`, and every retired entry frees its full size,
so the stats need no `PendingBytes`. The listing runs lock-free, so the
retire step re-checks the links under the hit lock as it re-checks the
last-use clock, and a link gained since the listing vetoes the entry
even when the hit's best-effort touch did not land. The contract is that
eviction never takes what a caller holds and its only cost is a
re-download.

**Commit 2, atelet.**

*Design.* The cache has one protection rule, the link rule above, and
both things that must survive a pass use it the same way. An actor
protects its runtime through the links in its own directory for as long
as it lives on the node. A worker protects its default runtime through
links in its own directory for as long as the pod exists. There is no
root set the pass consults, no reference count across pods, no prune,
and nothing to reconcile after a restart: the directory of a pod still
on the node keeps protecting, and the startup sweep that already removes
the directories of pods that are gone releases their links with them.
Several pods of one pool each hold their own links to the same entries;
the filesystem counts them, and the entry becomes evictable when the
last pod's directory goes.

*Directory layout.* Under each worker pod's directory, `<base
path>/workers/<pod UID>/`, which the syncer already owns for the pod's
sandbox runtimes file:

```
workers/<pod UID>/
  sandbox-runtimes.json
  default-runtime/
    <hash>/              one per default runtime the file advertises, by bundle hash
      gvisor/runsc …     hard links into the cache, made by the staging call Run uses
```

*Lifecycle.* The syncer owns the directories and the prewarmer only
fills them. Once a pod's file is written and registered, the syncer
makes `default-runtime/` match what the file names as defaults: the
directory of a default the file no longer names is removed, which
unlinks that runtime, and an empty directory is created for each default
that lacks one. Then it hands the prewarmer one item per default, the
hash and its directory. The hand-over comes after the build on purpose:
a build that fails, as a permanent runsc probe failure does, leaves the
file and so the directories as they were, with the previous default
still linked and still advertised. The prewarm starts late by only a
little: for a gVisor default the build's probe has already fetched the
release, so the prewarm is links and the pause image pull, and for a
micro-VM default the build takes milliseconds. The prewarm job checks
that its directory exists, stages into it with a variant of the staging
call that creates no directories, pulls the pause image beside it, and
on an error checks again: a directory gone, after a later sync or the
pod's removal, means the item was superseded, and it ends without a
retry and without recreating anything. A directory the syncer already
filled costs a stat per asset, so a re-sync with an unchanged default
downloads nothing and leaves the links alone. A failed download retries
with the existing backoff and gives up after eight attempts; the
directory stays, empty or partial, until the next sync, and a Run placed
on that default meanwhile fetches on demand and shares the download. One
queue and one worker drain the items in order, so defaults download one
at a time across the node and never compete for its bandwidth.

*Why the old default is not kept during the download.* By the time the
directories change, the file names the new default and the worker has
registered it, so the scheduler places nothing new on the old one.
Actors already running on it hold their own links, and restores of
snapshots taken on it go through the on-demand path like every
non-default version. Keeping its links would only hold cache space.

*GC loop.* `sandbox_asset_gc.go` adds `--sandbox-asset-cache-gc-period`
(5m, 0 disables), `--sandbox-asset-cache-max-bytes` (0 means unbounded:
no pass runs, warned at startup) and `--sandbox-asset-cache-gc-dry-run`.
A pass that finds the cache over budget asks the store to free down to
90% of it, oldest first. Same shape as the image GC: immediate first
pass, serialized passes, panic recovery, shortfall warnings that back
off to a periodic reminder.

*Metrics.* `ate.sandboxassetcache.size` (observable up/down counter,
bytes), `ate.sandboxassetcache.evictions` and
`ate.sandboxassetcache.gc.passes` by `ate.sandboxassetcache.gc.outcome`
(`complete`, `shortfall`, `skipped`); a pass within budget records
nothing, and a dry run records its pass but no evictions. Registry group
and entries added, `hack/verify/metrics.sh` passes, rows and a paragraph
in `docs/observability.md`.

**What a shortfall means.** Everything left is linked by an actor,
linked by a worker as its default, younger than the min age, or busy.
The size then measures what the node's actors and defaults need, not
waste; the budget is too small or the node hosts too many runtimes.

**Not in this PR.** A shared GC driver for the image and asset caches,
which would also give the image GC the metrics it lacks; filed as a
follow-up once this lands. Step 6, the summary doc, which agent-substrate#2792 carries
and which will be rewritten for this layout.

**Tests.** filecache: a linked file, a partly linked tree and a fully
linked tree all survive a pass; unlinked trees still go oldest first; an
entry linked between the listing and the retire, without a touch, is
vetoed at retire time. atelet: a prewarm links the runtime into its
directory and an eviction pass keeps the entry; a second prewarm of the
same directory fetches nothing; removing the pod directory makes the
entry evictable; a job whose directory is gone before it runs, or whose
pod goes while it stages, fetches nothing, recreates nothing, and is not
retried; the sync hands the default over only after the build, removes a
retired default's directory, keeps the current one and its contents
across a re-sync, hands nothing over and touches no directory when the
build fails, and the pod's removal takes the directories; the end-to-end
informer test asserts the link under the worker's directory; retry,
pause-image and attempt-timeout cases as before. GC: target table, flag
validation, quiet, skipped, complete and shortfall passes with the
metrics checked through a manual reader, the size callback, immediate
first pass, ticking, panic recovery.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

…etric family (agent-substrate#2889)

Follow-up to agent-substrate#2756, so that agent-substrate#2796 can give the snapshot cache a budget
without a third copy of the eviction loop.

**What moves.** The sandbox asset cache's eviction loop, flags,
instruments, target formula and shortfall backoff move from
`cmd/atelet/sandbox_asset_gc.go` into a package,
`cmd/atelet/internal/filecachegc`, written against the filecache store.
The loop's behavior is unchanged: immediate first pass, serialized
passes, panic recovery, dry run, budget with a 90% low water, shortfall
warnings that back off to a periodic reminder.

**How a cache uses it.**
- `filecachegc.NewFlags("sandbox-asset-cache", "sandbox asset cache")`
registers `--sandbox-asset-cache-gc-period`,
`--sandbox-asset-cache-max-bytes` and
`--sandbox-asset-cache-gc-dry-run`, so the asset cache's flags are the
same as before. The snapshot cache will call it once with its own
prefix.
- `filecachegc.NewInstruments(meter)` creates the instruments once per
process.
- `filecachegc.Start(ctx, name, store, flags, instruments)` reports the
cache's size, warns once when there is no budget, and runs the loop when
there is a budget and a period.

**One metric family.** `ate.filecache.size`, `ate.filecache.evictions`
and `ate.filecache.gc.passes`, labeled by `ate.filecache.name` (today
`sandbox_asset`) and, for passes, `ate.filecache.gc.outcome`, replace
`ate.sandboxassetcache.size`, `.evictions` and `.gc.passes`. The caches
run on one store and one loop, so one set of instruments describes them,
and "which cache is over budget on this node" is one query grouped by
name. The registry, `ateattr` and the observability guide are updated.
The name is `filecache` rather than something broader because the family
is about the filecache store; the image cache, which measures layers
against volume watermarks, is not a member and keeps its loop.

**Tests.** The asset GC's tests move into the package unchanged in what
they prove, plus one that puts two stores on the same instruments and
asserts two size series under their names.

**Docs and e2e.** The sandbox asset cache page from agent-substrate#2792 and the e2e
metrics suite follow the rename in the last commit: the suite asserts
`ate_filecache_size`, and the page's metrics table and runbook use the
new names and say where the label comes from.

**Follow-up.** agent-substrate#2796 adopts the driver for the snapshot cache and adds
the `snapshot` label value.

🤖 Generated with [Claude Code](https://claude.com/claude-code)