atelet: serve sandbox assets from the node-local file cache, keyed by sandbox runtime hash #2675

Summary

Sandbox assets (runsc, the gVisor release tree, the micro-VM kernel, VMM and guest image) are downloaded by atelet into flat files under static-files, shared by path across every actor on the node, and never pruned. A separate prewarmer fetches the default version of every SandboxConfig in the cluster, whether or not any worker on the node uses it. The filecache library from #690 (M1 merged in #1517) exists to replace this, and the worker sandbox syncer already resolves which SandboxConfig each local worker runs. This issue tracks moving assets onto that cache and onto the per-worker resolution.

Goals

• One content-addressed store per node for all sandbox assets, with a byte budget and eviction.
• Each actor holds hard links to its assets under its own directory. A running actor protects its assets by link count, so eviction can never break it.
• A node-wide index from sandbox runtime hash to asset record, built from the SandboxConfigs of the workers on the node. Run and Restore resolve assets through it. This prepares for the control-plane change that will replace sandbox_assets in RunRequest and RestoreRequest with the runtime hash the worker registered.
• Prewarm only the default version of each config a local worker uses, through the same code path Run uses, so a Run arriving mid-prewarm joins the in-flight download.
• atelet does not serve AteomHerder until the index is loaded after a restart.

Non-goals

• The proto change itself. Until it lands, atelet derives the hash from the request's sandbox_assets with the existing versionRuntimeHash and falls back to the request's record on an index miss, logging the miss.
• Prewarming non-default versions. They load on demand at Restore; the first restore of one on a node pays its download.
• Releasing index entries. Config churn is rare and entries are small, so the index keeps every hash for the life of the process.

Design notes

• The store root lives under the base path, on the same mount as the actors dir, so hits are hard links. A cross-mount setup degrades to copies and warns at startup.
• Assets are consumed by path from <actor dir>/sandbox-assets/. Both ateom validators and the kata process cleanup switch from the static-files prefix to the actor's own directory, which ateom already receives in every request. Rolled together with atelet.
• Prewarm runs on its own queue keyed by runtime hash, outside the syncer's write path, so worker registration never waits on a download. It links into a scratch directory and removes it, which reuses GetFileTo unchanged.
• The only new library method is a tree link-out for directory entries, for the extracted gVisor release.
• Legacy flat files in static-files are removed at startup. Open inodes survive unlink, and ateom receives asset paths on every later RPC.

Steps, one PR each

1. filecache: preserve exec bits on publish, tree link-out for directory entries, directory-aware sizing and eviction.
2. atelet: open the assets store, link assets per actor through it, update the ateom validators and kata cleanup, remove the legacy layout.
3. atelet: runtime hash index owned by the syncer, interim hashing of the request with logged fallback, startup gate with a readiness check.
4. atelet: default-version prewarm queue replaces sandbox_prewarm.go and the KVM capability gate.
5. atelet: GC driver for the assets store mirroring the image cache GC, with the index's default versions as the root set, plus metrics registry entries.
6. E2E under eviction pressure and a summarizing design doc.

Accepted trade-off

The startup gate means a permanently failing SandboxConfig watch also blocks Checkpoint and Terminate for actors already on the node, since they share the listener. The readiness reason and periodic warnings make this diagnosable. Gating only Run and Restore inside the handlers is the fallback if this proves common.

Related: #690, #1517, #1551.