fix(ateom-gvisor): isolate and evict checkpoint page cache to prevent actor cgroup OOM - #1917

…agent-substrate#1876)

### Summary

For a local pause resume (`CHECKPOINT_TYPE_LOCAL`), atelet no longer
stages the snapshot into `restore-state/`. It verifies the files in
`local-checkpoint/<snapshotName>/` and passes that directory to ateom as
`ActorDirs.restore_dir` with `preserve_restore_dir = true`. The
external/GCS `restore-state/` flow is unchanged.

### Why

`main` currently hard-links the snapshot into `restore-state/`, and the
only thing stopping ateom from mutating the actor's local restore point
is the `Nlink > 1` side effect of that hard-link. This PR replaces that
implicit guard with an explicit `preserve_restore_dir` flag: when set,
ateom never deletes or rewrites files in `restore_dir` (no
post-eager-restore drop of `memory-ranges`, copy-merge instead of
in-place merge on the next OnDemand checkpoint, no `config.json` rewrite
on a same-actor resume).

This also removes the staging step and its cross-filesystem copy
fallback, and gives lifecycle v2, where local snapshots are kept and
reused as a cache after upload, a clear read-only contract. Resume
latency and disk usage are on par with the current hard-link path.

Follow-ups:
- With `DATA_ON_GOLDEN` gone (agent-substrate#2105), `copyLocalCheckpoint` has no
remaining callers and will be removed in a separate PR.
- agent-substrate#1917 / agent-substrate#2030 should key their restore-file cleanup off
`preserve_restore_dir` rather than `Nlink`, since a local restore now
has a single link.

### Original benchmark vs the pre-hardlink full-copy baseline

Setup: glutton, 1 user, 512Mi working set, pause/resume lifecycle.
gVisor actors use 1536Mi so `pages.img` stays ~100% cached after pause
(same as MicroVM); MicroVM actors use 768Mi.

| | ResumeActor P50 | `download` (copy) | `ateom_restore` |
|---|---|---|---|
| gVisor 1536Mi, full-copy baseline | 740ms | ~310ms | ~370ms |
| gVisor 1536Mi, this PR | **430ms** | 0 | ~370ms |
| MicroVM 768Mi, full-copy baseline | 2900ms | ~1.8s (1.4–2.1s) | ~0.97s
(0.78–1.1s) |
| MicroVM 768Mi, this PR | **600ms** | 0 | ~0.57s (0.54–0.82s) |

ResumeActor P50 is client-side (locust). `download` and `ateom_restore`
are server-side medians from atelet's "Restore timing breakdown" log
(6–8 resumes per run).


- [x] Tests pass
- [ ] Appropriate changes to documentation are included in the PR

…agent-substrate#2796)

Replaces agent-substrate#1551, which was four commits from September that no longer
applied: the restore source message, the download path and the module
path all changed underneath it. Same design, ported onto today's main,
after agent-substrate#2756 and agent-substrate#2889, in four commits, plus three from review: the
sharing derived from the snapshot's owner, the gVisor worker evicting a
shared file's pages before unlinking it, and the manifest read through
the cache.

Milestone M2 of the node-local artifact cache from agent-substrate#690: an actor
restoring from a shared snapshot, the one a tag holds and every actor
created from it starts on, gets its files from a node-level cache
instead of its own download.

**Sharing is the control plane's knowledge.** `ObjectStorage` gains
`SnapshotSharing`, the property rather than the snapshot's kind, so new
shared snapshot kinds need no atelet change. ate-api-server sets it on
Restore from the snapshot's recorded owner: SHARED for a tag's snapshot,
borrowed at creation and immutable, PRIVATE for the actor's own, which
its next suspend replaces, and for an unrecorded owner; the local-copy
fallback object carries it too. Neither side infers it from the URI.

**The cache.** A `filecache.Store` under `/var/lib/ate/snapshot-cache`,
behind `--snapshot-cache-dir` (empty disables),
`--snapshot-cache-min-age` and `--snapshot-cache-fetch-timeout`, opened
at startup with crash debris swept before atelet serves. The directory
must be under the node state dir: the object-store plugin writes only
there, so a cache elsewhere could never be filled, and startup refuses
it. Every flag is validated whether or not the cache is enabled, and the
help text says which are read only while it is. The image cache's
base-path check moved to `nodepath.UnderBasePath`, which both caches
use.

**Eviction.** The cache runs on the eviction loop every file cache on
the node shares, `cmd/atelet/internal/filecachegc` from agent-substrate#2889:
`--snapshot-cache-gc-period`, `--snapshot-cache-max-bytes` and
`--snapshot-cache-gc-dry-run`, a budget with a 90% low water, least
recently used first, never an entry a restore still links. Its size,
evictions and passes appear under the existing `ate.filecache.*`
instruments as a second value of `ate.filecache.name`, `snapshot`; no
new instrument and no new loop. Every flag is validated whether or not
the cache is enabled, as the other flags in atelet are, and the help
text says which ones are read only while it is; a disabled cache starts
no loop and reports no size.

**The restore path.** A restore whose durable snapshot is declared
SHARED stages each file through the cache; concurrent restores of one
snapshot share a single download, later ones read disk. How a hit is
served depends on the sandbox class the request's runtime resolved to.
ateworker-gvisor gets hard links. Once the restore's page loading is
done the worker evicts the memory and filesystem images from the page
cache and unlinks them from its restore dir, keeping only the small
descriptors linked: a file with another link is treated like a preserved
restore dir, since unlinking it would free nothing and would leave its
pages charged to the actor's cgroup, the case agent-substrate#1917 fixed. So a link
protects an image's entry through the restore and the min age, not for
the actor's life; an entry evicted later is still read to the end
through the descriptors runsc holds open, and the next restore reads the
images from local disk. ateworker-microvm gets private hole-preserving
copies, not because a link would corrupt the cache, it replaces
`config.json` by rename and its delta merge copies when the inode is
shared, but because a link would cost more than it saves: an on-demand
restore keeps `memory-ranges` open for the VM's life, which would pin
the entry that long and turn every suspend's in-place merge into a full
copy, and the worker's page-cache eviction would drop the cached pages
for every other restore; agent-substrate#2948 tracks what links would take. EXDEV on a
link, from a cache bind-mounted apart from the restore dirs, degrades to
a copy, warned once per process. The manifest of a shared snapshot is
read through the cache too, so a restore that finds the snapshot on the
node makes no request to object storage; a private snapshot's manifest
is read fresh, as the commit marker it is. The cache key is the object's
address without its query, so a signed URL's credential reaches neither
the key, the cache's errors, `meta.json` nor the logs; and a restore
that fails after staging removes its restore-state dir, so the links do
not hold the cache's entries until the next lifecycle call. The fetch
goes through the object-store plugin as the direct path does, one file
into a scratch dir on the cache's filesystem; because the cache then
writes the restore dir by path rather than through the plugin's
`os.Root`, a symlink waiting there fails the restore instead of being
followed.

**Not here.** Private snapshots stay uncached by design. Links for
ateworker-microvm are agent-substrate#2948, a worker change. The open-and-sweep code
the two file caches share in shape is still duplicated, since the
snapshot cache can be disabled and degrades to copies across mounts
where the asset cache refuses to start.

**Tests.** ate-api functional: a golden restore declares SHARED, the
actor's own snapshot PRIVATE. ate-api unit: the detach-and-resume test
from agent-substrate#2852 compares the whole Restore snapshot, so its fixture now puts
the snapshot under the actor's own UID, and it expects PRIVATE for the
downloaded copy and nothing for a local one. atelet: flag validation,
enabled or not, including the eviction flags, the fetch timeout, and a
directory outside the node state dir; two shared manifest reads download
once and a private one reads fresh; open disabled, creating its root,
sweeping debris; the sharing gate and the class modes; through the
plugin-backed test atelet, two link restores share one download and one
inode and the file is read-only, copy mode gives private inodes and
survives a consumer's mutation, a nil cache downloads, a missing file's
error reaches the caller with nothing left behind, a plugin error
through the cached modes keeps its code and leaks no credential from a
signed URI, a restore that fails at the worker after staging leaves no
file in restore-state and no link on the cache's entry, four concurrent
restores download once, a file in the checkpoint subdirectory lands
there, and symlinks in place of the file or the subdirectory fail; on
Linux, a restore dir on another mount gets a copy with no second
download and one warning for two restores. ateworker-gvisor, on Linux:
staged images hard-linked elsewhere are evicted after the restore wait
and only then unlinked, with the other links intact.

🤖 Generated with [Claude Code](https://claude.com/claude-code)