atelet: implement DetachActor - #2852

…nt-substrate#2881)

A suspended actor with a local snapshot is pinned to its node. While
that node was full, its resume failed with `ResourceExhausted` even when
other nodes had room, until atelet detached the actor on its own after
`--snapshot-detach-delay`. Resume now detaches the actor itself, with
`DetachActor` from agent-substrate#2852, and restores it on another node.

Part of agent-substrate#589. This covers a full node. A node that is gone, the
scale-down case, is still open.

#### How the move works

Assignment still runs pinned to the node first. When it finds no room
there, and a worker on another node has room, resume:

1. Calls `DetachActor` on the pinned node. atelet uploads the latest
snapshot and deletes the local copies.
2. Records the detach, which marks the upload `COMPLETED` and clears the
locality and `assigned_node`.
3. Assigns a worker on any node and restores from the durable snapshot.

`DetachActor` is synchronous. If the snapshot isn't durable yet, the
resume waits for atelet to upload it.

#### What the restore keeps

On a worker of the same runtime, the restore keeps the snapshot's
fidelity. On a worker of another runtime, a MEMORY snapshot restores at
ROOTFS, as for any durable-only actor since agent-substrate#2880. Process memory is
lost, but rootfs writes and volumes are kept. ROOTFS and VOLUMES
snapshots keep their fidelity on any worker of the same sandbox class. A
template can't yet refuse the downgrade; agent-substrate#2818 would add that.

#### When the actor doesn't move

- No worker anywhere has room. The resume returns `ResourceExhausted` as
before, without calling atelet.
- The detach fails. The resume fails, and the actor stays SUSPENDED on
its node, not crashed, since it can still resume there.
- The node is gone. `DetachActor` can't reach atelet, so the resume
returns `Unavailable` and the actor stays pinned.

```mermaid
sequenceDiagram
    participant C as client
    participant A as ate-api-server
    participant L1 as atelet on node1 (full)
    participant O as object storage
    participant L2 as atelet on node2
    C->>A: ResumeActor
    A->>A: assign on node1: no room
    A->>A: node2 has room
    A->>L1: DetachActor
    L1->>O: upload latest snapshot
    L1->>L1: delete local copies
    L1-->>A: state with latest_durable only
    A->>A: mark COMPLETED, clear locality and assigned_node
    A->>A: reload restore source, assign on node2
    A->>L2: Restore(durable snapshot)
    L2->>O: download
    L2-->>A: ok
    A-->>C: RUNNING on node2
```

#### Design choices

- **Any runtime counts as room.**
- Memory is an optimization for GA, so a worker that can't restore the
snapshot's process memory still triggers the move, and the actor
restores there at ROOTFS.
  - Assignment still prefers a worker that keeps it.
- **No claim before the detach.**
- Claiming a worker first would leave the claim and the snapshot record
disagreeing if ate-api-server crashed before the detach.
  - In this order, every step ends in a state resume already handles.
- Two actors detached for one free place race for it, and the loser has
paid an upload, but it ends durable-only and its next resume schedules
it onto any node without another detach.

#### Also worth knowing

- A concurrent update fails the record write with `Aborted`, and the
next resume detaches again; atelet returns the same state. If the node
has room by then, the resume restores there from the upload instead, and
the record keeps the upload `PENDING`, because the detach cancelled
atelet's report.
- `DetachActor` errors map as the tag workflow maps `PromoteSnapshot`
errors, plus atelet's `Aborted` passed through.
- A resume that detaches records two scheduler samples, `no_capacity`
then `assigned`. A dashboard that reads `no_capacity` as a failed resume
will count these.
- No API changes.
- `TestResumeActor_RelocatesAfterDetach` checked `ResourceExhausted` at
a point where node2 already had room, which now detaches. The check
moved to before node2 exists.
- The functional fake's `DetachActor` returns the named actor's last
checkpointed snapshot, not the last of any actor.

…agent-substrate#2796)

Replaces agent-substrate#1551, which was four commits from September that no longer
applied: the restore source message, the download path and the module
path all changed underneath it. Same design, ported onto today's main,
after agent-substrate#2756 and agent-substrate#2889, in four commits, plus three from review: the
sharing derived from the snapshot's owner, the gVisor worker evicting a
shared file's pages before unlinking it, and the manifest read through
the cache.

Milestone M2 of the node-local artifact cache from agent-substrate#690: an actor
restoring from a shared snapshot, the one a tag holds and every actor
created from it starts on, gets its files from a node-level cache
instead of its own download.

**Sharing is the control plane's knowledge.** `ObjectStorage` gains
`SnapshotSharing`, the property rather than the snapshot's kind, so new
shared snapshot kinds need no atelet change. ate-api-server sets it on
Restore from the snapshot's recorded owner: SHARED for a tag's snapshot,
borrowed at creation and immutable, PRIVATE for the actor's own, which
its next suspend replaces, and for an unrecorded owner; the local-copy
fallback object carries it too. Neither side infers it from the URI.

**The cache.** A `filecache.Store` under `/var/lib/ate/snapshot-cache`,
behind `--snapshot-cache-dir` (empty disables),
`--snapshot-cache-min-age` and `--snapshot-cache-fetch-timeout`, opened
at startup with crash debris swept before atelet serves. The directory
must be under the node state dir: the object-store plugin writes only
there, so a cache elsewhere could never be filled, and startup refuses
it. Every flag is validated whether or not the cache is enabled, and the
help text says which are read only while it is. The image cache's
base-path check moved to `nodepath.UnderBasePath`, which both caches
use.

**Eviction.** The cache runs on the eviction loop every file cache on
the node shares, `cmd/atelet/internal/filecachegc` from agent-substrate#2889:
`--snapshot-cache-gc-period`, `--snapshot-cache-max-bytes` and
`--snapshot-cache-gc-dry-run`, a budget with a 90% low water, least
recently used first, never an entry a restore still links. Its size,
evictions and passes appear under the existing `ate.filecache.*`
instruments as a second value of `ate.filecache.name`, `snapshot`; no
new instrument and no new loop. Every flag is validated whether or not
the cache is enabled, as the other flags in atelet are, and the help
text says which ones are read only while it is; a disabled cache starts
no loop and reports no size.

**The restore path.** A restore whose durable snapshot is declared
SHARED stages each file through the cache; concurrent restores of one
snapshot share a single download, later ones read disk. How a hit is
served depends on the sandbox class the request's runtime resolved to.
ateworker-gvisor gets hard links. Once the restore's page loading is
done the worker evicts the memory and filesystem images from the page
cache and unlinks them from its restore dir, keeping only the small
descriptors linked: a file with another link is treated like a preserved
restore dir, since unlinking it would free nothing and would leave its
pages charged to the actor's cgroup, the case agent-substrate#1917 fixed. So a link
protects an image's entry through the restore and the min age, not for
the actor's life; an entry evicted later is still read to the end
through the descriptors runsc holds open, and the next restore reads the
images from local disk. ateworker-microvm gets private hole-preserving
copies, not because a link would corrupt the cache, it replaces
`config.json` by rename and its delta merge copies when the inode is
shared, but because a link would cost more than it saves: an on-demand
restore keeps `memory-ranges` open for the VM's life, which would pin
the entry that long and turn every suspend's in-place merge into a full
copy, and the worker's page-cache eviction would drop the cached pages
for every other restore; agent-substrate#2948 tracks what links would take. EXDEV on a
link, from a cache bind-mounted apart from the restore dirs, degrades to
a copy, warned once per process. The manifest of a shared snapshot is
read through the cache too, so a restore that finds the snapshot on the
node makes no request to object storage; a private snapshot's manifest
is read fresh, as the commit marker it is. The cache key is the object's
address without its query, so a signed URL's credential reaches neither
the key, the cache's errors, `meta.json` nor the logs; and a restore
that fails after staging removes its restore-state dir, so the links do
not hold the cache's entries until the next lifecycle call. The fetch
goes through the object-store plugin as the direct path does, one file
into a scratch dir on the cache's filesystem; because the cache then
writes the restore dir by path rather than through the plugin's
`os.Root`, a symlink waiting there fails the restore instead of being
followed.

**Not here.** Private snapshots stay uncached by design. Links for
ateworker-microvm are agent-substrate#2948, a worker change. The open-and-sweep code
the two file caches share in shape is still duplicated, since the
snapshot cache can be disabled and degrades to copies across mounts
where the asset cache refuses to start.

**Tests.** ate-api functional: a golden restore declares SHARED, the
actor's own snapshot PRIVATE. ate-api unit: the detach-and-resume test
from agent-substrate#2852 compares the whole Restore snapshot, so its fixture now puts
the snapshot under the actor's own UID, and it expects PRIVATE for the
downloaded copy and nothing for a local one. atelet: flag validation,
enabled or not, including the eviction flags, the fetch timeout, and a
directory outside the node state dir; two shared manifest reads download
once and a private one reads fresh; open disabled, creating its root,
sweeping debris; the sharing gate and the class modes; through the
plugin-backed test atelet, two link restores share one download and one
inode and the file is read-only, copy mode gives private inodes and
survives a consumer's mutation, a nil cache downloads, a missing file's
error reaches the caller with nothing left behind, a plugin error
through the cached modes keeps its code and leaks no credential from a
signed URI, a restore that fails at the worker after staging leaves no
file in restore-state and no link on the cache's entry, four concurrent
restores download once, a file in the checkpoint subdirectory lands
there, and symlinks in place of the file or the subdirectory fail; on
Linux, a restore dir on another mount gets a copy with no second
download and one warning for two restores. ateworker-gvisor, on Linux:
staged images hard-linked elsewhere are evicted after the restore wait
and only then unlinked, with the other links intact.

🤖 Generated with [Claude Code](https://claude.com/claude-code)