Enforce minimum snapshot fidelity when resuming actors and report restored fidelity - #2818

…nt-substrate#2881)

A suspended actor with a local snapshot is pinned to its node. While
that node was full, its resume failed with `ResourceExhausted` even when
other nodes had room, until atelet detached the actor on its own after
`--snapshot-detach-delay`. Resume now detaches the actor itself, with
`DetachActor` from agent-substrate#2852, and restores it on another node.

Part of agent-substrate#589. This covers a full node. A node that is gone, the
scale-down case, is still open.

#### How the move works

Assignment still runs pinned to the node first. When it finds no room
there, and a worker on another node has room, resume:

1. Calls `DetachActor` on the pinned node. atelet uploads the latest
snapshot and deletes the local copies.
2. Records the detach, which marks the upload `COMPLETED` and clears the
locality and `assigned_node`.
3. Assigns a worker on any node and restores from the durable snapshot.

`DetachActor` is synchronous. If the snapshot isn't durable yet, the
resume waits for atelet to upload it.

#### What the restore keeps

On a worker of the same runtime, the restore keeps the snapshot's
fidelity. On a worker of another runtime, a MEMORY snapshot restores at
ROOTFS, as for any durable-only actor since agent-substrate#2880. Process memory is
lost, but rootfs writes and volumes are kept. ROOTFS and VOLUMES
snapshots keep their fidelity on any worker of the same sandbox class. A
template can't yet refuse the downgrade; agent-substrate#2818 would add that.

#### When the actor doesn't move

- No worker anywhere has room. The resume returns `ResourceExhausted` as
before, without calling atelet.
- The detach fails. The resume fails, and the actor stays SUSPENDED on
its node, not crashed, since it can still resume there.
- The node is gone. `DetachActor` can't reach atelet, so the resume
returns `Unavailable` and the actor stays pinned.

```mermaid
sequenceDiagram
    participant C as client
    participant A as ate-api-server
    participant L1 as atelet on node1 (full)
    participant O as object storage
    participant L2 as atelet on node2
    C->>A: ResumeActor
    A->>A: assign on node1: no room
    A->>A: node2 has room
    A->>L1: DetachActor
    L1->>O: upload latest snapshot
    L1->>L1: delete local copies
    L1-->>A: state with latest_durable only
    A->>A: mark COMPLETED, clear locality and assigned_node
    A->>A: reload restore source, assign on node2
    A->>L2: Restore(durable snapshot)
    L2->>O: download
    L2-->>A: ok
    A-->>C: RUNNING on node2
```

#### Design choices

- **Any runtime counts as room.**
- Memory is an optimization for GA, so a worker that can't restore the
snapshot's process memory still triggers the move, and the actor
restores there at ROOTFS.
  - Assignment still prefers a worker that keeps it.
- **No claim before the detach.**
- Claiming a worker first would leave the claim and the snapshot record
disagreeing if ate-api-server crashed before the detach.
  - In this order, every step ends in a state resume already handles.
- Two actors detached for one free place race for it, and the loser has
paid an upload, but it ends durable-only and its next resume schedules
it onto any node without another detach.

#### Also worth knowing

- A concurrent update fails the record write with `Aborted`, and the
next resume detaches again; atelet returns the same state. If the node
has room by then, the resume restores there from the upload instead, and
the record keeps the upload `PENDING`, because the detach cancelled
atelet's report.
- `DetachActor` errors map as the tag workflow maps `PromoteSnapshot`
errors, plus atelet's `Aborted` passed through.
- A resume that detaches records two scheduler samples, `no_capacity`
then `assigned`. A dashboard that reads `no_capacity` as a failed resume
will count these.
- No API changes.
- `TestResumeActor_RelocatesAfterDetach` checked `ResourceExhausted` at
a point where node2 already had room, which now detaches. The check
moved to before node2 exists.
- The functional fake's `DetachActor` returns the named actor's last
checkpointed snapshot, not the last of any actor.