atelet,ateom: restore local pause checkpoints directly from LocalSnap - #1876

While benchmarking pause actor I ran into multiple bottlenecks.

I'm currently importing
agent-substrate#935 into my branch but
agent-substrate#1876 is another
alternative to solve the first bottleneck.

The 2nd bottleneck (primary target of this PR) is due to fsyncs being
called in writeFileAtomic() for sandbox-assets.json and SystemInfoVolume
files. Because these are new files that require ext4 filesystem metadata
updates, and we have ext4 mounted in ordered mode, calling fsync on the
file + directory causes all other pending writes (such as other
checkpoints) to be flushed. So the file creation ends up being blocked
behind other actor's checkpoint syncs.

These files do not actually need to be flushed to disk. SystemInfo is
recreatable. sandbox-assets.json is written during Run/Resume(), and
read during Checkpoint(). But once the checkpoint is finished, the
manifest is saved with the snapshot. If the node crashes between
Run/Resume and Checkpoint, the actor is crashed anyway.

### Single-Node Pause/Resume Benchmark Summary
**Configuration:** 17 workers, 15 actors, `glutton` (`512 MiB` RAM
capacity, `32 MiB` churn per cycle), `120s` run, `1.0s` wait, `100%`
pause (`0%` suspend)

| Metric | Baseline (`upstream/main`) | + Commit 1 (`os.Link` local
checkpoint) | + Commit 2 (remove `fsync` on ephemeral volumes) |
| :--- | :---: | :---: | :---: |
| **Completed Cycles** (`120s`) | `106.5` (`112` pauses / `101` resumes)
| `160.0` (`165` pauses / `155` resumes) | **`212.5`** (`214` pauses /
`211` resumes) |
| **Cycle Throughput** | `53.3 cycles/min` (`0.89/s`) | `80.0
cycles/min` (`1.33/s`, **+50.2%**) | **`106.3 cycles/min`** (`1.77/s`,
**2.00x**) |
| **`ResumeActor` p50** | `6,000 ms` | `4,100 ms` | **`210 ms`**
(**28.6x faster**) |
| **`ResumeActor` avg** | `7,023 ms` | `4,867 ms` | **`246 ms`**
(**28.5x faster**) |
| **`ResumeActor` p90 / p99** | `15,000 ms` / `19,000 ms` | `9,400 ms` /
`20,000 ms` | **`350 ms` / `590 ms`** |


- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR

---------

Co-authored-by: Jefftree <[email protected]>

… actor cgroup OOM (agent-substrate#1917)

## Summary

When a gVisor (`runsc`) sandbox is paused or suspended under a tight
memory limit (`ActorMemory`), writing the snapshot (`pages.img`) inside
the actor's own cgroup doubles memory pressure (`guest RAM + snapshot
page cache`), causing severe swap thrashing or cgroup OOM-kills.

This PR decouples snapshot I/O from the actor's memory cgroup and
deterministically reclaims snapshot page cache after restore:

1. **Checkpoint Cgroup Isolation**: Run `runsc checkpoint` in a
dedicated sibling cgroup(-_ckpt) so snapshot page cache is not charged
against the actor's memory limit during pause or suspend.
2. **Post-Restore Cache Reclamation**: After `runsc wait -restore`
completes, evict snapshot page cache (guarded by `cachestat(2)` so
`FADV_DONTNEED` never forces writeback). Staged files in `restore_dir`
are removed only when `preserve_restore_dir` is unset (GCS
`restore-state/`); for local resume (agent-substrate#1876) the files are kept and only
their cache is evicted.

## Notes

Rebased on agent-substrate#1876: file cleanup now keys off `preserve_restore_dir`
instead of `Nlink`, since a local restore no longer has a hard-link.
`DATA_ON_GOLDEN` case dropped after agent-substrate#2105.

## Benchmark

### 1. Memory-Constrained Local Pause/Resume
(`glutton_1user_500mi_gvisor_pause`)
1 user, 500 MiB active guest RAM (64 MiB dirtied before each pause), 600
MiB actor memory cgroup limit (`ActorMemory`), 1m duration.

| Metric | Baseline (`main`) | With Fix (agent-substrate#1917) | Improvement |
| :--- | :---: | :---: | :---: |
| **Completed Cycles (in 60s)** | 6 (1 actor OOM-crashed) | **23** |
**3.8x throughput** |
| **`PauseActor` Failure Rate** | 25.0% (2/8 failed, `runsc checkpoint`
OOM) | **0.0% (0/23)** | **Eliminates checkpoint OOM** |
| **`PauseActor` P50 / Avg / P95** | 3,400 ms / 3,245 ms / 5,800 ms |
**620 ms / 980 ms / 2,900 ms** | **-82% P50 (-70% Avg, -50% P95)** |
| **`ResumeActor` P50 / Avg / P95** | 3,000 ms / 2,957 ms / 3,000 ms |
**370 ms / 404 ms / 660 ms** | **-88% P50 (8.1x faster)** |

### 2. Memory-Constrained GCS Suspend/Resume
(`glutton_1user_500mi_gvisor_suspend`)
1 user, 500 MiB active guest RAM (64 MiB dirtied before each suspend),
600 MiB actor memory cgroup limit (`ActorMemory`), 1m duration.

| Metric | Baseline (`main`) | With Fix (agent-substrate#1917) | Improvement |
| :--- | :---: | :---: | :---: |
| **Completed Cycles (in 60s)** | 2 (5 actors OOM-crashed) | **9 (10
suspends)** | **4.5x–5x throughput** |
| **`SuspendActor` Failure Rate** | 83.3% (10/12 failed, `runsc
checkpoint` OOM) | **0.0% (0/10)** | **Eliminates checkpoint OOM** |
| **`SuspendActor` Latency (successful)** | 7,700 ms – 8,844 ms |
**2,800 ms P50 / 3,105 ms Avg / 4,400 ms P95** | **-64% to -68%** |
| **`ResumeActor` P50 / Avg / P95** | 1,800 ms / 1,715 ms / 1,815 ms
($N=2$) | **1,500 ms / 1,605 ms / 2,300 ms ($N=9$)** | **-17% P50 (-6%
Avg)** |

#### 3. Multi-User Local Pause/Resume
(`glutton_mem_512mi_4_user_gvisor_pause`)
4 concurrent users, 512 MiB active guest RAM (64 MiB dirtied before each
pause), 768 MiB actor memory cgroup limit (`ActorMemory`), 1m duration.

| Metric | Baseline (`main`) | With Fix (`agent-substrate#1917`) | Improvement |
| :--- | :---: | :---: | :---: |
| **Completed Cycles (in 60s)** | `20` (2 actors OOM-crashed) | **`27`**
| **+35% throughput** |
| **`PauseActor` Failure Rate** | **`15.4%`** (4/26 failed, `runsc
checkpoint` OOM) | **`0.0%`** (0/30) | **Eliminates checkpoint OOM** |
| **`PauseActor` P50 / Avg / P95** | `3,100 ms / 3,272 ms / 8,600 ms` |
**`1,000 ms / 1,008 ms / 1,400 ms`** | **-68% P50 (-69% Avg, -84% P95)**
|
| **`ResumeActor` P50 / Avg / P95** | `3,800 ms / 3,698 ms / 6,000 ms`
($N=20$) | **`5,000 ms / 4,365 ms / 5,600 ms`** ($N=27$) | **-7% P95** |

> **Note:** `ResumeActor` P50/Avg is slightly higher because `main`
OOM-crashes 2 of the 4 actors during `PauseActor`, whereas with `agent-substrate#1917`
all 4 actors stay alive and share the worker pod's 2-vCPU limit
concurrently (+35% more completed cycles).

- [x] Tests pass
- [ ] Appropriate changes to documentation are included in the PR