Skip to content

Remove onResume.fromData and the DATA_ON_GOLDEN restore scope - #2105

Merged
Zoe Zhao (zoez7) merged 4 commits into
agent-substrate:mainfrom
shrutiyam-glitch:issue-2027
Oct 2, 2026
Merged

Zoe Zhao (zoez7) merged 4 commits into
agent-substrate:mainfrom
shrutiyam-glitch:issue-2027

Conversation

@shrutiyam-glitch

@shrutiyam-glitch Shrutiya Mohan (shrutiyam-glitch) commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #2027

Actor lifecycle v2 will not restore an actor by combining the golden snapshot's guest state with the actor's own data snapshot. onResume.fromData: GOLDEN was the only way into that path, so this removes the setting and everything behind it.

Behavior changes

  • API: an ActorTemplate that sets snapshotConfig.onResume is rejected (unknown field). Templates already stored lose the field silently on read.
  • Resume of a DATA snapshot always starts the containers fresh from the image, with the durable dirs filled from the snapshot. Templates that used fromData: GOLDEN now pay app startup on resume and lose any in-memory state that came from the golden image.
  • Unchanged: FULL snapshots, a just-created actor (borrows the golden tag and restores it FULL, or cold-boots via Run if there is none), committing the golden actor (always FULL), and creating actors from a tag.
  • Restore path: a DATA_ON_GOLDEN restore used to need two manifest GETs and downloads from two snapshots; every restore now reads one manifest and one snapshot.
  • Metrics: ate.snapshot.scope takes only full, data or unknown.
  • Rollout: atelet rejects scope 3, so upgrade ateapi and atelet together. Per the pre-v1.0 policy there is no compatibility shim.

This is a Breaking change

  • Tests pass
  • Appropriate changes to documentation are included in the PR

Actor lifecycle v2 will not restore by combining the golden snapshot with
an actor's data snapshot. Drop SnapshotConfig.on_resume, OnResumeConfig and
ResumeSource, and the golden lookup on resume. A DATA snapshot now always
resumes by starting the containers afresh with its durable data restored.
atelet still accepts the DATA_ON_GOLDEN scope; a follow-up removes it.
With onResume gone from ActorTemplate, ateapi never asks for a DATA
snapshot to be restored on top of the golden snapshot. Remove
SNAPSHOT_SCOPE_DATA_ON_GOLDEN and RestoreRequest.base_config from the
atelet API and golden_snapshot_uri from the ateom API, along with the
golden manifest fetch, the combined download, and the data_on_golden
scope attribute. A restore now reads exactly one snapshot, FULL or DATA.
@zoez7
Zoe Zhao (zoez7) added this pull request to the merge queue Oct 2, 2026
Merged via the queue into agent-substrate:main with commit 6a35150 Oct 2, 2026
10 checks passed
Moe Abadi (moetemp) pushed a commit to moetemp/substrate that referenced this pull request Oct 3, 2026
…agent-substrate#1876)

### Summary

For a local pause resume (`CHECKPOINT_TYPE_LOCAL`), atelet no longer
stages the snapshot into `restore-state/`. It verifies the files in
`local-checkpoint/<snapshotName>/` and passes that directory to ateom as
`ActorDirs.restore_dir` with `preserve_restore_dir = true`. The
external/GCS `restore-state/` flow is unchanged.

### Why

`main` currently hard-links the snapshot into `restore-state/`, and the
only thing stopping ateom from mutating the actor's local restore point
is the `Nlink > 1` side effect of that hard-link. This PR replaces that
implicit guard with an explicit `preserve_restore_dir` flag: when set,
ateom never deletes or rewrites files in `restore_dir` (no
post-eager-restore drop of `memory-ranges`, copy-merge instead of
in-place merge on the next OnDemand checkpoint, no `config.json` rewrite
on a same-actor resume).

This also removes the staging step and its cross-filesystem copy
fallback, and gives lifecycle v2, where local snapshots are kept and
reused as a cache after upload, a clear read-only contract. Resume
latency and disk usage are on par with the current hard-link path.

Follow-ups:
- With `DATA_ON_GOLDEN` gone (agent-substrate#2105), `copyLocalCheckpoint` has no
remaining callers and will be removed in a separate PR.
- agent-substrate#1917 / agent-substrate#2030 should key their restore-file cleanup off
`preserve_restore_dir` rather than `Nlink`, since a local restore now
has a single link.

### Original benchmark vs the pre-hardlink full-copy baseline

Setup: glutton, 1 user, 512Mi working set, pause/resume lifecycle.
gVisor actors use 1536Mi so `pages.img` stays ~100% cached after pause
(same as MicroVM); MicroVM actors use 768Mi.

| | ResumeActor P50 | `download` (copy) | `ateom_restore` |
|---|---|---|---|
| gVisor 1536Mi, full-copy baseline | 740ms | ~310ms | ~370ms |
| gVisor 1536Mi, this PR | **430ms** | 0 | ~370ms |
| MicroVM 768Mi, full-copy baseline | 2900ms | ~1.8s (1.4–2.1s) | ~0.97s
(0.78–1.1s) |
| MicroVM 768Mi, this PR | **600ms** | 0 | ~0.57s (0.54–0.82s) |

ResumeActor P50 is client-side (locust). `download` and `ateom_restore`
are server-side medians from atelet's "Restore timing breakdown" log
(6–8 resumes per run).


- [x] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Davanum Srinivas (dims) pushed a commit to dims/substrate that referenced this pull request Oct 16, 2026
… actor cgroup OOM (agent-substrate#1917)

## Summary

When a gVisor (`runsc`) sandbox is paused or suspended under a tight
memory limit (`ActorMemory`), writing the snapshot (`pages.img`) inside
the actor's own cgroup doubles memory pressure (`guest RAM + snapshot
page cache`), causing severe swap thrashing or cgroup OOM-kills.

This PR decouples snapshot I/O from the actor's memory cgroup and
deterministically reclaims snapshot page cache after restore:

1. **Checkpoint Cgroup Isolation**: Run `runsc checkpoint` in a
dedicated sibling cgroup(-_ckpt) so snapshot page cache is not charged
against the actor's memory limit during pause or suspend.
2. **Post-Restore Cache Reclamation**: After `runsc wait -restore`
completes, evict snapshot page cache (guarded by `cachestat(2)` so
`FADV_DONTNEED` never forces writeback). Staged files in `restore_dir`
are removed only when `preserve_restore_dir` is unset (GCS
`restore-state/`); for local resume (agent-substrate#1876) the files are kept and only
their cache is evicted.

## Notes

Rebased on agent-substrate#1876: file cleanup now keys off `preserve_restore_dir`
instead of `Nlink`, since a local restore no longer has a hard-link.
`DATA_ON_GOLDEN` case dropped after agent-substrate#2105.

## Benchmark

### 1. Memory-Constrained Local Pause/Resume
(`glutton_1user_500mi_gvisor_pause`)
1 user, 500 MiB active guest RAM (64 MiB dirtied before each pause), 600
MiB actor memory cgroup limit (`ActorMemory`), 1m duration.

| Metric | Baseline (`main`) | With Fix (agent-substrate#1917) | Improvement |
| :--- | :---: | :---: | :---: |
| **Completed Cycles (in 60s)** | 6 (1 actor OOM-crashed) | **23** |
**3.8x throughput** |
| **`PauseActor` Failure Rate** | 25.0% (2/8 failed, `runsc checkpoint`
OOM) | **0.0% (0/23)** | **Eliminates checkpoint OOM** |
| **`PauseActor` P50 / Avg / P95** | 3,400 ms / 3,245 ms / 5,800 ms |
**620 ms / 980 ms / 2,900 ms** | **-82% P50 (-70% Avg, -50% P95)** |
| **`ResumeActor` P50 / Avg / P95** | 3,000 ms / 2,957 ms / 3,000 ms |
**370 ms / 404 ms / 660 ms** | **-88% P50 (8.1x faster)** |

### 2. Memory-Constrained GCS Suspend/Resume
(`glutton_1user_500mi_gvisor_suspend`)
1 user, 500 MiB active guest RAM (64 MiB dirtied before each suspend),
600 MiB actor memory cgroup limit (`ActorMemory`), 1m duration.

| Metric | Baseline (`main`) | With Fix (agent-substrate#1917) | Improvement |
| :--- | :---: | :---: | :---: |
| **Completed Cycles (in 60s)** | 2 (5 actors OOM-crashed) | **9 (10
suspends)** | **4.5x–5x throughput** |
| **`SuspendActor` Failure Rate** | 83.3% (10/12 failed, `runsc
checkpoint` OOM) | **0.0% (0/10)** | **Eliminates checkpoint OOM** |
| **`SuspendActor` Latency (successful)** | 7,700 ms – 8,844 ms |
**2,800 ms P50 / 3,105 ms Avg / 4,400 ms P95** | **-64% to -68%** |
| **`ResumeActor` P50 / Avg / P95** | 1,800 ms / 1,715 ms / 1,815 ms
($N=2$) | **1,500 ms / 1,605 ms / 2,300 ms ($N=9$)** | **-17% P50 (-6%
Avg)** |

#### 3. Multi-User Local Pause/Resume
(`glutton_mem_512mi_4_user_gvisor_pause`)
4 concurrent users, 512 MiB active guest RAM (64 MiB dirtied before each
pause), 768 MiB actor memory cgroup limit (`ActorMemory`), 1m duration.

| Metric | Baseline (`main`) | With Fix (`agent-substrate#1917`) | Improvement |
| :--- | :---: | :---: | :---: |
| **Completed Cycles (in 60s)** | `20` (2 actors OOM-crashed) | **`27`**
| **+35% throughput** |
| **`PauseActor` Failure Rate** | **`15.4%`** (4/26 failed, `runsc
checkpoint` OOM) | **`0.0%`** (0/30) | **Eliminates checkpoint OOM** |
| **`PauseActor` P50 / Avg / P95** | `3,100 ms / 3,272 ms / 8,600 ms` |
**`1,000 ms / 1,008 ms / 1,400 ms`** | **-68% P50 (-69% Avg, -84% P95)**
|
| **`ResumeActor` P50 / Avg / P95** | `3,800 ms / 3,698 ms / 6,000 ms`
($N=20$) | **`5,000 ms / 4,365 ms / 5,600 ms`** ($N=27$) | **-7% P95** |

> **Note:** `ResumeActor` P50/Avg is slightly higher because `main`
OOM-crashes 2 of the 4 actors during `PauseActor`, whereas with `agent-substrate#1917`
all 4 actors stay alive and share the worker pod's 2-vCPU limit
concurrently (+35% more completed cycles).

- [x] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Remove the onResume.fromData field from ActorTemplate

2 participants