Repository navigation
Remove onResume.fromData and the DATA_ON_GOLDEN restore scope - #2105
Merged
Zoe Zhao (zoez7) merged 4 commits intoOct 2, 2026
Merged
Conversation
Actor lifecycle v2 will not restore by combining the golden snapshot with an actor's data snapshot. Drop SnapshotConfig.on_resume, OnResumeConfig and ResumeSource, and the golden lookup on resume. A DATA snapshot now always resumes by starting the containers afresh with its durable data restored. atelet still accepts the DATA_ON_GOLDEN scope; a follow-up removes it.
With onResume gone from ActorTemplate, ateapi never asks for a DATA snapshot to be restored on top of the golden snapshot. Remove SNAPSHOT_SCOPE_DATA_ON_GOLDEN and RestoreRequest.base_config from the atelet API and golden_snapshot_uri from the ateom API, along with the golden manifest fetch, the combined download, and the data_on_golden scope attribute. A restore now reads exactly one snapshot, FULL or DATA.
Shrutiya Mohan (shrutiyam-glitch)
force-pushed
the
issue-2027
branch
from
October 2, 2026 01:14
a769bc3 to
a87aaea
Compare
Shrutiya Mohan (shrutiyam-glitch)
marked this pull request as ready for review
October 2, 2026 01:40
Zoe Zhao (zoez7)
approved these changes
Oct 2, 2026
1 of 2 tasks
Moe Abadi (moetemp)
pushed a commit
to moetemp/substrate
that referenced
this pull request
Oct 3, 2026
…agent-substrate#1876) ### Summary For a local pause resume (`CHECKPOINT_TYPE_LOCAL`), atelet no longer stages the snapshot into `restore-state/`. It verifies the files in `local-checkpoint/<snapshotName>/` and passes that directory to ateom as `ActorDirs.restore_dir` with `preserve_restore_dir = true`. The external/GCS `restore-state/` flow is unchanged. ### Why `main` currently hard-links the snapshot into `restore-state/`, and the only thing stopping ateom from mutating the actor's local restore point is the `Nlink > 1` side effect of that hard-link. This PR replaces that implicit guard with an explicit `preserve_restore_dir` flag: when set, ateom never deletes or rewrites files in `restore_dir` (no post-eager-restore drop of `memory-ranges`, copy-merge instead of in-place merge on the next OnDemand checkpoint, no `config.json` rewrite on a same-actor resume). This also removes the staging step and its cross-filesystem copy fallback, and gives lifecycle v2, where local snapshots are kept and reused as a cache after upload, a clear read-only contract. Resume latency and disk usage are on par with the current hard-link path. Follow-ups: - With `DATA_ON_GOLDEN` gone (agent-substrate#2105), `copyLocalCheckpoint` has no remaining callers and will be removed in a separate PR. - agent-substrate#1917 / agent-substrate#2030 should key their restore-file cleanup off `preserve_restore_dir` rather than `Nlink`, since a local restore now has a single link. ### Original benchmark vs the pre-hardlink full-copy baseline Setup: glutton, 1 user, 512Mi working set, pause/resume lifecycle. gVisor actors use 1536Mi so `pages.img` stays ~100% cached after pause (same as MicroVM); MicroVM actors use 768Mi. | | ResumeActor P50 | `download` (copy) | `ateom_restore` | |---|---|---|---| | gVisor 1536Mi, full-copy baseline | 740ms | ~310ms | ~370ms | | gVisor 1536Mi, this PR | **430ms** | 0 | ~370ms | | MicroVM 768Mi, full-copy baseline | 2900ms | ~1.8s (1.4–2.1s) | ~0.97s (0.78–1.1s) | | MicroVM 768Mi, this PR | **600ms** | 0 | ~0.57s (0.54–0.82s) | ResumeActor P50 is client-side (locust). `download` and `ateom_restore` are server-side medians from atelet's "Restore timing breakdown" log (6–8 resumes per run). - [x] Tests pass - [ ] Appropriate changes to documentation are included in the PR
1 of 2 tasks
Davanum Srinivas (dims)
pushed a commit
to dims/substrate
that referenced
this pull request
Oct 16, 2026
… actor cgroup OOM (agent-substrate#1917) ## Summary When a gVisor (`runsc`) sandbox is paused or suspended under a tight memory limit (`ActorMemory`), writing the snapshot (`pages.img`) inside the actor's own cgroup doubles memory pressure (`guest RAM + snapshot page cache`), causing severe swap thrashing or cgroup OOM-kills. This PR decouples snapshot I/O from the actor's memory cgroup and deterministically reclaims snapshot page cache after restore: 1. **Checkpoint Cgroup Isolation**: Run `runsc checkpoint` in a dedicated sibling cgroup(-_ckpt) so snapshot page cache is not charged against the actor's memory limit during pause or suspend. 2. **Post-Restore Cache Reclamation**: After `runsc wait -restore` completes, evict snapshot page cache (guarded by `cachestat(2)` so `FADV_DONTNEED` never forces writeback). Staged files in `restore_dir` are removed only when `preserve_restore_dir` is unset (GCS `restore-state/`); for local resume (agent-substrate#1876) the files are kept and only their cache is evicted. ## Notes Rebased on agent-substrate#1876: file cleanup now keys off `preserve_restore_dir` instead of `Nlink`, since a local restore no longer has a hard-link. `DATA_ON_GOLDEN` case dropped after agent-substrate#2105. ## Benchmark ### 1. Memory-Constrained Local Pause/Resume (`glutton_1user_500mi_gvisor_pause`) 1 user, 500 MiB active guest RAM (64 MiB dirtied before each pause), 600 MiB actor memory cgroup limit (`ActorMemory`), 1m duration. | Metric | Baseline (`main`) | With Fix (agent-substrate#1917) | Improvement | | :--- | :---: | :---: | :---: | | **Completed Cycles (in 60s)** | 6 (1 actor OOM-crashed) | **23** | **3.8x throughput** | | **`PauseActor` Failure Rate** | 25.0% (2/8 failed, `runsc checkpoint` OOM) | **0.0% (0/23)** | **Eliminates checkpoint OOM** | | **`PauseActor` P50 / Avg / P95** | 3,400 ms / 3,245 ms / 5,800 ms | **620 ms / 980 ms / 2,900 ms** | **-82% P50 (-70% Avg, -50% P95)** | | **`ResumeActor` P50 / Avg / P95** | 3,000 ms / 2,957 ms / 3,000 ms | **370 ms / 404 ms / 660 ms** | **-88% P50 (8.1x faster)** | ### 2. Memory-Constrained GCS Suspend/Resume (`glutton_1user_500mi_gvisor_suspend`) 1 user, 500 MiB active guest RAM (64 MiB dirtied before each suspend), 600 MiB actor memory cgroup limit (`ActorMemory`), 1m duration. | Metric | Baseline (`main`) | With Fix (agent-substrate#1917) | Improvement | | :--- | :---: | :---: | :---: | | **Completed Cycles (in 60s)** | 2 (5 actors OOM-crashed) | **9 (10 suspends)** | **4.5x–5x throughput** | | **`SuspendActor` Failure Rate** | 83.3% (10/12 failed, `runsc checkpoint` OOM) | **0.0% (0/10)** | **Eliminates checkpoint OOM** | | **`SuspendActor` Latency (successful)** | 7,700 ms – 8,844 ms | **2,800 ms P50 / 3,105 ms Avg / 4,400 ms P95** | **-64% to -68%** | | **`ResumeActor` P50 / Avg / P95** | 1,800 ms / 1,715 ms / 1,815 ms ($N=2$) | **1,500 ms / 1,605 ms / 2,300 ms ($N=9$)** | **-17% P50 (-6% Avg)** | #### 3. Multi-User Local Pause/Resume (`glutton_mem_512mi_4_user_gvisor_pause`) 4 concurrent users, 512 MiB active guest RAM (64 MiB dirtied before each pause), 768 MiB actor memory cgroup limit (`ActorMemory`), 1m duration. | Metric | Baseline (`main`) | With Fix (`agent-substrate#1917`) | Improvement | | :--- | :---: | :---: | :---: | | **Completed Cycles (in 60s)** | `20` (2 actors OOM-crashed) | **`27`** | **+35% throughput** | | **`PauseActor` Failure Rate** | **`15.4%`** (4/26 failed, `runsc checkpoint` OOM) | **`0.0%`** (0/30) | **Eliminates checkpoint OOM** | | **`PauseActor` P50 / Avg / P95** | `3,100 ms / 3,272 ms / 8,600 ms` | **`1,000 ms / 1,008 ms / 1,400 ms`** | **-68% P50 (-69% Avg, -84% P95)** | | **`ResumeActor` P50 / Avg / P95** | `3,800 ms / 3,698 ms / 6,000 ms` ($N=20$) | **`5,000 ms / 4,365 ms / 5,600 ms`** ($N=27$) | **-7% P95** | > **Note:** `ResumeActor` P50/Avg is slightly higher because `main` OOM-crashes 2 of the 4 actors during `PauseActor`, whereas with `agent-substrate#1917` all 4 actors stay alive and share the worker pod's 2-vCPU limit concurrently (+35% more completed cycles). - [x] Tests pass - [ ] Appropriate changes to documentation are included in the PR
This was referenced Oct 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2027
Actor lifecycle v2 will not restore an actor by combining the golden snapshot's guest state with the actor's own data snapshot.
onResume.fromData: GOLDENwas the only way into that path, so this removes the setting and everything behind it.Behavior changes
snapshotConfig.onResumeis rejected (unknown field). Templates already stored lose the field silently on read.fromData: GOLDENnow pay app startup on resume and lose any in-memory state that came from the golden image.Runif there is none), committing the golden actor (always FULL), and creating actors from a tag.ate.snapshot.scopetakes onlyfull,dataorunknown.This is a Breaking change