microvm: evict snapshot page cache on restore to prevent cgroup memory pressure by chw120 · Pull Request #2030 · agent-substrate/substrate

Chenyi Wang (chw120) · 2026-10-01T00:14:35Z

Summary

After RestoreWorkload loads a MicroVM snapshot into guest memory, the snapshot files (memory-ranges and filesystem tarballs) remain cached in the worker pod's cgroup memory—either because staged files in restore-state/ are cleaned up too late (or not at all for .tar archives), or because preserved local checkpoints in local-checkpoint/ keep their page cache resident. Under tight pod memory limits or high density, this leftover page cache overlaps with active guest RAM and subsequent checkpoints, triggering kernel dirty-page throttling (balance_dirty_pages) and memory-reclaim stalls.

This PR proactively reclaims snapshot page cache during RestoreWorkload before resuming the guest, without introducing synchronous disk I/O stalls.

Changes

Immediate disposal / cache eviction of restore files before Resume: Right after guest memory is loaded and filesystem layers are extracted (before client.Resume), temporary single-link files in restore-state/ (memory-ranges, rootfs-upper-.tar, durable-dir-.tar) are truncated and removed so the kernel discards their cached pages in-memory without disk writeback. When preserve_restore_dir is set (direct local pause restore) or files are hard-linked (Nlink > 1), the underlying snapshot files are preserved and only clean cached pages are evicted.
Non-blocking background page-cache eviction: A lightweight background pass inspects remaining snapshot files (such as preserved local-checkpoint/ snapshots) via cachestat and calls POSIX_FADV_DONTNEED strictly when pages are clean (Dirty == 0 && Writeback == 0), reclaiming cgroup memory as soon as background flushes finish without blocking on disk writeback.

Benchmarks

Evaluated head-to-head on a 15-node c3-highcpu-4 GKE cluster comparing main vs this PR (14b5d346).

1. Memory-Constrained Local Pause/Resume (`glutton_1user_512mi_uvm_pause_768mi`)

1 user, 512 MiB active guest RAM (64 MiB dirtied before each pause), 768 MiB worker pod memory limit, 1m duration.

Metric	Baseline (`main`)	This PR	Delta
Total Requests / Throughput	58 (`1.00 req/s`)	69 (`1.19 req/s`)	+19.0%
`PauseActor` Avg	1,522 ms	756 ms	-50.3% (2.0x faster)
`PauseActor` P50 / P75	620 ms / 2,600 ms	780 ms / 910 ms	+25.8% / -65.0%
`PauseActor` P90 / P99	4,000 ms / 5,000 ms	970 ms / 1,300 ms	-75.8% (4.1x) / -74.0% (3.8x)
`ResumeActor` Avg / P50	573 ms / 610 ms	459 ms / 430 ms	-19.9% / -29.5%
`ResumeActor` P90 / P99	830 ms / 830 ms	540 ms / 630 ms	-34.9% / -24.1%

Note: On main, PauseActor exhibits a bimodal distribution—fast (~620 ms) for the first 1–2 cycles while free cgroup memory remains, followed by 2.6s–5.0s synchronous writeback stalls once un-evicted page cache fills the 768 MiB cgroup. Evicting the snapshot cache smooths PauseActor into a steady ~750–970 ms distribution, cutting average latency in half (-50.3%) and P90 by 4.1x.

2. High-Density GCS Suspend/Resume (`m1_density_512mi_uvm`)

50 concurrent users, 500 actors, 60 workers across 15 nodes (4 vCPU /node), 512 MiB guest RAM, 1 GiB worker memory limit, 5m duration.

Metric	Baseline (`main`)	This PR	Delta
Total Requests / Throughput	3,676 (`12.29 req/s`)	3,809 (`12.74 req/s`)	+3.6%
Total Failures	132 (`3.59%`)	108 (`2.84%`)	-18.2% fewer failures
`ResumeActor` Failure Rate	21 / 97 (`21.65%`)	13 / 112 (`11.61%`)	-46.4% failure rate
`ResumeActor` Avg / P75	11,757 ms / 12,000 ms	9,189 ms / 9,700 ms	-21.8% / -19.2%
`ResumeActor` P90 / P99	36,000 ms / 62,000 ms	18,000 ms / 48,000 ms	-50.0% (2.0x) / -22.6%
`FirstResume` Avg / P90 / P99	2,727 ms / 6,400 ms / 30,000 ms	2,073 ms / 4,100 ms / 20,000 ms	-24.0% / -35.9% / -33.3%
`SuspendActor` Failure Rate	45 / 518 (`8.69%`)	35 / 544 (`6.43%`)	-26.0% failure rate
`SuspendActor` Avg / P50	9,925 ms / 7,600 ms	9,532 ms / 6,900 ms	-4.0% / -9.2%

3. High-Density Local Pause/Resume (`m1_density_512mi_uvm_pause`)

50 concurrent users, 500 actors, 60 workers across 15 nodes (4 vCPU /node), 512 MiB guest RAM, 1 GiB worker memory limit, 5m duration.

Metric	Baseline (`main`)	This PR
Aggregated P50	230 ms	230 ms
`ResumeActor` P50 / Avg	5,100 ms / 6,601 ms	4,600 ms (-9.8%) / 6,804 ms
`ResumeActorFirstResume` P50	1,400 ms	1,400 ms
`PauseActor` P50	6,800 ms	7,800 ms

Note: In m1_density_512mi_uvm_pause, the 1 GiB worker pod limit already fits the 512 MiB guest RAM plus a hard-linked local checkpoint, and unlike GCS Suspend/Resume (where single-link staged files are truncated in-memory before writeback), local Pause checkpoints must persist to disk—so latency under 50 concurrent users on 4-vCPU nodes is dominated by node CPU contention and shared disk write bandwidth rather than page-cache pressure.

Testing

Unit tests in cmd/ateom-microvm/snapshotcache_test.go.

Chenyi Wang (chw120) · 2026-10-01T00:16:14Z

/assign Walter Fender (@cheftako)

ajaysundar.k (ajaysundark) · 2026-10-02T18:02:36Z

/cc

Wu Shuwen (dajiaohuang) · 2026-10-03T07:07:09Z

There is an interaction with merged #1876: the direct-local restore path sets ActorDirs.RestoreDir to the checkpoint directory and sends PreserveRestoreDir=true. In this PR, that call has restoreDir == actorDirs.GetRestoreDir(), so the preserved-source guard does not match; a single-link checkpoint then reaches the truncate-and-remove branch. That destroys the local checkpoint needed for another resume. Could this cleanup receive and honor PreserveRestoreDir? #1876's body calls out this follow-up explicitly.

…agent-substrate#1876) ### Summary For a local pause resume (`CHECKPOINT_TYPE_LOCAL`), atelet no longer stages the snapshot into `restore-state/`. It verifies the files in `local-checkpoint/<snapshotName>/` and passes that directory to ateom as `ActorDirs.restore_dir` with `preserve_restore_dir = true`. The external/GCS `restore-state/` flow is unchanged. ### Why `main` currently hard-links the snapshot into `restore-state/`, and the only thing stopping ateom from mutating the actor's local restore point is the `Nlink > 1` side effect of that hard-link. This PR replaces that implicit guard with an explicit `preserve_restore_dir` flag: when set, ateom never deletes or rewrites files in `restore_dir` (no post-eager-restore drop of `memory-ranges`, copy-merge instead of in-place merge on the next OnDemand checkpoint, no `config.json` rewrite on a same-actor resume). This also removes the staging step and its cross-filesystem copy fallback, and gives lifecycle v2, where local snapshots are kept and reused as a cache after upload, a clear read-only contract. Resume latency and disk usage are on par with the current hard-link path. Follow-ups: - With `DATA_ON_GOLDEN` gone (agent-substrate#2105), `copyLocalCheckpoint` has no remaining callers and will be removed in a separate PR. - agent-substrate#1917 / agent-substrate#2030 should key their restore-file cleanup off `preserve_restore_dir` rather than `Nlink`, since a local restore now has a single link. ### Original benchmark vs the pre-hardlink full-copy baseline Setup: glutton, 1 user, 512Mi working set, pause/resume lifecycle. gVisor actors use 1536Mi so `pages.img` stays ~100% cached after pause (same as MicroVM); MicroVM actors use 768Mi. | | ResumeActor P50 | `download` (copy) | `ateom_restore` | |---|---|---|---| | gVisor 1536Mi, full-copy baseline | 740ms | ~310ms | ~370ms | | gVisor 1536Mi, this PR | **430ms** | 0 | ~370ms | | MicroVM 768Mi, full-copy baseline | 2900ms | ~1.8s (1.4–2.1s) | ~0.97s (0.78–1.1s) | | MicroVM 768Mi, this PR | **600ms** | 0 | ~0.57s (0.54–0.82s) | ResumeActor P50 is client-side (locust). `download` and `ateom_restore` are server-side medians from atelet's "Restore timing breakdown" log (6–8 resumes per run). - [x] Tests pass - [ ] Appropriate changes to documentation are included in the PR

Chenyi Wang (chw120) · 2026-10-07T06:28:24Z

There is an interaction with merged #1876: the direct-local restore path sets ActorDirs.RestoreDir to the checkpoint directory and sends PreserveRestoreDir=true. In this PR, that call has restoreDir == actorDirs.GetRestoreDir(), so the preserved-source guard does not match; a single-link checkpoint then reaches the truncate-and-remove branch. That destroys the local checkpoint needed for another resume. Could this cleanup receive and honor PreserveRestoreDir? #1876's body calls out this follow-up explicitly.

Rebased onto main (#1876) and updated maybeDropStagedMemoryImage / dropStagedMemoryImage / dropStagedSnapshotFile to key off preserveRestoreDir: when preserveRestoreDir is true, snapshot files are preserved on disk and we only evict clean page cache. Please review again. Thanks.

Chenyi Wang (chw120) · 2026-10-07T06:36:44Z

Thanks Walter Fender (@cheftako) for the quick review. I resolved all comments and it's ready to review again.

Wu Shuwen (dajiaohuang) · 2026-10-07T07:02:22Z

Thanks for addressing the PreserveRestoreDir interaction — the updated guard makes sense. Looking good.

…#2030 review fixes

…#2030

Walter Fender (cheftako) · 2026-10-08T20:26:49Z

Minor nits with constants would be nice. Otherwise
/lgtm

Benjamin Elder (BenTheElder)

agent caught some more things ...

Benjamin Elder (BenTheElder) · 2026-10-08T23:56:26Z

+	}
+	defer f.Close()
+	var cstat unix.Cachestat_t
+	if err := unix.Cachestat(uint(f.Fd()), &unix.CachestatRange{}, &cstat, 0); err != nil {


🤖 question 🟢 – cachestat needs Linux 6.5+. On an older node kernel every call here fails and nothing gets evicted except the truncate path, with no log. What kernels do we expect on GKE nodes? If older ones matter we should probably log once when cachestat isn't available, so the no-op is visible.

I don't think we require 6.5+ yet for anything else in the repo yet?

Right, nothing else requires 6.5+, and this doesn't hard-require it either — on < 6.5 kernels cachestat(2) returns ENOSYS and gracefully degrades to a no-op (logging a one-time warning), while the Truncate(0) path still works on all kernels. (GKE COS nodes are already on 6.6+.)

Local development with limactl is pinned to 6.17 ... 6.18 kernels have broken nested virt on arm64.

We also kind of need to consider other clusters, not just GKE. It looks like this degrades safely now.

Benjamin Elder (BenTheElder) · 2026-10-08T23:56:27Z

+		if !evictSnapshotFiles(targets) {
+			return
+		}
+		for _, delay := range []time.Duration{500 * time.Millisecond, 1500 * time.Millisecond, 3000 * time.Millisecond} {


🤖 should-fix 🟡 – I don't think these retries can catch the case they exist for. Dirty pages are skipped, and the kernel won't write back a dirty page until it is older than dirty_expire_centisecs (30s by default) unless background writeback kicks in first. A pause snapshot resumed within a few seconds is almost certainly still dirty for this whole ~5s window, so these passes are probably no-ops for it. The measured win likely comes mostly from the truncate path.

If the checkpoint side writes back and evicts right after the snapshot is written, these files are already clean by restore time and the retries can go away. Otherwise starting writeback here first, with sync_file_range or fdatasync in the goroutine, would make this reliable without blocking the restore.

In glutton_1user_512mi_uvm_pause_768mi (pure local pause/resume, which doesn't use the truncate path), cachestat + retries cut PauseActor P90 from 4,000 ms to 970 ms (avg 1,522 ms -> 756 ms).
With dirty_background_ratio = 10%, a 768 MiB cgroup triggers background writeback at ~25–76 MiB of dirty pages, so writing a 512 MiB snapshot starts kernel writeback immediately rather than waiting 30s for dirty_expire_centisecs, and the retries reclaim those pages once clean.

I also tested evicting on the checkpoint side or forcing immediate writeback on restore, and both regressed 1-user local pause/resume from ~410 ms to ~2.4 s: evicting on checkpoint loses the warm page cache that local resume reads from, and unconditional fdatasync on restore forces 512 MiB of disk writes during rapid pause/resume cycles even when there is no memory pressure.

Benjamin Elder (BenTheElder)

I think we can just walk RestoreDir and not try to walk all local snapshots in ateom, so it doesn't need to know about the atelet paths hardcoded.

I think we can just do this on the checkpoint side as well, which would simplify some things, we don't need the delayed retry, nlink check, etc.

An older rough cut of this:
main...BenTheElder:substrate:microvm-snapshot-nocache

Chenyi Wang (chw120) · 2026-10-09T01:42:04Z

I think we can just walk RestoreDir and not try to walk all local snapshots in ateom, so it doesn't need to know about the atelet paths hardcoded.

I think we can just do this on the checkpoint side as well, which would simplify some things, we don't need the delayed retry, nlink check, etc.

An older rough cut of this: main...BenTheElder:substrate:microvm-snapshot-nocache

Thanks Benjamin Elder (@BenTheElder) Haven Xia (@HavenXia) for the comments. Updated to only walk restoreDir (dropped localCheckpointsDir, the Nlink check, and extraDirs).

I keep eviction on the restore side rather than fsync + FADV_DONTNEED on checkpoint for two reasons:

Keep local pause/resume warm: Evicting on checkpoint forces a synchronous disk flush on pause and turns the next local resume into a cold disk read (~410 ms -> ~2.4 s in 1-user tests).
Reading the snapshot during RestoreWorkload puts it back into page cache: Even if we evict on checkpoint, reading the snapshot from disk during restore populates the page cache again, so we still need to evict after restore so the running actor doesn't hold both guest RAM and snapshot cache.

…y pressure

Benjamin Elder (BenTheElder)

I think this is OK on older kernels but keep in mind that local dev is stuck at 6.17 for the immediate future. This is in the dev docs.

Benjamin Elder (BenTheElder) · 2026-10-14T00:27:34Z

+		return
+	}
+	snapshotcache.DropFiles(targets)
+	slog.InfoContext(ctx, "dropped the staged memory image (eager restore needs no merge base)")


🤖 nit 🟢 – This logs "dropped" whenever FindFiles matched something, even if every DropFile skipped the file or failed to remove it. The old code only logged success when the remove succeeded. Having DropFile return whether it removed the file would keep the log honest.

Benjamin Elder (BenTheElder) · 2026-10-14T00:27:34Z

+				_ = f.Close()
+				continue
+			}
+			anyRemaining = true


🤖 nit 🟢 – Cache is measured before the fadvise, so a file that was fully clean and just got evicted still sets anyRemaining. Every restore then pays one extra 500ms sleep and a second pass that finds nothing. Setting anyRemaining only when the file is skipped for dirty or writeback pages avoids that.

Benjamin Elder (BenTheElder) · 2026-10-14T00:27:34Z

+		if err == nil && os.SameFile(info, t.Info) && info.Size() == t.Info.Size() {
+			var cstat unix.Cachestat_t
+			if err := cachestatFn(uint(f.Fd()), &unix.CachestatRange{}, &cstat, 0); err != nil {
+				if errors.Is(err, unix.ENOSYS) {


🤖 nit 🟢 – Only ENOSYS is logged. A seccomp profile that rejects cachestat with EPERM would turn eviction into a silent no-op. Logging any cachestat error once would make that visible too.

Benjamin Elder (BenTheElder) · 2026-10-14T00:28:48Z

+	// Dropping ahead of Resume means a Resume, agent dial, or wakeup failure leaves
+	// restore-state empty; atelet re-downloads with O_TRUNC and resetActorDirs
+	// wipes the directory on retry, so this costs a re-download on retry rather
+	// than correctness.


🤖 nit 🟢 – This comment names atelet internals (restore-state, O_TRUNC, resetActorDirs). That's the same coupling as the localCheckpointsDir thread, just in prose, and it will go stale when atelet changes. Maybe just say a failure after this point costs a re-download of the snapshot on retry?

Benjamin Elder (BenTheElder) · 2026-10-14T00:35:29Z

+	}
+	defer f.Close()
+	var cstat unix.Cachestat_t
+	if err := unix.Cachestat(uint(f.Fd()), &unix.CachestatRange{}, &cstat, 0); err != nil {


Local development with limactl is pinned to 6.17 ... 6.18 kernels have broken nested virt on arm64.

Benjamin Elder (BenTheElder) · 2026-10-14T00:39:57Z

I think these can be fixed in follow-ups, but please let's fix them.

Chenyi Wang (chw120) · 2026-10-14T04:22:08Z

I think these can be fixed in follow-ups, but please let's fix them.

Thanks Ben for the review! Addressed all the follow-up comments in #1917.

Chenyi Wang (chw120) changed the title ~~ateom-microvm: evict snapshot page cache via POSIX_FADV_DONTNEED~~ ateom-microvm: evict restored snapshot page cache via POSIX_FADV_DONTNEED Oct 1, 2026

Chenyi Wang (chw120) changed the title ~~ateom-microvm: evict restored snapshot page cache via POSIX_FADV_DONTNEED~~ perf(ateom-microvm): evict snapshot page cache via truncate/FADV_DONTNEED with cachestat(2) Oct 1, 2026

Chenyi Wang (chw120) changed the title ~~perf(ateom-microvm): evict snapshot page cache via truncate/FADV_DONTNEED with cachestat(2)~~ microvm: evict snapshot page cache on restore to prevent cgroup memory pressure Oct 1, 2026

Chenyi Wang (chw120) force-pushed the uvm-snapshot-fadvise branch from 14b5d34 to c34338d Compare October 1, 2026 06:06

Chenyi Wang (chw120) requested review from Walter Fender (cheftako) and removed request for Walter Fender (cheftako) October 1, 2026 06:08

Walter Fender (cheftako) reviewed Oct 2, 2026

View reviewed changes

Chenyi Wang (chw120) mentioned this pull request Oct 3, 2026

atelet,ateom: restore local pause checkpoints directly from LocalSnap #1876

Merged

1 of 2 tasks

Chenyi Wang (chw120) force-pushed the uvm-snapshot-fadvise branch from c34338d to c24607c Compare October 7, 2026 06:16

Chenyi Wang (chw120) added a commit to chw120/substrate that referenced this pull request Oct 7, 2026

ateom-gvisor: align checkpoint cache eviction with PR agent-substrate…

022c516

…#2030 review fixes

Chenyi Wang (chw120) added a commit to chw120/substrate that referenced this pull request Oct 7, 2026

ateom-gvisor: align staged restore file tests with PR agent-substrate…

6948306

…#2030

Walter Fender (cheftako) reviewed Oct 8, 2026

View reviewed changes

Comment thread cmd/ateom-microvm/snapshotcache.go Resolved

Walter Fender (cheftako) reviewed Oct 8, 2026

View reviewed changes

Comment thread cmd/ateom-microvm/snapshotcache_test.go Resolved

Walter Fender (cheftako) reviewed Oct 8, 2026

View reviewed changes

Comment thread cmd/ateom-microvm/snapshotcache_test.go Resolved

Chenyi Wang (chw120) force-pushed the uvm-snapshot-fadvise branch from c24607c to ea88409 Compare October 8, 2026 23:11

Dmitry Berkovich (dberkov) enabled auto-merge October 8, 2026 23:50

Dmitry Berkovich (dberkov) previously approved these changes Oct 8, 2026

View reviewed changes

Dmitry Berkovich (dberkov) added this pull request to the merge queue Oct 8, 2026

Benjamin Elder (BenTheElder) reviewed Oct 8, 2026

View reviewed changes

Comment thread cmd/ateom-microvm/actorpath.go Resolved

Benjamin Elder (BenTheElder) removed this pull request from the merge queue due to a manual request Oct 8, 2026

Benjamin Elder (BenTheElder) reviewed Oct 8, 2026

View reviewed changes

Benjamin Elder (BenTheElder) reviewed Oct 9, 2026

View reviewed changes

Comment thread cmd/ateom-microvm/snapshotcache.go Resolved

Benjamin Elder (BenTheElder) reviewed Oct 9, 2026

View reviewed changes

Chenyi Wang (chw120) dismissed Dmitry Berkovich (dberkov)’s stale review via 0629008 October 9, 2026 01:09

Chenyi Wang (chw120) force-pushed the uvm-snapshot-fadvise branch from ea88409 to 0629008 Compare October 9, 2026 01:09

Aditya Shantanu (aditya-shantanu) mentioned this pull request Oct 12, 2026

fix(ateom-gvisor): isolate and evict checkpoint page cache to prevent actor cgroup OOM #1917

Merged

1 of 2 tasks

Aditya Shantanu (aditya-shantanu) reviewed Oct 12, 2026

View reviewed changes

Comment thread cmd/ateom-microvm/snapshotcache.go Resolved

Comment thread cmd/ateom-microvm/snapshotcache_test.go Resolved

Comment thread cmd/ateom-microvm/restore.go Resolved

microvm: evict snapshot page cache on restore to prevent cgroup memor…

9d7582d

…y pressure

Chenyi Wang (chw120) force-pushed the uvm-snapshot-fadvise branch from 0629008 to 9d7582d Compare October 13, 2026 20:40

Benjamin Elder (BenTheElder) reviewed Oct 14, 2026

View reviewed changes

Benjamin Elder (BenTheElder) approved these changes Oct 14, 2026

View reviewed changes

Benjamin Elder (BenTheElder) added kind/feature An enhancement / feature request or implementation area/microVM labels Oct 14, 2026

Benjamin Elder (BenTheElder) added this pull request to the merge queue Oct 14, 2026

Merged via the queue into agent-substrate:main with commit 1485a77 Oct 14, 2026
14 checks passed

Benjamin Elder (BenTheElder) mentioned this pull request Oct 14, 2026

microvm: drop the staged memory image off the resume path #1998

Closed

2 tasks done

Chenyi Wang (chw120) deleted the uvm-snapshot-fadvise branch October 14, 2026 04:22

Conversation

Chenyi Wang (chw120) commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Benchmarks

1. Memory-Constrained Local Pause/Resume (glutton_1user_512mi_uvm_pause_768mi)

2. High-Density GCS Suspend/Resume (m1_density_512mi_uvm)

3. High-Density Local Pause/Resume (m1_density_512mi_uvm_pause)

Testing

Uh oh!

Chenyi Wang (chw120) commented Oct 1, 2026

Uh oh!

ajaysundar.k (ajaysundark) commented Oct 2, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Wu Shuwen (dajiaohuang) commented Oct 3, 2026

Uh oh!

Chenyi Wang (chw120) commented Oct 7, 2026

Uh oh!

Chenyi Wang (chw120) commented Oct 7, 2026

Uh oh!

Wu Shuwen (dajiaohuang) commented Oct 7, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Walter Fender (cheftako) commented Oct 8, 2026

Uh oh!

Uh oh!

Uh oh!

Benjamin Elder (BenTheElder) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Benjamin Elder (BenTheElder) left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Chenyi Wang (chw120) commented Oct 9, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Benjamin Elder (BenTheElder) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Chenyi Wang (chw120) commented Oct 1, 2026 •

edited

Loading

1. Memory-Constrained Local Pause/Resume (`glutton_1user_512mi_uvm_pause_768mi`)

2. High-Density GCS Suspend/Resume (`m1_density_512mi_uvm`)

3. High-Density Local Pause/Resume (`m1_density_512mi_uvm_pause`)

Benjamin Elder (BenTheElder) left a comment •

edited

Loading