Skip to content

amd64 runsc default (gVisor nightly 2026-05-19) fails OpenClaw/NemoClaw checkpoint/restore on substrate #2036

Closed

Description

@Matcham89

Description

kagent's default amd64 gVisor runsc (gs://gvisor/releases/nightly/2026-05-19/x86_64/runsc, set in go/core/pkg/app/app.go) cannot restore node.js OpenClaw/NemoClaw actors on the substrate runtime. The golden snapshot checkpoints fine, but ResumeActor → runsc restore of the openclaw sub-container fails with inconsistent private memory files on restore, so the AgentHarness is stuck ActorReady=False (ActorResuming) indefinitely. The bug reproduces only on the amd64 build of the runsc nightly — arm64 restores fine with identical code.

Root cause

A gVisor checkpoint/restore bug in the amd64 build of the pinned 2026-05-19 runsc nightly. It is fixed in a later gVisor nightly: 2026-06-02 checkpoints AND restores successfully, and is before the 2026-06-13 build that breaks checkpoint outright — so 2026-06-02 is a working window. The defect is in the runsc binary, not the substrate version or the kagent controller (confirmed below).

Steps to reproduce

  1. substrate 0.0.6 + kagent 0.9.8 on an amd64 cluster, default runsc (2026-05-19).
  2. Create AgentHarness {runtime: substrate, backend: openclaw} with a valid gatewayTokenSecretRef + modelConfigRef.
  3. RunWorkload (boot) OK → CheckpointWorkload OK → ActorTemplateReady=True.
  4. ResumeActor → runsc restore of the openclaw container fails.

Outcome (actual)

Restore fails on amd64 and the harness never reaches Ready:

FATAL ERROR: starting sub-container [/bin/sh -c ... openclaw gateway run --port 80 --allow-unconfigured]:
  inconsistent private memory files on restore:
    savedMFOwners = [pause:/], mfmap = map[openclaw:/:unwasteSmall: ...]
ate-api: ResumeActor ... workflow failed at step CallAteletRestore:
  while creating workload from golden snapshot: rpc error: code = Internal desc = internal server error

Expected outcome

The OpenClaw/NemoClaw actor restores from its golden snapshot and the AgentHarness reaches Ready / ActorRunning — as it already does on arm64, and as it does on amd64 once runsc is bumped to 2026-06-02 (a fresh harness then reaches Ready / ActorRunning in ~104s).

Evidence / isolation

  • Cross-arch: the identical stack (same ateom build, same runsc date, same nemoclaw/sandbox-base@sha256:d52bee… image, same systrap platform) restores on arm64 (Kind, Apple Silicon) and fails on amd64 (Talos). Not the substrate version — rebuilding ateom-gvisor from substrate HEAD on amd64 reproduces the same failure.
  • amd64 runsc sweep: 2026-05-19 ❌, 2026-05-26 ❌ (both inconsistent private memory files), 2026-06-02 ✅ (checkpoint + restore), 2026-06-13 breaks checkpoint.
  • Caveat: the arm64 repro ran on a different host kernel (Docker Desktop VM), so an arch-vs-kernel contribution can't be 100% ruled out. The provable, actionable claim independent of that: 2026-05-19 amd64 fails to restore, 2026-06-02 fixes it and still checkpoints.

Suggested fix

Bump the amd64 runsc default to 2026-06-02 (sha256 efd12935f6654c91a1389710eb8dfa4d12b6b9be00db87526dc2eb584ad00119); leave arm64 unchanged. Fix submitted in #2035.

Workaround (helm, amd64 only — arm64 default is fine):

controller:
  substrate:
    runscAMD64URL: "gs://gvisor/releases/nightly/2026-06-02/x86_64/runsc"
    runscAMD64SHA256: "efd12935f6654c91a1389710eb8dfa4d12b6b9be00db87526dc2eb584ad00119"

Note: the runsc-compatibility coupling is conceptually owned by kagent-dev/substrate (its C/R is built against a specific runsc version), but that repo has issues disabled, so filing here alongside the fix.

Environment

  • Talos Linux v1.12.2 (Sidero), amd64 · Kubernetes v1.35.0
  • kagent v0.9.8 · substrate chart 0.0.6 (ateom-gvisor v0.0.6) · gVisor platform systrap

Activity

  1. added a commit that references this issue on Jun 17, 2026
    e9e4e34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions