Skip to content

[bug-bash] ResumeActor attempts ROOTFS restore across CPU architectures instead of dropping to VOLUMES #3230

Open

Description

When ResumeActor relaxes constraints.RequiredRuntime because no worker matches the snapshot's ActorSandboxRuntime, it clears RequiredRuntime completely (hardware.Requirement{} in assignWorkerAttempt) and degrades the restore to SNAPSHOT_FIDELITY_ROOTFS in ensureAteletRestored. Likewise, requiredSnapshotRuntime returns nil for SNAPSHOT_FIDELITY_ROOTFS snapshots.

SNAPSHOT_FIDELITY_ROOTFS only survives sandbox_bundle_hash and CPU feature/model changes within the same CPU architecture. Across architectures (amd64 vs. arm64), any ELF binary written or modified in the container's rootfs upper layer cannot execute on the target architecture.

Steps to Reproduce

  1. Deploy atelet, objectstore-plugin, ateworker-gvisor, and the sandbox demo built for linux/amd64,linux/arm64 on a cluster with both amd64 and arm64 nodes, with two WorkerPools matching workload: sandbox (sandbox-amd64 at replicas: 1, sandbox-arm64 at replicas: 0).
  2. Create and wake a sandbox actor on amd64.
  3. Copy the amd64 entrypoint binary into the container's rootfs upper layer (plus a marker file under /root) and suspend the actor (--mode=sync).
  4. Scale sandbox-amd64 to 0 and sandbox-arm64 to 1, then run kubectl ate resume actor.

Actual Behavior

The scheduler assigns the actor to the arm64 worker and ensureAteletRestored sends RestoreRequest with SNAPSHOT_FIDELITY_ROOTFS. The amd64 upper-layer entrypoint is overlaid onto the arm64 container rootfs and the container fails to start with exec format error.

Expected Behavior

Either:

  1. Keep architecture as a requirement for SNAPSHOT_FIDELITY_ROOTFS and degrade to SNAPSHOT_FIDELITY_VOLUMES when the assigned worker's architecture differs from the snapshot's (so the actor starts on arm64 at VOLUMES using the arm64 image's own entrypoint, with /root upper-layer files gone), or
  2. Document in docs/api-guide.md that all WorkerPools selected by an ActorTemplate must share a single CPU architecture.

Note: This could be a temporary fix for now. We should look into how to define the fallback (default) fidelity based on individual attribute values.

Activity

  1. shrutiyam-glitch commented on Oct 26, 2026

    @shrutiyam-glitch
    CollaboratorAuthor
  2. dberkov commented on Oct 26, 2026

    @dberkov
    Collaborator

    We need to fix it; however, it is not an M3 blocker since the chance someone will mix architectures in the same cluster is low

  3. added this to the M3 milestone on Oct 27, 2026
  4. removed this from the M3 milestone on Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions