Skip to content

[Bug]atelet: microvm SandboxConfig promotes unverified defaultVersion to sandbox-runtimes.json #3139

Closed

Description

What happened?

When a SandboxConfig version has an invalid asset url or sha256, atelet handles gvisor and microvm differently in workerSandboxSyncer.versionAttrs (cmd/atelet/worker_sandbox_versions.go).

On gvisor, versionAttrs calls gvisorCPUFeatures, which calls s.assets.ensureSandboxAssets before returning attributes. If the defaultVersion fails to download or fails SHA256 verification, buildFile returns an error before s.write runs, so existing sandbox-runtimes.json files stay intact. If a restorable version fails, buildFile omits that version from sandbox-runtimes.json and returns errIncomplete.

On microvm, versionAttrs only calls hostCPUIdentity(), which reads /proc/cpuinfo and does not call ensureSandboxAssets. s.write immediately overwrites sandbox-runtimes.json with the unverified defaultVersion or restorable version. The worker watches that file and re-registers with ate-api-server advertising a runtime whose assets cannot be staged. Cold boots and restores placed on that worker then fail in stageSandboxAssets during Run or Restore.

Expected Behavior

versionAttrs should stage microvm version assets through ensureSandboxAssets before including the version in sandbox-runtimes.json, matching gvisor:

  • If the defaultVersion assets fail to download or verify, buildFile returns an error without overwriting sandbox-runtimes.json.
  • If a restorable version's assets fail to download or verify, buildFile omits that version from sandbox-runtimes.json and returns errIncomplete so workerSandboxSyncer retries with backoff.

Steps to Reproduce

  1. Deploy Substrate with a microvm WorkerPool (sandboxConfigRef: microvm-default).
  2. Patch SandboxConfig microvm-default to add a version v2 with a wrong sha256 (0000...) and set spec.defaultVersion: v2.
  3. Inspect the worker's advertised runtimes (kubectl ate get workers -o yaml) or sandbox-runtimes.json on the node. The worker advertises v2's sandbox_bundle_hash as its defaultRuntime even though atelet cannot fetch v2's assets.
  4. Create an actor on the pool (kubectl ate create actor ...). Placement selects the worker and Run fails in atelet when stageSandboxAssets rejects the sha256 mismatch.

Sandbox Runtime

microvm (Cloud Hypervisor)

Agent Substrate Version / Commit SHA

00f029145 (main)

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions