Skip to content

ocispec: make gVisor durable directory root mode 0777 - #3457

Merged
Tim Hockin (thockin) merged 1 commit into
agent-substrate:mainfrom
nybidari:gvisor-durdir-0777
Nov 4, 2026
Merged

Tim Hockin (thockin) merged 1 commit into
agent-substrate:mainfrom
nybidari:gvisor-durdir-0777

Conversation

@nybidari

@nybidari nybidari commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Part of #3226.

The API guide says a durableDir is created 0777, like an emptyDir. On gVisor its root is 01777. The volume is a tmpfs inside the sandbox, and its mount hint sets no mode, so it gets the tmpfs default. With the sticky bit, a user can't delete or rename another user's files in the volume's root.
A durable dir created before this keeps 01777 when it is restored from a VOLUMES or ROOTFS snapshot, since the snapshot records its mode.

Breaking Change:

A gVisor actor whose template has a durableDir can't restore a MEMORY snapshot taken before this change. On a MEMORY restore, runsc checks that the sandbox spec's mount hints match the checkpoint, except .source. The new dev.gvisor.spec.mount.<volume>.options key doesn't match, so runsc restore fails and the actor goes to CRASHED. This includes MEMORY golden snapshots of these templates, and actors suspended while their worker rolls during the upgrade. No API restores such a snapshot without its memory.
On upgrade:

  • Recreate affected templates so they take new golden snapshots.
  • Delete and recreate affected actors that hold a MEMORY snapshot from before the upgrade.
    A rollback breaks the same way. MEMORY snapshots taken after the upgrade fail to restore on the old worker. From v1.0.0, changes must keep upgrades and rollbacks working, so this has to land before then.

AI use:
AI-assisted, I reviewed and tested it.

  • Tests pass
  • Docs updated

Comment thread internal/ocispec/gvisor.go Resolved
gVisor durable directory volumes were mounted with tmpfs default 01777
because the mount hint specified no mode option. Set mode=0777 on the
mount hint annotations to match the API guide and microVM.

Signed-off-by: Nayana Bidari <[email protected]>
@BenTheElder

Copy link
Copy Markdown
Collaborator

We actually have a way to express snapshot compatibility constraints, though nothing in main is using it yet. This might be a good use case. I think we don't want to outright break memory snapshots on v1.0.0-rc.1 but we can incrementally switch to this.

cc Zoe Zhao (@zoez7) Huy Pham (@hdp617)

@zoez7

Copy link
Copy Markdown
Collaborator

Here's how to expression the snapshot compatibility:

  1. Each gvisor-worker needs to start reporting a new attribute, let's call it "root_mode", this will be saved in the worker's status field after RegisterWorkerRequest is processed.
  2. During restore, substrate already compares the attributes reported by the worker, with the ones stamped into the snapshots in internal/hardware/hardware.go. However I realized there is a bug in matchesCompat: it currently only iterates over r.compat.GetAttributes() (the snapshot's attributes) and ignores keys that only the worker has, so the new attribute doesn't cause a downgrade to "rootfs". I will add a small PR to fix this.

@zoez7

Copy link
Copy Markdown
Collaborator

Benjamin Elder (@BenTheElder) If you decide this is M3 blocking, AND don't want to outright break memory snapshots on v1.0.0-rc.1, #3501 becomes M3 blocking as well.

@thockin
Tim Hockin (thockin) added this pull request to the merge queue Nov 4, 2026
Merged via the queue into agent-substrate:main with commit 76cba32 Nov 4, 2026
5 checks passed
Benjamin Elder (BenTheElder) pushed a commit that referenced this pull request Nov 5, 2026
Part of #3226.

The API guide says a `durableDir` is created `0777`, like an emptyDir.
On gVisor its root is `01777`. The volume is a tmpfs inside the sandbox,
and its mount hint sets no mode, so it gets the tmpfs default. With the
sticky bit, a user can't delete or rename another user's files in the
volume's root.
A durable dir created before this keeps `01777` when it is restored from
a VOLUMES or ROOTFS snapshot, since the snapshot records its mode.

Breaking Change:

A gVisor actor whose template has a `durableDir` can't restore a MEMORY
snapshot taken before this change. On a MEMORY restore, runsc checks
that the sandbox spec's mount hints match the checkpoint, except
`.source`. The new `dev.gvisor.spec.mount.<volume>.options` key doesn't
match, so `runsc restore` fails and the actor goes to `CRASHED`. This
includes MEMORY golden snapshots of these templates, and actors
suspended while their worker rolls during the upgrade. No API restores
such a snapshot without its memory.
On upgrade:
- Recreate affected templates so they take new golden snapshots.
- Delete and recreate affected actors that hold a MEMORY snapshot from
before the upgrade.
A rollback breaks the same way. MEMORY snapshots taken after the upgrade
fail to restore on the old worker. From v1.0.0, changes must keep
upgrades and rollbacks working, so this has to land before then.

AI use:
AI-assisted, I reviewed and tested it.

- [x] Tests pass
- [ ] Docs updated

Signed-off-by: Nayana Bidari <[email protected]>
(cherry picked from commit 76cba32)
Signed-off-by: dberkov <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants