You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 46b6ea9
Browse filesBrowse the repository at this point in the historyBrowse files
docs: document sandbox class differences and fix stale passages (#3468)
Part of #3226
Add docs/reference/sandbox-classes.md comparing the gvisor and microvm
sandbox classes across CPU sizing, memory limits, per-container limits,
NET_RAW capabilities, system mounts, VOLUMES/ROOTFS snapshots and gVisor
fscheckpoint compatibility requirement for cross-version filesystem
restores. Update stale references across architecture, glossary,
quickstart, and guides.
AI use:
AI-assisted, I reviewed and tested it.
- [ ] Tests pass
- [x] Docs updated
---------
Signed-off-by: Nayana Bidari <[email protected]>
WorkerPool ..> Deployment : reconciled by atecontroller
305
316
Deployment "1" *-- "*" WorkerPod : manages
306
317
Actor "*" --> "1" ActorTemplate : actor_template
307
-
Actor "0..1" --> "0..1" Worker : runs on
318
+
Worker "1" *-- "*" ActorAssignment
319
+
ActorAssignment "0..1" --> "1" Actor : actor
308
320
Worker "1" --> "1" WorkerPod : maps to
309
321
```
310
322
@@ -343,9 +355,9 @@ The node-level subsystem manages the physical execution of sandboxes and the mov
343
355
344
356
### Sandbox Classes
345
357
346
-
A `WorkerPool` selects a **sandbox class** (`spec.sandboxClasses[].name`), and each class has a matching `ateworker` image. The sandbox binaries themselves are not baked into the worker image. They come at runtime from the cluster-scoped [`SandboxConfig`](../reference/api-guide.md#3-sandboxconfig-the-sandbox-itself) the `WorkerPool` resolves for its class (an explicit `configRef`, else the class default). Each snapshot records the sandbox runtime it was taken on, and a memory restore runs on a worker that still offers that runtime, so restores stay reproducible across runtime upgrades.
358
+
A `WorkerPool` selects a **sandbox class** (`spec.sandboxClasses[].name`), and each class has a matching `ateworker` image. The sandbox binaries themselves are not baked into the worker image. They come at runtime from the cluster-scoped [`SandboxConfig`](../reference/api-guide.md#3-sandboxconfig-the-sandbox-itself) the `WorkerPool` resolves for its class (an explicit `configRef`, else the class default). Each snapshot records the sandbox runtime it was taken on, and a memory restore runs on a worker that still offers that runtime, so restores stay reproducible across runtime upgrades. For workload-visible differences between the classes, see [Sandbox Classes](../reference/sandbox-classes.md).
347
359
348
-
***gVisor** (`ateworker-gvisor`, the default): Runs the workload under `runsc` for kernel-level sandboxing. Suspend and resume leverage gVisor's native checkpoint/restore of the sandboxed process tree. `DurableDir` volumes and each container's rootfs overlay are saved as separate filesystem checkpoint bundles (`fs/` and `rootfs/`). A `MEMORY` snapshot splits them out of the process checkpoint;`VOLUMES` and `ROOTFS` snapshots save only the bundles they need with `runsc fscheckpoint`, without memory, and restore them on a cold boot.
360
+
***gVisor** (`ateworker-gvisor`, the default): Runs the workload under `runsc` for kernel-level sandboxing. Suspend and resume use gVisor's native checkpoint/restore of the sandboxed process tree. `DurableDir` volumes and each container's rootfs overlay are saved as separate filesystem checkpoint bundles (`fs/` and `rootfs/`). A `MEMORY` snapshot splits them out of the process checkpoint.`VOLUMES` and `ROOTFS` snapshots save only the bundles they need with `runsc fscheckpoint`, without memory, and restore them on a cold boot.
349
361
350
362
***microVM** (`ateworker-microvm`): Runs the workload inside a [Kata Containers](https://katacontainers.io/) guest on the [Cloud Hypervisor](https://www.cloudhypervisor.org/) VMM. Suspend and resume capture a memory-only VM snapshot and restore it on-demand using `userfaultfd` memory demand-paging. Container rootfs writes are host-backed: the overlay is assembled on the host (read-only OCI image lower plus a per-actor writable upper) and served to the guest over the single virtio-fs share, so they cost reclaimable host page cache rather than guest RAM, and `ROOTFS` and `MEMORY` snapshots ship the upper as its own tar (a `ROOTFS` restore puts it back and cold-boots on it). `DurableDir` volumes travel over that same share and are likewise shipped as a tar, so a `VOLUMES` snapshot can capture them without any guest memory. Each volume is a subdirectory of the share, so an actor can have several at no extra cost in devices.
351
363
@@ -418,7 +430,7 @@ sequenceDiagram
418
430
Tunnel-->>Atelet: ready
419
431
Atelet-->>API: worker pod IP
420
432
API-->>Gateway: worker assignment
421
-
Gateway->>Tunnel: mTLS tunnel to worker port 443
433
+
Gateway->>Tunnel: mTLS CONNECT tunnel to worker port 8443
422
434
Tunnel->>A: forward request to Actor port
423
435
A-->>Tunnel: response
424
436
Tunnel-->>Gateway: response
@@ -530,21 +542,15 @@ After deletion, the state of the actor (i.e., memory+disk snapshots) is garbage
530
542
531
543
## State Management & Persistence
532
544
533
-
Agent Substrate distinguishes between two types of state, which are currently
534
-
captured together in a single versioned snapshot:
535
-
536
-
1.**Memory Snapshot**: The exact RAM state of the process.
545
+
A snapshot captures up to three cumulative layers of actor state, controlled by `snapshotConfig.preferredFidelity` on the `ActorTemplate`:
537
546
538
-
2.**Working Volume (Disk)**: The files written to the container's writable
539
-
layer (the "working memory").
547
+
1.**`VOLUMES`**: The contents of `DurableDir` volumes mounted into the actor's containers.
548
+
2.**`ROOTFS`**: `VOLUMES` plus files written to each container's writable rootfs overlay since boot.
549
+
3.**`MEMORY`**: `ROOTFS` plus the running process tree and RAM state (`MEMORY` is the default).
540
550
541
-
In the current implementation, both memory and disk states are tied to the
542
-
specific version of the code (ActorTemplate). This ensures strict consistency
543
-
during resumption.
551
+
`MEMORY` and `ROOTFS` state is tied to the `ActorTemplate` (`actorTemplateUid`). `MEMORY` state is also tied to the `sandboxRuntime` version it was captured under. Changing an actor's template while suspended keeps its `VOLUMES` data, and its next resume cold-boots the containers. Resume falls back to `ROOTFS` and cold-boots under the default runtime if no worker that offers the snapshot's runtime version has room.
544
552
545
-
Snapshots are stored durably in **Google Cloud Storage (GCS)**. This model
546
-
allows the physical compute resources in the `WorkerPool` to be fully reclaimed
547
-
while an actor is idle, without losing any process or filesystem progress.
553
+
Snapshots are stored durably in object storage (GCS or S3) through `objectstore-plugin`. This allows worker capacity in the `WorkerPool` to be reclaimed while an actor is suspended.
The template sets no `storageLocation`, so its snapshots go under ate-api-server's `--snapshot-location`. The kind profile points that at the local `rustfs` bucket, `gs://ate-snapshots/`.
75
+
76
76
## 3. Verify Your Running Actor
77
77
78
78
With `kubectl ate proxy actor` running in your first terminal, open a **second terminal** and send an HTTP `POST` request to wake and increment your actor:
Copy file name to clipboardExpand all lines: docs/guides/csi-deployment.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -307,7 +307,7 @@ The CSI Node plugin runs as a DaemonSet on each worker node and handles `NodeSta
307
307
308
308
### Mount Propagation Requirements
309
309
310
-
When the CSI Node plugin mounts an external volume (such as an NFS share or a formatted block device), the filesystem mount is created inside the plugin container. For Substrate's node supervisor (`atelet`) and worker sandboxes (`ateworker-gvisor`) on the host to see this filesystem mount, the following requirements must be met:
310
+
When the CSI Node plugin mounts an external volume (such as an NFS share or a formatted block device), the filesystem mount is created inside the plugin container. For Substrate's node supervisor (`atelet`) and worker sandboxes (`ateworker-gvisor`, `ateworker-microvm`) on the host to see this filesystem mount, the following requirements must be met:
311
311
312
312
1. **Bidirectional Mount Propagation (`mountPropagation: Bidirectional`):** The volume mount on the host Substrate target directory (e.g. `/var/lib/ate`) in the CSI Node plugin container must have `mountPropagation: Bidirectional`. This ensures that any mounts made by the CSI plugin inside the container propagate back to the host filesystem.
313
313
2. **Unix Domain Socket Accessibility:** The CSI Node plugin must place its Unix domain socket under `/var/lib/kubelet/plugins/<driverName>/` (or the path defined in `CSIDriverConfig.spec.nodeSocketOverride`), which is shared with `atelet`.
0 commit comments