Skip to content

Commit 46b6ea9

Browse files
authored
docs: document sandbox class differences and fix stale passages (agent-substrate#3468)
Part of agent-substrate#3226 Add docs/reference/sandbox-classes.md comparing the gvisor and microvm sandbox classes across CPU sizing, memory limits, per-container limits, NET_RAW capabilities, system mounts, VOLUMES/ROOTFS snapshots and gVisor fscheckpoint compatibility requirement for cross-version filesystem restores. Update stale references across architecture, glossary, quickstart, and guides. AI use: AI-assisted, I reviewed and tested it. - [ ] Tests pass - [x] Docs updated --------- Signed-off-by: Nayana Bidari <[email protected]>
1 parent 1261766 commit 46b6ea9

7 files changed

Lines changed: 142 additions & 52 deletions

File tree

‎docs/concepts/architecture.md‎

Lines changed: 32 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -224,6 +224,10 @@ administrative operations and actor environment definitions.
224224
to receive resumed actor states. Optional `spec.template` fields configure
225225
worker pod node selection, tolerations, priority class, and node affinity.
226226

227+
* **SandboxConfig** (Kubernetes CRD): A cluster-scoped resource that lists the
228+
sandbox runtime versions and assets for a sandbox class (`gvisor` or
229+
`microvm`). Each `WorkerPool` resolves a `SandboxConfig` for its class.
230+
227231
* **ActorTemplate** (ate API resource): An immutable definition of an
228232
actor-version. It encapsulates the container image, configuration, and
229233
environment required to generate a "golden" snapshot. ActorTemplates are
@@ -238,12 +242,13 @@ actors and workers. They are stored in a high-performance, low-latency state
238242
store (PostgreSQL) to support real-time operations.
239243

240244
* **Actor**: A specific instance of an ActorTemplate. An Actor record tracks
241-
its globally unique identifier, physical location (Worker IP), current
242-
state (RUNNING or SUSPENDED), and version-specific state metadata.
245+
its globally unique identifier, physical location (Worker IP), lifecycle
246+
state (such as `RUNNING` or `SUSPENDED`), and version-specific state metadata.
243247

244-
* **Worker**: A representation of a worker pod in the WorkerPool. It tracks
245-
the worker's unique identifier, current status (IDLE or BUSY), and the
246-
Actor it is currently hosting (if any).
248+
* **Worker**: A representation of a worker pod in a WorkerPool. It tracks
249+
the worker's unique identifier, status (`ACTIVE`, `UNAVAILABLE`, or
250+
`DRAINING`), capacity and allocated resources, and the Actors it currently
251+
hosts (each represented by an `ActorAssignment` subresource).
247252

248253
### Architectural Rationale
249254

@@ -277,10 +282,12 @@ classDiagram
277282
class WorkerPool {
278283
<<CRD>>
279284
}
285+
class SandboxConfig {
286+
<<CRD>>
287+
}
280288
class Deployment
281289
class WorkerPod {
282290
ateworker
283-
runsc
284291
}
285292
}
286293
@@ -290,21 +297,26 @@ classDiagram
290297
}
291298
class Actor {
292299
<<record>>
293-
status
294-
snapshotRefs
300+
status.state
301+
status.snapshots
295302
}
296303
class Worker {
297304
<<record>>
298-
actorName
299-
podIP
305+
status.state
306+
status.ips
307+
}
308+
class ActorAssignment {
309+
<<subresource>>
300310
}
301311
}
302312
303313
ActorTemplate ..> WorkerPool : worker_selector (labels)
314+
WorkerPool ..> SandboxConfig : sandboxClasses
304315
WorkerPool ..> Deployment : reconciled by atecontroller
305316
Deployment "1" *-- "*" WorkerPod : manages
306317
Actor "*" --> "1" ActorTemplate : actor_template
307-
Actor "0..1" --> "0..1" Worker : runs on
318+
Worker "1" *-- "*" ActorAssignment
319+
ActorAssignment "0..1" --> "1" Actor : actor
308320
Worker "1" --> "1" WorkerPod : maps to
309321
```
310322

@@ -343,9 +355,9 @@ The node-level subsystem manages the physical execution of sandboxes and the mov
343355

344356
### Sandbox Classes
345357

346-
A `WorkerPool` selects a **sandbox class** (`spec.sandboxClasses[].name`), and each class has a matching `ateworker` image. The sandbox binaries themselves are not baked into the worker image. They come at runtime from the cluster-scoped [`SandboxConfig`](../reference/api-guide.md#3-sandboxconfig-the-sandbox-itself) the `WorkerPool` resolves for its class (an explicit `configRef`, else the class default). Each snapshot records the sandbox runtime it was taken on, and a memory restore runs on a worker that still offers that runtime, so restores stay reproducible across runtime upgrades.
358+
A `WorkerPool` selects a **sandbox class** (`spec.sandboxClasses[].name`), and each class has a matching `ateworker` image. The sandbox binaries themselves are not baked into the worker image. They come at runtime from the cluster-scoped [`SandboxConfig`](../reference/api-guide.md#3-sandboxconfig-the-sandbox-itself) the `WorkerPool` resolves for its class (an explicit `configRef`, else the class default). Each snapshot records the sandbox runtime it was taken on, and a memory restore runs on a worker that still offers that runtime, so restores stay reproducible across runtime upgrades. For workload-visible differences between the classes, see [Sandbox Classes](../reference/sandbox-classes.md).
347359

348-
* **gVisor** (`ateworker-gvisor`, the default): Runs the workload under `runsc` for kernel-level sandboxing. Suspend and resume leverage gVisor's native checkpoint/restore of the sandboxed process tree. `DurableDir` volumes and each container's rootfs overlay are saved as separate filesystem checkpoint bundles (`fs/` and `rootfs/`). A `MEMORY` snapshot splits them out of the process checkpoint; `VOLUMES` and `ROOTFS` snapshots save only the bundles they need with `runsc fscheckpoint`, without memory, and restore them on a cold boot.
360+
* **gVisor** (`ateworker-gvisor`, the default): Runs the workload under `runsc` for kernel-level sandboxing. Suspend and resume use gVisor's native checkpoint/restore of the sandboxed process tree. `DurableDir` volumes and each container's rootfs overlay are saved as separate filesystem checkpoint bundles (`fs/` and `rootfs/`). A `MEMORY` snapshot splits them out of the process checkpoint. `VOLUMES` and `ROOTFS` snapshots save only the bundles they need with `runsc fscheckpoint`, without memory, and restore them on a cold boot.
349361

350362
* **microVM** (`ateworker-microvm`): Runs the workload inside a [Kata Containers](https://katacontainers.io/) guest on the [Cloud Hypervisor](https://www.cloudhypervisor.org/) VMM. Suspend and resume capture a memory-only VM snapshot and restore it on-demand using `userfaultfd` memory demand-paging. Container rootfs writes are host-backed: the overlay is assembled on the host (read-only OCI image lower plus a per-actor writable upper) and served to the guest over the single virtio-fs share, so they cost reclaimable host page cache rather than guest RAM, and `ROOTFS` and `MEMORY` snapshots ship the upper as its own tar (a `ROOTFS` restore puts it back and cold-boots on it). `DurableDir` volumes travel over that same share and are likewise shipped as a tar, so a `VOLUMES` snapshot can capture them without any guest memory. Each volume is a subdirectory of the share, so an actor can have several at no extra cost in devices.
351363

@@ -418,7 +430,7 @@ sequenceDiagram
418430
Tunnel-->>Atelet: ready
419431
Atelet-->>API: worker pod IP
420432
API-->>Gateway: worker assignment
421-
Gateway->>Tunnel: mTLS tunnel to worker port 443
433+
Gateway->>Tunnel: mTLS CONNECT tunnel to worker port 8443
422434
Tunnel->>A: forward request to Actor port
423435
A-->>Tunnel: response
424436
Tunnel-->>Gateway: response
@@ -530,21 +542,15 @@ After deletion, the state of the actor (i.e., memory+disk snapshots) is garbage
530542

531543
## State Management & Persistence
532544

533-
Agent Substrate distinguishes between two types of state, which are currently
534-
captured together in a single versioned snapshot:
535-
536-
1. **Memory Snapshot**: The exact RAM state of the process.
545+
A snapshot captures up to three cumulative layers of actor state, controlled by `snapshotConfig.preferredFidelity` on the `ActorTemplate`:
537546

538-
2. **Working Volume (Disk)**: The files written to the container's writable
539-
layer (the "working memory").
547+
1. **`VOLUMES`**: The contents of `DurableDir` volumes mounted into the actor's containers.
548+
2. **`ROOTFS`**: `VOLUMES` plus files written to each container's writable rootfs overlay since boot.
549+
3. **`MEMORY`**: `ROOTFS` plus the running process tree and RAM state (`MEMORY` is the default).
540550

541-
In the current implementation, both memory and disk states are tied to the
542-
specific version of the code (ActorTemplate). This ensures strict consistency
543-
during resumption.
551+
`MEMORY` and `ROOTFS` state is tied to the `ActorTemplate` (`actorTemplateUid`). `MEMORY` state is also tied to the `sandboxRuntime` version it was captured under. Changing an actor's template while suspended keeps its `VOLUMES` data, and its next resume cold-boots the containers. Resume falls back to `ROOTFS` and cold-boots under the default runtime if no worker that offers the snapshot's runtime version has room.
544552

545-
Snapshots are stored durably in **Google Cloud Storage (GCS)**. This model
546-
allows the physical compute resources in the `WorkerPool` to be fully reclaimed
547-
while an actor is idle, without losing any process or filesystem progress.
553+
Snapshots are stored durably in object storage (GCS or S3) through `objectstore-plugin`. This allows worker capacity in the `WorkerPool` to be reclaimed while an actor is suspended.
548554

549555
## Security & Isolation
550556

‎docs/get-started/quickstart.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -66,13 +66,13 @@ data:
6666
OTEL_METRIC_EXPORT_TIMEOUT: "10000"
6767
OTEL_LOGS_EXPORTER: otlp
6868
---
69-
# ActorTemplate snapshotConfig rendered for kind (local bucket)
69+
# ActorTemplate snapshotConfig from the counter demo
7070
snapshotConfig:
71-
onPause: SNAPSHOT_CONTENT_SCOPE_FULL
72-
onCommit: SNAPSHOT_CONTENT_SCOPE_FULL
73-
storageLocation: gs://ate-snapshots/ate-demo-counter/
71+
preferredFidelity: SNAPSHOT_FIDELITY_MEMORY
7472
```
7573
74+
The template sets no `storageLocation`, so its snapshots go under ate-api-server's `--snapshot-location`. The kind profile points that at the local `rustfs` bucket, `gs://ate-snapshots/`.
75+
7676
## 3. Verify Your Running Actor
7777

7878
With `kubectl ate proxy actor` running in your first terminal, open a **second terminal** and send an HTTP `POST` request to wake and increment your actor:

‎docs/guides/csi-deployment.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -307,7 +307,7 @@ The CSI Node plugin runs as a DaemonSet on each worker node and handles `NodeSta
307307

308308
### Mount Propagation Requirements
309309

310-
When the CSI Node plugin mounts an external volume (such as an NFS share or a formatted block device), the filesystem mount is created inside the plugin container. For Substrate's node supervisor (`atelet`) and worker sandboxes (`ateworker-gvisor`) on the host to see this filesystem mount, the following requirements must be met:
310+
When the CSI Node plugin mounts an external volume (such as an NFS share or a formatted block device), the filesystem mount is created inside the plugin container. For Substrate's node supervisor (`atelet`) and worker sandboxes (`ateworker-gvisor`, `ateworker-microvm`) on the host to see this filesystem mount, the following requirements must be met:
311311

312312
1. **Bidirectional Mount Propagation (`mountPropagation: Bidirectional`):** The volume mount on the host Substrate target directory (e.g. `/var/lib/ate`) in the CSI Node plugin container must have `mountPropagation: Bidirectional`. This ensures that any mounts made by the CSI plugin inside the container propagate back to the host filesystem.
313313
2. **Unix Domain Socket Accessibility:** The CSI Node plugin must place its Unix domain socket under `/var/lib/kubelet/plugins/<driverName>/` (or the path defined in `CSIDriverConfig.spec.nodeSocketOverride`), which is shared with `atelet`.

0 commit comments

Comments
 (0)