Skip to content

Commit 61ea6c6

Browse files
authored
site: publish per-page Markdown and fix llms.txt and llms-full.txt (#3435)
`ate.dev/llms.txt` linked to full Docsy HTML pages instead of standalone Markdown files, omitted page status markers and schema links, and still used the old `ateom` component name. Meanwhile, `raw-markdown.html` wrote into an HTML template buffer and expanded `include-file` with `replaceRE`, which HTML-escaped code blocks and blockquotes across `ate.dev/llms-full.txt` and "Copy as Markdown" (`&#39;`, `&#34;`, `&gt;`), stripped `$VAR` shell references as regexp capture groups, left relative links unrewritten, and kept decorative `<figure>` HTML blocks. - Add the `MARKDOWN` output format for doc pages (`site/layouts/docs/single.md`) so each page publishes a clean `index.md` alongside `index.html` and exposes `<link rel="alternate" type="text/markdown">` in `<head>`. - Return a plain string from `raw-markdown.html`, expand `include-file` with literal replacement, drop decorative diagram HTML, and rewrite relative links to absolute URLs across `llms-full.txt`, per-page `index.md`, and "Copy as Markdown". - Update `index.llms.txt` to link to each page's `index.md` build, include `[Status: ...]` labels from front matter, link the `ateapi.proto`, `ate.dev/v1` CRD, and OpenTelemetry Weaver metric schemas, and summarize the sandbox model and the split between `ate.dev/v1` Kubernetes CRDs and the `ate-api-server` gRPC API. - Replace leftover `ateom`, `ate-api`, and `herder` references in docs with `ateworker`, `atenet-router`, and `ate-api-server`, and add template tests in `hack/test-site-templates.sh`. AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Mofi Rahman <[email protected]>
1 parent 7cd9fe7 commit 61ea6c6

17 files changed

Lines changed: 167 additions & 45 deletions

File tree

‎docs/concepts/architecture.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -331,9 +331,9 @@ manage actor lifecycles.
331331

332332
The node-level subsystem manages the physical execution of sandboxes and the movement of snapshots.
333333

334-
* **atelet**: A lightweight supervisor running on each node as a DaemonSet. It acts as the "Herder," managing a pool of physical pods and communicating with the Control Plane.
334+
* **atelet**: A lightweight supervisor running on each node as a DaemonSet. It manages local worker pods and communicates with the Control Plane.
335335

336-
* **ateworker**: A specialized sandbox-herder container image — one per sandbox class (`ateworker-gvisor`, `ateworker-microvm`) — that runs inside the physical worker pods. It provides a gRPC interface for `atelet` to trigger `RunWorkload`, `CheckpointWorkload`, and `RestoreWorkload` operations. This separation ensures that the physical pod lifecycle remains decoupled from the sandboxed agent process.
336+
* **ateworker**: The worker container image (`ateworker-gvisor` or `ateworker-microvm`, one per sandbox class) that runs inside each physical worker pod. It provides a gRPC interface for `atelet` to trigger `RunWorkload`, `CheckpointWorkload`, and `RestoreWorkload` operations. This separation ensures that the physical pod lifecycle remains decoupled from the sandboxed agent process.
337337

338338
* **Lifecycle Management**: The `ateworker` process invokes the sandbox runtime to checkpoint or restore processes within the physical pod boundaries — `runsc` for gVisor, or the Kata + Cloud Hypervisor stack for microVMs. (Note: the gVisor backend currently requires a `runsc` version with the `--allow-connected-on-save` flag to work around a bug in networking resumption during checkpointing.)
339339

@@ -343,7 +343,7 @@ The node-level subsystem manages the physical execution of sandboxes and the mov
343343

344344
### Sandbox Classes
345345

346-
A `WorkerPool` selects a **sandbox class** (`spec.sandboxClasses[].name`), and each class has a matching `ateworker` herder image. The sandbox binaries themselves are not baked into the worker image. They come at runtime from the cluster-scoped [`SandboxConfig`](../reference/api-guide.md#3-sandboxconfig-the-sandbox-itself) the `WorkerPool` resolves for its class (an explicit `configRef`, else the class default). Each snapshot records the sandbox runtime it was taken on, and a memory restore runs on a worker that still offers that runtime, so restores stay reproducible across runtime upgrades.
346+
A `WorkerPool` selects a **sandbox class** (`spec.sandboxClasses[].name`), and each class has a matching `ateworker` image. The sandbox binaries themselves are not baked into the worker image. They come at runtime from the cluster-scoped [`SandboxConfig`](../reference/api-guide.md#3-sandboxconfig-the-sandbox-itself) the `WorkerPool` resolves for its class (an explicit `configRef`, else the class default). Each snapshot records the sandbox runtime it was taken on, and a memory restore runs on a worker that still offers that runtime, so restores stay reproducible across runtime upgrades.
347347

348348
* **gVisor** (`ateworker-gvisor`, the default): Runs the workload under `runsc` for kernel-level sandboxing. Suspend and resume leverage gVisor's native checkpoint/restore of the sandboxed process tree. `DurableDir` volumes and each container's rootfs overlay are saved as separate filesystem checkpoint bundles (`fs/` and `rootfs/`). A `MEMORY` snapshot splits them out of the process checkpoint; `VOLUMES` and `ROOTFS` snapshots save only the bundles they need with `runsc fscheckpoint`, without memory, and restore them on a cold boot.
349349

‎docs/get-started/overview.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -40,8 +40,8 @@ Substrate decouples the **Actor** (a stateful agent or workload instance) from t
4040
</div>
4141
<div class="as-diagram-node as-diagram-node--stack">
4242
<span class="as-diagram-node__tag">Pre-Warmed Worker Pod (WorkerPool)</span>
43-
<span class="mono as-diagram-node__title">Worker (ateom + Sandbox)</span>
44-
<span class="mono as-diagram-node__sub">ateom herder · atunnel mTLS · microVM / gVisor hosting Actor</span>
43+
<span class="mono as-diagram-node__title">Worker (ateworker + Sandbox)</span>
44+
<span class="mono as-diagram-node__sub">ateworker · atunnel mTLS · microVM / gVisor hosting Actor</span>
4545
</div>
4646
</div>
4747
<figcaption class="as-diagram-card__caption">
@@ -50,6 +50,6 @@ Substrate decouples the **Actor** (a stateful agent or workload instance) from t
5050
</figcaption>
5151
</figure>
5252

53-
* **Core Platform Terms at a Glance:** Each actor lives in an `atespace` (`<atespace>/<actor>`), traffic enters via `atenet`, node supervision and snapshot streaming are handled by `atelet`, and sandbox execution inside each warm `Worker` is managed by `ateom` (internally prefixed `ate` across binaries and CRDs). See the **[Glossary](../reference/glossary.md)** for complete definitions.
53+
* **Core Platform Terms at a Glance:** Each actor lives in an `atespace` (`<atespace>/<actor>`), traffic enters via `atenet-router`, node supervision and snapshot streaming are handled by `atelet`, and sandbox execution inside each warm `Worker` is managed by `ateworker` (internally prefixed `ate` across binaries and CRDs). See the **[Glossary](../reference/glossary.md)** for complete definitions.
5454
* **Framework Agnostic:** Runs standard OCI containers inside zero-trust sandboxes—supporting ADK, LangChain, Hermes, OpenClaw, MCP servers, and stateful coding environments out of the box.
55-
* **Ready to try it?** Jump straight to **[Quickstart](quickstart.md)** to launch your first actor locally on Kind, follow the **[GKE Installation Guide](https://docs.cloud.google.com/kubernetes-engine/ai-ml/install-overview-substrate)** to deploy on Google Kubernetes Engine, or explore the **[Architecture](../concepts/architecture.md)**.
55+
* **Ready to try it?** Jump straight to **[Quickstart](quickstart.md)** to launch your first actor locally on [kind](https://kind.sigs.k8s.io/), follow the **[GKE Installation Guide](https://docs.cloud.google.com/kubernetes-engine/ai-ml/install-overview-substrate)** to deploy on Google Kubernetes Engine, or explore the **[Architecture](../concepts/architecture.md)**.

‎docs/get-started/quickstart.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -5,25 +5,25 @@ weight: 2
55
aliases:
66
- "/docs/getting-started/"
77
description: >
8-
Deploy Agent Substrate on a local Kind cluster and invoke your first stateful actor.
8+
Deploy Agent Substrate on a local kind cluster and invoke your first stateful actor.
99
---
1010

1111
# Quickstart & Installation
1212

13-
Deploy the Agent Substrate control plane and the stateful **Counter** sample application on a local `kind` cluster.
13+
Deploy the Agent Substrate control plane and the stateful **Counter** sample application on a local [kind](https://kind.sigs.k8s.io/) cluster.
1414

1515
> [!NOTE]
16-
> **Prerequisites:** Ensure [Go](https://go.dev/doc/install), [`kubectl`](https://kubernetes.io/docs/tasks/tools/), and [Docker](https://www.docker.com/) are installed (`kind` is managed automatically via Go). A Linux host that will run more than about a thousand sandboxes also needs a larger kernel neighbor table (`net.ipv4.neigh.default.gc_thresh1/2/3` = 4096/16384/32768); `hack/create-kind-cluster.sh` warns when the host is at the default of 1024.
16+
> **Prerequisites:** Ensure [Go](https://go.dev/doc/install), [`kubectl`](https://kubernetes.io/docs/tasks/tools/), and [Docker](https://www.docker.com/) are installed ([`kind`](https://kind.sigs.k8s.io/) is managed automatically via Go). A Linux host that will run more than about a thousand sandboxes also needs a larger kernel neighbor table (`net.ipv4.neigh.default.gc_thresh1/2/3` = 4096/16384/32768); `hack/create-kind-cluster.sh` warns when the host is at the default of 1024.
1717
1818
## 1. Deploy Agent Substrate & Create an Actor
1919

2020
Run the following commands from the repository root to provision a local `kind` cluster, install Agent Substrate, and create a `counter` actor:
2121

2222
```bash
23-
# 1. Create a local Kind cluster and image registry (IPv4; IP_FAMILY=dual|ipv6 overrides)
23+
# 1. Create a local kind cluster and image registry (IPv4; IP_FAMILY=dual|ipv6 overrides)
2424
hack/create-kind-cluster.sh
2525

26-
# 2. Target the local Kind cluster in every shell that runs ate-setup
26+
# 2. Target the local kind cluster in every shell that runs ate-setup
2727
export ATE_INSTALL_KIND=true
2828

2929
# 3. Install Agent Substrate, PostgreSQL, and snapshot storage (rustfs)
@@ -66,7 +66,7 @@ data:
6666
OTEL_METRIC_EXPORT_TIMEOUT: "10000"
6767
OTEL_LOGS_EXPORTER: otlp
6868
---
69-
# ActorTemplate snapshotConfig rendered for Kind (local bucket)
69+
# ActorTemplate snapshotConfig rendered for kind (local bucket)
7070
snapshotConfig:
7171
onPause: SNAPSHOT_CONTENT_SCOPE_FULL
7272
onCommit: SNAPSHOT_CONTENT_SCOPE_FULL

‎docs/guides/authentication.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ title: "Authentication"
33
linkTitle: "Authentication"
44
weight: 3
55
description: >
6-
How clients authenticate to ate-api.
6+
How clients authenticate to ate-api-server over SPIFFE mTLS or OIDC JWTs.
77
---
88

99
# ate-api Authentication

‎docs/guides/authorization.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ title: "Authorization"
33
linkTitle: "Authorization"
44
weight: 4
55
description: >
6-
How ate-api checks every call against an OpenFGA model, and how to manage roles and access.
6+
How ate-api-server checks every call against an OpenFGA model, and how to manage roles and access.
77
---
88

99
# ate-api Authorization

‎docs/reference/api-guide.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ card_icon: "fa-solid fa-code"
66
aliases:
77
- "/docs/guides/api-guide/"
88
description: >
9-
Define WorkerPools and ActorTemplates, the workload blueprints of Agent Substrate.
9+
Define WorkerPools, ActorTemplates, and SandboxConfigs, and call the Control Plane gRPC API.
1010
---
1111

1212
# Substrate API Guide: WorkerPool & ActorTemplate
@@ -15,14 +15,14 @@ This guide explains how to configure Substrate resources to deploy high-density,
1515

1616
## 1. WorkerPool: The Physical Capacity
1717

18-
The `WorkerPool` defines the pool of physical "warm" compute capacity. It manages a fleet of standby pods (herders) that are ready to receive and execute actor states.
18+
The `WorkerPool` defines the pool of physical "warm" compute capacity. It manages a fleet of standby worker pods that are ready to receive and execute actor states.
1919

2020
### Specification (`WorkerPoolSpec`)
2121

2222
| Field | Type | Description |
2323
| :--- | :--- | :--- |
2424
| `replicas` | `int32` | **Required.** Number of physical standby pods to maintain in the cluster. |
25-
| `workerImage` | `string` | **Required.** The container image for the `ateworker` herder process (e.g. `ko://go.ate.dev/substrate/cmd/ateworker-gvisor`). |
25+
| `workerImage` | `string` | **Required.** The container image for the `ateworker` process (e.g. `ko://go.ate.dev/substrate/cmd/ateworker-gvisor`). |
2626
| `sandboxClasses` | `[]WorkerPoolSandboxClass` | **Required.** The sandbox runtime families the pool runs; exactly one entry today. Each entry has `name` (**required**, `gvisor` or `microvm`, immutable), which drives the worker pod shape (e.g. KVM device mounts, node placement), and `configRef.name` (optional), which names a cluster-scoped [`SandboxConfig`](#3-sandboxconfig-the-sandbox-itself) of the same class. Without `configRef`, the pool uses the class default (see [Class default](#class-default)). The controller records the result in `status.sandboxClasses`. The sandbox binaries an actor boots with come from the `SandboxConfig` the pool of its assigned worker resolved. |
2727
| `template` | `WorkerPoolPodTemplate` | **Optional.** Metadata, scheduling, and resource settings for worker workloads. |
2828

‎docs/reference/glossary.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ For how the pieces fit together, see the [Architecture](../concepts/architecture
2222
overall platform and runtime, while **`ate`** (originally *Agent Task /
2323
Execution Engine*) is the internal prefix used across Kubernetes namespaces
2424
(`ate-system`), control-plane binaries (`ate-api-server`, `atecontroller`),
25-
node and pod daemons (`atelet`, `ateom`), networking components (`atenet`,
25+
node and pod daemons (`atelet`, `ateworker`), networking components (`atenet`,
2626
`atunnel`), HTTP routing headers (`ate-target-actor`), and the `kubectl-ate`
2727
CLI plugin.
2828

‎hack/test-site-templates.sh‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -118,11 +118,23 @@ lacks main "${F}/override/index.html" 'as-status-banner--deprecated'
118118
lacks main "${F}/plain/index.html" 'as-status-banner'
119119
lacks main "${F}/index.html" 'as-status-banner'
120120

121-
# Copy as Markdown and llms-full.txt carry the status line.
121+
# Copy as Markdown, per-page index.md, and llms-full.txt carry unescaped
122+
# Markdown, preserve shell variables, and rewrite relative links.
122123
has main "${F}/experimental/index.html" '\*\*Status: Experimental\.\*\* This feature is experimental'
123-
has main llms-full.txt '(>|&gt;) \*\*Status: Deprecated\.\*\* Use the \[experimental page\]\(experimental\.md\) instead\.'
124+
has main "${F}/deprecated/index.md" '> \*\*Status: Deprecated\.\*\* Use the \[experimental page\]\(https://example\.org/docs/fixtures/experimental/index\.md\) instead\.'
125+
has main llms-full.txt '> \*\*Status: Deprecated\.\*\* Use the \[experimental page\]\(https://example\.org/docs/fixtures/experimental/index\.md\) instead\.'
126+
has main "${F}/plain/index.md" '[$]SHELL_VAR'
127+
has main llms-full.txt '[$]SHELL_VAR'
128+
lacks main "${F}/plain/index.md" '&gt;|&lt;|&#34;|&#39;'
129+
lacks main llms-full.txt '&gt;|&lt;|&#34;|&#39;'
124130
lacks main "${F}/plain/index.html" '\*\*Status:'
125131

132+
# llms.txt links to per-page Markdown files and prefixes non-GA statuses.
133+
has main llms.txt '\[Plain Page\]\(https://example\.org/docs/fixtures/plain/index\.md\):'
134+
has main llms.txt '\[Experimental Page\]\(https://example\.org/docs/fixtures/experimental/index\.md\): \[Status: Experimental\]'
135+
has main llms.txt '\[Deprecated Page\]\(https://example\.org/docs/fixtures/deprecated/index\.md\): \[Status: Deprecated\]'
136+
has main llms.txt '\[Planned Page\]\(https://example\.org/docs/fixtures/planned/index\.md\): \[Status: Planned\]'
137+
126138
# A status outside the allowed set fails the build and names the value.
127139
if build bogus testdata/hugo.bogus.yaml; then
128140
fail "bogus: build succeeded with status: experimantal"

‎site/README.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,21 @@ writes `data/release.json`.
7070
`hack/site.sh test` builds the site against `testdata/` and checks the rendered
7171
templates.
7272

73+
## Machine-readable documentation (`llms.txt`)
74+
75+
Hugo builds three plain-text outputs alongside the HTML pages:
76+
77+
- `/llms.txt` (`layouts/index.llms.txt`), a concise project overview and index
78+
linking to the Markdown build of each doc page and the repository's proto,
79+
CRD, and metric schemas;
80+
- `/llms-full.txt` (`layouts/index.llmsfull.txt`), a single-file compilation of
81+
every published doc page;
82+
- `/docs/<section>/<page>/index.md` (`layouts/docs/single.md`), the standalone
83+
Markdown version of each page, with relative links rewritten to absolute URLs.
84+
85+
All three share `layouts/_partials/raw-markdown.html` with the per-page
86+
"Copy as Markdown" button.
87+
7388
## Glossary hover cards
7489

7590
On doc pages, the first mention of each glossary term gets a hover card with

‎site/hugo.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,7 @@ outputFormats:
8787
outputs:
8888
home: [HTML, LLMS, LLMSFULL]
8989
section: [HTML]
90+
page: [HTML, MARKDOWN]
9091

9192
params:
9293
github_repo: "https://github.com/agent-substrate/substrate"

0 commit comments

Comments
 (0)