Repository navigation
Commit b5489e5
feat(examples): add a Fabric load client for GPU autoscaling runs (#14192)
## Failure
The GPU autoscaling example has no way to load the chart's GPUs through
the agents, which is what makes its autoscaler scale.
#14009 had load clients, but they sent each prompt with `kubectl exec`
into the sandbox pods, around the gateway's authentication, and read pod
names and runtime IDs from NemoClaw's `terraform.tfstate`.
With the `development` profile that example uses, nobody but NemoClaw
can authenticate to the gateway, so no supported client could reach the
agents.
## Decision
Follows #14191 (merged), on which this was stacked.
- `examples/kubernetes/gpu-autoscaling/load.sh`, a Bash script with the
OpenShell CLI, `curl` and `jq`. It reads `nemoclaw apply -o json` and
sends `PROMPTS` prompts across the agents, at most `CONCURRENCY` at a
time through `xargs -P`, then prints how many it sent and how many
replied, the median, 95th-percentile and longest latency overall and per
sandbox, up to 10 failures, and one reply per sandbox. It exits nonzero
if any request failed. Each sandbox takes one of the two paths that need
no state files or `kubectl exec`:
- **Hermes's API server**, when apply lists an `api` interface and an
`api` service URL for the sandbox: OpenAI chat completions to
`serviceUrls.SANDBOX.api`, through the gateway's port forward and with
its client certificate. The script reads the interface token once, with
`openshell sandbox exec … cat` and the reported `tokenPath`, into a
private temporary file.
- **`fabric-agent-ctl invoke`** for any other sandbox, such as OpenClaw
or Deep Agents, through `openshell sandbox exec` with
`agents.SANDBOX.runtimeId` (#14186).
- `two-runtimes.yaml`: a Hermes sandbox on the chart's vLLM release and
an OpenClaw sandbox on its Ollama release, the two that
`install-two-runtimes.sh` installs (#14191). It uses the `oidc` profile,
and declares Hermes's `api` interface with a `serviceExposures` entry
using `BEARER_PASSTHROUGH`, because OpenShell's default `STRIP` drops
the token Hermes checks. Its Envoy Service names are the ones Envoy
Gateway v1.8.3 created for those Gateways on kind; it shortens a
Gateway's name to 48 characters before the hash.
- `README.md`: **Drive Load Through the Agents**, with the apply and run
commands, the variables, and the limits: the gateway's one CPU for
`sandbox exec`, Fabric's one invocation at a time per runtime, Hermes's
10 concurrent runs per sandbox, what a service URL skips, and that the
CLI doesn't renew a client-credentials token (NVIDIA/OpenShell#4287).
From #14009's `e2e-fabric-load-test.py` and client scripts, this keeps
the prompts, the per-request timeout, the success check on
`result.status` and `result.output.response`, and per-sandbox counts. It
leaves out the HPA polling and the stop-at-ceiling logic, which belong
to the chart and `kubectl get hpa --watch`; the OpenClaw tool pinning,
which edited Fabric's generated configuration inside the sandbox; and
the timeline and CSV output.
## Validation
- `cargo ci` on `c6a581a9cf`: all selected steps passed in 135 s; test
ran 1,827 tests and lifecycle 101. `cargo test -p nemoclaw-onboarding
two_runtimes` ran the new file's onboarding journey.
- `shellcheck` 0.11 (`koalaman/shellcheck:stable`) on `load.sh`: no
findings.
- Live run on a local kind cluster (kind v0.33.0,
`kindest/node:v1.37.0`, Agent Sandbox 1.0.3, linux/arm64, no GPU), with
resources I created and removed afterwards:
- **Model:** the chart from nemoclaw-community `cb46db83ab` installed
with `helm`, with Ollama and `qwen3:0.6b` on CPU, no GPU request or node
selector, no ServiceMonitor or HPA, arm64 digests for Ollama 0.40.11 and
`node:22-bookworm-slim`, behind Envoy Gateway v1.8.3. Both sandboxes
used its one Envoy Service, with `model: qwen3:0.6b`.
- **Issuer:** Keycloak 26.4 in the cluster, at `https://<kind node
IP>:30443/realms/openshell` with a test CA, so that the gateway pod and
the host see the same issuer URL, with a confidential client whose
service account has the admin role in a `roles` claim and audience
`openshell`. The kind HCL cases' issuer serves only discovery and keys,
with no token endpoint, so the OpenShell CLI can't sign in through it.
- **Deployment:** `two-runtimes.yaml` with the context, issuer,
endpoints, model and image digests replaced, and `openclaw` and `hermes`
built for linux/arm64 and pushed to a registry on the kind network.
`nemoclaw apply -o json` with the bundle from this branch succeeded, and
reported `serviceUrls.user-0000.api` as
`https://nc-…--user-0000--api.openshell.localhost:8080/`, both runtime
IDs under `agents`, and the Hermes token path under `interfaces`.
- **CLI:** the port forward and client certificate from
`docs/kubernetes.md`, then `openshell gateway add … --oidc-client-id`
with `OPENSHELL_OIDC_CLIENT_SECRET` (OpenShell 0.1.6): `Authenticated
via client credentials`; `openshell status` showed `Authenticated
(OIDC)`.
- **Load:** `PROMPTS=12 CONCURRENCY=4 load.sh apply.json`:
```text
Sending 12 prompts to 2 sandboxes, at most 4 at a time:
user-0000: Hermes API at
https://nc-…--user-0000--api.openshell.localhost:8080/
user-0001: fabric-agent-ctl invoke, runtime runtime-1793794887745-82-2
Sent 12 prompts in 238 s: 12 replied, 0 failed.
Latency of replies: p50 51.9 s, p95 156 s, max 156 s.
user-0000 (api): 6/6 replied; p50 16.9 s, p95 87.6 s, max 87.6 s
user-0001 (invoke): 6/6 replied; p50 80.7 s, p95 156 s, max 156 s
reply #1 user-0000: A Kubernetes HorizontalPodAutoscaler is a service
that automatically scales its pods based on metrics.
reply #2 user-0001: The current system state shows no active exec
sessions or subagents, indicating no ongoing operations are in progress.
To proceed, please check your system or c
```
An earlier run of 8 prompts also got 8 replies. OpenClaw's replies don't
answer the prompts, likely because of `qwen3:0.6b` with OpenClaw's
tools; each invocation still reported `succeeded` with a response.
- **Failures:** with `TIMEOUT=5`, both requests were reported as failed
(`curl: (28) Operation timed out` and `timed out after 5 s`), and the
script exited 1.
- Not run: GPUs, autoscaling, vLLM, NIM, and Deep Agents. The Deep
Agents path is the same `fabric-agent-ctl invoke` command as OpenClaw's.
- Deferred to CI: `cargo ci live-kind` and `live-docker`; the change
adds no image or Kubernetes code.
Co-authored-by: maggiezha <[email protected]>1 parent b5e4a9a commit b5489e5
3 files changed
Lines changed: 306 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
| 12 | + | |
12 | 13 | | |
13 | 14 | | |
14 | 15 | | |
| |||
17 | 18 | | |
18 | 19 | | |
19 | 20 | | |
20 | | - | |
| 21 | + | |
21 | 22 | | |
22 | 23 | | |
23 | 24 | | |
| |||
98 | 99 | | |
99 | 100 | | |
100 | 101 | | |
101 | | - | |
| 102 | + | |
102 | 103 | | |
103 | 104 | | |
104 | 105 | | |
| |||
110 | 111 | | |
111 | 112 | | |
112 | 113 | | |
| 114 | + | |
| 115 | + | |
113 | 116 | | |
114 | 117 | | |
115 | 118 | | |
| |||
169 | 172 | | |
170 | 173 | | |
171 | 174 | | |
172 | | - | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
173 | 209 | | |
174 | 210 | | |
175 | 211 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
0 commit comments