Skip to content

Commit b5489e5

Browse files
cvmaggiezha
andauthored
feat(examples): add a Fabric load client for GPU autoscaling runs (#14192)
## Failure The GPU autoscaling example has no way to load the chart's GPUs through the agents, which is what makes its autoscaler scale. #14009 had load clients, but they sent each prompt with `kubectl exec` into the sandbox pods, around the gateway's authentication, and read pod names and runtime IDs from NemoClaw's `terraform.tfstate`. With the `development` profile that example uses, nobody but NemoClaw can authenticate to the gateway, so no supported client could reach the agents. ## Decision Follows #14191 (merged), on which this was stacked. - `examples/kubernetes/gpu-autoscaling/load.sh`, a Bash script with the OpenShell CLI, `curl` and `jq`. It reads `nemoclaw apply -o json` and sends `PROMPTS` prompts across the agents, at most `CONCURRENCY` at a time through `xargs -P`, then prints how many it sent and how many replied, the median, 95th-percentile and longest latency overall and per sandbox, up to 10 failures, and one reply per sandbox. It exits nonzero if any request failed. Each sandbox takes one of the two paths that need no state files or `kubectl exec`: - **Hermes's API server**, when apply lists an `api` interface and an `api` service URL for the sandbox: OpenAI chat completions to `serviceUrls.SANDBOX.api`, through the gateway's port forward and with its client certificate. The script reads the interface token once, with `openshell sandbox exec … cat` and the reported `tokenPath`, into a private temporary file. - **`fabric-agent-ctl invoke`** for any other sandbox, such as OpenClaw or Deep Agents, through `openshell sandbox exec` with `agents.SANDBOX.runtimeId` (#14186). - `two-runtimes.yaml`: a Hermes sandbox on the chart's vLLM release and an OpenClaw sandbox on its Ollama release, the two that `install-two-runtimes.sh` installs (#14191). It uses the `oidc` profile, and declares Hermes's `api` interface with a `serviceExposures` entry using `BEARER_PASSTHROUGH`, because OpenShell's default `STRIP` drops the token Hermes checks. Its Envoy Service names are the ones Envoy Gateway v1.8.3 created for those Gateways on kind; it shortens a Gateway's name to 48 characters before the hash. - `README.md`: **Drive Load Through the Agents**, with the apply and run commands, the variables, and the limits: the gateway's one CPU for `sandbox exec`, Fabric's one invocation at a time per runtime, Hermes's 10 concurrent runs per sandbox, what a service URL skips, and that the CLI doesn't renew a client-credentials token (NVIDIA/OpenShell#4287). From #14009's `e2e-fabric-load-test.py` and client scripts, this keeps the prompts, the per-request timeout, the success check on `result.status` and `result.output.response`, and per-sandbox counts. It leaves out the HPA polling and the stop-at-ceiling logic, which belong to the chart and `kubectl get hpa --watch`; the OpenClaw tool pinning, which edited Fabric's generated configuration inside the sandbox; and the timeline and CSV output. ## Validation - `cargo ci` on `c6a581a9cf`: all selected steps passed in 135 s; test ran 1,827 tests and lifecycle 101. `cargo test -p nemoclaw-onboarding two_runtimes` ran the new file's onboarding journey. - `shellcheck` 0.11 (`koalaman/shellcheck:stable`) on `load.sh`: no findings. - Live run on a local kind cluster (kind v0.33.0, `kindest/node:v1.37.0`, Agent Sandbox 1.0.3, linux/arm64, no GPU), with resources I created and removed afterwards: - **Model:** the chart from nemoclaw-community `cb46db83ab` installed with `helm`, with Ollama and `qwen3:0.6b` on CPU, no GPU request or node selector, no ServiceMonitor or HPA, arm64 digests for Ollama 0.40.11 and `node:22-bookworm-slim`, behind Envoy Gateway v1.8.3. Both sandboxes used its one Envoy Service, with `model: qwen3:0.6b`. - **Issuer:** Keycloak 26.4 in the cluster, at `https://<kind node IP>:30443/realms/openshell` with a test CA, so that the gateway pod and the host see the same issuer URL, with a confidential client whose service account has the admin role in a `roles` claim and audience `openshell`. The kind HCL cases' issuer serves only discovery and keys, with no token endpoint, so the OpenShell CLI can't sign in through it. - **Deployment:** `two-runtimes.yaml` with the context, issuer, endpoints, model and image digests replaced, and `openclaw` and `hermes` built for linux/arm64 and pushed to a registry on the kind network. `nemoclaw apply -o json` with the bundle from this branch succeeded, and reported `serviceUrls.user-0000.api` as `https://nc-…--user-0000--api.openshell.localhost:8080/`, both runtime IDs under `agents`, and the Hermes token path under `interfaces`. - **CLI:** the port forward and client certificate from `docs/kubernetes.md`, then `openshell gateway add … --oidc-client-id` with `OPENSHELL_OIDC_CLIENT_SECRET` (OpenShell 0.1.6): `Authenticated via client credentials`; `openshell status` showed `Authenticated (OIDC)`. - **Load:** `PROMPTS=12 CONCURRENCY=4 load.sh apply.json`: ```text Sending 12 prompts to 2 sandboxes, at most 4 at a time: user-0000: Hermes API at https://nc-…--user-0000--api.openshell.localhost:8080/ user-0001: fabric-agent-ctl invoke, runtime runtime-1793794887745-82-2 Sent 12 prompts in 238 s: 12 replied, 0 failed. Latency of replies: p50 51.9 s, p95 156 s, max 156 s. user-0000 (api): 6/6 replied; p50 16.9 s, p95 87.6 s, max 87.6 s user-0001 (invoke): 6/6 replied; p50 80.7 s, p95 156 s, max 156 s reply #1 user-0000: A Kubernetes HorizontalPodAutoscaler is a service that automatically scales its pods based on metrics. reply #2 user-0001: The current system state shows no active exec sessions or subagents, indicating no ongoing operations are in progress. To proceed, please check your system or c ``` An earlier run of 8 prompts also got 8 replies. OpenClaw's replies don't answer the prompts, likely because of `qwen3:0.6b` with OpenClaw's tools; each invocation still reported `succeeded` with a response. - **Failures:** with `TIMEOUT=5`, both requests were reported as failed (`curl: (28) Operation timed out` and `timed out after 5 s`), and the script exited 1. - Not run: GPUs, autoscaling, vLLM, NIM, and Deep Agents. The Deep Agents path is the same `fabric-agent-ctl invoke` command as OpenClaw's. - Deferred to CI: `cargo ci live-kind` and `live-docker`; the change adds no image or Kubernetes code. Co-authored-by: maggiezha <[email protected]>
1 parent b5e4a9a commit b5489e5

3 files changed

Lines changed: 306 additions & 3 deletions

File tree

‎examples/kubernetes/gpu-autoscaling/README.md‎

Lines changed: 39 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ NemoClaw runs only the gateway and the sandboxes, which reach the chart's Servic
99

1010
[openclaw-ollama.yaml](openclaw-ollama.yaml) declares two OpenClaw sandboxes, `user-0000` and `user-0001`, that use the chart's Ollama runtime.
1111
The chart also serves vLLM and NVIDIA NIM; [Choose a Pairing](#choose-a-pairing) lists what to change for those.
12+
[two-runtimes.yaml](two-runtimes.yaml) declares a Hermes sandbox on the chart's vLLM runtime and an OpenClaw sandbox on its Ollama runtime, set up for [load.sh](load.sh), which sends prompts to the agents to load the GPUs.
1213
No automated test covers this procedure; [current limits](../../../docs/limits.md#platforms-models-and-placement) lists what else is untested.
1314

1415
## Prerequisites
@@ -17,7 +18,7 @@ No automated test covers this procedure; [current limits](../../../docs/limits.m
1718
- **For the chart:** NVIDIA GPU nodes with the NVIDIA GPU Operator and its DCGM exporter, Metrics Server, `helm` and `kubectl` with a context that can install cluster-wide charts, and `git`.
1819
The chart pins its Ollama and Node.js images to linux/amd64 digests, so its pods need AMD64 GPU nodes.
1920
The chart's [README](https://github.com/NVIDIA/nemoclaw-community/blob/cb46db83abcadd8e069213e854b6e560149f3cdd/examples/recipes/nvidia/kubernetes-gpu-autoscaling/README.md#install-details) describes its scheduling, storage and runtime options.
20-
- **`jq`** and the pinned OpenShell CLI, to invoke an agent.
21+
- **`jq`** and the pinned OpenShell CLI, to invoke an agent, and **`curl`**, for `load.sh`.
2122

2223
## Install the Inference Chart
2324

@@ -98,7 +99,7 @@ DCGM_NAMESPACE=gpu-operator MAX_REPLICAS=8 ./scripts/install-two-runtimes.sh
9899
It installs the release `nemoclaw-gpu-vllm` in the namespace of the same name, and `nemoclaw-gpu-ollama` likewise, each with its own autoscaler whose ceiling is `MAX_REPLICAS`.
99100
Both ceilings count the same GPUs, so a pod that doesn't fit stays Pending until the other runtime scales down.
100101
For each release, use its name as the namespace and release in [Find the Service and Key](#find-the-service-and-key): its Gateway is `<release>-metrics-proxy` and its key Secret is `<release>-metrics-proxy-inference-api`.
101-
In the configuration, declare one inference provider and one inference per release, each with that release's key variable and model, and point each sandbox's `agent.inferenceRef` at the inference for its pairing.
102+
In the configuration, declare one inference provider and one inference per release, each with that release's key variable and model, and point each sandbox's `agent.inferenceRef` at the inference for its pairing, as [two-runtimes.yaml](two-runtimes.yaml) does with `GPU_VLLM_API_KEY` and `GPU_OLLAMA_API_KEY`.
102103

103104
## Deploy the Sandboxes
104105

@@ -110,6 +111,8 @@ cargo images build --platform linux/amd64 openclaw
110111
cargo images export-metadata nc-fabric:openclaw --output .local/openclaw.metadata.json
111112
```
112113

114+
For [two-runtimes.yaml](two-runtimes.yaml), build and export `hermes` as well, and set `HERMES_IMAGE_METADATA` and `OPENCLAW_IMAGE_METADATA` to the two bundles instead of `NEMOCLAW_AGENT_IMAGE_METADATA`.
115+
113116
Copy the file into the deployment directory:
114117

115118
```sh
@@ -169,7 +172,40 @@ kubectl --namespace nemoclaw-gpu get hpa --watch
169172
The autoscaler scales on GPU utilization or on average request latency; the chart's `scripts/set-hpa-metric.sh gpu_utilization` or `scripts/set-hpa-metric.sh latency_avg` switches an installed release between them without changing its runtime or model.
170173
Set `RELEASE` and `NAMESPACE` for a release other than `nemoclaw-gpu`.
171174

172-
This directory has no load client to drive many agents at once.
175+
## Drive Load Through the Agents
176+
177+
[load.sh](load.sh) sends prompts to every agent of a deployment, a bounded number at a time, and reports each request's latency and errors.
178+
It reaches the agents only through the gateway, as the OpenShell CLI, so the deployment must use the `oidc` profile, as [two-runtimes.yaml](two-runtimes.yaml) does.
179+
It sends each sandbox's prompts one of two ways:
180+
181+
- **Hermes's API server**, for a sandbox whose apply result lists an `api` interface and a service URL for it: an OpenAI chat completion to that URL, with the interface's token.
182+
Declare both in the sandbox, as `user-0000` in `two-runtimes.yaml` does: `harness.settings.interfaces.api` with its `port`, and a `serviceExposures` entry named `api` for that port with `authorizationMode: BEARER_PASSTHROUGH`, so that the gateway passes the token on to Hermes.
183+
The script reads the token once, with `openshell sandbox exec … cat` and the `tokenPath` that apply reports.
184+
- **`fabric-agent-ctl invoke`**, for any other sandbox, such as OpenClaw or Deep Agents: through `openshell sandbox exec`, with the runtime ID that apply reports under `agents`.
185+
186+
Apply with JSON output and keep the result, from the deployment directory:
187+
188+
```sh
189+
nemoclaw apply -o json --progress off --non-interactive --state-dir state deployment.yaml > apply.json
190+
```
191+
192+
Then, as in [Invoke an Agent](#invoke-an-agent), keep the gateway's port forward running and select the gateway and workspace, and run the script with the number of prompts and the most to send at once:
193+
194+
```sh
195+
PROMPTS=40 CONCURRENCY=8 ../../examples/kubernetes/gpu-autoscaling/load.sh apply.json
196+
```
197+
198+
A service URL names the gateway's own port, `8080`, under `openshell.localhost`; the script connects to the port forward instead, at `GATEWAY_ADDRESS`, `127.0.0.1:17671` by default, and presents the client certificate from `MTLS_DIR`, by default the directory where you copied it for the OpenShell CLI.
199+
`PROMPT_FILE` replaces the built-in prompts with one prompt per line, `TIMEOUT` limits each request, 300 seconds by default, and `RESULTS` names a file for one JSON line per request.
200+
The script prints how many prompts it sent and how many replied, the median, 95th-percentile and longest latency of replies, overall and per sandbox, up to 10 failures, and one reply per sandbox.
201+
It exits nonzero if any request failed.
202+
A request that reaches `TIMEOUT` counts as failed; the script stops waiting for it but doesn't cancel the agent's run.
203+
204+
- Each apply that starts a new runtime changes its runtime ID and Hermes's token: apply again with `-o json`, then rerun the script.
205+
- Each `fabric-agent-ctl invoke` is one `openshell sandbox exec` through the gateway, which NemoClaw limits to one CPU ([limits](../../../docs/limits.md)), and Fabric runs one invocation at a time per runtime, so concurrent prompts to one such sandbox wait in the sandbox, not on the GPUs.
206+
- Hermes's API server runs at most 10 requests at once per sandbox by default and answers `429` beyond that.
207+
- A service URL skips the gateway's sign-in: anyone who can reach the gateway's listener with its client certificate can reach Hermes's API, which only its token protects; see [who can use an interface](../../../docs/interfaces.md#who-can-use-an-interface).
208+
- The OpenShell CLI doesn't renew a token it got through client credentials ([NVIDIA/OpenShell#4287](https://github.com/NVIDIA/OpenShell/issues/4287)); for a run longer than the token's lifetime, sign in again first.
173209

174210
## Remove the Deployment
175211

Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
#!/usr/bin/env bash
2+
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
3+
# SPDX-License-Identifier: Apache-2.0
4+
#
5+
# Send prompts to the agents of an applied deployment, a bounded number at a
6+
# time, and report each request's latency and errors. See README.md, "Drive
7+
# Load Through the Agents".
8+
#
9+
# Usage: load.sh APPLY_JSON
10+
#
11+
# APPLY_JSON is the result of `nemoclaw apply -o json`. For each sandbox in
12+
# its `agents`:
13+
# - a sandbox with an `api` interface and a service URL for it, Hermes's API
14+
# server, gets OpenAI chat completions at that URL, with the interface's
15+
# token, which this script reads once at start with `openshell sandbox exec`;
16+
# - any other sandbox gets `fabric-agent-ctl invoke` with its runtime ID,
17+
# through `openshell sandbox exec`.
18+
#
19+
# The OpenShell CLI must select the deployment's gateway and workspace
20+
# (OPENSHELL_GATEWAY and OPENSHELL_WORKSPACE).
21+
#
22+
# Environment:
23+
# PROMPTS prompts to send, spread over the sandboxes in turn (20)
24+
# CONCURRENCY most requests in flight at once (4)
25+
# TIMEOUT seconds each request may take (300)
26+
# PROMPT_FILE file with one prompt per line, used in turn (built-in prompts)
27+
# GATEWAY_ADDRESS host:port where this host reaches the gateway, such as its
28+
# port forward (127.0.0.1:17671); service URLs connect there
29+
# MTLS_DIR the gateway's client certificate: ca.crt, tls.crt, tls.key
30+
# (the OpenShell CLI's directory for OPENSHELL_GATEWAY)
31+
# RESULTS file to write one JSON line per request to (none)
32+
#
33+
# Adapted from scripts/e2e-fabric-load-test.py in NVIDIA/NemoClaw#14009 at
34+
# ac69c22: its prompts, timeout and success checks, without kubectl exec or
35+
# reading NemoClaw's state.
36+
37+
set -euo pipefail
38+
39+
if [[ "${1:-}" == "--request" ]]; then
40+
# One request, run by xargs below: $2 is its number, from 1. A failed
41+
# command is the request's error, not the script's.
42+
set +e
43+
n="$2"
44+
count="$(wc -l <"$LOAD_DIR/sandboxes")"
45+
read -r sandbox mode agent runtime url < <(sed -n "$(((n - 1) % count + 1))p" "$LOAD_DIR/sandboxes")
46+
prompts="$(wc -l <"$LOAD_DIR/prompts")"
47+
prompt="$(sed -n "$(((n - 1) % prompts + 1))p" "$LOAD_DIR/prompts")"
48+
out="$LOAD_DIR/out.$n"
49+
start="$(date +%s%N)"
50+
ok=false error="" reply=""
51+
if [[ "$mode" == api ]]; then
52+
code="$(jq -n --arg p "$prompt" '{model: "hermes-agent", messages: [{role: "user", content: $p}]}' |
53+
curl -sS --max-time "$TIMEOUT" --cacert "$MTLS_DIR/ca.crt" --cert "$MTLS_DIR/tls.crt" --key "$MTLS_DIR/tls.key" \
54+
--connect-to "::$GATEWAY_ADDRESS" -H @"$LOAD_DIR/$sandbox.header" -H 'Content-Type: application/json' \
55+
--data-binary @- -o "$out" -w '%{http_code}' "${url%/}/v1/chat/completions" 2>"$out.err")" || true
56+
reply="$(jq -r '.choices[0].message.content // empty' "$out" 2>/dev/null || true)"
57+
if [[ "$code" == 200 && -n "$reply" ]]; then
58+
ok=true
59+
else
60+
error="HTTP ${code:-none}: $(cat "$out.err" "$out" 2>/dev/null | head -c 300 | tr '\n' ' ')"
61+
fi
62+
else
63+
jq -n --arg p "$prompt" '$p' |
64+
timeout "$TIMEOUT" openshell sandbox exec -n "$sandbox" --no-tty -- \
65+
fabric-agent-ctl invoke --agent "$agent" --runtime-id "$runtime" --input - >"$out" 2>"$out.err"
66+
status=$?
67+
reply="$(jq -r 'select(.result.status == "succeeded") | .result.output.response // empty' "$out" 2>/dev/null || true)"
68+
if [[ -n "$reply" ]]; then
69+
ok=true
70+
else
71+
error="$(jq -r '[.status, .result.status, ((.error // .result.error // {}) | .code, .stage)] | map(select(. != null)) | join(" ")' "$out" 2>/dev/null ||
72+
head -c 300 "$out.err" | tr '\n' ' ')"
73+
[[ "$status" != 124 ]] || error="timed out after $TIMEOUT s"
74+
[[ -n "$error" ]] || error="exit $status: $(head -c 300 "$out.err" | tr '\n' ' ')"
75+
fi
76+
fi
77+
end="$(date +%s%N)"
78+
jq -cn --argjson n "$n" --arg sandbox "$sandbox" --arg path "$mode" --argjson ok "$ok" \
79+
--argjson seconds "$(((end - start) / 1000000))e-3" --arg error "$error" --arg reply "$reply" \
80+
'{n: $n, sandbox: $sandbox, path: $path, ok: $ok, seconds: $seconds, error: $error, reply: $reply}' >"$LOAD_DIR/result.$n"
81+
rm -f "$out" "$out.err"
82+
exit 0
83+
fi
84+
85+
apply_json="${1:?usage: load.sh APPLY_JSON}"
86+
export PROMPTS="${PROMPTS:-20}" CONCURRENCY="${CONCURRENCY:-4}" TIMEOUT="${TIMEOUT:-300}"
87+
export GATEWAY_ADDRESS="${GATEWAY_ADDRESS:-127.0.0.1:17671}"
88+
export MTLS_DIR="${MTLS_DIR:-${XDG_CONFIG_HOME:-$HOME/.config}/openshell/gateways/${OPENSHELL_GATEWAY:-}/mtls}"
89+
for command in openshell curl jq; do
90+
command -v "$command" >/dev/null || { echo "load.sh: $command is required" >&2; exit 2; }
91+
done
92+
[[ "$PROMPTS" =~ ^[1-9][0-9]*$ && "$CONCURRENCY" =~ ^[1-9][0-9]*$ && "$TIMEOUT" =~ ^[1-9][0-9]*$ ]] ||
93+
{ echo "load.sh: PROMPTS, CONCURRENCY and TIMEOUT must be positive integers" >&2; exit 2; }
94+
jq -e '.outcome == "succeeded"' "$apply_json" >/dev/null ||
95+
{ echo "load.sh: $apply_json is not a successful apply result" >&2; exit 2; }
96+
97+
LOAD_DIR="$(mktemp -d)"
98+
export LOAD_DIR
99+
trap 'rm -rf "$LOAD_DIR"' EXIT
100+
umask 077
101+
102+
# One line per sandbox with a runtime: name, path, agent, runtime ID, URL.
103+
jq -r '.agents // {} | to_entries[] | select(.value.runtimeId) | [.key, .value.agent, .value.runtimeId] | @tsv' "$apply_json" |
104+
while IFS=$'\t' read -r sandbox agent runtime; do
105+
url="$(jq -r --arg s "$sandbox" '.serviceUrls[$s].api // empty' "$apply_json")"
106+
token_path="$(jq -r --arg s "$sandbox" '.interfaces // [] | map(select(.sandbox == $s and .interface == "api"))[0].tokenPath // empty' "$apply_json")"
107+
if [[ -n "$url" && -n "$token_path" ]]; then
108+
# The adapter writes a new token each time the runtime starts.
109+
token="$(openshell sandbox exec -n "$sandbox" --no-tty -- cat "$token_path" </dev/null)" ||
110+
{ echo "load.sh: can't read $sandbox's API token" >&2; exit 1; }
111+
printf 'Authorization: Bearer %s\n' "$token" >"$LOAD_DIR/$sandbox.header"
112+
echo "$sandbox api $agent $runtime $url"
113+
else
114+
echo "$sandbox invoke $agent $runtime -"
115+
fi
116+
done >"$LOAD_DIR/sandboxes"
117+
[[ -s "$LOAD_DIR/sandboxes" ]] || { echo "load.sh: $apply_json lists no agent with a runtime" >&2; exit 1; }
118+
if grep -q ' api ' "$LOAD_DIR/sandboxes"; then
119+
for file in ca.crt tls.crt tls.key; do
120+
[[ -r "$MTLS_DIR/$file" ]] || { echo "load.sh: $MTLS_DIR/$file is missing; set MTLS_DIR" >&2; exit 2; }
121+
done
122+
fi
123+
if [[ -n "${PROMPT_FILE:-}" ]]; then
124+
grep -v '^[[:space:]]*$' "$PROMPT_FILE" >"$LOAD_DIR/prompts"
125+
else
126+
printf '%s\n' \
127+
"In one sentence, what is a Kubernetes HorizontalPodAutoscaler?" \
128+
"In one sentence, what does GPU utilization measure?" \
129+
"In one sentence, what is vLLM?" >"$LOAD_DIR/prompts"
130+
fi
131+
132+
echo "Sending $PROMPTS prompts to $(wc -l <"$LOAD_DIR/sandboxes") sandboxes, at most $CONCURRENCY at a time:"
133+
awk '{ printf " %s: %s\n", $1, ($2 == "api" ? "Hermes API at " $5 : "fabric-agent-ctl invoke, runtime " $4) }' "$LOAD_DIR/sandboxes"
134+
started="$(date +%s)"
135+
seq 1 "$PROMPTS" | xargs -P "$CONCURRENCY" -I{} "$0" --request {} || true
136+
elapsed="$(($(date +%s) - started))"
137+
138+
cat "$LOAD_DIR"/result.* | jq -s 'sort_by(.n)' >"$LOAD_DIR/results.json"
139+
[[ -z "${RESULTS:-}" ]] || jq -c '.[]' "$LOAD_DIR/results.json" >"$RESULTS"
140+
jq -r --argjson elapsed "$elapsed" '
141+
def pct(p): sort | .[(length * p | ceil) - 1];
142+
def stats: map(.seconds) | if length == 0 then "-" else
143+
"p50 \(pct(0.5) * 10 | round / 10) s, p95 \(pct(0.95) * 10 | round / 10) s, max \(max * 10 | round / 10) s" end;
144+
"",
145+
"Sent \(length) prompts in \($elapsed) s: \(map(select(.ok)) | length) replied, \(map(select(.ok | not)) | length) failed.",
146+
"Latency of replies: \(map(select(.ok)) | stats).",
147+
"",
148+
(group_by(.sandbox)[] | " \(.[0].sandbox) (\(.[0].path)): \(map(select(.ok)) | length)/\(length) replied; \(map(select(.ok)) | stats)"),
149+
"",
150+
(map(select(.ok | not))[:10][] | " failed #\(.n) \(.sandbox) after \(.seconds) s: \(.error[:200])"),
151+
(map(select(.ok)) | group_by(.sandbox)[] | .[0] | " reply #\(.n) \(.sandbox): \(.reply | gsub("\\s+"; " ")[:160])")
152+
' "$LOAD_DIR/results.json"
153+
jq -e 'all(.ok)' "$LOAD_DIR/results.json" >/dev/null
Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
# yaml-language-server: $schema=../../../schemas/nemoclaw-v1alpha1.schema.json
4+
# A Hermes sandbox on the chart's vLLM release and an OpenClaw sandbox on its
5+
# Ollama release, which the nemoclaw-community kubernetes-gpu-autoscaling
6+
# chart's install-two-runtimes.sh installs on shared GPUs. The gateway uses the
7+
# oidc profile so that you, and load.sh, can reach the agents through the
8+
# OpenShell CLI. Hermes serves its OpenAI-compatible API, published at a URL
9+
# the gateway serves. NemoClaw creates nothing for the chart; see README.md.
10+
# Replace the deployment UID, the context, the issuer settings, and the image
11+
# digests before applying. Set GPU_VLLM_API_KEY and GPU_OLLAMA_API_KEY to the
12+
# two releases' inference keys, and NEMOCLAW_GATEWAY_TOKEN to a token from your
13+
# issuer, where you run nemoclaw. Write each image's metadata bundle with
14+
# `cargo images export-metadata` and set HERMES_IMAGE_METADATA and
15+
# OPENCLAW_IMAGE_METADATA to their paths; see docs/build.md.
16+
# Adapted from hermes-vllm.yaml and openclaw-ollama.yaml in
17+
# NVIDIA/NemoClaw#14009 at ac69c22.
18+
apiVersion: nemoclaw.nvidia.com/v1alpha1
19+
kind: NemoClawConfig
20+
metadata:
21+
name: kubernetes-gpu-autoscaling-two-runtimes
22+
uid: 7a1c3e52-9d4b-4f86-b0e2-3c5d8a6f1b94
23+
spec:
24+
gateway:
25+
management: managed
26+
runtime:
27+
provider: kubernetes
28+
endpoint: https://127.0.0.1:17671
29+
kubernetes:
30+
kubeconfig: {env: NEMOCLAW_CLUSTER_KUBECONFIG}
31+
context: replace-with-explicit-context
32+
namespace: nemoclaw-gpu-autoscaling
33+
authentication:
34+
profile: oidc
35+
# A ConfigMap in the namespace whose ca.crt verifies the issuer's
36+
# certificate; omit it for a publicly trusted issuer.
37+
caConfigMap: issuer-ca
38+
token: {env: NEMOCLAW_GATEWAY_TOKEN}
39+
config:
40+
openshell.gateway.oidc:
41+
issuer: https://login.example.com/realms/openshell
42+
audience: openshell
43+
roles_claim: groups
44+
admin_role: openshell-admins
45+
user_role: openshell-users
46+
inferenceProviders:
47+
# The Envoy Services for the two releases' Gateways. Envoy Gateway
48+
# shortens a long Gateway name before its hash; README.md shows how to
49+
# find each Service.
50+
gpu-vllm:
51+
provider: openai
52+
api: openai-completions
53+
endpoint: http://envoy-nemoclaw-gpu-vllm-nemoclaw-gpu-vllm-metrics-prox-62f61e15.envoy-gateway-system.svc.cluster.local:80/v1
54+
credential: {env: GPU_VLLM_API_KEY}
55+
gpu-ollama:
56+
provider: openai
57+
api: openai-completions
58+
endpoint: http://envoy-nemoclaw-gpu-ollama-nemoclaw-gpu-ollama-metrics-f83d8e1f.envoy-gateway-system.svc.cluster.local:80/v1
59+
credential: {env: GPU_OLLAMA_API_KEY}
60+
inferences:
61+
hermes:
62+
routes:
63+
- name: primary
64+
providerRef: gpu-vllm
65+
overrides:
66+
# The model each release serves; its inference.model value.
67+
model: nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8
68+
maxTokens: 2048
69+
openclaw:
70+
routes:
71+
- name: primary
72+
providerRef: gpu-ollama
73+
overrides:
74+
model: llama3.2:3b
75+
# OpenClaw otherwise requests more output tokens than the context allows.
76+
maxTokens: 2048
77+
sandboxes:
78+
user-0000:
79+
harness:
80+
kind: nvidia.fabric.hermes
81+
execution:
82+
timeoutSeconds: 300
83+
settings:
84+
interfaces:
85+
# Hermes's OpenAI-compatible API server, on loopback in the sandbox.
86+
api:
87+
port: 8642
88+
# Publishes the API server at a URL the gateway serves. Hermes checks
89+
# the bearer token itself, so the gateway passes it through.
90+
serviceExposures:
91+
api:
92+
targetPort: 8642
93+
authorizationMode: BEARER_PASSTHROUGH
94+
image:
95+
metadata: {env: HERMES_IMAGE_METADATA}
96+
ref: registry.example.com/nemoclaw/hermes@sha256:0000000000000000000000000000000000000000000000000000000000000000
97+
network:
98+
tier: isolated
99+
agent:
100+
name: assistant
101+
inferenceRef: hermes
102+
user-0001:
103+
harness:
104+
kind: nvidia.fabric.openclaw
105+
execution:
106+
timeoutSeconds: 300
107+
image:
108+
metadata: {env: OPENCLAW_IMAGE_METADATA}
109+
ref: registry.example.com/nemoclaw/openclaw@sha256:0000000000000000000000000000000000000000000000000000000000000000
110+
network:
111+
tier: isolated
112+
agent:
113+
name: assistant
114+
inferenceRef: openclaw

0 commit comments

Comments
 (0)