Skip to content

Commit c1b50e2

Browse files
committed
feat(e2e): add a chaos experiment harness
Add e2e/chaos, a harness that runs declarative resilience experiments against an OpenShell gateway and decides a verdict from recorded evidence: PASS, XFAIL, FAIL or ERROR. An experiment is one YAML file under e2e/chaos/experiments/ (apiVersion chaos/v1alpha1): the workloads to run as sandboxes, the operations to apply (kill, restart, pause, stop and start, policy changes), a flow that orders them in time, the metrics it is about, and per-instance and experiment-level checks, including bounds on scraped metrics. An expectations block ties checks to a GitHub issue, so a known bug ends the run in XFAIL and its fix shows up as XPASS. An adapter is a second YAML file under e2e/chaos/adapters/ that binds the harness to one system on one backend: the hooks that create, read, collect and delete a sandbox, the processes and containers or pods the built-in faults act on, adapter operations with their revert hooks, and what to observe. openshell-docker runs under e2e/with-docker-gateway.sh and is the one every result below came from. openshell-kubernetes declares the operations it cannot do and validates against every experiment, but has not been run against a cluster. A run writes e2e/chaos/runs/<experiment>/<run id>/ with every input, a timeline of state changes and operations, gateway and sandbox logs, scraped gateway metrics, the files each sandbox wrote and the verdict. From a run, chaos report draws an HTML report, chaos share writes a Markdown page with Mermaid charts for an issue or a post, chaos export writes the metrics as OpenMetrics, chaos dashboard builds the experiment's Grafana dashboard, and chaos evaluate decides the verdict again offline. grafana/ holds the Prometheus and Grafana compose stack. Nine experiments are included, all run on a local Docker gateway: - hello-world, smoke: harness sanity checks. PASS. - gateway-graceful-restart: SIGTERM the gateway and start it; every sandbox restarts and each counter resumes from its saved tick. PASS. - supervisor-pause: SIGSTOP every supervisor for 10 s; workloads with no network use run through untouched. PASS. - gateway-kill-keeps-state, policy-tighten-survives-gateway-restart: SIGKILL the gateway; the running sandboxes stop and never record an exit code. XFAIL, NVIDIA#4910. - sandbox-container-restart: docker restart one sandbox container; the sandbox never returns to Ready. XFAIL, NVIDIA#4828. - sandbox-stop-start: stop and start sandboxes for 25 minutes, timed against reconcile sweeps; one start ends StartFailed. XFAIL, NVIDIA#4981. - supervisor-kill: SIGKILL two supervisors; the sandboxes go to Error and stop, as documented. FAIL until the intended end state is settled. Outside e2e/chaos: e2e/with-docker-gateway.sh gains a metrics port and an opt-in OTLP endpoint (OPENSHELL_E2E_GATEWAY_OTLP=1), tasks/test.toml gains the e2e:chaos, e2e:chaos:preflight, e2e:chaos:check, e2e:chaos:share and e2e:chaos:grafana tasks, and CONTRIBUTING.md links the directory. Nothing runs in CI; e2e:chaos:check runs hello-world and smoke against a Docker gateway and fails unless both pass. Docs: e2e/chaos/README.md (overview, walkthrough, the run lifecycle, results, how to write an experiment), REFERENCE.md (the experiment and adapter schemas, CLI, run directory, timeline events, verdicts) and ADAPTER.md (how to write an adapter), with report, share and Grafana screenshots under images/. Signed-off-by: Murilo Pereira <[email protected]>
1 parent 9f1cdb4 commit c1b50e2

105 files changed

Lines changed: 17792 additions & 4 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CONTRIBUTING.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -386,6 +386,7 @@ These are the primary `mise` tasks for day-to-day development:
386386
| `proto/` | Protocol buffer definitions and [public API conventions](proto/README.md) |
387387
| `tasks/` | `mise` task definitions and build scripts |
388388
| `tests/testkit/` | Support library for SDK-driven end-to-end suites |
389+
| `e2e/chaos/` | [Chaos experiments](e2e/chaos/README.md) against e2e gateways |
389390
| `deploy/` | Dockerfiles, Helm chart, Kubernetes manifests |
390391
| `docs/` | Published Fern docs source, navigation, and content assets |
391392
| `fern/` | Fern site config, components, and theme assets |

‎e2e/chaos/.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
runs/
2+
.venv/

‎e2e/chaos/ADAPTER.md‎

Lines changed: 417 additions & 0 deletions
Large diffs are not rendered by default.

‎e2e/chaos/README.md‎

Lines changed: 1380 additions & 0 deletions
Large diffs are not rendered by default.

‎e2e/chaos/REFERENCE.md‎

Lines changed: 646 additions & 0 deletions
Large diffs are not rendered by default.

‎e2e/chaos/adapters/openshell-docker.yaml‎

Lines changed: 540 additions & 0 deletions
Large diffs are not rendered by default.

‎e2e/chaos/adapters/openshell-kubernetes.yaml‎

Lines changed: 491 additions & 0 deletions
Large diffs are not rendered by default.
Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
apiVersion: chaos/v1alpha1
5+
id: gateway-graceful-restart
6+
version: 2
7+
title: Counter instances resume after a graceful gateway restart
8+
hypothesis: >
9+
When the gateway process gets SIGTERM and is started again, its
10+
graceful-stop sweep stops the sandbox of every counter instance, the
11+
restarted gateway starts each sandbox again, and each instance's script
12+
resumes from its saved tick with no tick missing from its journal.
13+
description: >
14+
Two counter instances run. 15s after both have been observed running,
15+
gateway-restart stops and restarts the gateway. The checks encode current
16+
behaviour, observed on 2026-10-27: each instance's script is started a
17+
second time (the wrapper records two attempts), and the instance output
18+
directory keeps its files across the restart.
19+
default-adapter: openshell-docker
20+
timeout: 15m
21+
22+
workloads:
23+
- name: counter
24+
description: >
25+
Counts from 1 to $TICKS, one tick every $INTERVAL seconds. Each tick
26+
appends the line "<tick> <unix time>" to the instance's journal, then
27+
saves the tick number atomically. If the script is started again in the
28+
same output directory, it continues from the saved tick plus one.
29+
kind: shell
30+
instances: 2
31+
timeout: 6m
32+
env: {TICKS: "90", INTERVAL: "1"}
33+
run: |
34+
out=$CHAOS_EXPERIMENT_WORKLOAD_INSTANCE_OUTPUT_DIR
35+
start=$(cat "$out/counter" 2>/dev/null || echo 0)
36+
for i in $(seq $((start + 1)) "$TICKS"); do
37+
echo "$i $(date +%s)" >> "$out/journal"
38+
echo "$i" > "$out/counter.tmp" && mv "$out/counter.tmp" "$out/counter"
39+
sleep "$INTERVAL"
40+
done
41+
checks:
42+
- name: exited-ok
43+
description: >
44+
For each counter instance: the last attempt of the script exited
45+
with code 0. An instance with no recorded exit code fails.
46+
kind: exit-code
47+
equals: 0
48+
- name: journal-complete
49+
description: >
50+
For each counter instance: the journal has every tick from 1 to
51+
$TICKS. A tick may appear twice, because a script that is started
52+
again can repeat the tick the previous attempt was stopped on. A
53+
missing tick fails.
54+
kind: shell
55+
run: |
56+
journal=$CHAOS_EXPERIMENT_WORKLOAD_INSTANCE_OUTPUT_DIR/journal
57+
if ! missing=$(diff <(seq 1 "$TICKS") <(cut -d" " -f1 "$journal" | sort -un)); then
58+
echo "$missing" | head -5
59+
echo "want each tick from 1 to $TICKS at least once"
60+
exit 1
61+
fi
62+
- name: restarted
63+
description: >
64+
For each counter instance: .chaos/attempts has at least 2 lines, so
65+
the script was started again after the gateway restart.
66+
kind: shell
67+
run: |
68+
n=$(wc -l < "$CHAOS_EXPERIMENT_WORKLOAD_INSTANCE_OUTPUT_DIR/.chaos/attempts")
69+
[ "$n" -ge 2 ] || { echo "want >= 2 attempts, got $n"; exit 1; }
70+
71+
operations:
72+
- name: gateway-restart
73+
description: >
74+
Sends SIGTERM to the gateway process and, if it has not exited after
75+
60s, SIGKILL. Then runs the adapter's start hook, which waits until the
76+
gateway's /healthz returns 200, and records target.restarted.
77+
uses: restart
78+
target: gateway
79+
select: one
80+
grace-period: 60s
81+
82+
flow:
83+
parallel:
84+
- counter
85+
- serial: [{wait: {workload: counter, condition: started}}, {delay: 15s}, gateway-restart]
86+
87+
checks:
88+
- name: all-supervisors-connected
89+
description: >
90+
At some scrape all 2 counter instances had their supervisor session
91+
registered on the gateway, so the metric saw the full workload.
92+
kind: metric
93+
metric: openshell_server_supervisor_sessions
94+
value: max
95+
at-least: 2
96+
- name: database-healthy-after-restart
97+
description: >
98+
The gateway's database readiness probe passed in the last scrape of the
99+
run, after the restart.
100+
kind: metric
101+
metric: openshell_server_readiness_database_healthy
102+
value: last
103+
equals: 1
104+
105+
metrics:
106+
- family: openshell_server_supervisor_sessions
107+
description: >
108+
Supervisor sessions registered on the gateway: one per sandbox whose
109+
supervisor is connected. It follows the instance count while sandboxes
110+
run, and drops while a sandbox is restarting.
111+
- family: openshell_server_readiness_database_healthy
112+
description: >
113+
1 while the gateway's database probe passes. The gateway restarts with
114+
its database; a 0 after the restart would mean the gateway came back
115+
before its state did.
116+
- family: openshell_supervisor_policy_apply_results_total
117+
description: >
118+
Policy applies the supervisors reported, by outcome. A sandbox that
119+
reconnects to the restarted gateway applies its policy again, so the
120+
counter steps once per surviving sandbox.
Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
apiVersion: chaos/v1alpha1
5+
id: gateway-kill-keeps-state
6+
version: 3
7+
title: Sandboxes survive a hard gateway kill
8+
hypothesis: >
9+
After the gateway process gets SIGKILL, stays stopped for 15s and is started
10+
again, each counter instance's script runs to completion without being
11+
started again, and a sandbox created after every counter instance has ended
12+
runs its script to exit code 0.
13+
description: >
14+
Three counter instances run with restart-policy never. 20s after all three
15+
have been observed running, gateway-down kills the gateway. After every
16+
counter instance has ended, one fresh-sandbox instance is created, to show
17+
that the restarted gateway can create new sandboxes and not only serve
18+
existing ones. The checks encode desired behaviour.
19+
default-adapter: openshell-docker
20+
timeout: 15m
21+
expectations:
22+
- issue: NVIDIA/OpenShell#4910
23+
# The restarted gateway replaces each live supervisor, the sandbox runtime
24+
# rejects the replacement, and the driver stops the sandbox container.
25+
checks: [counter/exited-ok, counter/journal-complete]
26+
27+
workloads:
28+
- name: counter
29+
description: >
30+
Counts from 1 to $TICKS, one tick every $INTERVAL seconds. Each tick
31+
appends the line "<tick> <unix time>" to the instance's journal, then
32+
saves the tick number atomically. If the script is started again in the
33+
same output directory, it continues from the saved tick plus one.
34+
kind: shell
35+
instances: 3
36+
timeout: 5m
37+
env: {TICKS: "120", INTERVAL: "1"}
38+
options: {restart-policy: never}
39+
run: |
40+
out=$CHAOS_EXPERIMENT_WORKLOAD_INSTANCE_OUTPUT_DIR
41+
start=$(cat "$out/counter" 2>/dev/null || echo 0)
42+
for i in $(seq $((start + 1)) "$TICKS"); do
43+
echo "$i $(date +%s)" >> "$out/journal"
44+
echo "$i" > "$out/counter.tmp" && mv "$out/counter.tmp" "$out/counter"
45+
sleep "$INTERVAL"
46+
done
47+
checks:
48+
- name: exited-ok
49+
description: >
50+
For each counter instance: the last attempt of the script exited
51+
with code 0. An instance with no recorded exit code fails.
52+
kind: exit-code
53+
equals: 0
54+
- name: journal-complete
55+
description: >
56+
For each counter instance: the journal has the ticks 1 to $TICKS,
57+
each once and in order, so no tick was skipped or repeated.
58+
kind: shell
59+
run: |
60+
journal=$CHAOS_EXPERIMENT_WORKLOAD_INSTANCE_OUTPUT_DIR/journal
61+
if ! diff=$(diff <(seq 1 "$TICKS") <(cut -d" " -f1 "$journal")); then
62+
echo "$diff" | head -5
63+
echo "want the ticks 1 to $TICKS, each once, in order"
64+
exit 1
65+
fi
66+
- name: started-once
67+
description: >
68+
For each counter instance: .chaos/attempts has exactly 1 line, so
69+
the script ran once and was not started again.
70+
kind: shell
71+
run: |
72+
n=$(wc -l < "$CHAOS_EXPERIMENT_WORKLOAD_INSTANCE_OUTPUT_DIR/.chaos/attempts")
73+
[ "$n" -eq 1 ] || { echo "want 1 attempt, got $n"; exit 1; }
74+
75+
- name: fresh-sandbox
76+
description: >
77+
One instance, created after every counter instance has ended. Its
78+
script writes one file and exits 0.
79+
kind: shell
80+
instances: 1
81+
timeout: 1m
82+
run: echo ok > "$CHAOS_EXPERIMENT_WORKLOAD_INSTANCE_OUTPUT_DIR/hello"
83+
checks:
84+
- name: exited-ok
85+
description: >
86+
For the fresh-sandbox instance: the last attempt of the script exited
87+
with code 0. An instance with no recorded exit code fails.
88+
kind: exit-code
89+
equals: 0
90+
91+
operations:
92+
- name: gateway-down
93+
description: >
94+
Sends SIGKILL to the gateway process and keeps it stopped for 15s. Then
95+
runs the adapter's start hook, which waits up to 120s until the
96+
gateway's /healthz returns 200, and records operation.reverted. The gateway
97+
is unavailable for longer than 15s.
98+
uses: outage
99+
target: gateway
100+
select: one
101+
duration: 15s
102+
103+
flow:
104+
serial:
105+
- parallel:
106+
- counter
107+
- serial: [{wait: {workload: counter, condition: started}}, {delay: 20s}, gateway-down]
108+
- fresh-sandbox
109+
110+
checks:
111+
- name: gateway-recovered
112+
description: >
113+
The timeline has an operation.reverted event for gateway-down. The harness
114+
records it only after the restarted gateway answered /healthz with 200.
115+
kind: shell
116+
env: {OPERATION: gateway-down}
117+
run: |
118+
# Slurp: jq 1.6 -e only looks at the last line of a stream.
119+
jq -se --arg o "$OPERATION" 'any(.[]; .event == "operation.reverted" and .operation == $o)' \
120+
"$CHAOS_EXPERIMENT_RUN_DIR/timeline.jsonl" > /dev/null \
121+
|| { echo "$OPERATION was never reverted"; exit 1; }
122+
- name: database-healthy-after-restart
123+
description: >
124+
The gateway's database readiness probe passed in the last scrape of the
125+
run, after the restart.
126+
kind: metric
127+
metric: openshell_server_readiness_database_healthy
128+
value: last
129+
equals: 1
130+
131+
metrics:
132+
- family: openshell_server_supervisor_sessions
133+
description: >
134+
Supervisor sessions registered on the gateway: one per sandbox whose
135+
supervisor is connected. It follows the instance count while sandboxes
136+
run, and drops while a sandbox is restarting.
137+
- family: openshell_server_readiness_database_healthy
138+
description: >
139+
1 while the gateway's database probe passes. The gateway restarts with
140+
its database; a 0 after the restart would mean the gateway came back
141+
before its state did.
142+
- family: openshell_supervisor_policy_apply_results_total
143+
description: >
144+
Policy applies the supervisors reported, by outcome. A sandbox that
145+
reconnects to the restarted gateway applies its policy again, so the
146+
counter steps once per surviving sandbox.
Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
apiVersion: chaos/v1alpha1
5+
id: hello-world
6+
version: 1
7+
title: A sandbox says hello
8+
hypothesis: >
9+
One hello instance runs its script to exit code 0, and the file greeting in
10+
its output directory holds the line "hello, world".
11+
description: >
12+
The smallest experiment: one workload with one instance, two checks and no
13+
faults. The README walks through it. A FAIL or ERROR points at the harness,
14+
the adapter hooks or the environment. The checks encode desired behaviour.
15+
default-adapter: openshell-docker
16+
timeout: 5m
17+
18+
workloads:
19+
- name: hello
20+
description: Writes the line "hello, world" to the file greeting in its output directory.
21+
kind: shell
22+
instances: 1
23+
timeout: 2m
24+
run: |
25+
echo "hello, world" > "$CHAOS_EXPERIMENT_WORKLOAD_INSTANCE_OUTPUT_DIR/greeting"
26+
checks:
27+
- name: exited-ok
28+
description: "For each hello instance: the script exited with code 0."
29+
kind: exit-code
30+
equals: 0
31+
- name: said-hello
32+
description: 'For each hello instance: the file greeting holds the line "hello, world".'
33+
kind: shell
34+
run: |
35+
grep -qx "hello, world" "$CHAOS_EXPERIMENT_WORKLOAD_INSTANCE_OUTPUT_DIR/greeting"
36+
37+
flow: hello
38+
39+
metrics:
40+
- family: openshell_server_supervisor_sessions
41+
description: >
42+
Supervisor sessions registered on the gateway: one per sandbox whose
43+
supervisor is connected. It follows the instance count while sandboxes
44+
run, and drops while a sandbox is restarting.
45+
- family: openshell_server_grpc_requests_total
46+
description: >
47+
gRPC requests the gateway handled, by method and status. The harness's
48+
own calls (create, status, delete) are in it, so the rate shows the load
49+
the experiment puts on the gateway.

0 commit comments

Comments
 (0)