Skip to content

Commit 993a75e

Browse files
n1hilityclaude
andcommitted
CARRY: ci(forge): measure forward service connection cost on a runner
Add a manually dispatched workflow that brings up a single-node gateway from a published image set on ubuntu-latest (Docker compute driver, mTLS from generate-certs, file-backed SQLite on the runner disk), creates a sandbox serving loopback HTTP, and measures `openshell forward service` connection bursts of 1..64 through it. It runs once with the opendatahub images and once with the rh-forge images, whose only difference is the SQLite store running in WAL + synchronous=NORMAL, and renders a before/after table in the run summary together with the observed `PRAGMA journal_mode`, the raw fdatasync cost of the runner disk and the number of `connection limit reached` refusals. Evidence for NVIDIA#3494. Co-Authored-By: Claude Opus 5 <[email protected]> Signed-off-by: Jason T. Greene <[email protected]>
1 parent 918b95a commit 993a75e

5 files changed

Lines changed: 646 additions & 0 deletions

File tree

Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
name: forward-sweep
5+
6+
# Measures the per-connection cost of `openshell forward service` end to end
7+
# on a hosted runner, once with the opendatahub images (baseline) and once
8+
# with the rh-forge images (patched: SQLite store in WAL + synchronous=NORMAL),
9+
# and prints a before/after table. Evidence for NVIDIA/OpenShell#3494.
10+
#
11+
# Each set runs in its own job: the gateway and CLI binaries are extracted
12+
# from the images, a single-node gateway runs on the runner with the Docker
13+
# compute driver (mTLS, file-backed SQLite on the runner disk, supervisor
14+
# taken from that set's supervisor image), a sandbox serves loopback HTTP,
15+
# and scripts/forward-sweep/sweep.py opens bursts of TCP connections through
16+
# the forward. See scripts/forward-sweep/run-set.sh.
17+
18+
on:
19+
workflow_dispatch:
20+
inputs:
21+
baseline_tag:
22+
description: "quay.io/opendatahub/odh-openshell-* tag (baseline)"
23+
required: false
24+
type: string
25+
default: "v0.0.116-rhaiv.0"
26+
patched_tag:
27+
description: "ghcr.io/rh-forge/odh-openshell-* tag (patched)"
28+
required: false
29+
type: string
30+
default: "v0.0.116-rhaiv.0-forge.1"
31+
32+
permissions:
33+
contents: read
34+
packages: read
35+
36+
jobs:
37+
sweep:
38+
name: Sweep (${{ matrix.set }})
39+
runs-on: ubuntu-latest
40+
timeout-minutes: 45
41+
strategy:
42+
fail-fast: false
43+
matrix:
44+
include:
45+
- set: baseline
46+
prefix: quay.io/opendatahub
47+
- set: patched
48+
prefix: ghcr.io/rh-forge
49+
env:
50+
SET_NAME: ${{ matrix.set }}
51+
IMAGE_TAG: ${{ matrix.set == 'baseline' && inputs.baseline_tag || inputs.patched_tag }}
52+
IMAGE_PREFIX: ${{ matrix.prefix }}
53+
steps:
54+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
55+
with:
56+
persist-credentials: false
57+
58+
- name: Log in to ghcr.io
59+
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
60+
with:
61+
registry: ghcr.io
62+
username: ${{ github.actor }}
63+
password: ${{ secrets.GITHUB_TOKEN }}
64+
65+
- name: Run the sweep
66+
shell: bash
67+
env:
68+
OUT_DIR: ${{ runner.temp }}/forward-sweep-out
69+
run: |
70+
set -euo pipefail
71+
export GATEWAY_IMAGE="${IMAGE_PREFIX}/odh-openshell-gateway:${IMAGE_TAG}"
72+
export SUPERVISOR_IMAGE="${IMAGE_PREFIX}/odh-openshell-supervisor:${IMAGE_TAG}"
73+
export CLI_IMAGE="${IMAGE_PREFIX}/odh-openshell-cli:${IMAGE_TAG}"
74+
docker info --format 'docker {{.ServerVersion}} on {{.OperatingSystem}} ({{.Architecture}})'
75+
scripts/forward-sweep/run-set.sh
76+
77+
- name: Upload raw outputs
78+
if: always()
79+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
80+
with:
81+
name: forward-sweep-${{ matrix.set }}
82+
path: ${{ runner.temp }}/forward-sweep-out
83+
if-no-files-found: warn
84+
85+
report:
86+
name: Before/after report
87+
needs: sweep
88+
if: always()
89+
runs-on: ubuntu-latest
90+
timeout-minutes: 5
91+
steps:
92+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
93+
with:
94+
persist-credentials: false
95+
96+
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
97+
with:
98+
pattern: forward-sweep-*
99+
path: results
100+
101+
- name: Render report
102+
shell: bash
103+
run: |
104+
set -euo pipefail
105+
ls -R results
106+
if [ ! -f results/forward-sweep-baseline/results.json ] || [ ! -f results/forward-sweep-patched/results.json ]; then
107+
echo "::error::one or both sweep jobs produced no results.json; see the sweep job logs"
108+
exit 1
109+
fi
110+
python3 scripts/forward-sweep/report.py \
111+
--baseline results/forward-sweep-baseline/results.json \
112+
--patched results/forward-sweep-patched/results.json \
113+
| tee results/report.md >> "$GITHUB_STEP_SUMMARY"
114+
115+
- name: Upload report
116+
if: always()
117+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
118+
with:
119+
name: forward-sweep-report
120+
path: results
121+
if-no-files-found: warn

‎deploy/docker/README-forge.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,3 +20,29 @@ on tag pushes matching `v*-forge.*` (or by dispatch) and publishes
2020
`ghcr.io/rh-forge/odh-openshell-{gateway,supervisor,cli}:<tag>` plus
2121
`:sha-<commit>`. The tag, minus the leading `v`, is stamped into every binary
2222
through `OPENSHELL_VERSION`, so all three report the same version string.
23+
24+
## Forward connection sweep (`forward-sweep.yml`)
25+
26+
`.github/workflows/forward-sweep.yml` (manual dispatch) measures the cost of
27+
`openshell forward service` connections end to end on a hosted runner, once
28+
with the opendatahub images (`quay.io/opendatahub/odh-openshell-*`, baseline)
29+
and once with the rh-forge images (`ghcr.io/rh-forge/odh-openshell-*`, patched:
30+
SQLite store in WAL + `synchronous=NORMAL`), then prints a before/after table
31+
in the run summary. It is the evidence for
32+
[NVIDIA/OpenShell#3494](https://github.com/NVIDIA/OpenShell/issues/3494).
33+
34+
Each image set runs in its own job (`scripts/forward-sweep/run-set.sh`): the
35+
gateway and CLI binaries are extracted from the images; a single-node gateway
36+
runs on the runner with the Docker compute driver, mTLS from
37+
`openshell-gateway generate-certs`, and a file-backed SQLite database on the
38+
runner disk; the set's supervisor image is pre-pulled (the gateway extracts the
39+
supervisor through the Docker API, which has no registry credentials); a
40+
sandbox built from `python:3.13-slim` plus `iproute2` serves loopback HTTP;
41+
`scripts/forward-sweep/sweep.py` then opens bursts of 1, 6, 16, 32 and 64
42+
simultaneous connections (twice each) plus 10 sequential ones through the
43+
forward and records wall time, completions and mean latency. The job also
44+
reports the raw `fdatasync`/`dd oflag=dsync` cost of the runner disk, the
45+
number of `connection limit reached` refusals in the forward log, and the
46+
`PRAGMA journal_mode` of the gateway database (`delete` for baseline, `wal`
47+
for patched). Raw logs and `results.json` are uploaded as artifacts; the
48+
`report` job merges both into one table.

‎scripts/forward-sweep/report.py‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
#!/usr/bin/env python3
2+
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
3+
# SPDX-License-Identifier: Apache-2.0
4+
5+
"""Render a before/after markdown report from two forward-sweep results.json files."""
6+
7+
import argparse
8+
import json
9+
import sys
10+
11+
12+
def load(path):
13+
with open(path, encoding="utf-8") as handle:
14+
return json.load(handle)
15+
16+
17+
def fmt(value, suffix=""):
18+
return "-" if value is None else f"{value}{suffix}"
19+
20+
21+
def main():
22+
parser = argparse.ArgumentParser()
23+
parser.add_argument("--baseline", required=True)
24+
parser.add_argument("--patched", required=True)
25+
args = parser.parse_args()
26+
27+
base = load(args.baseline)
28+
new = load(args.patched)
29+
out = []
30+
31+
out.append("## openshell forward service: baseline vs patched")
32+
out.append("")
33+
out.append("| | baseline | patched |")
34+
out.append("|---|---|---|")
35+
for key, label in [
36+
("gateway_image", "gateway image"),
37+
("supervisor_image", "supervisor image"),
38+
("cli_image", "cli image"),
39+
("gateway_version", "gateway --version"),
40+
("journal_mode", "SQLite `PRAGMA journal_mode`"),
41+
("db_sidecars", "DB sidecar files"),
42+
("fsync_mean_ms", "fdatasync 4 KiB mean (ms)"),
43+
("dd_dsync", "dd oflag=dsync (200 x 4 KiB)"),
44+
("limit_hits", "`connection limit reached` in forward log"),
45+
("forward_warnings", "forward warnings total"),
46+
("runner", "runner"),
47+
]:
48+
out.append(f"| {label} | {fmt(base.get(key))} | {fmt(new.get(key))} |")
49+
out.append("")
50+
out.append(
51+
"| scenario | baseline completed | baseline wall (s) | baseline mean/conn (ms) "
52+
"| patched completed | patched wall (s) | patched mean/conn (ms) | wall speedup |"
53+
)
54+
out.append("|---|---:|---:|---:|---:|---:|---:|---:|")
55+
new_rows = {row["scenario"]: row for row in new.get("sweep", [])}
56+
for b in base.get("sweep", []):
57+
p = new_rows.get(b["scenario"], {})
58+
speedup = "-"
59+
if p.get("wall_s") and b.get("wall_s"):
60+
speedup = f"{b['wall_s'] / p['wall_s']:.1f}x"
61+
out.append(
62+
f"| {b['scenario']} | {b['completed']}/{b['n']} | {b['wall_s']} | {fmt(b['mean_ms'])} "
63+
f"| {fmt(p.get('completed'))}/{fmt(p.get('n'))} | {fmt(p.get('wall_s'))} | {fmt(p.get('mean_ms'))} "
64+
f"| {speedup} |"
65+
)
66+
out.append("")
67+
out.append(
68+
"Each concurrent row opens N TCP connections through the forward at the same instant; "
69+
"`completed` counts connections that received response bytes within the timeout. "
70+
"The gateway caps a sandbox at 20 concurrent SSH-session connections, so bursts above "
71+
"that can be refused by design; what matters is how quickly slots free up."
72+
)
73+
print("\n".join(out))
74+
return 0
75+
76+
77+
if __name__ == "__main__":
78+
sys.exit(main())

0 commit comments

Comments
 (0)