Project: NVIDIA/OpenShell
Issue: #2792 — normalize gateway configuration names
Branch: refactor/2792-normalize-gateway-config-names/jhjaggars
Report generated: 2026-09-04 15:46:15 UTC
This campaign tested the gateway schema-v2 configuration refactor against a frozen origin/main baseline. It combined deterministic regression tests, paired schema-v1/schema-v2 runtime comparisons, immutable artifact and source provenance, security review, package-startup safeguards, and explicit disposition of infrastructure-dependent checks.
The campaign recorded:
| Result | Count |
|---|---|
| Pass | 4 |
| Intentional change | 2 |
| Platform blocked | 24 |
| Regression | 0 |
| Total live-result records | 30 |
A platform_blocked result is not a pass. Every blocked result retains a named owner, execution lane, and blocker. Equal failures were never accepted as parity.
The strongest locally executable claims passed:
- Rootless Podman portable lifecycle parity.
- Gateway-wide process-option and configuration-precedence parity.
- Disposable kind Kubernetes core-option parity.
- In-tree and external-UDS compute-driver boundary parity.
- Security-reviewed intentional changes for TLS client-auth configuration and Podman PID-limit enforcement.
- Deterministic schema, cross-cutting, package-preflight, shell, Helm, Rust, Python, and TypeScript validation.
Real Docker, VM, MXC, qualified Kubernetes/security matrices, cross-cutting live service fixtures, and package install/upgrade/refresh lanes remain explicitly blocked pending their assigned infrastructure.
| Role | Commit |
|---|---|
Frozen baseline (origin/main) |
74960ebfaeec4673885089ed995fad902459749f |
| Candidate validation start | 8c868e430e9cd3284d7e274628419ab484ebcee0 |
| Final tested and committed branch HEAD | 9ccf6d9faf935438809b589db7492d5466786d59 |
| Final Step 10 runtime candidate | 4a39da510e4d278a24dd60291149519c9a570b46 |
| Step 10 evidence commit | 363d8540830b2ea294d43198daa2b7a283a2face |
| Step 11 evidence commit | 740f2eddca7ce758489701a1906006c22179c525 |
| Step 12 package-preflight commit | 9ccf6d9faf935438809b589db7492d5466786d59 |
The baseline and candidate used isolated worktrees and checkout-local build targets where required. Runtime comparisons isolated ports, databases, state, PKI, sockets, networks, namespaces, container stores, and driver processes.
The campaign enforced these rules:
- Deterministic tests preceded live validation.
- A live parity pass required successful baseline and candidate executions of the same observable oracle.
- A skip, equal failure, or partial success was not parity.
- Known flakes required an exact-source paired rerun; the oracle was not weakened.
- Candidate evidence binaries were freshly built from the recorded source and bound with SHA-256.
- Raw logs remained separate from normalized semantic results.
- Normalized results were independently tied to retained logs, launch manifests, configuration captures, package manifests, and staged artifacts.
- Mutable image references were rejected for parity execution; runtime references were digest-pinned.
- The same immutable sandbox artifact and supervisor dependency provenance were required across baseline/candidate and in-tree/external runs.
- External-driver gateways were driver-free and launched with a sanitized
env -ienvironment. - Docker and Podman remained separate lanes; a Libpod socket was not accepted as Docker evidence.
- Kubernetes tests used only a guarded disposable campaign-owned kind cluster. The active external OpenShift cluster and unrelated kind resources were not modified.
- Cleanup removed only campaign-owned resources and superseded artifacts.
- Security-sensitive changes and final evidence required Terra review approval before commit.
- Commits used Conventional Commit messages, DCO sign-off, and verified GPG signatures.
Phase 1 added 33 Rust/Python regression tests and expanded Helm and shell coverage. It also hardened:
crates/openshell-driver-vm/src/driver.rsdeploy/rpm/migrate-gateway-config.shtasks/scripts/gateway-toml.sh
Validation included focused Rust tests and Clippy, 20 Python tests, 137 Helm tests, shell suites, and mise run pre-commit.
Commit: 8c868e430e9cd3284d7e274628419ab484ebcee0 — test(config): expand schema v2 regression coverage
| Step | Outcome | Milestone commit |
|---|---|---|
| 1 | Built a machine-readable inventory of gateway, driver, producer, and package capabilities available on the frozen baseline. | db8c2a94, corrected by fd6b6a2d |
| 2 | Recorded all accepted semantic changes in a separate intentional-change ledger. | e8f94559 |
| 3 | Dispositioned suspected parity gaps before making runtime claims. | 0b502d84 |
| 4 | Added the paired schema-v1/schema-v2 execution and normalization harness. | a3860084 |
The rootless Podman lane passed the full lifecycle oracle:
- authenticated gateway status;
- create sandbox;
- observe Ready state;
- paginated list visibility;
- exact-output exec;
- delete sandbox;
- eventual empty-list confirmation.
Docker, Kubernetes, VM, and MXC portable lifecycle rows remained blocked at this stage. Kubernetes received a later dedicated disposable-cluster pass in Step 8.
Passed paired gateway-process validation for:
- configuration precedence;
- primary, health, and metrics listeners;
- auxiliary-listener disable behavior;
- SQLite persistence and reopen;
- rejection of TOML
database_urlwithout secret disclosure; - the deprecated single-value
OPENSHELL_DRIVERSenvironment alias; - canonical selector conflict and ambiguity handling.
TLS require_client_auth removal was accepted as an intentional change after security review. The old TOML field was accepted but ineffective; schema v2 rejects it and documents the actual derived policy:
- client CA without OIDC requires a client certificate;
- client CA with OIDC allows bearer-only clients and validates any presented certificate.
Podman option parity ran against isolated baseline and candidate gateways. Both variants matched for images, pull policy, CPU and memory, labels, bind/tmpfs mounts, bootstrap files, callback TLS, SSH socket behavior, health checks, and callback exec.
One difference was accepted as intentional: schema v2 correctly applies the configured OCI PID limit. The frozen baseline populated a Docker-specific field that Podman ignored and therefore retained its default.
Docker option validation remains assigned to a real Docker lane. Qualified Podman AppArmor, authenticated proxy, SPIFFE, and rootful user-namespace validation remains assigned to a dedicated security matrix.
A campaign-owned disposable kind cluster was provisioned over rootless Podman. Both variants reached Ready and passed exact-marker exec. Pod and PVC observations matched for:
- images and pull policies;
- pull Secret and ServiceAccount;
- TLS and token projections;
- host aliases and callback endpoint;
- SSH endpoint;
- combined/runc placement;
- AppArmor request;
- UID/GID and CPU/memory;
- shared storage;
- managed metadata and deletion.
The cluster openshell-parity-2792-step8 and its private test material were removed after validation. Managed/operator workspaces, sidecars, authenticated proxy, SPIRE, user namespaces, enforced AppArmor, image volumes, Kata, and GPU variants remain blocked on qualified infrastructure.
Baseline and candidate deterministic VM library suites passed. No live KVM/libkrun runtime bundle or runnable frozen/candidate VM driver was available, so VM lifecycle and VM guest-security claims remain platform_blocked rather than being promoted from deterministic coverage.
Commit: ecaf7e5a8527e85f5168b3ac0fadbac4d3bf4bd8
Step 10 established paired in-tree and external-UDS Podman compute-driver parity with hardened provenance.
The final accepted bundles are:
target/parity/step10-intree-4a39da51target/parity/step10-external-4a39da51
The comparison report is:
e2e/configs/gateway/schema-v2-compute-boundary-comparison.json- SHA-256:
65541eec5f642461a88b04b5459474fd7a475adeb7071a65a53fe183caad6a01
Evidence controls included:
- fresh exact-source gateway, CLI, conformance, Podman driver, and supervisor artifacts;
- pre-execution staging and post-execution unchanged-artifact checks;
- distinct baseline/candidate external-driver paths, inodes, and content;
- driver-free external gateways;
- sanitized
env -idriver launches; - separate UDS paths, Podman sockets, networks, callback TLS, and stores;
- exact exec stdout capture tied to a unique conformance run ID;
- authenticated preflight and zero exit status for every lifecycle action;
- one digest-pinned sandbox image across all four runs;
- matching supervisor base identity/reference, Dockerfile, and installed-package manifest;
- SHA-256 binding for binaries, logs, configurations, manifests, and retained evidence.
Several earlier bundles were rejected rather than promoted because they were partial, flaky, or incompletely attested. The final evidence received Terra APPROVE.
Frozen baseline and candidate deterministic suites passed:
| Suite | Baseline | Candidate |
|---|---|---|
openshell-server --lib |
1,455 passed, 8 ignored | 1,481 passed, 8 ignored |
openshell-gateway-interceptors |
49 passed | 49 passed |
openshell-supervisor-middleware |
85 passed | 85 passed |
multiplex_tls_integration |
5 passed | 5 passed |
edge_tunnel_auth |
5 passed | 5 passed |
openshell-otel-test-support |
Compiled; zero tests | Compiled; zero tests |
Tracked attestations bind source commits, clean-tree state, checkout-local targets, disabled compiler cache, tool versions, commands, full output, individual exits, and completion markers:
schema-v2-step11-baseline-attestation.txt— SHA-256df3c5e1dcb073eb8db0cb28377516523b331d44fc6f5a5fefab2e92273efaf51schema-v2-step11-candidate-attestation.txt— SHA-2562562bff73eba8042199597e814b618b12b93994d15e4ae36e902347347d95fed
These suites are deterministic preflight, not live service parity. The 11 auth, JWT, OTLP, interceptor, middleware, provider, inference, and credential rows therefore remain assigned to dedicated live lanes. Terra approved this disposition.
Historical Debian and Snap schema-v1 generators used dynamic paths and left no trustworthy package-default provenance marker. Automatic rewriting would risk destroying operator intent. Step 12 therefore implemented a read-only, fail-closed package preflight rather than unsafe migration inference.
New command:
openshell-gateway config preflight [--path PATH]The command:
- gives
--pathprecedence overOPENSHELL_GATEWAY_CONFIGand XDG discovery; - allows absent optional auto-discovery;
- fails for explicit or environment-selected missing paths;
- accepts valid strict schema v2;
- rejects schema v1, missing/future versions, malformed or semantically invalid v2, unreadable files, symlinks, directories, FIFOs, and other non-regular paths;
- uses no-follow, nonblocking, descriptor-based regular-file checks on Unix;
- runs read-only effective startup checks for rate-limit pairs, TLS relationships, interceptors, and supervisor middleware;
- does not construct compute drivers, connect transports, generate certificates, or create state;
- never includes raw TOML contents in Display, Debug,
miette::Report, or error source chains; - preserves every rejected file and emits stable category/version/path guidance plus a migration URL.
Package integration:
- Debian runs preflight before certificate generation.
- Snap preserves CLI
--config> environment > canonical$SNAP_COMMON/gateway.toml> optional discovery precedence. - Snap rejects ambiguous/missing selectors and does not start after preflight failure.
- RPM’s existing exact-default migration behavior remains unchanged.
Final focused validation included:
openshell-server --lib: 1,495 passed, 8 ignored;- gateway interceptor tests: 49 passed;
- supervisor middleware tests: 85 passed;
- TLS tests: 16 passed;
- Snap fake-wrapper and package-asset tests: passed;
- release/RPM Python tests: 17 passed;
- schema-v2 manifest/evidence tests: passed;
- full
mise run pre-commit: passed; - full
mise run test: passed; - TypeScript SDK: 103 tests passed.
Terra review found and drove remediation of:
- raw TOML leakage through retained error source chains;
- missing effective semantic validation;
- disabled-TLS compatibility under Snap defaults;
- client-CA-only startup mismatch;
- Snap CLI/environment config-selection mismatch;
- unsafe shell argument boundaries;
- evidence records that could have overstated package validation.
The final implementation and final evidence each received Terra APPROVE.
Commit: 9ccf6d9faf935438809b589db7492d5466786d59 — fix(packaging): preflight gateway config upgrades
| Step | Result | Lane |
|---|---|---|
| 5 | portable-lifecycle-podman |
local-linux-x86_64-rootless-podman-5.8.2 |
| 6 | gateway-wide-process-options |
local-linux-x86_64-rootless-podman-5.8.2 |
| 8 | kubernetes-core-option-parity |
kind-v1.36.1-rootless-podman-host-gateway |
| 10 | compute-driver-boundary-parity |
local-linux-x86_64-rootless-podman-5.8.2 |
| Step | Result | Reason |
|---|---|---|
| 6 | gateway-tls-client-auth-policy |
Removes a misleading, ineffective file option while preserving the established derived TLS policy. |
| 7 | podman-driver-option-parity |
Restores actual OCI PID-limit enforcement that the baseline silently failed to apply. |
| Step | Result | Assigned lane |
|---|---|---|
| 5 | portable-lifecycle-docker |
linux-x86_64-docker |
| 5 | portable-lifecycle-kubernetes |
ephemeral-kind-or-managed-kubernetes |
| 5 | portable-lifecycle-vm |
linux-x86_64-kvm-libkrun |
| 5 | portable-lifecycle-mxc |
windows-x64-and-windows-arm64-mxc |
| 7 | docker-driver-option-parity |
linux-x86_64-docker-with-apparmor-sub-lane |
| 7 | podman-qualified-security-option-parity |
linux-podman-apparmor-proxy-spiffe-rootful-userns |
| 8 | kubernetes-qualified-option-parity |
kubernetes-managed-operator-sidecar-proxy-spire-userns-apparmor-kata-gpu |
| 9 | vm-launch-and-resource-configuration |
linux-x86_64-kvm-libkrun |
| 9 | vm-guest-security-and-spiffe |
linux-x86_64-kvm-libkrun-proxy-spiffe |
| 11 | oidc-bearer-authentication |
linux-oidc-paired-keycloak-jwks |
| 11 | mtls-user-authentication |
linux-mtls-user-principal-paired |
| 11 | unsafe-unauthenticated-user-mode |
linux-auth-chain-paired-isolated |
| 11 | gateway-minted-sandbox-jwt |
linux-gateway-jwt-claims-paired |
| 11 | otlp-observability |
linux-otlp-grpc-collector-paired |
| 11 | gateway-interceptor-registration |
linux-gateway-interceptor-paired |
| 11 | supervisor-middleware-registration |
linux-supervisor-middleware-paired |
| 11 | provider-profile-sources |
linux-provider-profiles-interceptor-paired |
| 11 | inference-control-plane-configuration |
linux-inference-provider-routing-paired |
| 11 | credential-driver-selection-and-kek |
linux-credential-kek-paired |
| 11 | credential-driver-backend-tables |
kubernetes-vault-uds-credential-drivers-paired |
| 12 | rpm-package-upgrade |
fedora-rpm-prior-release-upgrade |
| 12 | debian-package-upgrade |
ubuntu-debian-prior-release-upgrade |
| 12 | snap-package-refresh |
ubuntu-snap-prior-release-refresh |
| 12 | homebrew-package-upgrade |
macos-homebrew-prior-release-upgrade |
- No Docker CLI or daemon was available. The Docker driver correctly rejected the Libpod socket.
- Rootless Podman lacked the AppArmor/rootful/proxy/SPIFFE conditions needed for the qualified security matrix.
- No prepared KVM/libkrun runtime bundle or runnable paired VM driver was available.
- MXC requires native Windows x64/arm64 infrastructure.
- Cross-cutting live checks require dedicated OIDC, mTLS, JWT, OTLP, interceptor, middleware, provider, inference, credential, Vault, or Kubernetes Secrets fixtures.
dpkg-debwas unavailable locally.- Real Snap refresh, RPM upgrade, Debian upgrade, and Homebrew upgrade behavior requires package-capable hosts.
- The active external OpenShift cluster and unrelated kind cluster were intentionally not used.
e2e/configs/gateway/schema-v2-capability-parity.tomle2e/configs/gateway/schema-v2-intentional-changes.tomle2e/configs/gateway/schema-v2-parity-gap-dispositions.tomle2e/configs/gateway/schema-v2-live-results.tomle2e/configs/gateway/schema-v2-kubernetes-core-comparison.jsone2e/configs/gateway/schema-v2-kubernetes-option-dispositions.tomle2e/configs/gateway/schema-v2-compute-boundary-comparison.jsone2e/configs/gateway/schema-v2-cross-cutting-dispositions.tomle2e/configs/gateway/schema-v2-step11-baseline-attestation.txte2e/configs/gateway/schema-v2-step11-candidate-attestation.txtpython/openshell/gateway_schema_v2_*_test.pye2e/parity/tasks/parity.toml
No regression is recorded in the final ledger. All locally executable paired or deterministic checks completed successfully, and the two observed semantic differences were reviewed and recorded as intentional changes rather than hidden as parity. Evidence with incomplete provenance or flaky execution was rejected and regenerated instead of being promoted.
The report does not claim that unavailable platforms passed. Twenty-four rows remain platform_blocked, each with explicit ownership and a qualifying lane. Those lanes are the remaining execution obligations for platform-specific confidence beyond the completed local campaign.
The final working tree was clean, the final package-preflight commit was GPG verified and DCO signed, and the preserved RPM schema-v1 fixture and migration regression test remained unchanged.