Skip to content

Instantly share code, notes, and snippets.

@jhjaggars
Created September 4, 2026 15:47
Show Gist options
  • Select an option

  • Save jhjaggars/d890ce0d377d8fc5187f61aa390bbf5c to your computer and use it in GitHub Desktop.

Select an option

Save jhjaggars/d890ce0d377d8fc5187f61aa390bbf5c to your computer and use it in GitHub Desktop.
OpenShell Gateway Schema-v2 Live Capability-Parity Testing Report (#2792)

OpenShell Gateway Schema-v2 Testing Report

Project: NVIDIA/OpenShell
Issue: #2792 — normalize gateway configuration names
Branch: refactor/2792-normalize-gateway-config-names/jhjaggars
Report generated: 2026-09-04 15:46:15 UTC

Executive summary

This campaign tested the gateway schema-v2 configuration refactor against a frozen origin/main baseline. It combined deterministic regression tests, paired schema-v1/schema-v2 runtime comparisons, immutable artifact and source provenance, security review, package-startup safeguards, and explicit disposition of infrastructure-dependent checks.

The campaign recorded:

Result Count
Pass 4
Intentional change 2
Platform blocked 24
Regression 0
Total live-result records 30

A platform_blocked result is not a pass. Every blocked result retains a named owner, execution lane, and blocker. Equal failures were never accepted as parity.

The strongest locally executable claims passed:

  • Rootless Podman portable lifecycle parity.
  • Gateway-wide process-option and configuration-precedence parity.
  • Disposable kind Kubernetes core-option parity.
  • In-tree and external-UDS compute-driver boundary parity.
  • Security-reviewed intentional changes for TLS client-auth configuration and Podman PID-limit enforcement.
  • Deterministic schema, cross-cutting, package-preflight, shell, Helm, Rust, Python, and TypeScript validation.

Real Docker, VM, MXC, qualified Kubernetes/security matrices, cross-cutting live service fixtures, and package install/upgrade/refresh lanes remain explicitly blocked pending their assigned infrastructure.

Immutable comparison points

Role Commit
Frozen baseline (origin/main) 74960ebfaeec4673885089ed995fad902459749f
Candidate validation start 8c868e430e9cd3284d7e274628419ab484ebcee0
Final tested and committed branch HEAD 9ccf6d9faf935438809b589db7492d5466786d59
Final Step 10 runtime candidate 4a39da510e4d278a24dd60291149519c9a570b46
Step 10 evidence commit 363d8540830b2ea294d43198daa2b7a283a2face
Step 11 evidence commit 740f2eddca7ce758489701a1906006c22179c525
Step 12 package-preflight commit 9ccf6d9faf935438809b589db7492d5466786d59

The baseline and candidate used isolated worktrees and checkout-local build targets where required. Runtime comparisons isolated ports, databases, state, PKI, sockets, networks, namespaces, container stores, and driver processes.

Test policy and acceptance rules

The campaign enforced these rules:

  1. Deterministic tests preceded live validation.
  2. A live parity pass required successful baseline and candidate executions of the same observable oracle.
  3. A skip, equal failure, or partial success was not parity.
  4. Known flakes required an exact-source paired rerun; the oracle was not weakened.
  5. Candidate evidence binaries were freshly built from the recorded source and bound with SHA-256.
  6. Raw logs remained separate from normalized semantic results.
  7. Normalized results were independently tied to retained logs, launch manifests, configuration captures, package manifests, and staged artifacts.
  8. Mutable image references were rejected for parity execution; runtime references were digest-pinned.
  9. The same immutable sandbox artifact and supervisor dependency provenance were required across baseline/candidate and in-tree/external runs.
  10. External-driver gateways were driver-free and launched with a sanitized env -i environment.
  11. Docker and Podman remained separate lanes; a Libpod socket was not accepted as Docker evidence.
  12. Kubernetes tests used only a guarded disposable campaign-owned kind cluster. The active external OpenShift cluster and unrelated kind resources were not modified.
  13. Cleanup removed only campaign-owned resources and superseded artifacts.
  14. Security-sensitive changes and final evidence required Terra review approval before commit.
  15. Commits used Conventional Commit messages, DCO sign-off, and verified GPG signatures.

Validation phases and step results

Phase 1 — deterministic schema-v2 regression coverage

Phase 1 added 33 Rust/Python regression tests and expanded Helm and shell coverage. It also hardened:

  • crates/openshell-driver-vm/src/driver.rs
  • deploy/rpm/migrate-gateway-config.sh
  • tasks/scripts/gateway-toml.sh

Validation included focused Rust tests and Clippy, 20 Python tests, 137 Helm tests, shell suites, and mise run pre-commit.

Commit: 8c868e430e9cd3284d7e274628419ab484ebcee0 — test(config): expand schema v2 regression coverage

Steps 1–4 — inventory, dispositions, and paired harness

Step Outcome Milestone commit
1 Built a machine-readable inventory of gateway, driver, producer, and package capabilities available on the frozen baseline. db8c2a94, corrected by fd6b6a2d
2 Recorded all accepted semantic changes in a separate intentional-change ledger. e8f94559
3 Dispositioned suspected parity gaps before making runtime claims. 0b502d84
4 Added the paired schema-v1/schema-v2 execution and normalization harness. a3860084

Step 5 — portable compute lifecycle

The rootless Podman lane passed the full lifecycle oracle:

  • authenticated gateway status;
  • create sandbox;
  • observe Ready state;
  • paginated list visibility;
  • exact-output exec;
  • delete sandbox;
  • eventual empty-list confirmation.

Docker, Kubernetes, VM, and MXC portable lifecycle rows remained blocked at this stage. Kubernetes received a later dedicated disposable-cluster pass in Step 8.

Step 6 — gateway-wide options and TLS policy

Passed paired gateway-process validation for:

  • configuration precedence;
  • primary, health, and metrics listeners;
  • auxiliary-listener disable behavior;
  • SQLite persistence and reopen;
  • rejection of TOML database_url without secret disclosure;
  • the deprecated single-value OPENSHELL_DRIVERS environment alias;
  • canonical selector conflict and ambiguity handling.

TLS require_client_auth removal was accepted as an intentional change after security review. The old TOML field was accepted but ineffective; schema v2 rejects it and documents the actual derived policy:

  • client CA without OIDC requires a client certificate;
  • client CA with OIDC allows bearer-only clients and validates any presented certificate.

Step 7 — driver option validation

Podman option parity ran against isolated baseline and candidate gateways. Both variants matched for images, pull policy, CPU and memory, labels, bind/tmpfs mounts, bootstrap files, callback TLS, SSH socket behavior, health checks, and callback exec.

One difference was accepted as intentional: schema v2 correctly applies the configured OCI PID limit. The frozen baseline populated a Docker-specific field that Podman ignored and therefore retained its default.

Docker option validation remains assigned to a real Docker lane. Qualified Podman AppArmor, authenticated proxy, SPIFFE, and rootful user-namespace validation remains assigned to a dedicated security matrix.

Step 8 — guarded Kubernetes core parity

A campaign-owned disposable kind cluster was provisioned over rootless Podman. Both variants reached Ready and passed exact-marker exec. Pod and PVC observations matched for:

  • images and pull policies;
  • pull Secret and ServiceAccount;
  • TLS and token projections;
  • host aliases and callback endpoint;
  • SSH endpoint;
  • combined/runc placement;
  • AppArmor request;
  • UID/GID and CPU/memory;
  • shared storage;
  • managed metadata and deletion.

The cluster openshell-parity-2792-step8 and its private test material were removed after validation. Managed/operator workspaces, sidecars, authenticated proxy, SPIRE, user namespaces, enforced AppArmor, image volumes, Kata, and GPU variants remain blocked on qualified infrastructure.

Step 9 — VM disposition

Baseline and candidate deterministic VM library suites passed. No live KVM/libkrun runtime bundle or runnable frozen/candidate VM driver was available, so VM lifecycle and VM guest-security claims remain platform_blocked rather than being promoted from deterministic coverage.

Commit: ecaf7e5a8527e85f5168b3ac0fadbac4d3bf4bd8

Step 10 — in-tree/external compute boundary

Step 10 established paired in-tree and external-UDS Podman compute-driver parity with hardened provenance.

The final accepted bundles are:

  • target/parity/step10-intree-4a39da51
  • target/parity/step10-external-4a39da51

The comparison report is:

  • e2e/configs/gateway/schema-v2-compute-boundary-comparison.json
  • SHA-256: 65541eec5f642461a88b04b5459474fd7a475adeb7071a65a53fe183caad6a01

Evidence controls included:

  • fresh exact-source gateway, CLI, conformance, Podman driver, and supervisor artifacts;
  • pre-execution staging and post-execution unchanged-artifact checks;
  • distinct baseline/candidate external-driver paths, inodes, and content;
  • driver-free external gateways;
  • sanitized env -i driver launches;
  • separate UDS paths, Podman sockets, networks, callback TLS, and stores;
  • exact exec stdout capture tied to a unique conformance run ID;
  • authenticated preflight and zero exit status for every lifecycle action;
  • one digest-pinned sandbox image across all four runs;
  • matching supervisor base identity/reference, Dockerfile, and installed-package manifest;
  • SHA-256 binding for binaries, logs, configurations, manifests, and retained evidence.

Several earlier bundles were rejected rather than promoted because they were partial, flaky, or incompletely attested. The final evidence received Terra APPROVE.

Step 11 — cross-cutting deterministic validation

Frozen baseline and candidate deterministic suites passed:

Suite Baseline Candidate
openshell-server --lib 1,455 passed, 8 ignored 1,481 passed, 8 ignored
openshell-gateway-interceptors 49 passed 49 passed
openshell-supervisor-middleware 85 passed 85 passed
multiplex_tls_integration 5 passed 5 passed
edge_tunnel_auth 5 passed 5 passed
openshell-otel-test-support Compiled; zero tests Compiled; zero tests

Tracked attestations bind source commits, clean-tree state, checkout-local targets, disabled compiler cache, tool versions, commands, full output, individual exits, and completion markers:

  • schema-v2-step11-baseline-attestation.txt — SHA-256 df3c5e1dcb073eb8db0cb28377516523b331d44fc6f5a5fefab2e92273efaf51
  • schema-v2-step11-candidate-attestation.txt — SHA-256 2562bff73eba8042199597e814b618b12b93994d15e4ae36e902347347d95fed

These suites are deterministic preflight, not live service parity. The 11 auth, JWT, OTLP, interceptor, middleware, provider, inference, and credential rows therefore remain assigned to dedicated live lanes. Terra approved this disposition.

Step 12 — package upgrade safety and config preflight

Historical Debian and Snap schema-v1 generators used dynamic paths and left no trustworthy package-default provenance marker. Automatic rewriting would risk destroying operator intent. Step 12 therefore implemented a read-only, fail-closed package preflight rather than unsafe migration inference.

New command:

openshell-gateway config preflight [--path PATH]

The command:

  • gives --path precedence over OPENSHELL_GATEWAY_CONFIG and XDG discovery;
  • allows absent optional auto-discovery;
  • fails for explicit or environment-selected missing paths;
  • accepts valid strict schema v2;
  • rejects schema v1, missing/future versions, malformed or semantically invalid v2, unreadable files, symlinks, directories, FIFOs, and other non-regular paths;
  • uses no-follow, nonblocking, descriptor-based regular-file checks on Unix;
  • runs read-only effective startup checks for rate-limit pairs, TLS relationships, interceptors, and supervisor middleware;
  • does not construct compute drivers, connect transports, generate certificates, or create state;
  • never includes raw TOML contents in Display, Debug, miette::Report, or error source chains;
  • preserves every rejected file and emits stable category/version/path guidance plus a migration URL.

Package integration:

  • Debian runs preflight before certificate generation.
  • Snap preserves CLI --config > environment > canonical $SNAP_COMMON/gateway.toml > optional discovery precedence.
  • Snap rejects ambiguous/missing selectors and does not start after preflight failure.
  • RPM’s existing exact-default migration behavior remains unchanged.

Final focused validation included:

  • openshell-server --lib: 1,495 passed, 8 ignored;
  • gateway interceptor tests: 49 passed;
  • supervisor middleware tests: 85 passed;
  • TLS tests: 16 passed;
  • Snap fake-wrapper and package-asset tests: passed;
  • release/RPM Python tests: 17 passed;
  • schema-v2 manifest/evidence tests: passed;
  • full mise run pre-commit: passed;
  • full mise run test: passed;
  • TypeScript SDK: 103 tests passed.

Terra review found and drove remediation of:

  • raw TOML leakage through retained error source chains;
  • missing effective semantic validation;
  • disabled-TLS compatibility under Snap defaults;
  • client-CA-only startup mismatch;
  • Snap CLI/environment config-selection mismatch;
  • unsafe shell argument boundaries;
  • evidence records that could have overstated package validation.

The final implementation and final evidence each received Terra APPROVE.

Commit: 9ccf6d9faf935438809b589db7492d5466786d59 — fix(packaging): preflight gateway config upgrades

Final live-result ledger

Passed

Step Result Lane
5 portable-lifecycle-podman local-linux-x86_64-rootless-podman-5.8.2
6 gateway-wide-process-options local-linux-x86_64-rootless-podman-5.8.2
8 kubernetes-core-option-parity kind-v1.36.1-rootless-podman-host-gateway
10 compute-driver-boundary-parity local-linux-x86_64-rootless-podman-5.8.2

Accepted intentional changes

Step Result Reason
6 gateway-tls-client-auth-policy Removes a misleading, ineffective file option while preserving the established derived TLS policy.
7 podman-driver-option-parity Restores actual OCI PID-limit enforcement that the baseline silently failed to apply.

Platform-blocked lanes

Step Result Assigned lane
5 portable-lifecycle-docker linux-x86_64-docker
5 portable-lifecycle-kubernetes ephemeral-kind-or-managed-kubernetes
5 portable-lifecycle-vm linux-x86_64-kvm-libkrun
5 portable-lifecycle-mxc windows-x64-and-windows-arm64-mxc
7 docker-driver-option-parity linux-x86_64-docker-with-apparmor-sub-lane
7 podman-qualified-security-option-parity linux-podman-apparmor-proxy-spiffe-rootful-userns
8 kubernetes-qualified-option-parity kubernetes-managed-operator-sidecar-proxy-spire-userns-apparmor-kata-gpu
9 vm-launch-and-resource-configuration linux-x86_64-kvm-libkrun
9 vm-guest-security-and-spiffe linux-x86_64-kvm-libkrun-proxy-spiffe
11 oidc-bearer-authentication linux-oidc-paired-keycloak-jwks
11 mtls-user-authentication linux-mtls-user-principal-paired
11 unsafe-unauthenticated-user-mode linux-auth-chain-paired-isolated
11 gateway-minted-sandbox-jwt linux-gateway-jwt-claims-paired
11 otlp-observability linux-otlp-grpc-collector-paired
11 gateway-interceptor-registration linux-gateway-interceptor-paired
11 supervisor-middleware-registration linux-supervisor-middleware-paired
11 provider-profile-sources linux-provider-profiles-interceptor-paired
11 inference-control-plane-configuration linux-inference-provider-routing-paired
11 credential-driver-selection-and-kek linux-credential-kek-paired
11 credential-driver-backend-tables kubernetes-vault-uds-credential-drivers-paired
12 rpm-package-upgrade fedora-rpm-prior-release-upgrade
12 debian-package-upgrade ubuntu-debian-prior-release-upgrade
12 snap-package-refresh ubuntu-snap-prior-release-refresh
12 homebrew-package-upgrade macos-homebrew-prior-release-upgrade

Known environmental constraints

  • No Docker CLI or daemon was available. The Docker driver correctly rejected the Libpod socket.
  • Rootless Podman lacked the AppArmor/rootful/proxy/SPIFFE conditions needed for the qualified security matrix.
  • No prepared KVM/libkrun runtime bundle or runnable paired VM driver was available.
  • MXC requires native Windows x64/arm64 infrastructure.
  • Cross-cutting live checks require dedicated OIDC, mTLS, JWT, OTLP, interceptor, middleware, provider, inference, credential, Vault, or Kubernetes Secrets fixtures.
  • dpkg-deb was unavailable locally.
  • Real Snap refresh, RPM upgrade, Debian upgrade, and Homebrew upgrade behavior requires package-capable hosts.
  • The active external OpenShift cluster and unrelated kind cluster were intentionally not used.

Files that anchor the evidence

  • e2e/configs/gateway/schema-v2-capability-parity.toml
  • e2e/configs/gateway/schema-v2-intentional-changes.toml
  • e2e/configs/gateway/schema-v2-parity-gap-dispositions.toml
  • e2e/configs/gateway/schema-v2-live-results.toml
  • e2e/configs/gateway/schema-v2-kubernetes-core-comparison.json
  • e2e/configs/gateway/schema-v2-kubernetes-option-dispositions.toml
  • e2e/configs/gateway/schema-v2-compute-boundary-comparison.json
  • e2e/configs/gateway/schema-v2-cross-cutting-dispositions.toml
  • e2e/configs/gateway/schema-v2-step11-baseline-attestation.txt
  • e2e/configs/gateway/schema-v2-step11-candidate-attestation.txt
  • python/openshell/gateway_schema_v2_*_test.py
  • e2e/parity/
  • tasks/parity.toml

Conclusion

No regression is recorded in the final ledger. All locally executable paired or deterministic checks completed successfully, and the two observed semantic differences were reviewed and recorded as intentional changes rather than hidden as parity. Evidence with incomplete provenance or flaky execution was rejected and regenerated instead of being promoted.

The report does not claim that unavailable platforms passed. Twenty-four rows remain platform_blocked, each with explicit ownership and a qualifying lane. Those lanes are the remaining execution obligations for platform-specific confidence beyond the completed local campaign.

The final working tree was clean, the final package-preflight commit was GPG verified and DCO signed, and the preserved RPM schema-v1 fixture and migration regression test remained unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment