Skip to content

Commit eb06a35

Browse files
committed
Merge upstream NVIDIA/OpenShell main
Sync from https://github.com/NVIDIA/OpenShell.git (main). Resolved two conflicts by unioning downstream and upstream changes: - tasks/scripts/docker-build-image.sh: keep both the `cli` and `supervisor|supervisor-sideload|supervisor-output` prebuilt-binary case arms. - tasks/scripts/stage-prebuilt-binaries.sh: `all` target expands to "gateway sandbox supervisor cli".
1 parent a5226f9 commit eb06a35

358 files changed

Lines changed: 60111 additions & 32486 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/skills/build-openshell-mxc-windows/SKILL.md‎

Lines changed: 11 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -115,7 +115,7 @@ The lane targets a Windows host with Visual Studio Build Tools and rustup.
115115
| Visual C++ ARM64 tools | `vswhere -latest -products * -requires Microsoft.VisualStudio.Component.VC.Tools.ARM64 -property installationPath` | Required for native ARM64 check, build, and tests and for x64-to-ARM64 check/build. Tests always require a native runner. |
116116
| Visual C++ ARM64 Spectre-mitigated libraries | `vswhere -latest -products * -requires Microsoft.VisualStudio.Component.VC.Runtimes.ARM64.Spectre -property installationPath` | Required by `regorus` through `msvc_spectre_libs`; the build fails when the selected MSVC toolset lacks `lib\spectre\arm64`. |
117117
| Visual C++ Clang tools | `vswhere -latest -products * -requires Microsoft.VisualStudio.Component.VC.Llvm.Clang -property installationPath` | Provides host-native `libclang.dll` for `bindgen` and `clang-cl.exe` for ARM64 crypto dependencies such as `aws-lc-sys`. On ARM64, the wrapper uses `VC\Tools\Llvm\Arm64\bin`. |
118-
| Visual C++ CMake tools | `vswhere -latest -products * -requires Microsoft.VisualStudio.Component.VC.CMake.Project -property installationPath` | Provides CMake and Ninja for bundled Z3 and other native dependencies. The x64-to-ARM64 path adds Ninja to `PATH`; Z3 uses MSVC's Visual Studio generator. |
118+
| Visual C++ CMake tools | `vswhere -latest -products * -requires Microsoft.VisualStudio.Component.VC.CMake.Project -property installationPath` | Provides CMake and Ninja for native dependencies. The x64-to-ARM64 path adds Ninja to `PATH`; Z3 uses an architecture-specific prebuilt release. |
119119
| Windows SDK | `where.exe rc.exe` from a Developer PowerShell | Install an SDK containing target libraries and ARM64 tools. |
120120
| Rust via rustup | `rustc --version` | Add each target being validated: `x86_64-pc-windows-msvc` and/or `aarch64-pc-windows-msvc`. The wrapper also adds the selected target. |
121121
| mise | `mise --version` | Used as a task runner only. |
@@ -135,6 +135,8 @@ from this skill.
135135
| `CARGO_TARGET_DIR` | `target` under repo root | Override Cargo output location. Use a short absolute path when x64-to-ARM64 builds approach Windows path-length limits. |
136136
| `Z3_LIBRARY_PATH_OVERRIDE` | unset | Directory containing an x64 system `libz3.lib`; not valid for ARM64. |
137137
| `Z3_SYS_Z3_HEADER` | unset | Full `z3.h` path required with a system Z3 library. |
138+
| `Z3_SYS_Z3_VERSION` | `4.16.0` | Pinned official prebuilt Z3 release selected by the wrapper. |
139+
| `READ_ONLY_GITHUB_TOKEN` | unset | Optional token for the Z3 release lookup; GitHub Actions supplies `github.token`. |
138140
| `RUSTC_WRAPPER` | inherited | The wrapper resolves an available command to an absolute path. If it is unavailable, the wrapper warns and continues without compiler caching. |
139141

140142
Legacy fork variables such as `OPENSHELL_UPSTREAM`,
@@ -203,8 +205,7 @@ jobs in the current mirror push run, or push a new mirrored commit. The binaries
203205
The ARM64 check/build steps in this x64-host contract are cross-builds. The
204206
wrapper discovers and adds host-native LLVM and Ninja to `PATH`, requires the
205207
ARM64 compiler and Spectre-mitigated libraries, lets ARM64 crypto crates select
206-
`clang-cl`, and builds bundled Z3 with native MSVC `cl.exe` and the Visual
207-
Studio generator.
208+
`clang-cl`, and downloads the official prebuilt ARM64 Z3 static library.
208209

209210
On ARM64 hosts, validate the native ARM64 check, build, and test path. The
210211
wrapper rejects test targets that do not match the host architecture, so x64
@@ -315,12 +316,13 @@ Useful log files:
315316
| `test-x86_64-pc-windows-msvc-unsupported-*.log` | Focused unsupported-driver contract output. |
316317
| `test-aarch64-pc-windows-msvc-unsupported-*.log` | Focused native ARM64 contract output. |
317318

318-
The first check builds bundled Z3 from source through `z3-sys`. Cargo stores the
319-
native build output in its target tree, so the Windows target cache reuses it.
320-
The resulting release executables do not require `libz3.dll`. The artifact
321-
report computes SHA256 through .NET directly and does not rely on the
322-
`Get-FileHash` module being available inside the mise-launched Windows
323-
PowerShell process.
319+
The first check downloads the pinned official Z3 archive for the target
320+
architecture through `z3-sys`. GitHub Actions authenticates the lookup with its
321+
read-only workflow token; local users can set `READ_ONLY_GITHUB_TOKEN` if an
322+
unauthenticated lookup is rate-limited. Cargo stores the extracted library in
323+
its target tree, so the Windows target cache reuses it. The artifact report
324+
computes SHA256 through .NET directly and does not rely on the `Get-FileHash`
325+
module being available inside the mise-launched Windows PowerShell process.
324326

325327
## Common Fix Patterns
326328

‎.agents/skills/build-openshell-mxc-windows/reference.md‎

Lines changed: 12 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -68,11 +68,11 @@ file.
6868
For ARM64, verify the Visual Studio instance contains the ARM64 MSVC tools,
6969
ARM64 Spectre-mitigated libraries, Clang tools, CMake tools, and a Windows SDK.
7070
Clang supplies host-native `libclang.dll` for `bindgen` and `clang-cl.exe` for
71-
ARM64 crypto dependencies such as `aws-lc-sys`. Native builds use the normal
72-
bundled-Z3 CMake path. An x64-to-ARM64 check/build discovers and adds
73-
host-native Ninja to `PATH`, builds bundled Z3 with native MSVC `cl.exe` and
74-
the Visual Studio generator, and lets the crypto crates select `clang-cl`. Use
75-
a short `CARGO_TARGET_DIR` if Windows path-length limits are reached.
71+
ARM64 crypto dependencies such as `aws-lc-sys`. Native and
72+
x64-to-ARM64 builds use the official prebuilt Z3 4.16.0 static library for the
73+
target architecture. An x64-to-ARM64 check/build discovers and adds host-native
74+
Ninja to `PATH`, while the crypto crates select `clang-cl`. Use a short
75+
`CARGO_TARGET_DIR` if Windows path-length limits are reached.
7676

7777
## Unsupported Driver Rules
7878

@@ -112,16 +112,18 @@ top-level workspace targets for check/test:
112112
--exclude openshell-driver-vault
113113
--exclude openshell-driver-vm
114114
--exclude openshell-sandbox
115-
--exclude openshell-supervisor-network
115+
--exclude openshell-supervisor
116116
--exclude openshell-supervisor-process
117117
--exclude openshell-vfio
118118
```
119119

120120
The gateway keeps platform configuration and unsupported-operation contracts
121-
without depending on the Docker, Kubernetes, Podman, sandbox supervisor,
122-
process supervisor, VM, or VFIO runtime crates. The Kubernetes Secrets and
123-
Vault libraries still compile as gateway dependencies; only their standalone
124-
Unix-socket binaries and package-level tests are excluded as top-level targets.
121+
without depending on the Docker, Kubernetes, Podman, sandbox runtime,
122+
standalone supervisor, supervisor process runtime, VM, or VFIO crates. The MXC
123+
driver does depend on the cross-platform supervisor network library for its host
124+
egress proxy. The Kubernetes Secrets and Vault libraries still compile as
125+
gateway dependencies; only their standalone Unix-socket binaries and
126+
package-level tests are excluded as top-level targets.
125127

126128
## Common Errors
127129

‎.agents/skills/helm-dev-environment/SKILL.md‎

Lines changed: 14 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -69,28 +69,15 @@ mise run helm:skaffold:dev
6969
mise run helm:skaffold:run
7070
```
7171

72-
**Supervisor sidecar topology** (build once and leave running):
73-
```bash
74-
mise run helm:skaffold:run:sidecar
75-
```
76-
77-
**Supervisor sidecar topology with TLS/mTLS enabled** (build once and leave running):
78-
```bash
79-
mise run helm:skaffold:run:sidecar-mtls
80-
```
81-
82-
Both commands build the `gateway` and `supervisor` images and deploy the OpenShell Helm
83-
chart. The sidecar profile renders an `openshell-network-init` init container for
84-
nftables setup and an `openshell-supervisor-network` runtime sidecar for proxying.
85-
Binary-aware policy mode runs that sidecar as UID 0 with `SYS_PTRACE` and
86-
`DAC_READ_SEARCH`; relaxed mode can run it as the configured proxy UID, which
87-
must be at least `1000` and distinct from the workload UID. The
88-
sidecar-mTLS profile reuses `ci/values-sidecar.yaml` and restores
89-
`server.disableTls=false` inline for Skaffold. The `pkiInitJob` hook (a pre-install
90-
Job that runs `openshell-gateway generate-certs`) generates mTLS secrets on first
91-
install. The default Skaffold values export gateway and Kubernetes-driver traces to
92-
the collector service installed by `helm:k3s:create`. Envoy Gateway opt-in; see the
93-
Optional Add-ons section below.
72+
The Skaffold flow builds distinct `gateway`, `sandbox`, and `supervisor` images
73+
and deploys the OpenShell Helm chart. The Kubernetes driver creates a
74+
capability-free workload Pod and a directly managed capability-free supervisor
75+
Pod. One namespace-wide NetworkPolicy denies direct egress from every OpenShell
76+
workload Pod. The
77+
`pkiInitJob` hook (a pre-install Job that runs `openshell-gateway generate-certs`)
78+
generates mTLS secrets on first install. The default Skaffold values export
79+
gateway and Kubernetes-driver traces to the collector service installed by
80+
`helm:k3s:create`. Envoy Gateway is opt-in; see the Optional Add-ons section.
9481

9582
The gateway Service uses ClusterIP. Access is via Envoy Gateway (port `8080`) or
9683
the unified local forwarding task:
@@ -102,9 +89,9 @@ mise run helm:k3s:forward
10289
The task forwards OTLP/gRPC to `http://127.0.0.1:4317` and the trace UI to
10390
`http://127.0.0.1:18888`. When Skaffold has deployed a Kubernetes gateway, it
10491
also forwards the gateway to `http://127.0.0.1:8090`; otherwise it continues
105-
with the collector ports only. A successful plaintext `helm:skaffold:run` or
106-
`helm:skaffold:run:sidecar` registers the gateway under the worktree-specific
107-
k3d cluster name and selects it as the active gateway. Keep the forwarding
92+
with the collector ports only. A successful plaintext `helm:skaffold:run`
93+
registers the gateway under the worktree-specific k3d
94+
cluster name and selects it as the active gateway. Keep the forwarding
10895
task running while using those endpoints.
10996

11097
### Viewing local traces
@@ -134,8 +121,7 @@ create the Secret named `openshell-ha-pg` with a `uri` key, then run
134121
### TLS behaviour
135122

136123
`ci/values-skaffold.yaml` sets `server.disableTls: true`, so Skaffold-based deploys run
137-
plaintext by default. To test sidecar topology with TLS enabled, use
138-
`mise run helm:skaffold:run:sidecar-mtls`.
124+
plaintext by default. Override `server.disableTls=false` to exercise TLS/mTLS.
139125

140126
| Mode | `server.disableTls` | Gateway scheme |
141127
|------|---------------------|----------------|
@@ -188,12 +174,6 @@ openshell sandbox list --gateway-endpoint https://localhost:8090
188174
mise run helm:skaffold:delete
189175
```
190176

191-
For a sidecar-profile deployment:
192-
193-
```bash
194-
mise run helm:skaffold:delete:sidecar
195-
```
196-
197177
### Delete the cluster entirely
198178

199179
```bash
@@ -292,7 +272,7 @@ SPIFFE JWT-SVIDs for dynamic provider token grants:
292272
`openshell.local` and adds a `ClusterSPIFFEID` that maps sandbox pod
293273
annotations to `spiffe://openshell.local/openshell/sandbox/<sandbox-id>`.
294274
OpenShell mounts the SPIFFE CSI Workload API socket at
295-
`/spiffe-workload-api/spire-agent.sock` into sandbox pods for provider token
275+
`/spiffe-workload-api/spire-agent.sock` only into supervisor Pods for provider token
296276
grants. Supervisor-to-gateway authentication remains on the Kubernetes
297277
ServiceAccount bootstrap and gateway-minted sandbox JWT path; the selected
298278
Kubernetes compute driver validates the projected token before the gateway
@@ -372,7 +352,6 @@ for dependencies still declared in `Chart.yaml`.
372352
| `deploy/helm/openshell/ci/values-gateway.yaml` | Envoy Gateway GRPCRoute + Gateway overlay |
373353
| `deploy/helm/openshell/ci/values-high-availability.yaml` | HA test overlay (`replicaCount: 2` with external PostgreSQL Secret) |
374354
| `deploy/helm/openshell/ci/values-keycloak.yaml` | Keycloak OIDC overlay |
375-
| `deploy/helm/openshell/ci/values-sidecar.yaml` | Supervisor sidecar topology overlay for Kubernetes e2e/dev |
376355
| `deploy/helm/openshell/ci/values-spire.yaml` | SPIFFE/SPIRE provider token grant overlay |
377356
| `deploy/helm/openshell/ci/values-spire-stack.yaml` | SPIRE hardened chart values for local dev |
378357
| `deploy/helm/openshell/ci/values-tls-disabled.yaml` | Lint-only: TLS + auth disabled (reverse-proxy edge termination) |

‎.agents/skills/sbom/SKILL.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ The OpenShell SBOM tooling produces source-tree CycloneDX JSON SBOMs using Syft,
1515

1616
SBOMs are **release artifacts only** -- they are generated on demand and not committed to the repository. Output lands in `deploy/sbom/output/` (gitignored).
1717

18-
Pushed gateway and supervisor images carry an SPDX SBOM and minimal SLSA provenance as OCI attestations. Branch E2E, Release Dev, and Release Tag image binaries embed cargo-auditable metadata, so their image SBOMs include linked Rust crates.
18+
Pushed gateway, sandbox, and supervisor images carry an SPDX SBOM and minimal SLSA provenance as OCI attestations. Branch E2E, Release Dev, and Release Tag image binaries embed cargo-auditable metadata, so their image SBOMs include linked Rust crates.
1919

2020
## Prerequisites
2121

‎.agents/skills/test-release-canary/SKILL.md‎

Lines changed: 12 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -13,18 +13,20 @@ The Release Canary (`.github/workflows/release-canary.yml`) smoke-tests the arti
1313

1414
| Job | Runner | Verifies |
1515
|---|---|---|
16-
| `macos` | `macos-latest-xlarge` | `install.sh` resolves the Homebrew formula, brew installs the cask, and `openshell status` reaches the brew-services–backed local gateway with the VM driver. |
17-
| `ubuntu` | `ubuntu-latest` | `install.sh` installs the Debian package, the post-install systemd user service starts, and `openshell status` reaches the local gateway with the Docker driver. |
18-
| `fedora` | `fedora:latest` container | `install.sh` installs the RPM packages, the local gateway starts under Podman, and `openshell status` succeeds. |
19-
| `ubuntu-snap` | `ubuntu-latest` | Downloads the Snap artifact from Release Dev, installs it with `--dangerous`, connects the required interfaces, and waits up to 30 seconds for the recovered local gateway. |
20-
| `kubernetes` | `ubuntu-latest` + kind | `helm install oci://ghcr.io/nvidia/openshell/helm-chart --version 0.0.0-dev` succeeds in a kind cluster, the gateway pod becomes Ready, port-forward exposes 8080, and the released CLI registers the in-cluster gateway and runs `openshell status` against it. |
16+
| `macos` | `macos-latest-xlarge` | Installs the dev Homebrew artifacts, reaches the VM gateway, and creates, executes in, and deletes a sandbox. |
17+
| `ubuntu` | `ubuntu-latest` | Installs the dev Debian package, reaches the Docker gateway, and creates, executes in, and deletes a sandbox. |
18+
| `fedora` | `fedora:latest` container | Installs the dev RPM packages, reaches the Podman gateway, and creates, executes in, and deletes a sandbox. |
19+
| `ubuntu-snap` | `ubuntu-latest` | Installs the Release Dev Snap, connects its interfaces, reaches the Docker gateway, and creates, executes in, and deletes a sandbox. |
20+
| `kubernetes` | `ubuntu-latest` + kind | Installs the dev Helm chart, reaches the in-cluster gateway, and creates, executes in, and deletes a sandbox using the published runtime images. |
2121

2222
All canary jobs disable anonymous OpenShell telemetry. Host package jobs inject
2323
`OPENSHELL_TELEMETRY_ENABLED=false` through the service environment, and the
2424
Kubernetes job installs with `server.telemetryEnabled=false`, so smoke traffic
2525
does not contribute to product usage metrics.
2626

27-
`install.sh` defaults to the *latest tagged* release — the canary is therefore checking that the most recent public release still installs, not the just-published `dev` build. The `kubernetes` job is the exception: it pins to `0.0.0-dev` chart + `:dev` images.
27+
The workflow sets `OPENSHELL_VERSION=dev`, so every `install.sh` job consumes the
28+
rolling dev release produced by the triggering workflow. Kubernetes pins the
29+
matching `0.0.0-dev` chart and `:dev` images.
2830

2931
The host-package jobs exercise fresh installs, not upgrades from a persisted
3032
schema-v1 gateway config. Validate Homebrew and RPM exact-default migration with
@@ -83,7 +85,7 @@ gh run view <run-id> --log-failed
8385

8486
## Iterating on the canary itself
8587

86-
When you change `release-canary.yml` on a branch, a manual dispatch on that branch tests *your branch's workflow logic* against *main's published artifacts* (`0.0.0-dev` chart, `:dev` images, latest tagged install.sh assets). This is what you want for iterating on the canary — you're validating that the canary still works against known-good artifacts.
88+
When you change `release-canary.yml` on a branch, a manual dispatch on that branch tests *your branch's workflow logic* against *main's published dev artifacts* (`0.0.0-dev` chart, `:dev` images, and the `dev` GitHub release). This is what you want for iterating on the canary — you're validating that the canary still works against known-good artifacts.
8789

8890
Note `install.sh` is pulled from `raw.githubusercontent.com/NVIDIA/OpenShell/${head_sha}/install.sh`, so changes to `install.sh` on your branch *are* exercised even though the binaries it downloads are from the latest public tag.
8991

@@ -92,7 +94,7 @@ Note `install.sh` is pulled from `raw.githubusercontent.com/NVIDIA/OpenShell/${h
9294
`Release Dev` publishes two chart versions for every dev build (see `.github/actions/release-helm-oci/action.yml:89-102`):
9395

9496
- `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev` — floating, overwritten on every main push.
95-
- `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev.<sha>` — immutable, `appVersion` set to the same SHA so it pulls `ghcr.io/nvidia/openshell/gateway:<sha>` and `:supervisor:<sha>`.
97+
- `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev.<sha>` — immutable, `appVersion` set to the same SHA so it pulls the matching `gateway`, `sandbox`, and `supervisor` images.
9698

9799
To smoke-test the chart for a specific dev build, dispatch `Release Dev` on the branch first, then run the kind canary steps locally pointed at the SHA-pinned chart (see "Local kind reproduction" below). The release-canary workflow itself does not currently expose `chart_version` / `image_tag` inputs.
98100

@@ -134,7 +136,8 @@ Loopback registration auto-derives the gateway name to `openshell` if `--name` i
134136

135137
| Symptom | Likely cause | Where to look |
136138
|---|---|---|
137-
| `macos`/`ubuntu`/`fedora` job fails on `install.sh` | Latest tagged release missing an asset, checksum mismatch, or `install.sh` regression on this branch. | Job log around the `curl … install.sh \| sh` step. |
139+
| `macos`/`ubuntu`/`fedora` job fails on `install.sh` | Dev release missing an asset, checksum mismatch, or `install.sh` regression on this branch. | Job log around the `curl … install.sh \| sh` step. |
140+
| Sandbox create or exec fails | Published sandbox and supervisor artifacts are missing, incompatible, or cannot establish the protected runtime channel. | Gateway logs plus Docker, Podman, VM, Snap, or Kubernetes runtime diagnostics for the job. |
138141
| `macos`/`ubuntu`/`fedora` job fails on `openshell status` | Local gateway service did not start (systemd/brew/podman). Often a driver issue. | Service logs in the job log; `OPENSHELL_COMPUTE_DRIVER` env in the "Ensure …" step. |
139142
| `ubuntu-snap` fails after interface connection | The gateway did not recover after Docker became available, or did not become reachable within the 30-second bound. | Failure diagnostics dump Snap service/connection/change state, gateway and snapd journals, Snap logs, and port 17670 listeners. |
140143
| `kubernetes` job fails on `helm install --wait` | Chart did not deploy in 5 min — usually image pull failure or readiness probe failing. | "Diagnostics on failure" step dumps `helm status`, manifest, pod describe, pod logs. |

0 commit comments

Comments
 (0)