Repository navigation
chore: import the Go SDK history from NVIDIA/OpenShell - #3
Merged
Merged
Conversation
…VIDIA/OpenShell#2271) * feat(sdk/go): add Go SDK foundation, types, and sandbox client (A) Add the Go SDK module with the full API contract and a working sandbox client as the first vertical slice. All other resource clients are present as stubs returning Unimplemented errors, to be replaced with real implementations in subsequent PRs. Contents: - Module setup (go.mod, Makefile, mise.toml) - All domain types (types/ package) - Full ClientInterface with all sub-client accessors - Shared infrastructure (errors, auth, gRPC connection, logging) - Sandbox client with converter and tests (fully functional) - Stub clients for remaining resources (exec, file, health, provider, profile, config, refresh, policy, service, ssh, tcp) Part of the Go SDK decomposition plan (NVIDIA/OpenShell#2270). Implements NVIDIA/OpenShell#2044. * fix(sdk/go): address review feedback on PR NVIDIA/OpenShell#2271 - Make scheme parsing drive transport selection: http:// uses plaintext gRPC, https:// or no scheme uses TLS. Add regression tests. - Add Resources and DriverConfig fields to SandboxTemplate and update both converter directions (SandboxFromProto/SandboxSpecToProto). - Regenerate proto bindings from current canonical proto sources to eliminate drift (SigV4/MCP fields, params matchers, reserved fields). - Run gofmt/goimports on all handwritten Go files. Signed-off-by: Roland Huß <[email protected]> * fix(sdk/go): address principal engineer review findings - Remove dead boolCount function that would fail golangci-lint (NVIDIA/OpenShell#1) - Emit EventAdded for the first watch event instead of EventModified, matching k8s watch semantics (NVIDIA/OpenShell#7) - Add mutex locking to all mock server methods that access the shared sandboxes map, fixing latent race conditions (NVIDIA/OpenShell#12) - Skip HealthCheck integration test that calls an unimplemented stub (NVIDIA/OpenShell#13) - Scope doc.go examples: mark sections for sub-clients not yet available in this PR with "available in a future release" (NVIDIA/OpenShell#4) - Document Config.Timeout/RetryPolicy/Logger and WatchOptions fields as reserved for future use (NVIDIA/OpenShell#2, NVIDIA/OpenShell#6) Signed-off-by: Roland Huß <[email protected]> * refactor(sdk/go): migrate mise config to centralized task include Move Go SDK mise configuration from standalone sdk/go/mise.toml into the project's centralized pattern: - Add Go tools (go, golangci-lint, protoc-gen-go, protoc-gen-go-grpc) to root mise.toml [tools] section - Create tasks/go.toml with all SDK tasks using go: namespace prefix and dir=sdk/go for working directory - Update sdk/go/Makefile to reference namespaced task names - Update proto:sync default path for monorepo layout Addresses review feedback from drew on PR NVIDIA/OpenShell#2271 regarding mise convention alignment. Signed-off-by: Roland Huß <[email protected]> * refactor(sdk/go): remove UPSTREAM_VERSION standalone repo artifact Remove sdk/go/proto/UPSTREAM_VERSION file and its exclusion from proto:check. This was a leftover from the standalone repo prototype. In a monorepo, proto drift is detectable via git diff between sdk/go/proto/ and proto/ directly. Signed-off-by: Roland Huß <[email protected]> * refactor(sdk/go): switch proto generation from protoc to buf Replace raw protoc invocations with buf for Go SDK proto code generation, aligning with the TS SDK approach (PR NVIDIA/OpenShell#2122). - Add repo-level buf.yaml declaring proto/ as the buf module with lint and breaking change detection config - Add sdk/go/buf.gen.yaml configuring buf to generate Go code directly from root proto/ (no more vendored .proto copies) - Delete vendored .proto source files from sdk/go/proto/ - Rewrite go:proto:gen and go:proto:check mise tasks to use buf - Remove go:proto:sync and go:proto:clean tasks (no longer needed) - Add proto target to sdk/go/Makefile - Add buf 1.72.0 to root mise.toml tool dependencies - Include options.proto in generation (was stripped from vendored copies) - Regenerate all .pb.go files via the new buf pipeline Signed-off-by: Roland Huß <[email protected]> * test(sdk/go): add proto-converter field coverage detection Use protobuf reflection to enumerate all fields on key proto messages (SandboxSpec, SandboxTemplate, SandboxStatus, SandboxCondition, SandboxPolicy) and compare against explicit handled/skipped sets in the converter tests. Unhandled fields produce warnings (t.Log), not failures, so proto contributors are not forced to fix SDK converters in the same PR. Stale entries in the handled set (removed proto fields) do fail, since they indicate the converter references something that no longer exists. A follow-up CI workflow will create GitHub issues when converter drift lands on main. Signed-off-by: Roland Huß <[email protected]> * fix(sdk/go): bump Go to 1.26 and fix errcheck lint violations The upstream go.mod now has `toolchain go1.26.4`, which requires Go 1.26 to build golangci-lint. Bump the mise.toml Go version from 1.25 to 1.26 and wrap deferred Close() calls in test helpers to satisfy errcheck. Assisted-By: 🤖 Claude Code * feat(sdk/go): add ObjectMeta fields (annotations, workspace, deletion_timestamp) Add three new proto ObjectMeta fields to Sandbox and Provider domain types: Annotations (map), Workspace (string), and DeletionTimestamp (*time.Time). Update converters in both directions, deep-copy maps at the proto/SDK boundary, and add TimeFromMillisPtr/MillisFromTimePtr helper functions. Assisted-By: 🤖 Claude Code * chore(sdk/go): regenerate proto bindings after rebase Pick up workspace fields from upstream PR NVIDIA/OpenShell#2445 (Wire authorization into workspace model). All request messages now include workspace parameter in the generated Go bindings. Assisted-By: 🤖 Claude Code * feat(sdk/go): add workspace scoping to all RPC interfaces Add workspace parameter to every sandbox-scoped RPC method across all interfaces (Sandbox, Exec, File, Service, SSH, TCP, Config, Policy, Provider, Profile, Refresh). The workspace string is passed as the second parameter after ctx, following the convention workspace then resource-name. Key changes: - SandboxInterface: all 10 methods gain workspace parameter - sandbox_client.go: passes Workspace field in every proto request - ListOptions: add AllWorkspaces field for cross-workspace queries - All stub interfaces updated to match new signatures - All sandbox client tests updated with "default" workspace Assisted-By: 🤖 Claude Code * chore(sdk/go): remove coverage.out from tracking Assisted-By: 🤖 Claude Code * fix(sdk/go): address review feedback from mrunalp - Add RefreshStrategyAWSStsAssumeRole to match proto enum value 6, fulfilling the "all domain types upfront" contract - Wrap context.DeadlineExceeded and context.Canceled in StatusError so IsDeadlineExceeded() and IsCancelled() helpers work correctly - Return error from mapToStruct/SandboxSpecToProto instead of silently discarding structpb.NewStruct failures on invalid template maps Signed-off-by: Roland Huss <[email protected]> * fix(sdk/go): address remaining review items - Wire go:ci into root ci task so SDK is tested in repository CI - Fix gofmt formatting on converter files - Add goimports to mise.toml tools - Add coverage.out to .gitignore - Add Go SDK section to AGENTS.md and CONTRIBUTING.md - Add regression tests for context-error wrapping (IsDeadlineExceeded, IsCancelled) and invalid template map rejection - Remove panic from SandboxToProto, return error instead Signed-off-by: Roland Huss <[email protected]> * fix(sdk/go): pin goimports version and update lockfile Pin goimports to 0.48.0 instead of "latest" and regenerate mise.lock to include the new entry. Signed-off-by: Roland Huss <[email protected]> * fix(sdk/go): TLS.Insecure means skip-verify, not plaintext Align TLS.Insecure semantics with the Rust SDK: Insecure: true now uses TLS with InsecureSkipVerify (skip cert verification) instead of switching to plaintext. Only the http:// scheme triggers plaintext. This fixes token auth against dev/k3d gateways: StaticToken and RefreshableToken require transport security, which real TLS (even with InsecureSkipVerify) satisfies, but plaintext does not. For http:// + token auth (dev gateways without TLS), wrap the auth provider to override RequireTransportSecurity, matching the Rust SDK's behavior where http:// accepts any auth mode. Transport decision table (matches Rust SDK crates/openshell-sdk): http:// + any TLS config -> plaintext (TLS config ignored) https:// + Insecure: true -> TLS, skip cert verify https:// + Insecure: false -> TLS, full verification no scheme -> same as https:// Signed-off-by: Roland Huss <[email protected]> * feat(sdk/go): add missing policy proto fields Add 6 previously silently dropped fields to the network policy types and converters, preventing security-relevant data loss on round-trip: NetworkEndpoint fields 19-23: - CredentialSigning: SigV4 re-signing mode - SigningService: AWS service name for SigV4 - SigningRegion: AWS region override for SigV4 - JsonRpcMaxBodyBytes: JSON-RPC body inspection limit - Mcp: MCP-specific policy options (new McpOptions type) L7Allow and L7DenyRule field 9: - Params: MCP params matcher map for tools/call filtering New type McpOptions with StrictToolNames and AllowAllKnownMcpMethods optional booleans matching the proto definitions. Signed-off-by: Roland Huss <[email protected]> * fix(sdk/go): enforce coverage test and extend to policy messages Change coverage_test.go from t.Logf (silent) to t.Errorf so that unhandled proto fields fail the test immediately. Add coverage tests for NetworkEndpoint (23 fields), L7Allow (8 fields), L7DenyRule (8 fields), and McpOptions (2 fields). Any new proto field that is not in the handled set or explicitly skipped now breaks the build, closing the silent-drift gap. Signed-off-by: Roland Huss <[email protected]> * ci(sdk/go): add Go SDK job to branch-checks workflow Add a Go SDK job to branch-checks.yml that runs mise run go:ci (lint, build, test, proto-check, docs-check) on every PR. This ensures the SDK is tested in CI, not just locally. Signed-off-by: Roland Huss <[email protected]> * fix(sdk/go): address should-fix review items NVIDIA/OpenShell#6 Fix broken godoc examples: add workspace parameter to all method calls in doc.go that were broken after workspace scoping. NVIDIA/OpenShell#7 Add Err field to Event[T]: Watch error events now carry the underlying error instead of discarding it. NVIDIA/OpenShell#8 Separate Unauthenticated from PermissionDenied: add ErrorUnauthenticated code and IsUnauthenticated() helper. gRPC Unauthenticated (401) now maps to its own code instead of collapsing into PermissionDenied (403). NVIDIA/OpenShell#9 Add Unwrap to StatusError: replace dead Details field with Cause error field. StatusError.Unwrap() returns Cause, enabling errors.Is/As unwrapping. FromGRPCError and contextError both populate Cause. Signed-off-by: Roland Huss <[email protected]> * ci(sdk/go): add go:format:check to CI pipeline Add gofmt format verification to go:ci. Catches unformatted Go files before they reach the PR. Fix formatting on coverage_test.go. Signed-off-by: Roland Huss <[email protected]> * chore(sdk/go): remove Makefile in favor of mise tasks All build, lint, test, and proto-gen tasks are already defined in tasks/go.toml and invoked via mise. The Makefile was a leftover that duplicated this and raised questions in review. Signed-off-by: Roland Huß <[email protected]> * feat(sdk/go): sync proto bindings and add credential handle support Regenerate Go proto bindings after rebase to pick up new CredentialHandle message and Provider.credential_handles and profile_workspace fields from upstream. Add domain types, converter support, and proto field coverage tests for Provider and CredentialHandle. Signed-off-by: Roland Huß <[email protected]> * fix(sdk/go): reject plaintext auth leak and fix watch error handling Reject http:// addresses when the auth provider requires transport security instead of silently stripping the requirement. Remove the insecureAuthWrapper that overrode RequireTransportSecurity. Fix watch stream error handling: use blocking send for terminal errors so they are never silently dropped when the channel is full, and wrap mid-stream errors with converter.FromGRPCError so SDK error helpers like IsUnavailable work on watch Event.Err. Signed-off-by: Roland Huß <[email protected]> * fix(sdk/go): address review findings from multi-agent code review - WaitReady now detects SandboxDeleting phase and returns immediately instead of polling indefinitely - Watch goroutine defers streamCancel() to prevent context leaks - Fix StopOnTerminal=false test to keep stream open (was wrong-reason pass due to stream ending, not StopOnTerminal logic) - Add EventDeleted test covering the Deleting phase branch - Add provider converter unit tests for CredentialHandle round-trip, nil handling, and empty maps Signed-off-by: Roland Huß <[email protected]> --------- Signed-off-by: Roland Huß <[email protected]> Signed-off-by: Roland Huss <[email protected]>
…enShell#2510) * feat(proxy): bind static credentials to provider endpoints Signed-off-by: John Myers <[email protected]> * test(e2e): verify static credential endpoint isolation Signed-off-by: John Myers <[email protected]> * docs(provider): explain static credential endpoint binding Signed-off-by: John Myers <[email protected]> * fix(e2e): use valid endpoint isolation fixtures Signed-off-by: John Myers <[email protected]> * docs(provider): explain static credential endpoint binding Signed-off-by: John Myers <[email protected]> * fix(credentials): preserve binding identity across rotations Signed-off-by: John Myers <[email protected]> * fix(proxy): enforce bindings across request lifecycle Signed-off-by: John Myers <[email protected]> * fix(proxy): close credential relay gaps Signed-off-by: John Myers <[email protected]> * docs(credentials): clarify binding failure behavior Signed-off-by: John Myers <[email protected]> * fix(credentials): hash selected provider profile scope Signed-off-by: John Myers <[email protected]> * fix(proxy): resolve credentials after request admission Signed-off-by: John Myers <[email protected]> * docs(credentials): clarify binding failure diagnostics Signed-off-by: John Myers <[email protected]> * fix(proxy): align single-route credential denials Signed-off-by: John Myers <[email protected]> * fix(credentials): harden endpoint-bound rotation Signed-off-by: John Myers <[email protected]> * fix(credentials): enforce identity and authority binding Signed-off-by: John Myers <[email protected]> * fix(credentials): snapshot provider environment atomically Signed-off-by: John Myers <[email protected]> * test(e2e): include authority port in query proxy requests Signed-off-by: John Myers <[email protected]> * fix(credentials): close credential revocation gaps Signed-off-by: John Myers <[email protected]> * docs(proxy): explain authority mismatch diagnostics Signed-off-by: John Myers <[email protected]> * fix(credentials): enforce binding lifecycle invariants Signed-off-by: John Myers <[email protected]> * fix(provider): reject credential config collisions Signed-off-by: John Myers <[email protected]> * fix(network): capture credential scope atomically Signed-off-by: John Myers <[email protected]> * fix(network): distinguish origin and absolute targets Signed-off-by: John Myers <[email protected]> * fix(provider): isolate endpointless profile credentials Signed-off-by: John Myers <[email protected]> * fix(network): normalize IPv6 request authorities Signed-off-by: John Myers <[email protected]> * docs(credentials): clarify endpointless profile isolation Signed-off-by: John Myers <[email protected]> * feat(policy): bind endpointless provider credentials Signed-off-by: John Myers <[email protected]> * fix(credentials): use current GCP placeholder revision Signed-off-by: John Myers <[email protected]> * docs(providers): explain policy credential bindings Signed-off-by: John Myers <[email protected]> * test(credentials): cover endpointless fail-closed invariant Signed-off-by: John Myers <[email protected]> * test(policy): expect ambiguity rejection at creation Signed-off-by: John Myers <[email protected]> * test(server): authenticate rebased policy requests Signed-off-by: John Myers <[email protected]> * refactor(proxy): share credential mismatch finding builder Signed-off-by: John Myers <[email protected]> * test(credentials): cover malformed binding metadata Signed-off-by: John Myers <[email protected]> * test(credentials): verify multi-key endpoint isolation Signed-off-by: John Myers <[email protected]> * test(e2e): cover same-host credential path denial Signed-off-by: John Myers <[email protected]> * docs(credentials): document serialized refresh contract Signed-off-by: John Myers <[email protected]> * refactor(proxy): consolidate L7 log formatting Signed-off-by: John Myers <[email protected]> * perf(credentials): precompile endpoint binding patterns Signed-off-by: John Myers <[email protected]> * perf(credentials): share identity epoch revisions Signed-off-by: John Myers <[email protected]> * test(proxy): require explicit request default ports Signed-off-by: John Myers <[email protected]> * fix(policy): validate SigV4 credential sources Signed-off-by: John Myers <[email protected]> * fix(credentials): preserve endpoint bindings for credential handles Signed-off-by: John Myers <[email protected]> * feat(go-sdk): expose network credential bindings Signed-off-by: John Myers <[email protected]> --------- Signed-off-by: John Myers <[email protected]> Signed-off-by: John Myers <[email protected]> Co-authored-by: John Myers <[email protected]>
* feat(sandbox): add suspend and resume operations Signed-off-by: Seth Jennings <[email protected]> * fix(server): preserve lifecycle work after cancellation Signed-off-by: Seth Jennings <[email protected]> * fix(server): reconcile ambiguous lifecycle outcomes Signed-off-by: Seth Jennings <[email protected]> * fix(server): complete suspended session cleanup Signed-off-by: Seth Jennings <[email protected]> * fix(vm): preserve suspension state on resume failure Signed-off-by: Seth Jennings <[email protected]> * fix(server): retry retained lifecycle transitions Signed-off-by: Seth Jennings <[email protected]> * fix(server): clean sessions after suspend reconciliation Signed-off-by: Seth Jennings <[email protected]> * test(sandbox): cover deleting suspended sandbox Signed-off-by: Seth Jennings <[email protected]> * fix(kubernetes): preserve progressing sandbox suspension Signed-off-by: Seth Jennings <[email protected]> * fix(kubernetes): bound suspend status polling Signed-off-by: Seth Jennings <[email protected]> * fix(kubernetes): detect legacy sandbox suspension Signed-off-by: Seth Jennings <[email protected]> * fix(tui): render suspended sandbox phases Signed-off-by: Seth Jennings <[email protected]> * refactor(sandbox): rename suspend and resume lifecycle Signed-off-by: Seth Jennings <[email protected]> * perf(server): clean stopped sessions on transition Signed-off-by: Seth Jennings <[email protected]> * fix(kubernetes): fail fast on rejected stop Signed-off-by: Seth Jennings <[email protected]> * fix(compute): fence stale restart lifecycle events Signed-off-by: Seth Jennings <[email protected]> --------- Signed-off-by: Seth Jennings <[email protected]>
NVIDIA/OpenShell#2702) * feat(sdk/go): complete Go SDK with domain clients, auth, and hardening Add the Go SDK implementation covering all gateway RPC services with domain-typed clients, comprehensive OIDC authentication flows, fake test doubles, and proto converters. Domain clients: Sandbox, Provider, Exec, File, TCP, SSH, Policy, Profile, Health, Service, Config, Workspace, Inference, Refresh. Each client validates inputs, resolves sandboxes by name, and converts between domain types and proto at the boundary. Auth: OIDC authorization code (PKCE), device code (RFC 8628), and client credentials (RFC 6749 Section 4.4) flows with gateway config auto-resolution. Token refresh with singleflight deduplication and exponential backoff. Edge tunnel proxy for gRPC-over-WebSocket. Gateway: On-disk gateway discovery with user/system directory precedence, lazy token loading, and auth mode mapping. Testing: In-memory fake client with deep-copy isolation, watch broadcasting with filtering, and workspace-scoped object stores. Bufconn-based gRPC tests for all domain clients. Ref: NVIDIA/OpenShell#2044 Signed-off-by: Roland Huß <[email protected]> * feat(sdk/go): add gateway client options for logger, timeout, and retry Add WithLogger, WithTimeout, and WithRetryPolicy options to the gateway package's NewClient function. These forward the corresponding Config fields (Logger, Timeout, RetryPolicy) through to the underlying SDK client, giving callers full control over observability, connection timeouts, and retry behavior when constructing clients from on-disk gateway configurations. Also adds RetryPolicy type to types package and extends Config with the three new fields. * fix(mise): sync sccache lock entry Signed-off-by: Drew Newberry <[email protected]> --------- Signed-off-by: Roland Huß <[email protected]> Signed-off-by: Drew Newberry <[email protected]> Co-authored-by: Drew Newberry <[email protected]>
Use the same compact persona, workflow, impact, reproduction, and environment prompts for bug reports and feature requests. Keep logs optional and specific to bug reports. Remove filing-time agent diagnostics so maintainers can evaluate user needs apart from investigation output, which becomes stale over time. Require contributors to investigate current behavior after humans accept the work, and treat state:accepted or roadmap placement as that signal. Signed-off-by: Kris Hicks <[email protected]>
* fix(security): authenticate extension services Signed-off-by: Piotr Mlocek <[email protected]> * feat(extension-core): verify gateway JWTs Signed-off-by: Piotr Mlocek <[email protected]> * refactor(extension-core): keep inbound verification external This should become an extension SDK package. Signed-off-by: Piotr Mlocek <[email protected]> * fix(security): harden the extension authentication contract Follow-up hardening on the alpha extension authentication mechanism. Claim contract: - Extension tokens carry an explicit `typ` of `openshell-ext+jwt`. They share a signing key with sandbox-to-gateway admission tokens and were otherwise separated by audience alone, so a verifier that neglects to check `aud` could accept a gateway credential. The header is a second, independent discriminator. - Publish OIDC-shaped discovery at `/.well-known/openid-configuration` so a service configured with only the gateway URL can learn the exact expected issuer and the JWKS location. It is shaped, not compliant: `issuer` is the gateway identity, not the serving URL. Audience agreement: - `MiddlewareManifest` and `InterceptorManifest` gain `expected_audience`. The audience is otherwise configured independently on each side of the boundary, where a mismatch surfaces only as an opaque authentication failure on every call. OpenShell now compares the two and fails at startup. An empty field keeps the check off for existing services. Compatibility: - Add `allow_insecure_transport` per registration. Enabling gateway JWT signing previously made any plaintext endpoint a hard startup failure, including the endpoint form used in our own documentation. The opt-out attaches no credential, is refused by the gateway if a supervisor asks for one, and warns at every startup. - Make the transport requirement kind-aware. A middleware endpoint must be reachable from every sandbox supervisor, so only interceptors may use a gateway-local Unix socket. Credential lifecycle: - Replace the process-global slot map with a supervisor-owned `ExtensionCredentialStore` shared explicitly across the gateway connections the supervisor opens, removing test-order coupling. - Rotate only when a credential is missing or has passed four fifths of its lifetime. Configuration polling ran every ten seconds against fifteen-minute credentials, so each poll re-ran gateway effective-policy resolution and re-minted the gateway token. - Bound credential minting per sandbox, since each request resolves the caller's effective policy. Signed-off-by: Piotr Mlocek <[email protected]> * docs: record alpha extension authentication in RFC appendices Restore the RFC 0009 and 0010 bodies to their accepted text and move every extension-authentication update into appendices instead. An RFC records a decision at a point in time; superseding detail belongs alongside it rather than rewritten into it. RFC 0009's appendix carries the shared contract: claims, authorization, key distribution, the `allow_insecure_transport` replacement for the body's `allow_insecure`, and residual risks. RFC 0010's records only what differs for interceptors and links to it. The existing protocol-extensions appendix, which parked the phase 2 transport question, now points forward to what was built. Also document the audience handshake, the discovery endpoint, the `typ` requirement, and `jti` replay guidance in the extensibility and gateway configuration pages, and correct the middleware transport guidance: middleware endpoints must be reachable from sandbox supervisors, so Unix sockets are not an option there. Signed-off-by: Piotr Mlocek <[email protected]> * refactor(extension-core): abstract extension server trust Signed-off-by: Piotr Mlocek <[email protected]> * docs(core): update middleware manifest example Signed-off-by: Piotr Mlocek <[email protected]> * fix(extension-auth): preserve unsigned gateway compatibility Signed-off-by: Piotr Mlocek <[email protected]> * fix(extension-auth): reject cross-domain token replay Signed-off-by: Piotr Mlocek <[email protected]> --------- Signed-off-by: Piotr Mlocek <[email protected]>
) * feat(middleware): inspect websocket text messages Signed-off-by: Piotr Mlocek <[email protected]> * fix(middleware): address websocket review feedback Signed-off-by: Piotr Mlocek <[email protected]> * fix(network): bound websocket message assembly Signed-off-by: Piotr Mlocek <[email protected]> * fix(network): harden websocket upgrade lifecycle Signed-off-by: Piotr Mlocek <[email protected]> * refactor(middleware): unify in-process and remote transports Signed-off-by: Piotr Mlocek <[email protected]> * feat(middleware): support regex websocket redaction Signed-off-by: Piotr Mlocek <[email protected]> * fix(middleware): bound persistent streaming sessions Signed-off-by: Piotr Mlocek <[email protected]> * fix(middleware): accept websocket sequence gaps Signed-off-by: Piotr Mlocek <[email protected]> * refactor(middleware): refine websocket introspection contract Signed-off-by: Piotr Mlocek <[email protected]> * fix(middleware): clarify websocket preflight lifecycle Signed-off-by: Piotr Mlocek <[email protected]> * fix(middleware): clarify websocket coverage semantics Signed-off-by: Piotr Mlocek <[email protected]> * fix(network): type websocket frame failures Signed-off-by: Piotr Mlocek <[email protected]> * fix(network): return 503 when middleware admission is exhausted Signed-off-by: Piotr Mlocek <[email protected]> * fix(middleware): align streaming API contract Signed-off-by: Piotr Mlocek <[email protected]> * fix(middleware): clarify WebSocket event result scope Signed-off-by: Piotr Mlocek <[email protected]> * docs(rfc): simplify middleware revision history Signed-off-by: Piotr Mlocek <[email protected]> * feat(examples): add WebSocket content guard support Signed-off-by: Piotr Mlocek <[email protected]> * fix(middleware): unify binding payload limits Signed-off-by: Piotr Mlocek <[email protected]> * refactor(middleware): align payload limit terminology Signed-off-by: Piotr Mlocek <[email protected]> * fix(middleware): address websocket review feedback Signed-off-by: Piotr Mlocek <[email protected]> * fix(network): address websocket review findings Signed-off-by: Piotr Mlocek <[email protected]> * test(network): allow Linux handler setup in preflight regression Signed-off-by: Piotr Mlocek <[email protected]> * fix(network): harden websocket relay finalization Signed-off-by: Piotr Mlocek <[email protected]> * fix(network): inspect compressed websocket messages Signed-off-by: Piotr Mlocek <[email protected]> * test(network): stabilize compressed websocket regressions Signed-off-by: Piotr Mlocek <[email protected]> * fix(go-sdk): regenerate middleware protobuf binding Signed-off-by: Piotr Mlocek <[email protected]> * fix(middleware): clarify websocket skip lifecycle Signed-off-by: Piotr Mlocek <[email protected]> * fix(middleware): address WebSocket review feedback Signed-off-by: Piotr Mlocek <[email protected]> --------- Signed-off-by: Piotr Mlocek <[email protected]>
Test coverage artifact was accidentally committed. Already in .gitignore. Signed-off-by: Roland Huß <[email protected]>
…ell#2780) * fix(providers): keep refresh credential handles stable Signed-off-by: Mrunal Patel <[email protected]> * fix(providers): protect refresh-owned credentials Signed-off-by: Mrunal Patel <[email protected]> --------- Signed-off-by: Mrunal Patel <[email protected]>
…l#2493) * fix(policy): gate uninspected credentialed endpoints Signed-off-by: Adrien Langou <[email protected]> * refactor(cli): extract allowed-ip option parsing Signed-off-by: Adrien Langou <[email protected]> * fix(policy): gate endpointless credential bindings Signed-off-by: Adrien Langou <[email protected]> --------- Signed-off-by: Adrien Langou <[email protected]>
…DIA/OpenShell#2801) * feat(providers): store refresh credentials in credential drivers Signed-off-by: Mrunal Patel <[email protected]> * fix(providers): harden refresh credential lifecycle Signed-off-by: Mrunal Patel <[email protected]> * fix(providers): migrate legacy refresh secrets before skip Signed-off-by: Mrunal Patel <[email protected]> * refactor(providers): defer credential migration Signed-off-by: Mrunal Patel <[email protected]> * fix(providers): make refresh configuration atomic Signed-off-by: Mrunal Patel <[email protected]> --------- Signed-off-by: Mrunal Patel <[email protected]>
…l#2711) * feat(policy): accept explicit tcp endpoint protocol Signed-off-by: John Myers <[email protected]> * refactor(network): snapshot authoritative egress decisions Signed-off-by: John Myers <[email protected]> * docs(policy): document explicit tcp protocol Signed-off-by: John Myers <[email protected]> * docs(policy): defer transparent TCP release guidance Signed-off-by: John Myers <[email protected]> * chore(go): regenerate sandbox protobuf bindings Signed-off-by: John Myers <[email protected]> * fix(network): complete tcp egress foundation Signed-off-by: John Myers <[email protected]> * docs(policy): document explicit tcp contract Signed-off-by: John Myers <[email protected]> * fix(network): fail closed on authorization errors Signed-off-by: John Myers <[email protected]> * fix(podman): fence delayed exit events before restart Signed-off-by: John Myers <[email protected]> * fix(policy): validate network endpoint destinations Signed-off-by: John Myers <[email protected]> * test(providers): opt in tcp credential fixture Signed-off-by: John Myers <[email protected]> * fix(policy): require dns host for transparent tcp Signed-off-by: John Myers <[email protected]> --------- Signed-off-by: John Myers <[email protected]>
* feat(sandbox): add canonical main process Closes NVIDIA/OpenShell#2710 Persist and supervise one canonical workload per sandbox, attach sandbox connect to its retained session, and make every unexpected main-process exit terminal. Signed-off-by: Drew Newberry <[email protected]> * feat(sandbox): simplify canonical main process contract Signed-off-by: Drew Newberry <[email protected]> * fix(sandbox): preserve legacy VM main compatibility Signed-off-by: Drew Newberry <[email protected]> * fix(sandbox): preserve main status across driver updates Signed-off-by: Drew Newberry <[email protected]> * fix(sandbox): satisfy macOS process lint Signed-off-by: Drew Newberry <[email protected]> * fix(sandbox): gate Linux exit acknowledgement publisher Signed-off-by: Drew Newberry <[email protected]> * refactor(sandbox): simplify main process plumbing Signed-off-by: Drew Newberry <[email protected]> * fix(supervisor): make controlling tty ioctl portable Signed-off-by: Drew Newberry <[email protected]> * fix(supervisor): initialize canonical process environment Signed-off-by: Drew Newberry <[email protected]> * perf(supervisor): optimize retained main session Signed-off-by: Drew Newberry <[email protected]> * feat(sandbox): detach main session on ctrl-c Signed-off-by: Drew Newberry <[email protected]> * fix(sandbox): use explicit main detach keys Signed-off-by: Drew Newberry <[email protected]> * fix(dev): atomically stage Docker supervisor Signed-off-by: Drew Newberry <[email protected]> * fix(test): align Docker main environment assertion Signed-off-by: Drew Newberry <[email protected]> * feat(sdk): expose canonical main process fields Signed-off-by: Drew Newberry <[email protected]> --------- Signed-off-by: Drew Newberry <[email protected]>
…ell#2850) * fix(policy): bind reviews to applicable candidates Build and validate the exact effective-policy candidate before approval, bind review to live policy/provider/credential inputs, and preserve inspected endpoint contracts during mechanistic expansion. Closes NVIDIA/OpenShell#2821 Signed-off-by: John Myers <[email protected]> * fix(policy): canonicalize advisor review inputs Serialize nested protobuf maps in stable key order for proposal review tokens and effective-policy hashes. Narrow reused multi-port endpoint contracts to the denied port so advisor proposals cannot widen binary access. Add regressions for both cases. Signed-off-by: John Myers <[email protected]> * test(e2e): keep advisor sandbox running Create the issue 2821 regression sandbox detached with a durable canonical main process so policy denial, approval, and hot-reload checks run before lifecycle exit. Signed-off-by: John Myers <[email protected]> * fix(policy): apply reviewed draft batches atomically Signed-off-by: John Myers <[email protected]> --------- Signed-off-by: John Myers <[email protected]> Co-authored-by: John Myers <[email protected]>
…Shell#1970) * feat(provider): add ability to request token exchange instead of client credentials as OAuth grant_type Signed-off-by: Gordon Sim <[email protected]> * test(proxy): add further tests for token exchange Signed-off-by: Gordon Sim <[email protected]> * test(provider): add runnable example for token exchange Signed-off-by: Gordon Sim <[email protected]> * test(e2e): cover Podman token exchange grants Signed-off-by: Gordon Sim <[email protected]> * refactor(oauth): extract duplicated functionality from server and supervisor Signed-off-by: Gordon Sim <[email protected]> * fix(provider): evict nearest-to-expiry entry from intermediate token cache Signed-off-by: Gordon Sim <[email protected]> * doc(supervisor): add podman example for token exchange Signed-off-by: Gordon Sim <[email protected]> * fix(provider): withhold token-exchange subject credentials Signed-off-by: Gordon Sim <[email protected]> --------- Signed-off-by: Gordon Sim <[email protected]>
…hell#2907) * feat(sdk): add renewable client credentials auth Implement lazy OAuth client-credentials acquisition and renewal for the Python, TypeScript, and Go SDK clients, with shared security conformance coverage and service-account documentation.\n\nCloses NVIDIA/OpenShell#2803 Signed-off-by: Seth Jennings <[email protected]> * fix(sdk): address client credentials review Signed-off-by: Seth Jennings <[email protected]> * fix(sdk): address client credentials review Signed-off-by: Seth Jennings <[email protected]> --------- Signed-off-by: Seth Jennings <[email protected]>
…nShell#2887) * fix(providers): classify OAuth refresh failures Signed-off-by: Mrunal Patel <[email protected]> * test(providers): add Keycloak refresh e2e lane Signed-off-by: Mrunal Patel <[email protected]> * fix(providers): harden OAuth refresh recovery Signed-off-by: Mrunal Patel <[email protected]> * fix(providers): classify post-mint refresh failures Signed-off-by: Mrunal Patel <[email protected]> * fix(providers): tolerate malformed OAuth subtypes Signed-off-by: Mrunal Patel <[email protected]> --------- Signed-off-by: Mrunal Patel <[email protected]>
…#2985) Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.81.1 to 1.82.1. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.81.1...v1.82.1) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-version: 1.82.1 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ec (NVIDIA/OpenShell#2852) * feat(sandbox): add --no-login-shell to skip shell startup files on exec Signed-off-by: Artem Lytvyn <[email protected]> * chore(sdk/go): regenerate proto bindings for no_login_shell Signed-off-by: Artem Lytvyn <[email protected]> * test(supervisor-process): cover login-shell flag selection Signed-off-by: Artem Lytvyn <[email protected]> * feat(sandbox): gate --no-login-shell on supervisor SSH banner Signed-off-by: Artem Lytvyn <[email protected]> --------- Signed-off-by: Artem Lytvyn <[email protected]>
* feat(providers): make profiles authoritative Closes NVIDIA/OpenShell#1988 Signed-off-by: John Myers <[email protected]> * docs(providers): move profiles into provider navigation Signed-off-by: John Myers <[email protected]> * docs(providers): clarify provider attachment lifecycle Signed-off-by: John Myers <[email protected]> * fix(tui): scroll provider profile picker Signed-off-by: John Myers <[email protected]> * fix(providers): honor profile credential semantics Signed-off-by: John Myers <[email protected]> * fix(providers): prefer exact profile IDs Signed-off-by: John Myers <[email protected]> * fix(providers): harden authoritative profile adoption Signed-off-by: John Myers <[email protected]> * test(oidc): align provider fixtures with profiles Signed-off-by: John Myers <[email protected]> * fix(providers): preserve authoritative profile lifecycle Signed-off-by: John Myers <[email protected]> --------- Signed-off-by: John Myers <[email protected]> Co-authored-by: John Myers <[email protected]>
* feat(server): add sandbox workload templates Signed-off-by: Gordon Sim <[email protected]> * feat(go-sdk): add sandbox workload template support Signed-off-by: Gordon Sim <[email protected]> * feat(rust-sdk): add sandbox workload template support Signed-off-by: Gordon Sim <[email protected]> * feat(python-sdk): add sandbox workload template support Signed-off-by: Gordon Sim <[email protected]> * feat(typescript-sdk): add sandbox workload template support Signed-off-by: Gordon Sim <[email protected]> * docs(agents): document sandbox workload templates Signed-off-by: Gordon Sim <[email protected]> * fix(cli): support default GPU requests in sandbox templates Signed-off-by: Gordon Sim <[email protected]> * fix(server): cap sandbox templates per workspace Signed-off-by: Gordon Sim <[email protected]> * docs(architecture): document sandbox workload template boundaries Signed-off-by: Gordon Sim <[email protected]> * feat(cli+sdk): expose sandbox workload template provenance Signed-off-by: Gordon Sim <[email protected]> * fix(cli): include sandbox template annotations in output Signed-off-by: Gordon Sim <[email protected]> * feat(sandbox): add label selectors to template listing Signed-off-by: Gordon Sim <[email protected]> * test(e2e): cover sandbox template failure paths Signed-off-by: Gordon Sim <[email protected]> * fix(server): preserve command and ttl when creating sandbox from template Signed-off-by: Gordon Sim <[email protected]> * test(server): add field coverage test for template merge Signed-off-by: Gordon Sim <[email protected]> * fix(go-sdk): add pagination support to fake client Signed-off-by: Gordon Sim <[email protected]> * fix(cli): warn on env vars that looks like secrets Signed-off-by: Gordon Sim <[email protected]> * fix(sdk-ts): propagate sandbox workspace through lifecycle calls Signed-off-by: Gordon Sim <[email protected]> * fix(docs): update workspace management docs Signed-off-by: Gordon Sim <[email protected]> * fix(ts-sdk): support command and tty when creating from template Signed-off-by: Gordon Sim <[email protected]> * fix(go-sdk): support command and tty when creating from template Signed-off-by: Gordon Sim <[email protected]> * test(python-sdk): verify command and tty handling when creating from template Signed-off-by: Gordon Sim <[email protected]> * fix(go-sdk): update docs and ClientInterface Signed-off-by: Gordon Sim <[email protected]> * fix(server): validate sandbox create specs before I/O Signed-off-by: Gordon Sim <[email protected]> * fix(go-sdk): guard empty DNS-1123 label validation Signed-off-by: Gordon Sim <[email protected]> * fix(python-sdk): allow empty template builder mappings Signed-off-by: Gordon Sim <[email protected]> * fix(cli): align template GPU JSON default output Signed-off-by: Gordon Sim <[email protected]> --------- Signed-off-by: Gordon Sim <[email protected]>
) Signed-off-by: John Myers <[email protected]>
Signed-off-by: John Myers <[email protected]> Co-authored-by: John Myers <[email protected]>
…#3119) Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.82.1 to 1.83.1. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.82.1...v1.83.1) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-version: 1.83.1 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…l#3027) * feat(mcp): add version-aware wire profile metadata Signed-off-by: Shiju <[email protected]> * feat(policy): canonicalize MCP version allowlists Signed-off-by: Shiju <[email protected]> * fix(policy): align MCP policy tests with current main Signed-off-by: Shiju <[email protected]> * fix(policy): canonicalize supervisor protobuf ingress Materialize defaultable MCP revisions before ambiguity checks, OPA construction, and sidecar delivery. Reject invalid sidecar policies with bounded errors. Signed-off-by: Shiju <[email protected]> --------- Signed-off-by: Shiju <[email protected]>
* refactor(driver-docker): group runtime GPU capabilities Signed-off-by: Evan Lezar <[email protected]> * feat(compute): advertise resource capabilities Signed-off-by: Evan Lezar <[email protected]> * docs(compute): describe resource capabilities Signed-off-by: Evan Lezar <[email protected]> * fix(server): satisfy resource capability clippy lint Signed-off-by: Evan Lezar <[email protected]> --------- Signed-off-by: Evan Lezar <[email protected]>
…DIA/OpenShell#2863) * feat(sandbox): support rootfs tar as --from source for VM driver Accept flat rootfs tar archives (.tar, .tar.gz, .tgz) via the --from flag for VM-backed gateways. The CLI detects the archive extension, validates that the gateway uses the VM compute driver, and passes the tar path through driver_config. The VM driver copies the tar into its staging area and feeds it into the existing rootfs extraction and ext4 disk creation pipeline, skipping the container image pull/export steps. Closes NVIDIA/OpenShell#2175 Signed-off-by: Philippe Martin <[email protected]> * fix(sandbox): validate rootfs tar path at the VM driver boundary The rootfs_tar_path field in driver_config was passed from the API caller directly to tokio::fs::copy without validation. An authenticated user bypassing the CLI could supply arbitrary host paths (e.g. /dev/zero for disk exhaustion, or readable host files for data exfiltration). Introduce a trusted staging directory that the VM driver creates on startup and advertises via GetCapabilities. The CLI now copies the tar into the staging directory before creating the sandbox, and the driver validates that the received path is a regular file inside the staging root and within a configurable size limit (default 10 GiB) before any I/O. New VmDriverConfig options: - rootfs_tar_staging_dir: override the staging directory (default: <state_dir>/rootfs-tar-staging) - rootfs_tar_max_bytes: override the size limit (default: 10 GiB) Addresses GATOR-28b5152e-01. Signed-off-by: Philippe Martin <[email protected]> * fix(sandbox): request-scoped staging, size pre-check, and cleanup for rootfs tar Tighten the rootfs tar staging flow to address the remaining GATOR-01 obligations: - Request-scoped staging: the CLI creates a unique per-request subdirectory (req-<pid>) under the staging root instead of placing files directly in the shared directory. The driver enforces that the tar path is at depth 2 (staging_root/<subdir>/<file>), preventing cross-request path selection. - Size pre-check: the driver advertises rootfs_tar_max_bytes via GetCapabilities. The CLI reads this limit and rejects oversized files before copying, avoiding disk exhaustion in the staging directory. - Cleanup: the driver removes the request staging subdirectory after consuming the tar (on cache hit, copy success, or copy failure), ensuring staged data does not persist beyond the request. Signed-off-by: Philippe Martin <[email protected]> * fix(vm): restore rootfs-tar sandboxes from persisted image identity On restore or restart, the one-shot staged tar archive has already been cleaned up. Reading the persisted image identity from the sandbox state directory and resolving the cached disk path directly avoids re-accessing the deleted staging path. Addresses GATOR-168b9210-01. Signed-off-by: Philippe Martin <[email protected]> * fix(cli): use random staging dirs and enforce byte limit during rootfs tar copy Replace PID-based request staging directories with tempfile-generated random names to prevent collisions and make paths unpredictable. Replace bare tokio::fs::copy with a streaming copy loop that enforces the advertised max_bytes limit during transfer, closing the TOCTOU gap between the pre-copy size check and the actual copy. Signed-off-by: Philippe Martin <[email protected]> Signed-off-by: Philippe Martin <[email protected]> * fix(sandbox): issue rootfs tar staging slots from the gateway A caller could name any host path in `driver_config.vm.rootfs_tar_path`, which the privileged VM driver then read. The CLI-side locality check did not apply to direct API requests. The gateway now owns staging. `BeginRootfsTarStaging` allocates a request-scoped directory under the driver-advertised staging root and returns an opaque single-use token; `CreateSandbox` carries the token, and the gateway substitutes the path it allocated before dispatching to the driver. `template.driver_config.<driver>.rootfs_tar_path` is rejected outright in request validation, so a caller-supplied path never reaches privileged I/O. Tokens are bound to the issuing workspace and subject, consumed once, and expire after 30 minutes. Outstanding slots are capped per caller and overall, so one caller can neither exhaust the staging filesystem nor starve others. An RAII guard reclaims the directory on every failure path after consumption, and an age-gated sweep runs at startup and on each reconcile pass for directories whose driver died before its own cleanup. The token is stripped from the public sandbox before persistence: the stored copy is returned verbatim by GetSandbox, ListSandboxes and WatchSandbox to every member of the workspace. Also fixes two defects this exposed: - The CLI wrote `rootfs_tar_path` at the top level of `driver_config`, but the gateway forwards only `driver_config.<driver_name>`, silently dropping unmatched keys. The archive never reached the VM driver, so the documented `--from ./rootfs.tar` flow did not work at all. Config is now nested under `vm` and deep-merged, so a caller's existing VM settings survive instead of being clobbered by a shallow extend. - Staging previously required `GetGatewayInfo`, which is restricted to `platform_admin`, making the feature unusable for ordinary users on any RBAC-enabled gateway. The new RPC matches CreateSandbox at `sandbox:write` / `workspace_role: user`. `compute_driver.proto` is unchanged; the gateway reads the staging root from the capabilities it already stores. Refs NVIDIA/OpenShell#2175 Signed-off-by: Philippe Martin <[email protected]> * fix(vm): derive rootfs tar cache identity from archive contents The prepared-disk cache key combined the archive's full path with an mtime truncated to seconds, then mapped punctuation to `-`. Distinct paths such as `/tmp/a/b.tar` and `/tmp/a-b.tar` collapsed onto the same key and reused each other's disk, a rewrite within the same second kept stale contents, and a long path could exceed filesystem component limits. Identity is now a SHA-256 of the archive contents. This is also what makes the cache work at all now that the gateway allocates a fresh staging directory per request: a path-derived key would miss on every create. The archive is hashed, the cache checked, and only on a miss copied — so a hit skips writing a multi-gigabyte file. The copy is hashed as it is written and rejected if the digest differs from the first pass, which closes the window where the source changes during staging rather than approximating it with a re-stat. Refs NVIDIA/OpenShell#2175 Signed-off-by: Philippe Martin <[email protected]> * fix(vm): decompress gzip rootfs tar archives during staging `--from` accepts `.tar.gz` and `.tgz`, but the driver staged whatever bytes it was given and the guest image-prep VM extracts the staged file with a plain `tar -xpf`. Compressed sources therefore depended on the guest tar auto-detecting gzip, and the prepared disk was sized from the compressed length, which is far too small for the expanded rootfs. Staging now detects gzip from the archive's magic bytes -- the driver only ever sees a gateway-issued staging path, never the caller's file name -- and writes an uncompressed tar. The digest still covers the source bytes, so the "archive changed while staging" check is unaffected, and expansion is bounded by `rootfs_tar_max_bytes` so a compression bomb cannot fill the host disk. `extract_rootfs_archive_to` sniffs gzip as well, so the host-side extraction path matches. Adds unit coverage for gzip staging, bounded expansion, and gzip extraction, plus an e2e sandbox created from a gzip-compressed export. Signed-off-by: Philippe Martin <[email protected]> --------- Signed-off-by: Philippe Martin <[email protected]> Signed-off-by: Philippe Martin <[email protected]>
NVIDIA/OpenShell#3232) Introduce a single generic applier, options.Apply[T any, O ~func(*T)], in a new leaf package openshell/v1/internal/options, and route all option-application sites through it. Nil option entries are now ignored uniformly instead of panicking; non-nil options still apply in order, and each site keeps its existing pre-seeding and defaulting. - Add internal/options package with Apply and unit tests - Route the 15 application sites (root, types, oidc, fake, gateway, edge) through options.Apply; exported types.Apply*Options keep their signatures and delegate to it - Add one nil-handling test per affected package - Document the nil-option rule in README.md and the docs book No public API signature changes. Closes NVIDIA/OpenShell#3213 Signed-off-by: Roland Huß <[email protected]>
…l#3195) * refactor(inference): remove managed inference routes Closes NVIDIA/OpenShell#3172 Remove the inference route control plane, inference.local data path, built-in router crate, and SDK surface. Move inference workloads to explicitly imported provider profiles and native endpoints, with migration cleanup and updated tests and documentation. Signed-off-by: John Myers <[email protected]> * fix(policy): preserve alternate upstream isolation Restore the provider policy activation guard so legacy OpenAI and Anthropic providers configured for alternate base URLs do not grant egress to the built-in public vendor endpoints. Signed-off-by: John Myers <[email protected]> --------- Signed-off-by: John Myers <[email protected]>
…enShell#3661) * feat(server): place supervisor sessions by consistent hash and hand them off on shutdown Signed-off-by: divesh <[email protected]> * feat: route long-lived sandbox connections to the owning gateway replica Signed-off-by: divesh <[email protected]> * feat(helm): add opt-in per-replica GRPCRoute routing Signed-off-by: divesh <[email protected]> --------- Signed-off-by: divesh <[email protected]>
…VIDIA/OpenShell#4152) * fix(providers): bind multi-route grants to admitted endpoints Carry gateway-derived endpoint owners through policy and credential delivery. Select grants only from owners that admit the current request, refresh provider snapshots per request, and reject superseded installations before forwarding. Cover persistent routes, overlapping owners, denial, refresh, cache expiry and recovery with focused tests and a Podman regression. Signed-off-by: Shiju <[email protected]> * refactor(providers): simplify admitted-endpoint grant selection Share admitted endpoints between forwarding and credential authorization. Select grants from the live snapshot without rebuilding a locked map, reuse profile policy construction, and preserve freshness guards when grants are removed. Signed-off-by: Shiju <[email protected]> * fix(providers): preserve trusted owners across policy approval Sanitize authored owner metadata before composing effective policies, preserve derived owners through proposal approval, and normalize advisor endpoint comparisons. Account for gateway-only owner fields in SDK coverage and schema inventory. Signed-off-by: Shiju <[email protected]> * refactor(providers): share dynamic credential key layout and require a snapshot for admission Move the endpoint-bound credential key layout into openshell-core. DynamicCredentialKey encodes the key the gateway builds, and the supervisor reads its endpoint selector, credential identity, and revision-scoped form through the same module instead of parsing tab-separated strings in three places. Revision scoping becomes ProviderCredentialSnapshot::scoped_key. inject_for_admitted_owners now takes the pinned snapshot directly. The L7 admission path never acquires a grant from the live credential map, which production populates only alongside a snapshot. L4 forwarding keeps the selector-only path through inject_if_needed. Both share one acquire-and-rewrite step. Delete two relay tests whose behavior the Rego admission tests already prove. Add a test that grants for different headers are acquired only for admitting owners. Signed-off-by: Shiju <[email protected]> * fix(providers): share credential preparation across HTTP relays Apply endpoint-owned grants after policy and request middleware in REST, GraphQL, JSON-RPC/MCP and inspected plaintext forwarding. Preserve live provider and policy checks through guarded writes, reject missing owner metadata, and reject profiles that cannot inject dynamic HTTP credentials. Cover production stream dispatch, multiple endpoints, failure handling and plaintext ownership with focused regressions. Document upgrade ordering. Closes NVIDIA/OpenShell#3657 Signed-off-by: Shiju <[email protected]> * fix(providers): validate dynamic credential key encoding Reject control characters at profile import and key construction. Propagate invalid key errors through gateway environment resolution and reject malformed key layouts before token acquisition while preserving supported legacy keys. Consolidate duplicate relay success tests and share forward-proxy setup without removing ownership, canonicalization, refresh or guarded-write assertions. Document token-grant endpoint authority. Signed-off-by: Shiju <[email protected]> --------- Signed-off-by: Shiju <[email protected]>
…/OpenShell#4357) * feat(server): add operator-only provider credential retrieval Authenticate operators exclusively through verified direct gateway mTLS. Export selected runtime credentials with freshness validation and all-or-nothing delivery. Coordinate refresh across callers and replicas, preserve rotated grants across RPC cancellation, and fence credential delivery against reconfiguration. Add generated bindings, security and regression tests, and operator documentation. Signed-off-by: Seth Jennings <[email protected]> * feat(sdk): add curated operator credential retrieval Expose provider credential clients in Python and TypeScript with explicit workspace and key selection, freshness options, optional expiration, and preserved gateway errors. Reuse existing mTLS transports without introducing retries or secret caching. Add generated-wire and error-mapping tests, SDK package validation, and operator usage documentation. Signed-off-by: Seth Jennings <[email protected]> * fix(server): honor recovery state for queued automatic refreshes Signed-off-by: Seth Jennings <[email protected]> --------- Signed-off-by: Seth Jennings <[email protected]>
…hell#3339) * feat(providers): support stable credential placeholders Add credentials[].stable_placeholder so a running client keeps using an externally rotated static credential through the reference it already holds. Squashed replay of the earlier four commits onto upstream main 71c3cd957; re-derives the frozen schema fingerprints and adapts the provider environment revision test to the added capability argument. Signed-off-by: Shiju <[email protected]> * fix(e2e): apply stable-placeholder supervisor image override Use ManagedGateway's per-process supervisor image override so the fixture CA reaches the supervisor when runtime image environment variables take precedence over gateway TOML. Restore the wrapper's original image after each test, following the provider-readiness fixture. Signed-off-by: Shiju <[email protected]> * test(server): update public RPC schema fingerprint after merging main Signed-off-by: Shiju <[email protected]> --------- Signed-off-by: Shiju <[email protected]>
…ll#4265) * docs: document workspace and provider labels (fixes NVIDIA/OpenShell#4250) Signed-off-by: John Myers <[email protected]> * feat(providers): support CLI labels and selector filtering Signed-off-by: John Myers <[email protected]> * fix(providers): update schema inventory and scoped test environment Signed-off-by: John Myers <[email protected]> --------- Signed-off-by: John Myers <[email protected]>
…VIDIA/OpenShell#4106) * refactor(runtime): decouple supervisor access and boundary audit validation Signed-off-by: Drew Newberry <[email protected]> * refactor(runtime): keep supervisor process access boundary driven Signed-off-by: Drew Newberry <[email protected]> * feat(cli): support environment values from caller variables Signed-off-by: Drew Newberry <[email protected]> * refactor(runtime): extract driver-independent MXC foundations Signed-off-by: Drew Newberry <[email protected]> * refactor(runtime): defer UI policy support to MXC integration Signed-off-by: Drew Newberry <[email protected]> * fix(runtime): integrate current policy and SSH launch contracts Signed-off-by: Drew Newberry <[email protected]> * test(runtime): reconcile fixtures with updated main Signed-off-by: Drew Newberry <[email protected]> * fix(runtime): repair stacked branch CI contracts Signed-off-by: Drew Newberry <[email protected]> * test(runtime): update boundary fixtures for current contracts Signed-off-by: Drew Newberry <[email protected]> * docs(runtime): clarify scoped guards for launch inputs and delete Signed-off-by: Drew Newberry <[email protected]> * fix(runtime): validate foreground attachment intent before launch Signed-off-by: Drew Newberry <[email protected]> --------- Signed-off-by: Drew Newberry <[email protected]>
…l#4427) * feat(providers): report bounded token-grant failures Signed-off-by: Shiju <[email protected]> * test(providers): align policy metadata and TLS lockfile Signed-off-by: Shiju <[email protected]> * test(providers): fix trusted grant recovery fixture Signed-off-by: Shiju <[email protected]> * test(providers): enable denial API in recovery fixture Signed-off-by: Shiju <[email protected]> * refactor(providers): simplify token-grant failure reporting Remove duplication and dead paths from the token-grant failure reporting without changing classification, the workload-visible 502 response, or the wire format. - Build the gRPC failure envelope with tonic-types, store the typed failure in EndpointFailure, and share one request-ID rule and one endpoint-status authority check. - Merge the gateway exchange client wrapper, drop forwarding validators, and name the request field when a reported cause is rejected. - Inject fallback credentials only from live provider state and delete the unread per-connection credential copy. Keep one 502 writer per path and restore the base token-cache handling. - Check the token endpoint after the cache lookup, still before any credential acquisition. - Document 429 and 5xx classification as implemented, and drop tests that repeat retained assertions through the same code path. Signed-off-by: Shiju <[email protected]> * refactor(providers): trim token-grant failure tests and supervisor API Cut the token-grant failure tests to one test per distinct behavior across core, gateway, supervisor, CLI and Go, and replace the 794-line closure E2E with status assertions in the existing route-recovery E2E: the refused grant appears in sandbox status with the response's request ID, code, stage and upstream status, and the matching grant's later success clears it. - Pin the live provider snapshot with one helper used by inspected and fallback forwarding, and inject selector-matched grants through inject_for_selectors, which borrows the pinned context. Move the test-only wrapper into test_support. - Drop the 35-second wrappers around the gateway grant RPC. The shared channel bounds the connection attempt and the gateway bounds its token-service calls. - Log the token service's OAuth error and error_description, without control characters and limited to 256 characters each, with the request ID in the gateway or supervisor log. Status and the workload response still carry only the bounded cause. - Keep one check that an untrusted token-service certificate cannot assert a cause. Signed-off-by: Shiju <[email protected]> * fix(providers): restore test import and features lockfile The injection tests named EndpointObservationSender through the parent module, which no longer imports it. The features suite lockfile kept the tonic TLS dependency that the removed E2E fixture added. Signed-off-by: Shiju <[email protected]> * test(providers): fix body-bound fixture width and table type Pass the response-size width as a named argument; an implicitly captured constant panics at runtime. Name the mutation type in the validation table for Clippy. Signed-off-by: Shiju <[email protected]> * test(providers): pad exact-bound body without a format width MAX_TOKEN_RESPONSE_BYTES exceeds u16::MAX, the largest format width, so the padded fixture panicked. Pad with repeat_n as before. Signed-off-by: Shiju <[email protected]> * test(server): update public RPC schema hash after rebase Main added the ListProvidersRequest label selector. The merged public schema hash comes from the Branch Checks storage_proto test output. Signed-off-by: Shiju <[email protected]> * fix(providers): keep token-service error text out of grant logs A token service can echo a submitted credential in its OAuth error or error_description. Log only the classified failure code, HTTP status and request ID, and cover the canary echo in a dedicated test binary. Signed-off-by: Shiju <[email protected]> * docs(providers): tighten token grant failure documentation Signed-off-by: Shiju <[email protected]> * docs(providers): condense token grant failure guidance Signed-off-by: John Myers <[email protected]> --------- Signed-off-by: Shiju <[email protected]> Signed-off-by: John Myers <[email protected]> Co-authored-by: John Myers <[email protected]>
…ns (NVIDIA/OpenShell#3198) * feat(sdk/go)!: replace positional labels with functional Create options Move labels and annotations from separate parameter positions into functional options (WithLabels, WithAnnotations) on Create() and CreateFromTemplate(). Before: Create(ctx, ws, name, spec, labels, CreateOptions{Annotations: ann}) After: Create(ctx, ws, name, spec, WithLabels(labels), WithAnnotations(ann)) This unifies two resource metadata fields that were inconsistently placed (labels as positional, annotations in a struct) into a single extensible options pattern matching the existing LogOption convention. BREAKING CHANGE: Create() and CreateFromTemplate() signatures changed. All callers must use WithLabels() and WithAnnotations() functional options instead of positional labels and CreateOptions struct. Closes NVIDIA/OpenShell#2807 Signed-off-by: Roland Huß <[email protected]> * fix(sdk/go): ignore nil Create options Callers of the previous signature passed nil for the positional labels parameter to create an unlabeled sandbox: Create(ctx, ws, name, spec, nil) That call still compiles against the new variadic signature, because an untyped nil converts to CreateOption. ApplyCreateOptions then invoked it and panicked with a nil pointer dereference, so the migration was unsafe for existing consumers. Skip nil options in ApplyCreateOptions and cover the legacy trailing-nil call with regression tests against ApplyCreateOptions directly, the fake client (Create and CreateFromTemplate), and the gRPC client. Signed-off-by: Roland Huß <[email protected]> * refactor(sdk/go): apply Create options through the shared options helper Signed-off-by: Roland Huß <[email protected]> * docs(sdk/go): use WithLabels in CreateSandboxFromTemplate examples Signed-off-by: Roland Huß <[email protected]> --------- Signed-off-by: Roland Huß <[email protected]>
…on (NVIDIA/OpenShell#4320) Supervisors poll the gateway for configuration every 10 seconds. This PR adds an opt-in push mode (config_delivery_mode = "push"): supervisors still start by polling, and once their ConnectSupervisor session is accepted, the gateway delivers every later change over it and supervisors apply it and report the result. poll stays the default, and older supervisors and gateways keep polling. --------- Signed-off-by: Piotr Mlocek <[email protected]> Signed-off-by: divesh <[email protected]> Co-authored-by: divesh <[email protected]>
Signed-off-by: Eric Curtin <[email protected]>
* fix(sdk/go): accept CLI-compatible gateway names Signed-off-by: Sunil Kumar Vadlamani <[email protected]> * fix(sdk/go): enforce Windows-safe gateway names Signed-off-by: Sunil Kumar Vadlamani <[email protected]> --------- Signed-off-by: Sunil Kumar Vadlamani <[email protected]> Co-authored-by: Sunil Kumar Vadlamani <[email protected]>
…VIDIA/OpenShell#4678) * feat(middleware): authorize supervisor bindings at registration Let operators require and limit the operation and phase pairs that an external middleware may expose. Validate the same binding policy at gateway startup and when a supervisor discovers the service, and keep only authorized bindings in the registry used for dispatch. Preserve dynamic discovery and the existing wire configuration when no binding policy is set. Document that older supervisors must be upgraded before relying on binding restrictions. Signed-off-by: Free Wortley <[email protected]> * docs(skills): check which supervisor middleware hooks the operator authorizes Signed-off-by: Free Wortley <[email protected]> * fix(go): regenerate the sandbox protobuf binding Restore the comment separator emitted by the pinned generator. This keeps the checked-in binding byte-for-byte reproducible without changing the protobuf schema or Go behavior. Signed-off-by: Free Wortley <[email protected]> --------- Signed-off-by: Free Wortley <[email protected]>
* feat(sandbox): add suspend and resume lifecycle Signed-off-by: Drew Newberry <[email protected]> * fix(sandbox): preserve V8 state across failed resume launches Signed-off-by: Drew Newberry <[email protected]> * fix(sandbox): handle suspend and resume recovery races Signed-off-by: Drew Newberry <[email protected]> * refactor(example): move V8 suspend support to a dependent change Signed-off-by: Drew Newberry <[email protected]> * fix(sandbox): align suspend tests with current supervisor APIs Signed-off-by: Drew Newberry <[email protected]> * fix(sandbox): isolate suspend requests and resume compensation Signed-off-by: Drew Newberry <[email protected]> * fix(supervisor): allow restoration to wait until agent admission Signed-off-by: Drew Newberry <[email protected]> --------- Signed-off-by: Drew Newberry <[email protected]>
…NVIDIA/OpenShell#4852) * feat(sdk): route Go SDK exec and forward to the sandbox owner replica The Go SDK now does what the CLI already does for long-lived sandbox calls. The GetSandbox check that Exec.Run, Exec.Stream, Exec.Interactive, TCP.Forward and SSH.Tunnel already make also asks the gateway for the owner replica. The call then sends it in the x-openshell-replica header and retries once without the header when the edge proxy reports that the owner's route has no ready endpoints. No extra round trip is added. Without a reported owner the calls are unchanged. Part of NVIDIA/OpenShell#4223 Signed-off-by: divesh <[email protected]> * refactor(sdk): address Go replica routing review Share the header check between the routed attempt and its retry, keep the first-frame EOF rule in one bidi helper, reuse Get's request for the owner lookup, and document the header contract next to the replica routing docs. Part of NVIDIA/OpenShell#4223 Signed-off-by: divesh <[email protected]> * docs(kubernetes): refer to the SDKs collectively in replica routing Part of NVIDIA/OpenShell#4223 Signed-off-by: divesh <[email protected]> --------- Signed-off-by: divesh <[email protected]>
…es (NVIDIA/OpenShell#4873) * feat(gateway): add storage-preserving sandbox image updates Add image-only UpdateSandbox with conditional versions, durable replay, fenced pending intent, and Kubernetes storage retention. Enable non-destructive image repair after completed timeout cleanup and exercise both Kubernetes controller API lanes. Closes NVIDIA/OpenShell#4623 Signed-off-by: sjenning <[email protected]> * feat(gateway): add storage-preserving sandbox image updates OpenClaw-Publication: b2036b09-b697-4d65-aba9-5c96a3198f9c Signed-off-by: sjenning <[email protected]> * fix(gateway): preserve image update patch-release compatibility Move image updates to a separate authenticated service and retain the baseline public sandbox status. Persist image intent and revisions independently with atomic baseline-known lifecycle fencing and explicit post-upgrade activation. Preserve concurrent provisioning clocks, fence replaced attempts and delayed callbacks, and verify private-state transactions plus both Kubernetes controller API lanes. Signed-off-by: sjenning <[email protected]> * fix(gateway): preserve optional driver compatibility and repair CI Move image updates to a separate optional compute-driver service, preserving the baseline generated trait and streaming associated type. Add source and TCP/UDS transport regressions, capability negotiation coverage, and a default-unsupported in-process extension. Keep Go compatibility tests outside generated output and update production reflection expectations for the separate sandbox update service. Signed-off-by: sjenning <[email protected]> * fix(gateway): preserve baseline factory and descriptor APIs Restore the original in-process driver trait object and existing instance enum. Register optional image updates through a default factory hook without requiring changes to existing drivers or factories, and require registration plus capability before mutation. Accept OpenShell-only descriptors while retaining strict validation of present update services. Cover baseline registration, missing-updater no-write behavior, descriptor validation, and both live Kubernetes retention lanes. Signed-off-by: sjenning <[email protected]> * chore(ci): use upstream Kubernetes E2E defaults Drop the task-level fallback superseded by main's runner defaults and explicit e2e-conformance-only selector. Preserve current upstream task definitions and recovery waiver unchanged. Signed-off-by: sjenning <[email protected]> * fix(gateway): integrate image updates with backend protocol Adapt non-launch image updates to the upstream remote admission API while retaining resource-policy verification before forwarding. Cover both negotiated extensions and preservation of backend pins and persisted launch identity. Signed-off-by: sjenning <[email protected]> * fix(gateway): remove sandbox image state on deletion Signed-off-by: Gordon Sim <[email protected]> * feat(gateway): expand UpdateSandbox configuration updates Signed-off-by: Gordon Sim <[email protected]> * fix(sandbox): recover pending updates and address review feedback Signed-off-by: Gordon Sim <[email protected]> * perf(server): index pending sandbox updates for recovery Signed-off-by: Gordon Sim <[email protected]> * docs(sandbox): clarify stop behavior when updates are disabled Signed-off-by: Gordon Sim <[email protected]> * fix(server): scope deletion bypass to the matching sandbox update Signed-off-by: Gordon Sim <[email protected]> * fix(server): back off pending sandbox update recovery Signed-off-by: Gordon Sim <[email protected]> * ci: restore combined-chart Kubernetes e2e coverage Signed-off-by: Gordon Sim <[email protected]> * test(kubernetes): use uuid for update request IDs Signed-off-by: Gordon Sim <[email protected]> * fix(kubernetes): reject oversized annotations before accepting sandbox updates Signed-off-by: Gordon Sim <[email protected]> * docs(skills): correct pending sandbox update recovery guidance Signed-off-by: Gordon Sim <[email protected]> * fix(sandbox): preserve terminal results when stopping missing compute Signed-off-by: Gordon Sim <[email protected]> * fix(server): bound sandbox update attempts to unblock recovery and deletion Signed-off-by: Gordon Sim <[email protected]> * fix(server): remove unintended sandbox label count limit Signed-off-by: Gordon Sim <[email protected]> * fix(kubernetes): check known pod names during sandbox update preflight Signed-off-by: Gordon Sim <[email protected]> * docs: clarify sandbox update permissions and driver support Signed-off-by: Gordon Sim <[email protected]> --------- Signed-off-by: sjenning <[email protected]> Signed-off-by: Gordon Sim <[email protected]> Co-authored-by: sjenning <[email protected]>
…VIDIA/OpenShell#4936) * feat(supervisor): add independent application listener and endpoints Signed-off-by: Taylor Mutch <[email protected]> * test(supervisor): automate Kubernetes application listener e2e Signed-off-by: Taylor Mutch <[email protected]> * fix(supervisor): enforce app route limits and shutdown fencing Reject excess service declarations before persistence, fence registration against shutdown, and require the expected application Service count in listener E2E checks. Add focused regression coverage. Signed-off-by: divesh <[email protected]> Signed-off-by: Taylor Mutch <[email protected]> * test(supervisor): cover application streams on config rejection Verify rejected application configuration closes existing WebSockets without affecting another sandbox, and accepted recovery cannot revive old streams. Document raw SDK message migration and shared-owner selection before discovery. Signed-off-by: Taylor Mutch <[email protected]> * fix(app-listener): fix standalone lint and capture CNI setup failures Signed-off-by: Taylor Mutch <[email protected]> * fix(app-listener): preserve polling freshness and HTTP keepalive Signed-off-by: Taylor Mutch <[email protected]> --------- Signed-off-by: Taylor Mutch <[email protected]> Signed-off-by: divesh <[email protected]> Co-authored-by: divesh <[email protected]>
* fix(sdk/go): set up Listen tunnels in parallel Signed-off-by: Eric Curtin <[email protected]> * test(sdk/go): cover EOF on abandoned setup Signed-off-by: Eric Curtin <[email protected]> --------- Signed-off-by: Eric Curtin <[email protected]>
Signed-off-by: Yuedong Wu <[email protected]>
* fix(sdk/go): fall back to system active gateway Match the CLI: use the system active_gateway when the user one is missing or invalid. Signed-off-by: Eric Curtin <[email protected]> * test(sdk/go): cover system active gateway paths Signed-off-by: Eric Curtin <[email protected]> --------- Signed-off-by: Eric Curtin <[email protected]>
Merge the sdk/go history of NVIDIA/OpenShell at the freeze commit 8ed8bf0c907fd06572d616d64f22ede915d0c10f (after v0.1.7), filtered with hack/migration/extract-history.sh. Bare #N references are rewritten to NVIDIA/OpenShell#N. The repository's initial README is replaced by the SDK's. Tree: 49f0f2b (equal to sdk/go at the freeze commit). Signed-off-by: Roland Huß <[email protected]>
Contributor
Author
|
@SDAChess @pimlock @Gkrumbach07 This is the initial import of the golang SDK, directly from 1.0.7. It would be great if we could merge it as-is Important Merge with a MERGE COMMIT, not squash or rebase. 
that is to keep the imported history. |
This was referenced Nov 4, 2026
SDAChess
approved these changes
Nov 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Import the history of
sdk/gofrom NVIDIA/OpenShell into this repository, as the first step of the Go SDK move (NVIDIA/OpenShell#5031).8ed8bf0c9, the last coremaincommit on 2026-11-03, after the v0.1.7 release.git filter-repo --subdirectory-filter sdk/go. Authors, dates, and sign-offs are unchanged. Bare#Nreferences in commit messages now readNVIDIA/OpenShell#N, so they keep pointing at core.The module path is still
github.com/NVIDIA/OpenShell/sdk/gohere. The next PR makes the module standalone atgithub.com/ai-openshell/sdk-goand generates the bindings from core v0.1.7.Verification
From any core checkout, with the scripts from
hack/migration-tooling:The merge commit's tree is also
49f0f2b585dff0736df7a976ccf2d282a4bc5175.How to merge
Merge with a merge commit, not squash or rebase. The merge commit is what carries the 85 imported commits; a squash would drop the history this PR exists for.