Skip to content

Commit 4a459b9

Browse files
andre-mottaclaudemergify[bot]
authored
Consume the split OpenShell supervisor and sandbox runtime images (#432)
* Adapt agentic-ci to the split OpenShell supervisor and sandbox runtime OpenShell v0.0.116-rhaiv.8+ split the supervisor into odh-openshell-supervisor and odh-openshell-sandbox and moved the gateway to schema v2. Bump OPENSHELL_IMAGE_TAG to v0.0.116-rhaiv.11 and wire OPENSHELL_SANDBOX_RUNTIME_IMAGE into gateway.toml; render version 2 with the singular compute_driver and restart a running gateway when its config drifts; import vendored provider profiles, now that the gateway ships none; pass an explicit base policy at sandbox create, since the supervisor rejects the Hummingbird image policy; bake nameserver 127.0.0.53 into the sandbox images for the network=none workload; switch Vertex auth to the google-vertex-ai profile with placeholder bearer tokens (Claude Code via gateway mode, OpenCode via a loopback token-exchange stub) because the GCE metadata emulator was removed; drop inference hosts the provider layer already declares; make bump-versions.py require a tag in all four image repos; bump ACLI to 1.3.38~stable-1. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Andre Lustosa <[email protected]> * Disable agent-eval-harness for Claude runs in the OpenShell E2E The plugin's SessionStart hook bulk-installs MLflow with uv. uv reuses a policy-DNS synthetic IP past its ~30s mapping lifetime, and the resulting transparent_tcp_mapping_denied while other relays are open crashes the OpenShell supervisor (upstream NVIDIA/OpenShell#3396). Enable every other shipped plugin via AGENT_ENABLED_PLUGINS until the supervisor survives that denial. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Andre Lustosa <[email protected]> * Let the routed classifier complete on its route file Codex does not exit within the grace period after its final stream event, so the backend terminates it and only promotes that exit to success when its completion file exists. The session pointed that check at the skill verdict, which the classifier never writes, so every Codex classifier run on OpenShell fell back with "classifier exited with code -15". Pass _run/route.json as the classifier run's completion file. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Andre Lustosa <[email protected]> --------- Signed-off-by: Andre Lustosa <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
1 parent cae6287 commit 4a459b9

33 files changed

Lines changed: 1483 additions & 338 deletions

‎.claude/skills/debug-agentic-ci/references/symptoms.md‎

Lines changed: 43 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -141,13 +141,53 @@ Known failure patterns from this repo's history. Update this file when fixing bu
141141
- **Likely cause**: The sandbox was created with a short-lived canonical main process (e.g. `-- true`). OpenShell v0.0.111+ (#2726) treats the trailing argv as the sandbox's canonical process; when it exits, the supervisor shuts down before it can acknowledge policy v2. Fixed by using `--detach -- sleep infinity` so the main process stays alive while policy updates and agent commands run via `sandbox exec`.
142142
- **Where to look**: `backends/openshell/sandbox.py:create()` canonical process args
143143

144+
### Gateway never becomes healthy: gateway.toml rejected (`unknown field `compute_drivers`` or `unsupported gateway config version 1`)
145+
- **Symptom**: `agentic-ci run --backend openshell` fails with `RuntimeError: Gateway did not become healthy within 60s`; the gateway log shows either `TOML parse error ... unknown field `compute_drivers`, expected one of ... `compute_driver` ...` or `unsupported gateway config version 1; this build requires version 2; migrate legacy fields to the version 2 schema`.
146+
- **Likely cause**: OpenShell v0.0.116-rhaiv.8+ (upstream sync in opendatahub-io/openshell#44) moved gateway.toml to schema version 2: `[openshell] version = 2`, a singular `compute_driver` string instead of the `compute_drivers` list, and driver options only under `[openshell.drivers.<name>]`. Version 1 files are rejected outright. agentic-ci renders `version = 2` and `compute_driver = "podman"`.
147+
- **Where to look**: `backends/openshell/gateway.py` `_GATEWAY_TOML`, the gateway log under `~/.local/state/openshell/gateway-*.log`, the upstream migration guide at https://docs.nvidia.com/openshell/latest/reference/gateway-config#migrate-to-schema-version-2
148+
149+
### Provider creation fails: `provider profile 'google-cloud' not found; import a matching profile before using this provider type`
150+
- **Symptom**: `openshell provider create --type google-cloud|openai|anthropic` exits 1 during `agentic-ci run --backend openshell`, right after the gateway becomes healthy.
151+
- **Likely cause**: OpenShell v0.0.116-rhaiv.8+ removed the built-in provider profiles; a fresh gateway serves an empty catalog. agentic-ci imports the vendored profiles from `backends/openshell/profiles/` through `provider.ensure_profiles()` before creating the provider. If the error persists, check that `openshell provider list-profiles --global` shows the profile, that the YAML still lints (`openshell provider profile lint -f <file>`), and that the profile ID matches the `--type` value.
152+
- **Where to look**: `backends/openshell/provider.py` `ensure_profiles()` and `PROFILE_IDS`, `backends/openshell/profiles/*.yaml`, the upstream `providers/` directory in opendatahub-io/openshell for schema changes
153+
154+
### `openshell sandbox create` times out: `ConfigurationInvalid: Effective configuration could not be activated; replace the policy or repair attached providers`
155+
- **Symptom**: `sandbox provisioning timed out after 300s` with that status; both the supervisor and workload containers are up; `openshell policy get ci` says `no active policy configured`. Every E2E test then waits the full 300s, which looks like a hung job.
156+
- **Likely cause**: The supervisor log (`/var/log/openshell.<date>.log` inside the `openshell-supervisor-<id>` container) shows `Server returned no policy; attempting local discovery` followed by `Image policy is invalid; replace the sandbox policy to repair configuration`. The Hummingbird agentic images ship `/etc/openshell/policy.yaml` in a non-OpenShell schema (`sandbox:`/`permissions:` keys). OpenShell v0.0.116-rhaiv.8+ parses the discovered image policy strictly and rejects the sandbox instead of falling back to its restrictive default. agentic-ci now always passes `--policy` with `policy.BASE_POLICY` at create time, which skips image discovery entirely.
157+
- **Where to look**: `backends/openshell/sandbox.py:create()` `--policy` handling, `backends/openshell/policy.py:BASE_POLICY`, `podman exec openshell-supervisor-<id> cat /var/log/openshell.*.log` on the CI host, `openshell sandbox get ci -o json` `configuration_admission`
158+
159+
### Sandbox creation pulls `ghcr.io/nvidia/openshell/sandbox` or fails with a missing `openshell-sandbox` binary
160+
- **Symptom**: After bumping `OPENSHELL_IMAGE_TAG` to v0.0.116-rhaiv.8 or later, `openshell sandbox create` tries to pull `ghcr.io/nvidia/openshell/sandbox:<tag>` (unreachable from CI), or the supervisor container starts but the workload never runs `/openshell-sandbox`.
161+
- **Likely cause**: OpenShell split the supervisor image into `odh-openshell-supervisor` (runs in its own container) and `odh-openshell-sandbox` (static binary mounted into the workload). The gateway's podman driver needs both `supervisor_image` and `sandbox_runtime_image` in `gateway.toml`; without the second it falls back to the upstream ghcr.io default. agentic-ci renders both from `OPENSHELL_SUPERVISOR_IMAGE` and `OPENSHELL_SANDBOX_RUNTIME_IMAGE`, which the CI image sets from `OPENSHELL_IMAGE_TAG`. Conversely, tags before v0.0.116-rhaiv.8 have no `odh-openshell-sandbox` image, so `bump-versions.py` only selects tags published for all four repos.
162+
- **Where to look**: `images/ci/Containerfile.openshell` ENV lines, `backends/openshell/gateway.py:_render_podman_driver_section()`, `scripts/bump-versions.py:_quay_latest_openshell()`, `tests/e2e/e2e-openshell-sandbox.sh` image resolution
163+
144164
### OpenShell cleanup fails with `no such table: objects`
145165
- **Likely cause**: The local gateway used `sqlite::memory:` and lost its schema when SQLite replaced the connection during concurrent sandbox cleanup. The gateway now uses a temporary file-backed database so replacement connections share the same schema.
146166
- **Where to look**: `backends/openshell/gateway.py` database URL creation and cleanup, gateway logs around `DeleteSandbox` and compute-driver watch events
147167

148-
### `openshell policy set` rejects credential binding: "uses L4-only; configure L7 inspection or set allow_uninspected_credentials"
149-
- **Likely cause**: OpenShell v0.0.112+ validates that credentialed endpoints use L7 inspection. GCP endpoints use L4 CONNECT tunneling (required for Vertex AI gRPC streaming), so setting `credential_binding.provider` without `allow_uninspected_credentials: true` is now rejected. Fixed by adding `allow_uninspected_credentials: true` alongside the credential binding.
150-
- **Where to look**: `backends/openshell/policy.py:build_credential_binding_patch()`
168+
### `openshell policy update` rejects the sandbox policy: "network endpoint ambiguity validation failed ... overlap on port(s) 443 with conflicting metadata"
169+
- **Likely cause**: The sandbox policy declares a host that the attached provider profile already contributes as an L7 `rest` layer (`_provider_ci_gcp` / `_provider_ci_vertex`): `aiplatform.googleapis.com`, `api.anthropic.com`, or `api.openai.com`. Since OpenShell v0.0.116-rhaiv.8+ agentic-ci leaves those hosts out of `policy.AUTH_ENDPOINTS`; a repo `.agentic-ci/openshell-policy.yml` that re-adds them trips the same validation.
170+
- **Where to look**: `backends/openshell/policy.py:AUTH_ENDPOINTS`, the repo policy file, `openshell policy get ci --full -o json`
171+
172+
### Claude Code on Vertex fails inside OpenShell: `API Error: Could not refresh access token` or `Could not load the default credentials`
173+
- **Symptom**: The sandbox reaches `Ready`, the run starts, and the first API call fails. The workload container log shows `sandbox network notification denied ... Connection refused` for `127.0.0.1:8174`.
174+
- **Likely cause**: OpenShell v0.0.116-rhaiv.8+ removed the GCE metadata emulator (`openshell-sandbox` no longer serves `127.0.0.1:8174`), so SDK-side ADC discovery cannot mint a token. agentic-ci now uses the `google-vertex-ai` profile: the gateway injects `GOOGLE_VERTEX_AI_SERVICE_ACCOUNT_TOKEN` / `GOOGLE_VERTEX_AI_TOKEN` as a placeholder and the Claude env script sets `CLAUDE_CODE_SKIP_VERTEX_AUTH=1`, `ANTHROPIC_AUTH_TOKEN=<placeholder>`, and `ANTHROPIC_VERTEX_BASE_URL` (host plus `/v1`), so the request carries the placeholder as a bearer token and the supervisor proxy swaps in the real one. If this reappears, check that the env script still exports those three variables and that the profile's `binaries` include the harness launcher.
175+
- **Where to look**: `harness.py` `ClaudeCodeHarness.build_env_script_lines()`, `backends/openshell/provider.py` `_create_gcp_provider_*`, `backends/openshell/profiles/google-vertex-ai.yaml`, supervisor log `ALLOWED POST ... [policy:_provider_ci_gcp engine:l7]`
176+
177+
### OpenCode on Vertex fails inside OpenShell: `Was there a typo in the url or port?` or `Could not load the default credentials`
178+
- **Symptom**: The OpenCode run errors before any request reaches `aiplatform`; the supervisor log shows no `ALLOWED POST` for the model.
179+
- **Likely cause**: OpenCode's `@ai-sdk/google-vertex` mints a token through `google-auth-library` before every request. agentic-ci satisfies it with an `external_account` ADC whose `token_url` is `http://127.0.0.1:8175/token`, served by `agentic-ci vertex-token-stub` started from the env script. "typo in the url or port" means the stub is not listening: the sandbox image's agentic-ci predates the `vertex-token-stub` subcommand, `AGENTIC_CI_VERTEX_TOKEN` was empty (no provider placeholder injected) so the stub refused to start, or the sandbox base policy blocks `/tmp`. Check `/tmp/.agentic-ci-vertex-stub.log` inside the sandbox.
180+
- **Where to look**: `harness.py` `_OPENSHELL_VERTEX_ADC_STUB_LINES`, `vertex_token_stub.py`, `cli.py` `vertex-token-stub` dispatch, the sandbox image's installed agentic-ci version
181+
182+
### Routed Codex run falls back: `classifier exited with code -15`
183+
- **Symptom**: `run_routed_skill` on the OpenShell backend logs `stream completed (rc=-15) but verdict file .../verdict.json missing; keeping original exit code` right after the classifier wrote `_run/route.json`, then `Model routing fell back to <default>: classifier exited with code -15`.
184+
- **Likely cause**: Codex does not exit within the 10s grace period after its final stream event, so the backend terminates it (`-15`). The backend only promotes a terminated-after-completion run to 0 when its completion file exists. The classifier run now passes `completion_file=route.json` to the session, so the check looks for the route file rather than the skill verdict. If this reappears, check that `_router` in `skill.py` still wraps `session.run` with `completion_file`.
185+
- **Where to look**: `skill.py` `_AgentSession.run(completion_file=...)` and `_router`, `backend.py` `_resolve_exit_code()`, `routing.py` `classify()`
186+
187+
### Every host fails to resolve inside the OpenShell sandbox (`Could not resolve host`)
188+
- **Symptom**: `curl`/agent requests fail with DNS errors even for allowed endpoints; `/etc/resolv.conf` in the workload is empty while `ss -ltn` shows a listener on `127.0.0.53:53`.
189+
- **Likely cause**: With the supervisor/sandbox split the workload runs with `network=none` and the `openshell-sandbox` binary serves policy DNS on `127.0.0.53:53`. Podman writes no `resolv.conf` for `network=none`, the Hummingbird base ships an empty one, and the sandbox runs as uid 1001 so it cannot rewrite the root-owned file. The sandbox images now `COPY images/runner/shared/resolv.conf` (`nameserver 127.0.0.53`); a `RUN` that writes the file does not persist because buildah bind-mounts `resolv.conf` during builds.
190+
- **Where to look**: `images/runner/*/Containerfile.openshell`, `images/runner/shared/resolv.conf`, `tests/e2e/e2e-openshell-sandbox.sh` resolv.conf assertions
151191

152192
### OpenCode image build fails with `KeyError: 'repo'`
153193
- **Likely cause**: A marketplace plugin uses a `git-subdir` source with `url` and `path` instead of the legacy GitHub `repo` field. The OpenCode compatibility installer must resolve both source formats and search for skills relative to the configured subdirectory.

‎AGENTS.md‎

Lines changed: 13 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ src/agentic_ci/
3737
telemetry.py # Generic event transport for the OTLP trace pipeline
3838
routing.py # Difficulty classifier + model tier routing (run_routed_skill)
3939
models.py # Model registry: default model, routing tiers, effort levels per harness
40+
vertex_token_stub.py # Loopback GCP token-exchange stub for OpenCode in OpenShell
4041
otel.py # OTLP collector + token/cost summary
4142
```
4243

@@ -101,12 +102,19 @@ scripts/
101102
bump-versions.py — Bump pinned dependency versions in Containerfiles
102103
```
103104

104-
OpenShell is consumed from UBI9 artifacts: the CLI binary copied from
105+
OpenShell is consumed from four UBI9 artifacts sharing one tag
106+
(`OPENSHELL_IMAGE_TAG`): the CLI binary copied from
105107
`quay.io/opendatahub/odh-openshell-cli`, the gateway binary copied from
106-
`quay.io/opendatahub/odh-openshell-gateway`, and the supervisor pulled at
107-
runtime from `quay.io/opendatahub/odh-openshell-supervisor`. The OpenShell CI
108-
image is UBI9, the OpenShell sandbox images are Hummingbird,
109-
and the podman-path images stay on UBI10.
108+
`quay.io/opendatahub/odh-openshell-gateway`, the supervisor pulled at
109+
runtime from `quay.io/opendatahub/odh-openshell-supervisor`, and the sandbox
110+
runtime pulled at runtime from `quay.io/opendatahub/odh-openshell-sandbox`.
111+
Since v0.0.116-rhaiv.8 the supervisor runs in its own container and the
112+
sandbox runtime image supplies the `openshell-sandbox` binary mounted into
113+
each workload; both are selected through `OPENSHELL_SUPERVISOR_IMAGE` and
114+
`OPENSHELL_SANDBOX_RUNTIME_IMAGE`, rendered into `gateway.toml` by
115+
`backends/openshell/gateway.py`. The OpenShell CI image is UBI9, the
116+
OpenShell sandbox images are Hummingbird, and the podman-path images stay
117+
on UBI10.
110118

111119
All three sandbox images build directly on their respective hardened
112120
Hummingbird agentic images (`quay.io/aipcc/base-images/agentic/claude-code`,

‎README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -304,7 +304,8 @@ report token usage but do not produce a dollar estimate.
304304
| `GCLOUD_CREDENTIALS` | — | Raw JSON or base64 gcloud credentials |
305305
| `GCP_SERVICE_ACCOUNT_KEY` | — | Service account key: file path, raw JSON, or base64-encoded JSON |
306306
| `GOOGLE_APPLICATION_CREDENTIALS` | — | Path to ADC credentials file |
307-
| `OPENSHELL_SUPERVISOR_IMAGE` | `openshell/supervisor:dev` | OpenShell supervisor image (`openshell` backend only) |
307+
| `OPENSHELL_SUPERVISOR_IMAGE` | — | OpenShell supervisor image, runs beside each sandbox (`openshell` backend only; set in the CI image) |
308+
| `OPENSHELL_SANDBOX_RUNTIME_IMAGE` | — | OpenShell sandbox runtime image supplying the `openshell-sandbox` binary mounted into each sandbox (`openshell` backend only; set in the CI image) |
308309

309310
## Streaming Output
310311

0 commit comments

Comments
 (0)