Repository navigation
build(fabric): pin Fabric with the P0 interface adapters, OpenClaw 2026.9.34 and Hermes v0.22.5 - #14036
Merged
Conversation
…26.9.34 and Hermes v0.22.5 Step 1 of #14011. Pin Fabric 78e79bfc, the reference implementation of the P0 interface adapters (NVIDIA/NeMo-Fabric#541, #540, #546), fetched through the upstream repository. It sits on Fabric main 624e6876 (0.7.0). This commit enables no interfaces. - Pins: FABRIC_REVISION and FABRIC_SHA256, nemo-fabric-core 0.7.0 in three crates and Cargo.lock, and pinned documentation links. - OpenClaw: the adapter supports only 2026.9.34, so OPENCLAW_IMAGE moves to that release's index digest. - Hermes: v0.22.5 (bf0203a7), the release Fabric tests, and its image digest. Its lock overrides browser-harness's Pillow 12.3.0 pin with 13.0.0, which re-resolving the export rejects. The image now installs the exported closure without re-resolving, and Hermes's own pm.pip_check accepts only the uv pip check mismatches those overrides cause. aiohttp, web_dist and tui_dist still install. - Wheels: nemo-fabric-runtime and the adapter contract now depend on nemo-fabric-evidence, which the image builds from the same source. - Catalog: regenerated. OpenClaw and Hermes gain settings_schema.properties.interfaces; Claude, Codex and Pi gain insecure_http_model_hosts, so NemoClaw now lists plain-HTTP endpoint hosts for them as it does for OpenCode; OpenClaw gains the model setting supports_usage_in_streaming, which onboarding asks for. - fabric_agent: Fabric's new provenance section becomes an input like every other section; the provider schema snapshot gains its block. - Tests: contract fixtures regenerated; onboarding tests omit OpenClaw's interfaces setting and use a startup timeout within the new maximum of 75 seconds. - HCL scenarios: the fixture image's digest covers the bundled catalog, and the OpenShift scenario's inference addresses derive from it, so the case documents and openshift-external-two-providers.tftest.hcl follow the new catalog. Signed-off-by: Carlos Villela <[email protected]>
Live tests for #14011 found that OpenShell's sandbox refuses signals to a process group (EPERM), so stopping Hermes's interface children and Fabric's ACP agents, such as Deep Agents Code, failed. Fabric 6cad3504 fixes both: it merges the Hermes fix (7981efca) into the Deep Agents Code branch and fixes the ACP runner. Move every Fabric pin from 78e79bfc to 6cad3504: FABRIC_REVISION and FABRIC_SHA256, nemo-fabric-core in three crates and Cargo.lock, the bundled catalog, and pinned documentation links. No adapter descriptor changed, so the regenerated catalog differs only in fabric_revision and source_sha256. The catalog change moves the fixture image's digest again, so the HCL case documents and openshift-external-two-providers.tftest.hcl follow it. Signed-off-by: Carlos Villela <[email protected]>
Collaborator
Author
Authoring session — opening
|
cv
enabled auto-merge (squash)
October 31, 2026 00:08
Contributor
cv
added a commit
that referenced
this pull request
Oct 31, 2026
## Failure With the Fabric pin from #14036, OpenClaw's and Hermes's adapters can serve their native interfaces, but a NemoClaw deployment can't use them: planning doesn't check `harness.settings.interfaces` against the image's adapter, apply doesn't say how to reach an interface, and the docs don't cover it (#14011, #13360, #13973, #13969). Hermes's browser chat also fails with "out of pty devices", because the agent images' default filesystem policy doesn't allow pseudo-terminals. ## Decision - **Planning:** a new `harness_interfaces` check rejects interface names and fields the image's adapter doesn't declare in its descriptor's `settings_schema.properties.interfaces`. An adapter that declares none supports none. - **Apply output:** for each enabled interface, apply prints how to select the gateway and workspace, the `openshell forward service` command, the `openshell sandbox exec` command that reads the token from the runtime's `interfaces/` directory, and how to sign in. `-o json` carries the same facts under `interfaces`. NemoClaw never reads, prints or stores a token. - **Health:** no change was needed: the provider decides whether the agent runs from the runtime check alone, so a failed interface child doesn't change it. - **Per-image policy:** each image's adapter declaration may list extra read-write paths, which `catalog.py` adds to that image's default policy. Only the Hermes image declares `/dev/pts` and `/dev/ptmx`; every other image keeps its policy. - **Live test:** `interfaces_live` applies OpenClaw with its dashboard and Hermes with its API server and dashboard on a managed Docker gateway, with a model stand-in that needs a key, and checks sign-in, token rejection, the model key, browser chat under the default policy, a killed Hermes child leaving the runtime running, and a first `destroy` that succeeds. The live runner now also builds the Hermes image and installs the pinned OpenShell 0.1.6 CLI. - **Docs:** `docs/interfaces.md`, `docs/agents.md` and `docs/limits.md` cover enabling, forwarding, token retrieval, sign-in, the token's visibility to the agent's own tools, and Hermes's dashboard having no sign-in. Stacked on #14036. Part of #14011. ## Validation - `cargo ci` on this branch's tip: 1,787 unit tests and 161 lifecycle tests, plus fmt, clippy, build, schema and bundle, passed. - `cargo ci live-docker` with the next PR's commit on top: all 10 tests passed in 315.9 s, within the 360 s budget; `interfaces_live` passed in 51.7 s. - A manual test on Docker with a real model passed the OpenClaw and Hermes items; results are on #14011. - Deferred to CI: amd64 builds and `live-kind`, since this host is ARM64. --------- Signed-off-by: Carlos Villela <[email protected]>
cv
added a commit
that referenced
this pull request
Oct 31, 2026
<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. --> <!-- SPDX-License-Identifier: Apache-2.0 --> ## Failure `linux_amd64 / Live / Kind` failed intermittently after #14036 and #14037, on #14037 and #14024 (both passed on rerun). In `kubernetes_live::on_openshift_three_agents_keep_separate_identities_run_as_the_namespace_uid_and_answer_after_each_reapply`, one of three OpenClaw sandboxes failed with `start_failed`, and CI never showed why, because Fabric's message stayed in the runner's temporary directory. The cause is CPU starvation. Three agents start at once on four vCPUs, and each OpenClaw start took about 39 s on that runner, measured from the supervisor's relay logs. Fabric's OpenClaw adapter gave each startup step its own 30 s timeout, so the Gateway's readiness wait ran out at random. Local `live-kind` runs reproduced the same error with a shorter timeout, and start time tracked the Kind node's CPUs: about 7.5 s with 20, 11.4 s with 4 and 26 s with 2. The denied `catalog.openclaw.ai` lookup in the logs isn't the cause: OpenClaw starts it in the background after the Gateway is ready, and timing showed no difference with and without it. ## Decision - **Fix in Fabric's OpenClaw adapter, not NemoClaw.** `startup_timeout_seconds` is now one deadline for all of `start`, from checking the `openclaw` command to the Gateway becoming ready, with a 60 s default and the existing 75 s maximum. That's the rule NVIDIA/NeMo-Fabric#547 (L3) already sets for adapters that gain the setting. It's in the reference implementation as cv/NeMo-Fabric@a7c5d024. This PR pins Fabric `21ca8d3e`, which includes it, through NVIDIA's codeload URL as before. The catalog changes only OpenClaw's default and description, plus the revision and checksum. The HCL e2e cases' catalog-derived names and image digest are updated. - **Make the next failure diagnosable.** When a live Kubernetes apply fails, the test prints one redacted, length-limited line from each `diagnostics/fabric-SANDBOX.json`: stage, code, effects and message. Fabric's runtime server currently replaces the adapter's error with "Fabric could not establish an active runtime", so naming the cause also needs a Fabric change. - **Didn't raise a NemoClaw budget.** Setting `startup_timeout_seconds` in the live documents, or running fewer Kind tests in parallel, would have hidden the adapter's tight default for everyone else. - **One test adjusted:** `journey_tree_preview::invalid_fabric_setting_is_labeled_without_printing_its_value` checks that the value `42` never appears in the preview. The new revision contains "42", so the test now skips the "Fabric revision:" line, as it already skipped `sha256:` lines. ## Validation - Rebased on `v1` at `530e53d706` (#14024), then `cargo ci`: fmt, clippy, build, test 1791/1791, schema, bundle, lifecycle 161/161. - Before the rebase, on linux_arm64: - `cargo ci live-kind` passed three times, 7/7 each, once with the Kind node limited to 4 CPUs; - `cargo ci live-docker` passed. - The diagnostics unit test failed against an empty stub before the implementation. - `cargo metadata --locked` passes. `Cargo.lock` changes only the source line, because `cargo update -p nemo-fabric-core` also downgraded `windows-sys`. - Deferred to CI: `linux_amd64 / Live / Kind`, where the flake happened, since this machine is arm64. Part of #14011. --------- Signed-off-by: Carlos Villela <[email protected]>
This was referenced Oct 31, 2026
kaofelix
added a commit
that referenced
this pull request
Nov 2, 2026
## Failure The `verify-yaml-deployment` skill had drifted from `v1`. Fabric's pinned adapters now serve OpenClaw's Control UI and Hermes's API server and dashboard (#14036, #14037), and apply prints how to forward each one and the command that reads its token. The skill instead patched the OpenClaw adapter, read the gateway token from the host's `/proc` with sudo, set `port_range`, and said Hermes had no interfaces. A `maintain-verification-skill` pass also found errors in the generic features. After an exposure change, the skill reused the old runtime ID, which fails with `runtime_unavailable`. Its edit step overwrote existing exposures, and the credential-leak scan broke when no key was set. ## Decision - **openclaw-dashboard:** declare `harness.settings.interfaces.dashboard` with `public_origins`, read the token with the `Token` command apply prints, and check sign-in on `/control-ui-config.json` through the exposure and the printed forward. There's no patched image, `/proc` read, or `port_range`. - **hermes-interfaces (new):** the API server and dashboard through the printed forwards. The API token comes from its kept-state path, which stays the same across runtimes. - **deploy-and-answer:** check `error` and the runtime ID in the result, and scan for the key recursively. The model-name gotcha now says it applies to an existing endpoint, not a managed service. The `runtime_unavailable` gotcha now says to read the runtime ID again. - **export-reapply:** export with `--output`, plan the export first, and check that the runtime is unchanged and the **Agent interfaces** block matches. - **exposure-change:** keep existing exposures, expect the agent runtime to be replaced too, prove the old sandbox is gone, and read the new runtime ID before asking again. - **Agent files:** carry the new YAML edits, gaps, and proof rows, and the feature map gives the run order. ## Validation - Four read-only reviewers, one per feature file, checked each against source and docs at `24b35280fb`. - Every feature was driven live on a DGX Spark (linux/arm64, OpenShell 0.1.6) at `24b35280fb` against an existing vLLM, for OpenClaw and Hermes. - **OpenClaw:** - Apply printed **Service URLs** and **Agent interfaces**, and the `Token` command read the token. - `/control-ui-config.json` returned 401, 401, and 200 through both the exposure and the forward. - Browser chat replied. - **Hermes:** - The API server returned 401, 401, and its model list with the token, and a chat request answered. - The dashboard served its page, and `/api/health` reported no sign-in. - **Both agents:** - Export and reapply were unchanged. - The exposure change replaced the sandbox, and the agent answered under its new runtime ID. - Destroy confirmed, and no object with either run's UID remained. - I checked a few small command changes, including the rewritten exposure-change step, against those runs' results rather than re-driving them. The new step behaves the same in `sh` and `bash`. - Links, anchors, and YAML blocks resolve. `cargo ci fmt` and `fern check` pass, and `cargo ci` selects no bundle, lifecycle, or live steps for this Markdown-only change. - **Not run:** Deep Agents, managed model services, and targets other than DGX Spark.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Failure
NemoClaw can't offer the P0 agents' native interfaces (#14011): the pinned Fabric (
7aedb66) has no interface support in its OpenClaw, Hermes or Deep Agents Code adapters. Its OpenClaw and Hermes pins are also behind what the newer adapters support: Fabric's OpenClaw adapter now accepts only OpenClaw 2026.9.34, and its Hermes adapter targets Hermes v0.22.5.Decision
Pin Fabric
6cad3504, the reference implementation of NVIDIA/NeMo-Fabric#541, #540 and #546 oncv/NeMo-Fabric, on top of Fabricmain0.7.0 (624e6876). As #14011 allows, NemoClaw uses it until Fabric takes the changes, and fetches it through the upstream repository URLs, which serve the fork's commit.6cad3504includes the fix for stopping interface children and ACP agents inside OpenShell, which denies signals to a process group. This PR enables no interfaces; the next PR in the stack does.FABRIC_REVISIONandFABRIC_SHA256,nemo-fabric-corein three crates andCargo.lock, and the pinned documentation links.OPENCLAW_IMAGEmoves to the 2026.9.34 index digest.bf0203a7), its image digest, andHERMES-NOTICE.md. Hermes's lock overrides browser-harness's Pillow 12.3.0 pin with 13.0.0, which re-resolving the export rejects. The image now installs the exported closure without re-resolving, and Hermes's ownpm.pip_checkaccepts only the mismatches those overrides cause;image/NOTICE.mddocuments this.nemo-fabric-evidence, which the image builds from the same source.settings_schema.properties.interfaces. Claude, Codex and Pi now declareinsecure_http_model_hosts, so NemoClaw lists plain-HTTP endpoint hosts for them as it does for OpenCode. OpenClaw declaressupports_usage_in_streaming, which onboarding now asks about.fabric_agent: Fabric's newprovenancesection becomes an input like every other section, and the provider schema snapshot gains it.Part of #14011, under the P0 specification in NVIDIA/NeMo-Fabric#547.
Validation
cargo cion the rebased branch: fmt, clippy, build, 1,773 tests, schema, bundle and 161 lifecycle tests passed.cargo ci live-docker: passed.cargo metadata --locked: passed.v1, all 12 agent images and the dummy image built for linux/arm64 and passedcargo images qualify; the rebase didn't change image inputs.live-kind, since this host is ARM64.