Skip to content

build(fabric): pin Fabric with the P0 interface adapters, OpenClaw 2026.9.34 and Hermes v0.22.5 - #14036

Merged
cv merged 2 commits into
v1from
deps/fabric-interfaces-pin
Oct 31, 2026
Merged

cv merged 2 commits into
v1from
deps/fabric-interfaces-pin

Conversation

@cv

@cv cv commented Oct 31, 2026

Copy link
Copy Markdown
Collaborator

Failure

NemoClaw can't offer the P0 agents' native interfaces (#14011): the pinned Fabric (7aedb66) has no interface support in its OpenClaw, Hermes or Deep Agents Code adapters. Its OpenClaw and Hermes pins are also behind what the newer adapters support: Fabric's OpenClaw adapter now accepts only OpenClaw 2026.9.34, and its Hermes adapter targets Hermes v0.22.5.

Decision

Pin Fabric 6cad3504, the reference implementation of NVIDIA/NeMo-Fabric#541, #540 and #546 on cv/NeMo-Fabric, on top of Fabric main 0.7.0 (624e6876). As #14011 allows, NemoClaw uses it until Fabric takes the changes, and fetches it through the upstream repository URLs, which serve the fork's commit. 6cad3504 includes the fix for stopping interface children and ACP agents inside OpenShell, which denies signals to a process group. This PR enables no interfaces; the next PR in the stack does.

  • Pins: FABRIC_REVISION and FABRIC_SHA256, nemo-fabric-core in three crates and Cargo.lock, and the pinned documentation links.
  • OpenClaw: OPENCLAW_IMAGE moves to the 2026.9.34 index digest.
  • Hermes: v0.22.5 (bf0203a7), its image digest, and HERMES-NOTICE.md. Hermes's lock overrides browser-harness's Pillow 12.3.0 pin with 13.0.0, which re-resolving the export rejects. The image now installs the exported closure without re-resolving, and Hermes's own pm.pip_check accepts only the mismatches those overrides cause; image/NOTICE.md documents this.
  • Wheels: Fabric's runtime and adapter contract now depend on nemo-fabric-evidence, which the image builds from the same source.
  • Catalog: regenerated. OpenClaw and Hermes declare settings_schema.properties.interfaces. Claude, Codex and Pi now declare insecure_http_model_hosts, so NemoClaw lists plain-HTTP endpoint hosts for them as it does for OpenCode. OpenClaw declares supports_usage_in_streaming, which onboarding now asks about.
  • fabric_agent: Fabric's new provenance section becomes an input like every other section, and the provider schema snapshot gains it.
  • Tests: contract fixtures and the HCL e2e scenarios' catalog-derived resource names are regenerated; onboarding tests use a startup timeout within the adapters' new 75-second maximum.

Part of #14011, under the P0 specification in NVIDIA/NeMo-Fabric#547.

Validation

  • cargo ci on the rebased branch: fmt, clippy, build, 1,773 tests, schema, bundle and 161 lifecycle tests passed.
  • cargo ci live-docker: passed.
  • cargo metadata --locked: passed.
  • Before the rebase onto the current v1, all 12 agent images and the dummy image built for linux/arm64 and passed cargo images qualify; the rebase didn't change image inputs.
  • A manual test of the whole stack ran on Docker with a real model; results are on Offer the P0 agents' native interfaces: OpenClaw's Control UI, Hermes's dashboard and API, and Deep Agents Code's terminal #14011.
  • Deferred to CI: amd64 image builds and qualification, and live-kind, since this host is ARM64.

cv added 2 commits October 30, 2026 14:25
…26.9.34 and Hermes v0.22.5

Step 1 of #14011. Pin Fabric 78e79bfc, the reference implementation of
the P0 interface adapters (NVIDIA/NeMo-Fabric#541, #540, #546), fetched
through the upstream repository. It sits on Fabric main 624e6876
(0.7.0). This commit enables no interfaces.

- Pins: FABRIC_REVISION and FABRIC_SHA256, nemo-fabric-core 0.7.0 in
  three crates and Cargo.lock, and pinned documentation links.
- OpenClaw: the adapter supports only 2026.9.34, so OPENCLAW_IMAGE
  moves to that release's index digest.
- Hermes: v0.22.5 (bf0203a7), the release Fabric tests, and its image
  digest. Its lock overrides browser-harness's Pillow 12.3.0 pin with
  13.0.0, which re-resolving the export rejects. The image now installs
  the exported closure without re-resolving, and Hermes's own
  pm.pip_check accepts only the uv pip check mismatches those overrides
  cause. aiohttp, web_dist and tui_dist still install.
- Wheels: nemo-fabric-runtime and the adapter contract now depend on
  nemo-fabric-evidence, which the image builds from the same source.
- Catalog: regenerated. OpenClaw and Hermes gain
  settings_schema.properties.interfaces; Claude, Codex and Pi gain
  insecure_http_model_hosts, so NemoClaw now lists plain-HTTP endpoint
  hosts for them as it does for OpenCode; OpenClaw gains the model
  setting supports_usage_in_streaming, which onboarding asks for.
- fabric_agent: Fabric's new provenance section becomes an input like
  every other section; the provider schema snapshot gains its block.
- Tests: contract fixtures regenerated; onboarding tests omit OpenClaw's
  interfaces setting and use a startup timeout within the new maximum
  of 75 seconds.
- HCL scenarios: the fixture image's digest covers the bundled catalog,
  and the OpenShift scenario's inference addresses derive from it, so the
  case documents and openshift-external-two-providers.tftest.hcl follow
  the new catalog.

Signed-off-by: Carlos Villela <[email protected]>
Live tests for #14011 found that OpenShell's sandbox refuses signals to a
process group (EPERM), so stopping Hermes's interface children and
Fabric's ACP agents, such as Deep Agents Code, failed. Fabric 6cad3504
fixes both: it merges the Hermes fix (7981efca) into the Deep Agents
Code branch and fixes the ACP runner.

Move every Fabric pin from 78e79bfc to 6cad3504: FABRIC_REVISION and
FABRIC_SHA256, nemo-fabric-core in three crates and Cargo.lock, the
bundled catalog, and pinned documentation links. No adapter descriptor
changed, so the regenerated catalog differs only in fabric_revision and
source_sha256.

The catalog change moves the fixture image's digest again, so the HCL
case documents and openshift-external-two-providers.tftest.hcl follow it.

Signed-off-by: Carlos Villela <[email protected]>
@cv

cv commented Oct 31, 2026

Copy link
Copy Markdown
Collaborator Author
Authoring session — opening
Field Status Value Evidence or limit
Harness partial Pi, as a subagent in its own worktree Session record; it doesn't record Pi's version. A coordinating Pi session wrote the task prompts and is not counted here
Skills partial test-driven-development (complete read) Complete reads found in the record to the cutoff; skill listings injected at startup aren't stored
Instruction files partial AGENTS.md, WRITING.md, .github/PULL_REQUEST_TEMPLATE.md, docs/contributing/testing.md, docs/contributing/live-tests.md (complete reads) Reads found in the record; startup project-context injection isn't stored
Tool calls partial typescript: 152 Includes 3 failed calls; the typescript wrapper hides nested calls
Session window recorded 2026-10-30T13:41Z to 2026-10-30T21:30Z; 469 min First event to the final commit, including idle time between tasks
Compactions recorded 0 Complete record to the cutoff
Models recorded claude-opus-5-5 (Azure); claude-opus-5-5 (Bedrock) Assistant messages; moved to the second endpoint after rate limits
Reasoning levels recorded high Thinking level at session start; no changes

@github-actions

Copy link
Copy Markdown
Contributor

@cv
cv merged commit 6d6bf06 into v1 Oct 31, 2026
56 checks passed
@cv
cv deleted the deps/fabric-interfaces-pin branch October 31, 2026 00:25
@wscurran wscurran added area: integrations Third-party service integration behavior area: packaging Packages, images, registries, installers, or distribution area: providers Inference provider integrations and provider behavior area: security Security controls, permissions, secrets, or hardening chore Build, CI, dependency, or tooling maintenance integration: dcode LangChain Deep Code integration behavior integration: hermes Hermes integration behavior integration: openclaw OpenClaw integration behavior v1 NemoClaw v1 branch labels Oct 31, 2026
cv added a commit that referenced this pull request Oct 31, 2026
## Failure

With the Fabric pin from #14036, OpenClaw's and Hermes's adapters can
serve their native interfaces, but a NemoClaw deployment can't use them:
planning doesn't check `harness.settings.interfaces` against the image's
adapter, apply doesn't say how to reach an interface, and the docs don't
cover it (#14011, #13360, #13973, #13969). Hermes's browser chat also
fails with "out of pty devices", because the agent images' default
filesystem policy doesn't allow pseudo-terminals.

## Decision

- **Planning:** a new `harness_interfaces` check rejects interface names
and fields the image's adapter doesn't declare in its descriptor's
`settings_schema.properties.interfaces`. An adapter that declares none
supports none.
- **Apply output:** for each enabled interface, apply prints how to
select the gateway and workspace, the `openshell forward service`
command, the `openshell sandbox exec` command that reads the token from
the runtime's `interfaces/` directory, and how to sign in. `-o json`
carries the same facts under `interfaces`. NemoClaw never reads, prints
or stores a token.
- **Health:** no change was needed: the provider decides whether the
agent runs from the runtime check alone, so a failed interface child
doesn't change it.
- **Per-image policy:** each image's adapter declaration may list extra
read-write paths, which `catalog.py` adds to that image's default
policy. Only the Hermes image declares `/dev/pts` and `/dev/ptmx`; every
other image keeps its policy.
- **Live test:** `interfaces_live` applies OpenClaw with its dashboard
and Hermes with its API server and dashboard on a managed Docker
gateway, with a model stand-in that needs a key, and checks sign-in,
token rejection, the model key, browser chat under the default policy, a
killed Hermes child leaving the runtime running, and a first `destroy`
that succeeds. The live runner now also builds the Hermes image and
installs the pinned OpenShell 0.1.6 CLI.
- **Docs:** `docs/interfaces.md`, `docs/agents.md` and `docs/limits.md`
cover enabling, forwarding, token retrieval, sign-in, the token's
visibility to the agent's own tools, and Hermes's dashboard having no
sign-in.

Stacked on #14036. Part of #14011.

## Validation

- `cargo ci` on this branch's tip: 1,787 unit tests and 161 lifecycle
tests, plus fmt, clippy, build, schema and bundle, passed.
- `cargo ci live-docker` with the next PR's commit on top: all 10 tests
passed in 315.9 s, within the 360 s budget; `interfaces_live` passed in
51.7 s.
- A manual test on Docker with a real model passed the OpenClaw and
Hermes items; results are on #14011.
- Deferred to CI: amd64 builds and `live-kind`, since this host is
ARM64.

---------

Signed-off-by: Carlos Villela <[email protected]>
cv added a commit that referenced this pull request Oct 31, 2026
<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION &
AFFILIATES. All rights reserved. -->
<!-- SPDX-License-Identifier: Apache-2.0 -->

## Failure

`linux_amd64 / Live / Kind` failed intermittently after #14036 and
#14037, on #14037 and #14024 (both passed on rerun). In
`kubernetes_live::on_openshift_three_agents_keep_separate_identities_run_as_the_namespace_uid_and_answer_after_each_reapply`,
one of three OpenClaw sandboxes failed with `start_failed`, and CI never
showed why, because Fabric's message stayed in the runner's temporary
directory.

The cause is CPU starvation. Three agents start at once on four vCPUs,
and each OpenClaw start took about 39 s on that runner, measured from
the supervisor's relay logs. Fabric's OpenClaw adapter gave each startup
step its own 30 s timeout, so the Gateway's readiness wait ran out at
random. Local `live-kind` runs reproduced the same error with a shorter
timeout, and start time tracked the Kind node's CPUs: about 7.5 s with
20, 11.4 s with 4 and 26 s with 2. The denied `catalog.openclaw.ai`
lookup in the logs isn't the cause: OpenClaw starts it in the background
after the Gateway is ready, and timing showed no difference with and
without it.

## Decision

- **Fix in Fabric's OpenClaw adapter, not NemoClaw.**
`startup_timeout_seconds` is now one deadline for all of `start`, from
checking the `openclaw` command to the Gateway becoming ready, with a 60
s default and the existing 75 s maximum. That's the rule
NVIDIA/NeMo-Fabric#547 (L3) already sets for adapters that gain the
setting. It's in the reference implementation as
cv/NeMo-Fabric@a7c5d024. This PR pins Fabric
`21ca8d3e`, which includes it, through NVIDIA's codeload URL as before.
The catalog changes only OpenClaw's default and description, plus the
revision and checksum. The HCL e2e cases' catalog-derived names and
image digest are updated.
- **Make the next failure diagnosable.** When a live Kubernetes apply
fails, the test prints one redacted, length-limited line from each
`diagnostics/fabric-SANDBOX.json`: stage, code, effects and message.
Fabric's runtime server currently replaces the adapter's error with
"Fabric could not establish an active runtime", so naming the cause also
needs a Fabric change.
- **Didn't raise a NemoClaw budget.** Setting `startup_timeout_seconds`
in the live documents, or running fewer Kind tests in parallel, would
have hidden the adapter's tight default for everyone else.
- **One test adjusted:**
`journey_tree_preview::invalid_fabric_setting_is_labeled_without_printing_its_value`
checks that the value `42` never appears in the preview. The new
revision contains "42", so the test now skips the "Fabric revision:"
line, as it already skipped `sha256:` lines.

## Validation

- Rebased on `v1` at `530e53d706` (#14024), then `cargo ci`: fmt,
clippy, build, test 1791/1791, schema, bundle, lifecycle 161/161.
- Before the rebase, on linux_arm64:
- `cargo ci live-kind` passed three times, 7/7 each, once with the Kind
node limited to 4 CPUs;
  - `cargo ci live-docker` passed.
- The diagnostics unit test failed against an empty stub before the
implementation.
- `cargo metadata --locked` passes. `Cargo.lock` changes only the source
line, because `cargo update -p nemo-fabric-core` also downgraded
`windows-sys`.
- Deferred to CI: `linux_amd64 / Live / Kind`, where the flake happened,
since this machine is arm64.

Part of #14011.

---------

Signed-off-by: Carlos Villela <[email protected]>
kaofelix added a commit that referenced this pull request Nov 2, 2026
## Failure

The `verify-yaml-deployment` skill had drifted from `v1`. Fabric's
pinned adapters now serve OpenClaw's Control UI and Hermes's API server
and dashboard (#14036, #14037), and apply prints how to forward each one
and the command that reads its token. The skill instead patched the
OpenClaw adapter, read the gateway token from the host's `/proc` with
sudo, set `port_range`, and said Hermes had no interfaces. A
`maintain-verification-skill` pass also found errors in the generic
features. After an exposure change, the skill reused the old runtime ID,
which fails with `runtime_unavailable`. Its edit step overwrote existing
exposures, and the credential-leak scan broke when no key was set.

## Decision

- **openclaw-dashboard:** declare
`harness.settings.interfaces.dashboard` with `public_origins`, read the
token with the `Token` command apply prints, and check sign-in on
`/control-ui-config.json` through the exposure and the printed forward.
There's no patched image, `/proc` read, or `port_range`.
- **hermes-interfaces (new):** the API server and dashboard through the
printed forwards. The API token comes from its kept-state path, which
stays the same across runtimes.
- **deploy-and-answer:** check `error` and the runtime ID in the result,
and scan for the key recursively. The model-name gotcha now says it
applies to an existing endpoint, not a managed service. The
`runtime_unavailable` gotcha now says to read the runtime ID again.
- **export-reapply:** export with `--output`, plan the export first, and
check that the runtime is unchanged and the **Agent interfaces** block
matches.
- **exposure-change:** keep existing exposures, expect the agent runtime
to be replaced too, prove the old sandbox is gone, and read the new
runtime ID before asking again.
- **Agent files:** carry the new YAML edits, gaps, and proof rows, and
the feature map gives the run order.

## Validation

- Four read-only reviewers, one per feature file, checked each against
source and docs at `24b35280fb`.
- Every feature was driven live on a DGX Spark (linux/arm64, OpenShell
0.1.6) at `24b35280fb` against an existing vLLM, for OpenClaw and
Hermes.
- **OpenClaw:**
- Apply printed **Service URLs** and **Agent interfaces**, and the
`Token` command read the token.
- `/control-ui-config.json` returned 401, 401, and 200 through both the
exposure and the forward.
  - Browser chat replied.
- **Hermes:**
- The API server returned 401, 401, and its model list with the token,
and a chat request answered.
- The dashboard served its page, and `/api/health` reported no sign-in.
- **Both agents:**
  - Export and reapply were unchanged.
- The exposure change replaced the sandbox, and the agent answered under
its new runtime ID.
  - Destroy confirmed, and no object with either run's UID remained.
- I checked a few small command changes, including the rewritten
exposure-change step, against those runs' results rather than re-driving
them. The new step behaves the same in `sh` and `bash`.
- Links, anchors, and YAML blocks resolve. `cargo ci fmt` and `fern
check` pass, and `cargo ci` selects no bundle, lifecycle, or live steps
for this Markdown-only change.
- **Not run:** Deep Agents, managed model services, and targets other
than DGX Spark.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: integrations Third-party service integration behavior area: packaging Packages, images, registries, installers, or distribution area: providers Inference provider integrations and provider behavior area: security Security controls, permissions, secrets, or hardening chore Build, CI, dependency, or tooling maintenance integration: dcode LangChain Deep Code integration behavior integration: hermes Hermes integration behavior integration: openclaw OpenClaw integration behavior v1 NemoClaw v1 branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants