feat(nemoclaw-openshell-kubernetes): add Kubernetes and OpenShift Helm recipe by agudanv · Pull Request #166 · NVIDIA/nemoclaw-community

agudanv · 2026-09-03T15:52:00Z

Related Issue

N/A — this contribution adds a Kubernetes deployment tool and a separate NVIDIA SRE recipe.

Description

Adds two Helm entry points:

examples/tools/kubernetes-deployer: deploys the official, digest-pinned NemoClaw-managed Hermes image behind an OpenShell gateway on Kubernetes or OpenShift. It provides dashboard, API, and terminal access without invoking the NemoClaw CLI or bootstrap.
examples/recipes/nvidia/kubernetes-sre-assistant: layers the SRE skill, authenticated cluster API proxy, scoped RBAC, and optional model operations onto the deployer. The optional auto-heal controller is disabled by default; remediation requires explicit acknowledgement.

The deployer includes an explicit anonymous OIDC-discovery decision and startup probe, retained-state ownership checks before mutation, cert-generation hook cleanup, and an authenticated Helm-driven Sandbox teardown. Both examples declare NemoClaw: N/A and retain the managed-image version in their provenance sections.

Model API keys remain in referenced Kubernetes Secrets. The documented defaults and acknowledgements preserve the boundaries for cluster access, dashboard exposure, broader operations, and model deletion. No custom image build is required.

Maintainer Follow-up

Reload file-backed Kubernetes API tokens before each auto-heal request. Missing, unreadable, or empty files fail the request without reusing a stale token. Explicit token values retain precedence.
Restrict failed-Pod cleanup to Pods recorded in the approved finding, leaving unrelated workloads and recovered Pods unchanged.
Add ten runtime regression tests covering token rotation, projected symlink replacement, read failures and recovery, explicit-token precedence, cleanup scope, and dry-run behavior.
Merge current main without rewriting published history; regenerate the catalog indexes to resolve conflicts and update the catalog-count regression for the two new examples.
Correct stale README verification text and document the local modifications to the vendored auto-heal controller.

Original contribution and authorship remain credited to Anurag Guda.

Verification

Maintainer verification of the updated revision:

Deployer: python3.12 -m unittest tests/test_chart.py — 94 passed.
SRE recipe: python3.12 -m unittest discover -s tests -p 'test_*.py' — 53 passed, including the ten new runtime regressions. The test fixture rebuilds the sibling Helm dependency.
Repository: python -m unittest discover -s scripts/tests -p 'test_*.py' — 74 passed.
Repository: node --test scripts/tests/*.test.mjs — 14 passed.
python scripts/build_catalog.py --check — 24 examples across seven groups passed.
python3.12 -S scripts/check_license_headers.py --check — 859 checked source files passed.
python3.12 scripts/check_label_taxonomy.py --check — 63 labels passed.
Deployer Kubernetes Helm lint and Kubernetes/OpenShift template renders passed, with the documented API versions and offline OpenShift UID values.
git diff --check and staged whitespace checks passed.

Catalog commands and repository tests used uv run --no-project --python 3.12 --with-requirements scripts/catalog-requirements.txt. One initial repository test run found a stale catalog-count expectation after merging main; the updated expectation and complete rerun passed.

The example READMEs retain the contributor-reported OpenShift and standard-Kubernetes live evidence and its limits. The maintainer follow-up used local regression tests; it did not contact a cluster, run model inference, or repeat live token rotation, cleanup, installation, or teardown. Broader access, model deployment, metrics, and deletion retain their documented local/static verification limits.

Documentation Writer Review

Result: docs-updated.
Evidence: both example READMEs and this description now reflect the split entry points, current behavior, local test commands, and verification limits.
Reviewer: Codex, acting at the maintainer's request. Changed text was checked against the repository writing guide and controlled-word list.

Release And Compliance

Original contributor attribution and SPDX headers are preserved.
No new third-party dependency was added by the maintainer fixes.
No credentials or private deployment details were added.

Signed-off-by: Anurag Guda [email protected]
Signed-off-by: Apurv Kumaria [email protected]

Signed-off-by: Anurag Guda <[email protected]>

apurvvkumaria

Thanks for the substantial remediation. The earlier request-path, package-size, and third-party provenance concerns are resolved at da0aac833b8381903a2ae7279773beabff27fa64, and all current checks are green.

Approval is still blocked by the inline default-runtime, retained-state, teardown, classification/name, and catalog-metadata findings.

Please also update the Documentation Writer Review record: passed is not one of the template values (docs-updated, no-docs-needed, or blocked); the evidence should name the changed documentation paths; and the Reviewer field should clearly identify the final documentation reviewer.

Resolves scripts/tests/test_example_stack_facts.py by taking main's version and raising the unconfirmed count to 8 for the Kubernetes recipe this branch carries. Co-Authored-By: Claude Fable 5.1 <[email protected]>

Signed-off-by: Anurag Guda <[email protected]>

PicoNVIDIA · 2026-09-16T21:12:58Z

First install stalls with WaitForFirstConsumer storage

I hit a first-install deadlock on Helm v3.16.4 and K3s v1.34.11+k3s1 with local-path storage, testing this PR at cf03dcdd38e6a6627a9946b6cc9f4d9125c892f9. The gateway became Ready and seed completed, but the NemoClaw home PVC stayed Pending with no consumer.

In nemoclaw-home.yaml, the preparation Job is a post-install hook. Helm's --wait requires the PVC to bind before creating that Job, while WaitForFirstConsumer needs a consuming Pod before binding. Changing the hook weight won't break that cycle.

I unblocked installation by running the unchanged preparation Pod spec in an additional Job. The PVC bound and the original hooks completed, with --wait and the NemoClaw security gate still enabled.

Could preparation run before Helm's resource wait, with bootstrap explicitly waiting for it to succeed? A pre-install hook would also need its PVC and ServiceAccount available first. A fresh-install WFFC test would catch this; I haven't reproduced an upgrade failure with an already-bound claim.

PicoNVIDIA · 2026-09-16T21:13:00Z

K3s issuer-discovery prerequisite

On K3s v1.34.11+k3s1 with OpenShell 0.0.116, anonymous issuer discovery returned 401 despite the Kubernetes profile's public-discovery RBAC binding. The gateway crash-looped with OIDC discovery response parse failed.

K3s defaults to anonymous-auth=false when no authentication-config is supplied. RBAC grants access to an identity; it doesn't enable anonymous authentication.

In the disposable cluster, I kept the binding and added an AuthenticationConfiguration allowing anonymous requests only to /.well-known/openid-configuration and /openid/v1/jwks, including their trailing-slash variants. The canonical paths then returned 200, namespace and Secret requests still returned 401, and the gateway became Ready. OpenShell authentication stayed enabled. I didn't separately probe the trailing-slash variants.

Could the profile document this prerequisite, supported versions, and how to preserve existing authentication settings? The current bootstrap check runs after Helm waits for the gateway, too late for this failure. An earlier credential-free, TLS-verified check of discovery and the advertised JWKS URL would help.

OpenShell #3332 already improves the error message; this is about the recipe prerequisite, not another parsing-error report.

…nto codex/fix-pr166-autoheal Signed-off-by: Apurv Kumaria <[email protected]> # Conflicts: # examples/README.md # examples/tools/README.md

Signed-off-by: Apurv Kumaria <[email protected]>

apurvvkumaria

Reviewed and fixed the two auto-heal findings at 1ea40df: file-backed API credentials reload before each request, and failed-Pod cleanup is limited to the approved finding. Ten new regressions pass. Full local verification passed: 94 deployer tests, 53 recipe/runtime tests, 74 repository Python tests, 14 Node tests, both platform renders, catalog validation, SPDX, taxonomy, and whitespace checks. Required GitHub checks are green. The seven earlier review threads have been rechecked and resolved with supporting notes. The tool/recipe split and current artifact names are accepted in this maintainer review. Original attribution is preserved and published history was not rewritten. No new live cluster, inference, installation, or teardown run was performed; contributor-reported live evidence and the documented verification limits remain distinct from these local checks.

anuguda added 10 commits September 1, 2026 11:49

NemoClaw Kubernetes Deployment

d7f0792

Fix SRE proxy Secret rendering

5fa1265

Support projected SRE proxy token

89936bf

Harden OpenShift sandbox and SRE runtime

eb2f082

Add in-cluster Hermes terminal

26911ef

Merge branch 'NVIDIA:main' into aguda/nemoclaw-kubernetes

cde7ad7

NemoClaw Kubernetes Deployment

fa2ba75

Merge branch 'NVIDIA:main' into aguda/nemoclaw-kubernetes

66df036

docs: add NemoClaw architecture diagram

ed16f87

fix: resolve Kubernetes recipe review blockers

da0aac8

Signed-off-by: Anurag Guda <[email protected]>

agudanv marked this pull request as ready for review September 3, 2026 19:16

agudanv requested review from apurvvkumaria, cv, ericksoa and wscurran as code owners September 3, 2026 19:16

apurvvkumaria reviewed Sep 3, 2026

View reviewed changes

anuguda and others added 2 commits September 11, 2026 12:46

Merge NVIDIA main into aguda/nemoclaw-kubernetes

8d3a5c7

Resolves scripts/tests/test_example_stack_facts.py by taking main's version and raising the unconfirmed count to 8 for the Kubernetes recipe this branch carries. Co-Authored-By: Claude Fable 5.1 <[email protected]>

Fix: rearrange the recipe and kubernetes deployer

cf03dcd

Signed-off-by: Anurag Guda <[email protected]>

PicoNVIDIA mentioned this pull request Sep 16, 2026

Sandbox stuck in unrecoverable Error phase after supervisor session drops (SSH disconnect); stop/start refuse to act NVIDIA/OpenShell#3308

Closed

apurvvkumaria added 2 commits September 16, 2026 15:23

Merge branch 'main' of https://github.com/NVIDIA/nemoclaw-community i…

7252158

…nto codex/fix-pr166-autoheal Signed-off-by: Apurv Kumaria <[email protected]> # Conflicts: # examples/README.md # examples/tools/README.md

fix(kubernetes-sre-assistant): reload tokens and bound cleanup scope

1ea40df

Signed-off-by: Apurv Kumaria <[email protected]>

apurvvkumaria approved these changes Sep 16, 2026

View reviewed changes

apurvvkumaria merged commit ac08dd9 into NVIDIA:main Sep 16, 2026
9 checks passed

Navigation Menu

Repository navigation

feat(nemoclaw-openshell-kubernetes): add Kubernetes and OpenShift Helm recipe - #166

feat(nemoclaw-openshell-kubernetes): add Kubernetes and OpenShift Helm recipe#166
apurvvkumaria merged 14 commits into
NVIDIA:mainfrom
agudanv:aguda/nemoclaw-kubernetes

agudanv commented Sep 3, 2026 •

edited by apurvvkumaria

Loading

Uh oh!

apurvvkumaria left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

PicoNVIDIA commented Sep 16, 2026

Uh oh!

PicoNVIDIA commented Sep 16, 2026

Uh oh!

apurvvkumaria left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

agudanv commented Sep 3, 2026 • edited by apurvvkumaria Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related Issue

Description

Maintainer Follow-up

Verification

Documentation Writer Review

Release And Compliance

Uh oh!

apurvvkumaria left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

PicoNVIDIA commented Sep 16, 2026

First install stalls with WaitForFirstConsumer storage

Uh oh!

PicoNVIDIA commented Sep 16, 2026

K3s issuer-discovery prerequisite

Uh oh!

apurvvkumaria left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

agudanv commented Sep 3, 2026 •

edited by apurvvkumaria

Loading