Skip to content

Commit 3baa673

Browse files
gaveezyclaudecv
authored
feat(kubernetes): publish a managed gateway through a NodePort Service (#14324)
A managed Kubernetes or OpenShift gateway can be published at `spec.gateway.endpoint` through the OpenShell chart's own NodePort Service, with `spec.gateway.kubernetes.expose`, so clients such as the OpenShell CLI reach it there, as at a Docker gateway's published port. NemoClaw's own commands keep their port-forward. ## Problem A managed Kubernetes gateway has no endpoint to dial. The schema requires `https://127.0.0.1:PORT`, nothing listens there, and each provider port-forwards to the gateway pod while OpenTofu runs. People reach the gateway with `kubectl port-forward` and three copied certificate files. Part of #14313. This follows the decisions recorded on #14313 after this PR's design reviews: - The chart's own Service publishes the gateway, so NemoClaw bundles no Kubernetes provider. - LoadBalancer waits for chart inputs, which NVIDIA/OpenShell#5500 requests (#14421). - Kubernetes, not NemoClaw, checks the node-port range, and only the Service type is fixed after the first plan. - Other clients dial the published endpoint; NemoClaw's own plan, apply, and destroy keep their port-forward, so they still need `pods/portforward`. #14422 tracks an opt-in direct dial. Left for #14313: runs on clusters other than kind, including OpenShift. Sandbox service URLs on a published gateway still name `openshell.localhost` and the gateway pod's port, so they don't open from other machines. NemoClaw leaves them to OpenShell instead of working around it: #14316 waits on OpenShell to publish them at a configured origin (NVIDIA/OpenShell#4953). ## Change - `gateway.kubernetes.expose.service.type` takes `NodePort`. With `expose`, the endpoint is `https://`, a lowercase DNS name or IPv4 address, and a port; Kubernetes checks the port against the cluster's node-port range. Without `expose`, the `127.0.0.1` rule stays. The rule moves to the document schema, because the platform resources rebuild the Gateway definition from attributes that don't carry `expose`. - The Helm values set the chart's Service to `type: NodePort` with `nodePort` at the endpoint's port, which forwards to the gateway's TLS listener, so TLS and client certificates pass through. `service.metricsPort: 0` keeps the plaintext metrics port off the node: at 0 the pinned chart publishes no metrics port and the gateway serves no metrics (`templates/service.yaml`, `_toml.tpl`). Helm's wait and the release's existing binding cover the Service, so NemoClaw declares no Service of its own. - The chart's `pkiInitJob` names the endpoint host in the gateway certificate. - The providers' connection forwards a free loopback port to the gateway pod and names `127.0.0.1`, which the chart's certificate always covers, also when the endpoint names the published host. - Changing `expose` or its Service type after the first plan is refused, like changing the endpoint. - The intent record moves to version 21, so this release refuses state that an earlier build wrote. - LoadBalancer is left out: the pinned chart's Service takes no annotations, `loadBalancerSourceRanges` or `loadBalancerClass`, and its port is the gateway's own listening port, 8080. A limits row records it, #14421 tracks it, and NVIDIA/OpenShell#5500 asks for the chart inputs. - A chart contract scenario renders the pinned chart with an exposed gateway's values: the release's Service publishes only the gateway's port at the endpoint's node port, the gateway configuration has no metrics listener, and the certificate job names the host. - The `kind-kubernetes-oidc` live case publishes its gateway through the NodePort at the kind node's address, instead of a third case. Its CLI goes through the port-forward, and its probe dials the node's address with the gateway's client certificate and the token, as another client would; each agent must reply there. Live / Kind keeps its 300 s wall budget: in the last CI run before this change, three cases took 342.9 s on linux_arm64 and 368.4 s on linux_amd64, the NodePort case 94.5 s and 97.6 s. `cargo ci live-kind` replaces a published case's endpoint host with the node's address. - The Kubernetes guide, the limits, the release notes, and the test guides describe the change. Deliberately left out: an OpenShift Route, and a ClusterIP dial from a pod. ## Beyond CI - `cargo ci live-kind` ran both kind cases on linux_arm64 within the 300 s budget: 198.6 s wall, the OIDC case 77.9 s. With the chart's Service values removed, the same run's apply still passed through the port-forward and both replies through the node's address failed. - Unverified: a published gateway on a cluster other than kind, including OpenShift. That the gateway serves no metrics at `metricsPort: 0` is checked only in the rendered chart. --------- Signed-off-by: Hai Nguyen <[email protected]> Signed-off-by: Carlos Villela <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: Carlos Villela <[email protected]>
1 parent b094027 commit 3baa673

29 files changed

Lines changed: 654 additions & 55 deletions

File tree

‎.config/test-budgets.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,8 @@ live-docker:
6161
# the development case, which destroys and redeploys, took 137.9 s on
6262
# linux_amd64 and 136.9 s on linux_arm64; with the OIDC case (105.1 s and
6363
# 95.7 s) the step took 243.0 s and 232.6 s.
64+
# The OIDC case also publishes its gateway through the chart's NodePort
65+
# Service (#14313), instead of a third case that took 90-99 s on its own.
6466
live-kind:
6567
wall_seconds: 300
6668
test_seconds: 180

‎crates/nemoclaw-build/src/run_ci/live_kind.rs‎

Lines changed: 45 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -209,6 +209,9 @@ pub(super) async fn run_live_kind(
209209
let pushed = registry
210210
.push(&node_container, &image, &metadata, &pushed_metadata)
211211
.await?;
212+
// The host reaches the node at its address on the kind network, where a
213+
// NodePort Service publishes a case's gateway.
214+
let node = node_address(&node_container)?;
212215

213216
// Each kind HCL E2E case, bound to this cluster and agent image in new
214217
// namespaces, runs as its own nextest command: `hcl_live::case` runs one
@@ -232,7 +235,7 @@ pub(super) async fn run_live_kind(
232235
.join(format!("{case_name}.yaml")),
233236
)?,
234237
&live::uuid()?,
235-
&format!("https://127.0.0.1:{}", live::free_port()?),
238+
node,
236239
&format!("nc-hcl-{}", nemoclaw_build::hex(&namespace)),
237240
&format!("kind-{name}"),
238241
&pushed,
@@ -298,25 +301,56 @@ pub(super) async fn run_live_kind(
298301
result
299302
}
300303

304+
/// The endpoint of a case whose gateway the SDK reaches through a
305+
/// port-forward.
306+
const FORWARDED_ENDPOINT: &str = "https://127.0.0.1:17701";
307+
/// The endpoint's scheme and host in a case whose NodePort Service publishes
308+
/// the gateway. The case's node port stays.
309+
const PUBLISHED_ENDPOINT: &str = "https://192.0.2.1:";
310+
311+
/// The node's IPv4 address on its Docker network.
312+
fn node_address(container: &str) -> Result<std::net::Ipv4Addr> {
313+
let output = Command::new("docker")
314+
.args([
315+
"inspect",
316+
"--format",
317+
"{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}",
318+
container,
319+
])
320+
.stdin(Stdio::null())
321+
.output()?;
322+
String::from_utf8(output.stdout)?
323+
.split_whitespace()
324+
.find_map(|address| address.parse().ok())
325+
.ok_or_else(|| format!("cannot read the address of {container}").into())
326+
}
327+
301328
/// The kind case bound to one run: its UID, which is also its token audience,
302329
/// gateway endpoint, namespace (which also names the stub model's and the
303330
/// issuer's), kubeconfig context, agent image in the run's registry, and
304-
/// harness.
331+
/// harness. The endpoint is a free loopback port, or the node's address when
332+
/// a NodePort Service publishes the gateway.
305333
fn kind_case(
306334
template: &str,
307335
uid: &str,
308-
endpoint: &str,
336+
node: std::net::Ipv4Addr,
309337
namespace: &str,
310338
context: &str,
311339
image: &str,
312340
harness: &str,
313341
) -> Result<String> {
314342
let zero = format!("nc-registry:5000/agent@sha256:{}", "0".repeat(64));
343+
let endpoint = if template.contains(PUBLISHED_ENDPOINT) {
344+
(PUBLISHED_ENDPOINT, format!("https://{node}:"))
345+
} else {
346+
let port = live::free_port()?;
347+
(FORWARDED_ENDPOINT, format!("https://127.0.0.1:{port}"))
348+
};
315349
Ok(live::bind_case(
316350
template,
317351
&[
318352
("3f6d2b8e-1c4a-4e7f-9b2d-5a8c0e1f7d36", uid),
319-
("https://127.0.0.1:17701", endpoint),
353+
(endpoint.0, &endpoint.1),
320354
("nc-hcl-kind", namespace),
321355
("kind-nemoclaw", context),
322356
(&zero, image),
@@ -438,32 +472,37 @@ mod tests {
438472

439473
#[test]
440474
fn the_kind_cases_bind_their_cluster_namespace_and_image() {
475+
let node = std::net::Ipv4Addr::new(172, 18, 0, 9);
441476
let development =
442477
include_str!("../../../nemoclaw-e2e/cases/kind-kubernetes-development.yaml");
443478
let bound = kind_case(
444479
development,
445480
"u-1",
446-
"https://127.0.0.1:1",
481+
node,
447482
"nc-n",
448483
"kind-c",
449484
"nc-registry-1:5000/agent@sha256:1",
450485
"h",
451486
)
452487
.unwrap();
488+
assert!(bound.contains(" endpoint: https://127.0.0.1:"));
453489
assert!(bound.contains("http://model.nc-n-model.svc.cluster.local:8000/v1"));
454490
assert!(!bound.contains("nc-hcl-kind") && !bound.contains("@sha256:0000"));
491+
assert!(!bound.contains(FORWARDED_ENDPOINT) && !bound.contains("172.18.0.9"));
455492

456493
let template = include_str!("../../../nemoclaw-e2e/cases/kind-kubernetes-oidc.yaml");
457494
let bound = kind_case(
458495
template,
459496
"u-1",
460-
"https://127.0.0.1:1",
497+
node,
461498
"nc-n",
462499
"kind-c",
463500
"nc-registry-1:5000/agent@sha256:1",
464501
"h",
465502
)
466503
.unwrap();
504+
// A published gateway's endpoint names the node, at the case's port.
505+
assert!(bound.contains(" endpoint: https://172.18.0.9:30080\n"));
467506
assert!(bound.contains(" namespace: nc-n\n"));
468507
assert!(bound.contains("http://model.nc-n-model.svc.cluster.local:8000/v1"));
469508
assert!(bound.contains("https://issuer-oidc.nc-n-issuer.svc.cluster.local:8443"));

‎crates/nemoclaw-e2e/cases/kind-kubernetes-oidc.yaml‎

Lines changed: 12 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -9,9 +9,14 @@
99
# it can't read its signing key. openshift-oidc deploys the same agents with
1010
# the openshift profile on OpenShift. The gateway accepts tokens from an
1111
# existing OIDC issuer in another namespace, and the agent image comes from a
12-
# private registry through the pull Secret the operator created. cargo ci
13-
# live-kind gives each run a fresh identity, which is also the token
14-
# audience, port, namespace, kubeconfig context, registry, and agent image.
12+
# private registry through the pull Secret the operator created. The chart's
13+
# NodePort Service publishes the gateway at its endpoint: NemoClaw still
14+
# forwards a port to the gateway pod, and the probe dials the node's address
15+
# as another client would, so the gateway's certificate must name it. cargo ci live-kind gives each run a fresh
16+
# identity, which is also the token audience, namespace, kubeconfig context,
17+
# registry, and agent image, and replaces the endpoint's host with the node's
18+
# address. The node port stays: it is in the range's lower band, which
19+
# Kubernetes leaves for Services that name their port.
1520
apiVersion: nemoclaw.nvidia.com/v1alpha1
1621
kind: NemoClawConfig
1722
metadata:
@@ -22,7 +27,7 @@ spec:
2227
management: managed
2328
runtime:
2429
provider: kubernetes
25-
endpoint: https://127.0.0.1:17701
30+
endpoint: https://192.0.2.1:30080
2631
kubernetes:
2732
kubeconfig:
2833
env: NEMOCLAW_TEST_KUBECONFIG
@@ -35,6 +40,9 @@ spec:
3540
caConfigMap: issuer-oidc-ca
3641
token:
3742
env: NEMOCLAW_TEST_OIDC_TOKEN
43+
expose:
44+
service:
45+
type: NodePort
3846
config:
3947
openshell.gateway.oidc:
4048
issuer: https://issuer-oidc.nc-hcl-kind-issuer.svc.cluster.local:8443

‎crates/nemoclaw-e2e/tests/e2e/hcl/tests/kind-kubernetes-oidc.tftest.hcl‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,13 +3,17 @@
33

44
# Two agents on a managed gateway in kind, in a namespace annotated with
55
# OpenShift's UID range, so OpenShell runs each sandbox as its first UID:
6-
# apply, a reply from each agent's recorded runtime, an unchanged plan, and
7-
# destroy that keeps only the gateway's storage. cargo ci live-kind runs it.
6+
# apply, with the chart's NodePort Service publishing the gateway; a reply
7+
# from each agent's recorded runtime; an unchanged plan; and destroy that
8+
# keeps only the gateway's storage. cargo ci live-kind runs it.
89
# The stub's reply shows each request reached it; no model runs. The agent
910
# image comes from the run's private registry with the operator's pull
1011
# Secret, and the CLI and the probe reach the gateway with a token from an
1112
# existing OIDC issuer, so apply and the replies show that the gateway
12-
# accepts that issuer's tokens.
13+
# accepts that issuer's tokens. The CLI reaches the gateway through the SDK's
14+
# port-forward; the probe dials the node's address that the chart's NodePort
15+
# Service publishes, as another client would, with the gateway's client
16+
# certificate, and verifies the gateway's certificate there.
1317

1418
run "cluster" {
1519
module {

‎crates/nemoclaw-e2e/tests/e2e/hcl_live.rs‎

Lines changed: 14 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -106,10 +106,17 @@ async fn probe() {
106106
let runtime = runtime_id(&state, sandbox.name);
107107
let request: Value = serde_json::from_str(&input("REQUEST")).unwrap();
108108
// A managed Kubernetes gateway is reached through a port-forward that the
109-
// connection holds, as the SDK's providers reach it.
109+
// connection holds, as the SDK's providers reach it. A gateway published
110+
// through kubernetes.expose is dialed at its endpoint instead, as another
111+
// client would dial it, with the same client certificate and token.
112+
let published = document.spec.gateway.as_managed().and_then(|gateway| {
113+
gateway.kubernetes.as_ref()?.expose.as_ref()?;
114+
Some(gateway.endpoint.clone())
115+
});
110116
let (_forward, client) = match kubernetes_operations(&document, &state).await {
111117
Some(operations) => {
112-
let (forward, client) = kubernetes_gateway(&operations, &recorded).await;
118+
let (forward, client) =
119+
kubernetes_gateway(&operations, &recorded, published.as_deref()).await;
113120
(Some(forward), client)
114121
}
115122
None => {
@@ -261,7 +268,7 @@ async fn command() {
261268
let (exit_code, text) = match kubernetes_operations(&document, &state).await {
262269
Some(operations) => {
263270
let binding = sandbox_binding(&state, &sandbox);
264-
let (_forward, client) = kubernetes_gateway(&operations, &binding).await;
271+
let (_forward, client) = kubernetes_gateway(&operations, &binding, None).await;
265272
let (code, output) = client
266273
.exec_bound(&binding, command, Default::default(), 60)
267274
.await
@@ -458,10 +465,12 @@ fn visible(output: &[u8]) -> String {
458465

459466
/// The managed Kubernetes gateway that `binding` records, reached as the
460467
/// SDK's providers reach it: through the port-forward the returned connection
461-
/// holds, with the token and TLS files in its environment.
468+
/// holds, with the token and TLS files in its environment. With `published`,
469+
/// the client dials that endpoint instead of the forward.
462470
async fn kubernetes_gateway(
463471
operations: &nemoclaw_sdk::kubernetes::operations::Operations,
464472
binding: &nemoclaw_sdk::backend::Row,
473+
published: Option<&str>,
465474
) -> (nemoclaw_sdk::kubernetes::Connection, OpenShell) {
466475
use nemoclaw_sdk::{
467476
ObservationError,
@@ -484,7 +493,7 @@ async fn kubernetes_gateway(
484493
let connection = operations.connect(&spec).await.unwrap();
485494
let file = |env: &str| FileRef { env: env.into() };
486495
let gateway = Gateway::External(ExternalGateway {
487-
endpoint: connection.endpoint().to_owned(),
496+
endpoint: published.unwrap_or(connection.endpoint()).to_owned(),
488497
credential: Some(SecretRef {
489498
env: TOKEN_ENV.into(),
490499
}),

‎crates/nemoclaw-sdk/src/config/constraints.rs‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,10 @@ pub(crate) const GATEWAY_ENDPOINT: &str = "http://127.0.0.1:17681";
4242
/// The engine a managed gateway uses when none is configured: Docker's default socket.
4343
pub const GATEWAY_ENGINE: &str = "unix:///var/run/docker.sock";
4444
pub(crate) const KUBERNETES_GATEWAY_ENDPOINT: &str = r"^https://127\.0\.0\.1:(?:[1-9][0-9]{0,3}|[1-5][0-9]{4}|6[0-4][0-9]{3}|65[0-4][0-9]{2}|655[0-2][0-9]|6553[0-5])$";
45+
/// The address clients dial for a managed Kubernetes gateway with
46+
/// `kubernetes.expose`: a lowercase DNS name or IPv4 address and an explicit
47+
/// nonzero port.
48+
pub(crate) const KUBERNETES_EXPOSED_GATEWAY_ENDPOINT: &str = r"^https://[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?(?:\.[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?)*:(?:[1-9][0-9]{0,3}|[1-5][0-9]{4}|6[0-4][0-9]{3}|65[0-4][0-9]{2}|655[0-2][0-9]|6553[0-5])$";
4549
pub(crate) const NETWORK_TIER: &str = "isolated";
4650
/// Sandboxes in a document with a Kubernetes or OpenShift gateway. The cluster
4751
/// schedules their pods; a document is one gateway, namespace and state, so

‎crates/nemoclaw-sdk/src/config/kubernetes.rs‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,44 @@ pub struct ManagedKubernetes {
3737
#[serde(default, skip_serializing_if = "Vec::is_empty")]
3838
#[schemars(length(max = 32))]
3939
pub image_pull_secrets: Vec<String>,
40+
/// How clients outside the cluster reach the gateway. With it, `endpoint` is the address they dial and the gateway certificate names its host. NemoClaw's own commands still forward a free local port to the gateway pod. Its Service type is fixed after the first successful plan.
41+
#[serde(default, skip_serializing_if = "Option::is_none")]
42+
#[schemars(default, with = "KubernetesExpose")]
43+
pub expose: Option<KubernetesExpose>,
44+
}
45+
46+
/// Publication of a managed Kubernetes gateway's endpoint.
47+
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, schemars::JsonSchema)]
48+
#[serde(rename_all = "camelCase", deny_unknown_fields)]
49+
pub struct KubernetesExpose {
50+
/// The gateway chart's own Service, which publishes the endpoint's port and forwards it to the gateway pod. TLS and client certificates pass through to the gateway unchanged.
51+
pub service: ExposureService,
52+
}
53+
54+
/// The Service that publishes a managed Kubernetes gateway.
55+
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, schemars::JsonSchema)]
56+
#[serde(rename_all = "camelCase", deny_unknown_fields)]
57+
pub struct ExposureService {
58+
/// `NodePort` listens on the endpoint's port on every node; the port must be in the cluster's node-port range, 30000 through 32767 by default.
59+
#[serde(rename = "type")]
60+
pub service_type: ExposureServiceType,
61+
}
62+
63+
/// The Kubernetes Service type that publishes a managed gateway.
64+
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize, schemars::JsonSchema)]
65+
#[schemars(inline)]
66+
pub enum ExposureServiceType {
67+
/// A port on every node.
68+
NodePort,
69+
}
70+
71+
impl ExposureServiceType {
72+
/// The Service `spec.type` value.
73+
pub fn as_str(self) -> &'static str {
74+
match self {
75+
Self::NodePort => "NodePort",
76+
}
77+
}
4078
}
4179

4280
/// Authentication provisioned for a managed Kubernetes gateway.

‎crates/nemoclaw-sdk/src/config/schema/validation.rs‎

Lines changed: 17 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -356,7 +356,7 @@ pub(super) fn constrain(root: &mut Value, normalized: bool) {
356356
gateway,
357357
field,
358358
json!({"x-nemoclaw-default-rule": if field == "endpoint" {
359-
"Without kubernetes, omitted or empty selects the local HTTP endpoint. Kubernetes requires an explicit HTTPS loopback endpoint and port."
359+
"Without kubernetes, omitted or empty selects the local HTTP endpoint. Kubernetes requires an explicit HTTPS endpoint and port: 127.0.0.1 without kubernetes.expose, or the address clients dial with it."
360360
} else if field == "networkCIDR" {
361361
"Without kubernetes, omitted or empty selects 172.30.N.0/24, where N is the first byte of SHA-256(metadata.uid). Excluded by kubernetes."
362362
} else {
@@ -365,11 +365,14 @@ pub(super) fn constrain(root: &mut Value, normalized: bool) {
365365
);
366366
}
367367
gateway["if"] = json!({"required": ["kubernetes"]});
368+
// Whether the host must be 127.0.0.1 depends on kubernetes.expose,
369+
// a document rule below: platform resources rebuild this
370+
// definition from their attributes, which carry no exposure.
368371
gateway["then"] = json!({
369372
"required": ["endpoint"],
370-
"properties": {"endpoint": {"pattern": c::KUBERNETES_GATEWAY_ENDPOINT}},
373+
"properties": {"endpoint": {"pattern": c::KUBERNETES_EXPOSED_GATEWAY_ENDPOINT}},
371374
"allOf": [forbid(&["engine", "imagePullPolicy", "networkCIDR"])],
372-
"x-nemoclaw-error": "managed Kubernetes requires an explicit HTTPS 127.0.0.1 endpoint with a nonzero port and excludes local engine settings"
375+
"x-nemoclaw-error": "managed Kubernetes requires an explicit HTTPS endpoint with a lowercase host and a nonzero port and excludes local engine settings"
373376
});
374377
gateway["else"] = local;
375378
// Podman's socket depends on the host user, so it has no default.
@@ -426,6 +429,16 @@ pub(super) fn constrain(root: &mut Value, normalized: bool) {
426429
"else": at("spec/gateway/runtime/provider", json!({"enum":["docker", "podman"]}), false)
427430
}
428431
}));
432+
// Without kubernetes.expose, providers forward a free local port to the
433+
// gateway pod, and the certificate names 127.0.0.1.
434+
root["allOf"].as_array_mut().unwrap().push(json!({
435+
"if": at("spec/gateway", json!({
436+
"required": ["kubernetes"],
437+
"properties": {"kubernetes": {"not": {"required": ["expose"]}}}
438+
}), true),
439+
"then": at("spec/gateway/endpoint", json!({"pattern": c::KUBERNETES_GATEWAY_ENDPOINT}), true),
440+
"x-nemoclaw-error": "managed Kubernetes without kubernetes.expose requires an explicit HTTPS 127.0.0.1 endpoint with a nonzero port"
441+
}));
429442
// Cluster sandboxes need an explicit image, since no local engine supplies
430443
// a default; image metadata stands in for engine inspection only there.
431444
root["allOf"].as_array_mut().unwrap().push(json!({
@@ -462,7 +475,7 @@ pub(super) fn constrain(root: &mut Value, normalized: bool) {
462475
root["x-nemoclaw-parser-checks"] = json!([
463476
"Document::parse rejects YAML aliases, anchors, merge keys, all explicit tags (including core tags such as !!binary), duplicate keys, multiple documents, and input larger than 1 MiB. It applies the compiled input schema before defaulting; Document::validate applies the normalized schema and semantic checks, including for directly constructed Rust values.",
464477
"The parser checks the shape of every endpoint URL and the gateway endpoint's transport and address policy; OpenShell checks inference destinations when sandboxes connect. It also checks managed gateway port bounds, canonical private IPv4 /24 networks, local engine socket syntax, and publication address/port/network agreement.",
465-
"Managed Kubernetes requires a namespace and an authentication profile, and a kubeconfig environment reference and context together; omitting both selects the service account of the pod that runs NemoClaw. Agent Sandbox and a default StorageClass are platform prerequisites that are not checked. Its HTTPS endpoint is exactly 127.0.0.1 with an explicit port from 1 through 65535 and no path. Local engine fields and managed inference services are excluded; gateway.runtime.provider is kubernetes or openshift. OpenShift uses the upstream Kubernetes driver and requires platform-owned OpenShift security prerequisites. Namespace identity, ownership, and credential files are checked during operations.",
478+
"Managed Kubernetes requires a namespace and an authentication profile, and a kubeconfig environment reference and context together; omitting both selects the service account of the pod that runs NemoClaw. Agent Sandbox and a default StorageClass are platform prerequisites that are not checked. Without kubernetes.expose, its HTTPS endpoint is exactly 127.0.0.1 with an explicit port from 1 through 65535 and no path; with it, the endpoint names a lowercase DNS name or IPv4 address and an explicit port, which the chart's NodePort Service publishes; Kubernetes checks that the port is in the cluster's node-port range. Local engine fields and managed inference services are excluded; gateway.runtime.provider is kubernetes or openshift. OpenShift uses the upstream Kubernetes driver and requires platform-owned OpenShift security prerequisites. Namespace identity, ownership, and credential files are checked during operations.",
466479
"Explicit sandbox policies are also checked by the pinned OpenShell policy parser and validator, including protocol-specific rule semantics, process identities, filesystem paths, and destination address restrictions.",
467480
"Explicit filesystem grants must permit reads of the paths the selected image's runtime requires, such as Fabric's /opt/fabric; parent and read-write grants count. Planning checks them against the image's metadata; it does not resolve symlinks or establish runtime permissions.",
468481
"The schema requires an explicit default for multiple model choices. Rust checks unique route names, that the default names a route, and that native model and tool fields have valid structural shapes. Fabric validates adapter-specific combinations.",

0 commit comments

Comments
 (0)