Gateway restart regenerates TLS certificates, breaking existing sandbox connections

Problem

When the OpenShell gateway restarts (e.g., after openshell gateway start or system reboot), it regenerates its mTLS CA keypair. Existing sandbox pods still hold the old gateway CA certificate, so every subsequent connection attempt fails with:

[WARN] SSH connection: handshake verification failed

The only workaround is to destroy and recreate all sandboxes via nemoclaw onboard, which is disruptive and loses sandbox state.

Steps to Reproduce

Create a sandbox: nemoclaw onboard
Restart the gateway: openshell gateway start (after a stop or system reboot)
Try to connect: nemoclaw <name> connect
Connection fails with Command failed (exit 255)

Expected Behavior

Gateway restart should not invalidate existing sandbox connections. Options:

Persist the gateway CA keypair to a PersistentVolume so it survives restarts
Implement a certificate rotation mechanism where sandboxes can fetch the new gateway CA cert automatically

Environment

NemoClaw CLI / openshell 0.0.12
WSL2 Ubuntu on Windows 11
k3s (via Docker Desktop)

I have the same issue every time my computer restarts. The onboarding process just wipes the whole configuration.

I'm having the same problem. Every time my computer or gateway restarts, I have to onboard again, and I lose all my settings and configuration.

Workaround: always regenerate SSH config fresh — don't cache it

The TLS cert rotation breaks any cached SSH configs. The fix is to never persist openshell sandbox ssh-config output — regenerate it fresh on every connection attempt.

In any script that SSHes into the sandbox:

# Generate fresh SSH config each time — never cache this
SSH_CONF=$(mktemp /tmp/nemoclaw-XXXXXX.conf)
openshell sandbox ssh-config "$SANDBOX" > "$SSH_CONF"
ssh -T -F "$SSH_CONF" "openshell-${SANDBOX}" "your-command"
rm -f "$SSH_CONF"

This handles cert rotation transparently — ssh-config always returns the current valid config reflecting the new certificates. We call this at the top of every startup script and every SSH exec, so cert rotation after a gateway restart is never a problem.

Also helps: run openshell sandbox ssh-config as a health check before attempting anything. If it returns empty or errors, the gateway isn't ready yet — retry with backoff rather than failing.

This should be resolved by OpenShell #753, which persists sandbox state across gateway stop/start cycles. We'll close this issue once that lands and we validate it fixes the cert mismatch.

fix(onboard): upgrade OpenShell to 0.0.24 for sandbox survival across…

Additional root cause found: NSSH1 handshake secret mismatch in Sandbox CR

Beyond the TLS cert rotation, there's a separate failure mode: the gateway regenerates its OPENSHELL_SSH_HANDSHAKE_SECRET on restart, but the Sandbox custom resource (CR) retains the stale secret. This causes kex_exchange_identification: Connection closed by remote host even after the pod is bounced, because the CR templates the pod spec with the old value.

Workaround (tested on v0.0.14 gateway, helm release v4):

Read the gateway's current secret: kubectl exec openshell-0 -n openshell -- printenv OPENSHELL_SSH_HANDSHAKE_SECRET
Patch the Sandbox CR with the new value: kubectl get sandbox my-assistant -n openshell -o json → replace the secret → kubectl apply -f -
Delete the sandbox pod to force recreation with the patched CR

This could be automated in nemoclaw onboard or a startup script. We've added it to our 1-click startup to auto-detect and fix the mismatch.

Closing this as fixed for current supported NemoClaw/OpenShell versions.

Fixing PR chain:

feat(bootstrap): resume gateway from existing state and persist SSH handshake secret OpenShell#488 persisted gateway resume state and the SSH handshake secret across gateway stop/start cycles.
fix(bootstrap,server): persist sandbox state across gateway stop/start cycles OpenShell#739 persisted sandbox state across gateway stop/start cycles.
feat: sandbox survival across gateway restarts #1466 integrated the sandbox-survival path, including restart recovery and an E2E that verifies SSH works after gateway restart with no handshake failure.
fix(onboard): upgrade OpenShell to 0.0.24 for sandbox survival across restarts #1587 made onboarding upgrade too-old OpenShell installs so users are not left on versions without the persistence fix.
fix: clear stale SSH host keys after gateway restart #1087 cleared stale openshell-* SSH host keys after gateway restart/recreate so regenerated identity material does not get rejected by cached host fingerprints.

The remaining legacy caveat is documented: sandboxes created with OpenShell versions older than 0.0.24 may need OpenShell upgrade plus sandbox recreation. Current latest installs a newer validated OpenShell, and the sandbox-survival E2E explicitly covers this issue (#888/#1086).

test(e2e): migrate test-sandbox-survival.sh to vitest (#5332)

Navigation Menu

Repository navigation

Gateway restart regenerates TLS certificates, breaking existing sandbox connections #888

Problem

Steps to Reproduce

Expected Behavior

Environment

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Gateway restart regenerates TLS certificates, breaking existing sandbox connections #888

Description

Problem

Steps to Reproduce

Expected Behavior

Environment

Activity

rafaelluna commented on Mar 27, 2026

salmanadil84 commented on Mar 29, 2026

dknos commented on Mar 30, 2026

ericksoa commented on Apr 5, 2026

dknos commented on Apr 9, 2026

ericksoa commented on May 1, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions