Skip to content

Commit 15f537f

Browse files
committed
fix(kubernetes): separate legacy and runtime-initiated control services
Keep os-boundary selecting the workload and add os-supervisor for runtime-initiated connections. Bind the mode marker to its runtime generation so a restart by an older gateway does not inherit stale mode metadata. Preserve the legacy Service across creation and restart, validate the active endpoint, and drop the Service patch permission. Signed-off-by: Drew Newberry <[email protected]>
1 parent b02aa61 commit 15f537f

10 files changed

Lines changed: 384 additions & 114 deletions

File tree

‎.agents/skills/helm-dev-environment/SKILL.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,10 @@ and deploys the OpenShell Helm chart. The Kubernetes driver creates a
8282
capability-free workload Pod and a directly managed capability-free supervisor
8383
Pod. One namespace-wide NetworkPolicy denies direct egress from every OpenShell
8484
workload Pod. New sandbox generations use `runtime-initiated` control: the sandbox
85-
runtime dials its supervisor Pod, and the driver adds a per-sandbox
85+
runtime dials its supervisor through `os-supervisor-<id>`. The legacy
86+
`os-boundary-<id>` Service keeps selecting the workload for old gateways during
87+
rolling upgrades. New gateways create both Services and bind the control-mode
88+
annotation to its runtime generation. The driver adds a per-sandbox
8689
`os-control-<id>` NetworkPolicy for that path only. Existing running generations
8790
keep their connection until replaced; stop/start and gateway recovery that
8891
recreates both Pods upgrade legacy generations automatically. Use compatible

‎crates/openshell-driver-kubernetes/README.md‎

Lines changed: 17 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -106,21 +106,24 @@ enforcement, so operators must verify CNI support before running sandboxes.
106106
### Control connection direction
107107

108108
Every new runtime generation uses `runtime-initiated` control. The Sandbox
109-
annotation `openshell.ai/sandbox-runtime-control-connection` records the current
110-
generation's mode. A missing annotation identifies a legacy,
109+
annotation `openshell.ai/sandbox-runtime-control-connection` records the mode,
110+
bound to `openshell.ai/sandbox-runtime-control-generation`. A missing marker or
111+
a different current runtime generation identifies a legacy,
111112
`supervisor-initiated` generation, which keeps working until replaced. Stop/start
112-
upgrades it automatically: with both old Pods removed, the driver changes the
113-
Service selector in place, preserves its ClusterIP, and records the new mode
113+
upgrades it automatically: with both old Pods removed, the driver adds the
114+
supervisor Service and records the new mode and generation
114115
before releasing the replacement workload. Gateway recovery that recreates both
115116
Pods also upgrades the generation. Reconciliation never migrates a running
116117
generation. An unknown annotation value fails closed; there is no rollout flag.
117118

118119
Only the dial direction differs. In both modes the sandbox runtime presents the
119120
generation-pinned TLS server identity and serves gRPC, and the supervisor
120121
verifies that identity, acts as the gRPC client, and sends the Sandbox Protocol
121-
JWT. The per-sandbox Service `os-boundary-<id>` keeps one stable ClusterIP for
122-
the sandbox's lifetime; it selects the paired workload Pod for
123-
`supervisor-initiated` and the paired supervisor Pod for `runtime-initiated`.
122+
JWT. The per-sandbox Service `os-boundary-<id>` always selects the paired
123+
workload Pod. A separate `os-supervisor-<id>` Service always selects the paired
124+
supervisor Pod. New gateways create both Services, retaining the legacy
125+
Service's UID, ClusterIP, and selector across restarts so an older gateway can
126+
still replace the generation during a rolling upgrade.
124127

125128
For `runtime-initiated`:
126129

@@ -140,20 +143,20 @@ For `runtime-initiated`:
140143
namespace-wide policies are unchanged.
141144
- The policy's UID and resourceVersion are bound into the boundary resource
142145
claims and outer-fence evidence. Reconciliation suspends the workload if the
143-
policy disappears, diverges, or changes generation, or if the boundary
146+
policy disappears, diverges, or changes generation, or if the active control
144147
Service no longer selects the role the recorded mode requires.
145-
- The Service and control policy are owned by the Sandbox resource and are
148+
- Both Services and the control policy are owned by the Sandbox resource and are
146149
deleted only with that sandbox.
147150

148151
Deploy a gateway, supervisor image, and sandbox runtime image that all support
149152
`runtime-initiated` before using it. Older binaries reject the new descriptor
150153
shapes and fail closed.
151154

152-
Rollback is unsupported for `runtime-initiated` sandboxes. Already running Pods
153-
keep their established connection, but older binaries cannot start or restart
154-
these sandboxes: an older driver ignores the recorded mode while the boundary
155-
Service selects the supervisor, and older supervisors and runtimes reject the
156-
new descriptor variants. Delete them before rolling back, or roll forward.
155+
During a mixed-version rollout, an old gateway can restart a sandbox by
156+
creating its own compatible supervisor/runtime pair through the legacy
157+
Service. It preserves unknown annotations, so the mode marker is accepted only
158+
for its recorded generation. Do not replace an individual generation's
159+
supervisor or runtime binary with one that cannot read its descriptor.
157160

158161
Each sandbox generation uses two immutable bootstrap Secrets. A trusted init
159162
container stages the sandbox bootstrap into memory, and the sandbox removes it

0 commit comments

Comments
 (0)