|
| 1 | +# Brainstorm: K8s Watch Stream Crash Fix |
| 2 | + |
| 3 | +**Date:** 2026-07-10 |
| 4 | +**Status:** active |
| 5 | +**Issue:** [#2211](https://github.com/NVIDIA/OpenShell/issues/2211) |
| 6 | + |
| 7 | +## Problem Framing |
| 8 | + |
| 9 | +The Kubernetes driver's `watch_sandboxes()` crashes with a file descriptor leak when non-gateway `Sandbox` resources exist in the namespace. This happens when the Agent Sandbox operator's `SandboxWarmPool` controller creates `Sandbox` objects that the gateway did not create and does not expect. |
| 10 | + |
| 11 | +The gateway watch stream has no label selector, so it receives events for all `Sandbox` objects. Warm pool sandboxes lack the `openshell.ai/sandbox-id` label and the `sandbox-` name prefix, causing `sandbox_id_from_object()` to return an error. This error propagates as a stream failure, triggering a 2-second reconnect loop that leaks HTTP/2 connections until FD exhaustion crashes the process (13+ restarts observed). |
| 12 | + |
| 13 | +The same issue affects `list_sandboxes()`, which also uses unfiltered `ListParams::default()`. |
| 14 | + |
| 15 | +## Approaches Considered |
| 16 | + |
| 17 | +### A: Minimal targeted fix (chosen) |
| 18 | +Add a label selector (`openshell.ai/managed-by=openshell`) to both `watch_sandboxes()` and `list_sandboxes()` to filter at the API server level. Additionally, change the watch loop to skip (with `debug!` log) objects that fail `sandbox_from_object()` / `sandbox_id_from_object()` instead of sending errors to the channel. |
| 19 | + |
| 20 | +- Pros: Minimal blast radius (one file, two functions). Reduces API server traffic. Defense-in-depth against future unknown object types. |
| 21 | +- Cons: None significant. |
| 22 | + |
| 23 | +### B: Extract shared helper |
| 24 | +Same as A, but extract the label selector construction into a helper function to avoid duplication between `watch_sandboxes()` and `list_sandboxes()`. |
| 25 | + |
| 26 | +- Pros: DRY for selector construction. |
| 27 | +- Cons: Over-engineering for a one-liner format string used in two places. |
| 28 | + |
| 29 | +### C: Change sandbox_from_object signature |
| 30 | +Make `sandbox_from_object` return `Option<Sandbox>` instead of `Result<Sandbox, String>`, changing all callers to skip `None`. |
| 31 | + |
| 32 | +- Pros: Treats unknown objects as normal at the type level. |
| 33 | +- Cons: Larger change surface. Doesn't reduce API traffic. Loses error information for genuinely malformed objects. |
| 34 | + |
| 35 | +## Decision |
| 36 | + |
| 37 | +Approach A: minimal targeted fix with both label selector and defensive skip. |
| 38 | + |
| 39 | +## Key Requirements |
| 40 | + |
| 41 | +- Add `openshell.ai/managed-by=openshell` label selector to the `watcher::Config` in `watch_sandboxes()` (`driver.rs:717`) |
| 42 | +- Add the same label selector to `ListParams` in `list_sandboxes()` (`driver.rs:488`) |
| 43 | +- In the watch loop, all three event branches (Applied at line 741, Deleted at line 761, Restarted at line 782) must `debug!` log and continue instead of sending `Err` to the channel |
| 44 | +- PR targets `origin` (fork) first, then submitted upstream to `NVIDIA/OpenShell` |
| 45 | +- Unit tests for the skip behavior |
| 46 | + |
| 47 | +## Open Questions |
| 48 | + |
| 49 | +- Should the defensive skip use `debug!` or `warn!` level? (`debug!` seems right since the label filter should prevent this from happening in normal operation; it's only a fallback) |
0 commit comments