Skip to content

Commit 4aad039

Browse files
committed
docs: add brainstorm for K8s watch stream crash fix (NVIDIA#2211)
Assisted-By: 🤖 Claude Code Signed-off-by: Roland Huß <[email protected]>
1 parent e845b4c commit 4aad039

1 file changed

Lines changed: 49 additions & 0 deletions

File tree

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
# Brainstorm: K8s Watch Stream Crash Fix
2+
3+
**Date:** 2026-07-10
4+
**Status:** active
5+
**Issue:** [#2211](https://github.com/NVIDIA/OpenShell/issues/2211)
6+
7+
## Problem Framing
8+
9+
The Kubernetes driver's `watch_sandboxes()` crashes with a file descriptor leak when non-gateway `Sandbox` resources exist in the namespace. This happens when the Agent Sandbox operator's `SandboxWarmPool` controller creates `Sandbox` objects that the gateway did not create and does not expect.
10+
11+
The gateway watch stream has no label selector, so it receives events for all `Sandbox` objects. Warm pool sandboxes lack the `openshell.ai/sandbox-id` label and the `sandbox-` name prefix, causing `sandbox_id_from_object()` to return an error. This error propagates as a stream failure, triggering a 2-second reconnect loop that leaks HTTP/2 connections until FD exhaustion crashes the process (13+ restarts observed).
12+
13+
The same issue affects `list_sandboxes()`, which also uses unfiltered `ListParams::default()`.
14+
15+
## Approaches Considered
16+
17+
### A: Minimal targeted fix (chosen)
18+
Add a label selector (`openshell.ai/managed-by=openshell`) to both `watch_sandboxes()` and `list_sandboxes()` to filter at the API server level. Additionally, change the watch loop to skip (with `debug!` log) objects that fail `sandbox_from_object()` / `sandbox_id_from_object()` instead of sending errors to the channel.
19+
20+
- Pros: Minimal blast radius (one file, two functions). Reduces API server traffic. Defense-in-depth against future unknown object types.
21+
- Cons: None significant.
22+
23+
### B: Extract shared helper
24+
Same as A, but extract the label selector construction into a helper function to avoid duplication between `watch_sandboxes()` and `list_sandboxes()`.
25+
26+
- Pros: DRY for selector construction.
27+
- Cons: Over-engineering for a one-liner format string used in two places.
28+
29+
### C: Change sandbox_from_object signature
30+
Make `sandbox_from_object` return `Option<Sandbox>` instead of `Result<Sandbox, String>`, changing all callers to skip `None`.
31+
32+
- Pros: Treats unknown objects as normal at the type level.
33+
- Cons: Larger change surface. Doesn't reduce API traffic. Loses error information for genuinely malformed objects.
34+
35+
## Decision
36+
37+
Approach A: minimal targeted fix with both label selector and defensive skip.
38+
39+
## Key Requirements
40+
41+
- Add `openshell.ai/managed-by=openshell` label selector to the `watcher::Config` in `watch_sandboxes()` (`driver.rs:717`)
42+
- Add the same label selector to `ListParams` in `list_sandboxes()` (`driver.rs:488`)
43+
- In the watch loop, all three event branches (Applied at line 741, Deleted at line 761, Restarted at line 782) must `debug!` log and continue instead of sending `Err` to the channel
44+
- PR targets `origin` (fork) first, then submitted upstream to `NVIDIA/OpenShell`
45+
- Unit tests for the skip behavior
46+
47+
## Open Questions
48+
49+
- Should the defensive skip use `debug!` or `warn!` level? (`debug!` seems right since the label filter should prevent this from happening in normal operation; it's only a fallback)

0 commit comments

Comments
 (0)