Skip to content

Commit a605921

Browse files
committed
docs: add warm pool RFCs and brainstorm from feasibility study
Bring over the warm pool feasibility RFC, gRPC-push RFC, and brainstorm documents from the 6111-warm-pool-feasibility branch so the PoC branch is self-contained for reviewers. Signed-off-by: Roland Huß <[email protected]>
1 parent f20f49f commit a605921

7 files changed

Lines changed: 952 additions & 28 deletions

File tree

‎brainstorm/00-overview.md‎

Lines changed: 18 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Brainstorm Overview
22

3-
Last updated: 2026-07-13
3+
Last updated: 2026-07-11
44

55
## Sessions
66

@@ -10,10 +10,9 @@ Last updated: 2026-07-13
1010
| 02 | 2026-07-09 | cluster-setup | active | - | - |
1111
| 03 | 2026-07-09 | warm-pool-measurements | active | - | - |
1212
| 04 | 2026-07-09 | results-and-recommendations | active | - | - |
13-
| 05 | 2026-07-10 | k8s-watch-crash-fix | active | - | [#2211](https://github.com/NVIDIA/OpenShell/issues/2211) |
14-
| 06 | 2026-07-11 | warm-pool-grpc-poc | active | 002 | - |
13+
| 05 | 2026-07-10 | warm-pool-grpc-push | active | - | - |
14+
| 06 | 2026-07-11 | warm-pool-grpc-poc | active | - | - |
1515
| 07 | 2026-07-11 | warm-pool-sandbox-profile | active | - | - |
16-
| 08 | 2026-07-13 | warm-pool-e2e-ssh | active | - | - |
1716

1817
## Structure
1918

@@ -23,22 +22,28 @@ Last updated: 2026-07-13
2322
- **03** depends on: 02 (cluster must be running)
2423
- **04** depends on: 03 (measurements must be complete)
2524

26-
05 is a standalone bug fix discovered during the feasibility study.
27-
06 is the Milestone 1 gRPC PoC (gateway-side flow, spec 002).
28-
07 explores the SandboxProfile entity model for warm pool configuration.
29-
08 is the Milestone 2 plan for E2E SSH demo with full bootstrap extraction.
25+
05 is the RFC brainstorm for always-on supervisor with gRPC-push identity binding.
26+
27+
06-07 are prototype milestones incorporating NVIDIA feedback (2026-07-10):
28+
29+
- **06** (Milestone 1): ActivateSandbox gRPC PoC with unidentified supervisor. Proves the claim-time flow end-to-end with manual pool setup.
30+
- **07** (Milestone 2): SandboxProfile entity + workspace-scoped pool lifecycle. Builds on 06, makes pools mergeable upstream. Depends on Derek's workspace PR.
3031

3132
## Open Threads
3233

3334
- Does the Red Hat Agent Sandbox operator tech preview include extension CRDs? (from #01, #02)
34-
- Does env var injection at claim time trigger cold start? (from #01, #03) -- confirmed yes
35+
- Does env var injection at claim time trigger cold start? (from #01, #03)
3536
- Is KEP-753 (native sidecars) available on the target OpenShift version? (from #01, #02)
3637
- How does pool exhaustion behave (cold fallback vs. Pending)? (from #03)
3738
- Should findings be posted to upstream agent-sandbox repo? (from #04)
38-
- Should the defensive skip use `debug!` or `warn!` level? (from #05)
39-
- How much of run_sandbox() parameter setup can be shared vs reconstructed? (from #08)
40-
- Does the SSH relay path work with claimed pod IP or need Sandbox CRD name? (from #08)
41-
- What's the actual latency impact of proxy/nftables setup at activation time? (from #08)
39+
- What is the exact proto definition for ActivateSandbox? (from #05, #06)
40+
- How does the supervisor discover the gateway endpoint at activation time? (from #06)
41+
- Should the supervisor support an activation timeout to prevent resource waste? (from #06)
42+
- Coordinate with Craig's work on issue #1955 (legacy RPC cleanup) (from #06)
43+
- What is the relationship between SandboxProfile and SandboxTemplate? (from #07)
44+
- Does Derek's workspace PR define a workspace proto that references SandboxProfiles? (from #07)
45+
- How to handle pool config for workspaces spanning multiple namespaces? (from #07)
46+
- Pool utilization metrics: SandboxProfile status field vs. metric endpoint? (from #07)
4247

4348
## Parked Ideas
4449

‎brainstorm/01-warm-pool-feasibility.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ Start with raw Agent Sandbox measurements (no OpenShell code changes), then laye
4242

4343
1. **Fresh ROSA HCP cluster** provisioned via the ROSA plugin for consistent, isolated measurements
4444
2. **Red Hat build of Agent Sandbox operator** (tech preview) installed from OperatorHub for the extension CRDs
45-
3. **OpenShell deployed via Krzysztof's chart** (github.com/2000krysztof/Openshell-Openshift-Deploy) for fast setup
45+
3. **OpenShell deployed via the OpenShift deploy chart** (github.com/2000krysztof/Openshell-Openshift-Deploy) for fast setup
4646
4. **Baseline cold-start measurements** (vanilla sandbox creation, no pooling) as the control
4747
5. **Warm pool measurements** with varying configurations (readiness probe intervals, Pod Readiness Gates, sidecar readiness patterns)
4848
6. **Health check optimization experiments** including Knative-style readiness wrapping and KEP-580 Pod Readiness Gates

‎brainstorm/02-cluster-setup.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -17,14 +17,14 @@ Three components must work together:
1717

1818
### A: ROSA HCP with Upstream Agent Sandbox Manifests
1919

20-
Provision ROSA, install agent-sandbox from upstream manifests (both `manifest.yaml` and `extensions.yaml`), deploy OpenShell with Krzysztof's chart.
20+
Provision ROSA, install agent-sandbox from upstream manifests (both `manifest.yaml` and `extensions.yaml`), deploy OpenShell with the OpenShell OpenShift deploy chart.
2121

2222
- Pros: Uses upstream directly, most current version, full control over version
2323
- Cons: No operator lifecycle management, manual CRD updates, no OperatorHub integration
2424

2525
### B: ROSA HCP with Red Hat Agent Sandbox Operator (Tech Preview)
2626

27-
Provision ROSA, install the Red Hat build of Agent Sandbox from OperatorHub, deploy OpenShell with Krzysztof's chart.
27+
Provision ROSA, install the Red Hat build of Agent Sandbox from OperatorHub, deploy OpenShell with the OpenShell OpenShift deploy chart.
2828

2929
- Pros: Operator manages CRD lifecycle, OperatorHub integration, downstream supported path
3030
- Cons: Tech preview may not include extension CRDs yet, version may lag upstream
@@ -53,7 +53,7 @@ Install the Red Hat operator for the core Sandbox CRD, then layer upstream exten
5353
- Verify all four CRDs are served: `kubectl api-resources | grep agents`
5454

5555
3. **OpenShell deployment**
56-
- Clone github.com/2000krysztof/Openshell-Openshift-Deploy
56+
- Clone github.com/2000krysztof/Openshell-Openshift-Deploy at a pinned commit/tag for reproducibility
5757
- Run `deploy.sh` with default configuration
5858
- Verify: `openshell status` shows Connected
5959
- Verify: `openshell sandbox create --from base` succeeds (cold-start baseline works)
@@ -74,5 +74,5 @@ Install the Red Hat operator for the core Sandbox CRD, then layer upstream exten
7474

7575
- What OpenShift version does ROSA HCP currently offer that includes K8s 1.33+? Need to check `rosa list versions`.
7676
- Does the Red Hat Agent Sandbox operator tech preview install from the default OperatorHub catalog, or does it require a custom CatalogSource?
77-
- Does Krzysztof's deploy script handle OpenShift 4.20+ or does it need updates for newer SCC/security changes?
77+
- Does the deploy script handle OpenShift 4.20+ or does it need updates for newer SCC/security changes?
7878
- How much cluster capacity do we need? The warm pool experiments will create 5-20 pre-provisioned pods simultaneously.

‎brainstorm/04-results-and-recommendations.md‎

Lines changed: 9 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -8,10 +8,10 @@
88

99
After the measurements are complete, we need to synthesize findings into a document that serves two audiences:
1010

11-
1. **The OpenShell core team** (Derek, Murnal, Seth): What should OpenShell change to support warm pooling? Which approach from Issue #2157 is backed by data? What are the architectural constraints?
12-
2. **Our Red Hat team**: Is the upstream Agent Sandbox warm pooling viable for enterprise OpenShift deployments? What gaps exist in the Red Hat tech preview? What do we recommend for the beta timeline?
11+
1. **The OpenShell core team**: What should OpenShell change to support warm pooling? Which approach from Issue #2157 is backed by data? What are the architectural constraints?
12+
2. **The Red Hat integration team**: Is the upstream Agent Sandbox warm pooling viable for enterprise OpenShift deployments? What gaps exist in the Red Hat tech preview?
1313

14-
The document should be concrete enough to inform Issue #2157's design decisions and the Peter Steinberger demo conversation (July 21st).
14+
The document should be concrete enough to inform Issue #2157's design decisions.
1515

1616
## Approaches Considered
1717

@@ -38,7 +38,7 @@ Write the full report as a document, then distill key findings into a GitHub com
3838

3939
## Decision
4040

41-
**Approach C: Full report plus GitHub comment.** The report lives in our Obsidian vault as a reference. A distilled comment on #2157 shares our findings with the upstream community and positions our work as a concrete contribution to the warm pooling discussion.
41+
**Approach C: Full report plus GitHub comment.** The report is published as a standalone RFC in `rfc/` (see clarification in spec.md). A distilled comment on #2157 can be posted as a follow-up step to share findings with the upstream community.
4242

4343
## Report Structure
4444

@@ -96,19 +96,18 @@ Map findings to OpenShell's architecture:
9696

9797
### 6. Next Steps
9898
- Concrete list of upstream contributions (issues, PRs, RFCs)
99-
- Internal work items for the 60-day beta sprint
100-
- Demo plan for the Peter Steinberger meeting
99+
- Internal work items for the next sprint
100+
- Follow-up actions for stakeholder discussions
101101

102102
## Key Requirements
103103

104-
1. **Report saved to Obsidian vault** at `~/Obsidian/ro14nd/09-Meeting-Notes/` with date prefix
105-
2. **GitHub comment on #2157** with distilled findings (use prose:check before posting)
106-
3. **No Google Drive links** in the GitHub comment (public repo rule from CLAUDE.md)
104+
1. **Report saved to the configured notes vault** with a date prefix
105+
2. **RFC in `rfc/`** as the canonical, version-controlled results document
106+
3. **No Google Drive links** in any public-facing artifacts
107107
4. **Data tables with raw numbers**, not just qualitative assessments
108108
5. **Clear recommendation** for the OpenShell core team, not just "it depends"
109109

110110
## Open Questions
111111

112112
- Should we also post findings to the upstream Agent Sandbox repo (e.g., as a discussion or issue comment on Issue #390)?
113113
- Should the report include a proposed RFC outline for OpenShell warm pooling, or is that a separate step?
114-
- How much of this should feed into Derek's demo prep for Peter Steinberger?
Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
# Brainstorm: Warm Pool with Always-On Supervisor (gRPC-Push Identity)
2+
3+
**Date:** 2026-07-10
4+
**Status:** active
5+
6+
## Problem Framing
7+
8+
The existing warm pool RFC (`rfc/NNNN-warm-pool-feasibility/`) proposes annotation-based identity binding: the supervisor starts at claim time, detects its identity via the Kubernetes downward API, and bootstraps normally. This approach delivers ~3-4s claim latency (2.3s claim + 1.5s supervisor startup).
9+
10+
For workloads that need sub-1s sandbox provisioning, the supervisor startup at claim time is the bottleneck. The 1.5s breaks down as ~80ms for 8 gRPC calls and ~1.4s for process startup plus OPA policy compilation.
11+
12+
An alternative architecture keeps the supervisor always running in the warm pod. Global OPA policies (applicable to every sandbox) are pre-compiled at pool time. At claim time, the gateway pushes the sandbox identity directly to the supervisor via gRPC, the supervisor compiles only sandbox-specific policies, and the session activates. This eliminates process startup entirely and reduces OPA compilation to the per-sandbox delta.
13+
14+
This brainstorm scopes the alternative RFC, its relationship to the annotation RFC, and how gateway-managed pool lifecycle works across multiple sandbox images.
15+
16+
## Approaches Considered
17+
18+
### A: Two standalone RFCs with cross-references
19+
20+
Create a new RFC (`NNNN-warm-pool-grpc-push`) that stands independently alongside the existing annotation RFC. Both get an "Alternative Approaches" section with a comparison table. No ordering or primary/secondary framing.
21+
22+
- Pros: Each RFC is self-contained, reviewable independently. Maintainers can accept one, both, or neither.
23+
- Cons: Some duplication in shared context (motivation, measurements).
24+
25+
### B: Single RFC with two design variants
26+
27+
Merge both approaches into one RFC with variant sections.
28+
29+
- Pros: No duplication, single document.
30+
- Cons: Very long. Harder to discuss one variant without the other.
31+
32+
### C: Split RFC by concern (3 documents)
33+
34+
Shared pool infrastructure RFC plus two identity binding RFCs.
35+
36+
- Pros: Cleanest separation.
37+
- Cons: Three documents is heavy to manage.
38+
39+
## Decision
40+
41+
**Approach A: Two standalone RFCs with cross-references.** The shared context is short enough to duplicate. The pool management section (gateway-managed pools, multi-image config) lives in the new gRPC-push RFC since it's the one proposing gateway-managed pools. The annotation RFC can reference it.
42+
43+
## Key Requirements
44+
45+
### gRPC-Push RFC content
46+
47+
1. **Always-on supervisor in warm pods.** Supervisor starts at pool provisioning time. Pre-compiles global OPA policies. Exposes a new gRPC endpoint (`ActivateSandbox` or similar) for identity push.
48+
49+
2. **Two-tier OPA compilation.** Global policies (applicable to all sandboxes) compile at pool time. Sandbox-specific policies (per-sandbox rules, provider-specific constraints) compile at claim time after identity push. The split must be explicit in the RFC.
50+
51+
3. **Identity binding via gRPC push.** After the operator binds the claim, the gateway reads the pod IP from `.status.sandbox.podIPs`, connects directly to the supervisor's gRPC endpoint, and pushes sandbox identity (ID, name, policy config). No downward API, no annotation propagation delay.
52+
53+
4. **Gateway-managed pool lifecycle.** The gateway owns SandboxTemplate + SandboxWarmPool resource creation, scaling, and cleanup. Gateway config defines which images get pools and their sizes. Gateway reconciles pool state against desired config.
54+
55+
5. **Pool-per-image for known images.** Gateway config lists images with pool sizes. Example:
56+
```yaml
57+
warmPools:
58+
- image: ghcr.io/nvidia/openshell-community/sandboxes/base:latest
59+
replicas: 5
60+
- image: ghcr.io/nvidia/openshell-community/sandboxes/ollama:latest
61+
replicas: 2
62+
```
63+
64+
6. **Unknown images always cold-start.** Custom Dockerfiles, arbitrary image refs, first-time images go through the existing cold-start path. No auto-promotion in the initial design.
65+
66+
7. **Dynamic pool promotion as documented extension.** The RFC includes a "Future Extensions" section describing auto-promotion: gateway creates a pool for an image after seeing it N times. Not part of the initial design.
67+
68+
### Cross-referencing and comparison
69+
70+
8. **Both RFCs get an "Alternative Approaches" section.** Each links to the other with a comparison table covering:
71+
- Claim-to-ready latency
72+
- Supervisor complexity (idle-start vs always-on)
73+
- Gateway complexity (claim-only vs pool reconciler + gRPC client)
74+
- OPA compilation (full at claim vs pre-compiled global + delta)
75+
- Security surface (downward API vs new gRPC endpoint on supervisor)
76+
- Alignment with upstream (craig-kindo's validated approach vs novel)
77+
- Resource cost of idle pools (sleeping process vs running supervisor)
78+
79+
9. **Equal framing.** Neither RFC is "primary." Both are valid alternatives for different latency targets and complexity appetites.
80+
81+
### Upstream references (shared)
82+
83+
- [OpenShell#2157](https://github.com/NVIDIA/OpenShell/issues/2157): Feature issue for warm-pool provisioning
84+
- [OpenShell#1447](https://github.com/NVIDIA/OpenShell/issues/1447): craig-kindo's annotation-based implementation (validates the other RFC)
85+
- [agent-sandbox#1118](https://github.com/kubernetes-sigs/agent-sandbox/pull/1118): Operator adoption finalization improvement
86+
- [agent-sandbox#384](https://github.com/kubernetes-sigs/agent-sandbox/issues/384): Upstream file-based env injection tracking
87+
88+
## Open Questions
89+
90+
- What is the new gRPC endpoint name and proto definition for the supervisor identity push?
91+
- How does the gateway discover pool pod IPs before the supervisor has registered? (likely from `.status.sandbox.podIPs` on the adopted Sandbox resource)
92+
- What happens if the gRPC push fails (supervisor crashed, network partition)? Fall back to cold start or retry?
93+
- How does gateway config validation prevent conflicting pool definitions (same image, different pool sizes)?
94+
- Should the gateway pool reconciler run as a background loop or be event-driven (watch SandboxWarmPool status)?
95+
- How does the always-on supervisor handle image updates? The supervisor process is running an older binary while the pool template references a newer image.

0 commit comments

Comments
 (0)