Skip to content

Commit d506b07

Browse files
committed
feat: Add plan, tasks, and review guide for warm pool feasibility study
Spec clarified (cluster sizing, RFC deliverable, measurement automation, operator preference). Plan covers 4 execution phases across 53 tasks. REVIEWERS.md provides review guide for spec PR. Assisted-By: 🤖 Claude Code
1 parent 6dbfb52 commit d506b07

8 files changed

Lines changed: 643 additions & 7 deletions

File tree

‎CLAUDE.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,4 +3,5 @@
33
<!-- SPECKIT START -->
44
For additional context about technologies to be used, project structure,
55
shell commands, and other important information, read the current plan
6+
at specs/001-warm-pool-feasibility/plan.md
67
<!-- SPECKIT END -->
Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
# Review Guide: Warm Pool Feasibility Study
2+
3+
**Generated**: 2026-07-09 | **Spec**: [spec.md](spec.md)
4+
5+
## Why This Change
6+
7+
OpenShell's Kubernetes driver creates a fresh Sandbox CR for every sandbox request, paying 8-12 seconds for pod scheduling, image pull, init container execution, supervisor startup, and gateway registration. For agent harnesses like OpenClaw that create sandboxes per tool call, this latency is unusable. The upstream Agent Sandbox project (v0.5.0) provides extension CRDs for warm pooling (SandboxTemplate, SandboxWarmPool, SandboxClaim), but OpenShell has no awareness of these. We need to know if warm pooling can deliver sub-2s sandbox startup on OpenShift before investing in integration work.
8+
9+
## What Changes
10+
11+
This study adds an `experiments/` directory with shell-based measurement scripts, Kubernetes manifests for warm pool configurations, and a minimal Go sidecar binary for readiness experiments. It provisions a short-lived ROSA HCP cluster, runs 6 experiments measuring cold-start vs warm pool latency across readiness configurations, and produces an RFC in `rfc/` with data-backed architectural recommendations for OpenShell warm pool integration. No changes to OpenShell core code.
12+
13+
## How It Works
14+
15+
The study uses a layered measurement approach across 4 execution phases:
16+
17+
1. **Cluster Setup**: ROSA HCP with 3x m5.2xlarge workers, Red Hat Agent Sandbox tech preview operator (upstream fallback), OpenShell via Krzysztof's Helm chart, and image pre-pulling via DaemonSet.
18+
19+
2. **Measurement Library**: Shared shell functions (`experiments/lib/common.sh`) for nanosecond timestamp capture, CSV output, pod event collection, and p50/p90 computation. Each experiment script wraps kubectl with these functions.
20+
21+
3. **Experiments**: Six experiments progressively build understanding:
22+
- Cold-start baseline (control measurement)
23+
- Warm pool claim-to-ready with default and aggressive readiness probes
24+
- Pod Readiness Gates (KEP-580) as probe-free alternative
25+
- Sidecar readiness pattern (Knative-style, using KEP-753 native sidecars)
26+
- Env var injection at claim time (identity binding feasibility)
27+
- Combined best-configuration measurement
28+
29+
4. **RFC**: Results compiled into a standalone RFC following the OpenShell RFC template, structured for later distillation into a GitHub Issue #2157 comment.
30+
31+
## When It Applies
32+
33+
**Applies when**:
34+
- Evaluating whether OpenShell should adopt Agent Sandbox warm pooling
35+
- Understanding sandbox startup latency bottlenecks on OpenShift
36+
- Making architectural decisions about OpenShell's Kubernetes driver
37+
- Informing Issue #2157 design discussions with measured data
38+
39+
**Does not apply when**:
40+
- Podman/Docker single-player warm pooling (explicitly out of scope)
41+
- Production OpenShell code changes (this is measurement-only, no core changes)
42+
- Non-OpenShift Kubernetes distributions (results are OpenShift-specific)
43+
44+
## Key Decisions
45+
46+
1. **Layered measurement (no OpenShell code changes)**: Start with raw Agent Sandbox measurements, then layer OpenShell-specific concerns. Rejected "modify the Kubernetes driver first" because if the raw K8s layer is too slow, code changes would be wasted.
47+
48+
2. **Red Hat tech preview operator first, upstream fallback**: The tech preview may lack extension CRDs, but testing the downstream operator path is part of the study's value. Upstream manifests are applied on top only if needed.
49+
50+
3. **Shell scripts over Go benchmark harness**: N=10-20 per configuration is enough for a feasibility study. Shell scripts with `date +%s%N` and `kubectl wait` provide sufficient precision without over-engineering.
51+
52+
4. **Results as RFC (not Google Doc)**: The RFC template provides structured review and lives in the public repo. Google Docs are inaccessible to external contributors.
53+
54+
5. **3x m5.2xlarge workers**: Enough capacity for warm pool replicas and burst tests without hitting resource limits that would skew measurements. Short-lived cluster to control cost.
55+
56+
## Areas Needing Attention
57+
58+
- **Readiness probe interval assumption**: The brainstorm identifies the readiness probe interval as the dominant latency bottleneck. If this assumption is wrong (e.g., kubelet overhead dominates), the Readiness Gate and sidecar experiments may not show the expected improvement.
59+
- **Env var injection behavior is undocumented**: The SandboxClaim `envVarsInjectionPolicy` behavior is based on reading the CRD spec, not tested behavior. The actual controller implementation may differ.
60+
- **Sidecar readiness binary**: Building and pushing a custom container image adds complexity. If the sidecar experiment is not needed (readiness gates are sufficient), this work could be skipped.
61+
- **RFC number assignment**: The OpenShell RFC process requires maintainer-assigned numbers from an originating issue. A placeholder number is used until assignment.
62+
63+
## Open Questions
64+
65+
- Does the Red Hat Agent Sandbox operator tech preview include extension CRDs, or only the core Sandbox CRD? (Answered by T002/T003 during setup.)
66+
- What happens when the warm pool is exhausted? Does SandboxClaim stay Pending or fall back to cold start? (Answered by T024 during experiments.)
67+
- Is the sidecar experiment necessary if Pod Readiness Gates already eliminate the probe interval bottleneck? (Decided after T033 results.)
68+
69+
## Review Checklist
70+
71+
- [ ] Key decisions are justified
72+
- [ ] Measurement methodology is sound (N=10+, per-phase timestamps, CSV output)
73+
- [ ] Scope boundaries are clear (no OpenShell core changes, K8s only)
74+
- [ ] Success criteria are measurable (sub-2s target, p50/p90 computation)
75+
- [ ] Cluster sizing is appropriate for experiment scope
76+
- [ ] RFC structure follows the OpenShell RFC template
77+
- [ ] No Google Drive links referenced in public-facing artifacts
78+
79+
---
80+
81+
<!-- Code phase sections are appended below this line by the phase-manager command -->
Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# Data Model: Warm Pool Feasibility Study
2+
3+
## Entities
4+
5+
### MeasurementRun
6+
7+
A single sandbox creation or claim with captured timestamps.
8+
9+
| Field | Type | Description |
10+
|-------|------|-------------|
11+
| run_number | int | Sequential run number within an experiment |
12+
| experiment_id | string | Experiment identifier (e.g., `cold-start`, `warm-pool-default`) |
13+
| config_label | string | Configuration description (e.g., `probe-10s`, `readiness-gate`) |
14+
| create_timestamp | int64 | Nanosecond epoch when kubectl apply was issued |
15+
| ready_timestamp | int64 | Nanosecond epoch when pod/sandbox became Ready |
16+
| delta_ms | float | Latency in milliseconds (ready - create) |
17+
| phases | string | JSON-encoded phase breakdown from events |
18+
| pod_name | string | Name of the pod that was created or claimed |
19+
| status | enum | `success`, `timeout`, `error` |
20+
21+
### ExperimentConfig
22+
23+
A specific combination of settings being tested.
24+
25+
| Field | Type | Description |
26+
|-------|------|-------------|
27+
| experiment_id | string | Unique identifier |
28+
| description | string | Human-readable description |
29+
| readiness_method | enum | `probe-default`, `probe-1s`, `readiness-gate`, `sidecar` |
30+
| pool_size | int | Number of SandboxWarmPool replicas (0 for cold-start) |
31+
| env_injection | bool | Whether SandboxClaim includes env vars |
32+
| target_runs | int | Number of measurement runs to execute |
33+
34+
### PhaseTimestamp
35+
36+
Per-phase latency breakdown extracted from pod events.
37+
38+
| Field | Type | Description |
39+
|-------|------|-------------|
40+
| run_number | int | FK to MeasurementRun |
41+
| phase_name | string | `scheduled`, `image-pulled`, `init-complete`, `supervisor-ready`, `ssh-available` |
42+
| timestamp | int64 | Nanosecond epoch of phase completion |
43+
| delta_from_create_ms | float | Milliseconds from create to this phase |
44+
45+
## Relationships
46+
47+
```
48+
ExperimentConfig 1---* MeasurementRun
49+
MeasurementRun 1---* PhaseTimestamp
50+
```
51+
52+
## State Transitions
53+
54+
### MeasurementRun Lifecycle
55+
56+
```
57+
Created --> Running --> Success
58+
\--> Timeout (>60s)
59+
\--> Error
60+
```
61+
62+
### Warm Pool Pod Lifecycle
63+
64+
```
65+
Provisioned (warm, NotReady) --> Claimed (SandboxClaim created)
66+
--> Ready (readiness condition met)
67+
--> Terminated (cleanup)
68+
```
69+
70+
## CSV Output Format
71+
72+
```csv
73+
run,experiment,config,create_ts,ready_ts,delta_ms,pod,status
74+
1,cold-start,openshell-prepulled,1720000000000000000,1720000010500000000,10500.0,sandbox-abc123,success
75+
```
Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
# Implementation Plan: Warm Pool Feasibility Study
2+
3+
**Branch**: `6111-warm-pool-feasibility` | **Date**: 2026-07-09 | **Spec**: [spec.md](spec.md)
4+
**Input**: Feature specification from `specs/001-warm-pool-feasibility/spec.md`
5+
6+
## Summary
7+
8+
Evaluate Kubernetes Agent Sandbox warm pooling on a ROSA HCP OpenShift cluster to determine whether sandbox startup latency can be reduced from 8-12 seconds to under 2 seconds. The study provisions a short-lived cluster, runs 6 experiments with shell-based measurement scripts, and produces an RFC in `rfc/` with latency data and architectural recommendations for OpenShell integration.
9+
10+
## Technical Context
11+
12+
**Language/Version**: Bash (measurement scripts), Go (sidecar readiness binary), Markdown (RFC)
13+
**Primary Dependencies**: ROSA HCP (AAET profile), Agent Sandbox operator + extension CRDs, OpenShell Helm chart (Krzysztof's wrapper), kubectl, oc
14+
**Storage**: CSV files for measurement data, RFC markdown for results
15+
**Testing**: Manual validation of measurement scripts, N=10-20 runs per configuration
16+
**Target Platform**: ROSA HCP, OpenShift 4.20+ (K8s 1.33+), us-east-2, 3x m5.2xlarge workers
17+
**Project Type**: Feasibility study / spike
18+
**Performance Goals**: Measure sandbox startup latency, warm pool claim-to-ready target <2s
19+
**Constraints**: Short-lived cluster (tear down after experiments), no code changes to OpenShell core
20+
**Scale/Scope**: 6 experiments, 4+ configurations, 10-20 runs each, ~100 total measurement runs
21+
22+
## Constitution Check
23+
24+
*No project constitution defined (template only). Gate passes trivially.*
25+
26+
## Project Structure
27+
28+
### Documentation (this feature)
29+
30+
```text
31+
specs/001-warm-pool-feasibility/
32+
├── plan.md # This file
33+
├── spec.md # Feature specification
34+
├── research.md # Phase 0: resolved unknowns
35+
├── data-model.md # Phase 1: measurement data model
36+
├── quickstart.md # Phase 1: setup and execution guide
37+
└── tasks.md # Phase 2 output (generated by /speckit-tasks)
38+
```
39+
40+
### Source Code (repository root)
41+
42+
```text
43+
experiments/
44+
├── measure-cold-start.sh # Experiment 1: cold-start baseline
45+
├── measure-warm-pool.sh # Experiment 2: warm pool claim latency
46+
├── measure-readiness-gates.sh # Experiment 3: Pod Readiness Gates
47+
├── measure-sidecar-readiness.sh # Experiment 4: sidecar readiness pattern
48+
├── measure-env-injection.sh # Experiment 5: env var injection
49+
├── measure-combined.sh # Experiment 6: combined best configuration
50+
├── lib/
51+
│ ├── common.sh # Shared measurement functions (timestamps, CSV output)
52+
│ └── wait-ready.sh # Pod readiness wait with timeout
53+
├── manifests/
54+
│ ├── sandbox-template.yaml # SandboxTemplate for warm pool experiments
55+
│ ├── warm-pool.yaml # SandboxWarmPool configurations
56+
│ ├── sandbox-claim.yaml # SandboxClaim templates
57+
│ ├── readiness-gate-pod.yaml # Pod with custom ReadinessGate
58+
│ ├── sidecar-readiness.yaml # Sidecar container with readiness endpoint
59+
│ └── image-prepull-daemonset.yaml # DaemonSet for pre-pulling images
60+
├── sidecar/
61+
│ ├── main.go # Readiness sidecar binary (HTTP 503/200)
62+
│ ├── Dockerfile # Multi-stage build for scratch image
63+
│ └── Makefile # Build and push sidecar image
64+
└── results/
65+
└── *.csv # Raw measurement data (gitignored)
66+
67+
rfc/NNNN-warm-pool-feasibility/
68+
└── README.md # Results RFC (number assigned by maintainers)
69+
```
70+
71+
**Structure Decision**: Experiments live in `experiments/` at repo root for easy discovery. The sidecar binary is self-contained in `experiments/sidecar/`. Results CSV files are gitignored (raw data is synthesized into the RFC). The RFC follows the standard `rfc/NNNN-name/README.md` pattern.
72+
73+
## Global Constraints
74+
75+
These constraints from the spec apply to all tasks:
76+
77+
- **Kubernetes version**: 1.33+ (required for KEP-753 native sidecar containers GA)
78+
- **OpenShift version**: 4.20+ (maps to K8s 1.33)
79+
- **Cluster sizing**: 3 worker nodes, m5.2xlarge (8 vCPU, 32 GB each), us-east-2
80+
- **Cluster lifecycle**: Short-lived, tear down after experiments
81+
- **Operator preference**: Red Hat Agent Sandbox tech preview first, upstream v0.5.0 manifests as fallback
82+
- **Measurement format**: Shell scripts producing CSV output
83+
- **Scope**: Kubernetes-backed sandboxes only (no Podman/Docker)
84+
85+
## Measurement Library Interfaces
86+
87+
Functions exported by `experiments/lib/common.sh` (consumed by all experiment scripts):
88+
89+
```bash
90+
capture_timestamp() # Returns nanosecond epoch (date +%s%N)
91+
write_csv_header(file) # Writes CSV header row to file
92+
write_csv_row(file, run, config, create_ts, ready_ts, delta_ms, pod, status)
93+
collect_pod_events(pod) # Returns JSON phase breakdown from kubectl events
94+
compute_stats(csv_file) # Computes p50/p90 from CSV delta_ms column, prints summary
95+
```
96+
97+
Functions exported by `experiments/lib/wait-ready.sh`:
98+
99+
```bash
100+
wait_for_ready(resource, timeout_s) # Blocks until resource is Ready or timeout
101+
wait_for_pod_ready(pod, ns, timeout_s) # Blocks until pod condition Ready=True
102+
```
103+
104+
## Execution Phases
105+
106+
### Phase 1: Cluster Provisioning & Setup
107+
108+
1. Provision ROSA HCP cluster (3x m5.2xlarge, OpenShift 4.20+)
109+
2. Install Red Hat Agent Sandbox operator from OperatorHub (fallback: upstream manifests)
110+
3. Verify extension CRDs: SandboxTemplate, SandboxWarmPool, SandboxClaim
111+
4. Deploy OpenShell via Krzysztof's Helm chart wrapper
112+
5. Pre-pull sandbox images on all worker nodes (DaemonSet approach)
113+
6. Validate: cold-start sandbox creation works end-to-end
114+
115+
### Phase 2: Measurement Script Development
116+
117+
1. Build shared measurement library (`experiments/lib/common.sh`)
118+
2. Build cold-start measurement script (Experiment 1)
119+
3. Build warm pool measurement script (Experiments 2, 3)
120+
4. Build sidecar readiness binary and container
121+
5. Build sidecar/readiness-gate measurement scripts (Experiments 3, 4)
122+
6. Build env var injection measurement script (Experiment 5)
123+
7. Build combined measurement script (Experiment 6)
124+
125+
### Phase 3: Experiment Execution
126+
127+
1. Run Experiment 1: Cold-start baseline (N=10 pre-pulled, N=5 without)
128+
2. Run Experiment 2: Warm pool claim latency (default probes, aggressive probes)
129+
3. Run Experiment 3: Pod Readiness Gates
130+
4. Run Experiment 4: Sidecar readiness pattern
131+
5. Run Experiment 5: Env var injection
132+
6. Run Experiment 6: Combined best configuration
133+
134+
### Phase 4: Results & RFC
135+
136+
1. Compile CSV data into summary tables (p50, p90 per configuration)
137+
2. Write RFC following `rfc/0000-template/README.md` format
138+
3. Include: executive summary, experiment setup, results tables, architecture recommendations, gaps/risks, next steps
139+
4. Tear down cluster
140+
141+
## Risk Mitigations
142+
143+
| Risk | Mitigation |
144+
|------|------------|
145+
| Red Hat TP operator lacks extension CRDs | Fallback: upstream `extensions.yaml` on top of operator |
146+
| OpenShift 4.20 not available on ROSA HCP | Check `rosa list versions`, use latest available with K8s 1.33+ |
147+
| Krzysztof's chart incompatible with OpenShift 4.20 | Adjust chart values; worst case deploy OpenShell manually |
148+
| Sidecar readiness requires custom image | Build minimal Go binary, push to ghcr.io |
149+
| Env var injection triggers cold start | Document as constraint, propose alternative identity binding |
150+
151+
## Complexity Tracking
152+
153+
No constitution violations to justify.
Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
# Quickstart: Warm Pool Feasibility Study
2+
3+
## Prerequisites
4+
5+
- AWS credentials for AAET profile (`851725220677`)
6+
- `rosa` CLI authenticated
7+
- `oc` / `kubectl` CLI
8+
- `gh` CLI for GitHub operations
9+
- `jq` for JSON processing
10+
11+
## Step 1: Provision Cluster
12+
13+
```bash
14+
rosa:create warm-pool-study
15+
```
16+
17+
3 worker nodes, m5.2xlarge, us-east-2. Wait for cluster to be Ready (~15 min).
18+
19+
## Step 2: Install Agent Sandbox Operator
20+
21+
```bash
22+
# Check if Red Hat TP is on OperatorHub
23+
oc get packagemanifests -n openshift-marketplace | grep sandbox
24+
25+
# If available, install via Subscription
26+
# If not, apply upstream manifests
27+
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/agent-sandbox/v0.5.0/manifest.yaml
28+
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/agent-sandbox/v0.5.0/extensions.yaml
29+
30+
# Verify CRDs
31+
kubectl api-resources | grep agents
32+
```
33+
34+
## Step 3: Deploy OpenShell
35+
36+
```bash
37+
git clone https://github.com/2000krysztof/Openshell-Openshift-Deploy
38+
cd Openshell-Openshift-Deploy
39+
./deploy.sh
40+
openshell status
41+
```
42+
43+
## Step 4: Pre-pull Images
44+
45+
```bash
46+
# DaemonSet that pre-pulls sandbox images on all nodes
47+
kubectl apply -f experiments/image-prepull-daemonset.yaml
48+
kubectl rollout status ds/image-prepull
49+
```
50+
51+
## Step 5: Run Experiments
52+
53+
```bash
54+
# Cold-start baseline (N=10 pre-pulled, N=5 no pre-pull)
55+
./experiments/measure-cold-start.sh
56+
57+
# Warm pool with default probes
58+
./experiments/measure-warm-pool.sh --config default
59+
60+
# Warm pool with 1s probes
61+
./experiments/measure-warm-pool.sh --config aggressive
62+
63+
# Readiness gates
64+
./experiments/measure-readiness-gates.sh
65+
66+
# Sidecar readiness
67+
./experiments/measure-sidecar-readiness.sh
68+
69+
# Env var injection
70+
./experiments/measure-env-injection.sh
71+
```
72+
73+
## Step 6: Generate RFC
74+
75+
Compile CSV data into the RFC at `rfc/NNNN-warm-pool-feasibility/README.md`.
76+
77+
## Step 7: Tear Down
78+
79+
```bash
80+
rosa:delete warm-pool-study
81+
```

0 commit comments

Comments
 (0)