|
| 1 | +# Implementation Plan: Warm Pool Feasibility Study |
| 2 | + |
| 3 | +**Branch**: `6111-warm-pool-feasibility` | **Date**: 2026-07-09 | **Spec**: [spec.md](spec.md) |
| 4 | +**Input**: Feature specification from `specs/001-warm-pool-feasibility/spec.md` |
| 5 | + |
| 6 | +## Summary |
| 7 | + |
| 8 | +Evaluate Kubernetes Agent Sandbox warm pooling on a ROSA HCP OpenShift cluster to determine whether sandbox startup latency can be reduced from 8-12 seconds to under 2 seconds. The study provisions a short-lived cluster, runs 6 experiments with shell-based measurement scripts, and produces an RFC in `rfc/` with latency data and architectural recommendations for OpenShell integration. |
| 9 | + |
| 10 | +## Technical Context |
| 11 | + |
| 12 | +**Language/Version**: Bash (measurement scripts), Go (sidecar readiness binary), Markdown (RFC) |
| 13 | +**Primary Dependencies**: ROSA HCP (AAET profile), Agent Sandbox operator + extension CRDs, OpenShell Helm chart (Krzysztof's wrapper), kubectl, oc |
| 14 | +**Storage**: CSV files for measurement data, RFC markdown for results |
| 15 | +**Testing**: Manual validation of measurement scripts, N=10-20 runs per configuration |
| 16 | +**Target Platform**: ROSA HCP, OpenShift 4.20+ (K8s 1.33+), us-east-2, 3x m5.2xlarge workers |
| 17 | +**Project Type**: Feasibility study / spike |
| 18 | +**Performance Goals**: Measure sandbox startup latency, warm pool claim-to-ready target <2s |
| 19 | +**Constraints**: Short-lived cluster (tear down after experiments), no code changes to OpenShell core |
| 20 | +**Scale/Scope**: 6 experiments, 4+ configurations, 10-20 runs each, ~100 total measurement runs |
| 21 | + |
| 22 | +## Constitution Check |
| 23 | + |
| 24 | +*No project constitution defined (template only). Gate passes trivially.* |
| 25 | + |
| 26 | +## Project Structure |
| 27 | + |
| 28 | +### Documentation (this feature) |
| 29 | + |
| 30 | +```text |
| 31 | +specs/001-warm-pool-feasibility/ |
| 32 | +├── plan.md # This file |
| 33 | +├── spec.md # Feature specification |
| 34 | +├── research.md # Phase 0: resolved unknowns |
| 35 | +├── data-model.md # Phase 1: measurement data model |
| 36 | +├── quickstart.md # Phase 1: setup and execution guide |
| 37 | +└── tasks.md # Phase 2 output (generated by /speckit-tasks) |
| 38 | +``` |
| 39 | + |
| 40 | +### Source Code (repository root) |
| 41 | + |
| 42 | +```text |
| 43 | +experiments/ |
| 44 | +├── measure-cold-start.sh # Experiment 1: cold-start baseline |
| 45 | +├── measure-warm-pool.sh # Experiment 2: warm pool claim latency |
| 46 | +├── measure-readiness-gates.sh # Experiment 3: Pod Readiness Gates |
| 47 | +├── measure-sidecar-readiness.sh # Experiment 4: sidecar readiness pattern |
| 48 | +├── measure-env-injection.sh # Experiment 5: env var injection |
| 49 | +├── measure-combined.sh # Experiment 6: combined best configuration |
| 50 | +├── lib/ |
| 51 | +│ ├── common.sh # Shared measurement functions (timestamps, CSV output) |
| 52 | +│ └── wait-ready.sh # Pod readiness wait with timeout |
| 53 | +├── manifests/ |
| 54 | +│ ├── sandbox-template.yaml # SandboxTemplate for warm pool experiments |
| 55 | +│ ├── warm-pool.yaml # SandboxWarmPool configurations |
| 56 | +│ ├── sandbox-claim.yaml # SandboxClaim templates |
| 57 | +│ ├── readiness-gate-pod.yaml # Pod with custom ReadinessGate |
| 58 | +│ ├── sidecar-readiness.yaml # Sidecar container with readiness endpoint |
| 59 | +│ └── image-prepull-daemonset.yaml # DaemonSet for pre-pulling images |
| 60 | +├── sidecar/ |
| 61 | +│ ├── main.go # Readiness sidecar binary (HTTP 503/200) |
| 62 | +│ ├── Dockerfile # Multi-stage build for scratch image |
| 63 | +│ └── Makefile # Build and push sidecar image |
| 64 | +└── results/ |
| 65 | + └── *.csv # Raw measurement data (gitignored) |
| 66 | +
|
| 67 | +rfc/NNNN-warm-pool-feasibility/ |
| 68 | +└── README.md # Results RFC (number assigned by maintainers) |
| 69 | +``` |
| 70 | + |
| 71 | +**Structure Decision**: Experiments live in `experiments/` at repo root for easy discovery. The sidecar binary is self-contained in `experiments/sidecar/`. Results CSV files are gitignored (raw data is synthesized into the RFC). The RFC follows the standard `rfc/NNNN-name/README.md` pattern. |
| 72 | + |
| 73 | +## Global Constraints |
| 74 | + |
| 75 | +These constraints from the spec apply to all tasks: |
| 76 | + |
| 77 | +- **Kubernetes version**: 1.33+ (required for KEP-753 native sidecar containers GA) |
| 78 | +- **OpenShift version**: 4.20+ (maps to K8s 1.33) |
| 79 | +- **Cluster sizing**: 3 worker nodes, m5.2xlarge (8 vCPU, 32 GB each), us-east-2 |
| 80 | +- **Cluster lifecycle**: Short-lived, tear down after experiments |
| 81 | +- **Operator preference**: Red Hat Agent Sandbox tech preview first, upstream v0.5.0 manifests as fallback |
| 82 | +- **Measurement format**: Shell scripts producing CSV output |
| 83 | +- **Scope**: Kubernetes-backed sandboxes only (no Podman/Docker) |
| 84 | + |
| 85 | +## Measurement Library Interfaces |
| 86 | + |
| 87 | +Functions exported by `experiments/lib/common.sh` (consumed by all experiment scripts): |
| 88 | + |
| 89 | +```bash |
| 90 | +capture_timestamp() # Returns nanosecond epoch (date +%s%N) |
| 91 | +write_csv_header(file) # Writes CSV header row to file |
| 92 | +write_csv_row(file, run, config, create_ts, ready_ts, delta_ms, pod, status) |
| 93 | +collect_pod_events(pod) # Returns JSON phase breakdown from kubectl events |
| 94 | +compute_stats(csv_file) # Computes p50/p90 from CSV delta_ms column, prints summary |
| 95 | +``` |
| 96 | + |
| 97 | +Functions exported by `experiments/lib/wait-ready.sh`: |
| 98 | + |
| 99 | +```bash |
| 100 | +wait_for_ready(resource, timeout_s) # Blocks until resource is Ready or timeout |
| 101 | +wait_for_pod_ready(pod, ns, timeout_s) # Blocks until pod condition Ready=True |
| 102 | +``` |
| 103 | + |
| 104 | +## Execution Phases |
| 105 | + |
| 106 | +### Phase 1: Cluster Provisioning & Setup |
| 107 | + |
| 108 | +1. Provision ROSA HCP cluster (3x m5.2xlarge, OpenShift 4.20+) |
| 109 | +2. Install Red Hat Agent Sandbox operator from OperatorHub (fallback: upstream manifests) |
| 110 | +3. Verify extension CRDs: SandboxTemplate, SandboxWarmPool, SandboxClaim |
| 111 | +4. Deploy OpenShell via Krzysztof's Helm chart wrapper |
| 112 | +5. Pre-pull sandbox images on all worker nodes (DaemonSet approach) |
| 113 | +6. Validate: cold-start sandbox creation works end-to-end |
| 114 | + |
| 115 | +### Phase 2: Measurement Script Development |
| 116 | + |
| 117 | +1. Build shared measurement library (`experiments/lib/common.sh`) |
| 118 | +2. Build cold-start measurement script (Experiment 1) |
| 119 | +3. Build warm pool measurement script (Experiments 2, 3) |
| 120 | +4. Build sidecar readiness binary and container |
| 121 | +5. Build sidecar/readiness-gate measurement scripts (Experiments 3, 4) |
| 122 | +6. Build env var injection measurement script (Experiment 5) |
| 123 | +7. Build combined measurement script (Experiment 6) |
| 124 | + |
| 125 | +### Phase 3: Experiment Execution |
| 126 | + |
| 127 | +1. Run Experiment 1: Cold-start baseline (N=10 pre-pulled, N=5 without) |
| 128 | +2. Run Experiment 2: Warm pool claim latency (default probes, aggressive probes) |
| 129 | +3. Run Experiment 3: Pod Readiness Gates |
| 130 | +4. Run Experiment 4: Sidecar readiness pattern |
| 131 | +5. Run Experiment 5: Env var injection |
| 132 | +6. Run Experiment 6: Combined best configuration |
| 133 | + |
| 134 | +### Phase 4: Results & RFC |
| 135 | + |
| 136 | +1. Compile CSV data into summary tables (p50, p90 per configuration) |
| 137 | +2. Write RFC following `rfc/0000-template/README.md` format |
| 138 | +3. Include: executive summary, experiment setup, results tables, architecture recommendations, gaps/risks, next steps |
| 139 | +4. Tear down cluster |
| 140 | + |
| 141 | +## Risk Mitigations |
| 142 | + |
| 143 | +| Risk | Mitigation | |
| 144 | +|------|------------| |
| 145 | +| Red Hat TP operator lacks extension CRDs | Fallback: upstream `extensions.yaml` on top of operator | |
| 146 | +| OpenShift 4.20 not available on ROSA HCP | Check `rosa list versions`, use latest available with K8s 1.33+ | |
| 147 | +| Krzysztof's chart incompatible with OpenShift 4.20 | Adjust chart values; worst case deploy OpenShell manually | |
| 148 | +| Sidecar readiness requires custom image | Build minimal Go binary, push to ghcr.io | |
| 149 | +| Env var injection triggers cold start | Document as constraint, propose alternative identity binding | |
| 150 | + |
| 151 | +## Complexity Tracking |
| 152 | + |
| 153 | +No constitution violations to justify. |
0 commit comments