Repository navigation
Commit dbe29e6
committed
feat(server): add HA capacity metrics, scoped locks, and graceful drain
Multi-replica gateways need capacity signals, paced supervisor handoff,
and bounded shared-database contention for production scaling.
On SIGTERM every gateway reports 503 "draining" on /readyz and /health.
Peer-routed PostgreSQL gateways then drain their supervisor sessions
before stopping their listener. Supervisors reset their reconnect
backoff after an accepted session and jitter retries, and a new session
receives relays only after SessionAccepted is queued. Expose
per-replica session, relay, peer-RPC, mutation-lock, and watch-poller
metrics with bounded labels.
Replace the fleet-wide mutation guard with global/workspace/sandbox
intention locks while preserving the legacy global key for rolling
upgrades. Use a dedicated four-connection PostgreSQL lock pool with
bounded acquisition and return. Lock timeouts fail with UNAVAILABLE,
reason MUTATION_LOCK_TIMEOUT, and a one-second retry delay. Provider
refresh validates the credential expiry before staging and removes
staged credentials when lock acquisition fails. Reconcile startup
endpoint status per sandbox and batch watched resource-version reads in
groups of 1,000.
Add optional autoscaling/v2 HPA support and document drain budgets against
the chart's existing 30-second termination grace default. Give certgen
hook pods their own labels so gateway selectors and the HPA no longer
match them. Document placement, rollout, connection-pool, and HPA
limitations. Add PostgreSQL concurrency and stalled-connection tests,
credential cleanup coverage, Kubernetes rollout tests, and a
test:rust:postgres task.
Part of NVIDIA#3528
Signed-off-by: Emilien Macchi <[email protected]>1 parent 33a8eac commit dbe29e6
50 files changed
Lines changed: 9970 additions & 631 deletions
File tree
- .agents/skills/helm-dev-environment
- .config
- crates
- openshell-server
- src
- compute
- grpc
- policy
- persistence
- tests
- openshell-supervisor-process/src
- deploy/helm/openshell
- ci
- templates
- tests
- docs
- kubernetes
- observability
- sdk
- e2e/rust/tests
- skills/debug-openshell-cluster
- tasks
- scripts
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
134 | 134 | | |
135 | 135 | | |
136 | 136 | | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
137 | 141 | | |
138 | 142 | | |
139 | 143 | | |
| |||
271 | 275 | | |
272 | 276 | | |
273 | 277 | | |
274 | | - | |
275 | | - | |
| 278 | + | |
| 279 | + | |
276 | 280 | | |
277 | | - | |
278 | | - | |
| 281 | + | |
| 282 | + | |
279 | 283 | | |
280 | 284 | | |
281 | 285 | | |
282 | 286 | | |
283 | 287 | | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
284 | 296 | | |
285 | 297 | | |
286 | 298 | | |
| |||
446 | 458 | | |
447 | 459 | | |
448 | 460 | | |
| 461 | + | |
449 | 462 | | |
450 | 463 | | |
451 | 464 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
29 | 33 | | |
30 | 34 | | |
31 | 35 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
48 | 48 | | |
49 | 49 | | |
50 | 50 | | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
51 | 68 | | |
52 | 69 | | |
53 | 70 | | |
| |||
428 | 445 | | |
429 | 446 | | |
430 | 447 | | |
| 448 | + | |
431 | 449 | | |
432 | 450 | | |
433 | 451 | | |
| |||
0 commit comments