Skip to content

Commit dbe29e6

Browse files
committed
feat(server): add HA capacity metrics, scoped locks, and graceful drain
Multi-replica gateways need capacity signals, paced supervisor handoff, and bounded shared-database contention for production scaling. On SIGTERM every gateway reports 503 "draining" on /readyz and /health. Peer-routed PostgreSQL gateways then drain their supervisor sessions before stopping their listener. Supervisors reset their reconnect backoff after an accepted session and jitter retries, and a new session receives relays only after SessionAccepted is queued. Expose per-replica session, relay, peer-RPC, mutation-lock, and watch-poller metrics with bounded labels. Replace the fleet-wide mutation guard with global/workspace/sandbox intention locks while preserving the legacy global key for rolling upgrades. Use a dedicated four-connection PostgreSQL lock pool with bounded acquisition and return. Lock timeouts fail with UNAVAILABLE, reason MUTATION_LOCK_TIMEOUT, and a one-second retry delay. Provider refresh validates the credential expiry before staging and removes staged credentials when lock acquisition fails. Reconcile startup endpoint status per sandbox and batch watched resource-version reads in groups of 1,000. Add optional autoscaling/v2 HPA support and document drain budgets against the chart's existing 30-second termination grace default. Give certgen hook pods their own labels so gateway selectors and the HPA no longer match them. Document placement, rollout, connection-pool, and HPA limitations. Add PostgreSQL concurrency and stalled-connection tests, credential cleanup coverage, Kubernetes rollout tests, and a test:rust:postgres task. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <[email protected]>
1 parent 33a8eac commit dbe29e6

50 files changed

Lines changed: 9970 additions & 631 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/skills/helm-dev-environment/SKILL.md‎

Lines changed: 17 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -134,6 +134,10 @@ namespace with a `uri` key. For local manual testing, either create your own
134134
PostgreSQL Secret or use the e2e PostgreSQL fixture manifest in
135135
`e2e/kubernetes/postgres-fixture.yaml`.
136136

137+
Gateway pods in the `high-availability` profile use external PostgreSQL, so
138+
they drain their supervisor sessions when deleted or rolled and can take up to
139+
30 seconds to terminate.
140+
137141
For the `high-availability` profile, return to the repository root and apply the
138142
GatewayClass and BackendTrafficPolicy manifest after Skaffold has installed
139143
Envoy Gateway:
@@ -271,16 +275,24 @@ The kube e2e wrapper creates only one port-forward, to `svc/openshell`; it no
271275
longer forwards the unauthenticated health listener or runs a `/readyz` e2e
272276
target. `/readyz` remains covered by server unit/integration tests.
273277

274-
Use `mise run e2e:kubernetes:ha-rebalancing` for full-suite HA coverage. The
275-
task creates an external PostgreSQL fixture, installs Envoy Gateway, applies
278+
Use `mise run e2e:kubernetes:ha-rebalancing` for HA coverage. The task creates
279+
an external PostgreSQL fixture, installs Envoy Gateway, applies
276280
`deploy/kube/manifests/envoy-gateway-openshell.yaml`, enables the chart
277-
`GRPCRoute`, and runs the full Kubernetes e2e suite, including
278-
`kubernetes_ha_rebalancing`. That coverage validates sandbox create/watch and
281+
`GRPCRoute`, and runs the CLI conformance profile and the
282+
`kubernetes_ha_rebalancing` tests. That coverage validates sandbox create/watch and
279283
exec through the Envoy proxy while gateway replicas scale up, scale down, and
280284
rotate. It also keeps a long-running sandbox alive and runs upload/download
281285
operations while gateway pods roll, so file sync exercises the same relay retry
282286
path as interactive sessions.
283287

288+
`supervisor_sessions_redistribute_across_gateway_pod_rolls` restarts the
289+
gateway Deployment, reads `openshell_server_draining` from the terminating
290+
pods, and checks that the new pods' `openshell_server_supervisor_sessions` add
291+
up to the Ready sandbox count. It scrapes each pod through
292+
`kubectl get --raw /api/v1/namespaces/<ns>/pods/<pod>:9090/proxy/metrics`.
293+
Gateway pods take up to 30 seconds to terminate because each drains its
294+
sessions, and the HA tests are allowed ten minutes each.
295+
284296
If you reuse an existing Skaffold cluster for the full kube suite, make sure the
285297
chart has `server.hostGatewayIP` set so sandbox pods can resolve
286298
`host.openshell.internal` back to the test host. The e2e wrapper detects this on
@@ -446,6 +458,7 @@ for dependencies still declared in `Chart.yaml`.
446458
| `deploy/helm/openshell/ci/values-cert-manager.yaml` | cert-manager PKI overlay (opt-in; disables pkiInitJob) |
447459
| `deploy/helm/openshell/ci/values-gateway.yaml` | Envoy Gateway GRPCRoute + Gateway overlay |
448460
| `deploy/helm/openshell/ci/values-high-availability.yaml` | HA test overlay (`replicaCount: 2` with external PostgreSQL Secret) |
461+
| `deploy/helm/openshell/ci/values-autoscaling.yaml` | Render-only overlay for the optional gateway HorizontalPodAutoscaler (helm lint and helm-unittest) |
449462
| `deploy/helm/openshell/ci/values-keycloak.yaml` | Keycloak OIDC overlay |
450463
| `deploy/helm/openshell/ci/values-spire.yaml` | SPIFFE/SPIRE provider token grant overlay |
451464
| `deploy/helm/openshell/ci/values-spire-stack.yaml` | SPIRE hardened chart values for local dev |

‎.config/nextest.toml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,10 @@ kubernetes-ha = { max-threads = 1 }
2626
[[profile.e2e-kubernetes.overrides]]
2727
filter = "test(/gateway_(scale_and_rollout|pod_rolls)$/)"
2828
test-group = "kubernetes-ha"
29+
# Each rolled gateway pod drains its supervisor sessions for up to the 30s
30+
# termination grace period, so these tests get ten minutes, matching
31+
# HA_SYNC_TIMEOUT in kubernetes_ha_rebalancing.rs.
32+
slow-timeout = { period = "60s", terminate-after = 10 }
2933

3034
# Relative to the profile store dir (`e2e/rust/target/nextest/e2e-kubernetes/`).
3135
[profile.e2e-kubernetes.junit]

‎TESTING.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,23 @@ mise run test:rust # cargo test --workspace
4848

4949
Rust validation checks tracked Cargo lockfiles; run `mise run rust:lockfiles:check` to check them directly. If one is stale, refresh it with Cargo using its adjacent manifest, review the diff, and commit the update.
5050

51+
### PostgreSQL-backed tests
52+
53+
Tests that need a real PostgreSQL server, such as advisory-lock concurrency
54+
across two stores, are `#[ignore]`d and named `postgres_*`. Run them with:
55+
56+
```shell
57+
mise run test:rust:postgres
58+
```
59+
60+
The task starts a disposable PostgreSQL container with Docker or Podman
61+
(set `CONTAINER_ENGINE` to choose), runs the tests one at a time, and removes
62+
the container. Each test works in its own temporary schema. To use your own
63+
disposable database, set `OPENSHELL_TEST_POSTGRES_URL`. Never point it at a
64+
database that a running gateway uses: the tests take fleet-wide advisory locks.
65+
CI does not run these tests; the Kubernetes HA e2e suite covers PostgreSQL end
66+
to end.
67+
5168
### Native Windows validation
5269

5370
Use `mise run --skip-tools pre-commit` with the existing Rust/MSVC toolchain.
@@ -428,6 +445,7 @@ Available task variants:
428445
|---|---|
429446
| `e2e:kubernetes` | Default Rust e2e against Helm-deployed gateway |
430447
| `e2e:kubernetes:db` | All database backend scenarios (SQLite + external PostgreSQL) |
448+
| `e2e:kubernetes:ha-rebalancing` | Two gateway replicas behind Envoy with external PostgreSQL: scale, pod deletion, rollout drain and session redistribution, and file sync during pod rolls |
431449
| `e2e:kubernetes:sidecar` | Supervisor sidecar topology overlay |
432450
| `e2e:kubernetes:credential-drivers` | Kubernetes Secrets and Vault credential storage |
433451
| `e2e:kubernetes:workspace-managed` | Managed workspace mode (auto-created namespaces) |

0 commit comments

Comments
 (0)