You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 4097490
Browse filesBrowse the repository at this point in the historyBrowse files
feat(server): add HA capacity metrics, scoped locks, and graceful drain
Multi-replica gateways need capacity signals, paced supervisor handoff,
and bounded shared-database contention for production scaling.
On SIGTERM every gateway reports 503 "draining" on /readyz and /health.
Peer-routed PostgreSQL gateways then drain their supervisor sessions
before stopping their listener. Supervisors reset their reconnect
backoff after an accepted session and jitter retries, and a new session
receives relays only after SessionAccepted is queued. Expose
per-replica session, relay, peer-RPC, mutation-lock, and watch-poller
metrics with bounded labels.
Replace the fleet-wide mutation guard with global/workspace/sandbox
intention locks while preserving the legacy global key for rolling
upgrades. Use a dedicated four-connection PostgreSQL lock pool with
bounded acquisition and return. Lock timeouts fail with UNAVAILABLE,
reason MUTATION_LOCK_TIMEOUT, and a one-second retry delay. Provider
refresh validates the credential expiry before staging and removes
staged credentials when lock acquisition fails. Reconcile startup
endpoint status per sandbox and batch watched resource-version reads in
groups of 1,000.
Raise the Helm termination grace default to 30 seconds and add optional
autoscaling/v2 HPA support. Give certgen hook pods their own labels so
gateway selectors and the HPA no longer match them. Document placement,
rollout, connection-pool, and HPA limitations. Add a test:rust:postgres
task with PostgreSQL concurrency and stalled-connection regressions,
credential cleanup coverage, and Kubernetes rollout tests.
Part of NVIDIA#3528
Signed-off-by: Emilien Macchi <[email protected]>
|`deploy/helm/openshell/ci/values-high-availability.yaml`| HA test overlay (`replicaCount: 2` with external PostgreSQL Secret) |
461
+
|`deploy/helm/openshell/ci/values-autoscaling.yaml`| Render-only overlay for the optional gateway HorizontalPodAutoscaler (helm lint and helm-unittest) |
Copy file name to clipboardExpand all lines: TESTING.md
+18Lines changed: 18 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -48,6 +48,23 @@ mise run test:rust # cargo test --workspace
48
48
49
49
Rust validation checks tracked Cargo lockfiles; run `mise run rust:lockfiles:check` to check them directly. If one is stale, refresh it with Cargo using its adjacent manifest, review the diff, and commit the update.
50
50
51
+
### PostgreSQL-backed tests
52
+
53
+
Tests that need a real PostgreSQL server, such as advisory-lock concurrency
54
+
across two stores, are `#[ignore]`d and named `postgres_*`. Run them with:
55
+
56
+
```shell
57
+
mise run test:rust:postgres
58
+
```
59
+
60
+
The task starts a disposable PostgreSQL container with Docker or Podman
61
+
(set `CONTAINER_ENGINE` to choose), runs the tests one at a time, and removes
62
+
the container. Each test works in its own temporary schema. To use your own
63
+
disposable database, set `OPENSHELL_TEST_POSTGRES_URL`. Never point it at a
64
+
database that a running gateway uses: the tests take fleet-wide advisory locks.
65
+
CI does not run these tests; the Kubernetes HA e2e suite covers PostgreSQL end
66
+
to end.
67
+
51
68
### Native Windows validation
52
69
53
70
Use `mise run --skip-tools pre-commit` with the existing Rust/MSVC toolchain.
@@ -403,6 +420,7 @@ Available task variants:
403
420
|---|---|
404
421
|`e2e:kubernetes`| Default Rust e2e against Helm-deployed gateway |
405
422
|`e2e:kubernetes:db`| All database backend scenarios (SQLite + external PostgreSQL) |
423
+
|`e2e:kubernetes:ha-rebalancing`| Two gateway replicas behind Envoy with external PostgreSQL: scale, pod deletion, rollout drain and session redistribution, and file sync during pod rolls |
0 commit comments