Skip to content

Commit eb3eb15

Browse files
committed
docs: document scoped mutation locks and lock-pool sizing
In the High Availability guide, size PostgreSQL for 14 connections per gateway pod, 10 for data and 4 for the new lock pool, so three or more replicas now exceed what the PostgreSQL default leaves, and keep headroom for lock sessions that cancelled requests leave behind. Explain which changes lock a sandbox, a workspace, or the whole fleet, and that a lock wait over 10 seconds returns a retryable UNAVAILABLE with reason MUTATION_LOCK_TIMEOUT. A new upgrade subsection says to raise max_connections to the 14-per-pod size before upgrading, because the rollout already runs new pods at that count, and warns that older replicas keep serializing every mutation until the rollout finishes. Monitor Capacity gains the lock metrics and their queries, and names slow driver, credential, middleware, or profile source calls as another cause of lock waits. List the lock wait histogram and timeout counter, with their scope label, on the Gateway Metrics page. The counter also counts background work and leaves out lock connections that PostgreSQL does not open although at least a second of the wait remained. In the cluster debugging skill, replace the single advisory-lock paragraph with the lock levels and pool, what each timeout detail means, how a lock connection that PostgreSQL does not open shows up, when terminating a lock backend is safe, a pg_locks query that shows holders and waiters, and how to spot the global key. The per-pod greps also read the timeout counter, and new rows cover lock timeouts and lock connections that PostgreSQL does not open. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <[email protected]>
1 parent ac1ce1f commit eb3eb15

3 files changed

Lines changed: 188 additions & 19 deletions

File tree

‎docs/kubernetes/high-availability.mdx‎

Lines changed: 57 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -212,6 +212,7 @@ configuration, and access control.
212212
| Relay rejections | `openshell_server_relay_rejected_total` | Alert on any increase. |
213213
| Relay claims | `openshell_server_relay_claim_duration_seconds`, `openshell_server_relay_expired_total` | Supervisor connect-back time for exec, SSH, forwarding, and service traffic, including relays requested through peers. A high 99th percentile while rejections stay flat points at the supervisor or its node, not at relay capacity. Relays not claimed within 10 seconds are missing from the histogram and count as expired, so alert on any increase. |
214214
| Routed requests | `openshell_server_routed_request_attempts_total`, `openshell_server_peer_request_duration_seconds` | Local/peer relay setup mix and outbound peer rate, failures, and latency. Counts completed or cancelled attempts, including retries. Spikes of `grpc_code="unavailable"` during rollouts are expected. Do not use as an HPA target. |
215+
| Lock contention | `openshell_server_mutation_lock_wait_seconds`, `openshell_server_mutation_lock_timeouts_total` | Alert on any timeout. |
215216

216217
These queries assume that Prometheus labels each series with `namespace` and
217218
`pod`:
@@ -241,6 +242,12 @@ sum by (operation) (rate(openshell_server_routed_request_attempts_total{route="p
241242
242243
# 99th percentile peer request latency across the fleet.
243244
histogram_quantile(0.99, sum by (le, operation) (rate(openshell_server_peer_request_duration_seconds_bucket{outcome="success"}[5m])))
245+
246+
# 99th percentile mutation lock wait by scope. Watch scope="sandbox" during rollouts.
247+
histogram_quantile(0.99, sum by (le, scope) (rate(openshell_server_mutation_lock_wait_seconds_bucket[5m])))
248+
249+
# Mutation lock timeouts by scope. Alert on any increase.
250+
sum by (scope) (increase(openshell_server_mutation_lock_timeouts_total[10m]))
244251
```
245252

246253
Relay capacity is used on the replica that owns the sandbox's supervisor
@@ -250,6 +257,16 @@ stays counted for up to about 40 seconds, the 10-second claim timeout plus the
250257
30-second cleanup interval. Routed request counts include retries during
251258
rollouts.
252259

260+
Each supervisor that reconnects takes one or two short sandbox-scoped locks on
261+
the replica that receives it. When a gateway pod stops during a rollout, its
262+
supervisors reconnect to the other replicas, so watch
263+
`openshell_server_mutation_lock_wait_seconds` and
264+
`openshell_server_mutation_lock_timeouts_total` with `scope="sandbox"` on the
265+
receiving replicas. Rising waits or any timeout mean that those replicas
266+
receive reconnects faster than they can absorb them, or that slow compute
267+
driver, credential backend, middleware, or provider profile source calls during
268+
sandbox and provider changes hold their locks longer.
269+
253270
## Scale the Gateway
254271

255272
Scale the gateway by changing a fixed replica count or by letting a
@@ -380,7 +397,9 @@ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/openshell/pods
380397

381398
### Size PostgreSQL Connections
382399

383-
Each gateway pod opens up to 10 PostgreSQL connections on demand. The chart
400+
Each gateway pod opens up to 14 PostgreSQL connections on demand, 10 for data
401+
access and 4 for mutation locks. A request cancelled during a lock wait can
402+
leave its lock session open for up to 10 seconds, so keep headroom. The chart
384403
uses the Kubernetes default rolling update strategy for each workload kind, so
385404
the number of pods that run during a rollout depends on the kind.
386405

@@ -390,18 +409,18 @@ and each replaced pod can keep its connections throughout its termination
390409
grace period. A rollout can therefore run up to twice the replica count at
391410
once, and an eviction or an autoscaler scale-down during the rollout adds more
392411
terminating pods. Set PostgreSQL `max_connections` to at least
393-
`(2 × replicas + surge) × 10`, which leaves one surge of margin for those
412+
`(2 × replicas + surge) × 14`, which leaves one surge of margin for those
394413
pods, plus headroom for your other clients and administration. Use
395414
`autoscaling.maxReplicas` as the replica count when autoscaling is enabled.
396415

397416
| Replicas | Surge | Minimum `max_connections` |
398417
|---|---|---|
399-
| 2 | 1 | `(4 + 1) × 10 = 50` |
400-
| 3 | 1 | `(6 + 1) × 10 = 70` |
401-
| 4 | 1 | `(8 + 1) × 10 = 90` |
402-
| 6 | 2 | `(12 + 2) × 10 = 140` |
418+
| 2 | 1 | `(4 + 1) × 14 = 70` |
419+
| 3 | 1 | `(6 + 1) × 14 = 98` |
420+
| 4 | 1 | `(8 + 1) × 14 = 126` |
421+
| 6 | 2 | `(12 + 2) × 14 = 196` |
403422

404-
With a Deployment, five or more replicas exceed what the PostgreSQL default of
423+
With a Deployment, three or more replicas exceed what the PostgreSQL default of
405424
100 leaves for the gateway, because PostgreSQL reserves 3 connections for
406425
superusers. Raise `max_connections` or use a larger managed instance. Let one
407426
rollout finish before you start another, because each overlapping rollout adds
@@ -410,14 +429,44 @@ its own terminating pods.
410429
If you run several replicas as a StatefulSet with
411430
`workload.allowMultiReplicaStatefulSet`, a rollout adds no surge pods. It
412431
replaces one pod at a time and creates each replacement only after the old pod
413-
exits. Set `max_connections` to at least `replicas × 10`, using
432+
exits. Set `max_connections` to at least `replicas × 14`, using
414433
`autoscaling.maxReplicas` when autoscaling is enabled, plus headroom for your
415434
other clients and administration.
416435

417436
If a connection pooler sits between the gateway and PostgreSQL, use session
418437
pooling. The gateway holds session-level advisory locks, which transaction
419438
pooling breaks.
420439

440+
Mutations lock only what they change. Sandbox operations lock their own
441+
sandbox, provider and workspace-profile changes lock their workspace, and
442+
gateway-global policy, settings, and platform-profile changes lock the whole
443+
fleet. Sandbox start, stop, and delete lock only on the replica that runs them,
444+
and only a gateway-global change on that replica blocks them directly.
445+
Provider, workspace-profile, and other replicas' gateway-global changes delay
446+
them only while another operation on the same sandbox waits behind the change.
447+
Sandbox create, start, restart, and delete also take one fleet-wide lock while
448+
they provision or remove the sandbox's SSH host identity, so that step runs for
449+
one sandbox at a time across all replicas. For other
450+
mutations, a lock wait longer than 10 seconds returns `UNAVAILABLE` with the
451+
reason `MUTATION_LOCK_TIMEOUT`, and clients can retry the request. A request
452+
that carried a `request_id` leaves its admission unresolved, like any other
453+
error, so a retry with the same ID returns `REQUEST_OUTCOME_UNCERTAIN`. Observe
454+
resource state and reconcile effects, then send a new request with a new
455+
`request_id`. Refer to
456+
[Durable Request Admission](/sdk/api-errors#durable-request-admission).
457+
458+
### Upgrade from an Earlier Release
459+
460+
Each gateway pod now opens up to 14 PostgreSQL connections instead of 10. The
461+
upgrade rollout already runs new pods at that count, so raise `max_connections`
462+
to the minimum in [Size PostgreSQL Connections](#size-postgresql-connections)
463+
before you upgrade.
464+
465+
While gateways from the earlier release and the new release run together
466+
during the upgrade, the older replicas serialize every mutation across the
467+
fleet. Mutations can be slower and can briefly fail under load until the
468+
rollout finishes.
469+
421470
## Next Steps
422471

423472
- To expose the gateway through a highly available data path, refer to

‎docs/observability/gateway-metrics.mdx‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
# SPDX-License-Identifier: Apache-2.0
44
title: "Gateway Metrics"
55
sidebar-title: "Gateway Metrics"
6-
description: "Scrape Prometheus metrics from the OpenShell gateway, including supervisor session, relay capacity, and peer routing signals for multi-replica deployments."
6+
description: "Scrape Prometheus metrics from the OpenShell gateway, including supervisor session, relay capacity, peer routing, and mutation lock signals for multi-replica deployments."
77
keywords: "Generative AI, Cybersecurity, Observability, Metrics, Prometheus, Gateway, Kubernetes, High Availability"
88
---
99

@@ -175,6 +175,13 @@ that owns a sandbox's supervisor session:
175175
| `openshell_server_routed_request_attempts_total` | Counter | `operation`, `route`, `relay_kind`, `outcome`, `grpc_code` | Local relay setup and outbound peer attempts, counted once when they finish or are cancelled. Each retry counts separately. |
176176
| `openshell_server_peer_request_duration_seconds` | Histogram | `operation`, `outcome` | Latency of outbound peer requests only (`route="peer"`). For relays, until the owner's supervisor claimed the relay. |
177177

178+
Mutation locks:
179+
180+
| Metric | Type | Labels | Description |
181+
|---|---|---|---|
182+
| `openshell_server_mutation_lock_wait_seconds` | Histogram | `scope` | Time to acquire a mutation lock, recorded for successful acquisitions. |
183+
| `openshell_server_mutation_lock_timeouts_total` | Counter | `scope` | Acquisitions that waited longer than 10 seconds, including background work such as supervisor disconnect handling and provider credential refresh. API requests fail with `UNAVAILABLE` and reason `MUTATION_LOCK_TIMEOUT`, and [API Errors](/sdk/api-errors) describes how to retry them. A lock connection that PostgreSQL does not open, although at least a second of the wait remained, is not counted. |
184+
178185
The labels take these values:
179186

180187
- `operation` is `relay`, `report_provider_readiness`, `report_endpoint_status`,
@@ -210,6 +217,7 @@ The labels take these values:
210217
unchanged. The label is not named `code` because
211218
`openshell_server_grpc_requests_total` uses that name for the numeric status
212219
code.
220+
- `scope` is `global`, `workspace`, or `sandbox`.
213221

214222
Gauges and counters exist from startup. They start at `0`, except
215223
`openshell_server_relay_pending_capacity`, which starts at its limit. `openshell_server_routed_request_attempts_total` starts with seven

0 commit comments

Comments
 (0)