You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 71b9c52
Browse filesBrowse the repository at this point in the historyBrowse files
docs: document scoped mutation locks and lock-pool sizing
In the High Availability guide, size PostgreSQL for 14 connections per
gateway pod, 10 for data and 4 for the new lock pool, so three or more
replicas now exceed what the PostgreSQL default leaves, and keep
headroom for lock sessions that cancelled requests leave behind.
Explain which changes lock a sandbox, a workspace, or the whole fleet,
and that a lock wait over 10 seconds returns a retryable UNAVAILABLE
with reason MUTATION_LOCK_TIMEOUT. A new upgrade subsection says to
raise max_connections to the 14-per-pod size before upgrading, because
the rollout already runs new pods at that count, and warns that older
replicas keep serializing every mutation until the rollout finishes.
Monitor Capacity gains the lock metrics and their queries, and names
slow driver, credential, middleware, or profile source calls as another
cause of lock waits.
List the lock wait histogram and timeout counter, with their scope
label, on the Gateway Metrics page. The counter also counts background
work and leaves out lock connections that PostgreSQL does not open
although at least a second of the wait remained.
In the cluster debugging skill, replace the single advisory-lock
paragraph with the lock levels and pool, what each timeout detail
means, how a lock connection that PostgreSQL does not open shows up,
when terminating a lock backend is safe, a pg_locks query that shows
holders and waiters, and how to spot the global key. The per-pod greps
also read the timeout counter, and new rows cover lock timeouts and
lock connections that PostgreSQL does not open.
Part of #3528
Signed-off-by: Emilien Macchi <[email protected]>
Copy file name to clipboardExpand all lines: docs/kubernetes/high-availability.mdx
+55-8Lines changed: 55 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -204,6 +204,7 @@ configuration, and access control.
204
204
| Relay rejections | `openshell_server_relay_rejected_total` | Alert on any increase. |
205
205
| Relay claims | `openshell_server_relay_claim_duration_seconds`, `openshell_server_relay_expired_total` | Supervisor connect-back time for exec, SSH, forwarding, and service traffic, including relays requested through peers. A high 99th percentile while rejections stay flat points at the supervisor or its node, not at relay capacity. Relays not claimed within 10 seconds are missing from the histogram and count as expired, so alert on any increase. |
206
206
| Routed requests | `openshell_server_routed_request_attempts_total`, `openshell_server_peer_request_duration_seconds` | Local/peer relay setup mix and outbound peer rate, failures, and latency. Counts completed or cancelled attempts, including retries. Spikes of `grpc_code="unavailable"` during rollouts are expected. Do not use as an HPA target. |
207
+
| Lock contention | `openshell_server_mutation_lock_wait_seconds`, `openshell_server_mutation_lock_timeouts_total` | Alert on any timeout. |
207
208
208
209
These queries assume that Prometheus labels each series with `namespace` and
209
210
`pod`:
@@ -233,6 +234,12 @@ sum by (operation) (rate(openshell_server_routed_request_attempts_total{route="p
233
234
234
235
# 99th percentile peer request latency across the fleet.
235
236
histogram_quantile(0.99, sum by (le, operation) (rate(openshell_server_peer_request_duration_seconds_bucket{outcome="success"}[5m])))
237
+
238
+
# 99th percentile mutation lock wait by scope. Watch scope="sandbox" during rollouts.
239
+
histogram_quantile(0.99, sum by (le, scope) (rate(openshell_server_mutation_lock_wait_seconds_bucket[5m])))
240
+
241
+
# Mutation lock timeouts by scope. Alert on any increase.
242
+
sum by (scope) (increase(openshell_server_mutation_lock_timeouts_total[10m]))
236
243
```
237
244
238
245
Relay capacity is used on the replica that owns the sandbox's supervisor
@@ -242,6 +249,16 @@ stays counted for up to about 40 seconds, the 10-second claim timeout plus the
242
249
30-second cleanup interval. Routed request counts include retries during
243
250
rollouts.
244
251
252
+
Each supervisor that reconnects takes one or two short sandbox-scoped locks on
253
+
the replica that receives it. When a gateway pod stops during a rollout, its
254
+
supervisors reconnect to the other replicas, so watch
255
+
`openshell_server_mutation_lock_wait_seconds`and
256
+
`openshell_server_mutation_lock_timeouts_total`with `scope="sandbox"` on the
257
+
receiving replicas. Rising waits or any timeout mean that those replicas
258
+
receive reconnects faster than they can absorb them, or that slow compute
259
+
driver, credential backend, middleware, or provider profile source calls during
260
+
sandbox and provider changes hold their locks longer.
261
+
245
262
## Scale the Gateway
246
263
247
264
Scale the gateway by changing a fixed replica count or by letting a
@@ -374,7 +391,9 @@ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/openshell/pods
374
391
375
392
### Size PostgreSQL Connections
376
393
377
-
Each gateway pod opens up to 10 PostgreSQL connections on demand. The chart
394
+
Each gateway pod opens up to 14 PostgreSQL connections on demand, 10 for data
395
+
access and 4 for mutation locks. A request cancelled during a lock wait can
396
+
leave its lock session open for up to 10 seconds, so keep headroom. The chart
378
397
uses the Kubernetes default rolling update strategy for each workload kind, so
379
398
the number of pods that run during a rollout depends on the kind.
380
399
@@ -384,18 +403,18 @@ and each replaced pod can keep its connections throughout its termination
384
403
grace period. A rollout can therefore run up to twice the replica count at
385
404
once, and an eviction or an autoscaler scale-down during the rollout adds more
386
405
terminating pods. Set PostgreSQL `max_connections` to at least
387
-
`(2 × replicas + surge) × 10`, which leaves one surge of margin for those
406
+
`(2 × replicas + surge) × 14`, which leaves one surge of margin for those
388
407
pods, plus headroom for your other clients and administration. Use
389
408
`autoscaling.maxReplicas`as the replica count when autoscaling is enabled.
390
409
391
410
| Replicas | Surge | Minimum `max_connections` |
392
411
|---|---|---|
393
-
| 2 | 1 | `(4 + 1) × 10 = 50` |
394
-
| 3 | 1 | `(6 + 1) × 10 = 70` |
395
-
| 4 | 1 | `(8 + 1) × 10 = 90` |
396
-
| 6 | 2 | `(12 + 2) × 10 = 140` |
412
+
| 2 | 1 | `(4 + 1) × 14 = 70` |
413
+
| 3 | 1 | `(6 + 1) × 14 = 98` |
414
+
| 4 | 1 | `(8 + 1) × 14 = 126` |
415
+
| 6 | 2 | `(12 + 2) × 14 = 196` |
397
416
398
-
With a Deployment, five or more replicas exceed what the PostgreSQL default of
417
+
With a Deployment, three or more replicas exceed what the PostgreSQL default of
399
418
100 leaves for the gateway, because PostgreSQL reserves 3 connections for
400
419
superusers. Raise `max_connections` or use a larger managed instance. Let one
401
420
rollout finish before you start another, because each overlapping rollout adds
@@ -404,14 +423,42 @@ its own terminating pods.
404
423
If you run several replicas as a StatefulSet with
405
424
`workload.allowMultiReplicaStatefulSet`, a rollout adds no surge pods. It
406
425
replaces one pod at a time and creates each replacement only after the old pod
407
-
exits. Set `max_connections` to at least `replicas × 10`, using
426
+
exits. Set `max_connections` to at least `replicas × 14`, using
408
427
`autoscaling.maxReplicas`when autoscaling is enabled, plus headroom for your
409
428
other clients and administration.
410
429
411
430
If a connection pooler sits between the gateway and PostgreSQL, use session
412
431
pooling. The gateway holds session-level advisory locks, which transaction
413
432
pooling breaks.
414
433
434
+
Mutations lock only what they change. Sandbox operations lock their own
435
+
sandbox, provider and workspace-profile changes lock their workspace, and
436
+
gateway-global policy, settings, and platform-profile changes lock the whole
437
+
fleet. Sandbox start, stop, and delete lock only on the replica that runs them,
438
+
and only a gateway-global change on that replica blocks them directly.
439
+
Provider, workspace-profile, and other replicas' gateway-global changes delay
440
+
them only while another operation on the same sandbox waits behind the change.
441
+
For other
442
+
mutations, a lock wait longer than 10 seconds returns `UNAVAILABLE` with the
443
+
reason `MUTATION_LOCK_TIMEOUT`, and clients can retry the request. A request
444
+
that carried a `request_id` leaves its admission unresolved, like any other
445
+
error, so a retry with the same ID returns `REQUEST_OUTCOME_UNCERTAIN`. Observe
446
+
resource state and reconcile effects, then send a new request with a new
Copy file name to clipboardExpand all lines: docs/observability/gateway-metrics.mdx
+9-1Lines changed: 9 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,7 +3,7 @@
3
3
# SPDX-License-Identifier: Apache-2.0
4
4
title: "Gateway Metrics"
5
5
sidebar-title: "Gateway Metrics"
6
-
description: "Scrape Prometheus metrics from the OpenShell gateway, including supervisor session, relay capacity, and peer routing signals for multi-replica deployments."
6
+
description: "Scrape Prometheus metrics from the OpenShell gateway, including supervisor session, relay capacity, peer routing, and mutation lock signals for multi-replica deployments."
@@ -175,6 +175,13 @@ that owns a sandbox's supervisor session:
175
175
| `openshell_server_routed_request_attempts_total` | Counter | `operation`, `route`, `relay_kind`, `outcome`, `grpc_code` | Local relay setup and outbound peer attempts, counted once when they finish or are cancelled. Each retry counts separately. |
176
176
| `openshell_server_peer_request_duration_seconds` | Histogram | `operation`, `outcome` | Latency of outbound peer requests only (`route="peer"`). For relays, until the owner's supervisor claimed the relay. |
177
177
178
+
Mutation locks:
179
+
180
+
| Metric | Type | Labels | Description |
181
+
|---|---|---|---|
182
+
| `openshell_server_mutation_lock_wait_seconds` | Histogram | `scope` | Time to acquire a mutation lock, recorded for successful acquisitions. |
183
+
| `openshell_server_mutation_lock_timeouts_total` | Counter | `scope` | Acquisitions that waited longer than 10 seconds, including background work such as supervisor disconnect handling and provider credential refresh. API requests fail with `UNAVAILABLE` and reason `MUTATION_LOCK_TIMEOUT`, and [API Errors](/sdk/api-errors) describes how to retry them. A lock connection that PostgreSQL does not open, although at least a second of the wait remained, is not counted. |
| Kubernetes gateway pod pending | PVC unbound, taint, selector, or insufficient resources |`kubectl -n openshell describe pod <pod>`|
1005
1096
| Kubernetes sandbox pod stuck pending, workspace PVC unbound | Cluster has no default `StorageClass` and OpenShell does not set `storageClassName` on the workspace PVC (clusters with a default `StorageClass` bind fine without it) |`kubectl -n openshell describe pvc`; set `server.workspaceStorageClass` (gateway config `workspace_storage_class`) to a valid `StorageClass`|
1006
1097
| Kubernetes gateway pod crash loops | Missing secret, bad DB URL, bad TLS config |`kubectl -n openshell logs deployment/openshell -c openshell-gateway` or `kubectl -n openshell logs statefulset/openshell -c openshell-gateway`|
1098
+
| Mutating RPCs return `UNAVAILABLE` with "timed out waiting for a concurrent mutation" | Advisory-lock contention, a full 4-connection lock pool on one replica, a slow PostgreSQL, or older gateways still running during an upgrade |`openshell_server_mutation_lock_timeouts_total`, the `detail` field of `mutation lock acquisition timed out` in gateway logs, the `pg_locks` query in Step 6 |
1099
+
| Mutating RPCs return `INTERNAL` with "could not open a mutation lock connection" | PostgreSQL out of connection slots, restarting, or unreachable | PostgreSQL logs, `SELECT count(*) FROM pg_stat_activity` against `max_connections`, the connection sizing in the High Availability guide |
1007
1100
|`helm upgrade` fails with an `autoscaling.*` message | HPA values invalid: missing `resources.requests` (or `resources.limits`), `maxReplicas` above 1 without `server.externalDbSecret` (or on a StatefulSet without `workload.allowMultiReplicaStatefulSet`), no metric target, or min/max out of order. "`minReplicas` and `maxReplicas` are not set" means `--reuse-values` kept a release without the chart's autoscaling defaults | Fix the values named in the error; upgrade with `--reset-then-reuse-values` instead of `--reuse-values`|
1008
1101
| HPA shows `<unknown>` targets | No metrics-server for CPU/memory, or the metrics adapter does not serve the custom metric |`kubectl -n openshell describe hpa openshell`, `kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1`|
1009
1102
| One replica holds most sessions after a rollout | Expected: sessions stay where they reconnected |`openshell_server_supervisor_sessions` per pod; it fades as sandboxes are recreated |
0 commit comments