Skip to content

Commit 71b9c52

Browse files
committed
docs: document scoped mutation locks and lock-pool sizing
In the High Availability guide, size PostgreSQL for 14 connections per gateway pod, 10 for data and 4 for the new lock pool, so three or more replicas now exceed what the PostgreSQL default leaves, and keep headroom for lock sessions that cancelled requests leave behind. Explain which changes lock a sandbox, a workspace, or the whole fleet, and that a lock wait over 10 seconds returns a retryable UNAVAILABLE with reason MUTATION_LOCK_TIMEOUT. A new upgrade subsection says to raise max_connections to the 14-per-pod size before upgrading, because the rollout already runs new pods at that count, and warns that older replicas keep serializing every mutation until the rollout finishes. Monitor Capacity gains the lock metrics and their queries, and names slow driver, credential, middleware, or profile source calls as another cause of lock waits. List the lock wait histogram and timeout counter, with their scope label, on the Gateway Metrics page. The counter also counts background work and leaves out lock connections that PostgreSQL does not open although at least a second of the wait remained. In the cluster debugging skill, replace the single advisory-lock paragraph with the lock levels and pool, what each timeout detail means, how a lock connection that PostgreSQL does not open shows up, when terminating a lock backend is safe, a pg_locks query that shows holders and waiters, and how to spot the global key. The per-pod greps also read the timeout counter, and new rows cover lock timeouts and lock connections that PostgreSQL does not open. Part of #3528 Signed-off-by: Emilien Macchi <[email protected]>
1 parent b4884ee commit 71b9c52

3 files changed

Lines changed: 167 additions & 19 deletions

File tree

‎docs/kubernetes/high-availability.mdx‎

Lines changed: 55 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -204,6 +204,7 @@ configuration, and access control.
204204
| Relay rejections | `openshell_server_relay_rejected_total` | Alert on any increase. |
205205
| Relay claims | `openshell_server_relay_claim_duration_seconds`, `openshell_server_relay_expired_total` | Supervisor connect-back time for exec, SSH, forwarding, and service traffic, including relays requested through peers. A high 99th percentile while rejections stay flat points at the supervisor or its node, not at relay capacity. Relays not claimed within 10 seconds are missing from the histogram and count as expired, so alert on any increase. |
206206
| Routed requests | `openshell_server_routed_request_attempts_total`, `openshell_server_peer_request_duration_seconds` | Local/peer relay setup mix and outbound peer rate, failures, and latency. Counts completed or cancelled attempts, including retries. Spikes of `grpc_code="unavailable"` during rollouts are expected. Do not use as an HPA target. |
207+
| Lock contention | `openshell_server_mutation_lock_wait_seconds`, `openshell_server_mutation_lock_timeouts_total` | Alert on any timeout. |
207208

208209
These queries assume that Prometheus labels each series with `namespace` and
209210
`pod`:
@@ -233,6 +234,12 @@ sum by (operation) (rate(openshell_server_routed_request_attempts_total{route="p
233234
234235
# 99th percentile peer request latency across the fleet.
235236
histogram_quantile(0.99, sum by (le, operation) (rate(openshell_server_peer_request_duration_seconds_bucket{outcome="success"}[5m])))
237+
238+
# 99th percentile mutation lock wait by scope. Watch scope="sandbox" during rollouts.
239+
histogram_quantile(0.99, sum by (le, scope) (rate(openshell_server_mutation_lock_wait_seconds_bucket[5m])))
240+
241+
# Mutation lock timeouts by scope. Alert on any increase.
242+
sum by (scope) (increase(openshell_server_mutation_lock_timeouts_total[10m]))
236243
```
237244

238245
Relay capacity is used on the replica that owns the sandbox's supervisor
@@ -242,6 +249,16 @@ stays counted for up to about 40 seconds, the 10-second claim timeout plus the
242249
30-second cleanup interval. Routed request counts include retries during
243250
rollouts.
244251

252+
Each supervisor that reconnects takes one or two short sandbox-scoped locks on
253+
the replica that receives it. When a gateway pod stops during a rollout, its
254+
supervisors reconnect to the other replicas, so watch
255+
`openshell_server_mutation_lock_wait_seconds` and
256+
`openshell_server_mutation_lock_timeouts_total` with `scope="sandbox"` on the
257+
receiving replicas. Rising waits or any timeout mean that those replicas
258+
receive reconnects faster than they can absorb them, or that slow compute
259+
driver, credential backend, middleware, or provider profile source calls during
260+
sandbox and provider changes hold their locks longer.
261+
245262
## Scale the Gateway
246263

247264
Scale the gateway by changing a fixed replica count or by letting a
@@ -374,7 +391,9 @@ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/openshell/pods
374391

375392
### Size PostgreSQL Connections
376393

377-
Each gateway pod opens up to 10 PostgreSQL connections on demand. The chart
394+
Each gateway pod opens up to 14 PostgreSQL connections on demand, 10 for data
395+
access and 4 for mutation locks. A request cancelled during a lock wait can
396+
leave its lock session open for up to 10 seconds, so keep headroom. The chart
378397
uses the Kubernetes default rolling update strategy for each workload kind, so
379398
the number of pods that run during a rollout depends on the kind.
380399

@@ -384,18 +403,18 @@ and each replaced pod can keep its connections throughout its termination
384403
grace period. A rollout can therefore run up to twice the replica count at
385404
once, and an eviction or an autoscaler scale-down during the rollout adds more
386405
terminating pods. Set PostgreSQL `max_connections` to at least
387-
`(2 × replicas + surge) × 10`, which leaves one surge of margin for those
406+
`(2 × replicas + surge) × 14`, which leaves one surge of margin for those
388407
pods, plus headroom for your other clients and administration. Use
389408
`autoscaling.maxReplicas` as the replica count when autoscaling is enabled.
390409

391410
| Replicas | Surge | Minimum `max_connections` |
392411
|---|---|---|
393-
| 2 | 1 | `(4 + 1) × 10 = 50` |
394-
| 3 | 1 | `(6 + 1) × 10 = 70` |
395-
| 4 | 1 | `(8 + 1) × 10 = 90` |
396-
| 6 | 2 | `(12 + 2) × 10 = 140` |
412+
| 2 | 1 | `(4 + 1) × 14 = 70` |
413+
| 3 | 1 | `(6 + 1) × 14 = 98` |
414+
| 4 | 1 | `(8 + 1) × 14 = 126` |
415+
| 6 | 2 | `(12 + 2) × 14 = 196` |
397416

398-
With a Deployment, five or more replicas exceed what the PostgreSQL default of
417+
With a Deployment, three or more replicas exceed what the PostgreSQL default of
399418
100 leaves for the gateway, because PostgreSQL reserves 3 connections for
400419
superusers. Raise `max_connections` or use a larger managed instance. Let one
401420
rollout finish before you start another, because each overlapping rollout adds
@@ -404,14 +423,42 @@ its own terminating pods.
404423
If you run several replicas as a StatefulSet with
405424
`workload.allowMultiReplicaStatefulSet`, a rollout adds no surge pods. It
406425
replaces one pod at a time and creates each replacement only after the old pod
407-
exits. Set `max_connections` to at least `replicas × 10`, using
426+
exits. Set `max_connections` to at least `replicas × 14`, using
408427
`autoscaling.maxReplicas` when autoscaling is enabled, plus headroom for your
409428
other clients and administration.
410429

411430
If a connection pooler sits between the gateway and PostgreSQL, use session
412431
pooling. The gateway holds session-level advisory locks, which transaction
413432
pooling breaks.
414433

434+
Mutations lock only what they change. Sandbox operations lock their own
435+
sandbox, provider and workspace-profile changes lock their workspace, and
436+
gateway-global policy, settings, and platform-profile changes lock the whole
437+
fleet. Sandbox start, stop, and delete lock only on the replica that runs them,
438+
and only a gateway-global change on that replica blocks them directly.
439+
Provider, workspace-profile, and other replicas' gateway-global changes delay
440+
them only while another operation on the same sandbox waits behind the change.
441+
For other
442+
mutations, a lock wait longer than 10 seconds returns `UNAVAILABLE` with the
443+
reason `MUTATION_LOCK_TIMEOUT`, and clients can retry the request. A request
444+
that carried a `request_id` leaves its admission unresolved, like any other
445+
error, so a retry with the same ID returns `REQUEST_OUTCOME_UNCERTAIN`. Observe
446+
resource state and reconcile effects, then send a new request with a new
447+
`request_id`. Refer to
448+
[Durable Request Admission](/sdk/api-errors#durable-request-admission).
449+
450+
### Upgrade from an Earlier Release
451+
452+
Each gateway pod now opens up to 14 PostgreSQL connections instead of 10. The
453+
upgrade rollout already runs new pods at that count, so raise `max_connections`
454+
to the minimum in [Size PostgreSQL Connections](#size-postgresql-connections)
455+
before you upgrade.
456+
457+
While gateways from the earlier release and the new release run together
458+
during the upgrade, the older replicas serialize every mutation across the
459+
fleet. Mutations can be slower and can briefly fail under load until the
460+
rollout finishes.
461+
415462
## Next Steps
416463

417464
- To expose the gateway through a highly available data path, refer to

‎docs/observability/gateway-metrics.mdx‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
# SPDX-License-Identifier: Apache-2.0
44
title: "Gateway Metrics"
55
sidebar-title: "Gateway Metrics"
6-
description: "Scrape Prometheus metrics from the OpenShell gateway, including supervisor session, relay capacity, and peer routing signals for multi-replica deployments."
6+
description: "Scrape Prometheus metrics from the OpenShell gateway, including supervisor session, relay capacity, peer routing, and mutation lock signals for multi-replica deployments."
77
keywords: "Generative AI, Cybersecurity, Observability, Metrics, Prometheus, Gateway, Kubernetes, High Availability"
88
---
99

@@ -175,6 +175,13 @@ that owns a sandbox's supervisor session:
175175
| `openshell_server_routed_request_attempts_total` | Counter | `operation`, `route`, `relay_kind`, `outcome`, `grpc_code` | Local relay setup and outbound peer attempts, counted once when they finish or are cancelled. Each retry counts separately. |
176176
| `openshell_server_peer_request_duration_seconds` | Histogram | `operation`, `outcome` | Latency of outbound peer requests only (`route="peer"`). For relays, until the owner's supervisor claimed the relay. |
177177

178+
Mutation locks:
179+
180+
| Metric | Type | Labels | Description |
181+
|---|---|---|---|
182+
| `openshell_server_mutation_lock_wait_seconds` | Histogram | `scope` | Time to acquire a mutation lock, recorded for successful acquisitions. |
183+
| `openshell_server_mutation_lock_timeouts_total` | Counter | `scope` | Acquisitions that waited longer than 10 seconds, including background work such as supervisor disconnect handling and provider credential refresh. API requests fail with `UNAVAILABLE` and reason `MUTATION_LOCK_TIMEOUT`, and [API Errors](/sdk/api-errors) describes how to retry them. A lock connection that PostgreSQL does not open, although at least a second of the wait remained, is not counted. |
184+
178185
The labels take these values:
179186

180187
- `operation` is `relay`, `report_provider_readiness`, `report_endpoint_status`,
@@ -210,6 +217,7 @@ The labels take these values:
210217
unchanged. The label is not named `code` because
211218
`openshell_server_grpc_requests_total` uses that name for the numeric status
212219
code.
220+
- `scope` is `global`, `workspace`, or `sandbox`.
213221

214222
Gauges and counters exist from startup. They start at `0`, except
215223
`openshell_server_relay_pending_capacity`, which starts at its limit. `openshell_server_routed_request_attempts_total` starts with seven

‎skills/debug-openshell-cluster/SKILL.md‎

Lines changed: 103 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -470,18 +470,109 @@ kubectl -n <namespace> get deployment,service,pod -l app.kubernetes.io/name=<pos
470470
kubectl -n <namespace> logs deployment/<postgres-workload> --tail=200
471471
```
472472

473-
Multi-replica gateways serialize cross-object sandbox and provider mutations
474-
with a PostgreSQL advisory lock. If those RPCs stall while ordinary reads and
475-
health checks remain responsive, inspect long-running database sessions and
476-
advisory-lock waiters. Do not print the database URI or Secret contents into
477-
logs:
473+
Gateways that use an external PostgreSQL database guard cross-object
474+
mutations with PostgreSQL advisory locks at three levels: one global key, one
475+
key per workspace, and one key per sandbox, each taken shared or exclusive.
476+
Sandbox operations on different sandboxes do not wait on each other's locks;
477+
provider and workspace-profile changes block guarded sandbox mutations in
478+
their workspace; gateway-global policy and settings changes block every
479+
guarded mutation. Guarded sandbox mutations are sandbox create, provider
480+
attach and detach, sandbox-scoped policy and settings changes, and supervisor
481+
configuration, policy status, and endpoint status reports. Lifecycle paths
482+
(start, stop, delete, restart, supervisor connection and process-exit state,
483+
driver-watch updates, and reconcile) take only a process-local shared global
484+
and exclusive sandbox lock, with no time limit and no PostgreSQL lock, so only
485+
a global change on the same replica blocks them directly. Provider,
486+
workspace-profile, and other replicas' global changes delay them only while
487+
another operation on the same sandbox waits behind the change, holding that
488+
sandbox's key. Provisioning-deadline expiry also takes
489+
its workspace key locally, so provider and workspace-profile changes on the
490+
same replica can delay it. Each
491+
replica takes its advisory locks on a dedicated pool of 4 PostgreSQL
492+
connections, one per guard, so at most 4 guarded operations per replica hold
493+
or wait for PostgreSQL locks at once. A request cancelled during its lock wait
494+
frees its slot while its backend keeps waiting, with any keys it took, until
495+
the 10-second deadline, so `pg_locks` can briefly show more lock sessions from
496+
one pod. A guard that waits on a PostgreSQL lock
497+
keeps its connection while it waits, so contention on one key can fill the
498+
pool and make unrelated guarded operations on that replica queue for a
499+
connection, within the same 10-second limit.
500+
501+
A lock wait longer than 10 seconds returns `UNAVAILABLE` with reason
502+
`MUTATION_LOCK_TIMEOUT` and the message "... timed out waiting for a
503+
concurrent mutation; retry the request", logs
504+
`mutation lock acquisition timed out` with `scope`, `waited_ms`, and `detail`
505+
fields, and increments `openshell_server_mutation_lock_timeouts_total`. The
506+
`detail` field says where the wait stopped:
507+
508+
- `waiting for a local mutation lock`: another operation on the same replica
509+
holds a conflicting key.
510+
- `waiting for a mutation lock connection`: all 4 lock-pool connections of that
511+
replica were in use, by waiters or by holders slowed by a compute driver,
512+
credential backend, middleware, or provider profile source call, or a local
513+
wait left less than a second to open one. `pg_locks` shows no row for the
514+
timed-out operation. Look for the pod's granted or waiting advisory-lock
515+
rows by `client_addr`, and find the holder they wait on.
516+
- `waiting for a PostgreSQL advisory lock`, or PostgreSQL's own
517+
`canceling statement due to lock timeout`: a conflicting key is held on
518+
another connection, usually by another replica or by an older gateway during
519+
an upgrade. `pg_locks` shows the holder and the waiters.
520+
521+
A lock connection that PostgreSQL does not open, although at least a second of
522+
the 10-second limit remained, is not a lock timeout. The request fails with
523+
`INTERNAL` and "could not open a mutation lock connection", and the gateway
524+
logs `mutation lock acquisition failed` without incrementing the timeout
525+
counter. PostgreSQL refused the connection, ran out of connection slots, was
526+
starting up, or did not answer. Check the PostgreSQL logs for "too many
527+
clients already", "remaining connection slots are reserved", or "the database
528+
system is starting up", and compare
529+
`SELECT count(*) FROM pg_stat_activity` with `SHOW max_connections`.
530+
531+
A gateway log line `timed out returning PostgreSQL mutation lock connection;
532+
discarded connection` means returning a lock session stalled for 5 seconds and
533+
the gateway dropped it; that slot stays busy for up to those 5 seconds. The
534+
warning does not identify the PostgreSQL backend. Healthy guards also hold
535+
idle advisory-lock sessions while validation and writes use separate data
536+
connections. Neither an idle duration nor a matching `client_addr` proves
537+
that a holder is orphaned, even when it blocks a timed-out request.
538+
539+
Before using `SELECT pg_terminate_backend(<pid>)`, conclusively map that
540+
backend to its owning gateway process and confirm that process has stopped
541+
or can no longer write. If the owner is still running, stop it first and
542+
verify its exit; terminating or failing readiness alone is insufficient. Killing
543+
its lock session while it can still write removes exclusion from an active
544+
mutation. Recheck the backend PID and `backend_start` before terminating the
545+
confirmed orphan: pod IPs and PIDs can be reused, and a database proxy can
546+
hide several gateways behind one `client_addr`. If ownership cannot be
547+
established, investigate connectivity instead of choosing a backend by IP or
548+
idle state. Setting PostgreSQL
549+
`tcp_keepalives_idle`, `tcp_keepalives_interval`, and `tcp_keepalives_count`
550+
(for example 60, 10, and 6) bounds how long such sessions survive.
551+
552+
If mutations stall or time out while reads and health checks work, inspect
553+
advisory-lock holders and waiters. Do not print the database URI or Secret
554+
contents into logs:
478555

479556
```sql
480-
SELECT pid, granted, waitstart
481-
FROM pg_locks
482-
WHERE locktype = 'advisory';
557+
SELECT l.pid, l.mode, l.granted, l.waitstart, l.classid, l.objid,
558+
a.client_addr, a.backend_start, a.state,
559+
now() - a.state_change AS in_state_for
560+
FROM pg_locks l
561+
JOIN pg_stat_activity a USING (pid)
562+
WHERE l.locktype = 'advisory'
563+
ORDER BY l.granted, l.waitstart;
483564
```
484565

566+
Gateways hold these locks at session level outside any transaction, so a
567+
holder (`granted = t`) usually shows `state = idle`. It still holds the lock,
568+
and `in_state_for` approximates how long. `client_addr` is the holding
569+
gateway pod's IP, or the pooler's IP when a connection pooler sits in between.
570+
571+
The global key appears as `classid = 1330660686` and `objid = 1397247052`
572+
(key `0x4F50454E53484C4C`). Older gateways during a rolling upgrade take that
573+
key exclusively for every guarded mutation, so mutations can queue behind
574+
them until the rollout finishes.
575+
485576
For multi-replica gateway installs, supervisor and client session traffic may
486577
be served by a non-owner gateway replica and relayed to the current supervisor
487578
owner over the internal `PeerRelay` RPC. Check the headless peer Service,
@@ -527,7 +618,7 @@ for pod in $(kubectl -n openshell get pod \
527618
-o jsonpath='{range .items[?(@.spec.containers[0].name=="openshell-gateway")]}{.metadata.name}{" "}{end}'); do
528619
echo "${pod}"
529620
kubectl get --raw "/api/v1/namespaces/openshell/pods/${pod}:9090/proxy/metrics" \
530-
| grep -E '^openshell_server_(supervisor_sessions|relay_pending|relay_rejected_total|routed_request_attempts_total)'
621+
| grep -E '^openshell_server_(supervisor_sessions|relay_pending|relay_rejected_total|routed_request_attempts_total|mutation_lock_timeouts_total)'
531622
done
532623
kubectl -n openshell get hpa
533624
kubectl -n openshell describe hpa openshell
@@ -549,7 +640,7 @@ kubectl -n openshell port-forward pod/<gateway-pod> 9090:9090 >/dev/null &
549640
pf_pid=$!
550641
sleep 2
551642
curl -s http://localhost:9090/metrics \
552-
| grep -E '^openshell_server_(supervisor_sessions|relay_pending|relay_rejected_total|routed_request_attempts_total)'
643+
| grep -E '^openshell_server_(supervisor_sessions|relay_pending|relay_rejected_total|routed_request_attempts_total|mutation_lock_timeouts_total)'
553644
kill "${pf_pid}"
554645
```
555646

@@ -1004,6 +1095,8 @@ credential failures.
10041095
| Kubernetes gateway pod pending | PVC unbound, taint, selector, or insufficient resources | `kubectl -n openshell describe pod <pod>` |
10051096
| Kubernetes sandbox pod stuck pending, workspace PVC unbound | Cluster has no default `StorageClass` and OpenShell does not set `storageClassName` on the workspace PVC (clusters with a default `StorageClass` bind fine without it) | `kubectl -n openshell describe pvc`; set `server.workspaceStorageClass` (gateway config `workspace_storage_class`) to a valid `StorageClass` |
10061097
| Kubernetes gateway pod crash loops | Missing secret, bad DB URL, bad TLS config | `kubectl -n openshell logs deployment/openshell -c openshell-gateway` or `kubectl -n openshell logs statefulset/openshell -c openshell-gateway` |
1098+
| Mutating RPCs return `UNAVAILABLE` with "timed out waiting for a concurrent mutation" | Advisory-lock contention, a full 4-connection lock pool on one replica, a slow PostgreSQL, or older gateways still running during an upgrade | `openshell_server_mutation_lock_timeouts_total`, the `detail` field of `mutation lock acquisition timed out` in gateway logs, the `pg_locks` query in Step 6 |
1099+
| Mutating RPCs return `INTERNAL` with "could not open a mutation lock connection" | PostgreSQL out of connection slots, restarting, or unreachable | PostgreSQL logs, `SELECT count(*) FROM pg_stat_activity` against `max_connections`, the connection sizing in the High Availability guide |
10071100
| `helm upgrade` fails with an `autoscaling.*` message | HPA values invalid: missing `resources.requests` (or `resources.limits`), `maxReplicas` above 1 without `server.externalDbSecret` (or on a StatefulSet without `workload.allowMultiReplicaStatefulSet`), no metric target, or min/max out of order. "`minReplicas` and `maxReplicas` are not set" means `--reuse-values` kept a release without the chart's autoscaling defaults | Fix the values named in the error; upgrade with `--reset-then-reuse-values` instead of `--reuse-values` |
10081101
| HPA shows `<unknown>` targets | No metrics-server for CPU/memory, or the metrics adapter does not serve the custom metric | `kubectl -n openshell describe hpa openshell`, `kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1` |
10091102
| One replica holds most sessions after a rollout | Expected: sessions stay where they reconnected | `openshell_server_supervisor_sessions` per pod; it fades as sandboxes are recreated |

0 commit comments

Comments
 (0)