Repository navigation
ateapi: fail /readyz while the store is unreachable - #1989
Dmitry Berkovich (dberkov) merged 7 commits into
Conversation
Every RPC needs the store, so a replica that cannot reach it should leave the Service. Readiness gains recoverable checks next to the drain latch. Liveness is unchanged.
|
Julian Gutierrez Oschmann (@juli4n) Bowei Du (@bowei) PTAL when you get a chance. Thank you! |
|
Jeff Luo (@JeffLuoo) Krisztian F (@krisztianfekete) PTAL when you get a chance. Thank you! |
|
Merged main and resolved the serverboot.go conflict. PTAL. |
|
Michelle Au (@msau42) Dmitry Berkovich (@dberkov) PTAL when you get a chance. Thank you! |
| // Every RPC needs the store, so a replica that cannot reach it leaves the | ||
| // Service until it can. Liveness stays green: the pool reconnects on its own. | ||
| readiness := &serverboot.Readiness{} | ||
| readiness.AddCheck("store", pool.Ping) |
There was a problem hiding this comment.
🤖 should-fix 🟡 – pgxpool.Pool.Ping acquires a connection before it pings, and this is the shared read/write pool that every RPC and OpenFGA also draw from. When the pool is saturated, Acquire blocks past the 500 ms budget and /readyz answers 503, so the probe cannot tell a busy replica from an unreachable store.
Why this matters: three slow probes in a row (6 s at periodSeconds: 2) take a healthy replica out of the Service during a load spike. Its traffic lands on the remaining replicas, which are now more likely to saturate their own pools and fail their own probes. That is a readiness cascade, and it fires exactly when losing capacity hurts most. Today a busy replica keeps serving; after this change it can be evicted.
Fix: give the probe a connection that request traffic can never borrow. atepg.Connect already opens the watch and owner pools alongside the main one for the same isolation reason, so the smallest change is a fourth pool with MaxConns = 1 and MinConns = 0 built from the read/write DSN, closed in Persistence.Close with the others. Expose it as func (p *Persistence) Ping(ctx) error that calls Ping on that pool, and register persistence.Ping here instead of pool.Ping. Acquiring from a pool nothing else uses is instant, so the check fails only when PostgreSQL itself does not answer. One idle connection per replica is the whole cost.
There was a problem hiding this comment.
Done in 20f52b0: the probe uses its own one-connection pool.
|
Dmitry Berkovich (@dberkov) Addressed, PTAL. |
Fixes #1395
/readyznow fails while the store is unreachable, so the replica leaves the Service./healthzis unchanged.