Skip to content

ateapi: fail /readyz while the store is unreachable - #1989

Merged
Dmitry Berkovich (dberkov) merged 7 commits into
agent-substrate:mainfrom
ericcurtin:readyz-store-check
Oct 14, 2026
Merged

Dmitry Berkovich (dberkov) merged 7 commits into
agent-substrate:mainfrom
ericcurtin:readyz-store-check

Conversation

@ericcurtin

Copy link
Copy Markdown
Contributor

Fixes #1395

/readyz now fails while the store is unreachable, so the replica leaves the Service. /healthz is unchanged.

  • Tests pass
  • Appropriate changes to documentation are included in the PR

Every RPC needs the store, so a replica that cannot reach it should leave the Service. Readiness gains recoverable checks next to the drain latch. Liveness is unchanged.
@ericcurtin

Copy link
Copy Markdown
Contributor Author

Julian Gutierrez Oschmann (@juli4n) Bowei Du (@bowei) PTAL when you get a chance. Thank you!

@ericcurtin

Copy link
Copy Markdown
Contributor Author

Jeff Luo (@JeffLuoo) Krisztian F (@krisztianfekete) PTAL when you get a chance. Thank you!

@ericcurtin

Copy link
Copy Markdown
Contributor Author

Merged main and resolved the serverboot.go conflict. PTAL.

@ericcurtin

Copy link
Copy Markdown
Contributor Author

Michelle Au (@msau42) Dmitry Berkovich (@dberkov) PTAL when you get a chance. Thank you!

Comment thread cmd/ateapi/main.go Outdated
// Every RPC needs the store, so a replica that cannot reach it leaves the
// Service until it can. Liveness stays green: the pool reconnects on its own.
readiness := &serverboot.Readiness{}
readiness.AddCheck("store", pool.Ping)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 should-fix 🟡 – pgxpool.Pool.Ping acquires a connection before it pings, and this is the shared read/write pool that every RPC and OpenFGA also draw from. When the pool is saturated, Acquire blocks past the 500 ms budget and /readyz answers 503, so the probe cannot tell a busy replica from an unreachable store.

Why this matters: three slow probes in a row (6 s at periodSeconds: 2) take a healthy replica out of the Service during a load spike. Its traffic lands on the remaining replicas, which are now more likely to saturate their own pools and fail their own probes. That is a readiness cascade, and it fires exactly when losing capacity hurts most. Today a busy replica keeps serving; after this change it can be evicted.

Fix: give the probe a connection that request traffic can never borrow. atepg.Connect already opens the watch and owner pools alongside the main one for the same isolation reason, so the smallest change is a fourth pool with MaxConns = 1 and MinConns = 0 built from the read/write DSN, closed in Persistence.Close with the others. Expose it as func (p *Persistence) Ping(ctx) error that calls Ping on that pool, and register persistence.Ping here instead of pool.Ping. Acquiring from a pool nothing else uses is instant, so the check fails only when PostgreSQL itself does not answer. One idle connection per replica is the whole cost.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 20f52b0: the probe uses its own one-connection pool.

@ericcurtin

Copy link
Copy Markdown
Contributor Author

Dmitry Berkovich (@dberkov) Addressed, PTAL.

@dberkov
Dmitry Berkovich (dberkov) added this pull request to the merge queue Oct 14, 2026
Merged via the queue into agent-substrate:main with commit 34121f1 Oct 14, 2026
14 checks passed
@ericcurtin
Eric Curtin (ericcurtin) deleted the readyz-store-check branch October 14, 2026 16:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/api-machinery area/reliability kind/bug Something isn't working / bugfixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ateapi: a control plane that cannot reach its store serves 200 on /healthz and /readyz, stays Ready, and fails every RPC

3 participants