You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit c6340a4
Browse filesBrowse the repository at this point in the historyBrowse files
fix(deletion): give the scheduled drain the cold-start DB connector (#8146)
## Summary
The scheduled `buzz-admin deletions drain` Job never gets past its first
database connection in a hosted staging deployment. Every run exits 5
after exactly 3 seconds with `pool timed out while waiting for an open
connection`.
The drain starts in a fresh pod with no mesh sidecar, and that pod's
first connection can take longer than the relay's 3-second acquisition
budget. `connect_store()` in `buzz-deletion` inherits that budget from
`DbConfig::default()`, with no retry. In the same pod, the same command
run 15 seconds later succeeds. TCP to Postgres, Redis, and S3 from a
sidecar-less pod is fine.
#7770 fixed the same failure for `buzz-admin storage-snapshot`, but only
inside `buzz-admin`. This PR moves that connector into `buzz-db` as
`Db::connect_cold_start` and uses it from both run-once workers:
- **Pool settings:** a 30s acquisition budget, and no idle spare
connections (`min_connections: 0`).
- **Retries:** startup timeouts and transient transport errors retry at
most twice, after 2s and 5s, so startup waits at most 97s.
- **No retry:** configuration, authentication, TLS, and protocol errors
fail immediately, and the error message stays free of credentials.
- **Per-worker settings:** each worker keeps its own pool size.
Storage-snapshot stays at 1 connection, and deletion stays at
`BUZZ_DB_POOL_SIZE` (default 20). Each worker also keeps its session
timeouts.
- **Startup events:** the stderr events carry a worker prefix.
Storage-snapshot still emits `storage_snapshot_db_connect_*`, and the
deletion worker now emits `deletion_db_connect_*`.
- **Error text:** the exhaustion message is now `{worker} database
startup failed after …`. For storage-snapshot that reads
`storage_snapshot database startup failed …` instead of `storage
snapshot database startup failed …`. Nothing in this repo matches on
that string.
- **Session-setup failures:** SQLx retries a failed `after_connect` hook
(writer floor guard, session timeouts, isolation assertion) until the
acquire deadline. Those failures therefore surface as `timeout`, are
retried, and fail closed after at most 97s instead of immediately.
Relay pool defaults are unchanged. The deletion CLI's operator commands
(`list`, `approve`, …) use the same connector. They normally run inside
the warm relay container, where the first attempt succeeds.
**What got simpler:** there is now one cold-start connector instead of a
`buzz-admin`-private copy that the deletion worker couldn't reach.
`storage_snapshot_startup.rs` is gone, and its unit tests moved with the
code. The two Postgres startup tests now share one harness with the
drain's tests (`postgres_storage_snapshot.rs` →
`postgres_cold_start.rs`).
### Related issue
Follow-up to #7770 (same failure, other worker) and #7827 (the drain
CronJob). No duplicate found.
### Testing
All at head `1545de7b5`:
- **GitHub CI:** green on attempt 1, including `PostgreSQL Domain /
PostgreSQL Tests`.
- **Blox:**
- fmt, `clippy --workspace --all-targets -D warnings`, and the `buzz-db`
/ `buzz-admin` / `buzz-deletion` tests pass.
- All four `postgres_cold_start` tests pass, each against a freshly
migrated database.
- Falsification: with main's `connect_store` restored, the drain retry
test fails.
- **Agent review:** two rounds. All findings are fixed or disclosed; the
error-text change is listed above.
- **Agent exercise:** real `buzz-admin deletions list` / `deletions
drain` against local Postgres 17 behind a fault proxy.
- **5s delay:** succeeds on attempt 1 (3/3). Main fails at about 3.5s.
- **Dropped handshake, then 5s delay:** attempt 1 fails with
`UnexpectedEof`, then a 2s backoff, then attempt 2 succeeds (3/3).
- **Wrong password, missing database or malformed URL:** exit 5 after
one attempt.
- **Persistent rejection:** exit 5 after three attempts, with 2s and 5s
backoffs.
- The deletion queue stayed empty throughout.
- **Not covered:** an approved deletion, real S3 / Pod Identity, TLS and
session-setup failures, and a live 97s stall.
---------
Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
0 commit comments