Repository navigation
Commit bd73490
authored
ci: run PostgreSQL tests in isolated lane (#6730)
## Why
PR #6660 introduced four PostgreSQL-backed persistence tests and a
focused CI selector, but an exact test-name list does not automatically
cover future database tests. Broad ignored-test execution also exposed
shared-schema races and ambiguity between desired-state and
migration-applied schema expectations.
## What
- Establish a discoverable convention: PostgreSQL unit modules use
`postgres_tests`, PostgreSQL integration binaries use a `postgres_`
prefix, and unrelated external-infrastructure tests use an
`external_infra_` prefix.
- Add a dedicated nextest PostgreSQL profile and archive-backed CI job
covering the relevant crates without enumerating test names.
- Create a run-scoped desired-state source database and a unique
PostgreSQL database per test process for parallel-safe isolation, with
cleanup on success, failure, or interruption.
- Route destructive migration and migration-parity tests to clean
`template0` databases while desired-state tests clone the desired-state
database.
- Document discovery, schema modes, required role privileges, and a
bounded portable local runner.
- Keep the existing infrastructure-free unit-test jobs unchanged.
This is intentionally limited to test and CI harness behavior. It does
not move production database code, change persistence semantics, or
implement issue #20's broader shared test-utility refactor.
## Risk
Low production risk because all changes are confined to tests, CI
configuration, documentation, and test harness scripts.
Remaining operational risks:
- Cleanup retries `dropdb --force` five times across roughly four
seconds. Exhaustion warns with the database name but deliberately does
not mask the test result; individual PostgreSQL diagnostics are
suppressed.
- The seven-package boundary is duplicated between the runner and CI
archive and must remain synchronized if PostgreSQL tests move to a new
crate.
- Portable SHA fallback branches were exercised on Linux; no macOS Blox
workstation was needed for this Linux CI artifact.
- GitHub's unchanged generic Unit Tests job currently fails while
cold-linking `buzz-voice` because `sherpa-onnx-c-api` is absent. The
branch changes no voice/build/toolchain inputs; the exact parent
previously passed that job, and Blox passes the same infrastructure-free
suite (1,450/1,450). A prior-head one-job retry and the final-head run
both reproduced the hosted-runner failure.
## Verification
Author Blox workstation `2020088`:
- `cargo fmt --all -- --check`
- `just clippy`
- `just test-unit`: 1,450 passed; PostgreSQL tests remain skipped in
fast jobs
- PR #6660 focused tests: 4/4 passed through the final runner
- Full lane: 282/282 passed in 10.087s with a non-superuser role limited
to `CREATEDB`, `CREATEROLE`, and `pg_read_all_stats`
- 272 desired-state tests
- 10 migration-applied tests
- 6 explicitly filtered external-infrastructure tests
- Cleanup fault injection: a deliberately underprivileged 282-test run
produced the expected 277 passes and 5 failures, and the post-run
catalog audit found zero lane databases after retry cleanup. The
restored successful run also left zero lane databases.
GitHub exact final head `92c231e48f299ea2af23817763f5ae80ac013d68`:
- [PostgreSQL
Tests](https://github.com/block/buzz/actions/runs/32789030203/job/97630827823):
282/282 passed across 10 binaries in 98.886s; 1,240 skipped, including 6
via the profile filter
- The shared relay/PostgreSQL archive predecessor completed successfully
- [Unit
Tests](https://github.com/block/buzz/actions/runs/32789030203/job/97626930420):
unrelated `sherpa-onnx-c-api` native-link failure described above
- Independent exact-head review on separate Blox workstation `2022762`:
no substantive findings after all initial findings were addressed. The
reviewer independently verified cleanup retry behavior, exit-status
preservation, shell portability, syntax, and a clean exact-head
worktree.
## References
- Stacked on #6660 at exact head
`561de54be4d9c2b622b7c2aa5b61bc3068f47e2a`
- TheSentinel454#20
-
TheSentinel454#20 (comment)
Generated with Codex.
## Update — August 24, 2026 review follow-up
- Centralized the repeated PostgreSQL test URL resolution in
crate-local, test-only helpers without introducing the broader shared
utility refactor from issue #20.
- Restored descriptive hybrid/Redis test function names. Structural
`external_infra_*_tests` modules now own exclusion, and the nextest
filter only recognizes module path segments.
- Added a three-second source guard that scans every Rust file and fails
CI when an ignored PostgreSQL test would be omitted or an
external-infrastructure test would be included. Fixture tests cover
accepted modules/binaries and both failure modes.
- Removed the two implementation-plan documents.
Performance profiling on Blox workstation `2027352` (same prebuilt
archive and cargo-nextest 0.9.143 for every comparison):
- Current per-test database model, 8 workers: 10.15–10.40s across three
runs. The 283 `createdb` calls and 284 `dropdb` calls consumed
28.85–29.46 aggregate seconds, about 39–40% of aggregate test-process
duration.
- Reusing one database per worker without cleanup: 6.55–6.84s, about 35%
faster, but all three runs failed because global matcher-queue tests
inherited a quiesced community from an earlier test; the failing test
varied with scheduling.
- Reusing workers with `TRUNCATE … RESTART IDENTITY CASCADE`: all 282
tests passed, but total time regressed to 10.47–10.67s. The 265 truncate
calls consumed 24.34–25.45 aggregate seconds, so truncation merely
replaced most clone/drop cost.
- Raising concurrency to 16 workers reduced one isolated-database run to
8.79s, but 24 workers exposed a cluster-global `pg_stat_activity` race.
The profile remains at the proven-safe 8 workers.
Decision: retain unique per-test databases. Harness cleanup uses `dropdb
--if-exists --force`; it drops the database rather than truncating
tables or deleting rows.
Verification at local branch head
`60086c1f6ee34a193c342255a5e8e6293b60e988`:
- Discovery inventory: exactly 282 intended tests; all 6 hybrid/Redis
tests present and structurally excluded.
- PR #6660 focused persistence tests: 4/4 passed.
- Full PostgreSQL lane: 282/282 passed three times; 1,240 skipped each
run; wall time 9.72–9.96s without profiling shims.
- Non-ignored tests with a desired-state database: 1,234/1,234 passed;
one mesh-demo timeout from the first run passed immediately in
isolation.
- `cargo fmt --all -- --check` and all-target/all-feature clippy for
`buzz-db`, `buzz-deletion`, and `buzz-relay` passed.
Final independent re-review at exact head
`42097c0136ab3fa8f0efe5720ae829c04575b9b6` on separate Blox workstation
`2028455`: no substantive residual findings. The reviewer independently
exercised ordinary strings, zero/one/three-hash raw strings, line/block
comments, and string-contained lookalikes, then reran the full 353-file
scan, shell syntax, Python AST parsing, and diff check.
Independent review found that the source guard's initial regular
expression could miss raw-string ignore reasons and treat a commented
attribute as real. Final head `42097c0136ab3fa8f0efe5720ae829c04575b9b6`
parses valid ordinary/raw Rust string literals only at attributes found
in comment-sanitized source; regression fixtures cover both cases. The
fixture suite, Python compilation, shell syntax, diff check, and full
353-file repository scan pass on Blox.
## Update — August 25, 2026 restack
PR #6660 merged, so this follow-up was rebased from its former exact
parent onto current `origin/main`
(`8d2d0ff5ad42733e9949442c4b6358d0ba87f9a8`). The final candidate head
is `05dcc2ab28948e3ab79bb44839a69f2ba44648a2`; the PR no longer carries
#6660's pre-squash history.
Fresh exact-head verification on Blox workstation `2028572`:
- Discovery guard passed across 353 Rust files; inventory remained
exactly 282 intended PostgreSQL tests.
- PR #6660 focused persistence tests: 4/4 passed.
- Full PostgreSQL lane: 282/282 passed three times; 1,243 skipped each
run; wall time 9.70–9.87s.
- `cargo fmt --all -- --check` and all-target/all-feature clippy for
`buzz-db`, `buzz-deletion`, and `buzz-relay` passed.
## Update — August 25, 2026 final hosted verification
- [GitHub PostgreSQL
Tests](https://github.com/block/buzz/actions/runs/32801679226/job/97667835080):
282/282 passed in 53.787s; 1,243 skipped.
- The final guard scanned all 353 Rust files successfully before the
hosted lane ran.
- The shared relay/archive prerequisite and infrastructure-free Unit
Tests job both passed at exact final head
`42097c0136ab3fa8f0efe5720ae829c04575b9b6`.
---------
Signed-off-by: Luke Tornquist <[email protected]>
Signed-off-by: tornquist <[email protected]>1 parent 114dbf7 commit bd73490
62 files changed
Lines changed: 1619 additions & 442 deletions
File tree
- .config
- .github/workflows
- crates
- buzz-audit/src
- buzz-db
- src
- runtime
- store
- buzz-deletion/src
- buzz-push-gateway/src
- buzz-relay/src
- api
- admin
- git
- handlers
- buzz-search/tests
- buzz-workflow/src
- scripts
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
215 | 215 | | |
216 | 216 | | |
217 | 217 | | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
218 | 276 | | |
219 | 277 | | |
220 | 278 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
269 | 269 | | |
270 | 270 | | |
271 | 271 | | |
272 | | - | |
| 272 | + | |
273 | 273 | | |
274 | 274 | | |
275 | 275 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
30 | 33 | | |
31 | 34 | | |
32 | 35 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
183 | 183 | | |
184 | 184 | | |
185 | 185 | | |
186 | | - | |
| 186 | + | |
187 | 187 | | |
188 | 188 | | |
189 | 189 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1112 | 1112 | | |
1113 | 1113 | | |
1114 | 1114 | | |
1115 | | - | |
| 1115 | + | |
| 1116 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
426 | 426 | | |
427 | 427 | | |
428 | 428 | | |
429 | | - | |
430 | | - | |
431 | 429 | | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
432 | 433 | | |
433 | 434 | | |
434 | 435 | | |
435 | 436 | | |
436 | | - | |
| 437 | + | |
437 | 438 | | |
438 | 439 | | |
439 | 440 | | |
| |||
490 | 491 | | |
491 | 492 | | |
492 | 493 | | |
493 | | - | |
494 | | - | |
495 | 494 | | |
496 | 495 | | |
497 | 496 | | |
| |||
633 | 632 | | |
634 | 633 | | |
635 | 634 | | |
| 635 | + | |
| 636 | + | |
| 637 | + | |
| 638 | + | |
| 639 | + | |
| 640 | + | |
| 641 | + | |
| 642 | + | |
| 643 | + | |
| 644 | + | |
| 645 | + | |
| 646 | + | |
| 647 | + | |
| 648 | + | |
636 | 649 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
789 | 789 | | |
790 | 790 | | |
791 | 791 | | |
792 | | - | |
| 792 | + | |
793 | 793 | | |
794 | 794 | | |
795 | | - | |
796 | | - | |
797 | | - | |
798 | | - | |
799 | | - | |
800 | | - | |
801 | 795 | | |
802 | 796 | | |
803 | 797 | | |
804 | 798 | | |
805 | | - | |
| 799 | + | |
806 | 800 | | |
807 | 801 | | |
808 | 802 | | |
809 | 803 | | |
810 | 804 | | |
811 | 805 | | |
812 | 806 | | |
813 | | - | |
| 807 | + | |
814 | 808 | | |
815 | 809 | | |
816 | 810 | | |
| |||
973 | 967 | | |
974 | 968 | | |
975 | 969 | | |
976 | | - | |
977 | | - | |
| 970 | + | |
| 971 | + | |
| 972 | + | |
| 973 | + | |
| 974 | + | |
| 975 | + | |
| 976 | + | |
| 977 | + | |
978 | 978 | | |
979 | 979 | | |
980 | | - | |
| 980 | + | |
| 981 | + | |
| 982 | + | |
981 | 983 | | |
982 | 984 | | |
983 | 985 | | |
984 | 986 | | |
985 | 987 | | |
986 | | - | |
| 988 | + | |
| 989 | + | |
| 990 | + | |
987 | 991 | | |
988 | 992 | | |
989 | 993 | | |
| |||
1016 | 1020 | | |
1017 | 1021 | | |
1018 | 1022 | | |
1019 | | - | |
1020 | | - | |
| 1023 | + | |
| 1024 | + | |
| 1025 | + | |
| 1026 | + | |
1021 | 1027 | | |
1022 | 1028 | | |
1023 | 1029 | | |
1024 | | - | |
| 1030 | + | |
| 1031 | + | |
| 1032 | + | |
1025 | 1033 | | |
1026 | 1034 | | |
1027 | 1035 | | |
| |||
1038 | 1046 | | |
1039 | 1047 | | |
1040 | 1048 | | |
1041 | | - | |
| 1049 | + | |
1042 | 1050 | | |
1043 | 1051 | | |
1044 | 1052 | | |
| |||
1079 | 1087 | | |
1080 | 1088 | | |
1081 | 1089 | | |
1082 | | - | |
| 1090 | + | |
| 1091 | + | |
| 1092 | + | |
1083 | 1093 | | |
1084 | 1094 | | |
1085 | 1095 | | |
| |||
1100 | 1110 | | |
1101 | 1111 | | |
1102 | 1112 | | |
1103 | | - | |
| 1113 | + | |
1104 | 1114 | | |
1105 | 1115 | | |
1106 | 1116 | | |
| |||
1155 | 1165 | | |
1156 | 1166 | | |
1157 | 1167 | | |
1158 | | - | |
| 1168 | + | |
1159 | 1169 | | |
1160 | 1170 | | |
1161 | 1171 | | |
| |||
0 commit comments