Skip to content

Commit 322fdbf

Browse files
matt2eclaude
andauthored
test(relay): hold the replica's second aux page and move media's DB tests (#8740)
## Summary Two buzz-relay test fixes from the 23–30 Oct flake audit. Both are test-only. This is one of six PRs split from the reviewed `flakey-tests` branch (siblings below). ### `thread_window_bridge_restarts_both_aux_hops_after_replica_failure` (3 runs) - **Runs:** 40799617319, 40924179886 and 40930100994 (#8697, docs only), all on 10-29 in PostgreSQL Tests, attempt 1 (jobs 123208043033, 123632125866, 123651641392). None of the branches touches thread_window, and each run passed on a re-run. Signature: `failure_postgres_tests.rs:46`, "barrier must precede closure completion", left: 2, right: 1. - **Cause:** `pause_after_aux_page` polls the request until a `thread_window_aux` span closes, then expects exactly one. The fixture's 1,001 reactions make two replica pages at `AUX_LIMIT` 1000, and each page is one round trip on the transaction already held. A loaded runner can deschedule the test thread after page 2's query is written. Postgres then answers first, and the same poll reads page 2 too. - **Change:** `test_support` gains a `PausingProxy`, a trimmed copy of buzz-db's private one with only the hold-everything switch. The replica reader pool connects through it. `AuxPages` pauses it when the first aux span closes. That runs inside the request's poll, after page 1's response is consumed and before page 2's query is written, so page 2 stays pending however fast Postgres answers. The test resumes the proxy after it terminates the reader's backend. The held page then sees the termination, and the closure restarts on the writer, as before. The other three `pause_after_aux_page` callers read from the writer and pass `None`. ### `api::media` tests that need Postgres (fail every time Flake Check selects them) - **Run:** 40954304982 (#8658), 10-29, Flake check (Rust), job 123730434146. All six `test_state()` tests panic at the community-seed `expect` after the 30 s sqlx acquire timeout. - **Cause:** when a hunk lands outside a test body, Flake Check selects every test in the file. It skips only `postgres_tests::`, `postgres_*` binaries and `external_infra`, and its job has no database. These six tests seed `relay.example` through Postgres but sat in plain `mod tests` without `#[ignore]`. So any PR that changes non-test code in `media.rs` fails that lane, and at least 8 open PRs touch the file. No other lane ran these tests either. - **Change:** move `test_state`, the `media_get_*` helpers, `media_request` and the six tests into `tests::postgres_tests`. Mark each test `#[ignore = "requires Postgres"]`, and point `test_state` at `test_support::database_url()`, as the module's other states do. The PostgreSQL lane now runs them, and Flake Check skips them. The change also drops imports `mod tests` no longer uses, and updates the Justfile comment that gave `api::media` as the reason the infra-free lane is scoped to `api::admin`. - When `media.rs` is staged, the pre-commit secret scan reads the whole file and flags the module's existing public ES256 test key (`TEST_EC_PKCS8_PEM`). The key is now written on one line with a sadscan marker, as `invites.rs` already does. Its bytes are unchanged. ### Related issue None found. **Conflicts with #8451 and #8444**, which both already conflict with `main`. The one-line `TEST_EC_PKCS8_PEM` rewrite is the only new conflict hunk with each. The test move merges cleanly with both. ### Testing Verified locally: - Each of the four `pause_after_aux_page` tests passes when run alone against local Postgres through `scripts/postgres-test-run.sh`. - A forced check, not committed: after the barrier, the test gave page 2 another 500 ms over ten re-polls and asserted the page count stayed at 1. It passed with the pause and failed with `None`, with the CI signature (left: 2, right: 1). A true same-poll burst needs the OS to preempt between sqlx's write and read, so it was not forced directly. - `check-postgres-test-discovery.py` accepts the touched files and all 621 files under `crates/`, and `scripts/test-postgres-test-discovery.sh` passes. Each of the six media tests passes when run alone against local Postgres. - `cargo fmt --check` and `cargo clippy -p buzz-relay --all-targets -- -D warnings` are clean under 1.98.1. Only hosted CI can show the full lanes, which were not run locally. The six media tests have never run in CI, so this PR's PostgreSQL lane is their first hosted run. Pushed with `git push --no-verify`, because the pre-push suites fail when the laptop is under load. ### Readiness An agent review of the combined `flakey-tests` branch (b432453af) found no blockers. `buzz-review-completed` is not added, because the human test step (AGENTS.md "Before opening a PR", step 3) hasn't happened yet. ### Siblings - #8739 ci: Hermit fetch deadline and sccache fallback - #8741 test(mobile): RunnerTests title fade and alert wait - #8742 fix(desktop): deep-linked message hold (product change) - #8743 test(desktop): four specs racing scroll settles and focus - #8744 test(desktop): five settle classes and remote-owned-mentions 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Matt Toohey <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent a83926d commit 322fdbf

4 files changed

Lines changed: 328 additions & 230 deletions

File tree

‎Justfile‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -508,7 +508,8 @@ test-unit:
508508
# #[ignore]d Postgres suites — so these non-ignored tests ran in no lane
509509
# and a red one could ship green (exactly how a broken admin test slipped
510510
# past every gate once). Scoped to api::admin, not the whole buzz-relay
511-
# --lib, because api::media has non-ignored tests that require Postgres.
511+
# --lib, because much of handlers::req has non-ignored tests that require
512+
# Postgres.
512513
# DB-backed api::admin tests are #[ignore]d and run in the PostgreSQL
513514
# lane; the non-ignored ones reject before touching the database. Any
514515
# new non-ignored test here must stay DB-free: without a database, a

‎crates/buzz-relay/src/api/bridge/thread_window/postgres_tests/failure_postgres_tests.rs‎

Lines changed: 32 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
use super::*;
2+
use crate::test_support::PausingProxy;
23
use std::{
34
future::Future,
45
sync::atomic::{AtomicUsize, Ordering},
@@ -9,23 +10,31 @@ use tracing_subscriber::{layer::Context, prelude::*, registry::LookupSpan, Layer
910

1011
// Pause the HTTP future after the first real auxiliary page completes. This
1112
// observes a production span rather than replacing the database or closure.
12-
struct AuxPages(Arc<AtomicUsize>);
13+
// With a proxy, the first page's span close also pauses it. That runs inside
14+
// the request's poll, before the second page's query is written, so the second
15+
// page stays pending however fast the database answers.
16+
struct AuxPages(Arc<AtomicUsize>, Option<Arc<PausingProxy>>);
1317
impl<S> Layer<S> for AuxPages
1418
where
1519
S: tracing::Subscriber + for<'a> LookupSpan<'a>,
1620
{
1721
fn on_close(&self, id: tracing::Id, ctx: Context<'_, S>) {
18-
if ctx.span(&id).unwrap().metadata().name() == "thread_window_aux" {
19-
self.0.fetch_add(1, Ordering::SeqCst);
22+
if ctx.span(&id).unwrap().metadata().name() == "thread_window_aux"
23+
&& self.0.fetch_add(1, Ordering::SeqCst) == 0
24+
{
25+
if let Some(proxy) = &self.1 {
26+
proxy.set_paused(true);
27+
}
2028
}
2129
}
2230
}
2331

2432
async fn pause_after_aux_page<F: Future>(
2533
request: F,
34+
proxy: Option<Arc<PausingProxy>>,
2635
) -> std::pin::Pin<Box<impl Future<Output = F::Output>>> {
2736
let pages = Arc::new(AtomicUsize::new(0));
28-
let subscriber = tracing_subscriber::registry().with(AuxPages(pages.clone()));
37+
let subscriber = tracing_subscriber::registry().with(AuxPages(pages.clone(), proxy));
2938
let mut request = Box::pin(request.with_subscriber(subscriber));
3039
tokio::time::timeout(
3140
Duration::from_secs(5),
@@ -63,9 +72,17 @@ async fn thread_window_bridge_restarts_both_aux_hops_after_replica_failure() {
6372
let reader_name = format!("tw-reader-{}", Uuid::new_v4());
6473
let options: sqlx::postgres::PgConnectOptions =
6574
crate::test_support::database_url().parse().unwrap();
75+
// The replica reads through a proxy that holds the second aux page.
76+
let proxy =
77+
Arc::new(PausingProxy::spawn(options.get_host().to_owned(), options.get_port()).await);
6678
let reader = sqlx::postgres::PgPoolOptions::new()
6779
.max_connections(1)
68-
.connect_with(options.application_name(&reader_name))
80+
.connect_with(
81+
options
82+
.host("127.0.0.1")
83+
.port(proxy.port)
84+
.application_name(&reader_name),
85+
)
6986
.await
7087
.unwrap();
7188
let db = buzz_db::Db::from_pools(f.pool.clone(), reader.clone());
@@ -75,7 +92,11 @@ async fn thread_window_bridge_restarts_both_aux_hops_after_replica_failure() {
7592
let mut filter = f.filter();
7693
filter["until"] = json!(f.root.created_at.as_secs() + 1);
7794
filter["before_id"] = json!("00".repeat(32));
78-
let request = pause_after_aux_page(f.post(&f.keys, "/query", json!([filter]))).await;
95+
let request = pause_after_aux_page(
96+
f.post(&f.keys, "/query", json!([filter])),
97+
Some(proxy.clone()),
98+
)
99+
.await;
79100
let held: i64 = sqlx::query_scalar("SELECT count(*) FROM pg_stat_activity WHERE application_name=$1 AND xact_start IS NOT NULL")
80101
.bind(&reader_name).fetch_one(&f.pool).await.unwrap();
81102
assert_eq!(held, 1, "must actually hold the proved replica transaction");
@@ -87,6 +108,9 @@ async fn thread_window_bridge_restarts_both_aux_hops_after_replica_failure() {
87108
.execute(&f.pool)
88109
.await
89110
.unwrap();
111+
// Deliver the termination, so the held page fails and the closure
112+
// restarts on the writer.
113+
proxy.set_paused(false);
90114
let (status, body) = request.await;
91115
assert_eq!(status, StatusCode::OK, "{body}");
92116
assert_eq!(f.bounds(&body, &filter)["has_more"], false);
@@ -194,7 +218,7 @@ async fn assert_aux_access_change(grant: bool) {
194218
"pre-transition visibility control"
195219
);
196220

197-
let request = pause_after_aux_page(f.post(&f.keys, "/query", json!([filter]))).await;
221+
let request = pause_after_aux_page(f.post(&f.keys, "/query", json!([filter])), None).await;
198222
set_access(&f, aux_channel, grant).await;
199223
let current = f
200224
.state
@@ -331,7 +355,7 @@ async fn thread_window_batch_discards_earlier_windows_after_revocation() {
331355
let (status, before) = f.post(&f.keys, "/query", batch.clone()).await;
332356
assert_eq!(status, StatusCode::OK, "{before}");
333357
assert_eq!(ids(&before, Some(39007)).len(), 2);
334-
let request = pause_after_aux_page(f.post(&f.keys, "/query", batch.clone())).await;
358+
let request = pause_after_aux_page(f.post(&f.keys, "/query", batch.clone()), None).await;
335359
set_access(&f, f.channel, false).await;
336360
let (status, body) = request.await;
337361
assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE, "{body}");

0 commit comments

Comments
 (0)