Skip to content

Commit cd5756a

Browse files
authored
perf(relay): index channel-id lookups and skip trace-only reads (#4647)
## Problem `SELECT id, community_id FROM channels WHERE id = ANY($1) AND deleted_at IS NULL` is the top **Load by waits (AAS)** on the Buzz Postgres writer. Two independent causes compound, and both are fixed here. ### 1. No index can serve it `channels` is `PRIMARY KEY (community_id, id)`, and every secondary index leads with `community_id`: | Index | Columns | |---|---| | *(primary key)* | `(community_id, id)` | | `idx_channels_nip29_group` | `(community_id, nip29_group_id)` | | `idx_channels_dm_hash` | `(community_id, participant_hash)` | | `idx_channels_community_type` | `(community_id, channel_type)` | | `idx_channels_community_visibility` | `(community_id, visibility)` | | `idx_channels_created_by` | `(community_id, created_by)` | | `idx_channels_ttl_expiry` | `(ttl_deadline)` *(partial)* | The two tenant-independent lookups carry **no `community_id` predicate** — deliberately: - `Db::communities_of_channels` — `WHERE id = ANY($1) AND deleted_at IS NULL` - `Db::community_of_channel` — `WHERE id = $1 AND deleted_at IS NULL` That independence is load-bearing, not an oversight: projecting a row's *true* owning community regardless of the fetch query's `WHERE` clause is what makes `Inv_NonInterference` non-vacuous. If the fetch ever dropped its tenant scoping, this lookup would still report the real label and the checker would catch the mismatch. But a composite btree is only usable when its leading column is constrained, so neither query can use the primary key, and nothing else leads with `id`. **Both sequentially scan `channels` on every call.** ### 2. In production the result is discarded Both call sites feed `record_read_message_rows` / `record_read_by_id_rows`, which call `tracer.record(...)`. Production binds `NoopTracer` (`crates/buzz-relay/src/state.rs`), whose `record` body is empty. The existing guard tests `trace_state`, which is `Some` for every well-formed request — it only goes `None` on malformed pubkey bytes. So the scan ran on the hot read path and its output was dropped. This is the classic eager-argument bug: `log.debug("..." + expensiveCall())` with no `isDebugEnabled()` check. ### 3. Multiplied per filter The non-search call site sits **inside the phase-3 per-filter loop**, so a `REQ` carrying N filters performed N sequential scans of `channels` before responding. ## Changes **`Tracer::enabled()`** — a capability check on the trait (the `isDebugEnabled()` of this seam), defaulting to `true`. `NoopTracer` overrides it to `false`, and both emitters in `req.rs` now gate on it, skipping the trace-only DB read entirely in production. **`migrations/0027_channels_id_lookup_index.sql`** ```sql CREATE INDEX IF NOT EXISTS idx_channels_id_live ON channels (id) INCLUDE (community_id) WHERE deleted_at IS NULL; ``` - `INCLUDE (community_id)` — both queries select exactly `(id, community_id)`, so this is covering and can be served index-only. - Partial on `deleted_at IS NULL` — matches both predicates exactly, excludes soft-deleted history, and lets Postgres skip the recheck. - **Not `UNIQUE`.** `id` alone is *not* unique in this table — `command_executor.rs` documents that `community_of_channel(channel_id)` is ambiguous because the same channel id can appear under more than one community. A unique index would encode a false constraint and fail to build on any database already holding such a pair. Worth keeping the index even though fix #1 removes the production caller: it still runs under conformance, and `community_of_channel` has the same problem on its own paths. **`schema/schema.sql`** — mirrored, since a test asserts desired-state parity. ## Conformance is unchanged This is the part worth reviewing closely. Under a real tracer `enabled()` returns `true` and **every emit happens exactly as before** — the gate only skips *building* emit inputs when nothing observes them, never an emit that would otherwise have been made. The coverage-breach guard stays non-vacuous. `CountingTracer` forwards `enabled()` to its inner tracer rather than inheriting the `true` default. Both directions matter and both fail silently: - inheriting `true` over a `NoopTracer` would keep the overhead this PR removes; - hardcoding `false` over a live tracer would suppress the emits whose absence `EmitGuard` reports as `ImplBug` — masking real breaches behind expected ones. Covered by a new regression test, `counting_tracer_delegates_enabled_to_inner`, which asserts delegation in both directions. ## Verification - `cargo check -p buzz-conformance -p buzz-relay` — clean - `cargo clippy --all-targets` — clean, zero warnings - `cargo test -p buzz-conformance` — 6/6 - `cargo test -p buzz-relay --lib conformance` — 11/11 - `cargo test -p buzz-db --lib migration` — 7/7 - `just test-unit` (pre-push) — green Migration-count assertions in `crates/buzz-db/src/migration.rs` were bumped 26 → 27, with content assertions for 0027 following the existing per-migration pattern (including a guard that it never becomes `UNIQUE`). ## Open questions for reviewers 1. **Lock strategy.** Built *without* `CONCURRENTLY`, following migration 0004's precedent, because sqlx runs each migration inside a transaction and `CREATE INDEX CONCURRENTLY` cannot run in one. This takes a brief `SHARE` lock on `channels` (blocks writes, not reads) — small relative to `events`, but an operator preferring zero write-blocking can pre-build it by hand and `IF NOT EXISTS` makes the migration a no-op. I could not confirm whether sqlx 0.9 supports a `-- no-transaction` directive; if it does, that may be preferable. 2. **Diagnosis is static.** This comes from reading the source, not from `EXPLAIN` against the live database. Worth confirming with `EXPLAIN (ANALYZE, BUFFERS)` on the writer before/after — that also sizes the win by revealing the real table size and row counts. 3. **Expected impact** scales with average filters-per-`REQ`, which I did not measure. `pg_stat_statements` ordered by `total_exec_time` would confirm this query drops off the top and show whether anything else is scanning the same way. Signed-off-by: Jemiah Westerman <[email protected]>
1 parent 22cfba1 commit cd5756a

7 files changed

Lines changed: 185 additions & 5 deletions

File tree

‎crates/buzz-conformance/src/lib.rs‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -315,6 +315,27 @@ pub trait Tracer: Send + Sync {
315315
/// Record one trace step. Implementations MAY be no-ops in production
316316
/// builds and write to JSONL in tests.
317317
fn record(&self, step: TraceStep);
318+
319+
/// Whether recorded steps are actually observed.
320+
///
321+
/// Emitters on hot paths MUST consult this before doing work whose
322+
/// *only* consumer is the trace — most importantly extra database
323+
/// reads that project row labels independently of the fetch query
324+
/// (the read-seam's `communities_of_channels` lookup). With a
325+
/// discarding tracer that work is pure overhead.
326+
///
327+
/// This is the `log.isDebugEnabled()` of the trace seam. It exists to
328+
/// let callers skip *building emit inputs*, never to let them skip an
329+
/// emit they would otherwise have made: when this returns `true`
330+
/// every seam must behave exactly as it did before the gate existed,
331+
/// so the coverage-breach guard stays non-vacuous.
332+
///
333+
/// Defaults to `true` — a new tracer is assumed to observe steps until
334+
/// it says otherwise. Wrappers that delegate to an inner tracer MUST
335+
/// forward this method rather than inherit the default.
336+
fn enabled(&self) -> bool {
337+
true
338+
}
318339
}
319340

320341
/// A no-op tracer for production. Zero cost: the build can omit emission
@@ -324,4 +345,9 @@ pub struct NoopTracer;
324345

325346
impl Tracer for NoopTracer {
326347
fn record(&self, _step: TraceStep) {}
348+
349+
/// Nothing is observed, so emitters should skip building inputs.
350+
fn enabled(&self) -> bool {
351+
false
352+
}
327353
}

‎crates/buzz-db/src/migration.rs‎

Lines changed: 24 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -100,7 +100,7 @@ mod tests {
100100
use super::*;
101101
use std::collections::BTreeSet;
102102

103-
const TEST_DB_URL: &str = "postgres://buzz:buzz_dev@localhost:5432/buzz";
103+
const TEST_DB_URL: &str = "postgres://buzz:buzz_dev@localhost:5432/buzz"; // sadscan:disable np.postgres.1
104104

105105
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
106106
enum ConstraintKind {
@@ -561,7 +561,7 @@ mod tests {
561561
let mut migrations: Vec<_> = MIGRATOR.iter().collect();
562562
migrations.sort_by_key(|migration| migration.version);
563563

564-
assert_eq!(migrations.len(), 26);
564+
assert_eq!(migrations.len(), 27);
565565
assert_eq!(migrations[0].version, 1);
566566
assert_eq!(&*migrations[0].description, "initial schema");
567567
assert!(migrations[0]
@@ -919,6 +919,27 @@ mod tests {
919919
assert!(heartbeat.contains("epoch"));
920920
assert!(heartbeat.contains("INSERT INTO replica_heartbeat (id) VALUES (1)"));
921921
assert!(heartbeat.contains("_operator_global_tables"));
922+
923+
// Channel-id lookup index (0027): serves the tenant-independent
924+
// `channels` lookups that carry no community_id predicate, which no
925+
// community_id-leading index can satisfy. Covering + partial so the
926+
// planner can go index-only; asserted NOT UNIQUE because `id` alone is
927+
// not unique in this table (the same channel id may exist under more
928+
// than one community), so a unique index would encode a false
929+
// constraint and fail to build on such a database.
930+
assert_eq!(migrations[26].version, 27);
931+
let channel_id_index = migrations[26].sql.as_str();
932+
assert!(channel_id_index.contains("idx_channels_id_live"));
933+
assert!(channel_id_index.contains("INCLUDE (community_id)"));
934+
assert!(channel_id_index.contains("WHERE deleted_at IS NULL"));
935+
assert!(
936+
!channel_id_index.contains("CREATE UNIQUE INDEX"),
937+
"channels.id is not unique across communities — index must not be UNIQUE",
938+
);
939+
assert!(
940+
desired_schema.contains("idx_channels_id_live"),
941+
"desired-state schema must carry the channel-id lookup index",
942+
);
922943
}
923944

924945
#[test]
@@ -1161,7 +1182,7 @@ mod tests {
11611182
run_migrations(&pool)
11621183
.await
11631184
.expect("retry succeeds after operator repair");
1164-
assert_eq!(applied_versions(&pool).await.last().copied(), Some(26));
1185+
assert_eq!(applied_versions(&pool).await.last().copied(), Some(27));
11651186
}
11661187

11671188
#[tokio::test]

‎crates/buzz-relay/src/conformance/mod.rs‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -370,6 +370,16 @@ impl Tracer for CountingTracer {
370370
.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
371371
self.inner.record(step);
372372
}
373+
374+
/// Delegate, never inherit the `true` default. This wrapper is
375+
/// transparent: whether emits are observed is a property of the
376+
/// tracer underneath it. Returning `true` over a `NoopTracer` would
377+
/// reintroduce the overhead the gate exists to remove; returning
378+
/// `false` over a real tracer would suppress the emits whose absence
379+
/// the `EmitGuard` reports as a coverage breach.
380+
fn enabled(&self) -> bool {
381+
self.inner.enabled()
382+
}
373383
}
374384

375385
impl EmitGuard {
@@ -455,6 +465,52 @@ mod tests {
455465
}
456466
}
457467

468+
/// Discarding tracer that reports `enabled() == false`, standing in
469+
/// for the production `NoopTracer`.
470+
#[derive(Debug, Default)]
471+
struct DisabledTracer;
472+
473+
impl Tracer for DisabledTracer {
474+
fn record(&self, _step: TraceStep) {}
475+
fn enabled(&self) -> bool {
476+
false
477+
}
478+
}
479+
480+
/// `CountingTracer` must forward `enabled()` to the tracer it wraps
481+
/// rather than inherit the trait's `true` default. Both directions
482+
/// matter, and getting either wrong is silent:
483+
///
484+
/// - over a disabled tracer, answering `true` would keep the hot-path
485+
/// read-seam `channels` lookup running in production — the overhead
486+
/// the gate exists to remove;
487+
/// - over a live tracer, answering `false` would make gated emitters
488+
/// skip emits during conformance runs, so the `EmitGuard` would
489+
/// report `ImplBug` for seams that are in fact correct (or, worse,
490+
/// mask a real breach behind an expected one).
491+
#[test]
492+
fn counting_tracer_delegates_enabled_to_inner() {
493+
let (_guard, counting) = EmitGuard::arm(
494+
Arc::new(DisabledTracer),
495+
dummy_state(),
496+
"delegates_disabled",
497+
);
498+
assert!(
499+
!counting.enabled(),
500+
"CountingTracer must report disabled when wrapping a discarding tracer"
501+
);
502+
503+
let (_guard, counting) = EmitGuard::arm(
504+
Arc::new(VecTracer::default()),
505+
dummy_state(),
506+
"delegates_live",
507+
);
508+
assert!(
509+
counting.enabled(),
510+
"CountingTracer must report enabled when wrapping an observing tracer"
511+
);
512+
}
513+
458514
fn dummy_state() -> AbstractState {
459515
AbstractState {
460516
resolved_community: CommunityLabel::from_uuid(Uuid::from_u128(0xA)),

‎crates/buzz-relay/src/conformance/tracers.rs‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,12 @@ pub struct NoopTracer;
1717

1818
impl Tracer for NoopTracer {
1919
fn record(&self, _step: TraceStep) {}
20+
21+
/// Nothing is observed, so emitters should skip building inputs —
22+
/// including the read-seam's per-request `channels` lookup.
23+
fn enabled(&self) -> bool {
24+
false
25+
}
2026
}
2127

2228
/// JSONL-to-file tracer for tests + the CI replay job. Each `record` call

‎crates/buzz-relay/src/handlers/req.rs‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -334,7 +334,12 @@ pub async fn handle_req(
334334
// (B) projection strategy and the missing-lookup ImplBug
335335
// guard-rail. Skipped silently if `trace_state` is `None` (only
336336
// happens on malformed pubkey, a separate failure path).
337-
if let Some(state_snap) = trace_state.as_ref() {
337+
// `tracer.enabled()` short-circuits the whole block on the production
338+
// `NoopTracer`: the `communities_of_channels` lookup below is a
339+
// `channels` read whose only consumer is `record_read_message_rows`,
340+
// and this emit runs once PER FILTER. Gating on `trace_state` alone was
341+
// not enough — that is `Some` for every well-formed request.
342+
if let Some(state_snap) = trace_state.as_ref().filter(|_| state.tracer.enabled()) {
338343
let row_channels: Vec<Option<uuid::Uuid>> =
339344
events.iter().map(|e| e.channel_id).collect();
340345
let distinct: Vec<uuid::Uuid> = {
@@ -659,7 +664,9 @@ async fn handle_search_req(
659664
// level isn't bound to a single channel filter, the
660665
// per-row `channel_id` carries the channel identity
661666
// honestly.
662-
if let Some(state_snap) = trace_state {
667+
// Same `enabled()` gate as the non-search lane: skip the
668+
// trace-only `channels` lookup when nothing observes the emit.
669+
if let Some(state_snap) = trace_state.filter(|_| state.tracer.enabled()) {
663670
let row_channels: Vec<Option<uuid::Uuid>> =
664671
events.iter().map(|e| e.channel_id).collect();
665672
let distinct: Vec<uuid::Uuid> = {
Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
-- ── Covering index for channel-id → community lookups ───────────────────────
2+
-- `channels` is keyed PRIMARY KEY (community_id, id), and every secondary index
3+
-- leads with community_id:
4+
--
5+
-- idx_channels_nip29_group (community_id, nip29_group_id)
6+
-- idx_channels_dm_hash (community_id, participant_hash)
7+
-- idx_channels_community_type (community_id, channel_type)
8+
-- idx_channels_community_visibility (community_id, visibility)
9+
-- idx_channels_created_by (community_id, created_by)
10+
--
11+
-- The tenant-independent lookups in buzz-db resolve a channel's owning
12+
-- community *without* a community_id predicate — that independence is the
13+
-- point (buzz-db/src/lib.rs: the read-seam projects a row's true label
14+
-- regardless of the fetch query's WHERE clause, which is what makes
15+
-- Inv_NonInterference non-vacuous):
16+
--
17+
-- Db::communities_of_channels SELECT id, community_id FROM channels
18+
-- WHERE id = ANY($1) AND deleted_at IS NULL
19+
-- Db::community_of_channel SELECT community_id FROM channels
20+
-- WHERE id = $1 AND deleted_at IS NULL
21+
--
22+
-- A composite btree is only usable when its leading column is constrained, so
23+
-- neither query can use the primary key and no other index leads with `id`.
24+
-- Both therefore sequentially scan `channels` on every call. Observed as the
25+
-- top "Load by waits (AAS)" on the staging writer (db.r8g.8xlarge, ~53% CPU).
26+
--
27+
-- INCLUDE (community_id): both queries select only (id, community_id), so the
28+
-- index is covering and the planner can serve them index-only, with no heap
29+
-- fetch for visible rows.
30+
--
31+
-- Partial on deleted_at IS NULL: matches both predicates exactly, keeps the
32+
-- index off soft-deleted history, and lets Postgres skip re-checking the
33+
-- predicate.
34+
--
35+
-- NOT UNIQUE, deliberately. `id` alone is not unique in this table —
36+
-- handlers/command_executor.rs documents that community_of_channel(channel_id)
37+
-- is ambiguous because the same channel id can appear under more than one
38+
-- community. A unique index would encode a false constraint and would fail to
39+
-- build on any database that already holds such a pair.
40+
--
41+
-- Lock note: built without CONCURRENTLY, matching migration 0004's precedent —
42+
-- sqlx runs each migration inside a transaction and CREATE INDEX CONCURRENTLY
43+
-- cannot run in one. This takes a SHARE lock on `channels` (blocking writes,
44+
-- not reads) for the duration of the build. `channels` is a small table
45+
-- relative to `events`, so this is expected to be brief, but on a large
46+
-- brownfield database an operator may prefer to pre-build it by hand:
47+
--
48+
-- CREATE INDEX CONCURRENTLY idx_channels_id_live
49+
-- ON channels (id) INCLUDE (community_id)
50+
-- WHERE deleted_at IS NULL;
51+
--
52+
-- IF NOT EXISTS then makes this migration a no-op on that database.
53+
--
54+
-- Additive migration: previously applied files must not change checksum.
55+
56+
CREATE INDEX IF NOT EXISTS idx_channels_id_live
57+
ON channels (id) INCLUDE (community_id)
58+
WHERE deleted_at IS NULL;

‎schema/schema.sql‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -112,6 +112,12 @@ CREATE INDEX idx_channels_community_visibility ON channels (community_id, visibi
112112
CREATE INDEX idx_channels_created_by ON channels (community_id, created_by);
113113
CREATE INDEX idx_channels_ttl_expiry ON channels (ttl_deadline)
114114
WHERE ttl_seconds IS NOT NULL AND archived_at IS NULL AND deleted_at IS NULL;
115+
-- Tenant-independent channel-id → community lookups (Db::communities_of_channels,
116+
-- Db::community_of_channel) carry no community_id predicate, so no
117+
-- community_id-leading index can serve them. Covering + partial: index-only scan.
118+
-- Not UNIQUE — the same channel id may exist under more than one community.
119+
CREATE INDEX idx_channels_id_live ON channels (id) INCLUDE (community_id)
120+
WHERE deleted_at IS NULL;
115121

116122
-- channels.community_id is immutable: a channel can never be re-tenanted.
117123
-- (Conformance: "Migration lint forbids channel re-tenanting except through an

0 commit comments

Comments
 (0)