Skip to content

Commit c99872f

Browse files
TheSentinel454Elrond
andauthored
fix(relay): delete the startup NIP-33 d_tag backfill (#8468)
## Summary Delete the NIP-33 `d_tag` backfill that ran on every relay boot. On every boot, before the listener opened, the relay ran: ```sql SELECT DISTINCT community_id FROM events WHERE kind BETWEEN 30000 AND 39999 AND d_tag IS NULL ``` No index covers `d_tag IS NULL`. The only `d_tag` index is the partial `idx_events_parameterized`, which is `WHERE d_tag IS NOT NULL`. So this query scanned every partition of `events`. In production it averaged about 11s per boot, max 19s (`buzz_startup_phase_seconds{phase="d_tag_backfill"}`), and found nothing. These rows shouldn't exist anymore: - The backfill was the one-time migration for #246, the change that added `d_tag`. - In `crates/buzz-db/src`, every production write path sets the column for kinds 30000–39999 through `extract_d_tag`: the main insert, `replaceable.rs`, `channel_members.rs`, `relay_members.rs`, and `push.rs`. A new source test, `every_production_event_insert_names_d_tag`, now enforces that. - The `events` inserts that omit `d_tag` are all test fixtures. `scripts/backfill-d-tag.sql` is still there as the manual fallback for any database that predates #246 and somehow never booted a relay since. ### What got simpler - Removed `Db::backfill_d_tags` and its admission test. - Removed the source-shape assertion in `observability_source.rs`. - Removed the `d_tag_backfill` startup phase. - `StartupStep::ALL` is now a slice, so removing a step no longer means editing a hand-kept array length. ### Notes - The `buzz_startup_phase_seconds{phase="d_tag_backfill"}` series stops reporting. - **Rebased onto main after #8467 merged.** Both remove a startup phase. With the fixed-length `ALL` array the two merged cleanly in git into a 17-entry `[Self; 18]`, which doesn't compile. This PR makes `ALL` a slice. - **Before merging**, run this read-only query against production and confirm it returns `0`: `SELECT count(*) FROM events WHERE kind BETWEEN 30000 AND 39999 AND d_tag IS NULL`. It scans every partition, so use a read replica. If it isn't `0`, run `scripts/backfill-d-tag.sql` first. The cutover script copies legacy `d_tag` values as-is, so only this query proves historical data is clean. - **Self-hosted installs** should run the same check before upgrading. ### Related issue None found. Companion to #8467, which also targets relay startup time. ### Testing - New `every_production_event_insert_names_d_tag` (`crates/buzz-db/tests/observability_source.rs`): every production `INSERT INTO events` must name `d_tag` in its column list. Dropping `d_tag` from `push.rs` makes it fail and name the function. - Runtime E2E against a real relay: boot has no `d_tag_backfill` phase; kind:30023 replacement via WebSocket and HTTP keeps only the newer event, an unrelated `d` survives, and an older replay can't win (confirmed in SQL). - Stacked on #8467 at `501dfd83`: a fresh two-replica boot runs all enabled startup phases, with neither `nip43_reconcile` nor `d_tag_backfill` present. Both replicas pass readiness and return the same signed message over authenticated HTTP and WebSocket, and outsiders are denied. - Blox, at `501dfd83`: `just fmt-check clippy security-review-check file-size-check` passes. Every `test-unit` step and `just test` pass except the `buzz-acp` `git_bootstrap` native-git tests, which fail the same way at base `b6aa5539` on Blox, and one `buzz-agent` catalog timing test that fails only under full-suite load and passes when run alone. - `e27139de` adds only a doc comment on `every_production_event_insert_names_d_tag`, saying which SQL shapes the text match covers. No code changed, so the results above still apply. --------- Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
1 parent c9e35f8 commit c99872f

5 files changed

Lines changed: 148 additions & 118 deletions

File tree

‎crates/buzz-db/src/store/admission_postgres_tests.rs‎

Lines changed: 0 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -1003,41 +1003,6 @@ async fn fleet_sweep_returns_committed_results_when_database_drops_mid_page() {
10031003
);
10041004
}
10051005

1006-
#[tokio::test]
1007-
#[ignore = "requires Postgres"]
1008-
async fn d_tag_backfill_admits_active_tenant_and_skips_quiescing_tenant() {
1009-
let (db, communities) = fixture().await;
1010-
crate::test_support::reject_legacy_sql_fence_predicate(&db.pool).await;
1011-
for (index, community) in communities[..2].iter().enumerate() {
1012-
sqlx::query(
1013-
"INSERT INTO events (community_id, id, pubkey, created_at, kind, tags, content, sig) \
1014-
VALUES ($1, $2, $3, to_timestamp(1), 30023, '[[\"d\",\"tag\"]]'::jsonb, '', $4)",
1015-
)
1016-
.bind(community.as_uuid())
1017-
.bind(vec![index as u8; 32])
1018-
.bind(vec![2_u8; 32])
1019-
.bind(vec![3_u8; 64])
1020-
.execute(&db.pool)
1021-
.await
1022-
.expect("seed addressable event");
1023-
}
1024-
assert_eq!(db.backfill_d_tags().await.unwrap(), 1);
1025-
let active: Option<String> =
1026-
sqlx::query_scalar("SELECT d_tag FROM events WHERE community_id = $1")
1027-
.bind(communities[0].as_uuid())
1028-
.fetch_one(&db.pool)
1029-
.await
1030-
.unwrap();
1031-
let quiescing: Option<String> =
1032-
sqlx::query_scalar("SELECT d_tag FROM events WHERE community_id = $1")
1033-
.bind(communities[1].as_uuid())
1034-
.fetch_one(&db.pool)
1035-
.await
1036-
.unwrap();
1037-
assert_eq!(active.as_deref(), Some("tag"));
1038-
assert_eq!(quiescing, None);
1039-
}
1040-
10411006
#[tokio::test]
10421007
#[ignore = "requires Postgres"]
10431008
async fn match_claim_admits_active_tenant_and_skips_quiescing_tenant() {

‎crates/buzz-db/src/store/event.rs‎

Lines changed: 0 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -2776,53 +2776,6 @@ impl Db {
27762776
Ok(result)
27772777
}
27782778

2779-
/// Backfill `d_tag` for existing NIP-33 events (kind 30000–39999) that have `d_tag IS NULL`.
2780-
///
2781-
/// Idempotent — safe to call on every startup. No-ops when all rows are already populated.
2782-
/// Admits each community before updating its NIP-33 rows with NULL d_tag.
2783-
#[datastore_span(name = "backfill_d_tags", system = "postgresql")]
2784-
pub async fn backfill_d_tags(&self) -> Result<u64> {
2785-
let mut connection = crate::observability::acquire_writer(
2786-
&self.pool,
2787-
crate::observability::WriterOperation::Bootstrap,
2788-
)
2789-
.await?;
2790-
let communities: Vec<Uuid> = sqlx::query_scalar(
2791-
"SELECT DISTINCT community_id FROM events \
2792-
WHERE kind BETWEEN 30000 AND 39999 AND d_tag IS NULL",
2793-
)
2794-
.fetch_all(&mut *connection)
2795-
.await?;
2796-
drop(connection);
2797-
let mut updated = 0;
2798-
for id in communities {
2799-
let Some(mut tx) = crate::begin_community_sweep_transaction(
2800-
&self.pool,
2801-
CommunityId::from_uuid(id),
2802-
crate::observability::WriterOperation::Bootstrap,
2803-
)
2804-
.await?
2805-
else {
2806-
continue;
2807-
};
2808-
let result = sqlx::query(
2809-
"UPDATE events \
2810-
SET d_tag = COALESCE( \
2811-
(SELECT elem->>1 FROM jsonb_array_elements(tags) AS elem \
2812-
WHERE elem->>0 = 'd' LIMIT 1), \
2813-
'' \
2814-
) \
2815-
WHERE community_id = $1 AND kind BETWEEN 30000 AND 39999 AND d_tag IS NULL",
2816-
)
2817-
.bind(id)
2818-
.execute(tx.conn())
2819-
.await?;
2820-
tx.commit().await?;
2821-
updated += result.rows_affected();
2822-
}
2823-
Ok(updated)
2824-
}
2825-
28262779
/// Soft-delete NIP-29 discovery events for a channel created by a specific relay pubkey.
28272780
#[datastore_span(name = "soft_delete_discovery_events", system = "postgresql")]
28282781
pub async fn soft_delete_discovery_events(

‎crates/buzz-db/tests/observability_source.rs‎

Lines changed: 147 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -156,14 +156,6 @@ fn p0_pool_acquisitions_use_typed_operation_pairs_without_other() {
156156
assert!(event.contains("WriterOperation::Authorization"));
157157
assert!(event.contains("WriterOperation::SubscriptionHistory"));
158158
assert!(event.contains("ReaderOperation::SubscriptionHistory"));
159-
let backfill_d_tags = event
160-
.split_once("pub async fn backfill_d_tags")
161-
.expect("event store must expose the startup d-tag backfill")
162-
.1
163-
.split_once("/// Soft-delete NIP-29 discovery events")
164-
.expect("d-tag backfill must precede discovery deletion")
165-
.0;
166-
assert!(backfill_d_tags.contains("WriterOperation::Bootstrap"));
167159
let soft_delete_discovery = event
168160
.split_once("pub async fn soft_delete_discovery_events")
169161
.expect("event store must expose discovery-event deletion")
@@ -1633,6 +1625,12 @@ fn masker_blanks_every_literal_form_and_keeps_code() {
16331625
/// continuations, `\n`/`\t`/`\r` escapes, and with or without a `public.`
16341626
/// schema prefix.
16351627
fn inserts_events_row(source: &str) -> bool {
1628+
!events_insert_tails(source).is_empty()
1629+
}
1630+
1631+
/// `source` lowercased, with every run of whitespace, `\` line continuations,
1632+
/// and `\n`/`\t`/`\r` escapes collapsed to one space.
1633+
fn normalized_sql(source: &str) -> String {
16361634
let mut normalized = String::with_capacity(source.len());
16371635
let mut chars = source.chars().peekable();
16381636
while let Some(ch) = chars.next() {
@@ -1647,16 +1645,26 @@ fn inserts_events_row(source: &str) -> bool {
16471645
normalized.push(ch.to_ascii_lowercase());
16481646
}
16491647
}
1650-
["insert into events", "insert into public.events"]
1651-
.iter()
1652-
.any(|marker| {
1653-
normalized.match_indices(marker).any(|(index, marker)| {
1654-
normalized[index + marker.len()..]
1655-
.chars()
1656-
.next()
1657-
.is_none_or(|next| !(next.is_ascii_alphanumeric() || next == '_'))
1658-
})
1659-
})
1648+
normalized
1649+
}
1650+
1651+
/// The normalized SQL that follows each `INSERT INTO events` in `source`.
1652+
fn events_insert_tails(source: &str) -> Vec<String> {
1653+
let normalized = normalized_sql(source);
1654+
let mut tails = Vec::new();
1655+
for marker in ["insert into events", "insert into public.events"] {
1656+
for (index, marker) in normalized.match_indices(marker) {
1657+
let tail = &normalized[index + marker.len()..];
1658+
if tail
1659+
.chars()
1660+
.next()
1661+
.is_none_or(|next| !(next.is_ascii_alphanumeric() || next == '_'))
1662+
{
1663+
tails.push(tail.to_owned());
1664+
}
1665+
}
1666+
}
1667+
tails
16601668
}
16611669

16621670
/// An events insert must run both follow-ups: the push enqueue and the
@@ -2062,3 +2070,124 @@ fn every_production_event_insert_runs_the_roster_guard() {
20622070
"expected at least nine event writers, found {checked_writers}"
20632071
);
20642072
}
2073+
2074+
// The rolled-back replica-floor probe writes a kind 9 row, which has no `d`.
2075+
const EVENT_INSERT_D_TAG_EXCEPTIONS: [&str; 1] = ["pub async fn verify_floor_guard_behavior("];
2076+
2077+
/// Whether an `INSERT INTO events` tail names `d_tag` in an explicit column
2078+
/// list. An insert without a column list does not count.
2079+
fn events_insert_names_d_tag(tail: &str) -> bool {
2080+
tail.trim_start()
2081+
.strip_prefix('(')
2082+
.and_then(|columns| columns.split_once(')'))
2083+
.is_some_and(|(columns, _)| columns.split(',').any(|column| column.trim() == "d_tag"))
2084+
}
2085+
2086+
fn event_insert_d_tag_violations(production_source: &str) -> Vec<String> {
2087+
function_slices(production_source)
2088+
.into_iter()
2089+
.filter(|function_source| {
2090+
let header = function_header(function_source);
2091+
!EVENT_INSERT_D_TAG_EXCEPTIONS
2092+
.iter()
2093+
.any(|exception| header.starts_with(exception))
2094+
&& !events_insert_tails(function_source)
2095+
.iter()
2096+
.all(|tail| events_insert_names_d_tag(tail))
2097+
})
2098+
.map(|function_source| function_header(function_source).to_owned())
2099+
.collect()
2100+
}
2101+
2102+
#[test]
2103+
fn event_insert_d_tag_policy_rejects_writers_without_the_column() {
2104+
let source = r##"
2105+
pub(crate) async fn writer_without_d_tag(tx: &mut AdmittedTx) {
2106+
sqlx::query("INSERT INTO events (community_id, id, kind) VALUES ($1, $2, $3)");
2107+
}
2108+
pub(crate) async fn writer_with_d_tag(tx: &mut AdmittedTx) {
2109+
sqlx::query("INSERT INTO events \
2110+
(community_id, id, kind, d_tag) VALUES ($1, $2, $3, $4)");
2111+
}
2112+
pub(crate) async fn writer_with_compact_columns(tx: &mut AdmittedTx) {
2113+
sqlx::query("insert into public.events (community_id,id,kind,d_tag) VALUES ($1,$2,$3,$4)");
2114+
}
2115+
pub(crate) async fn writer_with_d_tag_only_in_values(tx: &mut AdmittedTx) {
2116+
sqlx::query("INSERT INTO events (community_id, id, kind) SELECT community_id, id, d_tag FROM x");
2117+
}
2118+
pub(crate) async fn writer_with_lookalike_column(tx: &mut AdmittedTx) {
2119+
sqlx::query("INSERT INTO events (community_id, id, kind, d_tag_hash) VALUES ($1, $2, $3, $4)");
2120+
}
2121+
pub(crate) async fn writer_without_column_list(tx: &mut AdmittedTx) {
2122+
sqlx::query("INSERT INTO events SELECT * FROM staged_events");
2123+
}
2124+
pub(crate) async fn second_insert_without_d_tag(tx: &mut AdmittedTx) {
2125+
sqlx::query("INSERT INTO events (community_id, id, d_tag) VALUES ($1, $2, $3)");
2126+
sqlx::query("INSERT INTO events (community_id, id) VALUES ($1, $2)");
2127+
}
2128+
pub(crate) async fn mention_writer(tx: &mut AdmittedTx) {
2129+
sqlx::query("INSERT INTO event_mentions (community_id, event_id) VALUES ($1, $2)");
2130+
}
2131+
pub async fn verify_floor_guard_behavior(pool: &PgPool) {
2132+
sqlx::query("INSERT INTO events (community_id, id, kind) VALUES ($1, $2, 9)");
2133+
}
2134+
#[cfg(test)]
2135+
mod tests {
2136+
fn fixture() {
2137+
sqlx::query("INSERT INTO events (community_id, id) VALUES ($1, $2)");
2138+
}
2139+
}
2140+
"##;
2141+
assert_eq!(
2142+
event_insert_d_tag_violations(&strip_cfg_test_items(source)),
2143+
[
2144+
"pub(crate) async fn writer_without_d_tag(tx: &mut AdmittedTx) {",
2145+
"pub(crate) async fn writer_with_d_tag_only_in_values(tx: &mut AdmittedTx) {",
2146+
"pub(crate) async fn writer_with_lookalike_column(tx: &mut AdmittedTx) {",
2147+
"pub(crate) async fn writer_without_column_list(tx: &mut AdmittedTx) {",
2148+
"pub(crate) async fn second_insert_without_d_tag(tx: &mut AdmittedTx) {",
2149+
],
2150+
"every production events insert must name `d_tag` in its column list"
2151+
);
2152+
}
2153+
2154+
/// Addressable events (kinds 30000-39999) replace by `d_tag`, and the only
2155+
/// index that serves that lookup is partial on `d_tag IS NOT NULL`. A writer
2156+
/// that leaves the column out stores NULL, so its rows never replace. Nothing
2157+
/// repairs them after the fact, so every production writer must name the column.
2158+
///
2159+
/// This is a text match, not a proof. It checks the column list of literal
2160+
/// `INSERT INTO [public.]events (...)` SQL inside function bodies under each
2161+
/// crate's `src/`. It does not see `ONLY`, quoted or `format!`-built table
2162+
/// names, a comment between the keywords, or SQL held in a constant outside a
2163+
/// function, and it does not check that the bound value is non-NULL. Exempt
2164+
/// probes are skipped by function header, so every insert inside them is too.
2165+
#[test]
2166+
fn every_production_event_insert_names_d_tag() {
2167+
let mut checked_writers = 0;
2168+
let mut violations = Vec::new();
2169+
for path in production_event_writer_source_files() {
2170+
let source = std::fs::read_to_string(&path).expect("read source file");
2171+
let production = strip_cfg_test_items(&source);
2172+
checked_writers += function_slices(&production)
2173+
.into_iter()
2174+
.filter(|function| inserts_events_row(function))
2175+
.count();
2176+
violations.extend(
2177+
event_insert_d_tag_violations(&production)
2178+
.into_iter()
2179+
.map(|header| format!("{}: {header}", path.display())),
2180+
);
2181+
}
2182+
assert!(
2183+
violations.is_empty(),
2184+
"production `INSERT INTO events` writers must name `d_tag` in their column list: \
2185+
{violations:?}"
2186+
);
2187+
// The same writer set as the roster-guard scan. A lower count means the
2188+
// scan stopped seeing a writer, not that one was removed safely.
2189+
assert!(
2190+
checked_writers >= 9,
2191+
"expected at least nine event writers, found {checked_writers}"
2192+
);
2193+
}

‎crates/buzz-relay/src/main.rs‎

Lines changed: 0 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -704,19 +704,6 @@ async fn run_relay_main(boot: BootTracker) -> anyhow::Result<()> {
704704
}
705705
step.finish();
706706

707-
// NIP-33: backfill d_tag for any existing parameterized replaceable events
708-
// that predate the column addition. Idempotent — no-ops when fully populated.
709-
let mut step = StepTimer::start(StartupStep::DTagBackfill);
710-
match db.backfill_d_tags().await {
711-
Ok(0) => {}
712-
Ok(n) => info!("Backfilled d_tag for {n} NIP-33 events"),
713-
Err(e) => {
714-
step.degrade();
715-
error!("Failed to backfill d_tags: {e}");
716-
}
717-
}
718-
step.finish();
719-
720707
let (audit, audit_metrics_pool) = if config.audit_enabled {
721708
let step = StepTimer::start(StartupStep::AuditConnect);
722709
let audit_pool = connect_audit_pool(&db_config)

‎crates/buzz-relay/src/startup_steps.rs‎

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -51,8 +51,6 @@ pub enum StartupStep {
5151
ReplicaFenceProbe,
5252
/// Ensure the deployment community, backfill the allowlist, and bootstrap the owner.
5353
MembershipBootstrap,
54-
/// Backfill NIP-33 `d_tag` values.
55-
DTagBackfill,
5654
/// Connect the audit database pool (only when audit is enabled).
5755
AuditConnect,
5856
/// Connect Redis commands and pub/sub.
@@ -79,14 +77,13 @@ pub enum StartupStep {
7977

8078
impl StartupStep {
8179
/// The complete `phase` label vocabulary.
82-
pub const ALL: [Self; 18] = [
80+
pub const ALL: &[Self] = &[
8381
Self::DbConnect,
8482
Self::DbMigrate,
8583
Self::PartitionEnsure,
8684
Self::DeletionFenceVerify,
8785
Self::ReplicaFenceProbe,
8886
Self::MembershipBootstrap,
89-
Self::DTagBackfill,
9087
Self::AuditConnect,
9188
Self::RedisConnect,
9289
Self::SearchConnect,
@@ -109,7 +106,6 @@ impl StartupStep {
109106
Self::DeletionFenceVerify => "deletion_fence_verify",
110107
Self::ReplicaFenceProbe => "replica_fence_probe",
111108
Self::MembershipBootstrap => "membership_bootstrap",
112-
Self::DTagBackfill => "d_tag_backfill",
113109
Self::AuditConnect => "audit_connect",
114110
Self::RedisConnect => "redis_connect",
115111
Self::SearchConnect => "search_connect",

0 commit comments

Comments
 (0)