Skip to content

Commit e097c42

Browse files
TheSentinel454Elrond
andauthored
feat(relay): time post-metrics-bind startup steps (#8132)
## Summary Relay pods on large deployments can take over 5 minutes to open their health port, and right now nothing shows which startup step is slow. Startup timing ends at `metrics_bind`, and the slow work all happens after it. This PR times every startup step from Postgres connect through public listener bind. It does not change startup behavior. - Each step logs `Startup phase started` and `Startup phase finished`, with `phase`, `elapsed_ms`, and `status` (`succeeded` / `degraded` / `failed`, the same values `buzz_process_lifecycle` records use). - Gauge `buzz_startup_phase_current{phase}` is `1` while a step runs and `0` after it ends. A pod that is still starting shows exactly which step it is on. - Gauge `buzz_startup_phase_seconds{phase}` records each finished step's duration. - The startup NIP-43 membership reconcile logs `NIP-43 startup reconciliation progress` every 5,000 communities, with `done`, `total`, `republished`, `failed`, and `elapsed_ms`. These are gauges rather than counters on purpose. Each value is written once per boot, and a monotonic-counter scrape drops a series' first sample, so a one-shot counter never shows up. ## Contract change `deploy/charts/buzz/README.md` ("Early-startup telemetry contract") said the early lifecycle phases are log-only. That still holds: `crypto_init` through `metrics_bind` are unchanged and emit no metrics. The README now also documents the post-bind steps. The guard in `boot_lifecycle.rs` now allows exactly the two `buzz_startup_phase_*` families. It still rejects any other startup, boot, or lifecycle family, and it also rejects any exported label value that names an early phase. ## Design - `crates/buzz-relay/src/startup_steps.rs` adds a closed `StartupStep` vocabulary (18 steps) and a `StepTimer` guard. - A step ends exactly once: - `finish()` → `succeeded` - `degrade()` then `finish()` → `degraded` (a non-fatal error branch) - dropped without finishing → `failed` (covers `?` early returns and panics) - The NIP-43 and large-roster reconciles now return `ReconcileSummary { repaired, failed }`. A sweep where any item fails ends its step `degraded`, and the final log line includes the `failed` count. NIP-FI warm ends `degraded` when any issuer did not warm. The deprecated `reconcile_nip43_membership_snapshots` wrapper still returns `repaired`. - Steps that are disabled by config (migrate, audit connect, NIP-FI warm, git probe, NIP-43 reconcile) are not timed, so their series never appear. - The gauges are set once, so they age out after the existing gauge idle timeout (2700 s with default intervals; logged as `idle_timeout_secs`). A step that runs longer than that loses its `_current` series. The README says so. - Progress logging runs only for the `Bootstrap` reconcile. The 60 s maintenance sweep stays quiet. ## What got simpler Little got simpler in the code: this mostly adds instrumentation. One CI gap closed along the way: the 54 `config` tests and the 9 non-ignored `boot_lifecycle` tests, which ran in no CI lane, now run in `test-unit`. The instrumentation is scoped as small as I could make it, and it is the evidence needed for the follow-up that should delete code: moving the per-community NIP-43 reconcile off the pre-bind path, after which the 900 s startup-probe override can go back to the chart default. ## Testing - `startup_steps` unit tests (`DebuggingRecorder`): - a running step is current and has no duration - a finished step clears current and records its elapsed time - dropped and degraded steps clear current and record a duration - log capture: `finish`, `degrade`+`finish`, drop, and panic each emit exactly one `Startup phase finished` record with the right `status`. Mutating `degrade()` to a no-op and removing the ends-once guard makes these tests fail. - the vocabulary is unique and shares no names with the early lifecycle phases - Tests that capture logs or check callsite interest now share `test_support::tracing_dispatch_lock()`. These are the `startup_steps` helpers, two `telemetry` tests, and the `config` and `bridge` log-capture helpers. tracing-core caches each callsite's interest across the whole process. Without the lock, the new panic test lost its record in 60 of 150 parallel runs of `telemetry:: startup_steps:: config::`. With the lock it failed 0 of 300. Removing the lock from `gauges()` brings the failures back (53 of 100 runs). - `nip43_progress_logs_every_interval_inside_the_startup_sweep_only` checks the cadence at 79,942 communities (15 progress lines, none at the start or end) and table-tests that the maintenance sweep never logs progress. - `boot_lifecycle::successful_main_emits_complete_lifecycle_and_reports_current_startup_step` boots the real relay binary against a TCP peer that never speaks Postgres. It asserts that `/metrics` shows `buzz_startup_phase_current{phase="db_connect"} 1` and no finished-step durations, and that no early phase leaks into metrics. - **CI routing.** `just test-unit` picks relay tests by hand-maintained filter. Before this PR, CI's filters selected none of the `startup_steps`, `telemetry`, `config` or non-ignored `boot_lifecycle` tests above; the NIP-43 cadence test is picked up by the existing `handlers::side_effects::tests::` clause. Its infra-free relay stanza now selects `startup_steps::tests::`, `telemetry::tests::` and `config::tests::` (anchored so `nip_fi_config` is not selected twice), and a new `cargo nextest run -p buzz-relay --test boot_lifecycle` line runs the non-ignored `boot_lifecycle` tests (they spawn the real binary and need no Postgres or Redis; the Postgres cases stay `#[ignore]`d). `scripts/run-tests.sh` mirrors both, running `telemetry` and `config` single-threaded because they mutate process env. ### Manual check 1. `just relay` (or run `buzz-relay` against local Postgres and Redis). 2. Once the relay is up: `curl -s localhost:9102/metrics | grep buzz_startup_phase`. Every step that ran has a `_seconds` value, and every `_current` value is 0. 3. In the relay logs, `"Startup phase finished"` lines show each step's `elapsed_ms`. --------- Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
1 parent bd1ff00 commit e097c42

13 files changed

Lines changed: 656 additions & 17 deletions

File tree

‎Justfile‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -481,8 +481,12 @@ test-unit:
481481
# (JWKS source + refresh-loop composition), which `--lib` cannot reach
482482
# because they live in the binary target; the nested
483483
# `tests::postgres_tests::` stays in the PostgreSQL lane.
484+
# The fourth clause adds the startup step timers (startup_steps) and the
485+
# env- and log-capture tests in telemetry and config. All three modules
486+
# are infra-free; config parses env into a struct and never connects.
487+
# `^config::` is anchored so it does not also select nip_fi_config.
484488
cargo nextest run -p buzz-relay --lib --bin buzz-relay \
485-
-E 'test(/^api::admin::/) + test(/^handlers::channel_authz::/) + test(/^handlers::moderation_authz::/) + test(/^handlers::side_effects::tests::/) + test(/^storage_sweep::tests::/) + test(/^nip_fi_core::tests::/) + test(/^nip_fi_http::tests::/) + test(/^nip_fi_config::tests::/) + test(/^readiness::tests::/) + test(/^router::tests::/) + test(/^api::parse_query_tests::/) + test(/^api::git::transport::off_mode_precedence_tests::/) + test(/^audio::join::tests::/) + test(/^audio::handler::tests::/) + test(/^nip_fi_gate::tests::/) + test(/^nip_fi_session::tests::/) + test(/^nip_fi_shadow::tests::/) + test(/^nip_fi_shadow_session::tests::/) + test(=state::tests::neither_a_confirmed_inactive_community_nor_a_failed_lookup_admits_the_socket) + test(=handlers::req::tests::timed_out_historical_read_deregisters_before_closed) + test(=handlers::req::tests::superseded_timeout_leaves_replacement_intact) + test(=handlers::req::tests::search_claim_retires_live_and_yields_to_replacement) + test(=handlers::req::tests::concurrent_claims_and_stale_teardowns_keep_the_last_owner) + test(=handlers::req::tests::timeout_closed_is_emitted_before_a_replacement_can_claim) + test(=handlers::req::tests::revoke_then_replacement_keeps_replacement_whole) + test(=handlers::req::tests::claims_after_connection_cleanup_are_refused) + test(=handlers::req::tests::dropped_terminal_frame_cancels_connection) + test(=handlers::req::tests::revoke_dropped_terminal_frame_cancels_connection) + (kind(bin) & (test(/^tests::/) + test(/^composition_tests::/) + test(/^env_filter_tests::/)) - test(/^tests::postgres_tests::/))'
489+
-E 'test(/^api::admin::/) + test(/^handlers::channel_authz::/) + test(/^handlers::moderation_authz::/) + test(/^handlers::side_effects::tests::/) + test(/^storage_sweep::tests::/) + test(/^nip_fi_core::tests::/) + test(/^nip_fi_http::tests::/) + test(/^nip_fi_config::tests::/) + test(/^readiness::tests::/) + test(/^router::tests::/) + test(/^api::parse_query_tests::/) + test(/^api::git::transport::off_mode_precedence_tests::/) + test(/^audio::join::tests::/) + test(/^audio::handler::tests::/) + test(/^nip_fi_gate::tests::/) + test(/^nip_fi_session::tests::/) + test(/^nip_fi_shadow::tests::/) + test(/^nip_fi_shadow_session::tests::/) + test(/^startup_steps::tests::/) + test(/^telemetry::tests::/) + test(/^config::tests::/) + test(=state::tests::neither_a_confirmed_inactive_community_nor_a_failed_lookup_admits_the_socket) + test(=handlers::req::tests::timed_out_historical_read_deregisters_before_closed) + test(=handlers::req::tests::superseded_timeout_leaves_replacement_intact) + test(=handlers::req::tests::search_claim_retires_live_and_yields_to_replacement) + test(=handlers::req::tests::concurrent_claims_and_stale_teardowns_keep_the_last_owner) + test(=handlers::req::tests::timeout_closed_is_emitted_before_a_replacement_can_claim) + test(=handlers::req::tests::revoke_then_replacement_keeps_replacement_whole) + test(=handlers::req::tests::claims_after_connection_cleanup_are_refused) + test(=handlers::req::tests::dropped_terminal_frame_cancels_connection) + test(=handlers::req::tests::revoke_dropped_terminal_frame_cancels_connection) + (kind(bin) & (test(/^tests::/) + test(/^composition_tests::/) + test(/^env_filter_tests::/)) - test(/^tests::postgres_tests::/))'
486490
# Note on audio::join::tests scope: the full suite is infra-free (no DB,
487491
# no Redis). The infra-free audio/FI regression witnesses — bootstrap
488492
# ordering barrier, CommitConfirmed arm, pending-close invisibility,
@@ -545,6 +549,12 @@ test-unit:
545549
+ test(=state::tests::disconnect_community_wins_reason_losing_nip_fi_does_not_enqueue_frame)
546550
+ test(=state::tests::manager_disconnect_sets_reason_enqueues_frame_then_cancels)
547551
+ test(/^api::nip_fi::/)'
552+
# boot_lifecycle spawns the real relay binary and asserts its startup
553+
# lifecycle, including that buzz_startup_phase_* reaches /metrics. Its
554+
# non-ignored tests need no Postgres or Redis; the Postgres cases are
555+
# #[ignore]d and stay in the PostgreSQL lane. Keep
556+
# scripts/run-tests.sh in step.
557+
cargo nextest run -p buzz-relay --test boot_lifecycle
548558
# ACP author-gate and queue tests protect the trust boundary between
549559
# relay events and agent prompts. They are infra-free; ignored lifecycle
550560
# tests remain excluded and run in their dedicated integration lanes.

‎crates/buzz-relay/src/api/bridge.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4831,6 +4831,7 @@ pub(crate) mod postgres_tests {
48314831
pubkey_hex: &str,
48324832
body: &[u8],
48334833
) -> (axum::http::StatusCode, String) {
4834+
let _tracing = crate::test_support::tracing_dispatch_lock();
48344835
let buf = Arc::new(Mutex::new(Vec::<u8>::new()));
48354836
let make_writer = CapturingMakeWriter {
48364837
buf: Arc::clone(&buf),

‎crates/buzz-relay/src/config.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1627,6 +1627,8 @@ mod tests {
16271627
) -> (Result<Config, ConfigError>, String) {
16281628
use std::sync::{Arc, Mutex};
16291629

1630+
let _tracing = crate::test_support::tracing_dispatch_lock();
1631+
16301632
#[derive(Clone)]
16311633
struct CapturingMakeWriter {
16321634
buf: Arc<Mutex<Vec<u8>>>,

‎crates/buzz-relay/src/handlers/side_effects.rs‎

Lines changed: 83 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -3112,6 +3112,15 @@ pub enum Nip43ReconciliationPurpose {
31123112
Maintenance,
31133113
}
31143114

3115+
/// Per-item counts from one snapshot reconciliation sweep.
3116+
#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)]
3117+
pub struct ReconcileSummary {
3118+
/// Snapshots rebuilt and republished.
3119+
pub repaired: usize,
3120+
/// Items whose reconciliation failed; each was logged and counted.
3121+
pub failed: usize,
3122+
}
3123+
31153124
/// Preserve the original maintenance reconciliation API for downstream callers.
31163125
#[deprecated(note = "use reconcile_nip43_membership_snapshots_with_purpose")]
31173126
pub async fn reconcile_nip43_membership_snapshots(state: &Arc<AppState>) -> anyhow::Result<usize> {
@@ -3120,20 +3129,34 @@ pub async fn reconcile_nip43_membership_snapshots(state: &Arc<AppState>) -> anyh
31203129
Nip43ReconciliationPurpose::Maintenance,
31213130
)
31223131
.await
3132+
.map(|summary| summary.repaired)
31233133
}
31243134

31253135
/// Reconcile NIP-43 snapshots with explicit startup or maintenance attribution.
31263136
pub async fn reconcile_nip43_membership_snapshots_with_purpose(
31273137
state: &Arc<AppState>,
31283138
purpose: Nip43ReconciliationPurpose,
3129-
) -> anyhow::Result<usize> {
3139+
) -> anyhow::Result<ReconcileSummary> {
31303140
let communities = match purpose {
31313141
Nip43ReconciliationPurpose::Bootstrap => state.db.bootstrap_community_hosts().await?,
31323142
Nip43ReconciliationPurpose::Maintenance => state.db.active_community_hosts().await?,
31333143
};
31343144
let mut reconciled = 0usize;
3135-
3136-
for community in communities {
3145+
let mut failed = 0usize;
3146+
let total = communities.len();
3147+
let started = std::time::Instant::now();
3148+
3149+
for (index, community) in communities.into_iter().enumerate() {
3150+
if nip43_progress_due(purpose, index, total) {
3151+
info!(
3152+
done = index,
3153+
total,
3154+
republished = reconciled,
3155+
failed,
3156+
elapsed_ms = u64::try_from(started.elapsed().as_millis()).unwrap_or(u64::MAX),
3157+
"NIP-43 startup reconciliation progress"
3158+
);
3159+
}
31373160
let community_id = buzz_core::CommunityId::from_uuid(community.id);
31383161
let host = community.host;
31393162
let result = async {
@@ -3171,6 +3194,7 @@ pub async fn reconcile_nip43_membership_snapshots_with_purpose(
31713194
Ok(true) => reconciled += 1,
31723195
Ok(false) => {}
31733196
Err(error) => {
3197+
failed += 1;
31743198
metrics::counter!("buzz_nip43_membership_reconciliation_failures_total")
31753199
.increment(1);
31763200
warn!(%community_id, %host, %error, "NIP-43 membership reconciliation failed");
@@ -3179,7 +3203,24 @@ pub async fn reconcile_nip43_membership_snapshots_with_purpose(
31793203
}
31803204

31813205
metrics::counter!("buzz_nip43_membership_reconciliations_total").increment(reconciled as u64);
3182-
Ok(reconciled)
3206+
Ok(ReconcileSummary {
3207+
repaired: reconciled,
3208+
failed,
3209+
})
3210+
}
3211+
3212+
/// Communities between startup NIP-43 reconciliation progress logs.
3213+
const NIP43_BOOTSTRAP_PROGRESS_INTERVAL: usize = 5_000;
3214+
3215+
/// Whether a sweep logs progress before processing community `done` of
3216+
/// `total`. Only the startup sweep logs (the 60 s maintenance sweep stays
3217+
/// quiet): every [`NIP43_BOOTSTRAP_PROGRESS_INTERVAL`] communities, never at
3218+
/// the start (nothing done yet) and never past the end.
3219+
fn nip43_progress_due(purpose: Nip43ReconciliationPurpose, done: usize, total: usize) -> bool {
3220+
matches!(purpose, Nip43ReconciliationPurpose::Bootstrap)
3221+
&& done > 0
3222+
&& done < total
3223+
&& done.is_multiple_of(NIP43_BOOTSTRAP_PROGRESS_INTERVAL)
31833224
}
31843225

31853226
/// Publish a kind:13534 relay membership list event (NIP-43).
@@ -3315,7 +3356,7 @@ pub async fn publish_nip43_member_removed(
33153356
/// never resolves a channel against a neighboring tenant.
33163357
pub async fn reconcile_large_channel_member_snapshots(
33173358
state: &Arc<AppState>,
3318-
) -> anyhow::Result<usize> {
3359+
) -> anyhow::Result<ReconcileSummary> {
33193360
const LEGACY_ROSTER_LIMIT: i64 = 1_000;
33203361

33213362
let relay_pubkey = state.relay_keypair.public_key();
@@ -3328,6 +3369,7 @@ pub async fn reconcile_large_channel_member_snapshots(
33283369
.await?;
33293370
let relay_pubkey_hex = relay_pubkey.to_hex();
33303371
let mut reconciled = 0usize;
3372+
let mut failed = 0usize;
33313373

33323374
for candidate in candidates {
33333375
let result = async {
@@ -3352,6 +3394,7 @@ pub async fn reconcile_large_channel_member_snapshots(
33523394
Ok(true) => reconciled += 1,
33533395
Ok(false) => {}
33543396
Err(error) => {
3397+
failed += 1;
33553398
metrics::counter!("buzz_channel_roster_reconciliation_failures_total").increment(1);
33563399
warn!(
33573400
community_id = %candidate.community_id,
@@ -3365,7 +3408,10 @@ pub async fn reconcile_large_channel_member_snapshots(
33653408
}
33663409

33673410
metrics::counter!("buzz_channel_roster_reconciliations_total").increment(reconciled as u64);
3368-
Ok(reconciled)
3411+
Ok(ReconcileSummary {
3412+
repaired: reconciled,
3413+
failed,
3414+
})
33693415
}
33703416

33713417
/// Reconcile channels that exist in the DB but don't have kind:39000 events.
@@ -3799,6 +3845,37 @@ pub async fn publish_nipia_unarchived(
37993845
mod tests {
38003846
use super::*;
38013847

3848+
#[test]
3849+
fn nip43_progress_logs_every_interval_inside_the_startup_sweep_only() {
3850+
use Nip43ReconciliationPurpose::{Bootstrap, Maintenance};
3851+
const STEP: usize = NIP43_BOOTSTRAP_PROGRESS_INTERVAL;
3852+
let total = 79_942;
3853+
let due: Vec<usize> = (0..total)
3854+
.filter(|&done| nip43_progress_due(Bootstrap, done, total))
3855+
.collect();
3856+
assert_eq!(due.first(), Some(&STEP));
3857+
assert_eq!(due.len(), total / STEP);
3858+
assert!(due.iter().all(|done| done.is_multiple_of(STEP)));
3859+
3860+
// (purpose, done, total, expected)
3861+
for (purpose, done, total, expected) in [
3862+
(Bootstrap, STEP, total, true),
3863+
(Maintenance, STEP, total, false),
3864+
(Bootstrap, 0, total, false),
3865+
(Bootstrap, STEP - 1, total, false),
3866+
(Bootstrap, STEP, STEP, false),
3867+
(Bootstrap, STEP, STEP + 1, true),
3868+
] {
3869+
assert_eq!(
3870+
nip43_progress_due(purpose, done, total),
3871+
expected,
3872+
"done={done} total={total} bootstrap={}",
3873+
matches!(purpose, Bootstrap)
3874+
);
3875+
}
3876+
assert!(!(0..total).any(|done| nip43_progress_due(Maintenance, done, total)));
3877+
}
3878+
38023879
#[test]
38033880
fn workflow_deletion_retry_matches_authorized_dispatch() {
38043881
let keys = nostr::Keys::generate();

‎crates/buzz-relay/src/lib.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,8 @@ pub mod push_runtime;
6262
pub mod readiness;
6363
/// Axum router construction.
6464
pub mod router;
65+
/// Timing for startup work between metrics bind and serving.
66+
pub mod startup_steps;
6567
/// Shared application state.
6668
pub mod state;
6769
pub mod storage_sweep;

0 commit comments

Comments
 (0)