Skip to content

Commit deec496

Browse files
wpfleger96Duncan
andauthored
fix(relay): align NIP-11 max_limit with REQ ceiling (#3635)
Buzz's NIP-11 document advertised `limitation.max_limit: 10_000`, but the effective websocket REQ page ceiling was `1_000` — a 10x lie. The websocket REQ path never sets `EventQuery::max_limit`, so `query_events` applied its own `unwrap_or(1000)` clamp to every historical query. Only the COUNT fallback (`apply_count_fallback_limit`) ever raises that clamp. A client that trusts the advertised value asks for 10,000 events, silently receives 1,000, and — with no error and no continuation signal — reads that short page as exhaustion. Up to 9,000 events are dropped without anyone noticing. `MAX_HISTORICAL_LIMIT = 2_000` in `handlers/req.rs` was dead weight for the same reason: nothing clamped to 2,000 could survive the DB's 1,000 clamp one layer down. ## Change `buzz_db::DEFAULT_MAX_PAGE_LIMIT` (`1_000`) is now the single source of truth. It is the `query_events` clamp default, the value both REQ clamp sites use, and the value advertised as NIP-11 `max_limit`. `MAX_HISTORICAL_LIMIT` is removed rather than re-pointed — an alias for a constant used four lines away adds a name without adding meaning. The NIP-50 search path carries a second, independent bound. It clamps its emission target to the shared ceiling like any other REQ, but how many FTS candidates it will scan was bounded separately, by a bare 10-page loop over 100-hit pages. That product only coincidentally equalled the ceiling, so raising the ceiling — or shrinking a page — would shrink the scan relative to what clients may now request, degrading search quality while nothing in the code registered the change. The page count is now ceiling-divided from `DEFAULT_MAX_PAGE_LIMIT` over a named `SEARCH_PAGE_SIZE`, so the scan budget tracks the advertised ceiling by construction. That budget is a resource policy, not a delivery promise. It bounds candidates *scanned*, not events *emitted*: post-filtering (NIP-01 match, channel access, reader visibility, dedup) discards an unpredictable share of every page, so a search result smaller than the requested limit remains possible. This is not a NIP-11 violation — `max_limit` is defined as a clamp the relay applies to a requested `limit`, not a guaranteed count in the response. Two guards hold the pair together: - `req_filter_limit_clamps_to_advertised_nip11_max_limit` reads `max_limit` back out of a built `RelayInfo` and asserts the REQ path clamps to exactly that number. - `search_scan_capacity_covers_advertised_nip11_max_limit` asserts the scan budget covers exactly one advertised ceiling's worth of candidates — no less, and with no spare page of slack, so the derivation can't be quietly replaced by a hand-tuned constant that happens to pass today. ## Behavior Websocket behavior is unchanged: 1,000 was already the real ceiling on every path, including NIP-50. The advertisement now tells the truth about it. Raising the effective limit is a capacity decision and is deliberately not made here. The generic HTTP bridge's page-2+ offsets do change, as a consequence of the corrected clamp. `extract_page_offset` sizes a page from `query.limit` *before* the DB clamp applies, so an absent limit previously produced an offset of 2,000 and a requested 1,500 produced 1,500 — while the page actually returned held at most 1,000 rows. Both now produce 1,000. This corrects paging that had been skipping rows the previous page never returned; `extract_page_offset_sizes_pages_from_clamped_limit` locks it down. ## Scope note The bridge's per-endpoint ceilings — `BRIDGE_WINDOW_MAX_LIMIT` (200) for channel windows and `BRIDGE_THREAD_MAX_LIMIT` (500) for thread reads — are endpoint contracts on a non-NIP-01 transport, not values NIP-11 speaks for, and are unchanged. Fixes #3757 --------- Signed-off-by: Will Pfleger <[email protected]> Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
1 parent 119fb24 commit deec496

5 files changed

Lines changed: 147 additions & 24 deletions

File tree

‎crates/buzz-db/src/event.rs‎

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,13 @@ use buzz_core::{CommunityId, StoredEvent};
1717

1818
use crate::error::{DbError, Result};
1919

20+
/// Largest page [`query_events`] will return when [`EventQuery::max_limit`] is
21+
/// unset — the effective ceiling on any client-requested `limit`.
22+
///
23+
/// This is the value the relay advertises as NIP-11 `limitation.max_limit`, so
24+
/// the advertised ceiling and the enforced one cannot drift.
25+
pub const DEFAULT_MAX_PAGE_LIMIT: i64 = 1_000;
26+
2027
/// Optional filters for [`query_events`].
2128
#[derive(Debug, Clone)]
2229
pub struct EventQuery {
@@ -67,9 +74,9 @@ pub struct EventQuery {
6774
/// channel-less global events. Applied before SQL `LIMIT` so access-filtered
6875
/// historical pages have exact exhaustion semantics.
6976
pub channel_ids: Option<Vec<uuid::Uuid>>,
70-
/// Override the default limit clamp (1000). Used by COUNT fallback path
71-
/// which needs to fetch all matching events for post-filter counting.
72-
/// When None, the default clamp of 1000 applies.
77+
/// Override the default page clamp ([`DEFAULT_MAX_PAGE_LIMIT`]). Used by
78+
/// the COUNT fallback path, which needs to fetch all matching events for
79+
/// post-filter counting. When None, the default clamp applies.
7380
pub max_limit: Option<i64>,
7481
/// Shared-gated visibility reader: when set, append an SQL visibility
7582
/// clause for every kind in [`SHARED_GATED_KINDS`] before ORDER/LIMIT so
@@ -357,7 +364,7 @@ pub(crate) async fn query_events_on(
357364
return Ok(vec![]);
358365
}
359366

360-
let clamp = q.max_limit.unwrap_or(1000);
367+
let clamp = q.max_limit.unwrap_or(DEFAULT_MAX_PAGE_LIMIT);
361368
let limit_val = q.limit.unwrap_or(100).min(clamp);
362369
let offset_val = q.offset.unwrap_or(0);
363370

‎crates/buzz-db/src/lib.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,7 @@ pub mod user;
5555
pub mod workflow;
5656

5757
pub use error::{DbError, Result};
58-
pub use event::{EventQuery, ReactionEventInsertOutcome};
58+
pub use event::{EventQuery, ReactionEventInsertOutcome, DEFAULT_MAX_PAGE_LIMIT};
5959

6060
use chrono::{DateTime, Utc};
6161
use sqlx::postgres::{PgConnection, PgPoolOptions};

‎crates/buzz-relay/src/api/bridge.rs‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3042,6 +3042,27 @@ mod tests {
30423042
assert_eq!(extract_page_offset(&raw, None), None);
30433043
}
30443044

3045+
/// Offsets are sized from the *clamped* limit the DB will honor, not from
3046+
/// what the client asked for. `filter_to_query_params` clamps an absent or
3047+
/// over-ceiling `limit` to `DEFAULT_MAX_PAGE_LIMIT` (guarded in
3048+
/// `handlers::req::tests::req_filter_limit_clamps_to_advertised_nip11_max_limit`)
3049+
/// and that clamped value is what arrives here — so page N starts exactly
3050+
/// N-1 full pages in. Sizing from an unclamped limit would step past rows
3051+
/// the previous page never returned.
3052+
#[test]
3053+
fn extract_page_offset_sizes_pages_from_clamped_limit() {
3054+
let clamped = buzz_db::DEFAULT_MAX_PAGE_LIMIT;
3055+
3056+
assert_eq!(
3057+
extract_page_offset(&serde_json::json!({ "page": 2 }), Some(clamped)),
3058+
Some(clamped)
3059+
);
3060+
assert_eq!(
3061+
extract_page_offset(&serde_json::json!({ "page": 3 }), Some(clamped)),
3062+
Some(clamped * 2)
3063+
);
3064+
}
3065+
30453066
#[test]
30463067
fn extract_depth_limit_valid() {
30473068
let raw = serde_json::json!({ "depth_limit": 3 });

‎crates/buzz-relay/src/handlers/req.rs‎

Lines changed: 108 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,6 @@ use crate::connection::{AuthState, ConnectionState};
2222
use crate::protocol::RelayMessage;
2323
use crate::state::AppState;
2424

25-
const MAX_HISTORICAL_LIMIT: i64 = 2_000;
2625
const MAX_SUBSCRIPTIONS: usize = 1024;
2726

2827
/// Maximum `query_events` calls in flight per multi-filter REQ / bridge query.
@@ -416,10 +415,24 @@ pub async fn handle_req(
416415
);
417416
}
418417

419-
/// Handle a NIP-50 search REQ: query Postgres FTS, fetch full events, deliver results, EOSE.
420-
/// Search subscriptions are one-shot — no persistent subscription is registered.
418+
/// FTS candidate hits fetched per page. Pages are always full regardless of
419+
/// the requested limit — post-filtering discards an unpredictable share of
420+
/// hits, so the scan fetches candidates in full pages rather than sizing
421+
/// pages to the request.
422+
const SEARCH_PAGE_SIZE: u32 = 100;
423+
421424
/// Maximum FTS pages to fetch per filter (prevents unbounded loops).
422-
const MAX_SEARCH_PAGES: u32 = 10;
425+
///
426+
/// Derived from the advertised page ceiling rather than fixed: the scan
427+
/// budget is a resource policy — at most one advertised page ceiling's worth
428+
/// of candidates per filter — and deriving it keeps the budget tracking the
429+
/// ceiling if the ceiling ever moves. This bounds candidates *scanned*, not
430+
/// events *emitted*: post-filtering (NIP-01 match, channel access, reader
431+
/// visibility, dedup) can discard any number of candidates, so a result
432+
/// smaller than the requested limit remains possible and is not a NIP-11
433+
/// violation — `max_limit` promises a clamp on the request, not a count in
434+
/// the response.
435+
const MAX_SEARCH_PAGES: u32 = (buzz_db::DEFAULT_MAX_PAGE_LIMIT as u32).div_ceil(SEARCH_PAGE_SIZE);
423436

424437
/// Resolve request-local channel access, repairing a stale cache-negative.
425438
///
@@ -501,6 +514,8 @@ pub(crate) fn build_search_channel_scope_filter(
501514
})
502515
}
503516

517+
/// Handle a NIP-50 search REQ: query Postgres FTS, fetch full events, deliver results, EOSE.
518+
/// Search subscriptions are one-shot — no persistent subscription is registered.
504519
#[allow(clippy::too_many_arguments)]
505520
async fn handle_search_req(
506521
sub_id: &str,
@@ -535,8 +550,8 @@ async fn handle_search_req(
535550

536551
let limit = filter
537552
.limit
538-
.map(|l| (l as u32).min(MAX_HISTORICAL_LIMIT as u32))
539-
.unwrap_or(MAX_HISTORICAL_LIMIT as u32);
553+
.map(|l| (l as u32).min(buzz_db::DEFAULT_MAX_PAGE_LIMIT as u32))
554+
.unwrap_or(buzz_db::DEFAULT_MAX_PAGE_LIMIT as u32);
540555

541556
if limit == 0 {
542557
continue; // NIP-01: limit 0 means "no results from this filter"
@@ -583,13 +598,11 @@ async fn handle_search_req(
583598
let since = filter.since.map(|s| s.as_secs() as i64);
584599
let until = filter.until.map(|u| u.as_secs() as i64);
585600

586-
// Paginate: keep fetching pages until we've emitted `limit` results
587-
// or exhausted the search result set. This ensures post-filtering
588-
// doesn't silently reduce the result count below the requested limit.
601+
// Paginate: keep fetching pages until we've emitted `limit` results or
602+
// exhausted the search result set. Post-filtering discards an unpredictable
603+
// share of each page, so continuing past short yields gives the scan a
604+
// chance — not a guarantee — of filling the requested limit.
589605
let mut emitted: u32 = 0;
590-
// Always fetch full pages (100) regardless of limit — post-filtering
591-
// may discard many hits, so we need headroom to fill the requested limit.
592-
let per_page: u32 = 100;
593606

594607
for page in 1..=MAX_SEARCH_PAGES {
595608
if emitted >= limit {
@@ -605,7 +618,7 @@ async fn handle_search_req(
605618
since,
606619
until,
607620
page,
608-
per_page,
621+
per_page: SEARCH_PAGE_SIZE,
609622
mode: buzz_search::SearchMode::FullText,
610623
};
611624

@@ -617,9 +630,9 @@ async fn handle_search_req(
617630
}
618631
};
619632

620-
// A short page is the last page: FTS returns up to `per_page` hits,
621-
// so fewer than that means the result set is exhausted.
622-
let exhausted = search_result.hits.len() < per_page as usize;
633+
// A short page is the last page: FTS returns up to a full page of
634+
// hits, so fewer than that means the result set is exhausted.
635+
let exhausted = search_result.hits.len() < SEARCH_PAGE_SIZE as usize;
623636
let page_empty = search_result.hits.is_empty();
624637

625638
let hit_ids: Vec<[u8; 32]> =
@@ -878,8 +891,8 @@ fn filter_to_query_params(
878891
.and_then(|u| chrono::DateTime::from_timestamp(u.as_secs() as i64, 0));
879892
let limit = filter
880893
.limit
881-
.map(|l| (l as i64).min(MAX_HISTORICAL_LIMIT))
882-
.unwrap_or(MAX_HISTORICAL_LIMIT);
894+
.map(|l| (l as i64).min(buzz_db::DEFAULT_MAX_PAGE_LIMIT))
895+
.unwrap_or(buzz_db::DEFAULT_MAX_PAGE_LIMIT);
883896

884897
// Push author filter into SQL. Single-author uses the indexed `pubkey` column;
885898
// multi-author uses the `authors` IN-list pushdown added in the pure-nostr PR.
@@ -1418,6 +1431,83 @@ mod tests {
14181431
)
14191432
}
14201433

1434+
/// NIP-11 `limitation.max_limit` as this relay actually advertises it.
1435+
fn advertised_max_limit() -> i64 {
1436+
crate::nip11::RelayInfo::build(
1437+
None,
1438+
None,
1439+
false,
1440+
crate::config::DEFAULT_MAX_FRAME_BYTES,
1441+
None,
1442+
)
1443+
.limitation
1444+
.expect("limitation")
1445+
.max_limit
1446+
.expect("max_limit") as i64
1447+
}
1448+
1449+
#[test]
1450+
fn req_filter_limit_clamps_to_advertised_nip11_max_limit() {
1451+
let advertised = advertised_max_limit();
1452+
1453+
let community = buzz_core::tenant::CommunityId::from_uuid(uuid::Uuid::new_v4());
1454+
1455+
// A filter asking for more than the relay advertises is clamped down to
1456+
// exactly the advertised ceiling — the NIP-11 document is the promise,
1457+
// this is the enforcement.
1458+
let greedy = filter_to_query_params(
1459+
&Filter::new().limit(advertised as usize * 10),
1460+
None,
1461+
community,
1462+
);
1463+
assert_eq!(greedy.limit, Some(advertised));
1464+
1465+
// A filter with no `limit` gets the same ceiling, not something larger.
1466+
let unbounded = filter_to_query_params(&Filter::new(), None, community);
1467+
assert_eq!(unbounded.limit, Some(advertised));
1468+
1469+
// Neither sets `max_limit`, so `query_events` applies its own default
1470+
// clamp. That default must equal the advertised value too, or the
1471+
// clamp above would be undone one layer down.
1472+
assert_eq!(greedy.max_limit, None);
1473+
assert_eq!(unbounded.max_limit, None);
1474+
assert_eq!(buzz_db::DEFAULT_MAX_PAGE_LIMIT, advertised);
1475+
1476+
// Under-ceiling requests are honored verbatim.
1477+
let modest = filter_to_query_params(&Filter::new().limit(10), None, community);
1478+
assert_eq!(modest.limit, Some(10));
1479+
}
1480+
1481+
/// The NIP-50 search path clamps its emission target to the advertised
1482+
/// ceiling like every other REQ, but the number of candidates it will scan
1483+
/// is bounded a second time by the page budget. This pins the resource
1484+
/// policy: the budget covers exactly one advertised page ceiling's worth of
1485+
/// candidates — no less (a ceiling raise must not silently shrink the scan
1486+
/// relative to what clients may request) and no hand-tuned spare (the budget
1487+
/// must stay derived, not drift back into a magic number). It deliberately
1488+
/// does NOT claim search fills the emitted limit — post-filtering can
1489+
/// discard any number of candidates.
1490+
#[test]
1491+
fn search_scan_capacity_covers_advertised_nip11_max_limit() {
1492+
let advertised = advertised_max_limit();
1493+
let capacity = i64::from(MAX_SEARCH_PAGES) * i64::from(SEARCH_PAGE_SIZE);
1494+
1495+
assert!(
1496+
capacity >= advertised,
1497+
"NIP-50 scans at most {capacity} candidates ({MAX_SEARCH_PAGES} pages of \
1498+
{SEARCH_PAGE_SIZE}) but NIP-11 advertises {advertised} — the scan budget \
1499+
no longer covers the advertised ceiling"
1500+
);
1501+
1502+
// The budget is derived, not hand-tuned: one page under the derived
1503+
// count must be insufficient, or the ceiling could rise without the
1504+
// page count following it.
1505+
assert!(
1506+
capacity - i64::from(SEARCH_PAGE_SIZE) < advertised,
1507+
"scan budget has a spare page of slack — derive it from the ceiling"
1508+
);
1509+
}
1510+
14211511
#[test]
14221512
fn count_fallback_fetches_one_extra_candidate() {
14231513
let mut query =

‎crates/buzz-relay/src/nip11.rs‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,11 @@ pub struct RelayLimitation {
8989

9090
/// Canonical `RelayLimitation` advertised by this relay.
9191
///
92+
/// `max_limit` is [`buzz_db::DEFAULT_MAX_PAGE_LIMIT`], the same constant the
93+
/// REQ path clamps filter limits to, so the advertised ceiling and the
94+
/// enforced one cannot drift (see
95+
/// `handlers::req::tests::req_filter_limit_clamps_to_advertised_nip11_max_limit`).
96+
///
9297
/// `auth_required` is always `true`: the REQ, EVENT, and COUNT handlers
9398
/// unconditionally reject connections that are not in
9499
/// `AuthState::Authenticated`. This is independent of the REST API token
@@ -103,7 +108,7 @@ fn relay_limitation(max_message_length: usize) -> RelayLimitation {
103108
max_message_length: Some(max_message_length as u64),
104109
max_subscriptions: Some(1024),
105110
max_filters: Some(10),
106-
max_limit: Some(10_000),
111+
max_limit: Some(buzz_db::DEFAULT_MAX_PAGE_LIMIT as u32),
107112
max_subid_length: Some(256),
108113
min_pow_difficulty: None,
109114
auth_required: true,

0 commit comments

Comments
 (0)