Skip to content

Commit e094092

Browse files
wesbillmanCarl
andauthored
fix(channels): return complete member rosters (block#5765)
## Summary - return complete channel rosters instead of truncating at 1,000 members - chunk `event_mentions` inserts inside one transaction so large kind `39002` snapshots remain discoverable by every `p` tag - add a targeted `buzz-admin reconcile-channels --channel <uuid>` force-republish path for stale discovery snapshots - cover a 1,501-member roster, 11,000-tag mention index, and kind `39002` tag construction past member 1,000 ## Why The relay builds NIP-29 discovery and several authorization decisions from `get_members()`, but that helper silently returned only the first 1,000 active members. Desktop then counted the truncated kind `39002` event, while late members could be rejected by roster-scanning member actions. Removing the roster cap exposes PostgreSQL's 65,535 bind-parameter ceiling in mention indexing, so the insert is chunked transactionally to preserve all-or-nothing indexing. The existing reconcilers only fill missing discovery events. The targeted admin option bypasses the separately known 1,000-channel reconciliation-list ceiling and replaces an existing channel snapshot using the configured production relay key. ## Attribution This supersedes and builds on block#3166 by @LordMelkor. Thank you for identifying the roster boundary and contributing the original complete-roster and mention-index patch. The production roster/query changes and the two PostgreSQL regressions retain that work's shape; this PR rebases it onto current `main`, adds relay coverage, and adds the targeted repair operation requested for rollout. ## Validation Exact pushed head: `24d02e4f3824150ed84913c9d230e675502e5b12` - `cargo check -p buzz-db -p buzz-admin` - `cargo test -p buzz-db channel::tests::get_members_returns_full_roster_beyond_1000 -- --ignored --exact --nocapture` - `cargo test -p buzz-db feed::tests::insert_mentions_indexes_rosters_past_bind_parameter_cap -- --ignored --exact --nocapture` - `cargo test -p buzz-relay --lib handlers::side_effects::tests::group_members_snapshot_keeps_members_past_one_thousand -- --exact` - `cargo run -q -p buzz-admin -- reconcile-channels --help` - mandatory pre-push hook: branch-skew, desktop checks/typecheck/tests, mobile tests, Rust tests, and desktop Tauri checks all passed on the pushed head ## Rollout 1. Deploy the relay/backend build. 2. Run `buzz-admin reconcile-channels --channel <general-channel-uuid>` with `BUZZ_RELAY_PRIVATE_KEY` configured. 3. Verify the replacement kind `39002` roster count matches the active database membership count. No schema migration or desktop release is required. Fixes block#3156 Supersedes block#3166 --------- Signed-off-by: Wes <[email protected]> Co-authored-by: Carl <c7ebe626f000404285d3686e1dc74cc07cc60a9754a150041ba132e14bd3e2ec@buzz.block.builderlab.xyz>
1 parent 514195b commit e094092

9 files changed

Lines changed: 544 additions & 83 deletions

File tree

‎Cargo.lock‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎TESTING.md‎

Lines changed: 42 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -155,9 +155,49 @@ buzz messages thread --channel "$CHANNEL" --event "$EVENT_ID" | jq .
155155
156156
A successful run prints `{"event_id":"…","accepted":true,"message":""}` for
157157
the send, and the message body in the `get` output. `thread` returns `[]`
158-
for a leaf message — populated only after a reply comes in (see §5).
158+
for a leaf message — populated only after a reply comes in (see §6).
159159

160-
### 5. Going deeper
160+
### 5. Verify a roster beyond 1,000 members
161+
162+
Use the focused live-relay script when changing channel membership, discovery,
163+
or reconciliation. It proves the three boundaries that DB-only tests cannot:
164+
the relay-served kind 39002 includes a member at roster position 1,501, that
165+
identity can publish a channel message, and targeted reconciliation preserves
166+
its discoverability.
167+
168+
Run this only against an isolated local database. The script inserts fixture
169+
members directly, then drives discovery and messaging through the release CLI
170+
and relay. Keep the release relay from step 3 running and use its configured
171+
relay key for authoritative replacement:
172+
173+
```bash
174+
export PATH="$PWD/target/release:$PATH"
175+
export DATABASE_URL="postgres://buzz:buzz_dev@localhost:5432/buzz_roster_e2e"
176+
export BUZZ_RELAY_URL="http://localhost:3030" # match the relay from step 3
177+
export RELAY_URL="ws://localhost:3030"
178+
export BUZZ_RELAY_PRIVATE_KEY="<same key used by buzz-relay>"
179+
180+
scripts/e2e-large-channel-roster.sh
181+
```
182+
183+
Success is directly observable as four `PASS` lines. The first and fourth
184+
include a member count greater than 1,000 and the same late-member pubkey; the
185+
second includes the accepted kind 9 event ID, and the third proves targeted
186+
repair left kind 39000/39001 IDs and tags unchanged:
187+
188+
```text
189+
PASS discovery-before-republish channel=<uuid> members=1502 late_pubkey=<hex>
190+
PASS late-member-action event_id=<hex>
191+
PASS targeted-repair-preserves-metadata-and-admin-events channel=<uuid>
192+
PASS discovery-after-republish channel=<uuid> members=1502 late_pubkey=<hex>
193+
```
194+
195+
The script refuses debug binaries and refuses a `buzz` or `buzz-admin` resolved
196+
outside this checkout's `target/release`. It also requires the targeted admin
197+
operation to use `BUZZ_RELAY_PRIVATE_KEY`; never substitute an ephemeral signer
198+
for an authoritative replacement.
199+
200+
### 6. Going deeper
161201

162202
For full coverage of every CLI command (54 subcommands across 12 groups),
163203
follow [`crates/buzz-cli/TESTING.md`](crates/buzz-cli/TESTING.md).

‎crates/buzz-admin/Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,4 +35,5 @@ rustls = { version = "0.23", default-features = false, features = ["ring", "std"
3535
tracing = { workspace = true }
3636
sqlx = { workspace = true }
3737
url = { workspace = true }
38+
uuid = { workspace = true }
3839
clap = { version = "4", features = ["derive"] }

‎crates/buzz-admin/src/main.rs‎

Lines changed: 88 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -88,12 +88,17 @@ enum Command {
8888
#[command(subcommand)]
8989
command: deletions::DeletionsCommand,
9090
},
91-
/// Emit kind:39000/39002 events for channels missing them.
91+
/// Emit missing kind:39000/39001/39002 channel discovery events, or
92+
/// republish only a targeted channel's kind:39002 roster.
9293
///
93-
/// Channels created via direct SQL (seed scripts, pre-migration data) won't
94-
/// have Nostr discovery events. This command creates them so pure-nostr
95-
/// clients can see those channels. Idempotent — safe to run multiple times.
94+
/// Without `--channel`, only channels missing discovery metadata are
95+
/// reconciled. With `--channel`, only that channel's member snapshot is
96+
/// replaced; canonical metadata and admin events remain untouched.
9697
ReconcileChannels {
98+
/// Optional channel UUID to force-republish.
99+
#[arg(long)]
100+
channel: Option<String>,
101+
97102
/// Relay private key (hex) for signing events. Falls back to
98103
/// BUZZ_RELAY_PRIVATE_KEY env var. If neither is set, generates
99104
/// an ephemeral key (events will be unverifiable after restart).
@@ -156,8 +161,8 @@ async fn run(cli: Cli) -> Result<i32> {
156161
command: ProductFeedbackCommand::List { limit },
157162
} => cmd_list_product_feedback(limit).await,
158163
Command::Deletions { command } => deletions::run(command).await,
159-
Command::ReconcileChannels { relay_key } => {
160-
reconcile_channels(relay_key).await?;
164+
Command::ReconcileChannels { channel, relay_key } => {
165+
reconcile_channels(channel, relay_key).await?;
161166
Ok(0)
162167
}
163168
}
@@ -466,14 +471,26 @@ async fn resolve_admin_tenant(db: &Db) -> Result<TenantContext> {
466471
Ok(TenantContext::resolved(record.id, record.host))
467472
}
468473

469-
async fn reconcile_channels(relay_key_arg: Option<String>) -> Result<()> {
474+
async fn reconcile_channels(
475+
channel_arg: Option<String>,
476+
relay_key_arg: Option<String>,
477+
) -> Result<()> {
470478
use buzz_core::kind::KIND_NIP29_GROUP_ADMINS;
471479
use buzz_db::event::EventQuery;
472480

473481
let db = connect_db().await?;
474482

475-
// Resolve relay signing key: arg > env > ephemeral
476-
let relay_keys = match relay_key_arg.or_else(|| std::env::var("BUZZ_RELAY_PRIVATE_KEY").ok()) {
483+
// Resolve relay signing key: arg > env > ephemeral. Force-republish must
484+
// never use an ephemeral key because it replaces an existing authoritative
485+
// snapshot.
486+
let configured_relay_key =
487+
relay_key_arg.or_else(|| std::env::var("BUZZ_RELAY_PRIVATE_KEY").ok());
488+
if channel_arg.is_some() && configured_relay_key.is_none() {
489+
return Err(anyhow::anyhow!(
490+
"--channel requires --relay-key or BUZZ_RELAY_PRIVATE_KEY"
491+
));
492+
}
493+
let relay_keys = match configured_relay_key {
477494
Some(key_hex) => {
478495
Keys::parse(&key_hex).map_err(|e| anyhow::anyhow!("invalid relay key: {e}"))?
479496
}
@@ -490,7 +507,21 @@ async fn reconcile_channels(relay_key_arg: Option<String>) -> Result<()> {
490507
};
491508

492509
let tenant = resolve_admin_tenant(&db).await?;
493-
let channels = db.list_channels(tenant.community(), None).await?;
510+
let target_channel = channel_arg
511+
.as_deref()
512+
.map(uuid::Uuid::parse_str)
513+
.transpose()
514+
.map_err(|e| anyhow::anyhow!("invalid --channel UUID: {e}"))?;
515+
let channels = if let Some(target) = target_channel {
516+
vec![db
517+
.get_channel(tenant.community(), target)
518+
.await
519+
.map_err(|_| {
520+
anyhow::anyhow!("channel {target} not found in community {}", tenant.host())
521+
})?]
522+
} else {
523+
db.list_channels(tenant.community(), None).await?
524+
};
494525
if channels.is_empty() {
495526
println!("No channels in database.");
496527
return Ok(());
@@ -513,57 +544,64 @@ async fn reconcile_channels(relay_key_arg: Option<String>) -> Result<()> {
513544
.await
514545
.unwrap_or_default();
515546

516-
if !existing.is_empty() {
547+
if !existing.is_empty() && target_channel.is_none() {
517548
skipped += 1;
518549
continue;
519550
}
520551

521552
let members = db.get_members(tenant.community(), channel.id).await?;
522553

523-
// kind:39000 — channel metadata
524-
{
525-
let mut tags: Vec<Tag> = vec![Tag::parse(["d", &channel_id_str])?];
526-
tags.push(Tag::parse(["name", &channel.name])?);
527-
if let Some(ref desc) = channel.description {
528-
if !desc.is_empty() {
529-
tags.push(Tag::parse(["about", desc])?);
554+
// A targeted repair is deliberately roster-only. kind:39000 metadata
555+
// is richer than this legacy backfill builder, and kind:39001 is not
556+
// part of the stale-roster incident; replacing either can destroy
557+
// canonical state. Full backfill still creates all three event kinds
558+
// for channels with no discovery metadata.
559+
if target_channel.is_none() {
560+
// kind:39000 — channel metadata
561+
{
562+
let mut tags: Vec<Tag> = vec![Tag::parse(["d", &channel_id_str])?];
563+
tags.push(Tag::parse(["name", &channel.name])?);
564+
if let Some(ref desc) = channel.description {
565+
if !desc.is_empty() {
566+
tags.push(Tag::parse(["about", desc])?);
567+
}
530568
}
569+
if channel.visibility == "private" {
570+
tags.push(Tag::parse(["private"])?);
571+
} else {
572+
tags.push(Tag::parse(["public"])?);
573+
}
574+
if channel.channel_type == "dm" {
575+
tags.push(Tag::parse(["hidden"])?);
576+
}
577+
tags.push(Tag::parse(["closed"])?);
578+
tags.push(Tag::parse(["t", &channel.channel_type])?);
579+
580+
let event = EventBuilder::new(Kind::Custom(39000), "")
581+
.tags(tags)
582+
.sign_with_keys(&relay_keys)
583+
.map_err(|e| anyhow::anyhow!("sign kind:39000: {e}"))?;
584+
db.replace_addressable_event(tenant.community(), &event, Some(channel.id))
585+
.await?;
531586
}
532-
if channel.visibility == "private" {
533-
tags.push(Tag::parse(["private"])?);
534-
} else {
535-
tags.push(Tag::parse(["public"])?);
536-
}
537-
if channel.channel_type == "dm" {
538-
tags.push(Tag::parse(["hidden"])?);
539-
}
540-
tags.push(Tag::parse(["closed"])?);
541-
tags.push(Tag::parse(["t", &channel.channel_type])?);
542587

543-
let event = EventBuilder::new(Kind::Custom(39000), "")
544-
.tags(tags)
545-
.sign_with_keys(&relay_keys)
546-
.map_err(|e| anyhow::anyhow!("sign kind:39000: {e}"))?;
547-
db.replace_addressable_event(tenant.community(), &event, Some(channel.id))
548-
.await?;
549-
}
550-
551-
// kind:39001 — admins
552-
{
553-
let mut tags: Vec<Tag> = vec![Tag::parse(["d", &channel_id_str])?];
554-
for m in members
555-
.iter()
556-
.filter(|m| m.role == "owner" || m.role == "admin")
588+
// kind:39001 — admins
557589
{
558-
let pk = hex::encode(&m.pubkey);
559-
tags.push(Tag::parse(["p", &pk, &m.role])?);
590+
let mut tags: Vec<Tag> = vec![Tag::parse(["d", &channel_id_str])?];
591+
for m in members
592+
.iter()
593+
.filter(|m| m.role == "owner" || m.role == "admin")
594+
{
595+
let pk = hex::encode(&m.pubkey);
596+
tags.push(Tag::parse(["p", &pk, &m.role])?);
597+
}
598+
let event = EventBuilder::new(Kind::Custom(KIND_NIP29_GROUP_ADMINS as u16), "")
599+
.tags(tags)
600+
.sign_with_keys(&relay_keys)
601+
.map_err(|e| anyhow::anyhow!("sign kind:39001: {e}"))?;
602+
db.replace_addressable_event(tenant.community(), &event, Some(channel.id))
603+
.await?;
560604
}
561-
let event = EventBuilder::new(Kind::Custom(KIND_NIP29_GROUP_ADMINS as u16), "")
562-
.tags(tags)
563-
.sign_with_keys(&relay_keys)
564-
.map_err(|e| anyhow::anyhow!("sign kind:39001: {e}"))?;
565-
db.replace_addressable_event(tenant.community(), &event, Some(channel.id))
566-
.await?;
567605
}
568606

569607
// kind:39002 — members

‎crates/buzz-db/src/channel.rs‎

Lines changed: 86 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -687,7 +687,12 @@ pub async fn membership_pairs(
687687
.collect()
688688
}
689689

690-
/// Returns all active members of the given channel.
690+
/// Returns all active members of the given channel, ordered by `joined_at`.
691+
///
692+
/// The roster is returned in full and is never truncated: callers use it to
693+
/// build the kind 39002 (NIP-29 group members) snapshot and to resolve actor
694+
/// roles for admin-event authorization, so a partial list silently hides late
695+
/// joiners from channel discovery and makes them read as non-members.
691696
///
692697
/// Returns an empty list if the channel has been soft-deleted.
693698
pub async fn get_members(
@@ -702,7 +707,6 @@ pub async fn get_members(
702707
JOIN channels c ON cm.community_id = c.community_id AND cm.channel_id = c.id AND c.deleted_at IS NULL
703708
WHERE cm.community_id = $1 AND cm.channel_id = $2 AND cm.removed_at IS NULL
704709
ORDER BY cm.joined_at ASC
705-
LIMIT 1000
706710
"#,
707711
)
708712
.bind(community_id.as_uuid())
@@ -1532,7 +1536,7 @@ mod tests {
15321536
use crate::user::{ensure_user, set_agent_owner};
15331537
use nostr::Keys;
15341538

1535-
const TEST_DB_URL: &str = "postgres://buzz:buzz_dev@localhost:5432/buzz";
1539+
const TEST_DB_URL: &str = "postgres://buzz:buzz_dev@localhost:5432/buzz"; // sadscan:disable np.postgres.1 -- local test-only credentials
15361540

15371541
async fn setup_pool() -> PgPool {
15381542
PgPool::connect(TEST_DB_URL)
@@ -1933,6 +1937,85 @@ mod tests {
19331937
assert_eq!(channel_ids.len(), channel_count as usize);
19341938
}
19351939

1940+
/// `get_members` must return the complete roster, not a truncated prefix.
1941+
///
1942+
/// The relay builds the kind 39002 (NIP-29 group members) snapshot and every
1943+
/// admin role lookup from this list, so a cap silently hides late joiners:
1944+
/// their clients never discover the channel, and an owner past the cutoff
1945+
/// reads as a non-member.
1946+
#[tokio::test]
1947+
#[ignore = "requires Postgres"]
1948+
async fn get_members_returns_full_roster_beyond_1000() {
1949+
let database_url =
1950+
std::env::var("BUZZ_TEST_DATABASE_URL").unwrap_or_else(|_| TEST_DB_URL.to_string());
1951+
let pool = PgPool::connect(&database_url)
1952+
.await
1953+
.expect("connect to test DB");
1954+
let community_id = make_test_community(&pool).await;
1955+
let community = CommunityId::from_uuid(community_id);
1956+
let creator = random_pubkey();
1957+
1958+
// create_test_channel also inserts the creator as the first (owner) member.
1959+
let channel = create_test_channel(
1960+
&pool,
1961+
community_id,
1962+
"high-volume-roster",
1963+
ChannelType::Stream,
1964+
ChannelVisibility::Open,
1965+
None,
1966+
&creator,
1967+
None,
1968+
)
1969+
.await
1970+
.expect("create test channel");
1971+
1972+
// Bulk-insert additional members with strictly increasing `joined_at`, so
1973+
// member N lands at roster position N (the creator holds position 0).
1974+
// The final member is an owner joining well past the old 1000-row cutoff.
1975+
let extra_members = 1_500;
1976+
sqlx::query(
1977+
r#"
1978+
INSERT INTO channel_members (community_id, channel_id, pubkey, role, joined_at)
1979+
SELECT
1980+
$1,
1981+
$2,
1982+
decode(lpad(to_hex(n), 64, '0'), 'hex'),
1983+
(CASE WHEN n = $3 THEN 'owner' ELSE 'member' END)::member_role,
1984+
NOW() + (n || ' seconds')::interval
1985+
FROM generate_series(1, $3) n
1986+
"#,
1987+
)
1988+
.bind(community_id)
1989+
.bind(channel.id)
1990+
.bind(extra_members)
1991+
.execute(&pool)
1992+
.await
1993+
.expect("insert high-volume channel members");
1994+
1995+
let members = get_members(&pool, community, channel.id)
1996+
.await
1997+
.expect("load channel members");
1998+
1999+
assert_eq!(
2000+
members.len(),
2001+
extra_members as usize + 1,
2002+
"get_members truncated the roster"
2003+
);
2004+
2005+
// The last joiner sits at the final roster position — past any
2006+
// 1000-row cap — which also pins the documented `joined_at` ordering.
2007+
let late_owner = hex::decode(format!("{:064x}", extra_members)).expect("hex pubkey");
2008+
let late = members.last().expect("roster is non-empty");
2009+
assert_eq!(
2010+
late.pubkey, late_owner,
2011+
"member who joined after the 1000th must be present and ordered last"
2012+
);
2013+
assert_eq!(
2014+
late.role, "owner",
2015+
"role of a late-joining owner must resolve correctly"
2016+
);
2017+
}
2018+
19362019
/// A random non-admin, non-owner user cannot remove someone else's bot.
19372020
#[tokio::test]
19382021
#[ignore = "requires Postgres"]

0 commit comments

Comments
 (0)