Skip to content

[bug] NIP-43 maintenance sweep enumerates tombstoned communities, logging "community write fenced" every ~60s forever after a completed deletion #7558

Closed

Description

@mitchallen

Describe the bug

After a whole-community deletion completes successfully via buzz-admin deletions, the relay's periodic NIP-43 membership reconciliation sweep keeps enumerating the deleted community's tombstone row and attempting a guarded write against it. The community's own deletion fence refuses the write, and the relay logs a WARN once per maintenance cycle — roughly every 60 seconds — indefinitely.

The deletion itself is fine. Every checkpoint completed, all fenced tables were purged, and verify_cross_store_absence passed. The row simply remains in communities as deletion_state = 'tombstone' while the request sits at stage retention_pending, awaiting physical retention expiry — which is by design. The sweep just doesn't know to skip it.

Root cause looks like the community enumeration helper, which selects every row with no deletion_state predicate:

// crates/buzz-db/src/store/usage.rs:399  (identical on main @ 051c3a27)
let rows = sqlx::query_as::<_, (Uuid, String)>("SELECT id, host FROM communities")

Both usage_community_hosts() (Maintenance) and bootstrap_community_hosts() (Bootstrap) route through community_hosts_with_operation, so reconcile_nip43_membership_snapshots_with_purpose in crates/buzz-relay/src/handlers/side_effects.rs receives tombstoned communities and hands each to publish_nip43_membership_list, which the fence then rejects.

Steps to reproduce

  1. Have two communities on a relay, one stranded/empty (ours came from a RELAY_URL authority change — minime.local:3000 vs minime:3000).
  2. Delete the stranded one through the supported control plane: buzz-admin deletions submit, then approve, then run. Confirm it reaches a completed state with all checkpoints completed.
  3. Leave the relay running and watch the logs: docker logs --tail 2000 <relay> | grep "NIP-43 membership reconciliation failed".
  4. A WARN appears every ~60s, forever. Restarting the relay does not clear it — the row is still in communities.

Expected behavior

The maintenance sweep should only reconcile communities that are actually serving — i.e. filter to deletion_state = 'active' (excluding quiescing, fenced, and tombstone), either in community_hosts_with_operation or in the sweep itself. A community that has been deliberately deleted and fenced should not be a reconciliation target, and failing to write to it is expected rather than exceptional.

Version and platform

  • Buzz version: relay at commit 3c7f288c60d67df78577b237e27c3dfc8831aaa1, image ghcr.io/block/buzz@sha256:919280d68dc2b61d91d92136e9ab19566c40b6e1afc1937df8986d73abe2de2c. Verified still present on main @ 051c3a270be9 by reading both files — no fix in between.
  • OS: macOS 26.6.2 (Apple Silicon), Docker inside a Colima VM. Relay deployed from deploy/compose.

Logs / additional context

{"timestamp":"2026-09-10T14:01:29.922464Z","level":"WARN","message":"NIP-43 membership reconciliation failed","community_id":"43b71a51-c183-40c9-a6e1-7f223271f2c7","host":"minime.local:3000","error":"database error: error returned from database: community write fenced: community 43b71a51-c183-40c9-a6e1-7f223271f2c7 generation 1","target":"buzz_relay::handlers::side_effects"}

Cadence measured over the last 2,000 log lines: 278 occurrences, inter-arrival gaps min 59.6s / max 60.0s / avg 59.9s. First occurrence 2026-09-06T03:34:36Z, which is 59 seconds after the deletion committed at 2026-09-06T03:33:37Z.

Community rows:

                  id                  |       host        | deletion_state | gen |          deleted_at
--------------------------------------+-------------------+----------------+-----+-------------------------------
 43b71a51-c183-40c9-a6e1-7f223271f2c7 | minime.local:3000 | tombstone      |   1 | 2026-09-06 03:33:37.066095+00
 0c170e88-e9b7-451f-ba33-137131e71318 | minime:3000       | active         |   0 |

Impact is limited but not zero: unbounded WARN noise in relay logs, a monotonically climbing buzz_nip43_membership_reconciliation_failures_total, and — the reason I'd argue it's worth fixing — it permanently poisons that metric and log channel as a signal, so a genuine reconciliation failure on a live community is much harder to notice. The active community is unaffected; health checks, NIP-42 auth and normal client traffic are all fine throughout.

This appears distinct from #5433, which concerns reconciliation failing for a live community after events are deleted (causing channel-creation 403s). This one is about a deliberately deleted community that should no longer be swept at all.

Activity

  1. Squalthor commented on Sep 14, 2026

    @Squalthor

    Independent reproduction on a self-hosted deployment (confirms this is not environment-specific):

    Setup

    • arm64 host (Raspberry Pi 4), image ghcr.io/block/buzz:main pinned at 4cd82f5 (relay reports version 0.2.1)
    • Community deleted through buzz-admin deletions (submit -> run); request reached the terminal stage retention_pending, and the communities row is deletion_state = 'tombstone' with deleted_at set
    • All community_deletion_checkpoints completed, including postgres_purged and logically_verified; 0 channels / 0 events / 0 relay members remain for that community

    Observed

    • WARN NIP-43 membership reconciliation failed exactly once per maintenance cycle, indefinitely, with:
      error: database error: error returned from database: community write fenced: community <uuid> generation 1 at line 3897
    • Confirmed root cause matches the analysis here: community_hosts() in crates/buzz-db/src/store/usage.rs runs SELECT id, host FROM communities with no deletion_state filter, so the tombstoned (and fenced) row is handed to publish_nip43_membership_list on every sweep and is rejected.
    • Raising BUZZ_NIP43_RECONCILE_INTERVAL_SECS only changes the log cadence (1/60s -> 1/900s); it does not stop the failure, as expected.

    Note on the filter in #7630
    deletion_state is NOT NULL DEFAULT 'active', so deletion_state = 'active' cannot drop live rows. The added archived_at IS NULL / deleted_at IS NULL conditions look correct for the same reason (both are nullable, and a live community has neither set).

    Not data-affecting: the active community on the same relay reconciles normally (its NIP-43 snapshots continue to be published), so this is log/metric noise rather than a functional break. Happy to re-test once #7630 lands.

  2. added 4 commits that reference this issue on Oct 2, 2026
    189e2b6
    7814c8e
    cd09eef
    7f6ffd5
  3. added a commit that references this issue on Oct 7, 2026
    22a41be
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions