perf(relay): keep 1 s read caches warm when the pool is saturated - #9012

…hole fleet (#9106)

## What

Adds Review-Proven Rule 9 to `AGENTS.md`: **design every client-driven
path for the whole fleet running it at once.**

Before an author adds or changes a connect, reconnect, renewal, poll, or
catch-up path, they:
- count the relay reads it costs per client, and keep reconnect to a
delta from the client's cursor
- jitter its start and back off with a cap
- prefer an event-driven refresh with a long backstop over a short poll
- on the relay, check every cache on a connect or auth path at a 0% hit
rate, with single-flight misses that survive a saturated pool

## Why

Relay load tests keep finding the same class of defect. A path is cheap
for one client, but when a rollout drain, node eviction, relay restart,
or network blip makes every client run it in the same few seconds, it
collapses the relay. Each bullet cites a merged PR that fixed an
instance: #4542, #8846, #9012, block/buzz-app#2215 and #2308.

Unlike rules 1–8, this one comes from load tests rather than review
mining, and the rule text says so. The section intro and the Reviewing
bullet now say the rules list includes non-review sources.

## What got simpler

Nothing in code. This is one rule added to the existing list. It turns a
recurring load-test finding into a check made at planning and review
time, so it doesn't keep resurfacing as a late load-test result.

## Validation

Docs-only. I checked the diff, the Markdown rendering, and every cited
PR number and title.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Luke Tornquist <[email protected]>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5.5 <[email protected]>

)

## What

Socket admission (root and audio) and the ephemeral write fence now
share one cache of "is this community active?". An entry is served for 1
s after its read checked out a pool connection, and concurrent misses
for one community share one read. Archive and deletion drop the entry
before they close the community's sockets.

## Why

Every new socket runs `is_community_active` before it can authenticate.
In a reconnect storm that means thousands of identical reads for one
community. A read that times out in the pool refuses the socket
(`community active check failed; refusing the socket`), and the client
then reconnects and adds more load. On a local cold-start reconnect rig,
this refusal count roughly tripled once #9012 freed pool capacity for
other reads. It is now one of the top causes of connect failures.

## How

- `buzz_db::Db::is_community_active_read_at` returns the answer together
with the instant its connection was checked out, following the #9012
`ReadAt` pattern. `is_community_active` now calls it, and the datastore
span stays under its existing name.
- `community_active::CommunityActiveCache` holds an entry per community
and a `SharedReads` gate. An entry is served for
`COMMUNITY_ACTIVE_MAX_AGE` (1 s) after its read. Both answers are
cached. A cached "inactive" fails safe and lasts at most 1 s after an
unarchive. Errors are not cached.
- **The archive sweep stays exact.** Admission registers a socket before
it checks, so the sweep closes every socket registered before it. A
socket that registers after the sweep must see the community as
inactive. `apply_community_disconnect` invalidates the entry before the
fence and sweep. Invalidation bumps an epoch, and a fill whose read
began before the bump is not published. A post-sweep socket therefore
reads afresh. It cannot take a pre-archive answer from the cache or from
a shared in-flight read. Host invalidations (archive and unarchive
writes, local and received) also drop the entry.
- **Bound when the notification is missed.** A pod that misses the Redis
disconnect admits sockets from a cached "active" for at most 1 s after
the archive commits. Lifecycle revalidation then closes them. This
matches the 1 s host-routing bound in `docs/community-auth-upgrades.md`
("archive and deletion … cached tenant binding can lag those changes by
up to one second"). Deletion keeps its own write fence
(`guard_community_write`).
- Lifecycle revalidation and persistent ingest still read uncached.

## What got simpler

The ephemeral fence's separate `serving_active_cache` is gone. It ran
the same SQL predicate through `is_serving_active`, used a 10 s TTL and
had no invalidation. The connect path, the WebSocket ephemeral path and
HTTP typing now share one cache with one 1 s bound and one invalidation
path. That leaves one fewer cache concept, and the ephemeral fence goes
from 10 s stale to at most 1 s.

## Tests

- `community_active::tests` (unit):
- 16 concurrent misses cost one read, and an entry past its max age is
re-read.
- An invalidation during an in-flight read fences that fill. A waiter
that joined the read reads afresh and gets "inactive".
  - Errors are not cached.
-
`connection::tests::bootstrap_admits_from_the_cached_community_answer`:
with the database unreachable, a cached "active" admits a socket through
the real `handle_connection`, and the socket gets its AUTH challenge.
- `postgres_tests::cache_misses_behind_a_held_pool_share_one_fresh_read`
now also covers this cache. With a one-connection pool held past every
max age, eight concurrent admission checks cost one DB read, and the
entry lands fresh.
-
`postgres_tests::archived_community_is_refused_after_invalidation_or_max_age`:
after an applied archive, the next check refuses at once. An archive
this pod never hears about refuses once the entry passes 1 s.
- The ephemeral and typing fence tests now seed the new cache.
- Mutation-checked on Blox. Each of these turns a test red: dropping the
epoch fence, the max-age filter, the single-flight gate, the connect
path's cached read, or the disconnect invalidation; stamping `read_at`
before the pool wait; and calling the uncached read on the ephemeral
path.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Luke Tornquist <[email protected]>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5.5 <[email protected]>

## Summary

A cold client in 280 channels can issue 280 separate membership reads,
each checking out a database connection. Cache the user's complete
active-membership set per community so those channel checks share one
read.

Keep the existing membership rules and 10-second freshness bound,
measured from connection checkout. Moka now accounts for snapshot size
with a 64 MiB weighted budget per relay. Each snapshot holds at most
16,384 UUIDs. The database pool controls query concurrency; there is no
separate 16-load limiter. Channel discovery keeps its separate filtering
rules. No schema change or migration.

### Request flow

1. Look up the community/user entry in Moka. A fresh complete snapshot
answers channel checks by binary search.
2. A fresh `PointChecks` marker sends each check to the database; it
stores no permission decision.
3. On a miss, create a lightweight, weighted loading entry. Requests for
that user share its lock.
4. Recheck after acquiring the lock. If waiting for that user’s lock
times out, use a point check.
5. Fetch at most 16,385 membership IDs. Above 16,384, retain a
`PointChecks` marker instead of a partial list.
6. Otherwise, sort the complete list into a compact immutable UUID
array.
7. Replace the loading entry through Moka. Its weigher now includes the
completed payload plus estimated bookkeeping.
8. Let Moka manage admission, eviction, and 10-second expiry. Membership
changes remove the relevant user or community entries.

Publication and invalidation share a short synchronous lock. A loader
publishes only while its original loading lock still identifies the
current entry. An old load can finish its original request, but cannot
repopulate an invalidated entry. Database failures propagate;
cancellation releases the user lock. Both database-query futures
allocate only when used, which keeps nested handler futures small. Cache
hits allocate neither query.

The budget covers estimated retained cache weight. Moka enforces it on a
best-effort basis; query buffers and snapshots held by active requests
can add memory. This is not a strict process-memory ceiling. Empty
entries and fallback markers have nonzero weight. There is no separate
memory-reservation or eviction implementation.

### Operational notes

- A newly granted membership can be absent from another relay's cached
snapshot until the Redis invalidation arrives, or until the 10-second
freshness bound expires if it is lost. Cached write checks can briefly
deny access; explicit channel subscriptions already confirm missing
membership against the database.
- `buzz_membership_cache_hits_total` counts channel checks served from a
fresh snapshot. `buzz_membership_cache_misses_total` counts bounded
full-set load attempts. Timeouts now use point checks, recorded by
`buzz_membership_cache_fallbacks_total{reason="wait_timeout"}`. The
other fallback reason is `oversize`; the `load_limit` path has been
removed. Before/after hit ratios are not directly comparable.
- Keep `channel_id` in membership invalidation messages for
compatibility with older relay instances during rolling deployments,
even though the new local cache invalidates by user.

### Related issue

Follow-up to #9012.

### Testing

**Current head: `a79a40cc5a6e4c7c611ed1b6bcbdd8b8c0490ef2`, including
main `f82944e3180e28502f5ea599ac9973aa6398ddb3`.** Commit `393b32583`
fixes the large-membership test fixture by refreshing planner statistics
after bulk seeding. Before that fix, the boundary test failed locally
and was the sole PostgreSQL test failure in GitHub CI (1,409/1,410
passed); the repaired test passes in 3.54 seconds including setup and
all checks. An isolated copy of the production schema/query at 16,384
memberships selected a quadratic join without statistics and a hash join
after `ANALYZE`; the latter executed in 10.86 ms. The cap and freshness
rules are unchanged.

At this exact head, the full regular buzz-db/buzz-relay suite passed
1,963/1,963 and the full PostgreSQL suite passed 1,310/1,310. This
includes both previously failing Git-auth cases and the repaired
membership boundary test (2.302 seconds including setup and checks).
Rebuilt signed live checks also passed. Full local `just ci` passed
(exit 0). GitHub CI is green: 79 successful checks and 31 skipped, with
none pending or failed at verification. The [verified matched A/B at
this
head](#9060 (comment))
against main `f82944e31` is complete: four 6,600-client arms in both
orders show 90.98–91.24% fewer membership-cache queries and 73.76–74.23%
fewer authorization checkouts. Auth error rates improved in both orders;
auth p99 and overall operation errors were mixed. Average relay RSS
increased 1.6–4.3%, with flat to slightly lower sampled peaks;
PostgreSQL average PSS decreased 1.7–5.2%, with mixed peaks. The
16,384-row production query executed in 24.953 ms after fixture ANALYZE.
Signed pool-4 and 4,097/16,384/16,385-membership probes passed. See the
report for raw counts, memory, provenance and limitations. The
[completed A/B at the earlier `6f0358875`
head](#9060 (comment))
is reported separately; it does not establish current-head performance.

Users above 16,384 memberships still use uncached authoritative point
checks. The higher cap reduces how many users hit that fallback; it does
not eliminate the cost. Each maximum-size list contains 256 KiB of UUIDs
plus overhead within the unchanged 64 MiB estimated cache budget.
Temporary allocations and retained readers can add memory outside that
budget.

Agent-run signed WebSocket test at
`a79a40cc5a6e4c7c611ed1b6bcbdd8b8c0490ef2`, using the rebuilt relay and
an isolated database: non-members cannot post; grants allow both private
channels; removal blocks writes and delivery from A while B continues;
re-add and resubscription restore A. Delivery checks completed in 1.776
s after removal and 0.466 s after re-add. A user grown above 5,000
memberships used snapshots with no oversize fallback. After growing
beyond 16,384, authoritative point checks still enforced removal and
re-add; four oversize fallbacks were recorded. The relay stopped and all
three test ports closed. Binary SHA-256:
`5b69a5aed07e9beef6bd6d2c5661c07fbdf1901dedb1729d27ab5e86a6b8d6a3`.

Build this head with `cargo build -p buzz-relay --bin buzz-relay` and
start an isolated local relay as described in
[TESTING.md](https://github.com/block/buzz/blob/6f0358875709961e2c694835eda32b0573cce47c/TESTING.md).
Reproduce with two accounts and two private channels: check denial
before membership, grant both, remove one, then re-add. Subscribe to
both channels during the test. At the current head, verify a user with
5,000 memberships still uses snapshots, then give the second account
more than 16,384 memberships and repeat the permission checks through
the fallback path. This was agent verification; human confirmation is
still pending.

#### Historical load test, before the memory-bound refactor

These measurements apply to `519509451`, before weighted eviction, the
per-user cap, and bounded loads. The refactored version was subsequently
measured at `6f0358875`; see the linked report above. The current-head
comparison is complete and reported separately
[above](#9060 (comment)).

Compared base `f9ef721e688847ed0c1512081160b6e42cdf92fd` with head
`519509451c10d94074f597124d19f2254f0b96d1` on a dedicated Linux
workstation. Ran base → fix, then fix → base. Each arm used 6,600
clients, 100 connects/second, six generator shards, and a 300-second
hold. Both versions used pool size 50, the same notices index, CPU
allocation, restored fixture, and load generator
(`4813b8cd15411355f05599e263ab15dd04e7afcc`). All four arms and 24 shard
reports completed; recorded binary hashes, fixture manifests, index DDL,
and shard configurations were verified.

| Metric | First pair: base | First pair: fix | Reverse pair: base |
Reverse pair: fix |
|---|---:|---:|---:|---:|
| Per-channel membership query calls | 1,319,382 | 7,205 | 1,367,525 |
7,465 |
| All membership-related query calls | 1,419,429 | 216,639 | 1,457,453 |
220,253 |
| Membership query execution time | 19.55 s | 10.55 s | 18.89 s | 10.74
s |
| Authorization pool checkouts | 1,667,944 | 457,649 | 1,690,624 |
453,910 |
| Average relay CPU, cores | 3.43 | 2.32 | 2.29 | 2.00 |
| Server authentication timeouts | 5,613 | 3,832 | 3,152 | 3,120 |
| Client authentication p99, highest shard | 8.378 s | 7.010 s | 6.349 s
| 6.606 s |

The cache reduced per-channel membership reads by 99.45%, all
membership-related calls by 84.7–84.9%, and authorization checkouts by
72.6–73.2%. Relay CPU was 12.7–32.4% lower. These runs support an
efficiency improvement, but do not establish a repeatable
authentication-latency or timeout improvement.

The pool still saturated. Mean authorization checkout wait increased
from 176.6 to 486.2 ms in the first pair and 118.6 to 442.6 ms in
reverse order. Removing cheap checkouts changes that average's
denominator; it does not isolate latency changes for equivalent
requests. Both versions retained client errors. Memory usage did not
consistently improve.

Each arm restored the fixture and started with a cold relay cache, but
Linux page cache was not dropped. The first baseline arm performed
materially worse than later arms. Operation counts differed with
reconnects and failures. Reported p99 is the maximum shard p99, not a
merged percentile. No perf profile was collected.

Reproduction with the reviewed harness (build exact commits above as
`base` and `fix`; optional delayed profiler disabled and complete
query/metric collection enabled):

```bash
RUN=pr9060_ab EXTRA_ENVS="BUZZ_DB_POOL_SIZE=50" \
ARMS="current:scen=bbcold.toml:rate=100:relay=base:sql=nidx.sql current:scen=bbcold.toml:rate=100:relay=fix:sql=nidx.sql" \
bash ~/lt-src/bb.sh

RUN=pr9060_ba EXTRA_ENVS="BUZZ_DB_POOL_SIZE=50" \
ARMS="current:scen=bbcold.toml:rate=100:relay=fix:sql=nidx.sql current:scen=bbcold.toml:rate=100:relay=base:sql=nidx.sql" \
bash ~/lt-src/bb.sh
```

Generated with Codex

---------

Signed-off-by: Ravneet Arora <[email protected]>
Co-authored-by: maincourse <37ce551b493d4a998caf030929c47bce0b6a0e43f1178b3a63c4151d7c16b76e@buzz.block.builderlab.xyz>