perf(acp): reduce observer CPU and lock overhead - #9381

…small

* origin/main:
  perf(acp): reduce observer CPU and lock overhead (#9381)
  ci: fix main — stale RunCause call sites and Playwright on ubuntu-26.04 (#9418)
  fix(mobile): keep shortcut taps and the member picker from wedging (#9388)
  fix(mobile/ios): close notification code blocks only at line-ending fences (#9349)
  fix(media): run thumbnail regeneration tests in CI and cap rebuild pixels (#9198)
  fix(mobile): never show emoji shortcodes in reaction pills (#9411)
  fix(mobile): isolate channel header scroll rebuilds (#9400)

Signed-off-by: Ravneet Arora <[email protected]>

Replaces #9424 with the same release-profile change on a branch in this
repository so the review workflows can run.

## Summary

Reduce release codegen units from 16 to 4 and enable thin link-time
optimization (LTO). Local measurements show about **7% smaller
binaries**, with a **14% increase in dependency-cached release rebuild
time** (+3.1 seconds). Clean release build time was roughly flat.

`codegen-units = 4` seems like a good balance. With thin LTO fixed, one
unit cut stripped sizes a further 7–10%, but increased clean build time
by 38% and cached release rebuild time by 67.5%. CPU and Activity trials
showed no clear speedup.

Fewer codegen units let LLVM optimize larger portions of each crate
together. Lower counts typically favor runtime performance and smaller
binaries over compilation speed. Thin LTO also enables optimization
across crate boundaries. Runtime gains depend on the workload.

Release keeps `opt-level = 3`, the highest numeric optimization level
and Cargo's existing default. Development builds stay unchanged. Keep
the custom `ci` profile at 16 codegen units with LTO off, preserving its
existing settings. Sprig keeps its separate size-focused profile. CI
jobs that use ordinary `--release` will use the new settings.

These measurements cover `buzz` and `buzz-acp`, not the desktop bundle.
The desktop app has its own Cargo workspace; its bundled CLI picks up
this change after a pin update. Our end-to-end runtime trials were
broadly flat; the measurements below demonstrate the size and build-time
tradeoff.

### Related issue

N/A — no duplicate release LTO/codegen PR found. This PR contains only
the profile change and is independent of #9413.

### Testing

#### Baseline versus four codegen units

Compared identical source and lockfile at
`b4de4479485fc3d57628ed509ed40c4278e63132`, using Rust 1.98.1 on an
Apple M5 Max (18 cores, 128 GiB RAM). Both arms used `opt-level=3`,
disabled incremental compilation and no compiler-cache wrapper. The
baseline used LTO off and 16 codegen units; the candidate used thin LTO
and four.

| Median / artifact | Baseline | Thin LTO, 4 units | Change |
| --- | ---: | ---: | ---: |
| Clean release build (2 runs each) | 47.98 s | 46.71 s | -2.7% |
| Dependency-cached release rebuild (3 each) | 22.55 s | 25.68 s |
+13.9% |
| No-op release build (5 each) | 0.469 s | 0.461 s | Roughly unchanged |
| `buzz` executable | 14.516 MiB | 13.580 MiB | -6.4% |
| `buzz-acp` executable | 14.580 MiB | 13.487 MiB | -7.5% |

Build command: `cargo build --offline --locked --release -p buzz-acp -p
buzz-cli --timings`. Set `CARGO_INCREMENTAL=0`,
`CARGO_PROFILE_RELEASE_LTO=false|thin` and
`CARGO_PROFILE_RELEASE_CODEGEN_UNITS=16|4` for each arm, with separate
`CARGO_TARGET_DIR` directories. Clean runs used empty build directories
in sequential ABBA order, with downloaded dependencies and filesystem
caches warm. Rebuilds cleaned only `buzz-acp` and `buzz-cli` outside the
timed region. Sizes are Cargo-built executables before the desktop
packager's stripping step.

Stripping copies with `xcrun strip -S -x` kept the size benefit: `buzz`
fell from 12.550 to 11.805 MiB (-5.9%), and `buzz-acp` from 12.360 to
11.501 MiB (-7.0%). This does not measure a signed desktop installer.

Both release executables passed `--help` smoke checks. Profile
inspection confirmed the custom `ci` and `sprig` settings are unchanged.

#### One codegen unit versus four

A separate follow-up compared four versus one on
`a314975ce7bf9aa00baebfef874b6e69474a95cf`, which includes the merged
observer CPU/locking changes (#9381). Both arms used thin LTO,
`opt-level=3`, the same Rust 1.98.1 toolchain and hardware, and
identical source and lockfile. This is a separate source snapshot from
the first comparison; the tables should not be combined into a three-arm
comparison.

| Median / artifact | Thin LTO, 4 units | Thin LTO, 1 unit | Change |
| --- | ---: | ---: | ---: |
| Clean release build (2 runs each) | 45.93 s | 63.35 s | +37.9% |
| Dependency-cached release rebuild (3 each) | 22.38 s | 37.48 s |
+67.5% |
| Stripped `buzz` | 11.789 MiB | 10.923 MiB | -7.3% |
| Stripped `buzz-acp` | 11.485 MiB | 10.320 MiB | -10.1% |
| Activity p95 latency (median of 5 runs each) | 952 ms | 961 ms | No
clear change |

Both sets of binaries were stripped identically with `xcrun strip -S
-x`. Repeat the build command above with
`CARGO_PROFILE_RELEASE_LTO=thin` and
`CARGO_PROFILE_RELEASE_CODEGEN_UNITS=4` or `1`, retaining the same cache
and ordering controls.

All 60 CPU trials and 10 Activity trials passed. The CPU cases exercised
production packing and queue ingestion functions; their differences were
within run variation. Activity exercised real ACP subprocesses, observer
publication, signing/encryption, the browser broker and production UI
against a modeled relay. All 500 markers rendered exactly once, and all
2,250 heavy events were delivered. Its one-second publication interval
limits visible latency; these local results are not production
percentiles.

Generated with AI assistance

---------

Signed-off-by: Sanket Kanjalkar <[email protected]>
Co-authored-by: Codex <[email protected]>

Replaces #9366 on a branch in this repository and resolves its conflicts
with main.

## Summary

The ACP observer kept up to 1,000 raw events in both replay and live
buffers. Large valid messages could retain gigabytes before downstream
trimming ran.

Fit telemetry before either buffer retains it. Cap replay at 4 MiB of
serialized JSON and keep the existing live window. Its per-event limit
bounds live JSON below 64 MiB.

Preserve the CPU and locking improvements from #9381: prepare large
payloads and copy snapshots outside the replay lock. Finalize sequence
and timestamp with publication. Account for that final metadata,
including sequence digit-width changes, in the byte limit.

The tradeoffs are extra work per emit, shorter replay history for large
events, and elided oversized telemetry. Agent protocol messages remain
unchanged.

### Related issue

Fixes #7987. Existing publisher byte limits act downstream of these
buffers.

### Testing

The opt-in probe exercises the real in-process emit, snapshot and
stalled-subscriber paths:

```sh
CARGO_INCREMENTAL=0 CARGO_PROFILE_RELEASE_LTO=thin CARGO_PROFILE_RELEASE_CODEGEN_UNITS=4 \
  cargo test --release -p buzz-acp --lib observer::tests::retention_probe -- --exact --ignored --nocapture
```

Compared baseline `a314975ce` with candidate `c2df3cb56` on macOS arm64,
Rust 1.98.1. The baseline observer, fitter, workspace manifest and
lockfile match the candidate's parent `8f3f4e266`. Both arms use
`opt-level=3`, thin LTO and four codegen units, with incremental
compilation and compiler-cache wrappers off. This profile is fixed for
the comparison; this PR does not change build settings.

Values are medians of three fresh-process runs per version and workload,
in alternating order. Peak RSS comes from `/usr/bin/time -l` around each
test executable; compilation is excluded. Each version was rebuilt
separately and its executable and workload identity verified.

| Workload | Peak RSS before → after | Emit time per event before →
after |
| --- | --- | --- |
| 10,000 × 1 KiB | 14.0 → 14.1 MiB | 0.47 → 0.89 µs |
| 1,000 × 60,000 bytes | 199.4 → 80.8 MiB | 8.54 → 22.53 µs |
| 1,000 × 1 MiB | 3,025.8 → 26.4 MiB | 123.50 → 307.89 µs |

For the 1 MiB workload, replay JSON fell from 1,000.2 MiB to 3.997 MiB.
Live JSON fell from 1,000.2 MiB to 5.91 MiB. Replay retained 676 fitted
events; the live subscriber received all 1,000.

Emit time includes payload construction. Peak RSS includes snapshot and
receive copies made by the probe. JSON budgets are not exact heap
limits. These local measurements show the memory/CPU tradeoff;
concurrent producer throughput and native app latency were not
benchmarked. See [the probe
instructions](https://github.com/block/buzz/blob/c2df3cb567a3766aa796c50369903b09a8f04036/crates/buzz-acp/TESTING.md#observer-retention-measurement)
for workloads, controls and limits.

Human verification remains pending.

Generated with AI assistance

Signed-off-by: Sanket Kanjalkar <[email protected]>
Co-authored-by: Codex <[email protected]>

Co-authored-by: mboston.Engineer <932d3b6ebea6cc83e091551ae46d1c7e7f8f7d0fa8e9cef5e6facc8544af413b@buzz.block.builderlab.xyz>

* origin/main:
  fix(mobile): require Builderlab consent for existing identities (#8823)
  feat(mobile): use HTTP/2 on iOS through app-wide URLSessions (#8897)
  fix(relay): publish a git push only when refs changed; check the protocol by refinement (#9425)
  fix(mobile): show shared chats under Solo (#9348)
  chore(release): release Buzz Desktop version 0.5.29 (#9456)
  feat(mobile): render messages with gpt_markdown 1.3.4 plusparse (#8945)
  fix(acp): bound observer retention by bytes (#7987) (#9436)
  Tune release codegen for faster/smaller binaries (#9434)
  feat(desktop): add dismissible banner pointing to the new buzz app (#9379)
  fix(mobile): keep publishing presence while the app is open (#9441)
  feat(mobile): share into Buzz from the iOS share sheet (#9433)
  fix(mobile): qualify every namesake mention and keep authored mention casing (#9325)
  fix(mobile): match browser GIF timing in the iOS reaction sheet (#9437)
  fix(mobile): play short-delay animation frames at browser speed (#9432)
  buzz-search: read profile names without the profile_name_fields function (#9397)
  buzz-db: build the community deletion lock key in Rust (#9396)
  fix(cli): pair for the active community, skip stopped desktops, accept a bare relay host (#9417)
  test(mobile): run push tests without a gateway dart-define (#9423)
  fix(mobile): never pop Home when archiving from channel settings (#9387)
  perf(acp): reduce observer CPU and lock overhead (#9381)

Signed-off-by: Matthew Boston <[email protected]>

…count-backfill

Co-authored-by: mboston.Engineer <932d3b6ebea6cc83e091551ae46d1c7e7f8f7d0fa8e9cef5e6facc8544af413b@buzz.block.builderlab.xyz>

* bostonaholic--agent-reply-count:
  fix(mobile): require Builderlab consent for existing identities (#8823)
  feat(mobile): use HTTP/2 on iOS through app-wide URLSessions (#8897)
  fix(relay): publish a git push only when refs changed; check the protocol by refinement (#9425)
  fix(mobile): show shared chats under Solo (#9348)
  chore(release): release Buzz Desktop version 0.5.29 (#9456)
  feat(mobile): render messages with gpt_markdown 1.3.4 plusparse (#8945)
  fix(acp): bound observer retention by bytes (#7987) (#9436)
  Tune release codegen for faster/smaller binaries (#9434)
  feat(desktop): add dismissible banner pointing to the new buzz app (#9379)
  fix(mobile): keep publishing presence while the app is open (#9441)
  feat(mobile): share into Buzz from the iOS share sheet (#9433)
  fix(mobile): qualify every namesake mention and keep authored mention casing (#9325)
  fix(mobile): match browser GIF timing in the iOS reaction sheet (#9437)
  fix(mobile): play short-delay animation frames at browser speed (#9432)
  buzz-search: read profile names without the profile_name_fields function (#9397)
  buzz-db: build the community deletion lock key in Rust (#9396)
  fix(cli): pair for the active community, skip stopped desktops, accept a bare relay host (#9417)
  test(mobile): run push tests without a gateway dart-define (#9423)
  fix(mobile): never pop Home when archiving from channel settings (#9387)
  perf(acp): reduce observer CPU and lock overhead (#9381)

Signed-off-by: Matthew Boston <[email protected]>

Replaces #9413 with the packing optimization refreshed against main on a
branch in this repository.

## Summary

Busy agent activity streams repeatedly build and serialize a growing
JSON batch just to check whether the next event fits. Reuse cached event
sizes, measure the newest outer envelope, and build the batch payload
once per completed frame.

Preserve the wire format, greedy packing, per-channel order, channel
barriers, byte budget, and one-frame-per-second relay cadence.
Exact-limit coverage checks escaping, UTF-8, optional metadata, and
array separators. Move the queue tests into a separate module to satisfy
the file-size rule; their test names stay unchanged.

### Related issue

Related to #7987. This builds on the merged fitting/locking work in
#9381 and retention limits in #9436. Closest earlier batching work:
#4917. No duplicate packing fix found among open PRs.

### Testing

The ignored probe fills the production queue before timing, drains every
event, and checks complete FIFO delivery, frame byte limits, and empty
byte accounting.

```sh
CARGO_INCREMENTAL=0 CARGO_PROFILE_RELEASE_LTO=thin CARGO_PROFILE_RELEASE_CODEGEN_UNITS=4 \
  cargo test --release -p buzz-acp --lib observer::packing_tests::publish_queue_drain_probe -- \
  --exact --ignored --nocapture

BUZZ_OBSERVER_PROBE_EVENTS=100 cargo test --release -p buzz-acp --lib \
  observer::packing_tests::publish_queue_drain_probe -- --exact --ignored --nocapture
```

Five alternating fresh release processes per arm and workload compared
baseline `59ae028a3` with candidate `237b928b9` on macOS arm64, Rust
1.98.1. Both arms used `opt-level=3`, thin LTO and four codegen units,
with incremental compilation and compiler-cache wrappers off. Each
executable was rebuilt separately and its probe identity verified.
Medians:

| Events | Before | After | Frames in both arms |
| --- | ---: | ---: | ---: |
| 100 | 6.760 ms | 0.108 ms | 1 |
| 10,000 | 1,597.509 ms | 7.844 ms | 30 |

The one-event control measured 2.04 → 2.21 microseconds. The timed
region covers packing only; ingestion, validation, encryption, publish
pacing, relay delivery, and rendering are excluded. These local results
do not establish an overall app latency improvement. No timing threshold
was added to CI.

Native and deployed-relay latency, independent review, and human
verification remain pending. Keep this PR in draft.

Generated with AI assistance

Signed-off-by: Sanket Kanjalkar <[email protected]>
Co-authored-by: Codex <[email protected]>