Skip to content

Commit 5511b56

Browse files
wesbillmanCarlkiranmagic7
authored
fix(relay): exclude ephemeral activity from message quota (#7736)
## Summary Stop charging ephemeral WebSocket activity against the per-minute `Messages` quota. Presence, typing, and observer traffic should not consume the allowance intended for persistable event attempts. - Use the existing `buzz_core::kind::is_ephemeral` predicate (kinds `20000..=29999`) at the second admission gate. All authenticated EVENT/REQ/COUNT traffic still consumes the shared `WsEvents` budget first. Non-ephemeral EVENT attempts remain charged, including attempts later rejected by authorization or validation. - Add bounded `bucket` labels (`messages`, `ws_operations`, `api_calls`) to `buzz_admission_rejections_total`. HTTP changes are metric labeling only. Wire rejection text remains byte-for-byte unchanged, preserving existing client parsers. - Add real-Redis admission regressions and select them in the existing relay integration CI job. No quota increases, identity/keying changes, client changes, new limiter, or deployment. Downstream authorization and fail-closed Redis behavior are unchanged. ### Related issue Duplicate search found overlapping work: open #4912 and closed #4902. They propose the same ephemeral-accounting correction alongside broader wire/client/config changes. The relay fix is a deliberately narrower four-file implementation on the current admission module, with metrics-only diagnostics and real-Redis regression coverage. A separate test-only CI repair is documented below. It does not close or claim to supersede those PRs. Related symptom: #7411. This PR does **not** resolve subscription bursts exhausting the shared WS budget; that budget is intentionally preserved. It also does not establish or fix the cause of HTTP quota exhaustion. ### Testing Validation ran against base `779af8886caae1317b4de962082429867ab61503` plus the exact patch now committed as `70ea0528594e25e6dcb40cd74620bd5ef0177592` (binary diff SHA-256 `5064e1cbb87ecb9eff10336bae14b912cac442e1edc247aa4f093de30e4f9717`). Normal commit/push hooks subsequently passed on the committed head, without bypass. Passed: - Independent review and rerun: 11 rejection tests (including real Redis) plus 4 admission tests. Covers ephemeral range boundaries, human/agent budgets, persistable events, ephemeral admission after message exhaustion, mixed EVENT/REQ/COUNT flood exhaustion, Redis outage, exact wire payloads, and bucket metrics. Three mutations were caught: reintroducing ephemeral billing, exempting stored events, and bypassing WS rejection. - `just test-unit`: all 17 runner groups; `just test-integration`: all 3 runner groups. Auth integration reported no tests; ignored database integration tests are not claimed as executed. - Relay all-target Clippy, formatting, workflow actionlint, pre-commit/signoff hooks, and all applicable pre-push lanes (`push-head-scope`, `branch-skew`, `file-size-check`, `rust-tests`, `desktop-tauri-checks`). - Release relay/CLI/admin build and isolated release-binary workflow: channel creation, signed root/reply sends, exact-ID readback, and thread retrieval. Debug and release WS smoke admitted three presence events with a Messages limit of two, admitted two persistable events, rejected the third, still admitted presence afterward, and rejected author mismatch. Debug Prometheus confirmed `bucket="messages"`. **Outstanding validation gap:** the additional `cargo test -p buzz-relay --no-fail-fast` run was not green: library 1,055 passed / 2 failed / 100 ignored; binary 13 passed / 1 ignored; boot integration 9 passed. Failures were `api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo` (504 instead of 200) and `telemetry::tests::trace_context_lookup_does_not_enable_callsites` (tracing assertion). Neither failure was diagnosed or repaired in this scope, and this PR does not claim a fully green package suite. `just ci` was not run locally; hosted CI remains to validate the PR. All runtime tests used isolated local services. Owned services have been stopped; nothing was deployed and no production data was touched. ### CI cancellation-test repair At the original relay head, hosted Rust unit tests failed on the pre-existing cancellation harness race tracked in #4945. With owner authorization, this PR now adapts the one-file repair from #5003 (original commit `cc8dc6124957acfd67fe2336396b58e0ff43ec60`), preserving author credit. #5003's old hosted workflows failed without starting jobs; their status is not used as validation evidence. Commit `55fe5f16618a51d03ad48d6f2f97b2f8b463607a` holds the fake provider response until a successful cancellation acknowledgement, then replays all pre-acknowledgement usage/prompt frames in order. It retains the cancellation and usage-before-response assertions and explicitly requires prompt completion. No production code changes in this follow-up. Validation on parent `70ea052859` plus this test patch, before the pre-commit-only assertion wrapping: full `cargo test -p buzz-agent --no-fail-fast` passed (715 tests, one ignored); 100 targeted repetitions passed with four concurrent workers; all-target buzz-agent Clippy passed. The unmodified parent also passed 50 local repetitions, so this is not claimed as a local reproduction of the intermittent CI failure. The synchronization defect was confirmed from the source, original CI log, and #4945. Hosted CI must validate the updated head. Independent source review cleared exact commit `55fe5f16618a51d03ad48d6f2f97b2f8b463607a`. Normal pre-commit/signoff and all applicable pre-push hooks passed without bypass. The original relay-package mesh-demo and telemetry failures remain disclosed above and were not modified by this test-only repair. --------- Signed-off-by: Wes <[email protected]> Co-authored-by: Carl <32a2e2c9d428ee08902cab75d956da2c1d235a22d4766b0dd4138bf6e2e5db1d@buzz.block.builderlab.xyz> Co-authored-by: kiranmagic7 <[email protected]>
1 parent 4e65148 commit 5511b56

5 files changed

Lines changed: 304 additions & 23 deletions

File tree

‎.github/workflows/_ci-relay.yml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -497,6 +497,14 @@ jobs:
497497
--archive-file target/ci/backend-integration-tests.tar.zst \
498498
-E 'package(buzz-relay) and test(/api::git::settings::tests::external_infra::/)' \
499499
--run-ignored ignored-only
500+
- name: Relay quota accounting and rejection regressions
501+
env:
502+
BUZZ_TEST_REDIS_URL: redis://localhost:6379
503+
run: |
504+
cargo nextest run \
505+
--archive-file target/ci/backend-integration-tests.tar.zst \
506+
-E 'package(buzz-relay) and (test(/^rejection::/) or test(/^admission::/))' \
507+
--run-ignored all
500508
- name: Workflow message provenance unit tests
501509
# The relay's workflow_sink suite is not selected by the infra-free
502510
# unit job. Its ignored database cases run in the isolated PostgreSQL

‎crates/buzz-agent/tests/fake_llm.rs‎

Lines changed: 23 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1320,7 +1320,7 @@ async fn mid_turn_usage_includes_earlier_turns() {
13201320
/// Setup: round 1 is a tool call WITH usage (tokens are captured). After the
13211321
/// tool_call_update notification (proving round 1 is fully processed), we gate
13221322
/// the round-2 LLM response behind a `oneshot` barrier that only releases after
1323-
/// cancel is sent. This guarantees the turn exits with `stopReason: "cancelled"`
1323+
/// cancel is acknowledged. This guarantees the turn exits with `stopReason: "cancelled"`
13241324
/// deterministically, even on a slow CI worker.
13251325
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
13261326
async fn cancelled_turn_with_usage_emits_notification_before_response() {
@@ -1335,7 +1335,7 @@ async fn cancelled_turn_with_usage_emits_notification_before_response() {
13351335
// in-flight TCP request can resolve. The queue is empty for round 2, so the
13361336
// agent receives the fallback "no canned response" body which it treats as
13371337
// an LLM error; the cancel check at the round boundary fires first because
1338-
// the gate is only released after cancel is enqueued.
1338+
// the gate is only released after cancel is acknowledged.
13391339
let responses = vec![openai_tool_call_with_usage(
13401340
"call_cancel_test",
13411341
"fake__noop",
@@ -1371,7 +1371,7 @@ async fn cancelled_turn_with_usage_emits_notification_before_response() {
13711371
}
13721372
}
13731373
// For request 2+ (round 2), wait for the gate to open before
1374-
// responding. This ensures cancel is sent before round 2 resolves,
1374+
// responding. This ensures cancel is processed before round 2 resolves,
13751375
// making stopReason: cancelled deterministic.
13761376
if req_num >= 2 {
13771377
let rx = gate.lock().await.take();
@@ -1415,20 +1415,27 @@ async fn cancelled_turn_with_usage_emits_notification_before_response() {
14151415
})
14161416
.await;
14171417

1418-
// Now send cancel and release the round-2 gate. Cancel is enqueued before
1419-
// round 2 can respond, so the turn exits with stopReason: cancelled.
1418+
// Writing to stdin does not prove the agent processed cancel. Keep the
1419+
// provider blocked until the acknowledgement, retaining all earlier frames
1420+
// so usage/prompt ordering is checked even if the turn finishes before ACK.
14201421
let c_id = h.send("session/cancel", json!({"sessionId": sid})).await;
1422+
let (frames_before_cancel_ack, cancel_ack) =
1423+
recv_until_with_drain(&mut h, |v| v["id"] == json!(c_id)).await;
1424+
assert_eq!(cancel_ack.get("result"), Some(&Value::Null), "{cancel_ack}");
1425+
assert!(cancel_ack.get("error").is_none(), "{cancel_ack}");
14211426
let _ = gate_tx.send(()); // unblock round 2
14221427

14231428
let mut saw_usage_before_prompt_response = false;
14241429
let mut saw_usage = false;
1425-
let mut saw_cancel_ok = false;
14261430
let mut saw_prompt_response = false;
1427-
for _ in 0..40 {
1428-
let v = h.recv().await;
1429-
if v["id"] == json!(c_id) {
1430-
saw_cancel_ok = true;
1431-
} else if is_usage_update(&v) {
1431+
let mut pending_frames = VecDeque::from(frames_before_cancel_ack);
1432+
let frame_budget = 40 + pending_frames.len();
1433+
for _ in 0..frame_budget {
1434+
let v = match pending_frames.pop_front() {
1435+
Some(v) => v,
1436+
None => h.recv().await,
1437+
};
1438+
if is_usage_update(&v) {
14321439
saw_usage = true;
14331440
if !saw_prompt_response {
14341441
saw_usage_before_prompt_response = true;
@@ -1441,11 +1448,14 @@ async fn cancelled_turn_with_usage_emits_notification_before_response() {
14411448
"turn must end with stopReason: cancelled"
14421449
);
14431450
}
1444-
if saw_usage && saw_prompt_response && saw_cancel_ok {
1451+
if saw_usage && saw_prompt_response {
14451452
break;
14461453
}
14471454
}
1448-
assert!(saw_cancel_ok, "session/cancel was not acknowledged");
1455+
assert!(
1456+
saw_prompt_response,
1457+
"session/prompt did not finish after cancel"
1458+
);
14491459
assert!(
14501460
saw_usage,
14511461
"expected usage_update notification for cancelled turn with observed tokens"

‎crates/buzz-relay/src/api/bridge.rs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -39,14 +39,14 @@ pub(crate) async fn enforce_http_admission(
3939
{
4040
Ok(()) => Ok(()),
4141
Err(crate::admission::AdmissionError::Exceeded { reset_in_secs }) => {
42-
metrics::counter!("buzz_admission_rejections_total", "transport" => "http", "reason" => "quota").increment(1);
42+
metrics::counter!("buzz_admission_rejections_total", "transport" => "http", "reason" => "quota", "bucket" => "api_calls").increment(1);
4343
Err(api_error(
4444
StatusCode::TOO_MANY_REQUESTS,
4545
&format!("rate-limited: quota exceeded; retry in {reset_in_secs}s"),
4646
))
4747
}
4848
Err(crate::admission::AdmissionError::Unavailable) => {
49-
metrics::counter!("buzz_admission_rejections_total", "transport" => "http", "reason" => "unavailable").increment(1);
49+
metrics::counter!("buzz_admission_rejections_total", "transport" => "http", "reason" => "unavailable", "bucket" => "api_calls").increment(1);
5050
Err(api_error(
5151
StatusCode::SERVICE_UNAVAILABLE,
5252
"rate-limited: shared admission unavailable",

‎crates/buzz-relay/src/rejection.rs‎

Lines changed: 21 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -75,11 +75,12 @@ pub(crate) async fn enforce_ws_admission(
7575
ws_limit,
7676
)
7777
.await;
78-
if !send_admission_result(conn, ws_result, msg) {
78+
if !send_admission_result(conn, ws_result, msg, "ws_operations") {
7979
return false;
8080
}
8181

82-
if is_event {
82+
if matches!(msg, ClientMessage::Event(event) if !buzz_core::kind::is_ephemeral(event.kind.as_u16() as u32))
83+
{
8384
let message_limit = if is_agent {
8485
limits.agent_standard_messages_per_min
8586
} else {
@@ -94,9 +95,9 @@ pub(crate) async fn enforce_ws_admission(
9495
message_limit,
9596
)
9697
.await;
97-
// The per-minute message quota only applies to EVENTs, and its
98-
// rejection must be as correlatable as the burst quota's.
99-
if !send_admission_result(conn, message_result, msg) {
98+
// Only persistable EVENT attempts consume the minute quota. Ephemeral activity
99+
// still passes the shared WS flood limit above and normal event auth.
100+
if !send_admission_result(conn, message_result, msg, "messages") {
100101
return false;
101102
}
102103
}
@@ -115,20 +116,21 @@ fn send_admission_result(
115116
conn: &ConnectionState,
116117
result: Result<(), AdmissionError>,
117118
msg: &ClientMessage,
119+
bucket: &'static str,
118120
) -> bool {
119121
let target = rejection_target_for(msg);
120122
match result {
121123
Ok(()) => true,
122124
Err(AdmissionError::Exceeded { reset_in_secs }) => {
123-
metrics::counter!("buzz_admission_rejections_total", "transport" => "websocket", "reason" => "quota").increment(1);
125+
metrics::counter!("buzz_admission_rejections_total", "transport" => "websocket", "reason" => "quota", "bucket" => bucket).increment(1);
124126
conn.send(request_rejection_message(
125127
target,
126128
&format!("rate-limited: quota exceeded; retry in {reset_in_secs}s"),
127129
));
128130
false
129131
}
130132
Err(AdmissionError::Unavailable) => {
131-
metrics::counter!("buzz_admission_rejections_total", "transport" => "websocket", "reason" => "unavailable").increment(1);
133+
metrics::counter!("buzz_admission_rejections_total", "transport" => "websocket", "reason" => "unavailable", "bucket" => bucket).increment(1);
132134
conn.send(request_rejection_message(
133135
target,
134136
"rate-limited: shared admission unavailable",
@@ -198,6 +200,7 @@ mod tests {
198200
&conn,
199201
Err(AdmissionError::Exceeded { reset_in_secs: 7 }),
200202
&msg,
203+
"ws_operations",
201204
);
202205

203206
assert!(!admitted, "an over-quota frame is not admitted");
@@ -227,7 +230,12 @@ mod tests {
227230
let (conn, mut rx) = test_conn();
228231
let (msg, event_id) = parsed_event_message();
229232

230-
send_admission_result(&conn, Err(AdmissionError::Unavailable), &msg);
233+
send_admission_result(
234+
&conn,
235+
Err(AdmissionError::Unavailable),
236+
&msg,
237+
"ws_operations",
238+
);
231239

232240
let frame = sent_frame(&mut rx);
233241
assert_eq!(frame[0], "OK");
@@ -247,6 +255,7 @@ mod tests {
247255
&conn,
248256
Err(AdmissionError::Exceeded { reset_in_secs: 7 }),
249257
&msg,
258+
"ws_operations",
250259
);
251260

252261
let frame = sent_frame(&mut rx);
@@ -269,6 +278,7 @@ mod tests {
269278
&conn,
270279
Err(AdmissionError::Exceeded { reset_in_secs: 7 }),
271280
&msg,
281+
"ws_operations",
272282
);
273283

274284
let frame = sent_frame(&mut rx);
@@ -333,3 +343,6 @@ mod tests {
333343
assert_eq!(frame[1], "history-abc");
334344
}
335345
}
346+
347+
#[cfg(test)]
348+
mod quota_tests;

0 commit comments

Comments
 (0)