Skip to content

Commit 9911c99

Browse files
TheSentinel454Elrondclaude
committed
fix(relay): wait out the leader's pool wait at shared-read gates
Every gate waited at most its cache's 1 s max age for the leader. When the leader itself queued for a pool connection longer than that, which is the saturated case these gates target, its waiters gave up, found no entry and each queued its own read behind it: N misses still cost about N reads. Checkout stamping already keeps the leader's entry fresh however long it waited, so the max age no longer needs to bound the wait. Each gate now waits the writer pool's acquire timeout, which bounds the leader's pool wait, plus SHARED_READ_STATEMENT_WAIT (1 s) for its statement. A leader whose statement hangs still stalls waiters only that long, not the 120 s statement timeout. The wait moves from each call site into SharedReads, so the three gates cannot drift apart. The held-pool test now sends eight concurrent misses per cache behind a connection held past the max age and asserts one database read each, as well as the checkout stamp. The open-set hung-leader test bounds its pool acquire timeout so its waiters still give up within the test's window. Also says "connection checkout" in the cache docs that still said the read "began". Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Luke Tornquist <[email protected]>
1 parent 06850f7 commit 9911c99

2 files changed

Lines changed: 159 additions & 83 deletions

File tree

‎crates/buzz-relay/src/shared_reads.rs‎

Lines changed: 25 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -16,37 +16,44 @@ use tokio::sync::{Mutex, OwnedMutexGuard};
1616
/// One gate per key, held by the caller that is reading that key.
1717
pub(crate) struct SharedReads<K> {
1818
gates: moka::sync::Cache<K, Arc<Mutex<()>>>,
19+
wait: Duration,
1920
}
2021

2122
impl<K> SharedReads<K>
2223
where
2324
K: Hash + Eq + Send + Sync + 'static,
2425
{
25-
/// Gates for up to `max_capacity` keys; an idle gate is dropped after a
26-
/// minute.
27-
pub(crate) fn new(max_capacity: u64) -> Self {
26+
/// Gates for up to `max_capacity` keys, each waited on for at most
27+
/// `wait`; an idle gate is dropped after a minute.
28+
///
29+
/// `wait` must cover the holder's wait for a pool connection, which is
30+
/// what grows in a storm, plus its statement: a waiter that gives up
31+
/// while the holder still queues for a connection only queues behind it
32+
/// with a duplicate read.
33+
pub(crate) fn new(max_capacity: u64, wait: Duration) -> Self {
2834
Self {
2935
gates: moka::sync::Cache::builder()
3036
.max_capacity(max_capacity)
3137
.time_to_idle(Duration::from_secs(60))
3238
.build(),
39+
wait,
3340
}
3441
}
3542

3643
/// Takes `key`'s gate, or waits for its holder to finish.
3744
///
3845
/// Returns the guard when no read of `key` is in flight: the caller reads
3946
/// and holds the guard until its cache entry is written. Otherwise waits
40-
/// up to `wait` for the holder and returns `None`; the caller then
41-
/// rechecks its cache and, if the holder failed, was cancelled or is
42-
/// still reading, reads on its own without the gate. A read that hangs
43-
/// therefore stalls each waiter for at most `wait`.
44-
pub(crate) async fn turn(&self, key: K, wait: Duration) -> Option<OwnedMutexGuard<()>> {
47+
/// for the holder, at most this gate's `wait`, and returns `None`; the
48+
/// caller then rechecks its cache and, if the holder failed, was
49+
/// cancelled or is still reading, reads on its own without the gate. A
50+
/// read that hangs therefore stalls each waiter for at most `wait`.
51+
pub(crate) async fn turn(&self, key: K) -> Option<OwnedMutexGuard<()>> {
4552
let gate = self.gates.get_with(key, || Arc::new(Mutex::new(())));
4653
match Arc::clone(&gate).try_lock_owned() {
4754
Ok(guard) => Some(guard),
4855
Err(_) => {
49-
drop(tokio::time::timeout(wait, gate.lock()).await);
56+
drop(tokio::time::timeout(self.wait, gate.lock()).await);
5057
None
5158
}
5259
}
@@ -67,16 +74,16 @@ mod tests {
6774

6875
#[tokio::test]
6976
async fn one_caller_reads_while_the_rest_wait_for_it() {
70-
let reads = Arc::new(SharedReads::<u8>::new(16));
71-
let leader = reads.turn(1, Duration::from_secs(5)).await;
77+
let reads = Arc::new(SharedReads::<u8>::new(16, Duration::from_secs(5)));
78+
let leader = reads.turn(1).await;
7279
assert!(leader.is_some(), "the first caller takes the gate");
7380

7481
// A second key has its own gate.
75-
assert!(reads.turn(2, Duration::from_secs(5)).await.is_some());
82+
assert!(reads.turn(2).await.is_some());
7683

7784
let waiter = {
7885
let reads = Arc::clone(&reads);
79-
tokio::spawn(async move { reads.turn(1, Duration::from_secs(5)).await })
86+
tokio::spawn(async move { reads.turn(1).await })
8087
};
8188
tokio::task::yield_now().await;
8289
assert!(!waiter.is_finished(), "a waiter blocks while the read runs");
@@ -87,15 +94,15 @@ mod tests {
8794
);
8895

8996
// Once the gate is free the next caller reads again.
90-
assert!(reads.turn(1, Duration::from_secs(5)).await.is_some());
97+
assert!(reads.turn(1).await.is_some());
9198
}
9299

93100
#[tokio::test(start_paused = true)]
94101
async fn a_waiter_stops_waiting_for_a_hung_read() {
95-
let reads = SharedReads::<u8>::new(16);
96-
let _hung = reads.turn(1, Duration::from_secs(1)).await;
102+
let reads = SharedReads::<u8>::new(16, Duration::from_secs(4));
103+
let _hung = reads.turn(1).await;
97104
let started = tokio::time::Instant::now();
98-
assert!(reads.turn(1, Duration::from_secs(1)).await.is_none());
99-
assert_eq!(started.elapsed(), Duration::from_secs(1));
105+
assert!(reads.turn(1).await.is_none());
106+
assert_eq!(started.elapsed(), Duration::from_secs(4));
100107
}
101108
}

0 commit comments

Comments
 (0)