feat: more flash indexer optimizations [DYN-2164]#6305
Conversation
Signed-off-by: PeaBrane <[email protected]>
Signed-off-by: PeaBrane <[email protected]>
Signed-off-by: PeaBrane <[email protected]>
Signed-off-by: PeaBrane <[email protected]>
Signed-off-by: PeaBrane <[email protected]>
Signed-off-by: PeaBrane <[email protected]>
Signed-off-by: PeaBrane <[email protected]>
Signed-off-by: PeaBrane <[email protected]>
Signed-off-by: PeaBrane <[email protected]>
Signed-off-by: PeaBrane <[email protected]> Co-authored-by: Cursor <[email protected]>
Signed-off-by: PeaBrane <[email protected]> Co-authored-by: Cursor <[email protected]>
Signed-off-by: PeaBrane <[email protected]>
WalkthroughThis PR adds benchmark-specific alternative indexer implementations and optimizes data structure performance across the codebase by replacing standard HashMap/HashSet with FxHash equivalents and updating DashMap to use FxBuildHasher for faster hashing. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~45 minutes Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (5)
lib/kv-router/src/nested_map.rs (1)
303-310: Pre-existing TOCTOU onworker_blocks— considerentryAPI.Lines 305–310 have a check-then-act pattern (
contains_key→insert→get().unwrap()) that's racy if two threads hit the sameworkerkey simultaneously. The sticky routing inThreadPoolIndexermakes this safe in practice, but using theentryAPI would be both cleaner and formally correct:♻️ Suggested simplification
- if !worker_blocks.contains_key(&worker) { - worker_blocks.insert(worker, RwLock::new(FxHashMap::default())); - } - - let worker_blocks_entry = worker_blocks.get(&worker).unwrap(); - let mut worker_map = worker_blocks_entry.write(); + let worker_blocks_entry = worker_blocks + .entry(worker) + .or_insert_with(|| RwLock::new(FxHashMap::default())); + let mut worker_map = worker_blocks_entry.write();🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@lib/kv-router/src/nested_map.rs` around lines 303 - 310, The current contains_key → insert → get().unwrap() pattern on worker_blocks is racy; replace it with the HashMap entry API: use worker_blocks.entry(worker).or_insert_with(|| RwLock::new(FxHashMap::default())) to obtain the RwLock for that worker atomically, then call .write() on the returned entry to get worker_map; update the code paths that reference worker_blocks, worker, RwLock, and FxHashMap accordingly to remove the check-then-act and the unwrap.lib/kv-router/src/naive_indexers.rs (4)
119-143: Actor threads spawn a full Tokio runtime just formpsc::Receiver::recv().Each actor creates a dedicated OS thread + a
current_threadTokio runtime solely to.awaiton thempsc::Receiver. Since the inner logic is entirely synchronous, a simpler approach would be to usestd::sync::mpsc(orflume) and avoid the Tokio runtime overhead altogether. This is bench-only code so it's not blocking, but the extra runtime may add noise to benchmark measurements.Also applies to: 323-347
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@lib/kv-router/src/naive_indexers.rs` around lines 119 - 143, The actor thread currently spawns a full Tokio current_thread runtime and uses rt.block_on to await rx.recv(), which is unnecessary because NaiveNestedMapInner and message handling (ActorMessage variants: Event, Match, RemoveWorker) are fully synchronous; replace the async mpsc::Receiver usage with a synchronous channel (std::sync::mpsc or flume) and convert the thread closure to a plain blocking loop that calls recv() (or recv_timeout if needed) to pull messages and handle them by calling inner.apply_event, inner.find_matches (sending replies), and inner.remove_worker, removing the Tokio runtime creation (tokio::runtime::Builder, rt.block_on) and simplifying the spawn to a standard thread loop over the sync receiver.
175-177: Silently dropped send errors inapply_event.Both
apply_eventimplementations discard theResultfromself.tx.send(...). If the actor thread panics or the channel fills up (bounded at 2048), events are silently lost with no log or metric. For benchmarking correctness validation, consider at least atracing::warn!on failure.Also applies to: 379-381
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@lib/kv-router/src/naive_indexers.rs` around lines 175 - 177, The apply_event implementations silently drop the Result from self.tx.send(ActorMessage::Event(...)), which loses events if the channel is closed or full; update both apply_event (the one using self.tx.send(ActorMessage::Event(event)).await and the other occurrence around the second implementation) to handle the Result: match or if let Err(e) = self.tx.send(...).await and emit a tracing::warn! (including context like "failed to send ActorMessage::Event" and the error) and/or increment a metric counter for dropped events, so failures are observable and include the error details for debugging.
244-260: Redundant re-check of position 0 in the loop.
activeis cloned fromself.index.get(&sequence[0])on line 238, then the loop starting at line 244 re-checkssequence[0](sinceenumerate()starts at 0). The first iteration is always a no-op. Starting fromskip(1)and initializing scores accordingly would be slightly cleaner:♻️ Suggested fix
- for (depth, local_hash) in sequence.iter().enumerate() { + for (depth, local_hash) in sequence.iter().enumerate().skip(1) { let empty = HashSet::new(); let workers_here = self.index.get(local_hash).unwrap_or(&empty); active.retain(|w| { if workers_here.contains(w) { true } else { scores.scores.insert(*w, depth as u32); false } }); if active.is_empty() { break; } }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@lib/kv-router/src/naive_indexers.rs` around lines 244 - 260, The loop redundantly re-checks sequence[0] because active was already cloned from self.index.get(&sequence[0]); change the iterator to skip the first element (use sequence.iter().enumerate().skip(1)) so the first no-op iteration is removed, keep the body that looks up workers_here via self.index.get(local_hash) and updates scores.scores and active as before (symbols to edit: active, sequence, self.index, scores.scores, the enumerate loop). Ensure behavior is otherwise unchanged.
47-67:find_matchesdoesn't verify prefix/positional ordering — intentional but worth a note.This checks set membership (
blocks.contains(local_hash)) without verifying that blocks appear at the correct positions in the sequence. A worker with blocks{A, C}queried with[A, B, C]correctly returns depth 1, but[C, A]would also return depth 2 even though the prefix order differs. The doc says "O(W × D) perfind_matches" which matches, but this semantic gap vs. the production indexer could produce misleading benchmark comparisons if the query workload has reordered tokens.Given this is bench-only and the doc already caveats "cut corners," this is just a heads-up for interpreting results.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@lib/kv-router/src/naive_indexers.rs` around lines 47 - 67, The current find_matches implementation only checks set membership via blocks.contains(LocalBlockHash) and thus counts matches regardless of positional/prefix order; update find_matches to verify positional ordering by using each worker's ordered block sequence (self.index entry) rather than set membership: build or use a mapping from LocalBlockHash to its index in the worker's block list and then, for each query sequence, only increment depth while the next sequence hash maps to the next consecutive index (ensuring prefix/positional order), then insert the final depth into OverlapScores; adjust any types as needed so blocks is accessible as an ordered sequence or index map instead of an unordered set.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@lib/kv-router/benches/mooncake_bench.rs`:
- Around line 63-72: The InvertedIndex enum variant declares num_event_workers
and documents multi-threaded behavior, but the value is never used: update the
constructors and builder to pass and honor num_event_workers (propagate the
field from InvertedIndex into the built index instance) or, if the index must
remain single-threaded, remove the field and update docs to avoid confusion;
specifically modify InvertedIndex::from_name (which currently hard-codes 0) to
parse/retain num_event_workers and change build() to use that field when
spawning event workers, or delete num_event_workers and adjust the enum variant
documentation and any references in build()/from_name accordingly.
- Around line 1057-1066: The log-spaced durations loop divides by (n - 1), which
panics when args.sweep_steps == 1; add a guard on args.sweep_steps (n) before
computing durations: if n <= 1 produce a single duration (for example
args.sweep_min_ms as u64 or args.sweep_max_ms as u64, or the geometric mean via
((args.sweep_min_ms as f64).ln() + (args.sweep_max_ms as f64).ln())/2).
Otherwise run the existing map that uses (n - 1). Ensure you reference
args.sweep_steps (n), args.sweep_min_ms, args.sweep_max_ms and the durations
Vec<u64> when implementing the guard.
In `@lib/kv-router/src/indexer.rs`:
- Line 39: The new import line use rustc_hash::FxBuildHasher; in indexer.rs is
causing rustfmt import-ordering failures; run cargo fmt (or rustfmt) on the
repository and reorder imports in lib/kv-router/src/indexer.rs so they follow
rustfmt's default ordering (grouping std, extern crates, and local modules) and
ensure the FxBuildHasher import sits in the correct group—then commit the
formatted file.
---
Nitpick comments:
In `@lib/kv-router/src/naive_indexers.rs`:
- Around line 119-143: The actor thread currently spawns a full Tokio
current_thread runtime and uses rt.block_on to await rx.recv(), which is
unnecessary because NaiveNestedMapInner and message handling (ActorMessage
variants: Event, Match, RemoveWorker) are fully synchronous; replace the async
mpsc::Receiver usage with a synchronous channel (std::sync::mpsc or flume) and
convert the thread closure to a plain blocking loop that calls recv() (or
recv_timeout if needed) to pull messages and handle them by calling
inner.apply_event, inner.find_matches (sending replies), and
inner.remove_worker, removing the Tokio runtime creation
(tokio::runtime::Builder, rt.block_on) and simplifying the spawn to a standard
thread loop over the sync receiver.
- Around line 175-177: The apply_event implementations silently drop the Result
from self.tx.send(ActorMessage::Event(...)), which loses events if the channel
is closed or full; update both apply_event (the one using
self.tx.send(ActorMessage::Event(event)).await and the other occurrence around
the second implementation) to handle the Result: match or if let Err(e) =
self.tx.send(...).await and emit a tracing::warn! (including context like
"failed to send ActorMessage::Event" and the error) and/or increment a metric
counter for dropped events, so failures are observable and include the error
details for debugging.
- Around line 244-260: The loop redundantly re-checks sequence[0] because active
was already cloned from self.index.get(&sequence[0]); change the iterator to
skip the first element (use sequence.iter().enumerate().skip(1)) so the first
no-op iteration is removed, keep the body that looks up workers_here via
self.index.get(local_hash) and updates scores.scores and active as before
(symbols to edit: active, sequence, self.index, scores.scores, the enumerate
loop). Ensure behavior is otherwise unchanged.
- Around line 47-67: The current find_matches implementation only checks set
membership via blocks.contains(LocalBlockHash) and thus counts matches
regardless of positional/prefix order; update find_matches to verify positional
ordering by using each worker's ordered block sequence (self.index entry) rather
than set membership: build or use a mapping from LocalBlockHash to its index in
the worker's block list and then, for each query sequence, only increment depth
while the next sequence hash maps to the next consecutive index (ensuring
prefix/positional order), then insert the final depth into OverlapScores; adjust
any types as needed so blocks is accessible as an ordered sequence or index map
instead of an unordered set.
In `@lib/kv-router/src/nested_map.rs`:
- Around line 303-310: The current contains_key → insert → get().unwrap()
pattern on worker_blocks is racy; replace it with the HashMap entry API: use
worker_blocks.entry(worker).or_insert_with(|| RwLock::new(FxHashMap::default()))
to obtain the RwLock for that worker atomically, then call .write() on the
returned entry to get worker_map; update the code paths that reference
worker_blocks, worker, RwLock, and FxHashMap accordingly to remove the
check-then-act and the unwrap.
Signed-off-by: PeaBrane <[email protected]>
We are already using that since #6305 so no Cargo.lock changes, but we're using it as a side effect of a different package so it's not pinned. This PR pins it. Closes: #7328 Signed-off-by: Graham King <[email protected]>
Signed-off-by: PeaBrane <[email protected]> Co-authored-by: Cursor <[email protected]>
Switch all hash maps in the kv-router indexers to
rustc-hash(FxHash) for faster hashing, add naive indexer baselines for benchmarking, and extend the benchmark harness with sweep/compare modes.Changes
HashMap/HashSetand defaultDashMaphashers withFxHashMap/FxHashSet/FxBuildHasheracrossconcurrent_radix_tree,indexer,nested_map,radix_tree, andprotocols.naive_indexers.rs,bench-gated) —NaiveNestedMap(worker → set<hash>) andInvertedIndex(hash → set<worker>) for benchmarking the performance progression described in the blog.--sweepplots throughput vs p99 latency across log-spaced durations;--compareruns multiple indexers on the same plot. Addedplottersdev-dependency.