Skip to content

feat: more flash indexer optimizations [DYN-2164]#6305

Merged
PeaBrane merged 21 commits into
mainfrom
rupei/flash-indexer-blog
Feb 18, 2026
Merged

feat: more flash indexer optimizations [DYN-2164]#6305
PeaBrane merged 21 commits into
mainfrom
rupei/flash-indexer-blog

Conversation

@PeaBrane

@PeaBrane PeaBrane commented Feb 14, 2026

Copy link
Copy Markdown
Contributor

Switch all hash maps in the kv-router indexers to rustc-hash (FxHash) for faster hashing, add naive indexer baselines for benchmarking, and extend the benchmark harness with sweep/compare modes.

Changes

  • FxHash everywhere — replace HashMap/HashSet and default DashMap hashers with FxHashMap/FxHashSet/FxBuildHasher across concurrent_radix_tree, indexer, nested_map, radix_tree, and protocols.
  • Naive indexer baselines (naive_indexers.rs, bench-gated) — NaiveNestedMap (worker → set<hash>) and InvertedIndex (hash → set<worker>) for benchmarking the performance progression described in the blog.
  • Benchmark sweep & compare modes--sweep plots throughput vs p99 latency across log-spaced durations; --compare runs multiple indexers on the same plot. Added plotters dev-dependency.

Signed-off-by: PeaBrane <[email protected]>
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Feb 14, 2026
Comment thread blog/flash_indexer.md Outdated
Comment thread blog/flash_indexer.md Outdated
Comment thread blog/flash_indexer.md Outdated
Comment thread blog/flash_indexer.md Outdated
Comment thread blog/flash_indexer.md Outdated
Comment thread blog/flash_indexer.md Outdated
Comment thread blog/flash_indexer.md Outdated
Signed-off-by: PeaBrane <[email protected]>
Signed-off-by: PeaBrane <[email protected]>
Signed-off-by: PeaBrane <[email protected]>
Signed-off-by: PeaBrane <[email protected]>
Signed-off-by: PeaBrane <[email protected]>
Signed-off-by: PeaBrane <[email protected]>
Signed-off-by: PeaBrane <[email protected]>
Co-authored-by: Cursor <[email protected]>
@PeaBrane PeaBrane marked this pull request as ready for review February 18, 2026 07:48
@PeaBrane PeaBrane requested a review from a team as a code owner February 18, 2026 07:48
Signed-off-by: PeaBrane <[email protected]>
Co-authored-by: Cursor <[email protected]>
Signed-off-by: PeaBrane <[email protected]>
@coderabbitai

coderabbitai Bot commented Feb 18, 2026

Copy link
Copy Markdown
Contributor

Walkthrough

This PR adds benchmark-specific alternative indexer implementations and optimizes data structure performance across the codebase by replacing standard HashMap/HashSet with FxHash equivalents and updating DashMap to use FxBuildHasher for faster hashing.

Changes

Cohort / File(s) Summary
Benchmark Infrastructure
lib/kv-router/Cargo.toml, lib/kv-router/benches/mooncake_bench.rs, lib/kv-router/src/lib.rs, lib/kv-router/src/naive_indexers.rs
Added rustc-hash and plotters dependencies; introduced two new naive indexer implementations (NaiveNestedMap and InvertedIndex) with single-threaded actor-based designs; extended benchmark harness with sweep-mode CLI options, comparative plotting, throughput/latency metrics collection, and support for multiple indexer variants.
Hash Container Optimization
lib/kv-router/src/concurrent_radix_tree.rs, lib/kv-router/src/indexer.rs, lib/kv-router/src/nested_map.rs, lib/kv-router/src/protocols.rs, lib/kv-router/src/radix_tree.rs
Systematically replaced HashMap/HashSet with FxHashMap/FxHashSet throughout; updated DashMap initializations to use FxBuildHasher; swapped std::sync::RwLock to parking_lot::RwLock in nested_map.rs; updated public type signatures in OverlapScores, SeqEntry, PositionalIndexer, and RadixTree/RadixBlock.
Code Cleanup
lib/llm/src/kv_router.rs
Replaced explicit empty OverlapScores construction with OverlapScores::new() call.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~45 minutes

Poem

🐰 Hashes swift as rustc-grain,
FxHashMaps flow through our domain,
Benchmarks bloom with plots so bright,
Alternative indexers take flight—
Optimization hops abound tonight! 🌙

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning No pull request description was provided. The template requires Overview, Details, Where should the reviewer start, and Related Issues sections. Add a comprehensive description covering: the optimization changes made, files to review first, and any related GitHub issues using action keywords (Closes/Fixes/Resolves/Relates to).
✅ Passed checks (2 passed)
Check name Status Explanation
Title check ✅ Passed The title 'feat: more flash indexer optimizations' clearly describes the main change: performance optimizations to the flash indexer, aligning with changes across multiple indexer implementations and supporting infrastructure.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Nitpick comments (5)
lib/kv-router/src/nested_map.rs (1)

303-310: Pre-existing TOCTOU on worker_blocks — consider entry API.

Lines 305–310 have a check-then-act pattern (contains_keyinsertget().unwrap()) that's racy if two threads hit the same worker key simultaneously. The sticky routing in ThreadPoolIndexer makes this safe in practice, but using the entry API would be both cleaner and formally correct:

♻️ Suggested simplification
-        if !worker_blocks.contains_key(&worker) {
-            worker_blocks.insert(worker, RwLock::new(FxHashMap::default()));
-        }
-
-        let worker_blocks_entry = worker_blocks.get(&worker).unwrap();
-        let mut worker_map = worker_blocks_entry.write();
+        let worker_blocks_entry = worker_blocks
+            .entry(worker)
+            .or_insert_with(|| RwLock::new(FxHashMap::default()));
+        let mut worker_map = worker_blocks_entry.write();
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@lib/kv-router/src/nested_map.rs` around lines 303 - 310, The current
contains_key → insert → get().unwrap() pattern on worker_blocks is racy; replace
it with the HashMap entry API: use worker_blocks.entry(worker).or_insert_with(||
RwLock::new(FxHashMap::default())) to obtain the RwLock for that worker
atomically, then call .write() on the returned entry to get worker_map; update
the code paths that reference worker_blocks, worker, RwLock, and FxHashMap
accordingly to remove the check-then-act and the unwrap.
lib/kv-router/src/naive_indexers.rs (4)

119-143: Actor threads spawn a full Tokio runtime just for mpsc::Receiver::recv().

Each actor creates a dedicated OS thread + a current_thread Tokio runtime solely to .await on the mpsc::Receiver. Since the inner logic is entirely synchronous, a simpler approach would be to use std::sync::mpsc (or flume) and avoid the Tokio runtime overhead altogether. This is bench-only code so it's not blocking, but the extra runtime may add noise to benchmark measurements.

Also applies to: 323-347

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@lib/kv-router/src/naive_indexers.rs` around lines 119 - 143, The actor thread
currently spawns a full Tokio current_thread runtime and uses rt.block_on to
await rx.recv(), which is unnecessary because NaiveNestedMapInner and message
handling (ActorMessage variants: Event, Match, RemoveWorker) are fully
synchronous; replace the async mpsc::Receiver usage with a synchronous channel
(std::sync::mpsc or flume) and convert the thread closure to a plain blocking
loop that calls recv() (or recv_timeout if needed) to pull messages and handle
them by calling inner.apply_event, inner.find_matches (sending replies), and
inner.remove_worker, removing the Tokio runtime creation
(tokio::runtime::Builder, rt.block_on) and simplifying the spawn to a standard
thread loop over the sync receiver.

175-177: Silently dropped send errors in apply_event.

Both apply_event implementations discard the Result from self.tx.send(...). If the actor thread panics or the channel fills up (bounded at 2048), events are silently lost with no log or metric. For benchmarking correctness validation, consider at least a tracing::warn! on failure.

Also applies to: 379-381

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@lib/kv-router/src/naive_indexers.rs` around lines 175 - 177, The apply_event
implementations silently drop the Result from
self.tx.send(ActorMessage::Event(...)), which loses events if the channel is
closed or full; update both apply_event (the one using
self.tx.send(ActorMessage::Event(event)).await and the other occurrence around
the second implementation) to handle the Result: match or if let Err(e) =
self.tx.send(...).await and emit a tracing::warn! (including context like
"failed to send ActorMessage::Event" and the error) and/or increment a metric
counter for dropped events, so failures are observable and include the error
details for debugging.

244-260: Redundant re-check of position 0 in the loop.

active is cloned from self.index.get(&sequence[0]) on line 238, then the loop starting at line 244 re-checks sequence[0] (since enumerate() starts at 0). The first iteration is always a no-op. Starting from skip(1) and initializing scores accordingly would be slightly cleaner:

♻️ Suggested fix
-        for (depth, local_hash) in sequence.iter().enumerate() {
+        for (depth, local_hash) in sequence.iter().enumerate().skip(1) {
             let empty = HashSet::new();
             let workers_here = self.index.get(local_hash).unwrap_or(&empty);
 
             active.retain(|w| {
                 if workers_here.contains(w) {
                     true
                 } else {
                     scores.scores.insert(*w, depth as u32);
                     false
                 }
             });
 
             if active.is_empty() {
                 break;
             }
         }
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@lib/kv-router/src/naive_indexers.rs` around lines 244 - 260, The loop
redundantly re-checks sequence[0] because active was already cloned from
self.index.get(&sequence[0]); change the iterator to skip the first element (use
sequence.iter().enumerate().skip(1)) so the first no-op iteration is removed,
keep the body that looks up workers_here via self.index.get(local_hash) and
updates scores.scores and active as before (symbols to edit: active, sequence,
self.index, scores.scores, the enumerate loop). Ensure behavior is otherwise
unchanged.

47-67: find_matches doesn't verify prefix/positional ordering — intentional but worth a note.

This checks set membership (blocks.contains(local_hash)) without verifying that blocks appear at the correct positions in the sequence. A worker with blocks {A, C} queried with [A, B, C] correctly returns depth 1, but [C, A] would also return depth 2 even though the prefix order differs. The doc says "O(W × D) per find_matches" which matches, but this semantic gap vs. the production indexer could produce misleading benchmark comparisons if the query workload has reordered tokens.

Given this is bench-only and the doc already caveats "cut corners," this is just a heads-up for interpreting results.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@lib/kv-router/src/naive_indexers.rs` around lines 47 - 67, The current
find_matches implementation only checks set membership via
blocks.contains(LocalBlockHash) and thus counts matches regardless of
positional/prefix order; update find_matches to verify positional ordering by
using each worker's ordered block sequence (self.index entry) rather than set
membership: build or use a mapping from LocalBlockHash to its index in the
worker's block list and then, for each query sequence, only increment depth
while the next sequence hash maps to the next consecutive index (ensuring
prefix/positional order), then insert the final depth into OverlapScores; adjust
any types as needed so blocks is accessible as an ordered sequence or index map
instead of an unordered set.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@lib/kv-router/benches/mooncake_bench.rs`:
- Around line 63-72: The InvertedIndex enum variant declares num_event_workers
and documents multi-threaded behavior, but the value is never used: update the
constructors and builder to pass and honor num_event_workers (propagate the
field from InvertedIndex into the built index instance) or, if the index must
remain single-threaded, remove the field and update docs to avoid confusion;
specifically modify InvertedIndex::from_name (which currently hard-codes 0) to
parse/retain num_event_workers and change build() to use that field when
spawning event workers, or delete num_event_workers and adjust the enum variant
documentation and any references in build()/from_name accordingly.
- Around line 1057-1066: The log-spaced durations loop divides by (n - 1), which
panics when args.sweep_steps == 1; add a guard on args.sweep_steps (n) before
computing durations: if n <= 1 produce a single duration (for example
args.sweep_min_ms as u64 or args.sweep_max_ms as u64, or the geometric mean via
((args.sweep_min_ms as f64).ln() + (args.sweep_max_ms as f64).ln())/2).
Otherwise run the existing map that uses (n - 1). Ensure you reference
args.sweep_steps (n), args.sweep_min_ms, args.sweep_max_ms and the durations
Vec<u64> when implementing the guard.

In `@lib/kv-router/src/indexer.rs`:
- Line 39: The new import line use rustc_hash::FxBuildHasher; in indexer.rs is
causing rustfmt import-ordering failures; run cargo fmt (or rustfmt) on the
repository and reorder imports in lib/kv-router/src/indexer.rs so they follow
rustfmt's default ordering (grouping std, extern crates, and local modules) and
ensure the FxBuildHasher import sits in the correct group—then commit the
formatted file.

---

Nitpick comments:
In `@lib/kv-router/src/naive_indexers.rs`:
- Around line 119-143: The actor thread currently spawns a full Tokio
current_thread runtime and uses rt.block_on to await rx.recv(), which is
unnecessary because NaiveNestedMapInner and message handling (ActorMessage
variants: Event, Match, RemoveWorker) are fully synchronous; replace the async
mpsc::Receiver usage with a synchronous channel (std::sync::mpsc or flume) and
convert the thread closure to a plain blocking loop that calls recv() (or
recv_timeout if needed) to pull messages and handle them by calling
inner.apply_event, inner.find_matches (sending replies), and
inner.remove_worker, removing the Tokio runtime creation
(tokio::runtime::Builder, rt.block_on) and simplifying the spawn to a standard
thread loop over the sync receiver.
- Around line 175-177: The apply_event implementations silently drop the Result
from self.tx.send(ActorMessage::Event(...)), which loses events if the channel
is closed or full; update both apply_event (the one using
self.tx.send(ActorMessage::Event(event)).await and the other occurrence around
the second implementation) to handle the Result: match or if let Err(e) =
self.tx.send(...).await and emit a tracing::warn! (including context like
"failed to send ActorMessage::Event" and the error) and/or increment a metric
counter for dropped events, so failures are observable and include the error
details for debugging.
- Around line 244-260: The loop redundantly re-checks sequence[0] because active
was already cloned from self.index.get(&sequence[0]); change the iterator to
skip the first element (use sequence.iter().enumerate().skip(1)) so the first
no-op iteration is removed, keep the body that looks up workers_here via
self.index.get(local_hash) and updates scores.scores and active as before
(symbols to edit: active, sequence, self.index, scores.scores, the enumerate
loop). Ensure behavior is otherwise unchanged.
- Around line 47-67: The current find_matches implementation only checks set
membership via blocks.contains(LocalBlockHash) and thus counts matches
regardless of positional/prefix order; update find_matches to verify positional
ordering by using each worker's ordered block sequence (self.index entry) rather
than set membership: build or use a mapping from LocalBlockHash to its index in
the worker's block list and then, for each query sequence, only increment depth
while the next sequence hash maps to the next consecutive index (ensuring
prefix/positional order), then insert the final depth into OverlapScores; adjust
any types as needed so blocks is accessible as an ordered sequence or index map
instead of an unordered set.

In `@lib/kv-router/src/nested_map.rs`:
- Around line 303-310: The current contains_key → insert → get().unwrap()
pattern on worker_blocks is racy; replace it with the HashMap entry API: use
worker_blocks.entry(worker).or_insert_with(|| RwLock::new(FxHashMap::default()))
to obtain the RwLock for that worker atomically, then call .write() on the
returned entry to get worker_map; update the code paths that reference
worker_blocks, worker, RwLock, and FxHashMap accordingly to remove the
check-then-act and the unwrap.

Comment thread lib/kv-router/benches/mooncake_bench.rs
Comment thread lib/kv-router/benches/mooncake_bench.rs
Comment thread lib/kv-router/src/indexer.rs Outdated
Signed-off-by: PeaBrane <[email protected]>
@PeaBrane PeaBrane merged commit c5c6a55 into main Feb 18, 2026
83 of 84 checks passed
@PeaBrane PeaBrane deleted the rupei/flash-indexer-blog branch February 18, 2026 17:13
@PeaBrane PeaBrane changed the title feat: more flash indexer optimizations feat: more flash indexer optimizations [DYN-2164] Feb 21, 2026
grahamking added a commit that referenced this pull request Mar 13, 2026
We are already using that since #6305 so no Cargo.lock changes,
but we're using it as a side effect of a different package so
it's not pinned. This PR pins it.

Closes: #7328

Signed-off-by: Graham King <[email protected]>
yao531441 pushed a commit to yao531441/dynamo that referenced this pull request May 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

actions documentation Improvements or additions to documentation feat size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants