Repository navigation
feat(file): a coalescing IO service for planning-protocol scans - #10463
Closed
joseph-isaacs wants to merge 1 commit into
Closed
joseph-isaacs wants to merge 1 commit into
joseph-isaacs wants to merge 1 commit into
Conversation
Merging this PR will improve performance by 15%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | WallTime | threshold_neon[64] |
5.6 µs | 4.5 µs | +25.71% |
| ⚡ | WallTime | threshold_neon[80] |
5.6 µs | 4.9 µs | +14.9% |
| ⚡ | WallTime | filter_neon[64] |
3.3 µs | 2.8 µs | +14.9% |
| ⚡ | WallTime | starts_with_avx512[8, VarBin] |
46.2 µs | 40.7 µs | +13.57% |
| ⚡ | WallTime | threshold_neon[8] |
3.4 µs | 3 µs | +10.79% |
| ⚡ | WallTime | starts_with_avx512[64, VarBin] |
59.7 µs | 53.9 µs | +10.76% |
Tip
Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ji/scan-v2-io (22871ce) with develop (ce04bf9)
joseph-isaacs
force-pushed
the
ji/scan-v2-plan
branch
from
October 13, 2026 12:52
ad53eaf to
d4d0acd
Compare
joseph-isaacs
force-pushed
the
ji/scan-v2-io
branch
from
October 13, 2026 12:52
82aa9ef to
27bffeb
Compare
joseph-isaacs
force-pushed
the
ji/scan-v2-io
branch
from
October 13, 2026 13:11
27bffeb to
3c37da1
Compare
Serves the planning protocol's `IoService` reads of a file: - `FileIoService` coalesces the ranges a batch of requests or announcements asks for, reads each range once for every session interested in it, and withdraws a dropped session's interest. `FileScanIo` serves the same protocol through the file's segment source and read driver. - Local files reuse their handles and admit native reads up to a bound; object-store reads stream completions as they land. - The read driver keeps its coalescing scratch between passes and can leave optional requests out of coalescing. - Segment caches can answer synchronously from `get_if_ready`. - Request tracing ids for diagnostics. A file opened over a reader carries the service as `VortexFile::scan_io`. Nothing drives it yet; the planning scan that does follows separately. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01P1oQtzTgSzkvrGDKisoNko Signed-off-by: Claude <[email protected]>
joseph-isaacs
force-pushed
the
ji/scan-v2-io
branch
from
October 15, 2026 10:47
3c37da1 to
22871ce
Compare
This was referenced Oct 15, 2026
Contributor
Author
|
Closing as superseded: this is the same change as #10548. V2 scans on Generated by Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Carved out of #10339 alongside the plan PR (#10462) and the planning PR. This PR is independent: it targets
developdirectly and does not depend on #10462. It adds the IO layer that the planner-driven scan reads through. Nothing drives it until the planning PR.Changes
FileIoService. Coalesces the ranges a batch of planning-protocol requests or announcements asks for. Each range is read once for every session interested in it, and a dropped session's interest is withdrawn.FileScanIo. Serves the same protocol through the file's segment source and read driver, sharing pending reads between live consumers.VORTEX_SCAN_IO_COALESCE_OPTIONAL).SegmentCache::get_if_ready.VortexFile::scan_io;FileIoServiceis opt-in through its env settings.Tests:
segments::{io_service, scan_io}.Checks run:
cargo check -p vortex-io -p vortex-file -p vortex-layout --all-targets --all-features(after moving ontodevelop)cargo clippy -p vortex-io -p vortex-file -p vortex-layout --all-targets --all-features -- -D warnings,cargo +nightly-2026-09-10 fmt,cargo test -p vortex-io -p vortex-file -p vortex-layout --lib🤖 Generated with Claude Code
https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9