Repository navigation
Commit 39a66b1
test(parquet): add I/O tracking test for Bloom filter loading baseline (apache#26897)
# Which issue does this PR close?
Part of apache#25797. Baseline for apache#25811.
# Rationale for this change
In apache#25811 we evaluate Parquet bloom filters incrementally during row
group pruning to avoid fetching filters for later columns when an
earlier column already rejects the row group.
Per review discussion on apache#25811, this PR adds a baseline test to `main`
using a tracking `AsyncFileReader` (similar to `arrow-rs` async reader
tests). It records that `main` currently fetches all candidate column
bloom filters upfront (2 requests for a 2-column query where the first
column is absent).
Once this merges, apache#25811 will rebase on top and show the skipped fetch
directly in the test diff.
# What changes are included in this PR?
- Adds `RangeTracker`, `TrackingAsyncReader`, and
`TrackingReaderFactory` in
`datafusion/datasource-parquet/src/opener/mod.rs` to record requested
byte ranges.
- Adds `test_bloom_filter_io_pattern_baseline` asserting current upfront
fetch behavior on `main`.
# Are these changes tested?
Yes, via `test_bloom_filter_io_pattern_baseline`.
# Are there any user-facing changes?
No.
---------
Co-authored-by: Andrew Lamb <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>1 parent 320a7d2 commit 39a66b1
4 files changed
Lines changed: 415 additions & 0 deletions
File tree
- datafusion/core/tests
- datasource
- parquet
- io
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
12 | 15 | | |
13 | 16 | | |
14 | 17 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
0 commit comments