Repository navigation
Commit 90b55e3
test(parquet): add I/O access tests for parquet pruning, filter pushdown and TopK (apache#27022)
## Which issue does this PR close?
- Part of apache#26682
- Follow on to apache#26897
## Rationale for this change
We have several issues / PRs open that contemplate changing the I/O
pattern such as
- apache#24393
- apache#20324
- apache#26387
- apache#26418
- apache#23492
- apache#24086
- apache#25811
I want to have a test that clearly shows how those proposals effect the
scan
## What changes are included in this PR?
Add some new tests that show the Parquet I/O patterns. For example I
think this is quite clear now the difference in I/O between filter
pushdown and not
```rust
/// Without filter pushdown, the scan fetches every column of every row group
#[tokio::test]
async fn test_filter_pushdown_io_disabled() {
let test = filter_pushdown_test(false).await;
assert_snapshot!(test.query("SELECT * FROM t WHERE val = 7").await, @r"
--- Query: SELECT * FROM t WHERE val = 7 (1 rows) ---
Request 1
Read Parquet footer (offsets 0..1402)
Request 2
Read dictionary page and data page 0 for column 'id' (rg 0) (offsets 4..51)
Read dictionary page and data page 0 for column 'val' (rg 0) (offsets 51..98)
Request 3
Read dictionary page and data page 0 for column 'id' (rg 1) (offsets 98..145)
Read dictionary page and data page 0 for column 'val' (rg 1) (offsets 145..192)
Request 4
Read dictionary page and data page 0 for column 'id' (rg 2) (offsets 192..239)
Read dictionary page and data page 0 for column 'val' (rg 2) (offsets 239..286)
Produce batch (1 rows)
");
}
/// **With** filter pushdown, the scan first fetches and evaluates the filter
/// column (`val`) for each row group, and then fetches the remaining columns
/// (`id`) only for the row groups with matching rows (row group 1).
#[tokio::test]
async fn test_filter_pushdown_io_enabled() {
let test = filter_pushdown_test(true).await;
assert_snapshot!(test.query("SELECT * FROM t WHERE val = 7").await, @r"
--- Query: SELECT * FROM t WHERE val = 7 (1 rows) ---
Request 1
Read Parquet footer (offsets 0..1402)
Request 2
Read dictionary page and data page 0 for column 'val' (rg 0) (offsets 51..98)
Request 3
Read dictionary page and data page 0 for column 'val' (rg 1) (offsets 145..192)
Request 4
Read dictionary page for column 'id' (rg 1) (offsets 98..124)
Read data page 0 for column 'id' (rg 1) (offsets 124..145)
Produce batch (1 rows)
Request 5
Read dictionary page and data page 0 for column 'val' (rg 2) (offsets 239..286)
");
}
```
## What is the testing strategy for this PR?
CI
## Are there any user-facing changes?
No
---------
Co-authored-by: mohitgurav20 <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>1 parent 22aedf0 commit 90b55e3
6 files changed
Lines changed: 880 additions & 65 deletions
File tree
- datafusion/core/tests/parquet/io
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
6 | 24 | | |
7 | 25 | | |
8 | 26 | | |
| |||
21 | 39 | | |
22 | 40 | | |
23 | 41 | | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
24 | 46 | | |
25 | 47 | | |
26 | | - | |
27 | | - | |
28 | | - | |
29 | | - | |
30 | | - | |
31 | | - | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
32 | 61 | | |
33 | 62 | | |
34 | 63 | | |
| |||
43 | 72 | | |
44 | 73 | | |
45 | 74 | | |
46 | | - | |
47 | | - | |
48 | | - | |
49 | | - | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
50 | 85 | | |
51 | 86 | | |
52 | 87 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
0 commit comments