Repository navigation
Commit caaf0fa
authored
bench: measure projected Parquet scans across file boundaries (apache#27016)
## Which issue does this PR close?
Part of apache#26682.
## Rationale for this change
When each Parquet file has one row group, a reader can hide the next
file's data request by starting it while decoding the current file. A
benchmark that scans several partitions concurrently can hide the same
delay even when each reader waits for its current file to finish. This
gives apache#23492 and the cross-file experiment on apache#24086 a shared,
single-partition workload.
## What changes are included in this PR?
- Extend `sql_query_with_io` with a projected scan over 20
single-row-group files in one scan partition. The same SQL and bytes run
against an in-memory store and a store with 100 ms of simulated read
latency. Planning and data generation happen outside the timed loop.
Each execution checks the sum of the selected column over 20 million
rows.
- Add a two-file object-store test. The full projection must read the
selected column chunk once from each file and return all four values.
With `LIMIT 1`, only one file's payload is needed. File statistics are
disabled so the scan gets both files, and the assertions allow either
file order.
## What is the testing strategy for this PR?
`cargo test -p datafusion --test core_integration object_store_access`
passed all 24 tests. The extended workspace suite passed 13,396 Rust
tests and all 545 SQL logic files. Workspace formatting and Clippy with
all targets and features passed. To reproduce the benchmark, run
```sh
cargo bench -p datafusion --bench sql_query_with_io -- parquet_cross_file_projection
```
Local Criterion baseline on the PR branch, using a release build on a
shared host, four pinned CPUs and ten samples per store.
| Store | Estimated scan time | 95% confidence interval |
| --- | ---: | ---: |
| In memory | 24.6 ms | 24.3–24.9 ms |
| 100 ms per read request | 2.059 s | 2.057–2.060 s |
To check that the benchmark detects cross-file read-ahead, I also ran
the same workload against the [public cross-file prototype from
apache#24086](adriangb@64e67f5).
Both arms used the same release binary on four pinned CPUs, with
file-open depth 2, a 16 MiB read-ahead window and a shared 32 MiB
budget. Only the eager first-read cap changed, from 0 to 16 MiB. I ran
off/on/on/off, with ten samples per process. Each cell is the median of
its two process medians.
| Store on the prototype | Eager reads off | Eager reads on | Elapsed
time change |
| --- | ---: | ---: | ---: |
| In memory | 29.22 ms | 30.73 ms | 5.2% longer |
| 100 ms per read request | 2.065 s | 1.036 s | 49.8% shorter |
The prototype roughly halved elapsed time on the simulated high-latency
store and added about 1.5 ms on the in-memory control. Both
configurations produced the same expected sum. A real object-store
workload could have a different balance of I/O and decoding.
## Are there any user-facing changes?
No. This adds benchmark and integration-test coverage.1 parent ea33cce commit caaf0fa
2 files changed
Lines changed: 132 additions & 6 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
| 19 | + | |
19 | 20 | | |
20 | 21 | | |
21 | 22 | | |
| |||
67 | 68 | | |
68 | 69 | | |
69 | 70 | | |
70 | | - | |
| 71 | + | |
71 | 72 | | |
72 | 73 | | |
73 | 74 | | |
| |||
101 | 102 | | |
102 | 103 | | |
103 | 104 | | |
104 | | - | |
105 | | - | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
106 | 109 | | |
107 | 110 | | |
108 | 111 | | |
109 | 112 | | |
110 | 113 | | |
111 | 114 | | |
112 | | - | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
113 | 118 | | |
114 | 119 | | |
115 | 120 | | |
| |||
167 | 172 | | |
168 | 173 | | |
169 | 174 | | |
170 | | - | |
| 175 | + | |
| 176 | + | |
171 | 177 | | |
172 | 178 | | |
173 | 179 | | |
174 | 180 | | |
175 | 181 | | |
176 | 182 | | |
177 | 183 | | |
178 | | - | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
179 | 188 | | |
180 | 189 | | |
181 | 190 | | |
| |||
244 | 253 | | |
245 | 254 | | |
246 | 255 | | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
247 | 287 | | |
248 | 288 | | |
249 | 289 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
961 | 961 | | |
962 | 962 | | |
963 | 963 | | |
| 964 | + | |
| 965 | + | |
| 966 | + | |
| 967 | + | |
| 968 | + | |
| 969 | + | |
| 970 | + | |
| 971 | + | |
| 972 | + | |
| 973 | + | |
| 974 | + | |
| 975 | + | |
| 976 | + | |
| 977 | + | |
| 978 | + | |
| 979 | + | |
| 980 | + | |
| 981 | + | |
| 982 | + | |
| 983 | + | |
| 984 | + | |
| 985 | + | |
| 986 | + | |
| 987 | + | |
| 988 | + | |
| 989 | + | |
| 990 | + | |
| 991 | + | |
| 992 | + | |
| 993 | + | |
| 994 | + | |
| 995 | + | |
| 996 | + | |
| 997 | + | |
| 998 | + | |
| 999 | + | |
| 1000 | + | |
| 1001 | + | |
| 1002 | + | |
| 1003 | + | |
| 1004 | + | |
| 1005 | + | |
| 1006 | + | |
| 1007 | + | |
| 1008 | + | |
| 1009 | + | |
| 1010 | + | |
| 1011 | + | |
| 1012 | + | |
| 1013 | + | |
| 1014 | + | |
| 1015 | + | |
| 1016 | + | |
| 1017 | + | |
| 1018 | + | |
| 1019 | + | |
| 1020 | + | |
| 1021 | + | |
| 1022 | + | |
| 1023 | + | |
| 1024 | + | |
| 1025 | + | |
| 1026 | + | |
| 1027 | + | |
| 1028 | + | |
| 1029 | + | |
| 1030 | + | |
| 1031 | + | |
| 1032 | + | |
| 1033 | + | |
| 1034 | + | |
| 1035 | + | |
| 1036 | + | |
| 1037 | + | |
| 1038 | + | |
| 1039 | + | |
| 1040 | + | |
| 1041 | + | |
| 1042 | + | |
| 1043 | + | |
| 1044 | + | |
| 1045 | + | |
| 1046 | + | |
| 1047 | + | |
| 1048 | + | |
| 1049 | + | |
964 | 1050 | | |
965 | 1051 | | |
966 | 1052 | | |
| |||
0 commit comments