Repository navigation
Commit 23339c5
committed
fix(parquet): gate runtime RG-prune rebuild on is_at_row_group_boundary
ClickBench Q24 / Q26 (`SELECT … WHERE x <> '' ORDER BY ts LIMIT 10`)
were failing with:
Parquet error: into_builder called mid-row-group;
check is_at_row_group_boundary() first
`transition()` in `push_decoder.rs` re-enters Step 2 on every loop
iteration — including the iterations where Step 3 returned `NeedsData`
and we pushed byte ranges but the decoder has not yet handed back a
reader. At those moments the decoder sits in `ReadingRowGroup` state
but `is_at_row_group_boundary()` is `false`, and arrow-rs's
`into_builder` errors out (it can only rebuild at a clean RG boundary).
Step 2's prune-and-rebuild block now skips when the decoder is mid-RG.
Step 3 still drives the decoder forward in that iteration, and the next
boundary re-enters Step 2 with the pruner in the same state (the pruner
is stateful but idempotent — re-evaluating costs one cached
`pp.prune` per RG, no rebuild). No semantic change for queries that
were already passing.
Add `dynamic_rg_pruner_does_not_call_into_builder_mid_row_group`: a
20-RG × 50-row file plus `ORDER BY v ASC LIMIT 10` gives the pruner
multiple boundaries to attempt rebuilds and reliably trips the failure
mode on the pre-fix code.1 parent 1b55225 commit 23339c5
2 files changed
Lines changed: 94 additions & 8 deletions
File tree
- datafusion
- core/tests/parquet
- datasource-parquet/src
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
207 | 207 | | |
208 | 208 | | |
209 | 209 | | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
210 | 283 | | |
211 | 284 | | |
212 | 285 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
315 | 315 | | |
316 | 316 | | |
317 | 317 | | |
318 | | - | |
319 | | - | |
320 | | - | |
321 | | - | |
322 | | - | |
323 | | - | |
324 | | - | |
325 | | - | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
326 | 339 | | |
327 | 340 | | |
328 | 341 | | |
| |||
0 commit comments