Repository navigation
Commit 0419942
authored
[arrow] Fix pushdown errors (duckdb#27319)
Fixes duckdb/duckdb-python#663, duckdb/duckdb-python#665,
duckdb/duckdb-python#666 and the engine half of duckdb#664.
The `arrow_scan` function refuses filter pushdown per column through
`supports_pushdown_type`, because the Arrow producer applies the pushed
filter itself and never gets it re-checked. pyarrow cannot evaluate
filters on string or binary view arrays, on run-end encoded arrays, or
on narrow decimals. The gate missed three cases, so filters reached
pyarrow and failed loud, in two cases with an INTERNAL error that
invalidates the database.
This PR closes the three gaps:
- A dictionary column was judged by its index type, so a dictionary of
`string_view`, `binary_view` or a narrow decimal was pushed although the
plain type is refused. The gate now judges a dictionary by its value
type. A dictionary of view values does not turn pushdown off for the
table, since pyarrow filters a dictionary by its indices.
- A run-end encoded column kept the `run_ends, values` struct as its
type info while its duck type became the values' type, so reading the
string info at planning threw an INTERNAL error for any filter or Top-N
on the table. The gate now checks run-end encoding before it reads any
type info, and treats it like a view type: pushdown is off for the whole
table.
- Join filter pushdown never consulted `supports_pushdown_type`, so
dynamic join filters reached the producer for columns the static filter
path keeps above the scan. `GetPushdownFilterTargets` now drops the
refused columns and keeps the rest; a scan with no accepted column gets
no target. This applies to every caller, Top-N included. Top-N's filter
above the scan was an optional filter that passed every row, so nothing
is lost; measured parity.
Tests hand-build the Arrow columns DuckDB's own export never produces
and scan them through a producer that records which filters it receives.
Notes:
- The run-end encoded type info inconsistency also breaks binding of
run-end encoded lists and mistypes run-end encoded structs. That is a
separate fix.
- None of these is a regression; all reproduce on 1.5.6.3 files changed
Lines changed: 421 additions & 12 deletions
File tree
- src
- function/table
- optimizer
- test/arrow
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
213 | 213 | | |
214 | 214 | | |
215 | 215 | | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
216 | 222 | | |
217 | 223 | | |
218 | 224 | | |
| |||
266 | 272 | | |
267 | 273 | | |
268 | 274 | | |
269 | | - | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
270 | 282 | | |
271 | 283 | | |
272 | 284 | | |
| |||
277 | 289 | | |
278 | 290 | | |
279 | 291 | | |
280 | | - | |
| 292 | + | |
281 | 293 | | |
282 | 294 | | |
283 | 295 | | |
| |||
286 | 298 | | |
287 | 299 | | |
288 | 300 | | |
289 | | - | |
| 301 | + | |
290 | 302 | | |
291 | 303 | | |
292 | 304 | | |
293 | | - | |
| 305 | + | |
294 | 306 | | |
295 | 307 | | |
296 | 308 | | |
297 | 309 | | |
298 | 310 | | |
299 | 311 | | |
300 | | - | |
| 312 | + | |
301 | 313 | | |
302 | | - | |
| 314 | + | |
303 | 315 | | |
304 | 316 | | |
305 | 317 | | |
| |||
309 | 321 | | |
310 | 322 | | |
311 | 323 | | |
312 | | - | |
313 | | - | |
314 | | - | |
315 | | - | |
316 | | - | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
317 | 327 | | |
318 | 328 | | |
319 | 329 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
233 | 233 | | |
234 | 234 | | |
235 | 235 | | |
| 236 | + | |
236 | 237 | | |
237 | 238 | | |
238 | 239 | | |
239 | 240 | | |
240 | 241 | | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
241 | 248 | | |
242 | 249 | | |
243 | 250 | | |
| |||
254 | 261 | | |
255 | 262 | | |
256 | 263 | | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
257 | 268 | | |
258 | | - | |
| 269 | + | |
259 | 270 | | |
260 | 271 | | |
261 | 272 | | |
| |||
0 commit comments