Repository navigation
Disable pushdown for arrow tables that contain view columns - #20703
Conversation
|
The main issue I see is that we add yet another obscure option that does something extremely specific to our Arrow integration, and the problem of obscure options outside the default path, is that people will most likely never set them (not sure about the AI though, hehe) . It also surprises me a bit that these types still do not have their filter kernel in the main Arrow library, as they have been around for quite some time. One thing we could consider doing is to have this information coded in our library integration instead. IIRC, for pyarrow we actually construct these filters in Python land. The same would be necessary for any other library we would integrate with, right? Can we signal to DuckDB, to not to push down certain types depending on which scanner we have (e.g., if pyarrow don't pushdown these types in core). Might be a bit more infrastructure work, but we avoid a new setting. If you think that would be too much work, I would personally prefer to just skip these view filters entirely, and then do what I described here whenever some library actually implements them. |
|
Can't we just implement this logic in |
That is exactly where it is implemented in this PR, see https://github.com/duckdb/duckdb/pull/20703/changes#diff-ce1b80ec09a97c097d88eb9725ba4628316e107be28de8b3459cee69956c67a7L282 Discussed offline with @pdet: we won't add the option, when a table has a string or binary view we don't push down the filter. I'll update the PR now. |
evertlammerts
changed the title
|
LGTM, can you just merge with 1.5 again to see if CI passes? |
74392ea to
7d57546
Compare
Rebased! |
|
Thanks. Failure is unrelated! |
Date: 2026-02-03 07:57:02 -0300 Disable pushdown for arrow tables that contain view columns (duckdb/duckdb#20703) remove outdated delta patch (duckdb/duckdb#20718) Fix JSON file_extension option (duckdb/duckdb#20751) Assign next batch index in union pipelines correctly (duckdb/duckdb#20785)
Date: 2026-02-03 07:57:02 -0300 Disable pushdown for arrow tables that contain view columns (duckdb/duckdb#20703) remove outdated delta patch (duckdb/duckdb#20718) Fix JSON file_extension option (duckdb/duckdb#20751) Assign next batch index in union pipelines correctly (duckdb/duckdb#20785)
Date: 2026-02-03 07:57:02 -0300 Disable pushdown for arrow tables that contain view columns (duckdb/duckdb#20703) remove outdated delta patch (duckdb/duckdb#20718) Fix JSON file_extension option (duckdb/duckdb#20751) Assign next batch index in union pipelines correctly (duckdb/duckdb#20785)
Date: 2026-02-03 07:57:02 -0300 Disable pushdown for arrow tables that contain view columns (duckdb/duckdb#20703) remove outdated delta patch (duckdb/duckdb#20718) Fix JSON file_extension option (duckdb/duckdb#20751) Assign next batch index in union pipelines correctly (duckdb/duckdb#20785)
This is an attempt at a workaround / fix for duckdb/duckdb-python#227.
Problem
Pyarrow's
array_filterkernel throwsNotImplementedwhen it's applied on a schema that has a view type, even if the filter itself is on a different type. Since we evaluate whether we can pushdown a filter based on the type of the column we filter on (supports_pushdown_type), we might still pushdown filters that fail during the scan.Note: this is an issue for binary and string views. List views do not seem to have this problem.
Approach
We refuse to push down filters if the table contains a view type. This is a coarse solution, but for now it solves the problem and given that Arrow's own compute kernels refuse to filter tables with views, it's not unreasonable.
Alternatives
arrow_filter_pushdown_with_view_types, default = false) that gates pushdown on schemas that includes a view type. By default, it checks whetherArrowScanFunctionData::arrow_tablecontains columns with view typers, and if so, disables pushdown. Users can enable pushdown by settingarrow_filter_pushdown_with_view_types = true. We deiced against this because we already havearrow_output_list_view,produce_arrow_string_viewandarrow_output_versionthat all play together in producing views. We should really clean that up.TableFunctionDatawith some notion of pushdown capabilities that can be checked by the table function when we decide whether we can pushdown a filter. This is a nicer option that allows more control per scanner, so that when we use e.g. pyarrow to push down we can set some conditions. It's also more invasive and relatively complex, so for now we don't do this.