Skip to content

Commit 124bcf1

Browse files
alambclaude
andcommitted
test: reproduce input_file_name() filter pushdown failure
`input_file_name()` is a source-dependent function that cannot be evaluated inside a parquet `RowFilter`. Unlike `file_row_index()`, it is not excluded from filter pushdown, so when `datafusion.execution.parquet.pushdown_filters` is enabled the predicate is pushed into the scan and the query fails with: Error evaluating filter predicate: input_file_name() is source dependent and cannot be evaluated directly This adds an slt test (currently failing) demonstrating the bug: the query should return rows 10, 20, 30 but errors instead. This matters because enabling `pushdown_filters` by default is unsafe until this is fixed. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
1 parent 36d1c98 commit 124bcf1

1 file changed

Lines changed: 28 additions & 0 deletions

File tree

‎datafusion/sqllogictest/test_files/input_file_name.slt‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -124,5 +124,33 @@ physical_plan
124124
04)------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=2
125125
05)--------DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/input_file_name/parquet/first.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/input_file_name/parquet/second.parquet]]}, projection=[input_file_name() as __datafusion_extracted_1, column1], file_type=parquet, predicate=input_file_name() LIKE %first.parquet
126126

127+
# Same query, but with parquet filter pushdown enabled.
128+
#
129+
# BUG: `input_file_name()` is a source-dependent function that cannot be
130+
# evaluated inside a parquet `RowFilter`. Unlike `file_row_index()` (which is
131+
# excluded from pushdown, see file_row_index.slt), `input_file_name()` is not
132+
# excluded, so with `pushdown_filters` enabled the predicate is pushed into the
133+
# scan and the query fails with:
134+
#
135+
# Error evaluating filter predicate: input_file_name() is source dependent
136+
# and cannot be evaluated directly
137+
#
138+
# It should instead return rows 10, 20, 30 (as it does with pushdown disabled
139+
# above). This is why enabling `pushdown_filters` by default is unsafe today.
140+
statement ok
141+
set datafusion.execution.parquet.pushdown_filters = true;
142+
143+
query I
144+
SELECT column1 FROM pq_table
145+
WHERE input_file_name() LIKE '%first.parquet'
146+
ORDER BY column1
147+
----
148+
10
149+
20
150+
30
151+
152+
statement ok
153+
reset datafusion.execution.parquet.pushdown_filters;
154+
127155
statement ok
128156
DROP TABLE pq_table;

0 commit comments

Comments
 (0)