You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Parquet pruning drops rows for CAST(int AS BOOLEAN) and CAST(int AS BYTEA) predicates #26981
Parquet statistics pruning drops row groups that contain matching rows when the predicate casts an integer column to BOOLEAN or BYTEA.
To Reproduce
CREATETABLEs(i INT) ASVALUES (-1), (0), (1), (255), (256);
COPY s TO '/tmp/i.parquet' STORED AS PARQUET;
CREATE EXTERNAL TABLE p STORED AS PARQUET LOCATION '/tmp/i.parquet';
Query
Actual (Parquet)
Expected (same as the in-memory table s)
SELECT count(*) FROM p WHERE CAST(i AS BOOLEAN) <= false
0
1
SELECT count(*) FROM p WHERE CAST(i AS BYTEA) < X'05'
0
3
Cause
Pruning casts the row group's min/max and compares them with the literal, which is only correct for casts that keep the order of the values. verify_support_type_for_prune in datafusion/pruning/src/pruning_predicate.rs allows any cast unless exactly one side is a string. Int → boolean (-1 and 1 are true, 0 is false) and int → binary (little-endian bytes) don't keep order, so the bounds -1 and 256 become true/true and X'FFFFFFFF'/X'00010000', and the row group is pruned. #26451 is the same cause for timestamp → time.
Expected behavior
Parquet returns the same rows as the in-memory table. I have a fix ready that makes pruning use is_order_preserving_cast, the rule sort elimination already uses.
Activity
fix: prune statistics only through casts that keep order (apache#26982)