You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Respect Spark string collations in predicate pushdown #10065
Spark 4 string comparisons and substring predicates can use a column’s collation. Vortex’s current UTF-8 comparisons and LIKE evaluation use binary, case-sensitive semantics. The Spark connector accepts StringType predicates without checking collation, so pushing a predicate on a case-insensitive column can discard matching rows.
Push string predicates only when Vortex’s operation matches the resolved Spark collation. Add explicit collation-aware bound operations if broader pushdown is desired; otherwise leave those predicates with Spark. Cover equality, ordering, IN, STARTS_WITH, ENDS_WITH, and CONTAINS in regression tests.
Activity