You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
IN filters of more than 16 values prune zones only by the set's overall min and max #101
SparkFilterToVortexExpression.convertIn (:157-204) turns an IN of up to 16 values (MAX_IN_DISJUNCTION) into an OR of equalities, which zone statistics can prune. Above 16 values it becomes a set lookup ANDed with the set's overall min/max, since "Vortex derives no zone bounds from a set".
So a 17-value IN keeps every zone between its smallest and largest value. On a file sorted by the filtered column, with values spread across the file, that's nearly every zone. A 16-value IN would have pruned all but the zones holding a value.
Example: a file sorted by a string key, read for about 80 keys spread through it. With the set lookup, almost every zone survives. With per-value pruning, only the zones holding one of the 80 keys would.
Proposal. Prune set membership per zone: keep a zone only if some value of the set falls within its [min, max]. That's a binary search over the sorted set for each zone, so large sets stay cheap. This likely belongs in the core expression's zone-statistics pruning rather than in the converter.
Activity