You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 8768c31
Browse filesBrowse the repository at this point in the historyBrowse files
Collapse an all-InList partitioned membership check into one InList
When every non-empty build partition pushes an `InList`, the routing `CASE`
is redundant. Routing is a deterministic function of the key columns, so
every build row holding key `K` lands in the same partition a probe row
holding `K` routes to; testing `K` against the union of all the lists
therefore accepts and rejects *precisely* what the `CASE` does. This is an
equality, not the relaxation the bounds merge is.
Collapsing drops the routing hash from the probe path entirely and leaves an
`InListExpr`, which unlike a `CASE` is a shape `datafusion/pruning`
understands (up to `max_in_list_size`).
The union is capped at 1 MiB: each partition's list is independently limited
by `hash_join_inlist_pushdown_max_size`, so the concatenation grows with the
partition count, and past some size the routed `CASE` — where a probe row
only ever probes one list — is the cheaper shape. Partitions that push a hash
map, and builds where a canceled partition makes the union incomplete, keep
the `CASE`.
Co-Authored-By: Claude Opus 5 <[email protected]>
0 commit comments