You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 3eb7cf7
Browse filesBrowse the repository at this point in the historyBrowse files
perf: collapse partitioned hash join InList dynamic filters into one IN list
In partitioned mode the hash join pushes a routed filter:
CASE hash_repartition % N WHEN i THEN bounds_i AND key IN (list_i) ... END
When every non-empty build partition pushes an InList and no partition is
canceled, the routing is redundant. Routing is a deterministic function of
the join keys, so a build row with key K is in the partition that a probe
row with key K routes to. A test of K against the union of all lists
therefore accepts the same rows as the routed CASE. The per-partition
bounds reject no additional rows either, because every key in a list is
inside the bounds of its partition.
The filter is now `key IN (union)`. This removes the per-row routing hash
from the probe side, and the pruning code can use an InList (up to
`max_in_list_size`), which it cannot do with a CASE.
The union is capped at 1 MiB. Each partition's list is limited
independently by `hash_join_inlist_pushdown_max_size`, so the union grows
with the partition count; past the cap the routed CASE, where a probe row
checks only one list, stays in use. Partitions that push a hash table, and
builds with a canceled partition, also keep the CASE.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
- DataSourceExec: file_groups={1 group: [[test.parquet]]}, projection=[a, b, e], file_type=test, pushdown_supported=true, predicate=DynamicFilter [ CASE hash_repartition % 12 WHEN 5 THEN a@0 >= ab AND a@0 <= ab AND b@1 >= bb AND b@1 <= bb AND struct(a@0, b@1) IN (SET) ([{c0:ab,c1:bb}]) WHEN 8 THEN a@0 >= aa AND a@0 <= aa AND b@1 >= ba AND b@1 <= ba AND struct(a@0, b@1) IN (SET) ([{c0:aa,c1:ba}]) ELSE false END ]
- DataSourceExec: file_groups={1 group: [[test.parquet]]}, projection=[a, b, e], file_type=test, pushdown_supported=true, predicate=DynamicFilter [ CASE range_partition WHEN 0 THEN a@0 >= aa AND a@0 <= aa AND b@1 >= ba AND b@1 <= ba AND struct(a@0, b@1) IN (SET) ([{c0:aa,c1:ba}]) WHEN 1 THEN a@0 >= ab AND a@0 <= ab AND b@1 >= bb AND b@1 <= bb AND struct(a@0, b@1) IN (SET) ([{c0:ab,c1:bb}]) ELSE false END ]
1389
+
- DataSourceExec: file_groups={1 group: [[test.parquet]]}, projection=[a, b, e], file_type=test, pushdown_supported=true, predicate=DynamicFilter [ CASE range_partition WHEN 0 THEN a@0 >= aa AND a@0 <= aa AND b@1 >= ba AND b@1 <= ba AND hash_lookup WHEN 1 THEN a@0 >= ab AND a@0 <= ab AND b@1 >= bb AND b@1 <= bb AND hash_lookup ELSE false END ]
02)--DataSourceExec: <slt:ignore>file_type=parquet, predicate=range_key@0 = 5 OR range_key@0 = 20<slt:ignore>metrics=[output_rows=2,<slt:ignore>]
1881
-
03)--DataSourceExec: <slt:ignore>file_type=parquet, predicate=DynamicFilter [ CASE range_partition WHEN 0 THEN range_key@0 >= 5 AND range_key@0 <= 5 AND range_key@0 IN (SET) ([5]) WHEN 2 THEN range_key@0 >= 20 AND range_key@0 <= 20 AND range_key@0 IN (SET) ([20]) ELSE false END ]<slt:ignore>metrics=[output_rows=2,<slt:ignore>]
1881
+
03)--DataSourceExec: <slt:ignore>file_type=parquet, predicate=DynamicFilter [ range_key@0 IN (SET) ([5, 20]) ]<slt:ignore>metrics=[output_rows=2,<slt:ignore>]
0 commit comments