Skip to content

Commit 70f2ed0

Browse files
adriangbclaude
andauthored
bench: add hj queries for partitioned joins on dense and sparse integer keys (#27090)
## Which issue does this PR close? - This PR does not close an issue. It adds a baseline for a follow-up PR that changes how a `Partitioned` hash join with dense integer keys filters its probe side. ## Rationale for this change No `hj` query runs a `Partitioned` join on integer keys whose build partitions are too large for an IN list. Such a join builds an `ArrayMap` per partition when the keys are dense, and a hash table when they are sparse. The suite has no query of either kind, so a change to how these joins filter the probe side has nothing to compare against. ## What changes are included in this PR? One commit. It adds two `hj` queries over TPC-H `orders`, joined to itself on `o_custkey` in `Partitioned` mode: | Query | Build side | Keys | |---|---|---| | Q26 | orders of the first 40K customers (~270K rows) | dense, about 10 rows per key | | Q27 | orders of customers whose key is a multiple of 37 | sparse | Both probe 15M rows with about a 2.7% hit rate. The runner now also registers `orders`. The queries set no Parquet options, so a caller can run them with or without filter pushdown, for example with `DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS=true`. ## What is the testing strategy for this PR? The change adds benchmark queries only. I ran both queries on TPC-H SF10 (16 partitions, 10 iterations) and checked their plans: both use `HashJoinExec: mode=Partitioned`. With pushdown on, Q26's probe scan gets a per-partition routed filter over `ArrayMap` partitions, and Q27's gets a merged hash lookup. | Query | `pushdown_filters=false` | `pushdown_filters=true` | |---|---|---| | Q26 (dense) | 50.0 ms | 93.5 ms | | Q27 (sparse) | 54.5 ms | 59.2 ms | ## Are there any user-facing changes? No. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent aeba33f commit 70f2ed0

1 file changed

Lines changed: 27 additions & 1 deletion

File tree

‎benchmarks/src/hj.rs‎

Lines changed: 27 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -505,6 +505,32 @@ const HASH_QUERIES: &[HashJoinQuery] = &[
505505
probe_size: "240K_multi_key_count",
506506
isolate_partitioned_join: true,
507507
},
508+
// Q26: Partitioned join on dense integer keys
509+
// Build Side: orders of the first 40K customers (~270K rows, ~27K distinct
510+
// dense keys, ~10 rows per key) | Probe Side: orders (15M rows), ~2.7% hit
511+
// rate. The probe keys are not sorted, so statistics prune nothing.
512+
HashJoinQuery {
513+
sql: r###"SELECT count(*)
514+
FROM (SELECT o_custkey as k FROM orders WHERE o_custkey <= 40000) b
515+
JOIN (SELECT o_custkey as k FROM orders) p ON b.k = p.k"###,
516+
density: 1.0,
517+
prob_hit: 0.027,
518+
build_size: "270K_dense_keys",
519+
probe_size: "15M",
520+
isolate_partitioned_join: true,
521+
},
522+
// Q27: Q26 with sparse integer keys: a similar build size and hit rate,
523+
// with keys 37 apart.
524+
HashJoinQuery {
525+
sql: r###"SELECT count(*)
526+
FROM (SELECT o_custkey as k FROM orders WHERE o_custkey % 37 = 0) b
527+
JOIN (SELECT o_custkey as k FROM orders) p ON b.k = p.k"###,
528+
density: 0.027,
529+
prob_hit: 0.027,
530+
build_size: "270K_sparse_keys",
531+
probe_size: "15M",
532+
isolate_partitioned_join: true,
533+
},
508534
];
509535

510536
impl RunOpt {
@@ -532,7 +558,7 @@ impl RunOpt {
532558
let ctx = SessionContext::new_with_config_rt(config, rt);
533559

534560
if let Some(path) = &self.path {
535-
for table in &["lineitem", "supplier", "nation", "customer"] {
561+
for table in &["lineitem", "orders", "supplier", "nation", "customer"] {
536562
let table_path = path.join(table);
537563
if !table_path.exists() {
538564
return exec_err!(

0 commit comments

Comments
 (0)