Repository navigation
Distinguish string equality, LIKE and NOT LIKE estimates #25614
Description
Describe the bug
String equality and pattern predicates all receive the same 20% fallback. LIKE
and NOT LIKE also get identical estimates even though they partition this
non-null input.
To Reproduce
From the repository root, with the CLI fix from
PR #25570 applied:
cargo build --profile ci --locked -p datafusion-benchmarks --bin dfbench
cargo install tpchgen-cli --version 1.1.1 --locked # if not already installed
repro_dir=$(mktemp -d)
tpchgen-cli --scale-factor 1 --format parquet \
--parquet-compression 'ZSTD(1)' --parts 1 --output-dir "$repro_dir/data"
cat > "$repro_dir/repro.sql" <<'SQL'
SET datafusion.execution.target_partitions = 1;
SET datafusion.optimizer.enable_dynamic_filter_pushdown = false;
SELECT o_orderkey FROM orders WHERE o_comment LIKE '%special%requests%';
SELECT o_orderkey FROM orders WHERE o_comment NOT LIKE '%special%requests%';
SELECT n_nationkey FROM nation WHERE n_name = 'GERMANY';
SELECT s_suppkey FROM supplier WHERE s_comment LIKE '%Customer%Complaints%';
SQL
target/ci/dfbench statistics \
--path "$repro_dir/data" --query_path "$repro_dir/repro.sql"Observed with tpchgen-cli 1.1.1 at
6c320561b5.
Inspect the SELECT reports; ignore the empty SET reports.
| SELECT | FilterExec node | Estimated rows | Actual rows |
|---|---|---|---|
| Orders LIKE | 0 |
300,000 | 16,082 |
| Orders NOT LIKE | 0 |
300,000 | 1,483,918 |
| Nation equality | 0 |
5 | 1 |
| Supplier LIKE | 0 |
2,000 | 4 |
Expected behavior
Use NDVs/frequencies for equality where available, distinguish pattern forms,
and make LIKE/NOT LIKE estimates complementary over non-null rows.
Additional context
The predicates come from TPC-H Q11/Q13/Q16. Related:
filter selectivity #14237.
Part of #25610.
Activity
fix: Estimate string equality and LIKE filter selectivity (apache#25835)
fix: Estimate string equality and LIKE filter selectivity (apache#25835)