Skip to content

Avoid inferring full anti-join coverage from range bounds #25617

Open

Description

@gabotechs

Describe the bug

An anti join estimates almost complete customer coverage from customer-key range
statistics in orders, despite a third of customers having no orders. Missing NDVs are
replaced by range bounds and treated as evidence of matching values.

To Reproduce

From the repository root, with the CLI fix from
PR #25570 applied:

cargo build --profile ci --locked -p datafusion-benchmarks --bin dfbench
cargo install tpchgen-cli --version 1.1.1 --locked # if not already installed
repro_dir=$(mktemp -d)
tpchgen-cli --scale-factor 1 --format parquet \
  --parquet-compression 'ZSTD(1)' --parts 1 --output-dir "$repro_dir/data"
cat > "$repro_dir/repro.sql" <<'SQL'
SET datafusion.execution.target_partitions = 1;
SET datafusion.optimizer.enable_dynamic_filter_pushdown = false;
SELECT c_custkey FROM customer WHERE NOT EXISTS (SELECT 1 FROM orders WHERE o_custkey = c_custkey);
SQL
target/ci/dfbench statistics \
  --path "$repro_dir/data" --query_path "$repro_dir/repro.sql"

Observed with tpchgen-cli 1.1.1 at
6c320561b5.
Inspect the SELECT reports; ignore the empty SET reports.

Operator Node Estimated rows Actual rows
HashJoinExec (anti) 0 1 50,004

Expected behavior

Distinguish NDV/range upper bounds from evidence of overlapping values. Use
measured distinct counts, constraints or an explicit uncertain-overlap fallback.

Additional context

Both scan row counts are exact: 150,000 customers and 1,500,000 orders. This
removes the extra predicates from TPC-H Q22.

Part of #25610.

Activity

  1. added theissue type on Sep 22, 2026
  2. LDM-A commented on Oct 14, 2026

    @LDM-A

    take

  3. added 2 commits that reference this issue on Oct 14, 2026
    2f204ec
    a53c4f4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions