Repository navigation
Commit 2abf69d
authored
fix(smj): handle floating-point keys across batch boundaries (apache#25134)
## Which issue does this PR close?
Closes apache#25133.
## Rationale for this change
Sort-merge mark, semi, and anti joins can fail when floating-point key
groups cross input batches. Float32/Float64 keys containing NaN can
panic, and Float16 keys can return an unsupported-type error, even
though the main merge scan supports these keys.
## What changes are included in this PR?
Use the existing `JoinKeyComparator` for batch-boundary comparisons
involving floating-point keys, matching the main merge scan's equality
semantics. Slice the right-hand keys to one row before constructing the
comparator; the left-hand keys are already single-row slices. This
bounds signed-zero normalization to the rows being compared.
Integer-only comparisons keep the existing scalar path. There are no
memory-accounting, buffering, configuration, dependency, or public-API
changes.
## What is the testing strategy for this PR?
`join_float_key_batch_boundaries` in
`datafusion/physical-plan/src/joins/sort_merge_join/tests.rs` exercises
actual one- and two-row input batches. Its 144 cases cover
Float16/32/64, left/right mark/semi/anti joins, both null-equality
modes, and ascending/descending order. Inputs include nulls, infinities,
signed zero, unmatched keys, and NaN groups spanning batches. Assertions
check exact output row IDs/order, key types, and mark values.
A physical-plan test is used because the reproducer needs precise input
batch boundaries, independent of SQL optimizer choices.
On unpatched Apache main (`40488988ad596c9b093ad60e1453430d803ce33c`)
plus only the new tests, all 144 cases fail: 96 NaN panics and 48
Float16 unsupported-type errors.
Local validation of `9ea2dea17b53b8ed2f8ea877df3d95604b87f450`:
- Physical-plan library: **2,096 passed**, including all 144 new
regression cases.
- Full extended workspace command below: **11,454 Rust tests passed, 0
failed, 8 ignored**; the SQL logic harness also completed **513/513
files** successfully.
- CLI: **107 reported passed, 0 failed**. Six environment-gated cases
did not exercise external storage.
- `cargo fmt --all -- --check`, `git diff --check`, `cargo clippy
--locked --all-targets --all-features -- -D warnings`, and the complete
`./dev/rust_lint.sh` suite, including documentation: **passed**. The
lint suite used the CI-pinned Hawkeye 7.0.0.
```sh
ulimit -n 4096
RUST_BACKTRACE=1 cargo test --locked --profile ci --no-fail-fast \
--exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-cli \
--workspace --lib --tests --bins \
--features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption
cargo test --locked --profile ci -p datafusion-cli
```
The initial workspace attempt failed four sort-only RSS runners
(`sort_no_mem_limit_runner` and `sort_with_mem_limit_{1,2,3}_runner`)
with a zero baseline RSS. All four passed an isolated serial rerun, and
the subsequent full workspace rerun passed without source/test changes.
The cause of the initial intermittent failures has not been established.
Supplemental standalone validation, not committed to this PR: an
independent row-by-row oracle passed 2,592 cases covering
asymmetric/empty input batches, filters, composite keys, both null
placements and the float/join options above. An allocation probe
confirmed the single-row slicing bounds comparator allocation
independently of the full input batch size (32 through 1,048,576 rows).
These checks supplement, rather than replace, the committed regression
tests.
The existing eight-case `sort_merge_join` release benchmark completed on
the exact upstream base and PR, with an identical harness, compiler and
lockfile. Runs were ordered base→PR and PR→base, using 30 samples,
one-second warmup and a three-second requested measurement time.
Short-run differences were mixed; the left-semi delta reversed from
+3.5% to -2.6%. Inner 1-to-1 was initially slower (+9.9% and +3.7%), so
it was repeated in three longer, alternating-order pairs (50 samples,
eight-second requested measurement time). The PR/base point-estimate
differences were -0.17%, +0.26% and -0.84%; Criterion reported change
within its noise threshold or no change. The initial slowdown did not
reproduce. These are integer-key control workloads on a development
machine, not a floating-point speedup benchmark; no speedup or general
performance-neutrality claim is made.
Upstream test workflows currently require maintainer approval
(`action_required`); local results above are not a claim that GitHub CI
or coverage checks have passed.
## Are there any user-facing changes?
Affected sort-merge joins can complete instead of failing at
floating-point batch boundaries. No public API changes.
AI-assisted contribution (OpenAI Codex).1 parent 935748d commit 2abf69d
2 files changed
Lines changed: 129 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1193 | 1193 | | |
1194 | 1194 | | |
1195 | 1195 | | |
| 1196 | + | |
| 1197 | + | |
| 1198 | + | |
| 1199 | + | |
| 1200 | + | |
| 1201 | + | |
| 1202 | + | |
| 1203 | + | |
| 1204 | + | |
| 1205 | + | |
| 1206 | + | |
| 1207 | + | |
| 1208 | + | |
| 1209 | + | |
| 1210 | + | |
1196 | 1211 | | |
1197 | 1212 | | |
1198 | 1213 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4353 | 4353 | | |
4354 | 4354 | | |
4355 | 4355 | | |
| 4356 | + | |
| 4357 | + | |
| 4358 | + | |
| 4359 | + | |
| 4360 | + | |
| 4361 | + | |
| 4362 | + | |
| 4363 | + | |
| 4364 | + | |
| 4365 | + | |
| 4366 | + | |
| 4367 | + | |
| 4368 | + | |
| 4369 | + | |
| 4370 | + | |
| 4371 | + | |
| 4372 | + | |
| 4373 | + | |
| 4374 | + | |
| 4375 | + | |
| 4376 | + | |
| 4377 | + | |
| 4378 | + | |
| 4379 | + | |
| 4380 | + | |
| 4381 | + | |
| 4382 | + | |
| 4383 | + | |
| 4384 | + | |
| 4385 | + | |
| 4386 | + | |
| 4387 | + | |
| 4388 | + | |
| 4389 | + | |
| 4390 | + | |
| 4391 | + | |
| 4392 | + | |
| 4393 | + | |
| 4394 | + | |
| 4395 | + | |
| 4396 | + | |
| 4397 | + | |
| 4398 | + | |
| 4399 | + | |
| 4400 | + | |
| 4401 | + | |
| 4402 | + | |
| 4403 | + | |
| 4404 | + | |
| 4405 | + | |
| 4406 | + | |
| 4407 | + | |
| 4408 | + | |
| 4409 | + | |
| 4410 | + | |
| 4411 | + | |
| 4412 | + | |
| 4413 | + | |
| 4414 | + | |
| 4415 | + | |
| 4416 | + | |
| 4417 | + | |
| 4418 | + | |
| 4419 | + | |
| 4420 | + | |
| 4421 | + | |
| 4422 | + | |
| 4423 | + | |
| 4424 | + | |
| 4425 | + | |
| 4426 | + | |
| 4427 | + | |
| 4428 | + | |
| 4429 | + | |
| 4430 | + | |
| 4431 | + | |
| 4432 | + | |
| 4433 | + | |
| 4434 | + | |
| 4435 | + | |
| 4436 | + | |
| 4437 | + | |
| 4438 | + | |
| 4439 | + | |
| 4440 | + | |
| 4441 | + | |
| 4442 | + | |
| 4443 | + | |
| 4444 | + | |
| 4445 | + | |
| 4446 | + | |
| 4447 | + | |
| 4448 | + | |
| 4449 | + | |
| 4450 | + | |
| 4451 | + | |
| 4452 | + | |
| 4453 | + | |
| 4454 | + | |
| 4455 | + | |
| 4456 | + | |
| 4457 | + | |
| 4458 | + | |
| 4459 | + | |
| 4460 | + | |
| 4461 | + | |
| 4462 | + | |
| 4463 | + | |
| 4464 | + | |
| 4465 | + | |
| 4466 | + | |
| 4467 | + | |
| 4468 | + | |
| 4469 | + | |
4356 | 4470 | | |
4357 | 4471 | | |
4358 | 4472 | | |
| |||
0 commit comments