Repository navigation
Commit 909f4ce
committed
fix: correlated scalar subquery returns NULL when HAVING filters out the row
A scalar subquery that aggregates with count(*) compensates unmatched
LEFT JOIN rows with the empty-input aggregate result, so outer rows with
no matching inner row get 0. That is correct in general, but when the
subquery has a HAVING clause (or a filter above the aggregate) that
references an outer column, an unmatched row can also mean the condition
rejected the aggregate output: the subquery produced no row, and the
scalar subquery must evaluate to NULL. Leaving the condition in the
join ON conflates the two cases, so a row whose real count fails the
HAVING still receives the default whenever the condition passes on it.
Record the outer-reference conditions of filters that sit directly above
a count-bug aggregate during decorrelation, leave them out of the join
ON when the subquery's output column gets a compensation expression, and
check them in the compensation CASE instead: the default (0 for count(*))
when they hold on the defaults for an unmatched row, NULL when they
reject a matched group, and the actual output otherwise. An output
whose own default is NULL has no compensation expression to check them
in, so for those the conditions keep filtering in the ON.
Closes apache#2650321 files changed
Lines changed: 2610 additions & 980 deletions
File tree
- datafusion
- core/tests/physical_optimizer
- optimizer/src
- physical-optimizer/src/ensure_requirements/enforce_sorting
- physical-plan/src
- aggregates
- joins/hash_join
- sqllogictest/test_files
- wasmtest/datafusion-wasm-app
- docs/source/contributor-guide
- test-utils/src/array_gen
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
43 | 43 | | |
44 | 44 | | |
45 | 45 | | |
46 | | - | |
| 46 | + | |
47 | 47 | | |
48 | 48 | | |
49 | 49 | | |
| |||
359 | 359 | | |
360 | 360 | | |
361 | 361 | | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
362 | 506 | | |
363 | 507 | | |
364 | 508 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
363 | 363 | | |
364 | 364 | | |
365 | 365 | | |
366 | | - | |
367 | | - | |
368 | | - | |
369 | | - | |
370 | | - | |
371 | | - | |
372 | | - | |
373 | | - | |
374 | | - | |
375 | | - | |
376 | | - | |
377 | | - | |
378 | | - | |
379 | | - | |
380 | | - | |
381 | | - | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
382 | 381 | | |
383 | 382 | | |
384 | 383 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
33 | | - | |
| 33 | + | |
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
| |||
402 | 402 | | |
403 | 403 | | |
404 | 404 | | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
405 | 411 | | |
406 | 412 | | |
407 | 413 | | |
| |||
417 | 423 | | |
418 | 424 | | |
419 | 425 | | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
420 | 432 | | |
421 | 433 | | |
422 | 434 | | |
423 | 435 | | |
424 | | - | |
425 | | - | |
426 | | - | |
427 | | - | |
428 | | - | |
429 | 436 | | |
430 | 437 | | |
431 | 438 | | |
| |||
1310 | 1317 | | |
1311 | 1318 | | |
1312 | 1319 | | |
| 1320 | + | |
| 1321 | + | |
| 1322 | + | |
| 1323 | + | |
| 1324 | + | |
| 1325 | + | |
| 1326 | + | |
| 1327 | + | |
| 1328 | + | |
| 1329 | + | |
| 1330 | + | |
| 1331 | + | |
| 1332 | + | |
| 1333 | + | |
| 1334 | + | |
| 1335 | + | |
| 1336 | + | |
| 1337 | + | |
| 1338 | + | |
| 1339 | + | |
| 1340 | + | |
| 1341 | + | |
| 1342 | + | |
| 1343 | + | |
| 1344 | + | |
| 1345 | + | |
1313 | 1346 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
102 | 102 | | |
103 | 103 | | |
104 | 104 | | |
105 | | - | |
106 | | - | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
107 | 108 | | |
108 | 109 | | |
109 | 110 | | |
| |||
427 | 428 | | |
428 | 429 | | |
429 | 430 | | |
430 | | - | |
431 | | - | |
432 | 431 | | |
433 | 432 | | |
434 | 433 | | |
435 | 434 | | |
436 | | - | |
437 | | - | |
438 | | - | |
439 | | - | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
440 | 439 | | |
441 | 440 | | |
442 | 441 | | |
| |||
446 | 445 | | |
447 | 446 | | |
448 | 447 | | |
449 | | - | |
450 | | - | |
451 | | - | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
452 | 453 | | |
453 | 454 | | |
454 | 455 | | |
| |||
0 commit comments