Repository navigation
Commit f391164
authored
[fix](scanner)(nereids) Harden FileScannerV2 and fix external COUNT pushdown semantics (#65548)
### What problem does this PR solve?
Issue Number: None
Related PR: None
This PR fixes two correctness areas in external file scans.
#### 1. Preserve external `COUNT(*)` / `COUNT(column)` semantics
External `COUNT(*)` retained an arbitrary scan slot after column
pruning. BE then inferred aggregate semantics from that post-pruning
scan shape and could treat the retained nullable placeholder as the
argument of `COUNT(column)`. This produced wrong results, including 9015
instead of 10000 rows for special ORC data and 116 instead of 219 rows
for Hive basic types.
The fix transports semantic COUNT arguments from Nereids through
`TPlanNode.push_down_count_slot_ids` to both file-scanner
implementations and their table/hybrid readers. The thrift field has an
explicit compatibility contract:
- absent: an old FE plan with unknown COUNT semantics; use the
conservative normal-scan path;
- present and empty: explicit `COUNT(*)` / row-count semantics;
- present and non-empty: explicit `COUNT(column)` arguments, which may
use metadata only when every mapping is proven safe.
Nereids no longer applies storage COUNT pushdown to cast arguments whose
null/error behavior would be lost. FE split planning also restricts
Iceberg/Paimon metadata-count split reduction and Hive/TVF no-split
shortcuts to explicit row-count semantics, so a `COUNT(column)` fallback
retains all data splits and normal scan parallelism. BE V1, V2, Hudi,
Paimon, and table-reader paths preserve the same argument state, reject
unsafe metadata shortcuts, and keep adaptive batching enabled for
real-row fallbacks.
#### 2. Harden FileScannerV2 schema and reader edge cases
FileScannerV2 treated valid empty splits as failures, could count
stopped EOF or malformed Native input as an empty file, localized
evolved nested predicates without fully preserving type/nullability
contracts, and rejected whole Parquet schemas when only unprojected
leaves used unsupported logical types. Parquet page-cache range metadata
also lost warm non-exact reuse while lacking a bound.
The fix distinguishes stopped reads, valid empty input, and
malformed/truncated input; keeps unsafe nested predicates above
`TableReader`; validates only semantically required Parquet projections
while preserving strict checks for real predicate and `COUNT(column)`
inputs; uses safe row-position/default carriers for placeholder-only
paths; and shares bounded per-file page-range indexes without adding a
process-wide lock to the `ReadAt` hot path.
### Release note
Fix wrong external `COUNT(*)` / `COUNT(column)` results and unsafe
metadata pushdown across Nereids, thrift, FE split planning, and BE file
readers. Also harden FileScannerV2 handling of empty or interrupted
input, evolved nested predicates, unsupported unprojected Parquet
logical types, COUNT placeholders, hybrid readers, and bounded
cross-reader page-cache range reuse.
### Check List (For Author)
- Test: Regression test / Unit Test
- BE ASAN unit coverage for V1/V2 COUNT semantics, table-level metadata
gates, schema-evolution fallbacks, COUNT placeholders, adaptive
batching, and Hudi/Paimon/Iceberg hybrid paths
- FE unit coverage for Nereids COUNT argument capture and
Hive/Iceberg/Paimon/TVF split-planning behavior
- `ColumnMapper*.*` BE ASAN suite (111 tests passed)
- Existing external regression cases `test_special_orc_formats` and
`test_hive_basic_type` reproduce the fixed COUNT(*) wrong results
- `test_file_scanner_v2_review_fixes` regression suite covers empty
input, evolved nested types, unsupported Parquet placeholders, and COUNT
fallback behavior
- Behavior changed: Yes (COUNT semantics are transported explicitly;
unsafe or ambiguous metadata COUNT plans fall back to normal scanning;
valid empty splits are skipped; stopped or malformed input is not
counted as empty; unsupported unprojected Parquet leaves no longer fail
unrelated scans)
- Does this need documentation: No1 parent 54e0e97 commit f391164
55 files changed
Lines changed: 2310 additions & 188 deletions
File tree
- be
- src
- exec
- operator
- scan
- format_v2
- native
- parquet
- table
- test
- exec/scan
- format_v2
- native
- parquet
- table
- fe/fe-core/src
- main/java/org/apache/doris
- datasource
- hive/source
- iceberg/source
- paimon/source
- tvf/source
- nereids
- glue/translator
- rules/implementation
- trees/plans/physical
- planner
- test/java/org/apache/doris
- datasource
- iceberg/source
- paimon/source
- tvf/source
- nereids/rules/rewrite
- gensrc/thrift
- regression-test
- data/external_table_p0/tvf
- suites/external_table_p0/tvf
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1046 | 1046 | | |
1047 | 1047 | | |
1048 | 1048 | | |
| 1049 | + | |
| 1050 | + | |
| 1051 | + | |
| 1052 | + | |
| 1053 | + | |
| 1054 | + | |
1049 | 1055 | | |
1050 | 1056 | | |
1051 | 1057 | | |
| |||
1231 | 1237 | | |
1232 | 1238 | | |
1233 | 1239 | | |
| 1240 | + | |
| 1241 | + | |
| 1242 | + | |
1234 | 1243 | | |
1235 | 1244 | | |
1236 | 1245 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
21 | 22 | | |
22 | 23 | | |
23 | 24 | | |
| |||
74 | 75 | | |
75 | 76 | | |
76 | 77 | | |
| 78 | + | |
77 | 79 | | |
78 | 80 | | |
79 | 81 | | |
| |||
252 | 254 | | |
253 | 255 | | |
254 | 256 | | |
| 257 | + | |
255 | 258 | | |
256 | 259 | | |
257 | 260 | | |
| |||
452 | 455 | | |
453 | 456 | | |
454 | 457 | | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
455 | 468 | | |
456 | 469 | | |
457 | 470 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
| 24 | + | |
24 | 25 | | |
25 | 26 | | |
26 | 27 | | |
| |||
80 | 81 | | |
81 | 82 | | |
82 | 83 | | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
83 | 88 | | |
84 | 89 | | |
85 | 90 | | |
| |||
334 | 339 | | |
335 | 340 | | |
336 | 341 | | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
337 | 358 | | |
338 | | - | |
339 | | - | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
340 | 364 | | |
341 | 365 | | |
342 | 366 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
272 | 272 | | |
273 | 273 | | |
274 | 274 | | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
275 | 279 | | |
276 | 280 | | |
277 | 281 | | |
| |||
322 | 326 | | |
323 | 327 | | |
324 | 328 | | |
| 329 | + | |
| 330 | + | |
325 | 331 | | |
326 | 332 | | |
327 | 333 | | |
| |||
394 | 400 | | |
395 | 401 | | |
396 | 402 | | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
397 | 417 | | |
398 | 418 | | |
399 | 419 | | |
| |||
423 | 443 | | |
424 | 444 | | |
425 | 445 | | |
426 | | - | |
427 | | - | |
428 | | - | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
429 | 452 | | |
430 | 453 | | |
431 | 454 | | |
| |||
438 | 461 | | |
439 | 462 | | |
440 | 463 | | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
441 | 474 | | |
442 | 475 | | |
443 | 476 | | |
| |||
458 | 491 | | |
459 | 492 | | |
460 | 493 | | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
461 | 509 | | |
462 | 510 | | |
463 | 511 | | |
| |||
468 | 516 | | |
469 | 517 | | |
470 | 518 | | |
| 519 | + | |
471 | 520 | | |
472 | 521 | | |
473 | 522 | | |
| |||
540 | 589 | | |
541 | 590 | | |
542 | 591 | | |
| 592 | + | |
| 593 | + | |
| 594 | + | |
| 595 | + | |
| 596 | + | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
543 | 600 | | |
544 | 601 | | |
545 | 602 | | |
| |||
809 | 866 | | |
810 | 867 | | |
811 | 868 | | |
812 | | - | |
813 | | - | |
814 | | - | |
815 | | - | |
| 869 | + | |
| 870 | + | |
| 871 | + | |
| 872 | + | |
| 873 | + | |
| 874 | + | |
| 875 | + | |
| 876 | + | |
| 877 | + | |
| 878 | + | |
| 879 | + | |
816 | 880 | | |
817 | 881 | | |
818 | 882 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
87 | 87 | | |
88 | 88 | | |
89 | 89 | | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
90 | 96 | | |
91 | 97 | | |
92 | 98 | | |
| |||
121 | 127 | | |
122 | 128 | | |
123 | 129 | | |
| 130 | + | |
124 | 131 | | |
125 | 132 | | |
126 | 133 | | |
| |||
138 | 145 | | |
139 | 146 | | |
140 | 147 | | |
| 148 | + | |
| 149 | + | |
141 | 150 | | |
142 | 151 | | |
143 | 152 | | |
| |||
181 | 190 | | |
182 | 191 | | |
183 | 192 | | |
| 193 | + | |
184 | 194 | | |
185 | 195 | | |
186 | 196 | | |
| |||
0 commit comments