Repository navigation
Commit 7e8a562
committed
[fix](fe) Preserve streaming scan ownership and count split policy
### What problem does this PR solve?
Issue Number: N/A
Related PR: #69298, #69986
Problem Summary: Cancelled Iceberg planning can retain an independent table-generation lease when task-group lookup throws before producer cleanup is registered. Resolve that executor before acquiring the lease. Hive and Iceberg also eagerly allocate a configured maximum batch capacity, so a large valid setting can fail with OOM even for empty work; share a bounded initial allocation while preserving the configured batch maximum. Hive streaming COUNT(*) decides whether to split one partition at a time, producing excessive ranges and footer reads when the selected scan already has enough files. Use the existing concurrent cancellable producers for bounded metadata lookahead, make the scan-wide decision after all lookahead workers finish, and reuse captured metadata before lazily listing further partitions. Preserve singleton listing, fair task admission, interruption, range backpressure and the eager COUNT policy.
### Release note
Avoid excessive Hive COUNT(*) scan ranges, premature memory allocation for large split batch settings, and leaked Iceberg planning references after cancellation.
### Check List (For Author)
- Test: Unit tests and packaging
- Reproduced the Iceberg lease leak and Hive COUNT range regression before repair; isolated the allocation failure in a separate 64MB JDK process.
- All 218 selected FE tests passed with no failures, errors or skips, using run-fe-ut.sh.
- Covered both scanner versions, Parquet/ORC, metadata reuse without file caching, few-file splitting, maximum batch settings, metadata failure and lookahead cancellation.
- Full FE packaging and Checkstyle passed using build.sh --fe -j4; git diff --check passed.
- Complete P0 and external-service regression were not run locally in this cycle.
- Behavior changed: Yes. Streaming COUNT(*) uses the selected scan's split decision, large batch limits allocate only for actual work, and failed admission acquires no independent generation reference. Scheduling and cancellation continue through existing producers.
- Does this need documentation: No1 parent 663c74d commit 7e8a562
5 files changed
Lines changed: 306 additions & 21 deletions
File tree
- fe/fe-core/src
- main/java/org/apache/doris/datasource
- hive/source
- iceberg/source
- test/java/org/apache/doris/datasource
- hive/source
- iceberg/source
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
| 28 | + | |
28 | 29 | | |
29 | 30 | | |
30 | 31 | | |
| |||
38 | 39 | | |
39 | 40 | | |
40 | 41 | | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
41 | 48 | | |
42 | 49 | | |
43 | 50 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
87 | 87 | | |
88 | 88 | | |
89 | 89 | | |
| 90 | + | |
90 | 91 | | |
91 | 92 | | |
| 93 | + | |
92 | 94 | | |
93 | 95 | | |
94 | 96 | | |
| |||
248 | 250 | | |
249 | 251 | | |
250 | 252 | | |
251 | | - | |
| 253 | + | |
252 | 254 | | |
253 | 255 | | |
254 | 256 | | |
| |||
274 | 276 | | |
275 | 277 | | |
276 | 278 | | |
277 | | - | |
278 | 279 | | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
279 | 352 | | |
280 | 353 | | |
281 | | - | |
| 354 | + | |
282 | 355 | | |
283 | 356 | | |
284 | 357 | | |
285 | 358 | | |
286 | 359 | | |
287 | 360 | | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
288 | 364 | | |
289 | 365 | | |
290 | 366 | | |
| |||
298 | 374 | | |
299 | 375 | | |
300 | 376 | | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
301 | 382 | | |
302 | 383 | | |
303 | 384 | | |
| |||
309 | 390 | | |
310 | 391 | | |
311 | 392 | | |
312 | | - | |
| 393 | + | |
313 | 394 | | |
314 | 395 | | |
315 | 396 | | |
| |||
349 | 430 | | |
350 | 431 | | |
351 | 432 | | |
352 | | - | |
353 | | - | |
354 | | - | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
355 | 444 | | |
356 | | - | |
357 | | - | |
358 | 445 | | |
359 | | - | |
| 446 | + | |
360 | 447 | | |
361 | 448 | | |
362 | 449 | | |
| |||
511 | 598 | | |
512 | 599 | | |
513 | 600 | | |
514 | | - | |
515 | | - | |
| 601 | + | |
516 | 602 | | |
517 | 603 | | |
518 | 604 | | |
| |||
525 | 611 | | |
526 | 612 | | |
527 | 613 | | |
| 614 | + | |
| 615 | + | |
| 616 | + | |
| 617 | + | |
| 618 | + | |
528 | 619 | | |
529 | 620 | | |
530 | 621 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1740 | 1740 | | |
1741 | 1741 | | |
1742 | 1742 | | |
| 1743 | + | |
1743 | 1744 | | |
1744 | 1745 | | |
1745 | 1746 | | |
| |||
1754 | 1755 | | |
1755 | 1756 | | |
1756 | 1757 | | |
1757 | | - | |
| 1758 | + | |
1758 | 1759 | | |
1759 | 1760 | | |
1760 | 1761 | | |
| |||
1774 | 1775 | | |
1775 | 1776 | | |
1776 | 1777 | | |
1777 | | - | |
| 1778 | + | |
1778 | 1779 | | |
1779 | 1780 | | |
1780 | 1781 | | |
| |||
0 commit comments