Repository navigation
Commit 9fb10c4
committed
[fix](fe) Update partition changelog row counts on publish
### What problem does this PR solve?
Issue Number: None
Related PR: #69850, #69896
Problem Summary: Stream P/Q snapshots read partition index row counts, but
changelog load deltas were stored in a separate lifetime counter in table
statistics. That counter was not read by the partition lookup, lost its
baseline after DROP STATS, and retained removed rows after TTL or partial
TRUNCATE.
Apply published changelog deltas directly to the partition index row-count
estimate. Periodic tablet statistics replace the estimate, and partition
replacement naturally discards it. Normalize each changelog tablet's rows
using its partition replica allocation and ignore late reports for removed
tablets or partitions so changelog rows cannot enter the base counter.
Remove the unused lifetime counter and cover the partition lookup and
statistics/partition lifecycle in AnalysisManagerTest.
Cloud commits and partial publish retries remain subject to the existing
asynchronous statistics limitations. A report that already includes the
load may temporarily cause an overestimate until the next report.
### Release note
Shared-nothing loads update partition changelog row-count estimates before
periodic tablet statistics arrive.
### Check List (For Author)
- Test: Unit Test
- AnalysisManagerTest: 28 passed
- TableStatsMetaTest: 15 passed
- FE Checkstyle validation passed
- Behavior changed: Yes, publish deltas update partition changelog estimates
- Does this need documentation: No1 parent fecb190 commit 9fb10c4
4 files changed
Lines changed: 128 additions & 74 deletions
File tree
- fe/fe-core/src
- main/java/org/apache/doris/statistics/analysis
- test/java/org/apache/doris/statistics/analysis
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1516 | 1516 | | |
1517 | 1517 | | |
1518 | 1518 | | |
1519 | | - | |
1520 | 1519 | | |
1521 | | - | |
1522 | | - | |
1523 | | - | |
1524 | | - | |
1525 | | - | |
| 1520 | + | |
| 1521 | + | |
| 1522 | + | |
| 1523 | + | |
| 1524 | + | |
| 1525 | + | |
| 1526 | + | |
| 1527 | + | |
| 1528 | + | |
| 1529 | + | |
| 1530 | + | |
| 1531 | + | |
| 1532 | + | |
| 1533 | + | |
| 1534 | + | |
| 1535 | + | |
| 1536 | + | |
| 1537 | + | |
| 1538 | + | |
| 1539 | + | |
| 1540 | + | |
| 1541 | + | |
| 1542 | + | |
| 1543 | + | |
| 1544 | + | |
| 1545 | + | |
1526 | 1546 | | |
1527 | 1547 | | |
1528 | 1548 | | |
| |||
1535 | 1555 | | |
1536 | 1556 | | |
1537 | 1557 | | |
1538 | | - | |
1539 | 1558 | | |
1540 | 1559 | | |
1541 | 1560 | | |
| |||
1563 | 1582 | | |
1564 | 1583 | | |
1565 | 1584 | | |
1566 | | - | |
1567 | | - | |
1568 | | - | |
1569 | | - | |
1570 | | - | |
1571 | 1585 | | |
1572 | 1586 | | |
1573 | 1587 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
99 | 99 | | |
100 | 100 | | |
101 | 101 | | |
102 | | - | |
103 | | - | |
104 | | - | |
105 | | - | |
106 | | - | |
107 | | - | |
108 | 102 | | |
109 | 103 | | |
110 | 104 | | |
| |||
397 | 391 | | |
398 | 392 | | |
399 | 393 | | |
400 | | - | |
401 | | - | |
402 | | - | |
403 | | - | |
404 | | - | |
405 | | - | |
406 | 394 | | |
407 | 395 | | |
408 | 396 | | |
| |||
426 | 414 | | |
427 | 415 | | |
428 | 416 | | |
429 | | - | |
430 | | - | |
431 | | - | |
432 | | - | |
433 | | - | |
434 | | - | |
435 | | - | |
436 | | - | |
437 | | - | |
438 | | - | |
439 | | - | |
440 | | - | |
441 | | - | |
442 | | - | |
443 | | - | |
444 | | - | |
445 | | - | |
446 | | - | |
447 | | - | |
448 | | - | |
449 | | - | |
450 | | - | |
451 | | - | |
452 | | - | |
453 | | - | |
454 | 417 | | |
455 | 418 | | |
456 | 419 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
25 | 29 | | |
26 | 30 | | |
| 31 | + | |
27 | 32 | | |
| 33 | + | |
28 | 34 | | |
29 | 35 | | |
30 | 36 | | |
| 37 | + | |
| 38 | + | |
31 | 39 | | |
32 | 40 | | |
33 | 41 | | |
| |||
52 | 60 | | |
53 | 61 | | |
54 | 62 | | |
| 63 | + | |
55 | 64 | | |
56 | 65 | | |
57 | 66 | | |
| |||
725 | 734 | | |
726 | 735 | | |
727 | 736 | | |
| 737 | + | |
| 738 | + | |
| 739 | + | |
| 740 | + | |
| 741 | + | |
| 742 | + | |
| 743 | + | |
| 744 | + | |
| 745 | + | |
| 746 | + | |
| 747 | + | |
| 748 | + | |
| 749 | + | |
| 750 | + | |
| 751 | + | |
| 752 | + | |
| 753 | + | |
| 754 | + | |
| 755 | + | |
| 756 | + | |
| 757 | + | |
| 758 | + | |
| 759 | + | |
| 760 | + | |
| 761 | + | |
| 762 | + | |
| 763 | + | |
| 764 | + | |
| 765 | + | |
| 766 | + | |
| 767 | + | |
| 768 | + | |
| 769 | + | |
| 770 | + | |
| 771 | + | |
| 772 | + | |
| 773 | + | |
| 774 | + | |
| 775 | + | |
| 776 | + | |
| 777 | + | |
| 778 | + | |
| 779 | + | |
| 780 | + | |
| 781 | + | |
| 782 | + | |
| 783 | + | |
| 784 | + | |
| 785 | + | |
| 786 | + | |
| 787 | + | |
| 788 | + | |
| 789 | + | |
| 790 | + | |
| 791 | + | |
| 792 | + | |
| 793 | + | |
| 794 | + | |
| 795 | + | |
| 796 | + | |
| 797 | + | |
| 798 | + | |
| 799 | + | |
| 800 | + | |
| 801 | + | |
| 802 | + | |
| 803 | + | |
| 804 | + | |
| 805 | + | |
| 806 | + | |
| 807 | + | |
| 808 | + | |
| 809 | + | |
| 810 | + | |
| 811 | + | |
| 812 | + | |
| 813 | + | |
| 814 | + | |
| 815 | + | |
| 816 | + | |
| 817 | + | |
| 818 | + | |
| 819 | + | |
| 820 | + | |
| 821 | + | |
| 822 | + | |
| 823 | + | |
| 824 | + | |
| 825 | + | |
| 826 | + | |
| 827 | + | |
| 828 | + | |
| 829 | + | |
728 | 830 | | |
729 | 831 | | |
730 | 832 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
28 | | - | |
29 | 28 | | |
30 | 29 | | |
31 | 30 | | |
| |||
94 | 93 | | |
95 | 94 | | |
96 | 95 | | |
97 | | - | |
98 | | - | |
99 | | - | |
100 | | - | |
101 | | - | |
102 | | - | |
103 | | - | |
104 | | - | |
105 | | - | |
106 | | - | |
107 | | - | |
108 | | - | |
109 | | - | |
110 | | - | |
111 | | - | |
112 | | - | |
113 | | - | |
114 | | - | |
115 | | - | |
116 | | - | |
117 | | - | |
118 | | - | |
119 | | - | |
120 | | - | |
121 | 96 | | |
122 | 97 | | |
123 | 98 | | |
| |||
0 commit comments