Repository navigation
Commit fecb190
committed
[improvement](ivm) Record the rows a load writes to a changelog index
### What problem does this PR solve?
Issue Number: None
Related PR: #69850
Problem Summary:
A table stream estimates the change rows a read returns from the row count of the partition's
changelog index: a read of the range between the stream offset and the end of the read returns at
most the change rows the changelog still holds. The count the frontend holds comes from the tablet
reports, which a backend sends every tablet_stat_update_interval_second (60s by default), so a read
which follows a load within that interval sees the count from before the load. The estimate falls
back to the row count of the table, which is safe but loose: it answers with the whole table where
the answer is a few change rows.
The frontend cannot derive the change rows from the rows it loaded into the base index: an update
writes change rows without adding a base row, and a delete adds none to the base but at least one to
the changelog. The backend knows them, so it now reports them.
- The publish task reports the delta rows of the changelog tablets as well, not only of the base
tablets: the report already visits the rowsets of every tablet a transaction wrote, and a
changelog tablet is one of them.
- AnalysisManager.replayUpdateRowsRecord routes those tablets to the table record by the index of
each tablet (from the tablet inverted index, which the followers have as well, so a replay
reproduces the routing without a new journal entry), and TableStatsMeta accumulates them per
index (Gson field "iur"). The base index rows are counted as before: the change rows are neither
added to them nor mistaken for them.
- TableStatsMeta.getRowCountWithDeltaRows returns the accumulated rows for a changelog index: an
analysis never collects its row count, so the rows of the loads are the count there.
A stream read consumes this through TableStatsMeta.getRowCountWithDeltaRows, which is where the
estimate of PR #69850 reads its counters from; until that lands, the recorded rows are visible in
the table statistics only.
### Release note
None
### Check List (For Author)
- Test: Unit test
- Unit test: TableStatsMetaTest, a new case (the rows a load writes to the changelog index are
accumulated per index and reported by getRowCountWithDeltaRows, while the base index rows are
not affected). TableStatsMetaTest 16/16 and AnalysisManagerTest 27/27 pass with the change.
- No regression exercises the reporting window: it needs a load followed by a read within the
tablet reporting interval, which the suites cannot time reliably. Every load now reports the
change rows as well, so the existing suites which read the row counts of a loaded table cover
the base index side of the change.
- Behavior changed: No user visible change; a load additionally reports the rows it wrote to the
changelog index.
- Does this need documentation: No1 parent 45d1598 commit fecb190
4 files changed
Lines changed: 100 additions & 0 deletions
File tree
- be/src/storage/task
- fe/fe-core/src
- main/java/org/apache/doris/statistics/analysis
- test/java/org/apache/doris/statistics/analysis
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
132 | 132 | | |
133 | 133 | | |
134 | 134 | | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
135 | 141 | | |
136 | 142 | | |
137 | 143 | | |
| |||
248 | 254 | | |
249 | 255 | | |
250 | 256 | | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
251 | 269 | | |
252 | 270 | | |
253 | 271 | | |
| |||
317 | 335 | | |
318 | 336 | | |
319 | 337 | | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
320 | 342 | | |
321 | 343 | | |
322 | 344 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
| 32 | + | |
32 | 33 | | |
33 | 34 | | |
34 | 35 | | |
| |||
1512 | 1513 | | |
1513 | 1514 | | |
1514 | 1515 | | |
| 1516 | + | |
| 1517 | + | |
1515 | 1518 | | |
| 1519 | + | |
1516 | 1520 | | |
| 1521 | + | |
| 1522 | + | |
| 1523 | + | |
| 1524 | + | |
| 1525 | + | |
| 1526 | + | |
1517 | 1527 | | |
1518 | 1528 | | |
1519 | 1529 | | |
| |||
1525 | 1535 | | |
1526 | 1536 | | |
1527 | 1537 | | |
| 1538 | + | |
1528 | 1539 | | |
1529 | 1540 | | |
1530 | 1541 | | |
| |||
1552 | 1563 | | |
1553 | 1564 | | |
1554 | 1565 | | |
| 1566 | + | |
| 1567 | + | |
| 1568 | + | |
| 1569 | + | |
| 1570 | + | |
1555 | 1571 | | |
1556 | 1572 | | |
1557 | 1573 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
99 | 99 | | |
100 | 100 | | |
101 | 101 | | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
102 | 108 | | |
103 | 109 | | |
104 | 110 | | |
| |||
391 | 397 | | |
392 | 398 | | |
393 | 399 | | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
394 | 406 | | |
395 | 407 | | |
396 | 408 | | |
| |||
414 | 426 | | |
415 | 427 | | |
416 | 428 | | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
417 | 454 | | |
418 | 455 | | |
419 | 456 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
| 28 | + | |
28 | 29 | | |
29 | 30 | | |
30 | 31 | | |
| |||
93 | 94 | | |
94 | 95 | | |
95 | 96 | | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
96 | 121 | | |
97 | 122 | | |
98 | 123 | | |
| |||
0 commit comments