Skip to content

Commit 1cb6c79

Browse files
committed
[improvement](ivm) Record the rows a load writes to a changelog index
### What problem does this PR solve? Issue Number: None Related PR: #69850 Problem Summary: A table stream estimates the change rows a read returns from the row count of the partition's changelog index: a read of the range between the stream offset and the end of the read returns at most the change rows the changelog still holds. The count the frontend holds comes from the tablet reports, which a backend sends every tablet_stat_update_interval_second (60s by default), so a read which follows a load within that interval sees the count from before the load. The estimate falls back to the row count of the table, which is safe but loose: it answers with the whole table where the answer is a few change rows. The frontend cannot derive the change rows from the rows it loaded into the base index: an update writes change rows without adding a base row, and a delete adds none to the base but at least one to the changelog. The backend knows them, so it now reports them. - The publish task reports the delta rows of the changelog tablets as well, not only of the base tablets: the report already visits the rowsets of every tablet a transaction wrote, and a changelog tablet is one of them. - AnalysisManager.replayUpdateRowsRecord routes those tablets to the table record by the index of each tablet (from the tablet inverted index, which the followers have as well, so a replay reproduces the routing without a new journal entry), and TableStatsMeta accumulates them per index (Gson field "iur"). The base index rows are counted as before: the change rows are neither added to them nor mistaken for them. - TableStatsMeta.getRowCountWithDeltaRows returns the accumulated rows for a changelog index: an analysis never collects its row count, so the rows of the loads are the count there. A stream read consumes this through TableStatsMeta.getRowCountWithDeltaRows, which is where the estimate of PR #69850 reads its counters from; until that lands, the recorded rows are visible in the table statistics only. ### Release note None ### Check List (For Author) - Test: Unit test - Unit test: TableStatsMetaTest, a new case (the rows a load writes to the changelog index are accumulated per index and reported by getRowCountWithDeltaRows, while the base index rows are not affected). TableStatsMetaTest 16/16 and AnalysisManagerTest 27/27 pass with the change. - No regression exercises the reporting window: it needs a load followed by a read within the tablet reporting interval, which the suites cannot time reliably. Every load now reports the change rows as well, so the existing suites which read the row counts of a loaded table cover the base index side of the change. - Behavior changed: No user visible change; a load additionally reports the rows it wrote to the changelog index. - Does this need documentation: No
1 parent 45d1598 commit 1cb6c79

4 files changed

Lines changed: 85 additions & 1 deletion

File tree

‎be/src/storage/task/engine_publish_version_task.cpp‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -243,8 +243,14 @@ Status EnginePublishVersionTask::execute() {
243243

244244
auto rowset_meta_ptr = rowset->rowset_meta();
245245
auto tablet_id = rowset_meta_ptr->tablet_id();
246-
if (_publish_version_req.base_tablet_ids.contains(tablet_id)) {
246+
if (_publish_version_req.base_tablet_ids.contains(tablet_id)
247+
|| tablet->is_changelog_tablet()) {
247248
// exclude delta rows in rollup tablets
249+
// The rows a load wrote to the changelog index are reported as well: the frontend
250+
// cannot derive them from the rows it loaded into the base tablets -- an update
251+
// writes change rows without adding a base row, and a delete adds none to the base
252+
// but at least one to the changelog. A table stream keeps the count to estimate how
253+
// many rows a read of the changelog returns.
248254
tablet_id_to_num_delta_rows.insert(
249255
{rowset_meta_ptr->tablet_id(), rowset_meta_ptr->num_rows()});
250256
}

‎fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/AnalysisManager.java‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@
2828
import org.apache.doris.catalog.ScalarType;
2929
import org.apache.doris.catalog.Table;
3030
import org.apache.doris.catalog.TableIf;
31+
import org.apache.doris.catalog.TabletMeta;
3132
import org.apache.doris.catalog.Tablet;
3233
import org.apache.doris.catalog.View;
3334
import org.apache.doris.catalog.info.PartitionNamesInfo;
@@ -1512,8 +1513,17 @@ public void replayUpdateRowsRecord(UpdateRowsEvent event) {
15121513
LOG.info("Tablet row count map is empty");
15131514
continue;
15141515
}
1516+
long changelogIndexId = olapTable.hasChangelog()
1517+
? olapTable.getChangelogIndexMeta().getIndexId() : -1;
15151518
long rowsForAllReplica = 0;
1519+
long changelogRowsForAllReplica = 0;
15161520
for (Entry<Long, Long> entry : tabletRows.entrySet()) {
1521+
if (changelogIndexId > 0 && isChangelogTablet(entry.getKey(), changelogIndexId)) {
1522+
// The change rows of the load, which are neither the rows loaded into the
1523+
// base index nor a lower bound of the table's rows: see TableStatsMeta.
1524+
changelogRowsForAllReplica += entry.getValue();
1525+
continue;
1526+
}
15171527
rowsForAllReplica += entry.getValue();
15181528
if (LOG.isDebugEnabled()) {
15191529
LOG.debug("Table id {}, tablet id {}, row count {}",
@@ -1525,6 +1535,7 @@ public void replayUpdateRowsRecord(UpdateRowsEvent event) {
15251535
+ "rows for all replica {}, tablets count {}",
15261536
olapTable.getName(), tableUpdateRows, replicaNum, rowsForAllReplica, tabletRows.size());
15271537
statsStatus.updatedRows.addAndGet(tableUpdateRows);
1538+
statsStatus.addIndexUpdatedRows(changelogIndexId, changelogRowsForAllReplica / replicaNum);
15281539
if (StatisticsUtil.enablePartitionAnalyze()) {
15291540
updatePartitionRows(olapTable, tabletRows, statsStatus, replicaNum);
15301541
}
@@ -1552,6 +1563,11 @@ public void replayUpdateRowsRecord(UpdateRowsEvent event) {
15521563
}
15531564
}
15541565

1566+
private static boolean isChangelogTablet(long tabletId, long changelogIndexId) {
1567+
TabletMeta tabletMeta = Env.getCurrentEnv().getTabletInvertedIndex().getTabletMeta(tabletId);
1568+
return tabletMeta != null && tabletMeta.getIndexId() == changelogIndexId;
1569+
}
1570+
15551571
protected void updatePartitionRows(OlapTable table, Map<Long, Long> originTabletToRows,
15561572
TableStatsMeta tableStats, short replicaNum) {
15571573
if (!table.isPartitionedTable()) {

‎fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/TableStatsMeta.java‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,12 @@ public class TableStatsMeta implements Writable, GsonPostProcessable {
9999
@SerializedName("newPartitionLoaded")
100100
public AtomicBoolean partitionChanged = new AtomicBoolean(false);
101101

102+
// Rows a load wrote to an index which the collected row counts do not describe, accumulated
103+
// since the record was created. The changelog index is one: no analysis collects its row count,
104+
// and the change rows a load writes are not the rows it loaded into the base index.
105+
@SerializedName("iur")
106+
private ConcurrentMap<Long, Long> indexUpdatedRows;
107+
102108
@SerializedName("userInjected")
103109
public boolean userInjected;
104110

@@ -391,6 +397,12 @@ public long getBaseIndexDeltaRowCount(OlapTable table) {
391397
* {@link #updatedRows} and reports them as delta rows.
392398
*/
393399
public synchronized long getRowCountWithDeltaRows(OlapTable table, long indexId) {
400+
if (isChangelogIndex(table, indexId)) {
401+
// The changelog index has no collected row count: the rows a load wrote to it are the
402+
// only count this record keeps, and they are added rather than taken as a lower bound
403+
// of the table's rows.
404+
return indexUpdatedRows().getOrDefault(indexId, 0L);
405+
}
394406
long rowCount = getRowCount(indexId);
395407
if (!keepsOneRowPerBaseRow(table, indexId)) {
396408
// The index aggregates, or it merges the rows of a unique key table: it has its own, smaller row
@@ -414,6 +426,31 @@ public synchronized long getRowCountWithDeltaRows(OlapTable table, long indexId)
414426
return rowCount + getBaseIndexDeltaRowCount(table);
415427
}
416428

429+
/**
430+
* Add the rows a load wrote to an index which has no collected row count of its own.
431+
*/
432+
public synchronized void addIndexUpdatedRows(long indexId, long rows) {
433+
if (rows <= 0) {
434+
return;
435+
}
436+
indexUpdatedRows().merge(indexId, rows, Long::sum);
437+
}
438+
439+
public synchronized long getIndexUpdatedRows(long indexId) {
440+
return indexUpdatedRows().getOrDefault(indexId, 0L);
441+
}
442+
443+
private static boolean isChangelogIndex(OlapTable table, long indexId) {
444+
return table.hasChangelog() && table.getChangelogIndexMeta().getIndexId() == indexId;
445+
}
446+
447+
private ConcurrentMap<Long, Long> indexUpdatedRows() {
448+
if (indexUpdatedRows == null) {
449+
indexUpdatedRows = new ConcurrentHashMap<>();
450+
}
451+
return indexUpdatedRows;
452+
}
453+
417454
public boolean isColumnsStatsEmpty() {
418455
return colToColStatsMeta == null || colToColStatsMeta.isEmpty();
419456
}

‎fe/fe-core/src/test/java/org/apache/doris/statistics/analysis/TableStatsMetaTest.java‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@
2525
import org.apache.doris.catalog.MaterializedIndexMeta;
2626
import org.apache.doris.catalog.OlapTable;
2727
import org.apache.doris.catalog.PrimitiveType;
28+
import org.apache.doris.catalog.TableIf;
2829
import org.apache.doris.common.Pair;
2930
import org.apache.doris.datasource.CatalogIf;
3031

@@ -93,6 +94,30 @@ void update() {
9394
Assertions.assertEquals(4, tableStatsMeta.rowCount);
9495
}
9596

97+
@Test
98+
void testTheChangelogRowsOfALoadAreAccumulated() {
99+
OlapTable table = mockOlapTable();
100+
long changelogIndexId = 10003L;
101+
MaterializedIndexMeta changelogMeta = Mockito.mock(MaterializedIndexMeta.class);
102+
Mockito.doReturn(changelogIndexId).when(changelogMeta).getIndexId();
103+
Mockito.doReturn(changelogMeta).when(table).getChangelogIndexMeta();
104+
Mockito.doReturn(true).when(table).hasChangelog();
105+
106+
TableStatsMeta meta = new TableStatsMeta();
107+
meta.addIndexUpdatedRows(changelogIndexId, 12);
108+
meta.addIndexUpdatedRows(changelogIndexId, 8);
109+
// Nothing to add: a load which wrote no change row must not create an entry.
110+
meta.addIndexUpdatedRows(changelogIndexId, 0);
111+
112+
// A changelog index has no collected row count, so the rows the loads wrote to it are the
113+
// count. They are neither the rows of the base index nor a lower bound of the table's rows:
114+
// an update adds a change row without adding a base row, a delete adds none to the base.
115+
Assertions.assertEquals(20, meta.getRowCountWithDeltaRows(table, changelogIndexId));
116+
Assertions.assertEquals(20, meta.getIndexUpdatedRows(changelogIndexId));
117+
Assertions.assertEquals(0, meta.getIndexUpdatedRows(BASE_INDEX_ID));
118+
Assertions.assertEquals(TableIf.UNKNOWN_ROW_COUNT, meta.getRowCountWithDeltaRows(table, BASE_INDEX_ID));
119+
}
120+
96121
@Test
97122
void testClearStaleIndexRowCount() {
98123
TableStatsMeta meta = new TableStatsMeta();

0 commit comments

Comments
 (0)