Skip to content

Commit fecb190

Browse files
committed
[improvement](ivm) Record the rows a load writes to a changelog index
### What problem does this PR solve? Issue Number: None Related PR: #69850 Problem Summary: A table stream estimates the change rows a read returns from the row count of the partition's changelog index: a read of the range between the stream offset and the end of the read returns at most the change rows the changelog still holds. The count the frontend holds comes from the tablet reports, which a backend sends every tablet_stat_update_interval_second (60s by default), so a read which follows a load within that interval sees the count from before the load. The estimate falls back to the row count of the table, which is safe but loose: it answers with the whole table where the answer is a few change rows. The frontend cannot derive the change rows from the rows it loaded into the base index: an update writes change rows without adding a base row, and a delete adds none to the base but at least one to the changelog. The backend knows them, so it now reports them. - The publish task reports the delta rows of the changelog tablets as well, not only of the base tablets: the report already visits the rowsets of every tablet a transaction wrote, and a changelog tablet is one of them. - AnalysisManager.replayUpdateRowsRecord routes those tablets to the table record by the index of each tablet (from the tablet inverted index, which the followers have as well, so a replay reproduces the routing without a new journal entry), and TableStatsMeta accumulates them per index (Gson field "iur"). The base index rows are counted as before: the change rows are neither added to them nor mistaken for them. - TableStatsMeta.getRowCountWithDeltaRows returns the accumulated rows for a changelog index: an analysis never collects its row count, so the rows of the loads are the count there. A stream read consumes this through TableStatsMeta.getRowCountWithDeltaRows, which is where the estimate of PR #69850 reads its counters from; until that lands, the recorded rows are visible in the table statistics only. ### Release note None ### Check List (For Author) - Test: Unit test - Unit test: TableStatsMetaTest, a new case (the rows a load writes to the changelog index are accumulated per index and reported by getRowCountWithDeltaRows, while the base index rows are not affected). TableStatsMetaTest 16/16 and AnalysisManagerTest 27/27 pass with the change. - No regression exercises the reporting window: it needs a load followed by a read within the tablet reporting interval, which the suites cannot time reliably. Every load now reports the change rows as well, so the existing suites which read the row counts of a loaded table cover the base index side of the change. - Behavior changed: No user visible change; a load additionally reports the rows it wrote to the changelog index. - Does this need documentation: No
1 parent 45d1598 commit fecb190

4 files changed

Lines changed: 100 additions & 0 deletions

File tree

‎be/src/storage/task/engine_publish_version_task.cpp‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -132,6 +132,12 @@ Status EnginePublishVersionTask::execute() {
132132
std::unique_ptr<ThreadPoolToken> token = _engine.tablet_publish_txn_thread_pool()->new_token(
133133
ThreadPool::ExecutionMode::CONCURRENT);
134134
std::unordered_map<int64_t, int64_t> tablet_id_to_num_delta_rows;
135+
// The changelog rowsets the transaction wrote, their rows reported once they are published.
136+
// They are collected across the retries of this call. A retry is a new call when the publish
137+
// request itself is retried, and a report whose task carries an error tablet is discarded as a
138+
// whole by the frontend (for the delta rows of the base tablets as well), so a tablet which
139+
// published in an earlier attempt contributes its rows with the next periodic report.
140+
std::vector<RowsetSharedPtr> attached_changelog_rowsets;
135141

136142
#ifndef NDEBUG
137143
if (UNLIKELY(_publish_version_req.partition_version_infos.empty())) {
@@ -248,6 +254,18 @@ Status EnginePublishVersionTask::execute() {
248254
tablet_id_to_num_delta_rows.insert(
249255
{rowset_meta_ptr->tablet_id(), rowset_meta_ptr->num_rows()});
250256
}
257+
// A load which writes a changelog keeps its changelog rowset on the base tablet's
258+
// transaction info instead of committing a changelog tablet of its own, so its rows are
259+
// reported separately below, once the publish tasks have finished: a merge-on-write
260+
// publication can still append changelog segments to that rowset. The frontend cannot
261+
// derive them from the rows it loaded into the base tablets -- an update writes change
262+
// rows without adding a base row, and a delete adds none to the base but at least one
263+
// to the changelog -- and it keeps the count to estimate how many rows a read of the
264+
// changelog returns.
265+
const ChangelogTxnInfo& attach_changelog = txn_info_it->second->attach_changelog;
266+
if (attach_changelog.rowset != nullptr) {
267+
attached_changelog_rowsets.push_back(attach_changelog.rowset);
268+
}
251269

252270
auto tablet_publish_txn_ptr = std::make_shared<TabletPublishTxnTask>(
253271
_engine, this, tablet, rowset, txn_info_it->second->attach_changelog,
@@ -317,6 +335,10 @@ Status EnginePublishVersionTask::execute() {
317335
}
318336
}
319337
}
338+
for (const RowsetSharedPtr& changelog_rowset : attached_changelog_rowsets) {
339+
const RowsetMetaSharedPtr& rowset_meta = changelog_rowset->rowset_meta();
340+
tablet_id_to_num_delta_rows.insert({rowset_meta->tablet_id(), rowset_meta->num_rows()});
341+
}
320342
_calculate_tbl_num_delta_rows(tablet_id_to_num_delta_rows);
321343

322344
if (!res.is<PUBLISH_VERSION_NOT_CONTINUOUS>()) {

‎fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/AnalysisManager.java‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@
2929
import org.apache.doris.catalog.Table;
3030
import org.apache.doris.catalog.TableIf;
3131
import org.apache.doris.catalog.Tablet;
32+
import org.apache.doris.catalog.TabletMeta;
3233
import org.apache.doris.catalog.View;
3334
import org.apache.doris.catalog.info.PartitionNamesInfo;
3435
import org.apache.doris.catalog.info.TableNameInfo;
@@ -1512,8 +1513,17 @@ public void replayUpdateRowsRecord(UpdateRowsEvent event) {
15121513
LOG.info("Tablet row count map is empty");
15131514
continue;
15141515
}
1516+
long changelogIndexId = olapTable.hasChangelog()
1517+
? olapTable.getChangelogIndexMeta().getIndexId() : -1;
15151518
long rowsForAllReplica = 0;
1519+
long changelogRowsForAllReplica = 0;
15161520
for (Entry<Long, Long> entry : tabletRows.entrySet()) {
1521+
if (changelogIndexId > 0 && isChangelogTablet(entry.getKey(), changelogIndexId)) {
1522+
// The change rows of the load, which are neither the rows loaded into the
1523+
// base index nor a lower bound of the table's rows: see TableStatsMeta.
1524+
changelogRowsForAllReplica += entry.getValue();
1525+
continue;
1526+
}
15171527
rowsForAllReplica += entry.getValue();
15181528
if (LOG.isDebugEnabled()) {
15191529
LOG.debug("Table id {}, tablet id {}, row count {}",
@@ -1525,6 +1535,7 @@ public void replayUpdateRowsRecord(UpdateRowsEvent event) {
15251535
+ "rows for all replica {}, tablets count {}",
15261536
olapTable.getName(), tableUpdateRows, replicaNum, rowsForAllReplica, tabletRows.size());
15271537
statsStatus.updatedRows.addAndGet(tableUpdateRows);
1538+
statsStatus.addIndexUpdatedRows(changelogIndexId, changelogRowsForAllReplica / replicaNum);
15281539
if (StatisticsUtil.enablePartitionAnalyze()) {
15291540
updatePartitionRows(olapTable, tabletRows, statsStatus, replicaNum);
15301541
}
@@ -1552,6 +1563,11 @@ public void replayUpdateRowsRecord(UpdateRowsEvent event) {
15521563
}
15531564
}
15541565

1566+
private static boolean isChangelogTablet(long tabletId, long changelogIndexId) {
1567+
TabletMeta tabletMeta = Env.getCurrentEnv().getTabletInvertedIndex().getTabletMeta(tabletId);
1568+
return tabletMeta != null && tabletMeta.getIndexId() == changelogIndexId;
1569+
}
1570+
15551571
protected void updatePartitionRows(OlapTable table, Map<Long, Long> originTabletToRows,
15561572
TableStatsMeta tableStats, short replicaNum) {
15571573
if (!table.isPartitionedTable()) {

‎fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/TableStatsMeta.java‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,12 @@ public class TableStatsMeta implements Writable, GsonPostProcessable {
9999
@SerializedName("newPartitionLoaded")
100100
public AtomicBoolean partitionChanged = new AtomicBoolean(false);
101101

102+
// Rows a load wrote to an index which the collected row counts do not describe, accumulated
103+
// since the record was created. The changelog index is one: no analysis collects its row count,
104+
// and the change rows a load writes are not the rows it loaded into the base index.
105+
@SerializedName("iur")
106+
private ConcurrentMap<Long, Long> indexUpdatedRows;
107+
102108
@SerializedName("userInjected")
103109
public boolean userInjected;
104110

@@ -391,6 +397,12 @@ public long getBaseIndexDeltaRowCount(OlapTable table) {
391397
* {@link #updatedRows} and reports them as delta rows.
392398
*/
393399
public synchronized long getRowCountWithDeltaRows(OlapTable table, long indexId) {
400+
if (isChangelogIndex(table, indexId)) {
401+
// The changelog index has no collected row count: the rows a load wrote to it are the
402+
// only count this record keeps, and they are added rather than taken as a lower bound
403+
// of the table's rows.
404+
return indexUpdatedRows().getOrDefault(indexId, 0L);
405+
}
394406
long rowCount = getRowCount(indexId);
395407
if (!keepsOneRowPerBaseRow(table, indexId)) {
396408
// The index aggregates, or it merges the rows of a unique key table: it has its own, smaller row
@@ -414,6 +426,31 @@ public synchronized long getRowCountWithDeltaRows(OlapTable table, long indexId)
414426
return rowCount + getBaseIndexDeltaRowCount(table);
415427
}
416428

429+
/**
430+
* Add the rows a load wrote to an index which has no collected row count of its own.
431+
*/
432+
public synchronized void addIndexUpdatedRows(long indexId, long rows) {
433+
if (rows <= 0) {
434+
return;
435+
}
436+
indexUpdatedRows().merge(indexId, rows, Long::sum);
437+
}
438+
439+
public synchronized long getIndexUpdatedRows(long indexId) {
440+
return indexUpdatedRows().getOrDefault(indexId, 0L);
441+
}
442+
443+
private static boolean isChangelogIndex(OlapTable table, long indexId) {
444+
return table.hasChangelog() && table.getChangelogIndexMeta().getIndexId() == indexId;
445+
}
446+
447+
private ConcurrentMap<Long, Long> indexUpdatedRows() {
448+
if (indexUpdatedRows == null) {
449+
indexUpdatedRows = new ConcurrentHashMap<>();
450+
}
451+
return indexUpdatedRows;
452+
}
453+
417454
public boolean isColumnsStatsEmpty() {
418455
return colToColStatsMeta == null || colToColStatsMeta.isEmpty();
419456
}

‎fe/fe-core/src/test/java/org/apache/doris/statistics/analysis/TableStatsMetaTest.java‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@
2525
import org.apache.doris.catalog.MaterializedIndexMeta;
2626
import org.apache.doris.catalog.OlapTable;
2727
import org.apache.doris.catalog.PrimitiveType;
28+
import org.apache.doris.catalog.TableIf;
2829
import org.apache.doris.common.Pair;
2930
import org.apache.doris.datasource.CatalogIf;
3031

@@ -93,6 +94,30 @@ void update() {
9394
Assertions.assertEquals(4, tableStatsMeta.rowCount);
9495
}
9596

97+
@Test
98+
void testTheChangelogRowsOfALoadAreAccumulated() {
99+
OlapTable table = mockOlapTable();
100+
long changelogIndexId = 10003L;
101+
MaterializedIndexMeta changelogMeta = Mockito.mock(MaterializedIndexMeta.class);
102+
Mockito.doReturn(changelogIndexId).when(changelogMeta).getIndexId();
103+
Mockito.doReturn(changelogMeta).when(table).getChangelogIndexMeta();
104+
Mockito.doReturn(true).when(table).hasChangelog();
105+
106+
TableStatsMeta meta = new TableStatsMeta();
107+
meta.addIndexUpdatedRows(changelogIndexId, 12);
108+
meta.addIndexUpdatedRows(changelogIndexId, 8);
109+
// Nothing to add: a load which wrote no change row must not create an entry.
110+
meta.addIndexUpdatedRows(changelogIndexId, 0);
111+
112+
// A changelog index has no collected row count, so the rows the loads wrote to it are the
113+
// count. They are neither the rows of the base index nor a lower bound of the table's rows:
114+
// an update adds a change row without adding a base row, a delete adds none to the base.
115+
Assertions.assertEquals(20, meta.getRowCountWithDeltaRows(table, changelogIndexId));
116+
Assertions.assertEquals(20, meta.getIndexUpdatedRows(changelogIndexId));
117+
Assertions.assertEquals(0, meta.getIndexUpdatedRows(BASE_INDEX_ID));
118+
Assertions.assertEquals(TableIf.UNKNOWN_ROW_COUNT, meta.getRowCountWithDeltaRows(table, BASE_INDEX_ID));
119+
}
120+
96121
@Test
97122
void testClearStaleIndexRowCount() {
98123
TableStatsMeta meta = new TableStatsMeta();

0 commit comments

Comments
 (0)