Skip to content

Commit 1183159

Browse files
yx-keithmorningmanclaude
authored andcommitted
[fix](fe) avoid concurrent tablet stat iteration failures (#63298)
### What problem does this PR solve? Issue Number: #59138 Problem Summary: TabletStatMgr.runAfterCatalogReady() is a periodic master-FE daemon that iterates every tablet/replica to pull statistics. When DDL runs concurrently with this daemon, two races fire: Iteration race (CME). MaterializedIndex.tablets and LocalTablet.replicas were plain ArrayLists whose getters returned the internal list. A concurrent addTablet / addReplica / deleteReplica (clone, repair, schema change, restore, report handler) during iteration triggered the fail-fast iterator and threw ConcurrentModificationException. TOCTOU race. In updateTabletStat, a getTabletMeta(id) != null check is followed by getReplica(id, beId). If the tablet is removed in between, getReplica hits Preconditions.checkState(...) and throws IllegalStateException. When the daemon throws, the current cycle leaves stale tablet/partition/table sizes and skewed MetricRepo metrics until the next cycle. Solution: Close the CME race for good with copy-on-write via a volatile snapshot. A first attempt returned a defensive copy (Lists.newArrayList(...)), but the copy itself iterates the source list and can still CME mid-copy — the window shrank but did not close. This PR instead: Makes LocalTablet.replicas and MaterializedIndex.tablets volatile. Writers (addReplica / deleteReplica / deleteReplicaByBackendId / addTablet / clearTabletsForRestore) are synchronized, build a new list, and atomically swap the volatile reference — they never mutate a list in place. Readers (getReplicas() / getTablets()) do a single volatile read and return an immutable snapshot (Collections.unmodifiableList). Iteration is lock-free and can never CME, and the hot read path no longer copies elements. synchronized on writers is required (not just volatile) because some write paths do not hold the OlapTable write lock — verified by tracing call sites: InternalCatalog.createPartitionWithIndices and RestoreJob.resetPartitionForRestore call addReplica/addTablet without the table write lock, so concurrent writers are real and a plain volatile field would allow lost updates. Writers are infrequent (DDL / repair / restore), so the lock cost is negligible; reads stay lock-free. TOCTOU race is handled by catching IllegalStateException around getReplica (kept from the original fix) and counting the skip via a new TabletStatMgr.staleTabletStatSkipped counter, which makes the race observable (>0 proves the window was actually hit) instead of relying solely on log scraping. Cloud path: CloudTabletStatMgr.updateStatInfo iterates tablet.getReplicas() and is covered by the same snapshot fix; its updateTabletStat uses getReplicasByTabletId (locked, returns empty list, no checkState) and is already safe. ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> --------- Co-authored-by: morningman <[email protected]> Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
1 parent 722d5a6 commit 1183159

13 files changed

Lines changed: 360 additions & 74 deletions

File tree

‎fe/fe-core/src/main/java/org/apache/doris/alter/MaterializedViewHandler.java‎

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -437,14 +437,22 @@ private RollupJobV2 createMaterializedViewJob(String rawSql, String mvName, Stri
437437
MaterializedIndex mvIndex = new MaterializedIndex(mvIndexId, IndexState.SHADOW);
438438
MaterializedIndex baseIndex = partition.getIndex(baseIndexId);
439439
short replicationNum = olapTable.getPartitionInfo().getReplicaAllocation(partitionId).getTotalReplicaNum();
440+
// All MV tablets of the same (partition, mv index) share the same TabletMeta;
441+
// build it once and bulk-publish to MaterializedIndex.tablets after the per-tablet
442+
// loop to keep copy-on-write O(n). TabletInvertedIndex registration stays
443+
// per-iteration because Tablet.addReplica(...) below needs the tablet present
444+
// in the inverted index.
445+
TabletMeta mvTabletMeta = new TabletMeta(
446+
dbId, tableId, partitionId, mvIndexId, mvSchemaHash, medium);
447+
List<Tablet> mvTabletsForPartition = Lists.newArrayListWithCapacity(baseIndex.getTablets().size());
448+
TabletInvertedIndex invertedIndex = Env.getCurrentInvertedIndex();
440449
for (Tablet baseTablet : baseIndex.getTablets()) {
441-
TabletMeta mvTabletMeta = new TabletMeta(
442-
dbId, tableId, partitionId, mvIndexId, mvSchemaHash, medium);
443450
long baseTabletId = baseTablet.getId();
444451
long mvTabletId = idGeneratorBuffer.getNextId();
445452

446453
Tablet newTablet = EnvFactory.getInstance().createTablet(mvTabletId);
447-
mvIndex.addTablet(newTablet, mvTabletMeta);
454+
invertedIndex.addTablet(mvTabletId, mvTabletMeta);
455+
mvTabletsForPartition.add(newTablet);
448456
addedTablets.add(newTablet);
449457

450458
mvJob.addTabletIdMap(partitionId, mvTabletId, baseTabletId);
@@ -499,6 +507,9 @@ private RollupJobV2 createMaterializedViewJob(String rawSql, String mvName, Stri
499507
}
500508
} // end for baseTablets
501509

510+
// Bulk-publish all MV tablets for this partition in one copy-on-write.
511+
mvIndex.appendTablets(mvTabletsForPartition);
512+
502513
mvJob.addMVIndex(partitionId, mvIndex);
503514

504515
if (LOG.isDebugEnabled()) {

‎fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java‎

Lines changed: 16 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,7 @@
6666
import org.apache.doris.catalog.Table;
6767
import org.apache.doris.catalog.TableIf.TableType;
6868
import org.apache.doris.catalog.Tablet;
69+
import org.apache.doris.catalog.TabletInvertedIndex;
6970
import org.apache.doris.catalog.TabletMeta;
7071
import org.apache.doris.cloud.qe.ComputeGroupException;
7172
import org.apache.doris.common.AnalysisException;
@@ -1680,14 +1681,23 @@ private void createJob(String rawSql, long dbId, OlapTable olapTable, Map<Long,
16801681
MaterializedIndex originIndex = partition.getIndex(originIndexId);
16811682
ReplicaAllocation replicaAlloc = olapTable.getPartitionInfo().getReplicaAllocation(partitionId);
16821683
Short totalReplicaNum = replicaAlloc.getTotalReplicaNum();
1684+
// All shadow tablets of the same (partition, shadow index) share the same TabletMeta;
1685+
// build it once and bulk-publish to MaterializedIndex.tablets after the per-tablet
1686+
// loop to keep copy-on-write O(n). TabletInvertedIndex registration stays
1687+
// per-iteration because Tablet.addReplica(...) below needs the tablet present
1688+
// in the inverted index.
1689+
TabletMeta shadowTabletMeta = new TabletMeta(dbId, tableId, partitionId, shadowIndexId,
1690+
newSchemaHash, medium);
1691+
List<Tablet> shadowTabletsForPartition = Lists.newArrayListWithCapacity(
1692+
originIndex.getTablets().size());
1693+
TabletInvertedIndex invertedIndex = Env.getCurrentInvertedIndex();
16831694
for (Tablet originTablet : originIndex.getTablets()) {
1684-
TabletMeta shadowTabletMeta = new TabletMeta(dbId, tableId, partitionId, shadowIndexId,
1685-
newSchemaHash, medium);
16861695
long originTabletId = originTablet.getId();
16871696
long shadowTabletId = idGeneratorBuffer.getNextId();
16881697

16891698
Tablet shadowTablet = EnvFactory.getInstance().createTablet(shadowTabletId);
1690-
shadowIndex.addTablet(shadowTablet, shadowTabletMeta);
1699+
invertedIndex.addTablet(shadowTabletId, shadowTabletMeta);
1700+
shadowTabletsForPartition.add(shadowTablet);
16911701
addedTablets.add(shadowTablet);
16921702

16931703
schemaChangeJob.addTabletIdMap(partitionId, shadowIndexId, shadowTabletId, originTabletId);
@@ -1745,6 +1755,9 @@ private void createJob(String rawSql, long dbId, OlapTable olapTable, Map<Long,
17451755
}
17461756
}
17471757

1758+
// Bulk-publish all shadow tablets for this partition in one copy-on-write.
1759+
shadowIndex.appendTablets(shadowTabletsForPartition);
1760+
17481761
schemaChangeJob.addPartitionShadowIndex(partitionId, shadowIndexId, shadowIndex);
17491762
} // end for partition
17501763
schemaChangeJob.addIndexSchema(shadowIndexId, originIndexId, newIndexName, newSchemaVersion, newSchemaHash,

‎fe/fe-core/src/main/java/org/apache/doris/backup/RestoreJob.java‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -107,6 +107,7 @@
107107

108108
import java.io.DataInput;
109109
import java.io.IOException;
110+
import java.util.ArrayList;
110111
import java.util.HashMap;
111112
import java.util.List;
112113
import java.util.Map;
@@ -1525,12 +1526,13 @@ protected Partition resetTabletForRestore(OlapTable localTbl, OlapTable remoteTb
15251526
int schemaHash = remoteTbl.getSchemaHashByIndexId(remoteIdx.getId());
15261527
int remotetabletSize = remoteIdx.getTablets().size();
15271528
remoteIdx.clearTabletsForRestore();
1529+
// Collect locally and bulk-publish to keep copy-on-write O(n) for the whole index.
1530+
List<Tablet> newTablets = new ArrayList<>(remotetabletSize);
15281531
for (int i = 0; i < remotetabletSize; i++) {
15291532
// generate new tablet id
15301533
long newTabletId = env.getNextId();
15311534
Tablet newTablet = EnvFactory.getInstance().createTablet(newTabletId);
1532-
// add tablet to index, but not add to TabletInvertedIndex
1533-
remoteIdx.addTablet(newTablet, null /* tablet meta */, true /* is restore */);
1535+
newTablets.add(newTablet);
15341536
// replicas
15351537
try {
15361538
Pair<Map<Tag, List<Long>>, TStorageMedium> beIdsAndMedium = Env.getCurrentSystemInfo()
@@ -1549,6 +1551,9 @@ protected Partition resetTabletForRestore(OlapTable localTbl, OlapTable remoteTb
15491551
return null;
15501552
}
15511553
}
1554+
// add tablets to index in one batch; TabletInvertedIndex registration
1555+
// is intentionally skipped on the restore path (rebuilt separately).
1556+
remoteIdx.appendTablets(newTablets);
15521557
}
15531558
return remotePart;
15541559
}

‎fe/fe-core/src/main/java/org/apache/doris/catalog/CloudTabletStatMgr.java‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -370,6 +370,7 @@ private void updateStatInfo(List<Long> dbIds) {
370370
long tabletIndexSize = 0L;
371371
long tabletSegmentSize = 0L;
372372

373+
// getReplicas() returns an immutable volatile snapshot; CME-safe under concurrent DDL.
373374
for (Replica replica : tablet.getReplicas()) {
374375
if (replica.getDataSize() > tabletDataSize) {
375376
tabletDataSize = replica.getDataSize();

‎fe/fe-core/src/main/java/org/apache/doris/catalog/LocalTablet.java‎

Lines changed: 57 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -31,16 +31,16 @@
3131
import org.apache.logging.log4j.Logger;
3232

3333
import java.util.ArrayList;
34+
import java.util.Collections;
3435
import java.util.Comparator;
35-
import java.util.Iterator;
3636
import java.util.List;
3737
import java.util.stream.LongStream;
3838

3939
public class LocalTablet extends Tablet {
4040
private static final Logger LOG = LogManager.getLogger(LocalTablet.class);
4141

4242
@SerializedName(value = "rs", alternate = {"replicas"})
43-
private List<Replica> replicas;
43+
private volatile List<Replica> replicas;
4444
@SerializedName(value = "lastCheckTime")
4545
private long lastCheckTime;
4646

@@ -109,13 +109,14 @@ public long getRemoteDataSize() {
109109
if (cooldownReplicaId <= 0) {
110110
return 0;
111111
}
112-
for (Replica r : replicas) {
112+
List<Replica> snapshot = replicas; // single volatile read; reuse below
113+
for (Replica r : snapshot) {
113114
if (r.getId() == cooldownReplicaId) {
114115
return r.getRemoteDataSize();
115116
}
116117
}
117118
// return replica with max remoteDataSize
118-
return replicas.stream().max(Comparator.comparing(Replica::getRemoteDataSize)).get().getRemoteDataSize();
119+
return snapshot.stream().max(Comparator.comparing(Replica::getRemoteDataSize)).get().getRemoteDataSize();
119120
}
120121

121122
@Override
@@ -223,29 +224,32 @@ public void setLastCheckTime(long lastCheckTime) {
223224
this.lastCheckTime = lastCheckTime;
224225
}
225226

226-
private boolean isLatestReplicaAndDeleteOld(Replica newReplica) {
227-
boolean delete = false;
227+
// Writers are synchronized on this tablet to prevent concurrent lost-update:
228+
// some callers (e.g. InternalCatalog.createTablets, RestoreJob) do NOT hold
229+
// the OlapTable write lock when modifying replicas.
230+
// Readers capture the volatile reference once and iterate freely — no lock needed.
231+
232+
@Override
233+
public synchronized void addReplica(Replica replica, boolean isRestore) {
234+
long version = replica.getVersion();
235+
long backendId = replica.getBackendIdWithoutException();
228236
boolean hasBackend = false;
229-
long version = newReplica.getVersion();
230-
Iterator<Replica> iterator = replicas.iterator();
231-
while (iterator.hasNext()) {
232-
Replica replica = iterator.next();
233-
if (replica.getBackendIdWithoutException() == newReplica.getBackendIdWithoutException()) {
237+
boolean deletedOld = false;
238+
List<Replica> current = replicas;
239+
List<Replica> next = new ArrayList<>(current.size() + 1);
240+
for (Replica r : current) {
241+
if (r.getBackendIdWithoutException() == backendId) {
234242
hasBackend = true;
235-
if (replica.getVersion() <= version) {
236-
iterator.remove();
237-
delete = true;
243+
if (r.getVersion() <= version) {
244+
deletedOld = true;
245+
continue; // drop stale replica
238246
}
239247
}
248+
next.add(r);
240249
}
241-
242-
return delete || !hasBackend;
243-
}
244-
245-
@Override
246-
public void addReplica(Replica replica, boolean isRestore) {
247-
if (isLatestReplicaAndDeleteOld(replica)) {
248-
replicas.add(replica);
250+
if (deletedOld || !hasBackend) {
251+
next.add(replica);
252+
replicas = next; // volatile write; readers see the new immutable snapshot
249253
if (!isRestore) {
250254
Env.getCurrentInvertedIndex().addReplica(id, replica);
251255
}
@@ -254,12 +258,13 @@ public void addReplica(Replica replica, boolean isRestore) {
254258

255259
@Override
256260
public List<Replica> getReplicas() {
257-
return this.replicas;
261+
// Volatile read: returns the current immutable snapshot; callers iterate without locking.
262+
return Collections.unmodifiableList(replicas);
258263
}
259264

260265
@Override
261266
public Replica getReplicaByBackendId(long backendId) {
262-
for (Replica replica : replicas) {
267+
for (Replica replica : replicas) { // single volatile read
263268
if (replica.getBackendIdWithoutException() == backendId) {
264269
return replica;
265270
}
@@ -268,26 +273,35 @@ public Replica getReplicaByBackendId(long backendId) {
268273
}
269274

270275
@Override
271-
public boolean deleteReplica(Replica replica) {
272-
if (replicas.contains(replica)) {
273-
replicas.remove(replica);
276+
public synchronized boolean deleteReplica(Replica replica) {
277+
List<Replica> current = replicas;
278+
if (current.contains(replica)) {
279+
List<Replica> next = new ArrayList<>(current);
280+
next.remove(replica);
281+
replicas = next; // volatile write
274282
Env.getCurrentInvertedIndex().deleteReplica(id, replica.getBackendIdWithoutException());
275283
return true;
276284
}
277285
return false;
278286
}
279287

280288
@Override
281-
public boolean deleteReplicaByBackendId(long backendId) {
282-
Iterator<Replica> iterator = replicas.iterator();
283-
while (iterator.hasNext()) {
284-
Replica replica = iterator.next();
289+
public synchronized boolean deleteReplicaByBackendId(long backendId) {
290+
List<Replica> current = replicas;
291+
List<Replica> next = new ArrayList<>(current.size());
292+
Replica found = null;
293+
for (Replica replica : current) {
285294
if (replica.getBackendIdWithoutException() == backendId) {
286-
iterator.remove();
287-
Env.getCurrentInvertedIndex().deleteReplica(id, backendId);
288-
return true;
295+
found = replica;
296+
} else {
297+
next.add(replica);
289298
}
290299
}
300+
if (found != null) {
301+
replicas = next; // volatile write
302+
Env.getCurrentInvertedIndex().deleteReplica(id, backendId);
303+
return true;
304+
}
291305
return false;
292306
}
293307

@@ -302,13 +316,17 @@ public boolean equals(Object obj) {
302316

303317
LocalTablet tablet = (LocalTablet) obj;
304318

305-
if (replicas != tablet.replicas) {
306-
if (replicas.size() != tablet.replicas.size()) {
319+
// Capture one snapshot per side so a concurrent writer cannot publish
320+
// a different list between size/contains/get calls below.
321+
List<Replica> thisReplicas = replicas;
322+
List<Replica> otherReplicas = tablet.replicas;
323+
if (thisReplicas != otherReplicas) {
324+
if (thisReplicas.size() != otherReplicas.size()) {
307325
return false;
308326
}
309-
int size = replicas.size();
327+
int size = thisReplicas.size();
310328
for (int i = 0; i < size; i++) {
311-
if (!tablet.replicas.contains(replicas.get(i))) {
329+
if (!otherReplicas.contains(thisReplicas.get(i))) {
312330
return false;
313331
}
314332
}
@@ -334,7 +352,7 @@ public boolean readyToBeRepaired(SystemInfoService infoService, TabletSchedCtx.P
334352
}
335353

336354
boolean allBeAliveOrDecommissioned = true;
337-
for (Replica replica : replicas) {
355+
for (Replica replica : replicas) { // single volatile read; iteration on the snapshot
338356
Backend backend = infoService.getBackend(replica.getBackendIdWithoutException());
339357
if (backend == null || (!backend.isAlive() && !backend.isDecommissioned())) {
340358
allBeAliveOrDecommissioned = false;

0 commit comments

Comments
 (0)