Skip to content

Commit 962b88f

Browse files
yx-keithmorningmanclaude
authored
[fix](fe) avoid concurrent tablet stat iteration failures (#63298)
### What problem does this PR solve? Issue Number: #59138 Problem Summary: TabletStatMgr.runAfterCatalogReady() is a periodic master-FE daemon that iterates every tablet/replica to pull statistics. When DDL runs concurrently with this daemon, two races fire: Iteration race (CME). MaterializedIndex.tablets and LocalTablet.replicas were plain ArrayLists whose getters returned the internal list. A concurrent addTablet / addReplica / deleteReplica (clone, repair, schema change, restore, report handler) during iteration triggered the fail-fast iterator and threw ConcurrentModificationException. TOCTOU race. In updateTabletStat, a getTabletMeta(id) != null check is followed by getReplica(id, beId). If the tablet is removed in between, getReplica hits Preconditions.checkState(...) and throws IllegalStateException. When the daemon throws, the current cycle leaves stale tablet/partition/table sizes and skewed MetricRepo metrics until the next cycle. Solution: Close the CME race for good with copy-on-write via a volatile snapshot. A first attempt returned a defensive copy (Lists.newArrayList(...)), but the copy itself iterates the source list and can still CME mid-copy — the window shrank but did not close. This PR instead: Makes LocalTablet.replicas and MaterializedIndex.tablets volatile. Writers (addReplica / deleteReplica / deleteReplicaByBackendId / addTablet / clearTabletsForRestore) are synchronized, build a new list, and atomically swap the volatile reference — they never mutate a list in place. Readers (getReplicas() / getTablets()) do a single volatile read and return an immutable snapshot (Collections.unmodifiableList). Iteration is lock-free and can never CME, and the hot read path no longer copies elements. synchronized on writers is required (not just volatile) because some write paths do not hold the OlapTable write lock — verified by tracing call sites: InternalCatalog.createPartitionWithIndices and RestoreJob.resetPartitionForRestore call addReplica/addTablet without the table write lock, so concurrent writers are real and a plain volatile field would allow lost updates. Writers are infrequent (DDL / repair / restore), so the lock cost is negligible; reads stay lock-free. TOCTOU race is handled by catching IllegalStateException around getReplica (kept from the original fix) and counting the skip via a new TabletStatMgr.staleTabletStatSkipped counter, which makes the race observable (>0 proves the window was actually hit) instead of relying solely on log scraping. Cloud path: CloudTabletStatMgr.updateStatInfo iterates tablet.getReplicas() and is covered by the same snapshot fix; its updateTabletStat uses getReplicasByTabletId (locked, returns empty list, no checkState) and is already safe. ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> --------- Co-authored-by: morningman <[email protected]> Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
1 parent 28b3da8 commit 962b88f

13 files changed

Lines changed: 360 additions & 74 deletions

File tree

‎fe/fe-core/src/main/java/org/apache/doris/alter/MaterializedViewHandler.java‎

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -443,14 +443,22 @@ private RollupJobV2 createMaterializedViewJob(String rawSql, String mvName, Stri
443443
MaterializedIndex mvIndex = new MaterializedIndex(mvIndexId, IndexState.SHADOW);
444444
MaterializedIndex baseIndex = partition.getIndex(baseIndexId);
445445
short replicationNum = olapTable.getPartitionInfo().getReplicaAllocation(partitionId).getTotalReplicaNum();
446+
// All MV tablets of the same (partition, mv index) share the same TabletMeta;
447+
// build it once and bulk-publish to MaterializedIndex.tablets after the per-tablet
448+
// loop to keep copy-on-write O(n). TabletInvertedIndex registration stays
449+
// per-iteration because Tablet.addReplica(...) below needs the tablet present
450+
// in the inverted index.
451+
TabletMeta mvTabletMeta = new TabletMeta(
452+
dbId, tableId, partitionId, mvIndexId, mvSchemaHash, medium);
453+
List<Tablet> mvTabletsForPartition = Lists.newArrayListWithCapacity(baseIndex.getTablets().size());
454+
TabletInvertedIndex invertedIndex = Env.getCurrentInvertedIndex();
446455
for (Tablet baseTablet : baseIndex.getTablets()) {
447-
TabletMeta mvTabletMeta = new TabletMeta(
448-
dbId, tableId, partitionId, mvIndexId, mvSchemaHash, medium);
449456
long baseTabletId = baseTablet.getId();
450457
long mvTabletId = idGeneratorBuffer.getNextId();
451458

452459
Tablet newTablet = EnvFactory.getInstance().createTablet(mvTabletId);
453-
mvIndex.addTablet(newTablet, mvTabletMeta);
460+
invertedIndex.addTablet(mvTabletId, mvTabletMeta);
461+
mvTabletsForPartition.add(newTablet);
454462
addedTablets.add(newTablet);
455463

456464
mvJob.addTabletIdMap(partitionId, mvTabletId, baseTabletId);
@@ -505,6 +513,9 @@ private RollupJobV2 createMaterializedViewJob(String rawSql, String mvName, Stri
505513
}
506514
} // end for baseTablets
507515

516+
// Bulk-publish all MV tablets for this partition in one copy-on-write.
517+
mvIndex.appendTablets(mvTabletsForPartition);
518+
508519
mvJob.addMVIndex(partitionId, mvIndex);
509520

510521
if (LOG.isDebugEnabled()) {

‎fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java‎

Lines changed: 16 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,7 @@
5454
import org.apache.doris.catalog.Table;
5555
import org.apache.doris.catalog.TableIf.TableType;
5656
import org.apache.doris.catalog.Tablet;
57+
import org.apache.doris.catalog.TabletInvertedIndex;
5758
import org.apache.doris.catalog.TabletMeta;
5859
import org.apache.doris.catalog.info.ColumnPosition;
5960
import org.apache.doris.catalog.info.IndexType;
@@ -2050,14 +2051,23 @@ private void createJob(String rawSql, long dbId, OlapTable olapTable, Map<Long,
20502051
MaterializedIndex originIndex = partition.getIndex(originIndexId);
20512052
ReplicaAllocation replicaAlloc = olapTable.getPartitionInfo().getReplicaAllocation(partitionId);
20522053
Short totalReplicaNum = replicaAlloc.getTotalReplicaNum();
2054+
// All shadow tablets of the same (partition, shadow index) share the same TabletMeta;
2055+
// build it once and bulk-publish to MaterializedIndex.tablets after the per-tablet
2056+
// loop to keep copy-on-write O(n). TabletInvertedIndex registration stays
2057+
// per-iteration because Tablet.addReplica(...) below needs the tablet present
2058+
// in the inverted index.
2059+
TabletMeta shadowTabletMeta = new TabletMeta(dbId, tableId, partitionId, shadowIndexId,
2060+
newSchemaHash, medium);
2061+
List<Tablet> shadowTabletsForPartition = Lists.newArrayListWithCapacity(
2062+
originIndex.getTablets().size());
2063+
TabletInvertedIndex invertedIndex = Env.getCurrentInvertedIndex();
20532064
for (Tablet originTablet : originIndex.getTablets()) {
2054-
TabletMeta shadowTabletMeta = new TabletMeta(dbId, tableId, partitionId, shadowIndexId,
2055-
newSchemaHash, medium);
20562065
long originTabletId = originTablet.getId();
20572066
long shadowTabletId = idGeneratorBuffer.getNextId();
20582067

20592068
Tablet shadowTablet = EnvFactory.getInstance().createTablet(shadowTabletId);
2060-
shadowIndex.addTablet(shadowTablet, shadowTabletMeta);
2069+
invertedIndex.addTablet(shadowTabletId, shadowTabletMeta);
2070+
shadowTabletsForPartition.add(shadowTablet);
20612071
addedTablets.add(shadowTablet);
20622072

20632073
schemaChangeJob.addTabletIdMap(partitionId, shadowIndexId, shadowTabletId, originTabletId);
@@ -2115,6 +2125,9 @@ private void createJob(String rawSql, long dbId, OlapTable olapTable, Map<Long,
21152125
}
21162126
}
21172127

2128+
// Bulk-publish all shadow tablets for this partition in one copy-on-write.
2129+
shadowIndex.appendTablets(shadowTabletsForPartition);
2130+
21182131
schemaChangeJob.addPartitionShadowIndex(partitionId, shadowIndexId, shadowIndex);
21192132
} // end for partition
21202133
schemaChangeJob.addIndexSchema(shadowIndexId, originIndexId, newIndexName, newSchemaVersion, newSchemaHash,

‎fe/fe-core/src/main/java/org/apache/doris/backup/RestoreJob.java‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -109,6 +109,7 @@
109109

110110
import java.io.DataInput;
111111
import java.io.IOException;
112+
import java.util.ArrayList;
112113
import java.util.HashMap;
113114
import java.util.List;
114115
import java.util.Map;
@@ -1553,12 +1554,13 @@ protected Partition resetTabletForRestore(OlapTable localTbl, OlapTable remoteTb
15531554
int schemaHash = remoteTbl.getSchemaHashByIndexId(remoteIdx.getId());
15541555
int remotetabletSize = remoteIdx.getTablets().size();
15551556
remoteIdx.clearTabletsForRestore();
1557+
// Collect locally and bulk-publish to keep copy-on-write O(n) for the whole index.
1558+
List<Tablet> newTablets = new ArrayList<>(remotetabletSize);
15561559
for (int i = 0; i < remotetabletSize; i++) {
15571560
// generate new tablet id
15581561
long newTabletId = env.getNextId();
15591562
Tablet newTablet = EnvFactory.getInstance().createTablet(newTabletId);
1560-
// add tablet to index, but not add to TabletInvertedIndex
1561-
remoteIdx.addTablet(newTablet, null /* tablet meta */, true /* is restore */);
1563+
newTablets.add(newTablet);
15621564
// replicas
15631565
try {
15641566
Pair<Map<Tag, List<Long>>, TStorageMedium> beIdsAndMedium = Env.getCurrentSystemInfo()
@@ -1577,6 +1579,9 @@ protected Partition resetTabletForRestore(OlapTable localTbl, OlapTable remoteTb
15771579
return null;
15781580
}
15791581
}
1582+
// add tablets to index in one batch; TabletInvertedIndex registration
1583+
// is intentionally skipped on the restore path (rebuilt separately).
1584+
remoteIdx.appendTablets(newTablets);
15801585
}
15811586
return remotePart;
15821587
}

‎fe/fe-core/src/main/java/org/apache/doris/catalog/CloudTabletStatMgr.java‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -366,6 +366,7 @@ private void updateStatInfo(List<Long> dbIds) {
366366
long tabletIndexSize = 0L;
367367
long tabletSegmentSize = 0L;
368368

369+
// getReplicas() returns an immutable volatile snapshot; CME-safe under concurrent DDL.
369370
for (Replica replica : tablet.getReplicas()) {
370371
if (replica.getDataSize() > tabletDataSize) {
371372
tabletDataSize = replica.getDataSize();

‎fe/fe-core/src/main/java/org/apache/doris/catalog/LocalTablet.java‎

Lines changed: 57 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -31,16 +31,16 @@
3131
import org.apache.logging.log4j.Logger;
3232

3333
import java.util.ArrayList;
34+
import java.util.Collections;
3435
import java.util.Comparator;
35-
import java.util.Iterator;
3636
import java.util.List;
3737
import java.util.stream.LongStream;
3838

3939
public class LocalTablet extends Tablet {
4040
private static final Logger LOG = LogManager.getLogger(LocalTablet.class);
4141

4242
@SerializedName(value = "rs", alternate = {"replicas"})
43-
private List<Replica> replicas;
43+
private volatile List<Replica> replicas;
4444
@SerializedName(value = "lastCheckTime")
4545
private long lastCheckTime;
4646

@@ -109,13 +109,14 @@ public long getRemoteDataSize() {
109109
if (cooldownReplicaId <= 0) {
110110
return 0;
111111
}
112-
for (Replica r : replicas) {
112+
List<Replica> snapshot = replicas; // single volatile read; reuse below
113+
for (Replica r : snapshot) {
113114
if (r.getId() == cooldownReplicaId) {
114115
return r.getRemoteDataSize();
115116
}
116117
}
117118
// return replica with max remoteDataSize
118-
return replicas.stream().max(Comparator.comparing(Replica::getRemoteDataSize)).get().getRemoteDataSize();
119+
return snapshot.stream().max(Comparator.comparing(Replica::getRemoteDataSize)).get().getRemoteDataSize();
119120
}
120121

121122
@Override
@@ -223,29 +224,32 @@ public void setLastCheckTime(long lastCheckTime) {
223224
this.lastCheckTime = lastCheckTime;
224225
}
225226

226-
private boolean isLatestReplicaAndDeleteOld(Replica newReplica) {
227-
boolean delete = false;
227+
// Writers are synchronized on this tablet to prevent concurrent lost-update:
228+
// some callers (e.g. InternalCatalog.createTablets, RestoreJob) do NOT hold
229+
// the OlapTable write lock when modifying replicas.
230+
// Readers capture the volatile reference once and iterate freely — no lock needed.
231+
232+
@Override
233+
public synchronized void addReplica(Replica replica, boolean isRestore) {
234+
long version = replica.getVersion();
235+
long backendId = replica.getBackendIdWithoutException();
228236
boolean hasBackend = false;
229-
long version = newReplica.getVersion();
230-
Iterator<Replica> iterator = replicas.iterator();
231-
while (iterator.hasNext()) {
232-
Replica replica = iterator.next();
233-
if (replica.getBackendIdWithoutException() == newReplica.getBackendIdWithoutException()) {
237+
boolean deletedOld = false;
238+
List<Replica> current = replicas;
239+
List<Replica> next = new ArrayList<>(current.size() + 1);
240+
for (Replica r : current) {
241+
if (r.getBackendIdWithoutException() == backendId) {
234242
hasBackend = true;
235-
if (replica.getVersion() <= version) {
236-
iterator.remove();
237-
delete = true;
243+
if (r.getVersion() <= version) {
244+
deletedOld = true;
245+
continue; // drop stale replica
238246
}
239247
}
248+
next.add(r);
240249
}
241-
242-
return delete || !hasBackend;
243-
}
244-
245-
@Override
246-
public void addReplica(Replica replica, boolean isRestore) {
247-
if (isLatestReplicaAndDeleteOld(replica)) {
248-
replicas.add(replica);
250+
if (deletedOld || !hasBackend) {
251+
next.add(replica);
252+
replicas = next; // volatile write; readers see the new immutable snapshot
249253
if (!isRestore) {
250254
Env.getCurrentInvertedIndex().addReplica(id, replica);
251255
}
@@ -254,12 +258,13 @@ public void addReplica(Replica replica, boolean isRestore) {
254258

255259
@Override
256260
public List<Replica> getReplicas() {
257-
return this.replicas;
261+
// Volatile read: returns the current immutable snapshot; callers iterate without locking.
262+
return Collections.unmodifiableList(replicas);
258263
}
259264

260265
@Override
261266
public Replica getReplicaByBackendId(long backendId) {
262-
for (Replica replica : replicas) {
267+
for (Replica replica : replicas) { // single volatile read
263268
if (replica.getBackendIdWithoutException() == backendId) {
264269
return replica;
265270
}
@@ -268,26 +273,35 @@ public Replica getReplicaByBackendId(long backendId) {
268273
}
269274

270275
@Override
271-
public boolean deleteReplica(Replica replica) {
272-
if (replicas.contains(replica)) {
273-
replicas.remove(replica);
276+
public synchronized boolean deleteReplica(Replica replica) {
277+
List<Replica> current = replicas;
278+
if (current.contains(replica)) {
279+
List<Replica> next = new ArrayList<>(current);
280+
next.remove(replica);
281+
replicas = next; // volatile write
274282
Env.getCurrentInvertedIndex().deleteReplica(id, replica.getBackendIdWithoutException());
275283
return true;
276284
}
277285
return false;
278286
}
279287

280288
@Override
281-
public boolean deleteReplicaByBackendId(long backendId) {
282-
Iterator<Replica> iterator = replicas.iterator();
283-
while (iterator.hasNext()) {
284-
Replica replica = iterator.next();
289+
public synchronized boolean deleteReplicaByBackendId(long backendId) {
290+
List<Replica> current = replicas;
291+
List<Replica> next = new ArrayList<>(current.size());
292+
Replica found = null;
293+
for (Replica replica : current) {
285294
if (replica.getBackendIdWithoutException() == backendId) {
286-
iterator.remove();
287-
Env.getCurrentInvertedIndex().deleteReplica(id, backendId);
288-
return true;
295+
found = replica;
296+
} else {
297+
next.add(replica);
289298
}
290299
}
300+
if (found != null) {
301+
replicas = next; // volatile write
302+
Env.getCurrentInvertedIndex().deleteReplica(id, backendId);
303+
return true;
304+
}
291305
return false;
292306
}
293307

@@ -302,13 +316,17 @@ public boolean equals(Object obj) {
302316

303317
LocalTablet tablet = (LocalTablet) obj;
304318

305-
if (replicas != tablet.replicas) {
306-
if (replicas.size() != tablet.replicas.size()) {
319+
// Capture one snapshot per side so a concurrent writer cannot publish
320+
// a different list between size/contains/get calls below.
321+
List<Replica> thisReplicas = replicas;
322+
List<Replica> otherReplicas = tablet.replicas;
323+
if (thisReplicas != otherReplicas) {
324+
if (thisReplicas.size() != otherReplicas.size()) {
307325
return false;
308326
}
309-
int size = replicas.size();
327+
int size = thisReplicas.size();
310328
for (int i = 0; i < size; i++) {
311-
if (!tablet.replicas.contains(replicas.get(i))) {
329+
if (!otherReplicas.contains(thisReplicas.get(i))) {
312330
return false;
313331
}
314332
}
@@ -334,7 +352,7 @@ public boolean readyToBeRepaired(SystemInfoService infoService, TabletSchedCtx.P
334352
}
335353

336354
boolean allBeAliveOrDecommissioned = true;
337-
for (Replica replica : replicas) {
355+
for (Replica replica : replicas) { // single volatile read; iteration on the snapshot
338356
Backend backend = infoService.getBackend(replica.getBackendIdWithoutException());
339357
if (backend == null || (!backend.isAlive() && !backend.isDecommissioned())) {
340358
allBeAliveOrDecommissioned = false;

0 commit comments

Comments
 (0)