You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 962b88f
Browse filesBrowse the repository at this point in the historyBrowse files
authored
[fix](fe) avoid concurrent tablet stat iteration failures (#63298)
### What problem does this PR solve?
Issue Number: #59138
Problem Summary:
TabletStatMgr.runAfterCatalogReady() is a periodic master-FE daemon that
iterates every tablet/replica to pull statistics. When DDL runs
concurrently with this daemon, two races fire:
Iteration race (CME). MaterializedIndex.tablets and LocalTablet.replicas
were plain ArrayLists whose getters returned the internal list. A
concurrent addTablet / addReplica / deleteReplica (clone, repair, schema
change, restore, report handler) during iteration triggered the
fail-fast iterator and threw ConcurrentModificationException.
TOCTOU race. In updateTabletStat, a getTabletMeta(id) != null check is
followed by getReplica(id, beId). If the tablet is removed in between,
getReplica hits Preconditions.checkState(...) and throws
IllegalStateException.
When the daemon throws, the current cycle leaves stale
tablet/partition/table sizes and skewed MetricRepo metrics until the
next cycle.
Solution:
Close the CME race for good with copy-on-write via a volatile snapshot.
A first attempt returned a defensive copy (Lists.newArrayList(...)), but
the copy itself iterates the source list and can still CME mid-copy —
the window shrank but did not close. This PR instead:
Makes LocalTablet.replicas and MaterializedIndex.tablets volatile.
Writers (addReplica / deleteReplica / deleteReplicaByBackendId /
addTablet / clearTabletsForRestore) are synchronized, build a new list,
and atomically swap the volatile reference — they never mutate a list in
place.
Readers (getReplicas() / getTablets()) do a single volatile read and
return an immutable snapshot (Collections.unmodifiableList). Iteration
is lock-free and can never CME, and the hot read path no longer copies
elements.
synchronized on writers is required (not just volatile) because some
write paths do not hold the OlapTable write lock — verified by tracing
call sites: InternalCatalog.createPartitionWithIndices and
RestoreJob.resetPartitionForRestore call addReplica/addTablet without
the table write lock, so concurrent writers are real and a plain
volatile field would allow lost updates. Writers are infrequent (DDL /
repair / restore), so the lock cost is negligible; reads stay lock-free.
TOCTOU race is handled by catching IllegalStateException around
getReplica (kept from the original fix) and counting the skip via a new
TabletStatMgr.staleTabletStatSkipped counter, which makes the race
observable (>0 proves the window was actually hit) instead of relying
solely on log scraping.
Cloud path: CloudTabletStatMgr.updateStatInfo iterates
tablet.getReplicas() and is covered by the same snapshot fix; its
updateTabletStat uses getReplicasByTabletId (locked, returns empty list,
no checkState) and is already safe.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
---------
Co-authored-by: morningman <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
0 commit comments