[fix](tabletScheduler) Fix addTablet dead lock in tabletScheduler by deardeng · Pull Request #45298 · apache/doris

deardeng · 2024-12-11T09:40:04Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

The conditions that need to be met to trigger the bug, with the second condition being somewhat difficult to trigger, are as follows:

The number of tablets that need to be fixed exceeds 2000 (in the pending queue);
The scheduling of the lowest priority in the pending queue has previously experienced a clone failure, with fewer than 3 failures, and has been put back into the pending queue. Additionally, a new scheduling request that happens to belong to the same table as the previous one has a higher priority than the previous scheduling.

The fix is to write the lock trylock in finalize TabletCtx. If the lock cannot be obtained, the current scheduling will fail and the next one will be rescheduled

Fix

"colocate group clone checker" #7557 daemon prio=5 os_prio=0 cpu=686.24ms elapsed=6719.45s tid=0x00007f3e6c039ab0 nid=0x17b08 waiting on condition  [0x00007f3ec77fe000]
(1 similar threads)
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
        - parking to wait for  <0x000010014d223908> (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync)
        at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:211)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:715)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:938)
        at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock([email protected]/ReentrantReadWriteLock.java:959)
        at org.apache.doris.common.lock.MonitoredReentrantReadWriteLock$WriteLock.lock(MonitoredReentrantReadWriteLock.java:98)
        at org.apache.doris.catalog.Table.writeLockIfExist(Table.java:211)
        at org.apache.doris.clone.TabletSchedCtx.releaseResource(TabletSchedCtx.java:940)
        at org.apache.doris.clone.TabletSchedCtx.releaseResource(TabletSchedCtx.java:898)
        at org.apache.doris.clone.TabletScheduler.releaseTabletCtx(TabletScheduler.java:1743)
        at org.apache.doris.clone.TabletScheduler.finalizeTabletCtx(TabletScheduler.java:1625)
        at org.apache.doris.clone.TabletScheduler.addTablet(TabletScheduler.java:287)
        - locked <0x0000100009429110> (a org.apache.doris.clone.TabletScheduler)
        at org.apache.doris.clone.ColocateTableCheckerAndBalancer.matchGroups(ColocateTableCheckerAndBalancer.java:563)
        at org.apache.doris.clone.ColocateTableCheckerAndBalancer.runAfterCatalogReady(ColocateTableCheckerAndBalancer.java:340)
        at org.apache.doris.common.util.MasterDaemon.runOneCycle(MasterDaemon.java:58)
        at org.apache.doris.common.util.Daemon.run(Daemon.java:119)

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

doris-robot · 2024-12-11T09:40:10Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

deardeng · 2024-12-11T09:40:14Z

run buildall

deardeng · 2024-12-11T10:37:32Z

run buildall

github-actions · 2024-12-11T10:41:43Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-12-11T11:07:41Z

TeamCity be ut coverage result:
Function Coverage: 38.75% (10102/26071)
Line Coverage: 29.67% (84706/285476)
Region Coverage: 28.74% (43486/151298)
Branch Coverage: 25.31% (22094/87306)
Coverage Report: http://coverage.selectdb-in.cc/coverage/627f681177ed6fb1882ef5bc62811a3a5e55b1d3_627f681177ed6fb1882ef5bc62811a3a5e55b1d3/report/index.html

yujun777

LGTM

github-actions · 2024-12-11T12:53:13Z

PR approved by anyone and no changes requested.

github-actions · 2024-12-12T02:35:30Z

PR approved by at least one committer and no changes requested.

deardeng · 2024-12-12T03:16:36Z

run buildall

doris-robot · 2024-12-12T03:54:26Z

TeamCity be ut coverage result:
Function Coverage: 38.75% (10102/26073)
Line Coverage: 29.68% (84750/285535)
Region Coverage: 28.75% (43501/151331)
Branch Coverage: 25.30% (22095/87332)
Coverage Report: http://coverage.selectdb-in.cc/coverage/627f681177ed6fb1882ef5bc62811a3a5e55b1d3_627f681177ed6fb1882ef5bc62811a3a5e55b1d3/report/index.html

deardeng · 2024-12-12T12:25:43Z

run buildall

github-actions · 2024-12-12T12:29:53Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-12-12T13:00:20Z

TeamCity be ut coverage result:
Function Coverage: 38.75% (10104/26074)
Line Coverage: 29.69% (84771/285557)
Region Coverage: 28.75% (43513/151351)
Branch Coverage: 25.31% (22105/87342)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ff8858499198f8129fe981738bd67fd04cfaddf6_ff8858499198f8129fe981738bd67fd04cfaddf6/report/index.html

deardeng · 2024-12-12T13:38:21Z

run buildall

github-actions · 2024-12-12T13:42:14Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-12-12T14:14:48Z

TeamCity be ut coverage result:
Function Coverage: 38.75% (10104/26074)
Line Coverage: 29.67% (84736/285557)
Region Coverage: 28.74% (43505/151351)
Branch Coverage: 25.31% (22102/87342)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bb891d5f23b8a45feed731919ad73cafebd8f77c_bb891d5f23b8a45feed731919ad73cafebd8f77c/report/index.html

yujun777 · 2024-12-13T04:00:11Z

LGTM

yujun777

LGTM

github-actions · 2024-12-13T04:07:55Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-12-13T04:31:49Z

TeamCity be ut coverage result:
Function Coverage: 38.76% (10109/26081)
Line Coverage: 29.71% (84857/285664)
Region Coverage: 28.77% (43554/151410)
Branch Coverage: 25.32% (22129/87388)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b792cb14e848f36519d599493cf8dbac169d9c0f_b792cb14e848f36519d599493cf8dbac169d9c0f/report/index.html

dataroaring

LGTM

github-actions · 2024-12-17T06:36:10Z

PR approved by at least one committer and no changes requested.

deardeng · 2024-12-17T13:11:07Z

run buildall

github-actions · 2024-12-17T13:15:34Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-12-17T13:51:28Z

TeamCity be ut coverage result:
Function Coverage: 38.85% (10126/26064)
Line Coverage: 29.77% (85137/285939)
Region Coverage: 28.81% (43652/151507)
Branch Coverage: 25.36% (22176/87462)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2c8b955cf996119d998054291dda72096be8b89b_2c8b955cf996119d998054291dda72096be8b89b/report/index.html

dataroaring

LGTM

github-actions · 2024-12-19T16:01:40Z

PR approved by at least one committer and no changes requested.

dataroaring

LGTM

…5298) The conditions that need to be met to trigger the bug, with the second condition being somewhat difficult to trigger, are as follows: 1. The number of tablets that need to be fixed exceeds 2000 (in the pending queue); 2. The scheduling of the lowest priority in the pending queue has previously experienced a clone failure, with fewer than 3 failures, and has been put back into the pending queue. Additionally, a new scheduling request that happens to belong to the same table as the previous one has a higher priority than the previous scheduling. The fix is to write the lock trylock in finalize TabletCtx. If the lock cannot be obtained, the current scheduling will fail and the next one will be rescheduled Fix ``` "colocate group clone checker" #7557 daemon prio=5 os_prio=0 cpu=686.24ms elapsed=6719.45s tid=0x00007f3e6c039ab0 nid=0x17b08 waiting on condition [0x00007f3ec77fe000] (1 similar threads) java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park([email protected]/Native Method) - parking to wait for <0x000010014d223908> (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync) at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:211) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:715) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:938) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock([email protected]/ReentrantReadWriteLock.java:959) at org.apache.doris.common.lock.MonitoredReentrantReadWriteLock$WriteLock.lock(MonitoredReentrantReadWriteLock.java:98) at org.apache.doris.catalog.Table.writeLockIfExist(Table.java:211) at org.apache.doris.clone.TabletSchedCtx.releaseResource(TabletSchedCtx.java:940) at org.apache.doris.clone.TabletSchedCtx.releaseResource(TabletSchedCtx.java:898) at org.apache.doris.clone.TabletScheduler.releaseTabletCtx(TabletScheduler.java:1743) at org.apache.doris.clone.TabletScheduler.finalizeTabletCtx(TabletScheduler.java:1625) at org.apache.doris.clone.TabletScheduler.addTablet(TabletScheduler.java:287) - locked <0x0000100009429110> (a org.apache.doris.clone.TabletScheduler) at org.apache.doris.clone.ColocateTableCheckerAndBalancer.matchGroups(ColocateTableCheckerAndBalancer.java:563) at org.apache.doris.clone.ColocateTableCheckerAndBalancer.runAfterCatalogReady(ColocateTableCheckerAndBalancer.java:340) at org.apache.doris.common.util.MasterDaemon.runOneCycle(MasterDaemon.java:58) at org.apache.doris.common.util.Daemon.run(Daemon.java:119) ```

…cheduler #45298 (#45769) Cherry-picked from #45298 Co-authored-by: deardeng <[email protected]>

…cheduler #45298 (#45768) Cherry-picked from #45298 Co-authored-by: deardeng <[email protected]>

…ache#45298) The conditions that need to be met to trigger the bug, with the second condition being somewhat difficult to trigger, are as follows: 1. The number of tablets that need to be fixed exceeds 2000 (in the pending queue); 2. The scheduling of the lowest priority in the pending queue has previously experienced a clone failure, with fewer than 3 failures, and has been put back into the pending queue. Additionally, a new scheduling request that happens to belong to the same table as the previous one has a higher priority than the previous scheduling. The fix is to write the lock trylock in finalize TabletCtx. If the lock cannot be obtained, the current scheduling will fail and the next one will be rescheduled Fix ``` "colocate group clone checker" apache#7557 daemon prio=5 os_prio=0 cpu=686.24ms elapsed=6719.45s tid=0x00007f3e6c039ab0 nid=0x17b08 waiting on condition [0x00007f3ec77fe000] (1 similar threads) java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park([email protected]/Native Method) - parking to wait for <0x000010014d223908> (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync) at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:211) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:715) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:938) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock([email protected]/ReentrantReadWriteLock.java:959) at org.apache.doris.common.lock.MonitoredReentrantReadWriteLock$WriteLock.lock(MonitoredReentrantReadWriteLock.java:98) at org.apache.doris.catalog.Table.writeLockIfExist(Table.java:211) at org.apache.doris.clone.TabletSchedCtx.releaseResource(TabletSchedCtx.java:940) at org.apache.doris.clone.TabletSchedCtx.releaseResource(TabletSchedCtx.java:898) at org.apache.doris.clone.TabletScheduler.releaseTabletCtx(TabletScheduler.java:1743) at org.apache.doris.clone.TabletScheduler.finalizeTabletCtx(TabletScheduler.java:1625) at org.apache.doris.clone.TabletScheduler.addTablet(TabletScheduler.java:287) - locked <0x0000100009429110> (a org.apache.doris.clone.TabletScheduler) at org.apache.doris.clone.ColocateTableCheckerAndBalancer.matchGroups(ColocateTableCheckerAndBalancer.java:563) at org.apache.doris.clone.ColocateTableCheckerAndBalancer.runAfterCatalogReady(ColocateTableCheckerAndBalancer.java:340) at org.apache.doris.common.util.MasterDaemon.runOneCycle(MasterDaemon.java:58) at org.apache.doris.common.util.Daemon.run(Daemon.java:119) ```

…cheduler apache#45298 (apache#45769) Cherry-picked from apache#45298 Co-authored-by: deardeng <[email protected]>

…ache#45298) The conditions that need to be met to trigger the bug, with the second condition being somewhat difficult to trigger, are as follows: 1. The number of tablets that need to be fixed exceeds 2000 (in the pending queue); 2. The scheduling of the lowest priority in the pending queue has previously experienced a clone failure, with fewer than 3 failures, and has been put back into the pending queue. Additionally, a new scheduling request that happens to belong to the same table as the previous one has a higher priority than the previous scheduling. The fix is to write the lock trylock in finalize TabletCtx. If the lock cannot be obtained, the current scheduling will fail and the next one will be rescheduled Fix ``` "colocate group clone checker" apache#7557 daemon prio=5 os_prio=0 cpu=686.24ms elapsed=6719.45s tid=0x00007f3e6c039ab0 nid=0x17b08 waiting on condition [0x00007f3ec77fe000] (1 similar threads) java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park([email protected]/Native Method) - parking to wait for <0x000010014d223908> (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync) at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:211) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:715) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:938) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock([email protected]/ReentrantReadWriteLock.java:959) at org.apache.doris.common.lock.MonitoredReentrantReadWriteLock$WriteLock.lock(MonitoredReentrantReadWriteLock.java:98) at org.apache.doris.catalog.Table.writeLockIfExist(Table.java:211) at org.apache.doris.clone.TabletSchedCtx.releaseResource(TabletSchedCtx.java:940) at org.apache.doris.clone.TabletSchedCtx.releaseResource(TabletSchedCtx.java:898) at org.apache.doris.clone.TabletScheduler.releaseTabletCtx(TabletScheduler.java:1743) at org.apache.doris.clone.TabletScheduler.finalizeTabletCtx(TabletScheduler.java:1625) at org.apache.doris.clone.TabletScheduler.addTablet(TabletScheduler.java:287) - locked <0x0000100009429110> (a org.apache.doris.clone.TabletScheduler) at org.apache.doris.clone.ColocateTableCheckerAndBalancer.matchGroups(ColocateTableCheckerAndBalancer.java:563) at org.apache.doris.clone.ColocateTableCheckerAndBalancer.runAfterCatalogReady(ColocateTableCheckerAndBalancer.java:340) at org.apache.doris.common.util.MasterDaemon.runOneCycle(MasterDaemon.java:58) at org.apache.doris.common.util.Daemon.run(Daemon.java:119) ```

[fix](tabletScheduler) Fix addTablet dead lock in tabletScheduler

48a78dd

add dp

627f681

yujun777 approved these changes Dec 11, 2024

View reviewed changes

github-actions Bot added the reviewed label Dec 11, 2024

dataroaring added dev/2.1.x dev/3.0.x usercase Important user case type label labels Dec 11, 2024

gavinchou previously approved these changes Dec 12, 2024

View reviewed changes

github-actions Bot added the approved Indicates a PR has been approved by one committer. label Dec 12, 2024

add ut

ff88584

deardeng dismissed gavinchou’s stale review via ff88584 December 12, 2024 12:25

github-actions Bot removed the approved Indicates a PR has been approved by one committer. label Dec 12, 2024

fix

bb891d5

yujun777 reviewed Dec 13, 2024

View reviewed changes

Comment thread fe/fe-core/src/test/java/org/apache/doris/clone/TabletHealthTest.java Resolved

yujun777 approved these changes Dec 13, 2024

View reviewed changes

dataroaring previously approved these changes Dec 17, 2024

View reviewed changes

github-actions Bot added the approved Indicates a PR has been approved by one committer. label Dec 17, 2024

fix ut compile error

2c8b955

deardeng dismissed dataroaring’s stale review via 2c8b955 December 17, 2024 13:10

github-actions Bot removed the approved Indicates a PR has been approved by one committer. label Dec 17, 2024

dataroaring approved these changes Dec 19, 2024

View reviewed changes

github-actions Bot added the approved Indicates a PR has been approved by one committer. label Dec 19, 2024

dataroaring approved these changes Dec 23, 2024

View reviewed changes

dataroaring merged commit 0527a2e into apache:master Dec 23, 2024

github-actions Bot mentioned this pull request Dec 23, 2024

branch-3.0: [fix](tabletScheduler) Fix addTablet dead lock in tabletScheduler #45298 #45768

Merged

github-actions Bot mentioned this pull request Dec 23, 2024

branch-2.1: [fix](tabletScheduler) Fix addTablet dead lock in tabletScheduler #45298 #45769

Merged

yiguolei pushed a commit that referenced this pull request Dec 24, 2024

branch-2.1: [fix](tabletScheduler) Fix addTablet dead lock in tabletS…

6d6473e

…cheduler #45298 (#45769) Cherry-picked from #45298 Co-authored-by: deardeng <[email protected]>

yiguolei added dev/2.1.8-merged and removed dev/2.1.x labels Dec 24, 2024

dataroaring pushed a commit that referenced this pull request Dec 25, 2024

branch-3.0: [fix](tabletScheduler) Fix addTablet dead lock in tabletS…

7da76ae

…cheduler #45298 (#45768) Cherry-picked from #45298 Co-authored-by: deardeng <[email protected]>

dataroaring added dev/3.0.4-merged and removed dev/3.0.x labels Dec 25, 2024

Conversation

deardeng commented Dec 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

doris-robot commented Dec 11, 2024

Uh oh!

deardeng commented Dec 11, 2024

Uh oh!

deardeng commented Dec 11, 2024

Uh oh!

github-actions Bot commented Dec 11, 2024

Uh oh!

doris-robot commented Dec 11, 2024

Uh oh!

yujun777 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Dec 11, 2024

Uh oh!

github-actions Bot commented Dec 12, 2024

Uh oh!

deardeng commented Dec 12, 2024

Uh oh!

doris-robot commented Dec 12, 2024

Uh oh!

deardeng commented Dec 12, 2024

Uh oh!

github-actions Bot commented Dec 12, 2024

Uh oh!

doris-robot commented Dec 12, 2024

Uh oh!

deardeng commented Dec 12, 2024

Uh oh!

github-actions Bot commented Dec 12, 2024

Uh oh!

doris-robot commented Dec 12, 2024

Uh oh!

yujun777 commented Dec 13, 2024

Uh oh!

Uh oh!

yujun777 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Dec 13, 2024

Uh oh!

doris-robot commented Dec 13, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Dec 17, 2024

Uh oh!

deardeng commented Dec 17, 2024

Uh oh!

github-actions Bot commented Dec 17, 2024

Uh oh!

doris-robot commented Dec 17, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Dec 19, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

deardeng commented Dec 11, 2024 •

edited

Loading