You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Performance] show proc statistic takes too long #6477
This query will calculate all replicas, which will takes very long on a large cluster.
Statistic don't need to calculate every time, one update in one minute is enough.
This could calculate in parallel.
which doris version do you use？ and how much replicas total in cluster？ how long dose it task for show proc one time？
Rebase on 0.13.11 and have the same impl at StatisticProcDir.java
We have many large scale clusters, many of which cost at least 5-10s querying /statistic. The biggest cluster has 45,000,000 replicas.
This proc is currently calculating twice on all the replicas per query because of the bad interface design. It has to calculate once when fetch the meta, and calculate again. Besides, this proc status doesn't need to refresh every time, maybe cache in a minute.
I think that it is needed to speed up caculating, but in most case, we need to know the real time cluster replica statistic, cache may be confused when we need to track some cluster problems
I think that it is needed to speed up caculating, but in most case, we need to know the real time cluster replica statistic, cache may be confused when we need to track some cluster problems
The cache is configurable, maybe set cache timeout 0 to disable. The cache is mostly to prevent high frequency query in a short time that may cause high calculation cost and heavy lock competitions, maybe 1s is close enough from real-time refreshing.
The cache is easy to implement, the state is restored in the previous version and may cause concurrent issue, so the cache is kind of fixing a concurrent bug rather than a new feature.
I think that it is needed to speed up caculating, but in most case, we need to know the real time cluster replica statistic, cache may be confused when we need to track some cluster problems
The cache is configurable, maybe set cache timeout 0 to disable. The cache is mostly to prevent high frequency query in a short time that may cause high calculation cost and heavy lock competitions, maybe 1s is close enough from real-time refreshing.
The cache is easy to implement, the state is restored in the previous version and may cause concurrent issue, so the cache is kind of fixing a concurrent bug rather than a new feature.
may we can provide two choice, sync or async, so that for high frequency query we can use async model, for other normal user, just speed up caculating and use sync model, what do you think about it?
may we can provide two choice, sync or async, so that for high frequency query we can use async model, for other normal user, just speed up caculating and use sync model, what do you think about it?
I'll disable the cache in default, or just remove it if you insist.
Activity
caiconghui commentedon Aug 20, 2021
which doris version do you use？ and how much replicas total in cluster？ how long dose it task for show proc one time？
ccoffline commentedon Aug 23, 2021
StatisticProcDir.java/statistic. The biggest cluster has 45,000,000 replicas.ccoffline commentedon Aug 23, 2021
I'll push a pr later on, this is just a notice.
caiconghui commentedon Aug 23, 2021
I think that it is needed to speed up caculating, but in most case, we need to know the real time cluster replica statistic, cache may be confused when we need to track some cluster problems
ccoffline commentedon Aug 24, 2021
The cache is configurable, maybe set cache timeout 0 to disable. The cache is mostly to prevent high frequency query in a short time that may cause high calculation cost and heavy lock competitions, maybe 1s is close enough from real-time refreshing.
The cache is easy to implement, the state is restored in the previous version and may cause concurrent issue, so the cache is kind of fixing a concurrent bug rather than a new feature.
caiconghui commentedon Aug 24, 2021
may we can provide two choice, sync or async, so that for high frequency query we can use async model, for other normal user, just speed up caculating and use sync model, what do you think about it?
ccoffline commentedon Aug 25, 2021
I'll disable the cache in default, or just remove it if you insist.
branch-selectdb-doris-3.1: [fix](temp-table) not clean temp table tem…