Add MemoryTracker for the background tasks#46089
Conversation
|
Check also |
| if (!canEnqueueBackgroundTask()) | ||
| { | ||
| if (out_disable_reason) | ||
| *out_disable_reason = fmt::format("Current background tasks memory usage ({}) is more than the limit ({})", |
There was a problem hiding this comment.
I'm afraid that it will be too easy to get "too many parts" if the soft limit is not high enough. Imagine that some big merges are currently executing and consume a big amount of memory. It will be impossible to assign small merges. So it would be better to make the limit flexible and give priority to small merges as we do in getMaxSourcePartsSizeForMerge.
However, it's more complicated for memory usage because memory usage should be O(1) and should not depend on the source parts size. The concern here is that big marges may hold memory for too long.
But there's a chance that it will just work as is (it's hard to predict), so let's just keep it in mind.
|
sorry for any disturbution. there may be some misbehaviors for this feature that I observed and want to report to you.
SELECT
now(),
metric,
formatReadableSize(value),
description
FROM system.metrics
WHERE metric = 'MergesMutationsMemoryTracking'
Query id: 70f4c034-70b4-4a46-bc0e-6cdaf433c0cc
┌───────────────now()─┬─metric────────────────────────┬─formatReadableSize(value)─┬─description──────────────────────────────────────────────────────────────────────────┐
│ 2023-03-27 00:11:08 │ MergesMutationsMemoryTracking │ 134.15 GiB │ Total amount of memory (bytes) allocated by background tasks (merges and mutations). │
└─────────────────────┴───────────────────────────────┴───────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────┘
while total available amount of memory is about 32 GiB |
|
@cangyin thank you for the feedback. Indeed, I initially forgot to update |
|
@novikd, you forgot to check the failed tests before merging |
7 participants
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Add MemoryTracker for the background tasks (merges and mutation). Introduces
merges_mutations_memory_usage_soft_limitandmerges_mutations_memory_usage_to_ram_ratiosettings that represent the soft memory limit for merges and mutations. If this limit is reached ClickHouse won't schedule new merge or mutation tasks. AlsoMergesMutationsMemoryTrackingmetric is introduced to allow observing current memory usage of background tasks.Closes #45710.
Documentation entry for user-facing changes