-
Notifications
You must be signed in to change notification settings - Fork 8.3k
Seesawing in MemoryTracking metric causing >20.3 to fail some writes and reads, unable to identify cause. #12583
Description
Summary
After upgrading a host that is just a collection Distributed tables to 20.4.X and 20.5.2.7, we observe the new MemoryTracking setting increasing substantially during writes, leading to individual writes and reads yielding exceptions: Memory limit (total) exceeded: would use 28.28 GiB (attempt to allocate chunk of 4197840 bytes), maximum: 28.28
System metrics show that no actual memory is being allocated to the same size.

Please let me know if there is any additional context, diagrams, or statements I should provide. I'm interested in knowing how to correlate the total MemoryTracking metric to anything else since system memory does not reflect the metric at all.
Context
This host has four different Distributed tables that are written to via the HTTP interface with insert_distributed_sync=1 set.
The host itself has no local tables.
The hosts with the downstream local tables are all 20.5.2.7.
We have downgraded the Distributed host in question back to 20.3.9.70 in order to continue writing to the clusters.