Skip to content

Seesawing in MemoryTracking metric causing >20.3 to fail some writes and reads, unable to identify cause. #12583

@JTCunning

Description

@JTCunning

Summary
After upgrading a host that is just a collection Distributed tables to 20.4.X and 20.5.2.7, we observe the new MemoryTracking setting increasing substantially during writes, leading to individual writes and reads yielding exceptions: Memory limit (total) exceeded: would use 28.28 GiB (attempt to allocate chunk of 4197840 bytes), maximum: 28.28

System metrics show that no actual memory is being allocated to the same size.
Screen Shot 2020-07-19 at 12 23 59 PM

Please let me know if there is any additional context, diagrams, or statements I should provide. I'm interested in knowing how to correlate the total MemoryTracking metric to anything else since system memory does not reflect the metric at all.

Context
This host has four different Distributed tables that are written to via the HTTP interface with insert_distributed_sync=1 set.
The host itself has no local tables.

The hosts with the downstream local tables are all 20.5.2.7.

We have downgraded the Distributed host in question back to 20.3.9.70 in order to continue writing to the clusters.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugConfirmed user-visible misbehaviour in official releasememoryWhen memory usage is higher than expected

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions