Skip to content

Balance merges across volumes in multi-volume JBOD configuration #16300

@alex-zaitsev

Description

@alex-zaitsev

Use case

Consider a storage policy containing one volume and several equally sized disks in the volume. If new disk is added to the volume, it gets unbalanced, because inserts and merges are distributed randomly. Here is an example:

│ default │     475466088448 │    177023565824 │
│ disk2   │     475466088448 │    106664300544 │
│ disk3   │     475466088448 │    116218195968 │
│ disk4   │     475466088448 │    174707056640 │
│ disk5   │     475466088448 │    237826211840 │
└─────────┴──────────────────┴─────────────────┘

We need a setting/option that would allow ClickHouse to balance merges across disks.

Describe the solution you'd like

ClickHouse should select the disk with maximum free space for the merge destination. Ideally, it should also subtract size of parts that are merged when estimating disk size.

@excitoon ^

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions