Skip to content

Speed-up parts loading and removal#6438

Merged
alexey-milovidov merged 5 commits intomasterfrom
speed-up-parts-removal
Aug 11, 2019
Merged

Speed-up parts loading and removal#6438
alexey-milovidov merged 5 commits intomasterfrom
speed-up-parts-removal

Conversation

@alexey-milovidov
Copy link
Copy Markdown
Member

@alexey-milovidov alexey-milovidov commented Aug 11, 2019

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

For changelog. Remove if this is non-significant change.

Category (leave one):

  • Performance Improvement

Short description (up to few sentences):
Allow to use multiple threads during parts loading and removal. This fixes #6372 and #6074.

@alexey-milovidov alexey-milovidov added the pr-performance Pull request with some performance improvements label Aug 11, 2019
@alexey-milovidov alexey-milovidov changed the title Speed-up parts removal Speed-up parts loading and removal Aug 11, 2019
@alexey-milovidov alexey-milovidov merged commit 504e615 into master Aug 11, 2019
@alexey-milovidov alexey-milovidov added the pr-feature Pull request with new product feature label Aug 13, 2019
@amosbird
Copy link
Copy Markdown
Collaborator

We can also refine the parallel loading process to handle JBOD better, with threads loading separate disks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-feature Pull request with new product feature pr-performance Pull request with some performance improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Option to increase concurrency on MergeTreeDataPart removal during Cleanup.

2 participants