Skip mutations of unrelated partitions in `StorageMergeTree` by excitoon · Pull Request #21326 · ClickHouse/ClickHouse

excitoon · 2021-03-01T05:06:24Z

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Changelog category (leave one):

Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Skipping mutations of different partitions in StorageMergeTree.

azat · 2021-04-03T16:05:13Z

src/Storages/StorageMergeTree.h

Worth a comment

src/Storages/StorageMergeTree.h

azat · 2021-04-05T07:29:16Z

tests/integration/test_mutations_with_merge_tree/test.py

Is this a problem? I mean why it tagged with FIXME?

This is no longer a problem.

azat · 2021-04-05T07:34:56Z

tests/integration/test_mutations_with_merge_tree/test.py

Worth checking system.parts to verify that parts wasn't cloned?
And also block_numbers.number/parts_to_do_names from the system.mutations?

azat · 2021-04-05T07:37:30Z

src/Storages/StorageMergeTree.cpp

Why this is required? Ok, I see, but worth a comment (and also describe why it should not be persistent on disk)

azat · 2021-04-05T20:52:05Z

tests/integration/test_mutations_with_merge_tree/test.py

BTW worth adding a test for ReplicatedMergeTree?

azat · 2021-04-06T06:39:30Z

Also with this patch the mutations will hang for non existing parts (since it will return nothing from selectPartsToMutate() and so nothing will be notified), i.e.:

CREATE TABLE test_mutations_in_partitions
(
    `date` Date,
    `a` UInt64,
    `b` String
)
ENGINE = MergeTree
PARTITION BY a
ORDER BY tuple();

INSERT INTO test_mutations_in_partitions SELECT
    '2019-07-29' AS date,
    number,
    toString(number)
FROM numbers(100);

ALTER TABLE test_mutations_in_partitions
    DELETE IN PARTITION 1000 WHERE 1
SETTINGS mutations_sync = 1;

Query id: cc447d87-d3df-4c9d-a53a-c3f58ea69a49

Timeout exceeded while receiving data from server. Waited for 300 seconds, timeout is 300 seconds.
Cancelling query.

azat · 2021-04-06T06:40:49Z

tests/integration/test_mutations_with_merge_tree/test.py

BTW how about adding simple stateless test instead of integration test?
For mutations_sync it can be even .sql file, while for non-sync it can be .sh

azat · 2021-04-06T09:44:11Z

src/Storages/StorageMergeTree.cpp

Also this place looks a little bit unsafe, maybe it worth moving commands into the for loop scope?

azat · 2021-06-17T08:45:49Z

src/Storages/StorageMergeTree.cpp

Maybe this triggering can this be done unconditionally?
Since if some parts was affected then it will not hit this section anyway, right?

I intentionally made that in order to not degrade performance on mutex each merge attempt for those who don't use this feature (who mutate always all data). Do you think it worth making unconditionally?

excitoon · 2021-07-26T02:57:53Z

Why did you send it to draft?

excitoon · 2021-07-27T12:33:27Z

Hi @azat. Can you approve a PR?

azat · 2021-07-27T19:35:28Z

@Mergifyio update

mergify · 2021-07-27T19:36:02Z

@azat is not allowed to run commands

azat · 2021-07-27T19:37:51Z

@excitoon I will take a look during this weak, in the mean time can you rebase the PR against master? (There are lots of tests failures, yes they seems unrelated, since server got KILLed, but checks should be passed before someone will merge it I guess)

P.S. Also I can't download binary since none of builds had been finished (even fasttest).

alesapin · 2021-10-26T11:07:53Z

@Mergifyio update

mergify · 2021-10-26T11:08:19Z

update

❌ Pull request can't be updated with latest base branch changes

Details

Mergify needs the author permission to update the base branch of the pull request.
excitoon-favorites needs to authorize modification on its head branch.
err-code: 9A53F

excitoon · 2021-10-30T15:58:51Z

@alesapin Can you please look at it?

alesapin · 2021-11-30T09:15:25Z

Ok, let's merge.

robot-clickhouse added the pr-improvement Pull request with some product improvements label Mar 1, 2021

excitoon marked this pull request as ready for review March 22, 2021 07:38

excitoon changed the title ~~Attempt to skip mutations of different partitions in StorageMergeTree.~~ Attempt to skip mutations of different partitions in StorageMergeTree Mar 29, 2021

azat reviewed Apr 5, 2021

View reviewed changes

azat reviewed Apr 6, 2021

View reviewed changes

filimonov added the altinity label Apr 6, 2021

excitoon force-pushed the flexiblemutations2 branch from b2851fa to 0064b91 Compare April 22, 2021 07:59

azat mentioned this pull request Apr 28, 2021

Partition pruning for mutations #12188

Closed

excitoon force-pushed the flexiblemutations2 branch from d5a44c8 to dbbe1bc Compare May 17, 2021 07:36

excitoon force-pushed the flexiblemutations2 branch 2 times, most recently from df5e6ef to c7978d2 Compare June 10, 2021 07:14

excitoon changed the title ~~Attempt to skip mutations of different partitions in StorageMergeTree~~ Skip mutations of different partitions in StorageMergeTree Jun 10, 2021

excitoon changed the title ~~Skip mutations of different partitions in StorageMergeTree~~ Skip mutations of unrelated partitions in StorageMergeTree Jun 10, 2021

azat reviewed Jun 17, 2021

View reviewed changes

alex-zaitsev mentioned this pull request Jun 30, 2021

DELETE/UPDATE IN PARTITION not working #25882

Closed

excitoon force-pushed the flexiblemutations2 branch from c7978d2 to a65cb38 Compare July 8, 2021 11:33

alexey-milovidov added the st-waiting-for-fix label Jul 13, 2021

alexey-milovidov marked this pull request as draft July 13, 2021 02:34

excitoon force-pushed the flexiblemutations2 branch from a65cb38 to 121dada Compare July 19, 2021 08:10

excitoon force-pushed the flexiblemutations2 branch from 121dada to 8472eb5 Compare July 26, 2021 02:58

excitoon requested a review from azat July 26, 2021 02:58

excitoon marked this pull request as ready for review July 26, 2021 02:58

excitoon force-pushed the flexiblemutations2 branch from 8472eb5 to feafbd4 Compare July 27, 2021 12:33

excitoon force-pushed the flexiblemutations2 branch 3 times, most recently from 4128aad to 6b0b2b9 Compare October 29, 2021 08:42

excitoon force-pushed the flexiblemutations2 branch from 6350a9f to a893f05 Compare November 15, 2021 07:51

excitoon requested a review from alesapin November 18, 2021 07:56

excitoon added 16 commits November 25, 2021 17:10

Attempt to skip mutations of different partitions in StorageMergeTree.

6599124

Fixed bug with frozen mutations.

aa3eecb

Improved diagnostics in tests.

407c1aa

Fixed duplication commands bug.

30cd133

Better tests.

0da55ca

Rebase fix.

390416b

Test fixes.

863f356

Minor fix.

e0a453a

Additional comments.

84ffcb4

Deadlock fix.

12f3620

Fixed wrong lock order.

ec1f909

Better comment.

7bbaa14

Synced clearOldMutations() with rest of changes.

c6e73d7

Rebase fix.

e414fa0

Rebase fix.

bb2f21b

Rebase fix.

100e61a

excitoon force-pushed the flexiblemutations2 branch from 7cd6430 to 100e61a Compare November 29, 2021 07:15

alesapin merged commit 438fa60 into ClickHouse:master Nov 30, 2021

excitoon mentioned this pull request Dec 1, 2021

Minor fixes for StorageMergeTree #32037

Merged

tavplubix mentioned this pull request Aug 24, 2022

Remove wrong code for skipping mutations in MergeTree #40589

Merged

ilejn mentioned this pull request Apr 19, 2023

Simplified partition mutations #48941

Open

Conversation

excitoon commented Mar 1, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

filimonov Nov 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

azat Apr 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

azat Apr 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

azat Apr 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

azat commented Apr 6, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

excitoon commented Jul 26, 2021

Uh oh!

excitoon commented Jul 27, 2021

Uh oh!

azat commented Jul 27, 2021

Uh oh!

mergify bot commented Jul 27, 2021

Uh oh!

azat commented Jul 27, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alesapin commented Oct 26, 2021

Uh oh!

mergify bot commented Oct 26, 2021

❌ Pull request can't be updated with latest base branch changes

Uh oh!

excitoon commented Oct 30, 2021

Uh oh!

alesapin commented Nov 30, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

filimonov Nov 30, 2021 •

edited

Loading

azat Apr 5, 2021 •

edited

Loading

azat Apr 5, 2021 •

edited

Loading

azat Apr 5, 2021 •

edited

Loading

azat commented Jul 27, 2021 •

edited

Loading