Skip to content

Delete parts by replacing them with empty parts#41145

Merged
CheSema merged 40 commits intoClickHouse:masterfrom
CheSema:lock-free-drop-partition
Nov 25, 2022
Merged

Delete parts by replacing them with empty parts#41145
CheSema merged 40 commits intoClickHouse:masterfrom
CheSema:lock-free-drop-partition

Conversation

@CheSema
Copy link
Copy Markdown
Member

@CheSema CheSema commented Sep 9, 2022

The issue:
#33457

Changelog category (leave one):

  • Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

This PR changes how followed queries delete parts: truncate table, alter table drop part, alter table drop partition. Now these queries make empty parts which cover old parts. This makes truncate query works without exclusive lock which means concurrent reads aren't locked. Also achieved durability in all those queries. If request is succeeded then no resurrected pars appear later. Note that atomicity is achieved only with transaction scope.

Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/

@robot-ch-test-poll robot-ch-test-poll added the pr-improvement Pull request with some product improvements label Sep 9, 2022
@den-crane
Copy link
Copy Markdown
Contributor

implements #33457
resolves #15742

@tavplubix
Copy link
Copy Markdown
Member

We can also create empty covering part on disk for ReplicatedMergeTree in order to fix #37664 (but we don't need to commit it to ZooKeeper and can even add it to the set of parts in Outdated state)

@CheSema CheSema force-pushed the lock-free-drop-partition branch 3 times, most recently from 779ccc3 to 875946b Compare September 14, 2022 21:41
@tavplubix
Copy link
Copy Markdown
Member

@CheSema CheSema force-pushed the lock-free-drop-partition branch from 875946b to c33c4ec Compare September 14, 2022 22:07
@CheSema
Copy link
Copy Markdown
Member Author

CheSema commented Sep 15, 2022

The test 01825_type_json_schema_race_long is failed.

I investigate it. At first look it seems like Object(‘json’) has to be wrapped in Tuple.

This PR seemed to be related #41290

UPD: test fixed with commit
208b65f5a067d14323c06f5c85aa8abfb09f2da1

UPD2:
Previous way to fix JSON is not right. It remembers all old types inside JSON object forever. But truncate table should forget the columns history in JSON object.
Here the right fix with test
593cf4a59848ebce434f874397b9bd787c03c3ec

@CheSema CheSema force-pushed the lock-free-drop-partition branch 3 times, most recently from 6ae9026 to 2e10652 Compare September 19, 2022 09:21
@CheSema CheSema marked this pull request as ready for review September 19, 2022 09:43
@CheSema CheSema force-pushed the lock-free-drop-partition branch 2 times, most recently from ec419d5 to 9a925e6 Compare September 22, 2022 23:25
@CheSema CheSema force-pushed the lock-free-drop-partition branch from f6d78d8 to 9f2c00d Compare November 23, 2022 15:19
@den-crane
Copy link
Copy Markdown
Contributor

Hooray. Dreams come true.

@CheSema
Copy link
Copy Markdown
Member Author

CheSema commented Dec 12, 2022

@CheSema, seems like it makes test_merge_tree_hdfs more flaky: https://play.clickhouse.com/play?

This should help with it.
#44154

@tavplubix
Copy link
Copy Markdown
Member

@CheSema, test_merge_tree_s3/test.py::test_attach_detach_partition is still flaky: link

jawm added a commit to jawm/ClickHouse that referenced this pull request Jun 6, 2023
I saw a server crash with error `Part 20230603_61687_61697_1 intersects
part 20230603_61690_61695_1 (state Active) It is a bug.
(LOGICAL_ERROR)`. This was using v22.12. I can see that since then the
error message was adjusted, but I don't believe the bug itself has been
resolved.

I believe the bug comes from this change (c976b28).
It's associated with this pull request: (ClickHouse#41145).
Specifically changes to the file `src/Storages/MergeTree/MergeTreeData.cpp`
seem maybe incorrect to me, since the logic used when a part is covered
by another part seems to have changed as described below.

The change modifies the function `renameTempPartAndReplaceImpl`, such
that a call to `getPartHierarchy` is used where previously there was a
call to `getActivePartsToReplace`. These two functions have a similar
implementation, but one notable difference -- in the latter, execution
breaks early if a covering part is found over the new part. In our outer
function, there was a condition looking for this covering part, which if
hit, would print the warning and return false.

Without the check, it instead continues, sees that there is an
intersecting part, and crashed with the exception. I believe that the
"intersecting" part would also be considered a "covering" part, as in
this case it covers the block range from before -> after the range of
the new part.

I believe a solution is to simply re-add the condition which checks for
a covering part, and return early in that case. However, my assessment
could be very wrong, I can't say I understand this code all that well. I
also don't really know how I could write a test for this either.

The other relevant detail; This occurred at the moment when I was
running a mutation on a *different* partition of the table. It happened
on two occasions, so I'm pretty sure connected to the mutation, although
I don't know why operations in different partitions would affect each
other in this way.
jawm added a commit to jawm/ClickHouse that referenced this pull request Jun 7, 2023
I saw a server crash with error `Part 20230603_61687_61697_1 intersects
part 20230603_61690_61695_1 (state Active) It is a bug.
(LOGICAL_ERROR)`. This was using v22.12. I can see that since then the
error message was adjusted, but I don't believe the bug itself has been
resolved.

I believe the bug comes from this change (c976b28).
It's associated with this pull request: (ClickHouse#41145).
Specifically changes to the file `src/Storages/MergeTree/MergeTreeData.cpp`
seem maybe incorrect to me, since the logic used when a part is covered
by another part seems to have changed as described below.

The change modifies the function `renameTempPartAndReplaceImpl`, such
that a call to `getPartHierarchy` is used where previously there was a
call to `getActivePartsToReplace`. These two functions have a similar
implementation, but one notable difference -- in the latter, execution
breaks early if a covering part is found over the new part. In our outer
function, there was a condition looking for this covering part, which if
hit, would print the warning and return false.

Without the check, it instead continues, sees that there is an
intersecting part, and crashed with the exception. I believe that the
"intersecting" part would also be considered a "covering" part, as in
this case it covers the block range from before -> after the range of
the new part.

I believe a solution is to simply re-add the condition which checks for
a covering part, and return early in that case. However, my assessment
could be very wrong, I can't say I understand this code all that well. I
also don't really know how I could write a test for this either.

The other relevant detail; This occurred at the moment when I was
running a mutation on a *different* partition of the table. It happened
on two occasions, so I'm pretty sure connected to the mutation, although
I don't know why operations in different partitions would affect each
other in this way.
@techkuz
Copy link
Copy Markdown
Contributor

techkuz commented Aug 15, 2023

does detach part also make an empty partition? (detach not listed in the affected methods)

@tavplubix
Copy link
Copy Markdown
Member

Yes, DETACH PART[ITION] works the same way as DROP PART[ITION]

@acmeguy
Copy link
Copy Markdown

acmeguy commented Jan 10, 2024

This seems to be an issue still when using a volume with a s3 disk and a cache.

Resurrected partitions are driving us crazy.

Any advice?

@filimonov
Copy link
Copy Markdown
Contributor

Resurrected partitions are driving us crazy.
Any advice?

Try to create a minimal repro and report the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-improvement Pull request with some product improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants