Add unique identifiers IMergeTreeDataPart structure#16033
Add unique identifiers IMergeTreeDataPart structure#16033alesapin merged 6 commits intoClickHouse:masterfrom nvartolomei:nv/parts-uuid
Conversation
robot-clickhouse
added
the
pr-not-for-changelog
alesapin
left a comment
alesapin
left a comment
There was a problem hiding this comment.
Maybe we can add UUID generation for parts in this PR?
|
A stress test failure, seems unrelated to this PR https://clickhouse-test-reports.s3.yandex.net/16033/7706b0562bcb470b1663f8a91f9b6fa0731bbb66/stress_test_(thread).html#fail1 |
alesapin
left a comment
alesapin
left a comment
There was a problem hiding this comment.
Ok, clear to me, need clarification for format_version.
There was a problem hiding this comment.
Other replicas will fetch part with UUID and get it from this file?
There was a problem hiding this comment.
The UUID now is exchanged via part exchange protocol. For InMemory parts it is read from the wire directly. For Disk parts: uuid.txt file is fetched and then uuid is loaded together with other metadata in loadColumnsChecksumsIndexes.
Adding it to ZK would is redundant but might be viewed as consistent with other log entires. Though using it as source of truth wouldn't be very convenient (pass it as an argument to Fetcher::fetchPart?).
For now uuids are not generated at all, they are present only if the part is updated manually (as you can see in the integration test). The only place where they can be seen today by an end user is in `system.parts` table. I was looking for hiding this column behind an option but couldn't find an easy way to do that. Likely this is also required for WAL, but need to think how not to break compatibility. Relates to #13574, #13574 Next 1: In the upcoming PR the plan is to integrate de-duplication based on these fingerprints in the query pipeline. Next 2: We'll enable automatic generation of uuids and come up with a way for conditionally sending uuids when processing distributed queries only when part movement is in progress.
Avoid breaking backwards compatibility by default for now.
We are breaking backwards compatibility anyway (but agted by a setting)
Not related to changes seems like a flaky test. |
4 participants
Add part unique identifiers. Not generated unless
assign_part_uuidsis set to1in MergeTree settings (can be done only if all replicas are upgraded).I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en
Changelog category (leave one):