@alexey-milovidov Any comments for this PR? By the way, I don't think the failed cases relate to my change.

@alexey-milovidov The failing case looks caused by unstable test environment.

Probably off topic:
Have you thought about more complex way to deduplicate inserts rather then data only hash? We can use some client insertID to deduplicate if client is able to set it on insert. And there can be several strategies

• data_hash - your current way to deduplicate. Good for data with high entropy
• id - client defined unique id for insert for further deduplication
• both - we use both id and data to search for duplicated inserts.

I'm conserned about scenarios where several inserts can have similar data. Currently there is only one solution to add one more column with something like UUID if your data can have same rows in different inserts.

@qoega Here's a scenario where several inserts have same data: The JDBC client sends insert command, but gets broken connection exception. The client cannot tell if the exception happened before the insert, so it retry insert the same data. The server gets the same data twice, so later analysis queries may get incorrect result.

The idea is clear to me, but I think we have to move this logic into StorageMergeTree level because in the current implementation it will work both for replicated and non replicated engines. I'll try to make some improvements.

Just give more information about duplicated records in clickHouse as below:

• We're consuming data from Kafka then saving to clickHouse (using Distributed table in cluster and ReplicatedMergeTree in each shard)
• Some times Kafka has duplicated data ( in producer if a writing process is not received responses from Kafka cluster and it has to write for the second message)

I see MergeTree is not useful in valid data due to it run background, data goes from invalid to valid then versa. The main purpose is exact data and it's hard to find a good solution to avoid duplicated data in a large of data (big data) if using clickHouse. Please advice! I hope we can handle the same solution in Cassandra.

#22514