Ok. The test: SELECT uniqCombined(number) FROM numbers_mt(10000000000)
will not be included in CI, because it's slow.

@azat what is the result of this test on your machine?

Ok. The test: SELECT uniqCombined(number) FROM numbers_mt(10000000000)
will not be included in CI, because it's slow.

Indeed, but if there were function like writeAggregation/readAggregation (the same as finalizeAggregation) this can be possible

what is the result of this test on your machine?

I tested using:

select uniq from (select uniqCombined(toString(number)) uniq from (select number from system.numbers limit 1e10))

• before: 1 427 486 203
• after for uint64 as a hash in HLL(String ch type): 10 017 420 795
• after for uint32 as hash in HLL (UInt64 ch type): ~5 545 308 725

P.S. I used toString here to force uint64 type (this looks odd to me that uint64 only for string but this is another story)

P.P.S. with numbers_mt it tooks 96.749 sec on my laptop.

this looks odd to me that uint64 only for string but this is another story

Actually looking into this more and more, and I cannot verify that this will change in-memory/on-disk representation, am I missing something?

I used toString here to force uint64 type (this looks odd to me that uint64 only for string but this is another story)

Can you point me to specific line in code?

Can you point me to specific line in code?

• all except string
  

  ClickHouse/dbms/src/AggregateFunctions/AggregateFunctionUniqCombined.h

  Line 102 in 15deedb

  struct AggregateFunctionUniqCombinedData : public AggregateFunctionUniqCombinedDataWithKey<UInt32, K>

• string:
  

  ClickHouse/dbms/src/AggregateFunctions/AggregateFunctionUniqCombined.h

  Line 108 in 15deedb

  struct AggregateFunctionUniqCombinedData<String, K> : public AggregateFunctionUniqCombinedDataWithKey<UInt64, K>

Actually looking into this more and more, and I cannot verify that this will change in-memory/on-disk representation, am I missing something?

Hm, changing for other types HashValueType to uint64 does not break reading HLL from the disk, will you mind if I will change this?

Tested with:

create table stats (items UInt64, uniq AggregateFunction(uniqCombined, String)) engine=TinyLog()
insert into stats select 1e10 as items, uniq from (select uniqCombinedState(toString(number)) uniq from (select number from system.numbers limit 1e10))
select items, uniqCombinedMerge(uniq) uniq from stats group by items
┌───────items─┬────────uniq─┬
│ 10000000000 │ 10017420795 │
└─────────────┴─────────────┴
# restart ch and change the typ
# same output

does not break reading HLL from the disk

But this will break uniqueness property of already added values. Maybe a separate function? Thoughts?

I've prepared another patch that will force uniqCombined to use 64-bit hash, once this will be resolved

uniqCombined has three states:

1. Small flat array.
2. Hash table.
3. HLL.

The sizes of these data structures are aligned to each other. Hash table (2) can grow only up to the size of HLL (3) and is converted to HLL when becomes too big.

Flat array and hash table stores hash values directly. But HLL does not store hash values - it does only store max number of leading/trailing zero bits across some subsets of hash values.

For first two states, using small hash function is perfectly fine and can give you better performance.
For HLL, it's better to use 64bit hash function. But as you need the value of hash function to convert from hash table to HLL, you cannot use different hashes.

The size of HLL is dependent on the size of hash function. For 32-bit hash function, HLL is using 5 bit cells; for 64-bit hash function, HLL is using 6-bit cells. (BTW, this is the origin of Log-Log name).

But you can use 32-bit hash function for HLL with 6-bit cells. In this case, one bit is always unused.

If different hash function was used for HLLs of the same size, you can deserialize this HLL but results would be incorrect. The same value will be counted twice in most cases. So, don't change the hash function without changing the function name (or some parameter).

If you use 32-bit hash function, you still can count more than 2^32 elements. You only have to apply correction for birthday paradox formula (If I remember correctly, we already have it in uniq and unqCombined). But it will work until about 25B values: #6078

You can radically change uniqCombined function to use 64-bit hash everywhere, but I suggest to make it under another name like uniqEnhanced.

Hash table.

Interesting, since there is 32-bit hash for non-string types, hash table (for medium cardinality) can be expanded by the power of 2 for non-string types, and this will make the value exact for up to 2^14 elements.

You can radically change uniqCombined function to use 64-bit hash everywhere

This will break compatibility, or am I missing something?

but I suggest to make it under another name like uniqEnhanced.

That patch introduces second parameter for it, but after thinking about this a little bit more, and uniqCombined64 came to my mind (I did not want to reserve uniqEnhanced to leave it for #2073)

PS.

:) SELECT uniqCombined(reinterpretAsString(number)) FROM numbers_mt(10000000000)

SELECT uniqCombined(reinterpretAsString(number))
FROM numbers_mt(10000000000)

┌─uniqCombined(reinterpretAsString(number))─┐
│                                9990159598 │                                                                                                                                                                                                                                                                                                                                                                                                               
└───────────────────────────────────────────┘

after this change.