Skip to content

feat: add quantile DDSketch#55305

Closed
srikanthccv wants to merge 10000 commits into
ClickHouse:masterfrom
srikanthccv:quantile-sketch
Closed

feat: add quantile DDSketch#55305
srikanthccv wants to merge 10000 commits into
ClickHouse:masterfrom
srikanthccv:quantile-sketch

Conversation

@srikanthccv

@srikanthccv srikanthccv commented Oct 7, 2023

Copy link
Copy Markdown
Contributor

Fixes #6739

Changelog category (leave one):

  • New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Add quantileDDSketch aggregate function as well as the corresponding quantilesDDSketch and medianDDSketch. It is based on the DDSketch https://www.vldb.org/pvldb/vol12/p2195-masson.pdf.

Documentation entry for user-facing changes

  • Documentation is written (mandatory for new features)

  • Motivation: It is very important to calculate accurate quantiles fast. The OpenTelemetry project is the emerging standard in the observability ecosystem, and the preferred way to record the latencies is ExponentialHistogram. Having this feature in ClickHouse makes it more adaptable.

  • Parameters: relative accuracy (or gamma in the paper)

  • Example use:

CREATE TABLE default.comparision
(
    `id` UInt64,
    `sketch` AggregateFunction(quantilesDDSketch(0.0003, 0.9), UInt64),
    `td` AggregateFunction(quantilesTDigest(0.5, 0.9, 0.99), UInt64),
    `bf` AggregateFunction(quantilesBFloat16(0.5, 0.9, 0.99), UInt64)
)
ENGINE = SummingMergeTree
ORDER BY id
SETTINGS index_granularity = 8192


insert into comparision
select
    number % 10000 as id,
    quantilesDDSketchState(0.0003, 0.9)(number) as sketch,
    quantilesTDigestState(0.5, 0.9, 0.99)(number) as td,
    quantilesBFloat16State(0.5, 0.9, 0.99)(number) as bf  
from numbers(100000000)
group by id;


SELECT
    quantilesDDSketchMerge(0.0003, 0.9)(sketch) AS sketch
FROM comparision

SELECT
    quantileTDigestMerge(0.9)(td) AS td
FROM comparision

SELECT
    quantileBFloat16Merge(0.9)(bf) AS bf
FROM comparision

Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/

PS:

We at SigNoz love ClickHouse.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

can be tested Allows running workflows for external contributors pr-feature Pull request with new product feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DDSketch Quantiles