added groupBitmapAnd, groupBitmapOr, groupBitmapXor by yuzhichang · Pull Request #7109 · ClickHouse/ClickHouse

yuzhichang · 2019-09-26T09:42:18Z

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

For changelog. Remove if this is non-significant change.

Category (leave one):

New Feature

Short description (up to few sentences):
added AND, OR, XOR aggregate function for bitmap column. This is the implementation of #6955.

groupBitmapAnd

Calculations the AND of a bitmap column, return cardinality of type UInt64, if add suffix -State, then return bitmap object.

groupBitmapAnd(expr)

Parameters

expr – An expression that results in AggregateFunction(groupBitmap, UInt*) type.

Return value

Value of the UInt64 type.

Example

DROP TABLE IF EXISTS bitmap_column_expr_test2;
CREATE TABLE bitmap_column_expr_test2
(
    tag_id String,
    z AggregateFunction(groupBitmap, UInt32)
)
ENGINE = MergeTree
ORDER BY tag_id;

INSERT INTO bitmap_column_expr_test2 VALUES ('tag1', bitmapBuild(cast([1,2,3,4,5,6,7,8,9,10] as Array(UInt32))));
INSERT INTO bitmap_column_expr_test2 VALUES ('tag2', bitmapBuild(cast([6,7,8,9,10,11,12,13,14,15] as Array(UInt32))));
INSERT INTO bitmap_column_expr_test2 VALUES ('tag3', bitmapBuild(cast([2,4,6,8,10,12] as Array(UInt32))));

SELECT groupBitmapAnd(z) FROM bitmap_column_expr_test2 WHERE like(tag_id, 'tag%');
┌─groupBitmapAnd(z)─┐
│               3   │
└───────────────────┘

SELECT groupBitmapAndMerge(z) FROM bitmap_column_expr_test2 WHERE like(tag_id, 'tag%');
┌─groupBitmapAnd(z)─┐
│               3   │
└───────────────────┘

SELECT arraySort(bitmapToArray(groupBitmapAndState(z))) FROM bitmap_column_expr_test2 WHERE like(tag_id, 'tag%');
┌─arraySort(bitmapToArray(groupBitmapAndState(z)))─┐
│ [6,8,10]                                         │
└──────────────────────────────────────────────────┘

groupBitmapOr

Calculations the OR of a bitmap column, return cardinality of type UInt64, if add suffix -State, then return bitmap object. This is equivalent to groupBitmapMerge.

groupBitmapOr(expr)

Parameters

expr – An expression that results in AggregateFunction(groupBitmap, UInt*) type.

Return value

Value of the UInt64 type.

Example

DROP TABLE IF EXISTS bitmap_column_expr_test2;
CREATE TABLE bitmap_column_expr_test2
(
    tag_id String,
    z AggregateFunction(groupBitmap, UInt32)
)
ENGINE = MergeTree
ORDER BY tag_id;

INSERT INTO bitmap_column_expr_test2 VALUES ('tag1', bitmapBuild(cast([1,2,3,4,5,6,7,8,9,10] as Array(UInt32))));
INSERT INTO bitmap_column_expr_test2 VALUES ('tag2', bitmapBuild(cast([6,7,8,9,10,11,12,13,14,15] as Array(UInt32))));
INSERT INTO bitmap_column_expr_test2 VALUES ('tag3', bitmapBuild(cast([2,4,6,8,10,12] as Array(UInt32))));

SELECT groupBitmapOr(z) FROM bitmap_column_expr_test2 WHERE like(tag_id, 'tag%');
┌─groupBitmapOr(z)─┐
│             15   │
└──────────────────┘

SELECT groupBitmapOrMerge(z) FROM bitmap_column_expr_test2 WHERE like(tag_id, 'tag%');
┌─groupBitmapOrMerge(z)─┐
│                  15   │
└───────────────────────┘

SELECT arraySort(bitmapToArray(groupBitmapOrState(z))) FROM bitmap_column_expr_test2 WHERE like(tag_id, 'tag%');
┌─arraySort(bitmapToArray(groupBitmapOrState(z)))─┐
│ [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15]           │
└─────────────────────────────────────────────────┘

groupBitmapXor

Calculations the XOR of a bitmap column, return cardinality of type UInt64, if add suffix -State, then return bitmap object.

groupBitmapOr(expr)

Parameters

expr – An expression that results in AggregateFunction(groupBitmap, UInt*) type.

Return value

Value of the UInt64 type.

Example

DROP TABLE IF EXISTS bitmap_column_expr_test2;
CREATE TABLE bitmap_column_expr_test2
(
    tag_id String,
    z AggregateFunction(groupBitmap, UInt32)
)
ENGINE = MergeTree
ORDER BY tag_id;

INSERT INTO bitmap_column_expr_test2 VALUES ('tag1', bitmapBuild(cast([1,2,3,4,5,6,7,8,9,10] as Array(UInt32))));
INSERT INTO bitmap_column_expr_test2 VALUES ('tag2', bitmapBuild(cast([6,7,8,9,10,11,12,13,14,15] as Array(UInt32))));
INSERT INTO bitmap_column_expr_test2 VALUES ('tag3', bitmapBuild(cast([2,4,6,8,10,12] as Array(UInt32))));

SELECT groupBitmapXor(z) FROM bitmap_column_expr_test2 WHERE like(tag_id, 'tag%');
┌─groupBitmapXor(z)─┐
│              10   │
└───────────────────┘

SELECT groupBitmapXorMerge(z) FROM bitmap_column_expr_test2 WHERE like(tag_id, 'tag%');
┌─groupBitmapXorMerge(z)─┐
│                   10   │
└────────────────────────┘

SELECT arraySort(bitmapToArray(groupBitmapXorState(z))) FROM bitmap_column_expr_test2 WHERE like(tag_id, 'tag%');
┌─arraySort(bitmapToArray(groupBitmapXorState(z)))─┐
│ [1,3,5,6,8,10,11,13,14,15]                       │
└──────────────────────────────────────────────────┘

alexey-milovidov · 2019-09-26T18:56:32Z

Let's make the names consistent with our existing groupBitAnd, groupBitOr, groupBitXor, groupArray functions (use "group" instead of "agg"):

groupBitmapAnd, groupBitmapOr, groupBitmapXor.

yuzhichang · 2019-09-27T05:24:10Z

@alexey-milovidov Renamed per your suggestion.

alexey-milovidov · 2019-10-08T19:51:39Z

Style check does not show any errors after merge.

abyss7 · 2019-10-10T14:51:55Z

Simple query SELECT groupBitmapOr(31) crashes the server. Can you please investigate and fix this?

akuzm · 2019-10-10T15:34:07Z

We've also observed a ThreadSanitizer failure in Stress test on current master:
https://clickhouse-test-reports.s3.yandex.net/7247/3fca2e42ca187bd4209726da87361e93882b503d/stress_test_(thread)/stderr.log

Looks more like a use-after-free.

akuzm · 2019-10-10T15:40:15Z

dbms/src/AggregateFunctions/AggregateFunctionGroupBitmap.h

+    void add(AggregateDataPtr place, const IColumn ** columns, size_t row_num, Arena *) const override
+    {
+        Data & data_lhs = this->data(place);
+        const Data & data_rhs = this->data(static_cast<const ColumnAggregateFunction &>(*columns[0]).getData()[row_num]);


Better to use assert_cast here instead of static_cast, like in other aggregate functions.

@akuzm I don't see how use-after-free happen at dbms/src/AggregateFunctions/AggregateFunctionGroupBitmap.h:74. Could you help on this?

@akuzm @abyss7 After replacing static_cast with assert_cast, SELECT groupBitmapOr(31) still crashes the server(segfault at AggregateFunctionGroupBitmap.h:74), but not the debug server(built with cmake -DCMAKE_BUILD_TYPE=Debug .., cli gets error Code: 368. DB::Exception: Received from localhost:9000. DB::Exception: Bad cast from type DB::ColumnVector<unsigned char> to DB::ColumnAggregateFunction. ).
Do you think this change is good enough?

The error message means that you are casting the column to a wrong type: ColumnVector<usigned char> to ColumnAggregateFunction. The next step would be to correct this wrong cast.

@akuzm It's hard for createAggregateFunctionBitmapL2 to distinguish groupBitmapOrMerge(AggregateFunction(groupBitmap, UInt32)) from groupBitmapOr(UInt32).
So I decide to remove support of groupBitmapOrMerge(AggregateFunction(groupBitmap, UInt32)).
Please review #7128.

akuzm · 2019-10-11T10:03:49Z

An AddressSanitizer failure in master:
https://clickhouse-test-reports.s3.yandex.net/7270/e25bdc8d25656486a73493ab0724342f40eeb27b/stress_test_(address)/stderr.log

Fixed #7109 groupBitmapOr(31) segfault

This was referenced Sep 26, 2019

Implementing Bitmap aggregate functions #6955

Closed

Slow performance for bitmap functions #6880

Closed

akuzm added the can be tested label Sep 26, 2019

yuzhichang changed the title ~~added aggBitmapAnd, aggBitmapOr, aggBitmapXor~~ added groupBitmapAnd, groupBitmapOr, groupBitmapXor Sep 27, 2019

added groupBitmapAnd, groupBitmapOr, groupBitmapXor

8945714

alexey-milovidov merged commit 4c1f017 into ClickHouse:master Oct 8, 2019

yuzhichang deleted the aggBitmapAnd branch October 9, 2019 00:27

akuzm reviewed Oct 10, 2019

View reviewed changes

zhang2014 mentioned this pull request Oct 12, 2019

parallel skip index scan #7057

Closed

KochetovNicolai added a commit that referenced this pull request Oct 12, 2019

Merge pull request #7289 from infinivision/fix_pull_7109

6f758ba

Fixed #7109 groupBitmapOr(31) segfault

akuzm added the pr-feature Pull request with new product feature label Oct 29, 2019

alexey-milovidov assigned akuzm Jun 6, 2021

alexey-milovidov mentioned this pull request Jun 6, 2021

Fix crash when chaining uniqStates #24523

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

added groupBitmapAnd, groupBitmapOr, groupBitmapXor#7109

added groupBitmapAnd, groupBitmapOr, groupBitmapXor#7109
alexey-milovidov merged 1 commit intoClickHouse:masterfrom
infinivision:aggBitmapAnd

yuzhichang commented Sep 26, 2019 •

edited

Loading

Uh oh!

alexey-milovidov commented Sep 26, 2019

Uh oh!

yuzhichang commented Sep 27, 2019

Uh oh!

alexey-milovidov commented Oct 8, 2019

Uh oh!

abyss7 commented Oct 10, 2019

Uh oh!

akuzm commented Oct 10, 2019

Uh oh!

akuzm Oct 10, 2019

Uh oh!

yuzhichang Oct 11, 2019

Uh oh!

yuzhichang Oct 11, 2019

Uh oh!

akuzm Oct 11, 2019

Uh oh!

yuzhichang Oct 12, 2019

Uh oh!

akuzm commented Oct 11, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

yuzhichang commented Sep 26, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

groupBitmapAnd

groupBitmapOr

groupBitmapXor

Uh oh!

alexey-milovidov commented Sep 26, 2019

Uh oh!

yuzhichang commented Sep 27, 2019

Uh oh!

alexey-milovidov commented Oct 8, 2019

Uh oh!

abyss7 commented Oct 10, 2019

Uh oh!

akuzm commented Oct 10, 2019

Uh oh!

akuzm Oct 10, 2019

Choose a reason for hiding this comment

Uh oh!

yuzhichang Oct 11, 2019

Choose a reason for hiding this comment

Uh oh!

yuzhichang Oct 11, 2019

Choose a reason for hiding this comment

Uh oh!

akuzm Oct 11, 2019

Choose a reason for hiding this comment

Uh oh!

yuzhichang Oct 12, 2019

Choose a reason for hiding this comment

Uh oh!

akuzm commented Oct 11, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yuzhichang commented Sep 26, 2019 •

edited

Loading