Skip to content

Slightly optimize very short queries with LowCardinality#14129

Merged
alexey-milovidov merged 1 commit intoClickHouse:masterfrom
CurtizJ:fix-performance
Aug 27, 2020
Merged

Slightly optimize very short queries with LowCardinality#14129
alexey-milovidov merged 1 commit intoClickHouse:masterfrom
CurtizJ:fix-performance

Conversation

@CurtizJ
Copy link
Copy Markdown
Member

@CurtizJ CurtizJ commented Aug 26, 2020 •
edited by alexey-milovidov
Loading

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Changelog category (leave one):

  • Performance Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Slightly optimize very short queries with LowCardinality.

Resolves #13864.

@robot-clickhouse robot-clickhouse added the pr-bugfix Pull request with bugfix, not backported by default label Aug 26, 2020
@CurtizJ
Copy link
Copy Markdown
Member Author

CurtizJ commented Aug 26, 2020

Look at QPS.
Before:

echo "SELECT col1, col2 FROM test PREWHERE col1 = '042fecec-ea68-1b26-00a4-a4f3b3987574'" | clickhouse-benchmark -c 1 -i 2000 -d 0 
Loaded 1 queries.

Queries executed: 2000.

localhost:9000, queries 2000, QPS: 735.422, RPS: 94134.037, MiB/s: 1.436, result RPS: 0.000, result MiB/s: 0.000.

0.000%		0.001 sec.	
10.000%		0.001 sec.	
20.000%		0.001 sec.	
30.000%		0.001 sec.	
40.000%		0.001 sec.	
50.000%		0.001 sec.	
60.000%		0.001 sec.	
70.000%		0.001 sec.	
80.000%		0.002 sec.	
90.000%		0.002 sec.	
95.000%		0.002 sec.	
99.000%		0.002 sec.	
99.900%		0.004 sec.	
99.990%		0.011 sec.

After:

echo "SELECT col1, col2 FROM test PREWHERE col1 = '042fecec-ea68-1b26-00a4-a4f3b3987574'" | clickhouse-benchmark -c 1 -i 2000 -d 0 
Loaded 1 queries.

Queries executed: 2000.

localhost:9000, queries 2000, QPS: 1030.173, RPS: 131862.092, MiB/s: 2.012, result RPS: 0.000, result MiB/s: 0.000.

0.000%		0.001 sec.	
10.000%		0.001 sec.	
20.000%		0.001 sec.	
30.000%		0.001 sec.	
40.000%		0.001 sec.	
50.000%		0.001 sec.	
60.000%		0.001 sec.	
70.000%		0.001 sec.	
80.000%		0.001 sec.	
90.000%		0.001 sec.	
95.000%		0.001 sec.	
99.000%		0.002 sec.	
99.900%		0.003 sec.	
99.990%		0.009 sec.

@CurtizJ
Copy link
Copy Markdown
Member Author

CurtizJ commented Aug 26, 2020

Currently it's impossible to add performance test for such short queries.

@alexey-milovidov alexey-milovidov added the pr-must-backport Pull request should be backported intentionally. Use this label with great care! label Aug 26, 2020
@robot-clickhouse robot-clickhouse added pr-performance Pull request with some performance improvements and removed pr-bugfix Pull request with bugfix, not backported by default labels Aug 26, 2020
@alexey-milovidov
Copy link
Copy Markdown
Member

alexey-milovidov commented Aug 26, 2020

I've read the fix but I don't understand how it's related to LowCardinality.

@alexey-milovidov alexey-milovidov self-assigned this Aug 26, 2020
@jameshartig
Copy link
Copy Markdown

jameshartig commented Aug 26, 2020

@CurtizJ looking forward to this! After your change, is there still a difference with experimental_use_processors?

@CurtizJ
Copy link
Copy Markdown
Member Author

CurtizJ commented Aug 26, 2020

I've read the fix but I don't understand how it's related to LowCardinality

The problem appeared, when we read dictionary of low cardinality. Since it has low size, a lot of marks point to one place, and previously we a lot of time was spent in loop, while trying to find mark to next compressed block.

@CurtizJ
Copy link
Copy Markdown
Member Author

CurtizJ commented Aug 26, 2020

After your change, is there still a difference with experimental_use_processors?

I can't check it now, because in new versions old code is already removed and experimental_use_processors is exists just for backward compatibility and does nothing.

@alexey-milovidov alexey-milovidov merged commit 9baa0fb into ClickHouse:master Aug 27, 2020
This was referenced Aug 27, 2020
Merged
Merged
Merged
Merged
robot-clickhouse pushed a commit that referenced this pull request Aug 27, 2020
Backport #14129 to 20.4: Slightly optimize very short queries with Lo…
3053df9 
…wCardinality
@robot-clickhouse robot-clickhouse mentioned this pull request Aug 27, 2020
Merged
robot-clickhouse pushed a commit that referenced this pull request Aug 27, 2020
Backport #14129 to 20.5: Slightly optimize very short queries with Lo…
985ee0b 
…wCardinality
@robot-clickhouse robot-clickhouse mentioned this pull request Aug 27, 2020
Merged
robot-clickhouse pushed a commit that referenced this pull request Aug 27, 2020
Backport #14129 to 20.7: Slightly optimize very short queries with Lo…
ca8f2ca 
…wCardinality
@robot-clickhouse robot-clickhouse mentioned this pull request Aug 27, 2020
Merged
robot-clickhouse pushed a commit that referenced this pull request Aug 27, 2020
Backport #14129 to 20.6: Slightly optimize very short queries with Lo…
428297c 
…wCardinality
@robot-clickhouse robot-clickhouse mentioned this pull request Aug 27, 2020
Merged
CurtizJ added a commit that referenced this pull request Aug 28, 2020
@CurtizJ
Merge pull request #14189 from ClickHouse/backport/20.6/14129
afb9aa7 
Backport #14129 to 20.6: Slightly optimize very short queries with LowCardinality
CurtizJ added a commit that referenced this pull request Aug 28, 2020
@CurtizJ
Merge pull request #14187 from ClickHouse/backport/20.5/14129
0f32e77 
Backport #14129 to 20.5: Slightly optimize very short queries with LowCardinality
CurtizJ added a commit that referenced this pull request Aug 28, 2020
@CurtizJ
Merge pull request #14186 from ClickHouse/backport/20.4/14129
2f66e60 
Backport #14129 to 20.4: Slightly optimize very short queries with LowCardinality
CurtizJ added a commit that referenced this pull request Aug 28, 2020
@CurtizJ
Merge pull request #14188 from ClickHouse/backport/20.7/14129
908a07e 
Backport #14129 to 20.7: Slightly optimize very short queries with LowCardinality
CurtizJ pushed a commit that referenced this pull request Aug 28, 2020
@alexey-milovidov @CurtizJ
Merge pull request #14129 from CurtizJ/fix-performance
e3561bc 
Slightly optimize very short queries with LowCardinality

(cherry picked from commit 9baa0fb)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alexey-milovidov alexey-milovidov alexey-milovidov approved these changes

Assignees

@alexey-milovidov alexey-milovidov

Labels

pr-must-backport Pull request should be backported intentionally. Use this label with great care! pr-performance Pull request with some performance improvements

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Performance degradation between 20.1 and 20.3 for large number of very short queries with LowCardinality fields

4 participants

@CurtizJ @alexey-milovidov @jameshartig @robot-clickhouse