Skip to content

Fix refcnt for unused MergeTree parts in SELECT queries#37913

Merged
CurtizJ merged 1 commit intoClickHouse:masterfrom
azat:parts-refcnt
Jun 21, 2022
Merged

Fix refcnt for unused MergeTree parts in SELECT queries#37913
CurtizJ merged 1 commit intoClickHouse:masterfrom
azat:parts-refcnt

Conversation

@azat
Copy link
Copy Markdown
Member

@azat azat commented Jun 7, 2022

Changelog category (leave one):

  • Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Fix refcnt for unused MergeTree parts in SELECT queries (may defer parts removal)

Before this patch SELECT queries hold parts even if they were not
required by select (had been eliminated by partition pruning).

This defers removing parts if you have long running queries.

This had been introduced in #23932, with introduction of
StorageSnapshotPtr.

Cc: @CurtizJ

@robot-clickhouse robot-clickhouse added the pr-improvement Pull request with some product improvements label Jun 7, 2022
Before this patch SELECT queries hold parts even if they were not
required by select (had been eliminated by partition pruning).

This defers removing parts if you have long running queries.

This had been introduced in ClickHouse#23932, with introduction of
StorageSnapshotPtr.

Signed-off-by: Azat Khuzhin <[email protected]>
@CurtizJ CurtizJ self-assigned this Jun 8, 2022
@azat
Copy link
Copy Markdown
Member Author

azat commented Jun 8, 2022

Integration tests (thread, actions) [4/4] — fail: 1, passed: 581, flaky: 1

  • test_keeper_multinode_simple/test.py::test_follower_restart - kazoo.exceptions.ConnectionLoss

Stateless tests (thread, actions) [1/3] — fail: 1, passed: 1390, skipped: 11

  • 01183_custom_separated_format_http - curl: (28) Operation timed out after 60001 milliseconds with 0 bytes received

Stress test (debug, actions) — Backward compatibility check: Error message in clickhouse-server.log (see bc_check_error_messages.txt)

2022.06.08 09:39:17.006005 [ 261869 ] {} <Error> MutateFromLogEntryTask: virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 47. DB::Exception: Missing columns: 'h' while processing query: 'h, b, g, d, a, c, f, e', required columns: 'h' 'a' 'b' 'g' 'd' 'c' 'f' 'e', maybe you meant: ['a','a','b','g','d','c','f','e']. (UNKNOWN_IDENTIFIER), Stack trace (when copying this message, always include the lines below):

From 00992_system_parts_race_condition_zookeeper_long

Will take a look later.

@azat azat requested a review from CurtizJ June 21, 2022 08:14
@CurtizJ
Copy link
Copy Markdown
Member

CurtizJ commented Jun 21, 2022

Tests with tsan were failing because of #38019. So, let's merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-improvement Pull request with some product improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants