Skip to content

Atomic metadata.#11745

Merged
alexey-milovidov merged 70 commits intomasterfrom
atomic_metadata5
Jun 23, 2020
Merged

Atomic metadata.#11745
alexey-milovidov merged 70 commits intomasterfrom
atomic_metadata5

Conversation

@alesapin
Copy link
Copy Markdown
Member

@alesapin alesapin commented Jun 17, 2020

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Changelog category (leave one):

  • Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Multiversion metadata for storages without structure locks.

alesapin added 30 commits June 15, 2020 19:55
@alesapin
Compilable setColumns, setConstraints, setIndices
af2fe2b
@alesapin
Working setColumns, setConstraints, setIndices
aa30649
@alesapin
Move set*Key methods to StorageInMemoryMetadata
5fc41c7
@alesapin
Working set*Keys methods
e667eb5
@alesapin
TTL methods in StorageInMemoryMetadata
33a74a3
@alesapin
All set methods in metadata
b47a732
@alesapin
Metadata in read and write methods of IStorage
36ba019
@alesapin
Merge branch 'master' into atomic_metadata5
eae901f
@alesapin
Copy some methods to metdata
0f286ac
@alesapin
Get column dependencies in StorageInMemoryMetadata
0bcd220
@alesapin
Move getSampleBlockNonMaterialized to StorageInMemoryMetadata
53cb521
@alesapin
getSampleBlockWithVirtuals in StorageInMemoryMetadata
08b9aa6
@alesapin
Compileable getSampleBlockWithColumns in StorageInMemoryMetadata
71f99a2
@alesapin
Seems to be working getSampleBlockWithColumns in StorageInMemoryMetadata
824d666
@alesapin
Buildable getSampleBlock in StorageInMemoryMetadata
1ddeb3d
@alesapin
getConstraints() in StorageInMemoryMetadata (suspicious commit, but p…
ccc2bda 
…retend to work)
@alesapin
Fix race condition
fa60903
@alesapin
Merge branch 'master' into atomic_metadata5
2fac329
@alesapin
Secondary indices in StorageInMemoryMetadata
62f2c17
@alesapin
Partition key in StorageInMemoryMetadata
ab61abc
@alesapin
Compilable sorting key in metadata
ba04d02
@alesapin
Fix race condition
eca6e90
@alesapin
Sampling key in StorageInMemoryMetadata
1da393b
@alesapin
Primary key in storage metadata
1afdebe
@alesapin
Missed change for primary key
7064a36
@alesapin
TTL in storage in memory metadata
ed8f3b2
@alesapin
Settings changes in StorageInMemoryMetadata
eaaef83
@alesapin
Select query in metadata
31abbe5
@alesapin
Check methods in metadata
33c27de
@alesapin
Better getVirtuals method
ef8781c
@alesapin
Copy link
Copy Markdown
Member Author

alesapin commented Jun 19, 2020 •
edited
Loading

TODO:

  • 1. Add a test with ALTER during long select. (We are absolutely non blocking now)
  • 2. Add comments.
  • 3. Realize what's wrong with 01213_alter_rename_column_zookeeper

@alesapin
Copy link
Copy Markdown
Member Author

alesapin commented Jun 19, 2020

Stress test (memory) — Killed by signal (in clickhouse-server.log)
FunctionExtractAllGroups in CI fest.

@alesapin alesapin marked this pull request as ready for review June 22, 2020 09:49
@alesapin alesapin changed the title [WIP] Atomic metadata. Atomic metadata. Jun 22, 2020
@alesapin
Copy link
Copy Markdown
Member Author

alesapin commented Jun 22, 2020 •
edited
Loading

  • Fix clang-tidy.

@alexey-milovidov alexey-milovidov self-assigned this Jun 22, 2020
@alesapin
Copy link
Copy Markdown
Member Author

alesapin commented Jun 23, 2020

Race condition in stress test. https://gist.github.com/alesapin/82c0f645d714eb74de255040c1d08fc2

@alesapin
Copy link
Copy Markdown
Member Author

alesapin commented Jun 23, 2020

Race condition in stress test. https://gist.github.com/alesapin/82c0f645d714eb74de255040c1d08fc2

Fix 292016e

@alesapin
Copy link
Copy Markdown
Member Author

alesapin commented Jun 23, 2020 •
edited
Loading 

020-06-23 12:29:41 00953_moving_functions:                                                 [ FAIL ] 0.15 sec. - result differs with reference:
2020-06-23 12:29:41 --- /usr/share/clickhouse-test/queries/0_stateless/00953_moving_functions.reference	2020-06-23 11:24:32.000000000 +0300
2020-06-23 12:29:41 +++ /tmp/clickhouse-test/0_stateless/00953_moving_functions.stdout	2020-06-23 12:29:41.321314019 +0300
2020-06-23 12:29:41 @@ -37,4 +37,4 @@
2020-06-23 12:29:41  k	groupArrayMovingSum(v)
2020-06-23 12:29:41  String	Array(UInt64)
2020-06-23 12:29:41  a	[0,1,3,6,10]
2020-06-23 12:29:41 -b	[0,1,3,6,10,15,21,28,36,45]
2020-06-23 12:29:41 +b	[5,11,18,26,35,35,36,38,41,45]

Flap?

@alesapin
Copy link
Copy Markdown
Member Author

alesapin commented Jun 23, 2020 

020-06-23 12:29:41 00953_moving_functions:                                                 [ FAIL ] 0.15 sec. - result differs with reference:
2020-06-23 12:29:41 --- /usr/share/clickhouse-test/queries/0_stateless/00953_moving_functions.reference	2020-06-23 11:24:32.000000000 +0300
2020-06-23 12:29:41 +++ /tmp/clickhouse-test/0_stateless/00953_moving_functions.stdout	2020-06-23 12:29:41.321314019 +0300
2020-06-23 12:29:41 @@ -37,4 +37,4 @@
2020-06-23 12:29:41  k	groupArrayMovingSum(v)
2020-06-23 12:29:41  String	Array(UInt64)
2020-06-23 12:29:41  a	[0,1,3,6,10]
2020-06-23 12:29:41 -b	[0,1,3,6,10,15,21,28,36,45]
2020-06-23 12:29:41 +b	[5,11,18,26,35,35,36,38,41,45]

Flap?

Flap

alexey-milovidov
alexey-milovidov reviewed Jun 23, 2020
View reviewed changes
src/Storages/System/StorageSystemTables.cpp Outdated
if (columns_mask[src_index++])
{
assert(table != nullptr);
assert(metadata_snapshot != nullptr);
Copy link
Copy Markdown
Member

@alexey-milovidov alexey-milovidov Jun 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very small inconsistency...

@alexey-milovidov alexey-milovidov merged commit 55eee9b into master Jun 23, 2020
@alexey-milovidov alexey-milovidov deleted the atomic_metadata5 branch June 23, 2020 19:28
@alexey-milovidov alexey-milovidov mentioned this pull request Jun 27, 2020
Closed
@zhang2014 zhang2014 mentioned this pull request Sep 17, 2020
Closed
azat added a commit to azat/ClickHouse that referenced this pull request Nov 3, 2020
@azat
Do not pass StoragePtr to ReadInOrderOptimizer::getInputOrder()
e7f15df 
Looks like this is not required anymore, since ClickHouse#11745
@azat azat mentioned this pull request Nov 3, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alexey-milovidov alexey-milovidov alexey-milovidov approved these changes

Assignees

@alexey-milovidov alexey-milovidov

Labels

pr-improvement Pull request with some product improvements

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@alesapin @alexey-milovidov