Skip to content

Allow clickhouse-local to run without system tables#18899

Merged
alexey-milovidov merged 1 commit intomasterfrom
clickhouse-local-no-system-tables
Jan 11, 2021
Merged

Allow clickhouse-local to run without system tables#18899
alexey-milovidov merged 1 commit intomasterfrom
clickhouse-local-no-system-tables

Conversation

@alexey-milovidov
Copy link
Copy Markdown
Member

@alexey-milovidov alexey-milovidov commented Jan 10, 2021

Changelog category (leave one):

  • Performance Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Add --no-system-table option for clickhouse-local to run without system tables. This avoids initialization of DateLUT that may take noticeable amount of time (tens of milliseconds) at startup.

@robot-clickhouse robot-clickhouse added the pr-performance Pull request with some performance improvements label Jan 10, 2021
@alexey-milovidov
Copy link
Copy Markdown
Member Author

alexey-milovidov commented Jan 11, 2021

@alesapin No good:

2021.01.10 17:54:10.588483 [ 229040 ] {} <Debug> system.text_log (29ec3865-0db2-4a55-834b-8fcd9c7fc97e) (MergerMutator): Merging 6 parts: from 202101_1_236_38 to 202101_1214_1276_12 into Wide
2021.01.10 17:54:10.588922 [ 229040 ] {} <Debug> system.text_log (29ec3865-0db2-4a55-834b-8fcd9c7fc97e) (MergerMutator): Selected MergeAlgorithm: Vertical
2021.01.10 17:54:10.589664 [ 229040 ] {} <Trace> MergeTreeSequentialSource: Reading 30 marks from part 202101_1_236_38, total 231097 rows starting from the beginning of the part
2021.01.10 17:54:10.590701 [ 229040 ] {} <Trace> MergeTreeSequentialSource: Reading 60 marks from part 202101_237_630_29, total 486482 rows starting from the beginning of the part
2021.01.10 17:54:10.591648 [ 229040 ] {} <Trace> MergeTreeSequentialSource: Reading 42 marks from part 202101_631_803_15, total 334259 rows starting from the beginning of the part
2021.01.10 17:54:10.592655 [ 229040 ] {} <Trace> MergeTreeSequentialSource: Reading 28 marks from part 202101_804_1009_19, total 216171 rows starting from the beginning of the part
2021.01.10 17:54:10.593577 [ 229040 ] {} <Trace> MergeTreeSequentialSource: Reading 16 marks from part 202101_1010_1213_25, total 111962 rows starting from the beginning of the part
2021.01.10 17:54:10.594493 [ 229040 ] {} <Trace> MergeTreeSequentialSource: Reading 4 marks from part 202101_1214_1276_12, total 22793 rows starting from the beginning of the part
2021.01.10 17:54:10.750617 [ 229040 ] {} <Trace> MergeTreeSequentialSource: Reading 30 marks from part 202101_1_236_38, total 231097 rows starting from the beginning of the part, column event_time_microseconds
2021.01.10 17:54:10.751308 [ 229040 ] {} <Trace> MergeTreeSequentialSource: Reading 60 marks from part 202101_237_630_29, total 486482 rows starting from the beginning of the part, column event_time_microseconds
2021.01.10 17:54:10.751918 [ 229040 ] {} <Trace> MergeTreeSequentialSource: Reading 42 marks from part 202101_631_803_15, total 334259 rows starting from the beginning of the part, column event_time_microseconds
2021.01.10 17:54:10.752542 [ 229040 ] {} <Trace> MergeTreeSequentialSource: Reading 28 marks from part 202101_804_1009_19, total 216171 rows starting from the beginning of the part, column event_time_microseconds
2021.01.10 17:54:10.753175 [ 229040 ] {} <Trace> MergeTreeSequentialSource: Reading 16 marks from part 202101_1010_1213_25, total 111962 rows starting from the beginning of the part, column event_time_microseconds
2021.01.10 17:54:10.753799 [ 229040 ] {} <Trace> MergeTreeSequentialSource: Reading 4 marks from part 202101_1214_1276_12, total 22793 rows starting from the beginning of the part, column event_time_microseconds
2021.01.10 17:54:11.277107 [ 229040 ] {} <Fatal> : Logical error: 'Incorrect mark rows for mark #174 (compressed offset 6655963, decompressed offset 0), actually in bin file 7610, in mrk file 8139'.
2021.01.10 17:54:11.278558 [ 247840 ] {} <Fatal> BaseDaemon: ########################################
2021.01.10 17:54:11.278710 [ 247840 ] {} <Fatal> BaseDaemon: (version 20.13.1.5638, build id: E38BE3A20EC1A0AF2C98E17AFEAC0D4311A2070F) (from thread 229040) (no query) Received signal Aborted (6)
2021.01.10 17:54:11.278823 [ 247840 ] {} <Fatal> BaseDaemon: 
2021.01.10 17:54:11.278975 [ 247840 ] {} <Fatal> BaseDaemon: Stack trace: 0x7f4158a4b18b 0x7f4158a2a859 0x1194496c 0x11944a11 0x1bd27bef 0x1bd1d1df 0x1bd1de47 0x1bd1dfd3 0x1be7acca 0x1bc9f210 0x1b93eb2d 0x1b94d1bf 0x1b94d10d 0x1b94d0bd 0x1b94d08d 0x1b94c1de 0x11985cd5 0x11984f65 0x1bbe1f3a 0x1bbe1eed 0x1bbe1e9d 0x1bbe1e6d 0x1bbe105e 0x11985cd5 0x11984f65 0x119ad358 0x119b8034 0x119b7ffd
2021.01.10 17:54:11.279216 [ 247840 ] {} <Fatal> BaseDaemon: 4. raise @ 0x4618b in /usr/lib/x86_64-linux-gnu/libc-2.31.so
2021.01.10 17:54:11.279367 [ 247840 ] {} <Fatal> BaseDaemon: 5. abort @ 0x25859 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
2021.01.10 17:54:11.279593 [ 247840 ] {} <Fatal> BaseDaemon: 6. ./obj-x86_64-linux-gnu/../src/Common/Exception.cpp:50: DB::handle_error_code(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x1194496c in /usr/bin/clickhouse
2021.01.10 17:54:11.279789 [ 247840 ] {} <Fatal> BaseDaemon: 7. ./obj-x86_64-linux-gnu/../src/Common/Exception.cpp:56: DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x11944a11 in /usr/bin/clickhouse
2021.01.10 17:54:11.323485 [ 247840 ] {} <Fatal> BaseDaemon: 8. ./obj-x86_64-linux-gnu/../src/Common/Exception.h:38: DB::Exception::Exception<unsigned long&, unsigned long&, unsigned long&, unsigned long, unsigned long>(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned long&, unsigned long&, unsigned long&, unsigned long&&, unsigned long&&) @ 0x1bd27bef in /usr/bin/clickhouse
2021.01.10 17:54:11.367532 [ 247840 ] {} <Fatal> BaseDaemon: 9. ./obj-x86_64-linux-gnu/../src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:457: DB::MergeTreeDataPartWriterWide::validateColumnOfFixedSize(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::IDataType const&) @ 0x1bd1d1df in /usr/bin/clickhouse
2021.01.10 17:54:11.411549 [ 247840 ] {} <Fatal> BaseDaemon: 10. ./obj-x86_64-linux-gnu/../src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:521: DB::MergeTreeDataPartWriterWide::finishDataSerialization(DB::MergeTreeDataPartChecksums&, bool) @ 0x1bd1de47 in /usr/bin/clickhouse
2021.01.10 17:54:11.454947 [ 247840 ] {} <Fatal> BaseDaemon: 11. ./obj-x86_64-linux-gnu/../src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp:529: DB::MergeTreeDataPartWriterWide::finish(DB::MergeTreeDataPartChecksums&, bool) @ 0x1bd1dfd3 in /usr/bin/clickhouse
2021.01.10 17:54:11.499575 [ 247840 ] {} <Fatal> BaseDaemon: 12. ./obj-x86_64-linux-gnu/../src/Storages/MergeTree/MergedColumnOnlyOutputStream.cpp:69: DB::MergedColumnOnlyOutputStream::writeSuffixAndGetChecksums(std::__1::shared_ptr<DB::IMergeTreeDataPart>&, DB::MergeTreeDataPartChecksums&, bool) @ 0x1be7acca in /usr/bin/clickhouse
2021.01.10 17:54:11.542945 [ 247840 ] {} <Fatal> BaseDaemon: 13. ./obj-x86_64-linux-gnu/../src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp:1052: DB::MergeTreeDataMergerMutator::mergePartsToTemporaryPart(DB::FutureMergedMutatedPart const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::BackgroundProcessListEntry<DB::MergeListElement, DB::MergeInfo>&, std::__1::shared_ptr<DB::RWLockImpl::LockHolderImpl>&, long, DB::Context const&, std::__1::unique_ptr<DB::IReservation, std::__1::default_delete<DB::IReservation> > const&, bool, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0x1bc9f210 in /usr/bin/clickhouse
2021.01.10 17:54:11.582249 [ 247840 ] {} <Fatal> BaseDaemon: 14. ./obj-x86_64-linux-gnu/../src/Storages/StorageMergeTree.cpp:793: DB::StorageMergeTree::mergeSelectedParts(std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, bool, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, DB::StorageMergeTree::MergeMutateSelectedEntry&, std::__1::shared_ptr<DB::RWLockImpl::LockHolderImpl>&) @ 0x1b93eb2d in /usr/bin/clickhouse
2021.01.10 17:54:11.623832 [ 247840 ] {} <Fatal> BaseDaemon: 15. ./obj-x86_64-linux-gnu/../src/Storages/StorageMergeTree.cpp:962: DB::StorageMergeTree::getDataProcessingJob()::$_4::operator()() @ 0x1b94d1bf in /usr/bin/clickhouse
2021.01.10 17:54:11.664456 [ 247840 ] {} <Fatal> BaseDaemon: 16. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:3519: decltype(std::__1::forward<DB::StorageMergeTree::getDataProcessingJob()::$_4&>(fp)()) std::__1::__invoke<DB::StorageMergeTree::getDataProcessingJob()::$_4&>(DB::StorageMergeTree::getDataProcessingJob()::$_4&) @ 0x1b94d10d in /usr/bin/clickhouse
2021.01.10 17:54:11.705085 [ 247840 ] {} <Fatal> BaseDaemon: 17. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/__functional_base:349: void std::__1::__invoke_void_return_wrapper<void>::__call<DB::StorageMergeTree::getDataProcessingJob()::$_4&>(DB::StorageMergeTree::getDataProcessingJob()::$_4&) @ 0x1b94d0bd in /usr/bin/clickhouse
2021.01.10 17:54:11.745371 [ 247840 ] {} <Fatal> BaseDaemon: 18. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1540: std::__1::__function::__alloc_func<DB::StorageMergeTree::getDataProcessingJob()::$_4, std::__1::allocator<DB::StorageMergeTree::getDataProcessingJob()::$_4>, void ()>::operator()() @ 0x1b94d08d in /usr/bin/clickhouse
2021.01.10 17:54:11.784811 [ 247840 ] {} <Fatal> BaseDaemon: 19. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1714: std::__1::__function::__func<DB::StorageMergeTree::getDataProcessingJob()::$_4, std::__1::allocator<DB::StorageMergeTree::getDataProcessingJob()::$_4>, void ()>::operator()() @ 0x1b94c1de in /usr/bin/clickhouse
2021.01.10 17:54:11.785255 [ 247840 ] {} <Fatal> BaseDaemon: 20. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1867: std::__1::__function::__value_func<void ()>::operator()() const @ 0x11985cd5 in /usr/bin/clickhouse
2021.01.10 17:54:11.785551 [ 247840 ] {} <Fatal> BaseDaemon: 21. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2473: std::__1::function<void ()>::operator()() const @ 0x11984f65 in /usr/bin/clickhouse
2021.01.10 17:54:11.825412 [ 247840 ] {} <Fatal> BaseDaemon: 22. ./obj-x86_64-linux-gnu/../src/Storages/MergeTree/BackgroundJobsExecutor.cpp:101: DB::IBackgroundJobExecutor::jobExecutingTask()::$_0::operator()() const @ 0x1bbe1f3a in /usr/bin/clickhouse
2021.01.10 17:54:11.865291 [ 247840 ] {} <Fatal> BaseDaemon: 23. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:3519: decltype(std::__1::forward<DB::IBackgroundJobExecutor::jobExecutingTask()::$_0&>(fp)()) std::__1::__invoke<DB::IBackgroundJobExecutor::jobExecutingTask()::$_0&>(DB::IBackgroundJobExecutor::jobExecutingTask()::$_0&) @ 0x1bbe1eed in /usr/bin/clickhouse
2021.01.10 17:54:11.905512 [ 247840 ] {} <Fatal> BaseDaemon: 24. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/__functional_base:349: void std::__1::__invoke_void_return_wrapper<void>::__call<DB::IBackgroundJobExecutor::jobExecutingTask()::$_0&>(DB::IBackgroundJobExecutor::jobExecutingTask()::$_0&) @ 0x1bbe1e9d in /usr/bin/clickhouse
2021.01.10 17:54:11.945612 [ 247840 ] {} <Fatal> BaseDaemon: 25. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1540: std::__1::__function::__alloc_func<DB::IBackgroundJobExecutor::jobExecutingTask()::$_0, std::__1::allocator<DB::IBackgroundJobExecutor::jobExecutingTask()::$_0>, void ()>::operator()() @ 0x1bbe1e6d in /usr/bin/clickhouse
2021.01.10 17:54:11.985414 [ 247840 ] {} <Fatal> BaseDaemon: 26. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1714: std::__1::__function::__func<DB::IBackgroundJobExecutor::jobExecutingTask()::$_0, std::__1::allocator<DB::IBackgroundJobExecutor::jobExecutingTask()::$_0>, void ()>::operator()() @ 0x1bbe105e in /usr/bin/clickhouse
2021.01.10 17:54:11.985822 [ 247840 ] {} <Fatal> BaseDaemon: 27. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1867: std::__1::__function::__value_func<void ()>::operator()() const @ 0x11985cd5 in /usr/bin/clickhouse
2021.01.10 17:54:11.986078 [ 247840 ] {} <Fatal> BaseDaemon: 28. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2473: std::__1::function<void ()>::operator()() const @ 0x11984f65 in /usr/bin/clickhouse
2021.01.10 17:54:11.986410 [ 247840 ] {} <Fatal> BaseDaemon: 29. ./obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:243: ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0x119ad358 in /usr/bin/clickhouse
2021.01.10 17:54:11.987348 [ 247840 ] {} <Fatal> BaseDaemon: 30. ./obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:124: void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()::operator()() const @ 0x119b8034 in /usr/bin/clickhouse
2021.01.10 17:54:11.988245 [ 247840 ] {} <Fatal> BaseDaemon: 31. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:3525: decltype(std::__1::forward<void>(fp)(std::__1::forward<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&>(fp0)...)) std::__1::__invoke_constexpr<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&...) @ 0x119b7ffd in /usr/bin/clickhouse
2021.01.10 17:54:12.996241 [ 247840 ] {} <Fatal> BaseDaemon: Calculated checksum of the binary: 52F1732EBC77E228F1C658DB4B71036D. There is no information about the reference checksum.
2021.01.10 17:54:23.098594 [ 154 ] {} <Fatal> Application: Child process was terminated by signal 6.

@alexey-milovidov alexey-milovidov mentioned this pull request Jan 11, 2021
Closed
@alexey-milovidov alexey-milovidov self-assigned this Jan 11, 2021
@alexey-milovidov alexey-milovidov merged commit 5cf711b into master Jan 11, 2021
@alexey-milovidov alexey-milovidov deleted the clickhouse-local-no-system-tables branch January 11, 2021 02:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@alexey-milovidov alexey-milovidov

Labels

pr-performance Pull request with some performance improvements

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@alexey-milovidov @robot-clickhouse