Add SYSTEM UNLOAD PRIMARY KEY#62738
Conversation
|
This is an automated comment for commit f6b19e1 with description of existing statuses. It's updated for the latest CI running ❌ Click here to open a full report in a separate page
Successful checks
|
|
Please update the description, |
The CI is injecting random values to detect flaky tests. Whenever the index_granularity was changed, the result did not have the right expectations.
|
I based In any case, I've made an attempt to make it pass for more scenarios, but I've failed. Since I'm not very savvy about the internals of your CI, I'd appreciate a hand to figure out whether this might really be an issue because of my changes or if you think there's a way to make a test for this feature that works no matter what parameters are used by the flakiness-looking CI job. |
|
Alright, I think I found the issue with the flaky tests. I'll check to see whether I can split the test in two to let a new test run sequentially so that no other tests interfere with it |
| std::scoped_lock lock(index_mutex); | ||
| index.clear(); | ||
| index_loaded = false; |
There was a problem hiding this comment.
What will happen if some queries are still using the index?
There was a problem hiding this comment.
That's an excellent question 🤔
I took a look and since read queries that use getIndex get a reference to underlying index as it is, inflight queries would be affected by the SYSTEM UNLOAD PRIMARY KEY that clears it.
There are several solutions to this, but the one that I found simpler to implement, to reason about and doesn't need expensive copies is to promote index to a shared_ptr. Here's the commit with it.
I saw that the IColumn interface is using a copy-on-write shared pointer, so my 2nd option was to create a new std::vector where I assign every column from the original ìndex. However, that implies increasing the refcount for every single column, so IMO it's cheaper and easier to understand doing that in the parent directly.
There was a problem hiding this comment.
Yes, using shared_ptr is good solution, I would do exactly the same
|
ClickHouse special build check failure is related, other failures are not |
Removing the reference for Index makes the const unnecessary. Constness for it is still preserved correctly because Columns are inmutable pointers to IColumn.
Fixed in f6b19e1 Leaving here for reference the command I used to ensure to run clang-tidy in parallel on all modified files of my branch: git diff --stat master..HEAD | grep -P "src.*\.cpp" | awk '{print $1}' | xargs -i -P0 bash -c "echo analyzing {}; clang-tidy-17 -p build_master {}" |
|
I think the tests that failed now are unrelated to my changes. Apart from that, I haven't figured out an easy way to test that unloading primary keys while some queries are using them interfere with each other. I've tested manually, but I don't think I can run easily create a functional test for it to verify its behavior. The only alternative I can think of is to do a unit test that does something along these lines: const auto & index = part->getIndex();
EXPECT_EQ(index->size(), 10);
part->unloadIndex();
EXPECT_EQ(index->size(), 10);
const auto & new_index = part->getIndex();
EXPECT_EQ(new_index->size(), 0);However, the issue is getting a Let me know what you think. |
|
Yes, failures are unrelated
This will be tested by Stress tests. They run 30 instances of Stateless tests in parallel and they don't care about |
| /// and we need to use the same set of index columns across all parts. | ||
| for (const auto & part : parts) | ||
| loaded_columns = std::min(loaded_columns, part.data_part->getIndex().size()); | ||
| loaded_columns = std::min(loaded_columns, part.data_part->getIndex()->size()); |
There was a problem hiding this comment.
Isn't this cached value now broken? If you change primary_key_ratio_of_unique_prefix_values_to_skip_suffix_columns and unload the index, the loaded_columns will be incorrect and any query using this IndexAccess might have bad values or crash (if some columns where removed).
loaded_columns is only ok if we also cache and save the index of all the parts.
There was a problem hiding this comment.
Confirmed by introducing sleep in getValue and doing concurrent setting change, queries and unloading of the PK. It hits the chassert:
{fa241230-2c8c-4b65-bff9-f84ee8f5feec} <Fatal> : Logical error: 'index->size() >= loaded_columns'
2024.05.14 14:34:41.614779 [ 229319 ] {} <Fatal> BaseDaemon: ########## Short fault info ############
2024.05.14 14:34:41.614935 [ 229319 ] {} <Fatal> BaseDaemon: (version 24.5.1.1, build id: 2DF79347B1B8FBD97ABA348E494E7B9B1CB78632, git hash: 11753dfbaebe7e929a43f7ba20d2e369430d9463) (from thread 228422) Received signal 6
2024.05.14 14:34:41.615089 [ 229319 ] {} <Fatal> BaseDaemon: Signal description: Aborted
2024.05.14 14:34:41.615182 [ 229319 ] {} <Fatal> BaseDaemon:
2024.05.14 14:34:41.615286 [ 229319 ] {} <Fatal> BaseDaemon: Stack trace: 0x00007ffff7e2de44 0x00007ffff7dd5a30 0x00007ffff7dbd4c3 0x00000000143c32da 0x000000002015f39d 0x000000002015c981 0x00000000201579c6 0x00000000200f3ef2 0x00000000200ffe2e 0x000000002010155c 0x00000000200a3f80 0x00000000200cdf31 0x000000001d39ce86 0x000000001d39cb71 0x000000001dad60a3 0x000000001dad03de 0x000000001f6ebd99 0x000000001f702cf2 0x00000000251c4b19 0x00000000251c537d 0x00000000253cd921 0x00000000253ca11a 0x00000000253c8bd5 0x00007ffff7e2bded 0x00007ffff7eaf0dc
2024.05.14 14:34:41.615412 [ 229319 ] {} <Fatal> BaseDaemon: ########################################
2024.05.14 14:34:41.615635 [ 229319 ] {} <Fatal> BaseDaemon: (version 24.5.1.1, build id: 2DF79347B1B8FBD97ABA348E494E7B9B1CB78632, git hash: 11753dfbaebe7e929a43f7ba20d2e369430d9463) (from thread 228422) (query_id: 2494f339-9bc1-41cc-b2d8-19c785d15966) (query: SELECT count() FROM t where not ignore(*);) Received signal Aborted (6)
2024.05.14 14:34:41.615898 [ 229319 ] {} <Fatal> BaseDaemon:
2024.05.14 14:34:41.616049 [ 229319 ] {} <Fatal> BaseDaemon: Stack trace: 0x00007ffff7e2de44 0x00007ffff7dd5a30 0x00007ffff7dbd4c3 0x00000000143c32da 0x000000002015f39d 0x000000002015c981 0x00000000201579c6 0x00000000200f3ef2 0x00000000200ffe2e 0x000000002010155c 0x00000000200a3f80 0x00000000200cdf31 0x000000001d39ce86 0x000000001d39cb71 0x000000001dad60a3 0x000000001dad03de 0x000000001f6ebd99 0x000000001f702cf2 0x00000000251c4b19 0x00000000251c537d 0x00000000253cd921 0x00000000253ca11a 0x00000000253c8bd5 0x00007ffff7e2bded 0x00007ffff7eaf0dc
2024.05.14 14:34:41.616288 [ 229319 ] {} <Fatal> BaseDaemon: 4. ? @ 0x00007ffff7e2de44
2024.05.14 14:34:41.616429 [ 229319 ] {} <Fatal> BaseDaemon: 5. ? @ 0x00007ffff7dd5a30
2024.05.14 14:34:41.616598 [ 229319 ] {} <Fatal> BaseDaemon: 6. ? @ 0x00007ffff7dbd4c3
2024.05.14 14:34:41.693317 [ 229319 ] {} <Fatal> BaseDaemon: 7. /mnt/ch/ClickHouse/src/Common/Exception.cpp:0: DB::abortOnFailedAssertion(String const&) @ 0x00000000143c32da
2024.05.14 14:34:41.937273 [ 229319 ] {} <Fatal> BaseDaemon: 8. /mnt/ch/ClickHouse/src/Processors/QueryPlan/PartsSplitter.cpp:150: (anonymous namespace)::IndexAccess::getValue(unsigned long, unsigned long) const @ 0x000000002015f39d
2024.05.14 14:34:42.285853 [ 229319 ] {} <Fatal> BaseDaemon: 9. /mnt/ch/ClickHouse/src/Processors/QueryPlan/PartsSplitter.cpp:751: (anonymous namespace)::splitIntersectingPartsRangesIntoLayers(DB::RangesInDataParts, unsigned long, std::shared_ptr<Poco::Logger> const&) @ 0x000000002015c981
2024.05.14 14:34:42.603459 [ 229319 ] {} <Fatal> BaseDaemon: 10. /mnt/ch/ClickHouse/src/Processors/QueryPlan/PartsSplitter.cpp:934: DB::splitPartsWithRangesByPrimaryKey(DB::KeyDescription const&, std::shared_ptr<DB::ExpressionActions>, DB::RangesInDataParts, unsigned long, std::shared_ptr<DB::Context const>, std::function<DB::Pipe (DB::RangesInDataParts)>&&, bool, bool) @ 0x00000000201579c6
2024.05.14 14:34:42.995839 [ 229319 ] {} <Fatal> BaseDaemon: 11. /mnt/ch/ClickHouse/src/Processors/QueryPlan/ReadFromMergeTree.cpp:807: DB::ReadFromMergeTree::spreadMarkRangesAmongStreams(DB::RangesInDataParts&&, unsigned long, std::vector<String, std::allocator<String>> const&) @ 0x00000000200f3ef2
2024.05.14 14:34:43.412668 [ 229319 ] {} <Fatal> BaseDaemon: 12. /mnt/ch/ClickHouse/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1935: DB::ReadFromMergeTree::spreadMarkRanges(DB::RangesInDataParts&&, unsigned long, DB::ReadFromMergeTree::AnalysisResult&, std::shared_ptr<DB::ActionsDAG>&) @ 0x00000000200ffe2e
2024.05.14 14:34:43.802381 [ 229319 ] {} <Fatal> BaseDaemon: 13. /mnt/ch/ClickHouse/src/Processors/QueryPlan/ReadFromMergeTree.cpp:2032: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000002010155c
2024.05.14 14:34:43.851744 [ 229319 ] {} <Fatal> BaseDaemon: 14. /mnt/ch/ClickHouse/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x00000000200a3f80
2024.05.14 14:34:43.997606 [ 229319 ] {} <Fatal> BaseDaemon: 15. /mnt/ch/ClickHouse/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x00000000200cdf31
2024.05.14 14:34:44.089048 [ 229319 ] {} <Fatal> BaseDaemon: 16. /mnt/ch/ClickHouse/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:223: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001d39ce86
2024.05.14 14:34:44.189719 [ 229319 ] {} <Fatal> BaseDaemon: 17. /mnt/ch/ClickHouse/src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:192: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001d39cb71
2024.05.14 14:34:44.426371 [ 229319 ] {} <Fatal> BaseDaemon: 18. /mnt/ch/ClickHouse/src/Interpreters/executeQuery.cpp:1198: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001dad60a3
2024.05.14 14:34:44.665615 [ 229319 ] {} <Fatal> BaseDaemon: 19. /mnt/ch/ClickHouse/src/Interpreters/executeQuery.cpp:1393: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001dad03de
2024.05.14 14:34:44.886185 [ 229319 ] {} <Fatal> BaseDaemon: 20. /mnt/ch/ClickHouse/src/Server/TCPHandler.cpp:522: DB::TCPHandler::runImpl() @ 0x000000001f6ebd99
2024.05.14 14:34:45.128180 [ 229319 ] {} <Fatal> BaseDaemon: 21. /mnt/ch/ClickHouse/src/Server/TCPHandler.cpp:2341: DB::TCPHandler::run() @ 0x000000001f702cf2
2024.05.14 14:34:45.138595 [ 229319 ] {} <Fatal> BaseDaemon: 22. /mnt/ch/ClickHouse/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x00000000251c4b19
2024.05.14 14:34:45.154126 [ 229319 ] {} <Fatal> BaseDaemon: 23. /mnt/ch/ClickHouse/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000251c537d
2024.05.14 14:34:45.170743 [ 229319 ] {} <Fatal> BaseDaemon: 24. /mnt/ch/ClickHouse/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000253cd921
2024.05.14 14:34:45.186580 [ 229319 ] {} <Fatal> BaseDaemon: 25. /mnt/ch/ClickHouse/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000253ca11a
2024.05.14 14:34:45.201583 [ 229319 ] {} <Fatal> BaseDaemon: 26. /mnt/ch/ClickHouse/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000253c8bd5
2024.05.14 14:34:45.201842 [ 229319 ] {} <Fatal> BaseDaemon: 27. ? @ 0x00007ffff7e2bded
2024.05.14 14:34:45.202040 [ 229319 ] {} <Fatal> BaseDaemon: 28. ? @ 0x00007ffff7eaf0dc
2024.05.14 14:34:45.202244 [ 229319 ] {} <Fatal> BaseDaemon: Integrity check of the executable skipped because the reference checksum could not be read.
2024.05.14 14:34:45.202518 [ 229319 ] {} <Fatal> BaseDaemon: This ClickHouse version is not official and should be upgraded to the official build.
2024.05.14 14:34:45.203110 [ 229319 ] {} <Fatal> BaseDaemon: Changed settings: min_compress_block_size = 1770298, max_compress_block_size = 241108, max_block_size = 45947, max_insert_threads = 13, max_threads = 7, max_read_buffer_size = 959253, use_uncompressed_cache = false, compile_aggregate_expressions = true, compile_sort_description = false, min_count_to_compile_sort_description = 0, group_by_two_level_threshold = 108124, group_by_two_level_threshold_bytes = 46225353, enable_memory_bound_merging_of_aggregation_results = false, min_chunk_bytes_for_parallel_parsing = 3608449, merge_tree_coarse_index_granularity = 23, min_bytes_to_use_direct_io = 7413113399, min_bytes_to_use_mmap_io = 10737418240, insert_deduplicate = true, merge_tree_read_split_ranges_into_intersecting_and_non_intersecting_injection_probability = 0.9700000286102295, http_response_buffer_size = 8163244, distributed_ddl_task_timeout = 120, joined_subquery_requires_alias = false, max_bytes_before_external_group_by = 1, max_bytes_before_external_sort = 6665888579, max_bytes_before_remerge_sort = 1315671325, max_execution_time = 0., max_expanded_ast_elements = 50000, join_algorithm = 'hash', max_memory_usage = 50000000000, memory_usage_overcommit_max_wait_microseconds = 50000, log_query_threads = false, log_comment = '03151_unload_index_race.sh', send_logs_level = 'warning', optimize_read_in_order = false, optimize_aggregation_in_order = true, aggregation_in_order_max_block_bytes = 29881666, read_in_order_two_level_merge_threshold = 70, max_partitions_per_insert_block = 100, optimize_if_transform_strings_to_enum = true, optimize_substitute_columns = true, optimize_append_index = true, enable_global_with_statement = true, database_replicated_initial_query_timeout_sec = 120, database_replicated_enforce_synchronous_settings = true, database_replicated_always_detach_permanently = true, distributed_ddl_output_mode = 'none', local_filesystem_read_method = 'read', merge_tree_compact_parts_min_granules_to_multibuffer_read = 69, filesystem_cache_segments_batch_size = 50, use_page_cache_for_disks_without_file_cache = true, allow_prefetched_read_pool_for_remote_filesystem = false, filesystem_prefetch_step_bytes = 104857600, filesystem_prefetch_step_marks = 50, filesystem_prefetch_min_bytes_for_single_read_task = 8388608, filesystem_prefetch_max_memory_usage = 134217728, filesystem_prefetches_limit = 0, insert_keeper_max_retries = 1000, insert_keeper_retry_initial_backoff_ms = 1, insert_keeper_retry_max_backoff_ms = 1, insert_keeper_fault_injection_probability = 0., optimize_distinct_in_order = false, session_timezone = 'Mexico/BajaSur', allow_experimental_database_replicated = true, input_format_null_as_default = false
I'm trying to find a consistent reproducer if possible to verify the fix and will send a PR
There was a problem hiding this comment.
In the end I didn't need the thread fuzzer sleep. Increasing the number of readers without sleep also reproduced the issue
Add
SYSTEM UNLOAD PRIMARY KEYfor a given table or for all tablesCloses #60643
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Add
SYSTEM UNLOAD PRIMARY KEYDocumentation entry for user-facing changes