Build with libcxx(abi) 15 by rschu1ze · Pull Request #42513 · ClickHouse/ClickHouse

rschu1ze · 2022-10-19T23:40:14Z

Changelog category (leave one):

Not for changelog (changelog entry is not required)

This is a prerequisite for #42245

Multiple things happen in this PR:

Submodules libcxx and libcxxabi are updated from 14 to 15, see Update to LLVM 15 libcxx#14 and Update to LLVM 15 libcxxabi#5
This broke some stuff:
- in submodule rocksdb: fixed with Re-shuffle class order to make it compile with libcxx 15 rocksdb#17
- in submodule vectorscan: fixed by patching stuff in libcxx, one of the two problems was reported to upstream here: assume_aligned vs. std::assume_aligned VectorCamp/vectorscan#134
- in ClickHouse, the main problem was that with libcxx 15 the compiler complains if typedefs are created for incomplete types. I am not sure if the standard mandates such behavior and if it previously worked by chance. Anyways, this problem hit in the aggregation code and in clickhouse-copier. I untangled the code so that it compiles now (as a side effect, compilation should now also be a little bit faster). These were mere refactorings, there are no semantic changes to the code.

src/Processors/Merges/Algorithms/AggregatingSortedAlgorithm.h

rschu1ze · 2022-10-21T09:25:44Z

clickhouse-client --query "select 1" coredumps in fasttest. Hm, works nicely for me locally.

rschu1ze · 2022-10-24T06:29:32Z

server.log:

2022.10.20 22:30:52.537137 [ 20411 ] {} <Fatal> BaseDaemon: Stack trace: 0xfa0a22f 0xfa03ef6 0xf972394 0xf97275f 0xb9582d6 0xb957926 0x10de455b 0x10de3981 0xb94706f 0x11043186 0xb93c2f0 0x110571b2 0xb9399a0 0x6fd47f1 0x7ff1ce4c2083 0x6fd3cae
2022.10.20 22:30:52.562551 [ 20411 ] {} <Fatal> BaseDaemon: 3.1. inlined from /fasttest-workspace/build/../../ClickHouse/contrib/libcxx/include/atomic:958: bool std::__1::__cxx_atomic_load[abi:v15003]<bool>(std::__1::__cxx_atomic_base_impl<bool> const*, std::__1::memory_order)
2022.10.20 22:30:52.562598 [ 20411 ] {} <Fatal> BaseDaemon: 3.2. inlined from /fasttest-workspace/build/../../ClickHouse/contrib/libcxx/include/atomic:1560: std::__1::__atomic_base<bool, false>::load[abi:v15003](std::__1::memory_order) const
2022.10.20 22:30:52.562602 [ 20411 ] {} <Fatal> BaseDaemon: 3.3. inlined from /fasttest-workspace/build/../../ClickHouse/contrib/libcxx/include/atomic:1564: std::__1::__atomic_base<bool, false>::operator bool[abi:v15003]() const
2022.10.20 22:30:52.562607 [ 20411 ] {} <Fatal> BaseDaemon: 3.4. inlined from /fasttest-workspace/build/../../ClickHouse/src/Common/ThreadPool.h:228: ~ThreadFromGlobalPoolImpl
2022.10.20 22:30:52.562610 [ 20411 ] {} <Fatal> BaseDaemon: 3. /fasttest-workspace/build/../../ClickHouse/src/Interpreters/ExternalLoader.cpp:1235: DB::ExternalLoader::PeriodicUpdater::enable(bool) @ 0xfa0a22f in /fasttest-workspace/build/programs/clickhouse
2022.10.20 22:30:52.569182 [ 20411 ] {} <Fatal> BaseDaemon: 4.1. inlined from /fasttest-workspace/build/../../ClickHouse/contrib/libcxx/include/__memory/shared_ptr.h:1508: weak_ptr<const DB::Context>
2022.10.20 22:30:52.569207 [ 20411 ] {} <Fatal> BaseDaemon: 4. /fasttest-workspace/build/../../ClickHouse/src/Interpreters/ExternalDictionariesLoader.cpp:26: DB::ExternalDictionariesLoader::ExternalDictionariesLoader(std::__1::shared_ptr<DB::Context const>) @ 0xfa03ef6 in /fasttest-workspace/build/programs/clickhouse
2022.10.20 22:30:52.641362 [ 20411 ] {} <Fatal> BaseDaemon: 5. /fasttest-workspace/build/../../ClickHouse/src/Interpreters/Context.cpp:1552: DB::Context::getExternalDictionariesLoaderUnlocked() @ 0xf972394 in /fasttest-workspace/build/programs/clickhouse
2022.10.20 22:30:52.711380 [ 20411 ] {} <Fatal> BaseDaemon: 6. /fasttest-workspace/build/../../ClickHouse/src/Interpreters/Context.cpp:0: DB::Context::loadOrReloadDictionaries(Poco::Util::AbstractConfiguration const&) @ 0xf97275f in /fasttest-workspace/build/programs/clickhouse
2022.10.20 22:30:52.756343 [ 20411 ] {} <Fatal> BaseDaemon: 7.1. inlined from /fasttest-workspace/build/../../ClickHouse/contrib/poco/Foundation/include/Poco/AutoPtr.h:231: Poco::AutoPtr<Poco::Util::AbstractConfiguration>::operator*()
2022.10.20 22:30:52.756377 [ 20411 ] {} <Fatal> BaseDaemon: 7. /fasttest-workspace/build/../../ClickHouse/programs/server/Server.cpp:1170: DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_4::operator()(Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool) const @ 0xb9582d6 in /fasttest-workspace/build/programs/clickhouse
2022.10.20 22:30:52.801080 [ 20411 ] {} <Fatal> BaseDaemon: 8.1. inlined from /fasttest-workspace/build/../../ClickHouse/contrib/poco/Foundation/include/Poco/AutoPtr.h:90: ~AutoPtr
2022.10.20 22:30:52.801119 [ 20411 ] {} <Fatal> BaseDaemon: 8.2. inlined from /fasttest-workspace/build/../../ClickHouse/contrib/libcxx/include/__functional/invoke.h:394: decltype(std::declval<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_4&>()(std::declval<Poco::AutoPtr<Poco::Util::AbstractConfiguration>>(), std::declval<bool>())) std::__1::__invoke[abi:v15003]<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_4&, Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool>(DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_4&, Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool&&)
2022.10.20 22:30:52.801146 [ 20411 ] {} <Fatal> BaseDaemon: 8.3. inlined from /fasttest-workspace/build/../../ClickHouse/contrib/libcxx/include/__functional/invoke.h:479: void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_4&, Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool>(DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_4&, Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool&&)
2022.10.20 22:30:52.801157 [ 20411 ] {} <Fatal> BaseDaemon: 8.4. inlined from /fasttest-workspace/build/../../ClickHouse/contrib/libcxx/include/__functional/function.h:235: std::__1::__function::__default_alloc_func<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_4, void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>::operator()[abi:v15003](Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool&&)
2022.10.20 22:30:52.801160 [ 20411 ] {} <Fatal> BaseDaemon: 8. /fasttest-workspace/build/../../ClickHouse/contrib/libcxx/include/__functional/function.h:716: void std::__1::__function::__policy_invoker<void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>::__call_impl<std::__1::__function::__default_alloc_func<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_4, void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>>(std::__1::__function::__policy_storage const*, Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool) @ 0xb957926 in /fasttest-workspace/build/programs/clickhouse
2022.10.20 22:30:52.809578 [ 20411 ] {} <Fatal> BaseDaemon: 9.1. inlined from /fasttest-workspace/build/../../ClickHouse/contrib/poco/Foundation/include/Poco/AutoPtr.h:90: ~AutoPtr

I have seen this error also in other PRs.

KochetovNicolai · 2022-10-25T12:23:00Z

src/Processors/Merges/Algorithms/SummingSortedAlgorithm.cpp

    extern const int CORRUPTED_DATA;
 }

-SummingSortedAlgorithm::ColumnsDefinition::ColumnsDefinition() = default;


ColumnsDefinition contains a std::vector<AggregateDescription> member. The vector destructor std::~vector() (called in ColumnsDefinition::~ColumnsDefinition()) does some bookkeeping which requires to know the full definition of AggregateDescription. To make this work, the three defaulted functions are here moved behind the definition of `AggregateDescription'. So I think the change makes sense in general.

The question remains why it used to work before. I see that they made a lot of methods of std::vector in libcxx15 constexpr, including the vector destructor. However, that would only be an explanation if ColumnsDefinition is used in a compile-time/constexpr context but this is not done here. Besides that and nothing else changed in std::vector, I don't really have a good explanation :-/

rschu1ze · 2022-10-27T07:23:02Z

I think this PR is ready-to-merge. Does anyone like to approve? Thanks!

devcrafter

LGTM

robot-clickhouse added pr-not-for-changelog This PR should not be mentioned in the changelog submodule changed At least one submodule changed in this PR. labels Oct 19, 2022

devcrafter self-assigned this Oct 20, 2022

antonio2368 reviewed Oct 20, 2022

View reviewed changes

src/Processors/Merges/Algorithms/AggregatingSortedAlgorithm.h Outdated Show resolved Hide resolved

devcrafter removed their assignment Oct 20, 2022

rschu1ze added 7 commits October 20, 2022 10:52

Build with libcxx(abi) 15

820e6b4

Up-inline ShardPartition, ShardPartitionPiece and TaskCluster

cdae708

Fix static_assert in random generator

f6de964

Un-inline TaskTableAndShard

df318d8

Separate TaskShard and TaskTable

50c932e

Un-clusterf$ck incomplete type errors

f435e4f

Update libcxx submodule

eda3955

rschu1ze force-pushed the update-libcxx-to-15 branch from 88fd21e to eda3955 Compare October 20, 2022 14:51

rschu1ze added 3 commits October 24, 2022 08:29

Merge branch 'master' into update-libcxx-to-15

c119cd2

Fix build

7243e12

Merge branch 'master' into update-libcxx-to-15

240637f

KochetovNicolai reviewed Oct 25, 2022

View reviewed changes

devcrafter self-assigned this Oct 25, 2022

rschu1ze added 2 commits October 25, 2022 21:41

Un-inline stuff

728342e

Fix clang-tidy

e996e2f

devcrafter approved these changes Oct 27, 2022

View reviewed changes

rschu1ze merged commit 31ef6c6 into master Oct 27, 2022

rschu1ze deleted the update-libcxx-to-15 branch October 27, 2022 08:30

azat mentioned this pull request Nov 11, 2022

Build libcxx and libcxxabi from llvm-project #42730

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build with libcxx(abi) 15#42513

Build with libcxx(abi) 15#42513
rschu1ze merged 12 commits intomasterfrom
update-libcxx-to-15

rschu1ze commented Oct 19, 2022 •

edited

Loading

Uh oh!

Uh oh!

rschu1ze commented Oct 21, 2022

Uh oh!

rschu1ze commented Oct 24, 2022

Uh oh!

KochetovNicolai Oct 25, 2022

Uh oh!

rschu1ze Oct 25, 2022

Uh oh!

rschu1ze commented Oct 27, 2022

Uh oh!

devcrafter left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

rschu1ze commented Oct 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Uh oh!

Uh oh!

rschu1ze commented Oct 21, 2022

Uh oh!

rschu1ze commented Oct 24, 2022

Uh oh!

KochetovNicolai Oct 25, 2022

Choose a reason for hiding this comment

Uh oh!

rschu1ze Oct 25, 2022

Choose a reason for hiding this comment

Uh oh!

rschu1ze commented Oct 27, 2022

Uh oh!

devcrafter left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

rschu1ze commented Oct 19, 2022 •

edited

Loading