Switch to LLVM/clang 16 (16.0.3) by azat · Pull Request #49678 · ClickHouse/ClickHouse

azat · 2023-05-08T20:56:47Z

Changelog category (leave one):

Build/Testing/Packaging Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Switch to LLVM/clang 16

Based on: #47709 (cc @alexey-milovidov )
Reverts: #43195
Fixes: #41500 (many thanks to @itrofimow for the patch!)
Closes: #46383 (cc @Felixoid )
Partially reverts: #42457/#40873 (workaround for emitting .debug_aranges), since patch had been merged to upstream) and included into LLVM 16
Blocks: #47783

Required fixes:

Since initial issue got resolved [1]. [1]: google/sanitizers#1540 Signed-off-by: Azat Khuzhin <[email protected]>

Signed-off-by: Azat Khuzhin <[email protected]>

…d-for-llvm-bug" With clang 16 this workaround should not be required anymore. This reverts commit 992729f, reversing changes made to d885b3c.

robot-ch-test-poll · 2023-05-08T20:57:29Z

This is an automated comment for commit e5c4eb3 with description of existing statuses. It's updated for the latest CI running
The full report is available here
The overall status of the commit is 🔴 failure

Check name	Description	Status
AST fuzzer	Runs randomly generated queries to catch program errors. The build type is optionally given in parenthesis. If it fails, ask a maintainer for help	🟢 success
CI running	A meta-check that indicates the running CI. Normally, it's in success or pending state. The failed status indicates some problems with the PR	🟢 success
ClickHouse build check	Builds ClickHouse in various configurations for use in further steps. You have to fix the builds that fail. Build logs often has enough information to fix the error, but you might have to reproduce the failure locally. The cmake options can be found in the build log, grepping for cmake. Use these options and follow the general build process	🟢 success
Compatibility check	Checks that clickhouse binary runs on distributions with old libc versions. If it fails, ask a maintainer for help	🟢 success
Docker image for servers	The check to build and optionally push the mentioned image to docker hub	🟢 success
Docs Check	Builds and tests the documentation	🟢 success
Fast test	Normally this is the first check that is ran for a PR. It builds ClickHouse and runs most of stateless functional tests, omitting some. If it fails, further checks are not started until it is fixed. Look at the report to see which tests fail, then reproduce the failure locally as described here	🟢 success
Flaky tests	Checks if new added or modified tests are flaky by running them repeatedly, in parallel, with more randomization. Functional tests are run 100 times with address sanitizer, and additional randomization of thread scheduling. Integrational tests are run up to 10 times. If at least once a new test has failed, or was too long, this check will be red. We don't allow flaky tests, read the doc	🟢 success
Install packages	Checks that the built packages are installable in a clear environment	🟢 success
Integration tests	The integration tests report. In parenthesis the package type is given, and in square brackets are the optional part/total tests	🔴 failure
Mergeable Check	Checks if all other necessary checks are successful	🟢 success
Performance Comparison	Measure changes in query performance. The performance test report is described in detail here. In square brackets are the optional part/total tests	🟢 success
Push to Dockerhub	The check for building and pushing the CI related docker images to docker hub	🟢 success
SQLancer	Fuzzing tests that detect logical bugs with SQLancer tool	🟢 success
Sqllogic	Run clickhouse on the sqllogic test set against sqlite and checks that all statements are passed	🟢 success
Stateful tests	Runs stateful functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc	🟢 success
Stateless tests	Runs stateless functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc	🔴 failure
Stress test	Runs stateless functional tests concurrently from several clients to detect concurrency-related errors	🟢 success
Style Check	Runs a set of checks to keep the code style clean. If some of tests failed, see the related log from the report	🟢 success
Unit tests	Runs the unit tests for different release types	🟢 success
Upgrade check	Runs stress tests on server version from last release and then tries to upgrade it to the version from the PR. It checks if the new server can successfully startup without any errors, crashes or sanitizer asserts	🟢 success

Otherwise you will get the following error: CMake Error at /usr/lib/llvm-16/lib/cmake/clang/ClangTargets.cmake:792 (message): The imported target "clangBasic" references the file "/usr/lib/llvm-16/lib/libclangBasic.a" but this file does not exist. Possible reasons include: * The file was deleted, renamed, or moved to another location. * An install or uninstall procedure did not complete successfully. * The installation package was faulty and contained "/usr/lib/llvm-16/lib/cmake/clang/ClangTargets.cmake" but not all the files it references. Call Stack (most recent call first): /usr/lib/llvm-16/lib/cmake/clang/ClangConfig.cmake:20 (include) generator/CMakeLists.txt:7 (Find_Package) Signed-off-by: Azat Khuzhin <[email protected]>

Signed-off-by: Azat Khuzhin <[email protected]>

By default woboq uses $version.$minor.$patch, while clang simply uses $version: CMake Error at generator/CMakeLists.txt:77 (message): Could not find any clang builtins headers in /usr/lib/llvm-16/lib/clang/16.0.4/include # ls -d /usr/lib/llvm-16/lib/clang/16/include /usr/lib/llvm-16/lib/clang/16/include Signed-off-by: Azat Khuzhin <[email protected]>

From the release notes [1]: To match GCC, __ppc64__ is no longer defined on PowerPC64 targets. Use __powerpc64__ instead. [1]: https://releases.llvm.org/16.0.0/tools/clang/docs/ReleaseNotes.html Signed-off-by: Azat Khuzhin <[email protected]>

"The -fexperimental-new-pass-manager and -fno-legacy-pass-manager flags have been removed. These flags have been silently ignored since Clang 15." Signed-off-by: Azat Khuzhin <[email protected]>

Signed-off-by: Azat Khuzhin <[email protected]>

…tFormat" Let's try to revert this quirk during upgrading to clang 16. This reverts commit c1e7016.

Conflicts: docker/test/codebrowser/Dockerfile

It is too noisy during upgrade: $ grep error: build_log.log | awk '{print $NF}' | sort | uniq -c 1 [bugprone-standalone-empty,-warnings-as-errors] 1 [misc-misplaced-const,-warnings-as-errors] 2147 [misc-use-anonymous-namespace,-warnings-as-errors] 6 [modernize-loop-convert,-warnings-as-errors] 9 [readability-redundant-string-cstr,-warnings-as-errors] Signed-off-by: Azat Khuzhin <[email protected]>

Signed-off-by: Azat Khuzhin <[email protected]>

CI: https://s3.amazonaws.com/clickhouse-test-reports/49678/6e9985a94d88d49169a48551ba8910c2b373d0de/stateless_tests__debug__[1_5].html Signed-off-by: Azat Khuzhin <[email protected]>

Signed-off-by: Azat Khuzhin <[email protected]>

…more time (sigh) This time, checked locally Signed-off-by: Azat Khuzhin <[email protected]>

NgramDistanceImpl::unrollLowering() relies on the fact that PODArray has padding and it is OK to access more items. Here is an MSan report: ==656==WARNING: MemorySanitizer: use-of-uninitialized-value 0 0x557fd825485f in DB::NgramDistanceImpl<4ul, char8_t, false, true, false>::vectorConstant(DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::PODArray<unsigned long, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, DB::PODArray<float, 4096ul, Allocator<false, false>, 63ul, 64ul>&) (/usr/bin/clickhouse+0x124d885f) (BuildId: 76773125d8739591c75d4f4d263a2ffe7ca96855) 1 0x557fd824eb83 in DB::FunctionsStringSimilarity<DB::NgramDistanceImpl<4ul, char8_t, false, true, false>, DB::NameNgramSearchCaseInsensitive>::executeImpl(std::__1::vector<DB::ColumnWithTypeAndName, std::__1::allocator<DB::ColumnWithTypeAndName>> const&, std::__1::shared_ptr<DB::IDataType const> const&, unsigned long) const (/usr/bin/clickhouse+0x124d2b83) (BuildId: 76773125d8739591c75d4f4d263a2ffe7ca96855) 2 0x557fd50023b7 in DB::FunctionToExecutableFunctionAdaptor::executeImpl() const (/usr/bin/clickhouse+0xf2863b7) (BuildId: 76773125d8739591c75d4f4d263a2ffe7ca96855) Uninitialized value was stored to memory at 0 0x557fd4f8da5a in __msan_memcpy (/usr/bin/clickhouse+0xf211a5a) (BuildId: 76773125d8739591c75d4f4d263a2ffe7ca96855) 1 0x557fd8253803 in DB::NgramDistanceImpl<4ul, char8_t, false, true, false>::vectorConstant(DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, DB::PODArray<unsigned long, 4096ul, Allocator<false, false>, 63ul, 64ul> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, DB::PODArray<float, 4096ul, Allocator<false, false>, 63ul, 64ul>&) (/usr/bin/clickhouse+0x124d7803) (BuildId: 76773125d8739591c75d4f4d263a2ffe7ca96855) 2 0x557fd824eb83 in DB::FunctionsStringSimilarity<DB::NgramDistanceImpl<4ul, char8_t, false, true, false>, DB::NameNgramSearchCaseInsensitive>::executeImpl(std::__1::vector<DB::ColumnWithTypeAndName, std::__1::allocator<DB::ColumnWithTypeAndName>> const&, std::__1::shared_ptr<DB::IDataType const> const&, unsigned long) const (/usr/bin/clickhouse+0x124d2b83) (BuildId: 76773125d8739591c75d4f4d263a2ffe7ca96855) 3 0x557fd50023b7 in DB::FunctionToExecutableFunctionAdaptor::executeImpl() const (/usr/bin/clickhouse+0xf2863b7) (BuildId: 76773125d8739591c75d4f4d263a2ffe7ca96855) Uninitialized value was stored to memory at 0 0x557fd4f8da5a in __msan_memcpy (/usr/bin/clickhouse+0xf211a5a) (BuildId: 76773125d8739591c75d4f4d263a2ffe7ca96855) 1 0x5580061699f5 in detail::memcpySmallAllowReadWriteOverflow15Impl(char*, char const*, long) build_docker/./src/Common/memcpySmall.h:42:13 2 0x5580061699f5 in memcpySmallAllowReadWriteOverflow15(void*, void const*, unsigned long) build_docker/./src/Common/memcpySmall.h:57:5 3 0x5580061699f5 in DB::ColumnString::replicate(DB::PODArray<unsigned long, 4096ul, Allocator<false, false>, 63ul, 64ul> const&) const build_docker/./src/Columns/ColumnString.cpp:462:13 4 0x558005d3fae4 in DB::ColumnConst::convertToFullColumn() const build_docker/./src/Columns/ColumnConst.cpp:48:18 Signed-off-by: Azat Khuzhin <[email protected]>

* u/master: enable used flags's reinit only when the hash talbe rehash Fix build of libfiu on clang-16 fix flaky test 02504_regexp_dictionary_ua_parser fix convertation Fix test that expected CH to apply a wrong optimization ActionsDAG: remove wrong optimization

azat · 2023-05-12T09:16:07Z

Test failures looks unrelated:

Integration tests (release) [2/4] — fail: 2, passed: 545, flaky: 0

E           Code: 117. DB::Exception: Received from 172.16.1.7:9000. DB::Exception: Data directory for table already containing data parts - probably it was unclean DROP table or manual intervention. You must either clear directory by hand or use ATTACH TABLE instead of CREATE TABLE if you need to use that parts.. Stack trace:

Stateless tests (release, analyzer) — fail: 1, passed: 5246, skipped: 17

It shows bunch of NOT_FAILED, but apparently it simply does not count it as an error (since in upstream it has the same - https://s3.amazonaws.com/clickhouse-test-reports/0/fc993f4c07ae3d14e5141d810a22322d0f444949/stateless_tests__release__analyzer_.html and check is green)

But, there is one failure of 00992_system_parts_race_condition_zookeeper_long while this test is flaky - play

rschu1ze · 2023-05-12T11:35:04Z

.gitmodules

-	url = https://github.com/VectorCamp/vectorscan
+	# FIXME: update once upstream fixes will be merged:
+	# - https://github.com/VectorCamp/vectorscan/pull/148
+	# - https://github.com/VectorCamp/vectorscan/pull/149


Just an interesting observation: "cmake/sanitize.cmake" sets -fsanitize-blacklist=${CMAKE_SOURCE_DIR}/tests/msan_suppressions.txt for msan. This file contains line fun:roseRunProgram that suppresses msan warnings in top-level (publicly callable) function "roseRunProgram ()" in rose/program_runtime.c. In other words, it achieves theoretically the same thing as VectorCamp/vectorscan#149.

But the GCC devs agree with your approach, https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61978#c1 😄.

Anyways, once this PR is in, I can try to clean out msan_suppressions.txt, it seems redundant and un-obvious to have an alternative way to suppress msan warnings.

Just an interesting observation: "cmake/sanitize.cmake" sets -fsanitize-blacklist=${CMAKE_SOURCE_DIR}/tests/msan_suppressions.txt for msan. This file contains line fun:roseRunProgram that suppresses msan warnings in top-level (publicly callable) function "roseRunProgram ()" in rose/program_runtime.c. In other words, it achieves theoretically the same thing as VectorCamp/vectorscan#149.

Hm, I totally forgot to look at MSan suppressions, thanks! I want check it again now, need to get full stack trace to compare why it does not work in debug build, and works in release.

There is actually one more fix for this - ClickHouse/boost#12, I tried to fix this in the small_vector, but since clang-16 it was not easy (I wrote about this in the vectorscan PR)

Anyways, once this PR is in, I can try to clean out msan_suppressions.txt, it seems redundant and un-obvious to have an alternative way to suppress msan warnings

Will take a look, but first, I need to check why it didn't work (maybe there is something more...)

Looks like you already did this - #49829, thanks!

But the question why suppressions does not work, it still open.

rschu1ze · 2023-05-12T11:42:51Z

src/Functions/FunctionsStringSimilarity.cpp

        if constexpr (case_insensitive)
        {
+#if defined(MEMORY_SANITIZER)
+            /// Due to PODArray padding accessing more elements should be OK


Speaking of tests/msan_suppressions.txt, it contains another suppression

# We apply std::tolower to uninitialized padding, but don't use the result, so # it is OK. Reproduce with "select ngramDistanceCaseInsensitive(materialize(''), '')" fun:tolower

Looks like msan16 no longer picks these suppressions up.

Cf. ClickHouse#49678 (comment)

Felixoid · 2023-05-12T16:47:01Z

.gitmodules

+	# FIXME: update once upstream fixes will be merged:
+	# - https://github.com/VectorCamp/vectorscan/pull/148
+	# - https://github.com/VectorCamp/vectorscan/pull/149
+	url = https://github.com/azat-ch/vectorscan


Sorry for being a bit late for the party, but there's no need to change the url. It is available all over the forks

VectorCamp/vectorscan@aaca65a

I am on it ...

@Felixoid I don't mind leaving my fork, but a) I guess it is some kind of unspoken rule to use ClickHouse namespace for forks and b) it makes sense most of the time, since if fork will be removed, then you will not be able to download it (though personally I - not going to remove this forks, but still).

I also don't mind too much. I would say the unspoken rule is "reference upstream if possible (= there are no custom patches), otherwise reference a fork in the ClickHouse organization".

Anyways, after making some silly mess, I now

created https://github.com/ClickHouse/vectorscan/

pushed your two fixes to branch ClickHouse/vectorscan/5.4.9 in this forked repo

and referenced that fork + commit from ClickHouse (here).

Your two pushes against vectorscan upstream remain as is (fingers crossed they are merged).

Felixoid · 2023-05-13T06:42:39Z

Unfortunately, there's another leftover https://github.com/ClickHouse/ClickHouse/actions/runs/4964847326/jobs/8885617747#step:6:781

And the report with full logs https://s3.amazonaws.com/clickhouse-test-reports/0/6df457dcaf32317090fc611138b5bd1e1ff5b8ce/woboq_build.html

Here's the last success build with 15 https://s3.amazonaws.com/clickhouse-test-reports/49701/4dcdcf4c11b5604f1c5d3121c9c6fea3e957b605/woboq_build.html

Here is how to launch it locally https://github.com/ClickHouse/ClickHouse/actions/runs/4964847326/jobs/8885617747#step:6:164

woboq codebrowser uses clang tooling, which adds clang system includes (in Linux::AddClangSystemIncludeArgs()), because none of (-nostdinc, -nobuiltininc) is set. And later it will complain with -Wpoison-system-directories for added by itself includes in InitHeaderSearch::AddUnmappedPath(), because they are starts from one of the following: - /usr/include - /usr/local/include The interesting thing here is that it got broken only after upgrading to llvm 16 (in ClickHouse#49678), and the reason for this is that clang 15 build has system includes that does not trigger the warning - "/usr/lib/clang/15.0.7/include", while clang 16 has "/usr/include/clang/16.0.4/include" So let's simply disable this warning, but only for woboq. Signed-off-by: Azat Khuzhin <[email protected]>

alexey-milovidov and others added 6 commits March 18, 2023 23:45

Use LLVM 16

89a76c2

Update one more place

0f7f620

Merge remote-tracking branch 'u/master' into build/llvm-16

c37fe64

Enable back thread fuzzuer under TSan

c525d13

Since initial issue got resolved [1]. [1]: google/sanitizers#1540 Signed-off-by: Azat Khuzhin <[email protected]>

Disable .debug_aranges workaround under ThinLTO for clang/llvm 16

cf987db

Signed-off-by: Azat Khuzhin <[email protected]>

Revert "Merge pull request ClickHouse#43195 from ClickHouse/workaroun…

e278081

…d-for-llvm-bug" With clang 16 this workaround should not be required anymore. This reverts commit 992729f, reversing changes made to d885b3c.

robot-ch-test-poll added the pr-build Pull request with build/testing/packaging improvement label May 8, 2023

azat mentioned this pull request May 8, 2023

Use LLVM 16 #47709

Closed

azat changed the title ~~Switch to LLVM/clang 16~~ Switch to LLVM/clang 16 (16.0.3) May 8, 2023

azat added 3 commits May 9, 2023 10:25

docker: remove symlink for libclang (not required anymore)

08b8b7e

Signed-off-by: Azat Khuzhin <[email protected]>

azat force-pushed the build/llvm-16 branch from 23569c9 to bf17c81 Compare May 9, 2023 08:33

azat changed the title ~~Switch to LLVM/clang 16 (16.0.3)~~ Switch to LLVM/clang 16 (16.0.4) May 9, 2023

azat changed the title ~~Switch to LLVM/clang 16 (16.0.4)~~ Switch to LLVM/clang 16 (16.0.3) May 9, 2023

azat added 2 commits May 9, 2023 12:20

Remove removed in clang 16 -fexperimental-new-pass-manager

6e9985a

"The -fexperimental-new-pass-manager and -fno-legacy-pass-manager flags have been removed. These flags have been silently ignored since Clang 15." Signed-off-by: Azat Khuzhin <[email protected]>

rschu1ze self-assigned this May 9, 2023

This comment was marked as outdated.

Sign in to view

azat added 3 commits May 9, 2023 20:28

Update instructions for building under OSX to clang-16

432250a

Signed-off-by: Azat Khuzhin <[email protected]>

Get back -fno-sanitize-address-use-after-scope quirk for llvm/clang-15

184efcb

Signed-off-by: Azat Khuzhin <[email protected]>

Revert "Suppress clang-analyzer-cplusplus.NewDelete in MsgPackRowInpu…

833652b

…tFormat" Let's try to revert this quirk during upgrading to clang 16. This reverts commit c1e7016.

This comment was marked as outdated.

Sign in to view

azat added 5 commits May 9, 2023 20:40

Merge remote-tracking branch 'u/master' into build/llvm-16

be39e8e

Conflicts: docker/test/codebrowser/Dockerfile

Suppress bugprone-standalone-empty in DatabaseMySQL

051d0e5

Signed-off-by: Azat Khuzhin <[email protected]>

Fix modernize-loop-convert clang-tidy warning

d03ae2a

Signed-off-by: Azat Khuzhin <[email protected]>

Update vectorscan library with fixes for MSan

88b07d1

Signed-off-by: Azat Khuzhin <[email protected]>

robot-ch-test-poll2 added the submodule changed At least one submodule changed in this PR. label May 10, 2023

Merge remote-tracking branch 'u/master' into build/llvm-16

a0cc5b7

azat marked this pull request as draft May 10, 2023 15:47

Update libfiu to fix the build

dde5357

Signed-off-by: Azat Khuzhin <[email protected]>

azat force-pushed the build/llvm-16 branch from 9666bb1 to dde5357 Compare May 10, 2023 17:35

This comment was marked as outdated.

Sign in to view

azat and others added 4 commits May 10, 2023 21:23

Merge branch 'master' into build/llvm-16

d56e142

Merge branch 'master' into build/llvm-16

91ffe8b

Update 02417_opentelemetry_insert_on_distributed_table reference one …

784e5c2

…more time (sigh) This time, checked locally Signed-off-by: Azat Khuzhin <[email protected]>

Fix MsgPackRowInputFormat.cpp build

9db7879

This comment was marked as outdated.

Sign in to view

azat added 2 commits May 11, 2023 16:30

rschu1ze reviewed May 12, 2023

View reviewed changes

rschu1ze marked this pull request as ready for review May 12, 2023 11:47

rschu1ze merged commit b9c185a into ClickHouse:master May 12, 2023

rschu1ze added a commit to rschu1ze/ClickHouse that referenced this pull request May 12, 2023

(wip) What happens if I remove everything in msan_suppressions?

fa3d564

Cf. ClickHouse#49678 (comment)

rschu1ze mentioned this pull request May 12, 2023

What happens if I remove everything in msan_suppressions? #49829

Merged

azat deleted the build/llvm-16 branch May 12, 2023 12:30

Felixoid reviewed May 12, 2023

View reviewed changes

azat mentioned this pull request May 18, 2023

Fix woboq codebrowser build with -Wno-poison-system-directories #49992

Merged

rschu1ze mentioned this pull request May 22, 2023

Woboq broken after Clang 16 upgrade #50077

Closed

rschu1ze mentioned this pull request Aug 25, 2023

Switch CI to Clang 17 #53831

Merged

alexey-milovidov mentioned this pull request Oct 21, 2023

Build base-images with llvm16 google/oss-fuzz#9397

Closed

Conversation

azat commented May 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Uh oh!

robot-ch-test-poll commented May 8, 2023 • edited by robot-ch-test-poll1 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

azat commented May 12, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Felixoid commented May 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

azat commented May 8, 2023 •

edited

Loading

robot-ch-test-poll commented May 8, 2023 •

edited by robot-ch-test-poll1

Loading

Felixoid commented May 13, 2023 •

edited

Loading