Skip to content

Switch to LLVM/clang 16 (16.0.3) #49678

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rschu1ze merged 33 commits into from
May 12, 2023
Merged

Switch to LLVM/clang 16 (16.0.3) #49678

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
89a76c2
Use LLVM 16
alexey-milovidov Mar 18, 2023
0f7f620
Update one more place
alexey-milovidov Mar 19, 2023
c37fe64
Merge remote-tracking branch 'u/master' into build/llvm-16
azat May 8, 2023
c525d13
Enable back thread fuzzuer under TSan
azat May 8, 2023
cf987db
Disable .debug_aranges workaround under ThinLTO for clang/llvm 16
azat May 8, 2023
e278081
Revert "Merge pull request #43195 from ClickHouse/workaround-for-llvm…
azat May 8, 2023
82536fe
docker: install libclang-16-dev for woboq code browser
azat May 9, 2023
08b8b7e
docker: remove symlink for libclang (not required anymore)
azat May 9, 2023
bf17c81
docker: fix path to clang builtin headers for woboq code browser
azat May 9, 2023
a794177
Use __powerpc64__ over __ppc64__ (clang 16 stops declaring __ppc64__)
azat May 9, 2023
6e9985a
Remove removed in clang 16 -fexperimental-new-pass-manager
azat May 9, 2023
432250a
Update instructions for building under OSX to clang-16
azat May 9, 2023
184efcb
Get back -fno-sanitize-address-use-after-scope quirk for llvm/clang-15
azat May 9, 2023
833652b
Revert "Suppress clang-analyzer-cplusplus.NewDelete in MsgPackRowInpu…
azat May 9, 2023
be39e8e
Merge remote-tracking branch 'u/master' into build/llvm-16
azat May 9, 2023
e1d7f46
Disable misc-use-anonymous-namespace check in clang-tidy
azat May 9, 2023
051d0e5
Suppress bugprone-standalone-empty in DatabaseMySQL
azat May 9, 2023
fa4099b
Update 02417_opentelemetry_insert_on_distributed_table for clang 16
azat May 9, 2023
d03ae2a
Fix modernize-loop-convert clang-tidy warning
azat May 9, 2023
0610865
Fix readability-redundant-string-cstr clang-tidy warning
azat May 9, 2023
d8dd50a
Fix misc-misplaced-const clang-tidy warning
azat May 9, 2023
9b3bcd0
Fix stack-use-after-scope in gtest_resource_manager_static
azat May 9, 2023
a15d088
Fix 02417_opentelemetry_insert_on_distributed_table (wrong sed was used)
azat May 10, 2023
2f9a7cb
Fix stack-use-after-scope in KafkaProducer::produce()
azat May 10, 2023
88b07d1
Update vectorscan library with fixes for MSan
azat May 10, 2023
a0cc5b7
Merge remote-tracking branch 'u/master' into build/llvm-16
azat May 10, 2023
dde5357
Update libfiu to fix the build
azat May 10, 2023
d56e142
Merge branch 'master' into build/llvm-16
azat May 10, 2023
91ffe8b
Merge branch 'master' into build/llvm-16
alexey-milovidov May 11, 2023
784e5c2
Update 02417_opentelemetry_insert_on_distributed_table reference one …
azat May 11, 2023
9db7879
Fix MsgPackRowInputFormat.cpp build
rschu1ze May 11, 2023
00fdfa1
Suppress MSan warning in NgramDistanceImpl::unrollLowering()
azat May 11, 2023
e5c4eb3
Merge remote-tracking branch 'u/master' into build/llvm-16
azat May 11, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .clang-tidy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,7 @@ Checks: '*,
-misc-no-recursion,
-misc-non-private-member-variables-in-classes,
-misc-confusable-identifiers, # useful but slooow
-misc-use-anonymous-namespace,

-modernize-avoid-c-arrays,
-modernize-concat-nested-namespaces,
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/nightly.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,8 +125,8 @@ jobs:
SONAR_SCANNER_VERSION: 4.8.0.2856
SONAR_SERVER_URL: "https://sonarcloud.io"
BUILD_WRAPPER_OUT_DIR: build_wrapper_output_directory # Directory where build-wrapper output will be placed
CC: clang-15
CXX: clang++-15
CC: clang-16
CXX: clang++-16
steps:
- name: Check out repository code
uses: ClickHouse/checkout@v1
Expand Down
5 changes: 4 additions & 1 deletion .gitmodules
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -267,7 +267,10 @@
url = https://github.com/ClickHouse/nats.c
[submodule "contrib/vectorscan"]
path = contrib/vectorscan
url = https://github.com/VectorCamp/vectorscan
# FIXME: update once upstream fixes will be merged:
# - https://github.com/VectorCamp/vectorscan/pull/148
# - https://github.com/VectorCamp/vectorscan/pull/149
Copy link
Copy Markdown
Member

@rschu1ze rschu1ze May 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just an interesting observation: "cmake/sanitize.cmake" sets -fsanitize-blacklist=${CMAKE_SOURCE_DIR}/tests/msan_suppressions.txt for msan. This file contains line fun:roseRunProgram that suppresses msan warnings in top-level (publicly callable) function "roseRunProgram ()" in rose/program_runtime.c. In other words, it achieves theoretically the same thing as VectorCamp/vectorscan#149.

But the GCC devs agree with your approach, https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61978#c1 😄.

Anyways, once this PR is in, I can try to clean out msan_suppressions.txt, it seems redundant and un-obvious to have an alternative way to suppress msan warnings.

Copy link
Copy Markdown
Member Author

@azat azat May 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just an interesting observation: "cmake/sanitize.cmake" sets -fsanitize-blacklist=${CMAKE_SOURCE_DIR}/tests/msan_suppressions.txt for msan. This file contains line fun:roseRunProgram that suppresses msan warnings in top-level (publicly callable) function "roseRunProgram ()" in rose/program_runtime.c. In other words, it achieves theoretically the same thing as VectorCamp/vectorscan#149.

Hm, I totally forgot to look at MSan suppressions, thanks! I want check it again now, need to get full stack trace to compare why it does not work in debug build, and works in release.

There is actually one more fix for this - ClickHouse/boost#12, I tried to fix this in the small_vector, but since clang-16 it was not easy (I wrote about this in the vectorscan PR)

Anyways, once this PR is in, I can try to clean out msan_suppressions.txt, it seems redundant and un-obvious to have an alternative way to suppress msan warnings

Will take a look, but first, I need to check why it didn't work (maybe there is something more...)

Copy link
Copy Markdown
Member Author

@azat azat May 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like you already did this - #49829, thanks!

But the question why suppressions does not work, it still open.

url = https://github.com/azat-ch/vectorscan
Copy link
Copy Markdown
Member

@Felixoid Felixoid May 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for being a bit late for the party, but there's no need to change the url. It is available all over the forks

VectorCamp/vectorscan@aaca65a

Copy link
Copy Markdown
Member

@rschu1ze rschu1ze May 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am on it ...

Copy link
Copy Markdown
Member Author

@azat azat May 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Felixoid I don't mind leaving my fork, but a) I guess it is some kind of unspoken rule to use ClickHouse namespace for forks and b) it makes sense most of the time, since if fork will be removed, then you will not be able to download it (though personally I - not going to remove this forks, but still).

Copy link
Copy Markdown
Member

@rschu1ze rschu1ze May 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also don't mind too much. I would say the unspoken rule is "reference upstream if possible (= there are no custom patches), otherwise reference a fork in the ClickHouse organization".

Anyways, after making some silly mess, I now

Your two pushes against vectorscan upstream remain as is (fingers crossed they are merged).

[submodule "contrib/c-ares"]
path = contrib/c-ares
url = https://github.com/ClickHouse/c-ares
Expand Down
7 changes: 0 additions & 7 deletions CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -342,13 +342,6 @@ if (COMPILER_CLANG)

set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fstrict-vtable-pointers")

if (CMAKE_CXX_COMPILER_VERSION VERSION_LESS 16)
# Set new experimental pass manager, it's a performance, build time and binary size win.
# Can be removed after https://reviews.llvm.org/D66490 merged and released to at least two versions of clang.
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fexperimental-new-pass-manager")
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fexperimental-new-pass-manager")
endif ()

# We cannot afford to use LTO when compiling unit tests, and it's not enough
# to only supply -fno-lto at the final linking stage. So we disable it
# completely.
Expand Down
13 changes: 10 additions & 3 deletions cmake/sanitize.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,16 @@ set (SAN_FLAGS "${SAN_FLAGS} -g -fno-omit-frame-pointer -DSANITIZER")

if (SANITIZE)
if (SANITIZE STREQUAL "address")
# LLVM-15 has a bug in Address Sanitizer, preventing the usage of 'sanitize-address-use-after-scope',
# see https://github.com/llvm/llvm-project/issues/58633
set (ASAN_FLAGS "-fsanitize=address -fno-sanitize-address-use-after-scope")
set (ASAN_FLAGS "-fsanitize=address -fsanitize-address-use-after-scope")
if (COMPILER_CLANG)
if (${CMAKE_CXX_COMPILER_VERSION} VERSION_GREATER_EQUAL 15 AND ${CMAKE_CXX_COMPILER_VERSION} VERSION_LESS 16)
# LLVM-15 has a bug in Address Sanitizer, preventing the usage
# of 'sanitize-address-use-after-scope', see [1].
#
# [1]: https://github.com/llvm/llvm-project/issues/58633
set (ASAN_FLAGS "${ASAN_FLAGS} -fno-sanitize-address-use-after-scope")
endif()
endif()
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${SAN_FLAGS} ${ASAN_FLAGS}")
set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} ${SAN_FLAGS} ${ASAN_FLAGS}")

Expand Down
8 changes: 5 additions & 3 deletions cmake/tools.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,13 +70,15 @@ if (LINKER_NAME)
if (NOT LLD_PATH)
message (FATAL_ERROR "Using linker ${LINKER_NAME} but can't find its path.")
endif ()
if (COMPILER_CLANG)
# This a temporary quirk to emit .debug_aranges with ThinLTO, can be removed after upgrade to clang-16
# This a temporary quirk to emit .debug_aranges with ThinLTO, it is only the case clang/llvm <16
if (COMPILER_CLANG AND CMAKE_CXX_COMPILER_VERSION VERSION_LESS 16)
set (LLD_WRAPPER "${CMAKE_CURRENT_BINARY_DIR}/ld.lld")
configure_file ("${CMAKE_CURRENT_SOURCE_DIR}/cmake/ld.lld.in" "${LLD_WRAPPER}" @ONLY)

set (CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} --ld-path=${LLD_WRAPPER}")
endif ()
else ()
set (CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} --ld-path=${LLD_PATH}")
endif()

endif ()

Expand Down
2 changes: 1 addition & 1 deletion contrib/vectorscan
Open in desktop
Submodule vectorscan updated 2 files
+1 −1 src/rose/program_runtime.c
+16 −2 src/util/small_vector.h
19 changes: 9 additions & 10 deletions docker/packager/packager
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -362,17 +362,16 @@ def parse_args() -> argparse.Namespace:
parser.add_argument(
"--compiler",
choices=(
"clang-15",
"clang-15-darwin",
"clang-15-darwin-aarch64",
"clang-15-aarch64",
"clang-15-aarch64-v80compat",
"clang-15-ppc64le",
"clang-15-amd64-compat",
"clang-15-freebsd",
"gcc-11",
"clang-16",
"clang-16-darwin",
"clang-16-darwin-aarch64",
"clang-16-aarch64",
"clang-16-aarch64-v80compat",
"clang-16-ppc64le",
"clang-16-amd64-compat",
"clang-16-freebsd",
),
default="clang-15",
default="clang-16",
help="a compiler to use",
)
parser.add_argument(
Expand Down
19 changes: 2 additions & 17 deletions docker/test/codebrowser/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,35 +10,20 @@ RUN sed -i "s|http://archive.ubuntu.com|$apt_archive|g" /etc/apt/sources.list

RUN apt-get update && apt-get --yes --allow-unauthenticated install libclang-${LLVM_VERSION}-dev libmlir-${LLVM_VERSION}-dev

# libclang-15-dev does not contain proper symlink:
#
# This is what cmake will search for:
#
# # readlink -f /usr/lib/llvm-15/lib/libclang-15.so.1
# /usr/lib/x86_64-linux-gnu/libclang-15.so.1
#
# This is what exists:
#
# # ls -l /usr/lib/x86_64-linux-gnu/libclang-15*
# lrwxrwxrwx 1 root root 16 Sep 5 13:31 /usr/lib/x86_64-linux-gnu/libclang-15.so -> libclang-15.so.1
# lrwxrwxrwx 1 root root 21 Sep 5 13:31 /usr/lib/x86_64-linux-gnu/libclang-15.so.15 -> libclang-15.so.15.0.0
# -rw-r--r-- 1 root root 31835760 Sep 5 13:31 /usr/lib/x86_64-linux-gnu/libclang-15.so.15.0.0
#
ARG TARGETARCH
RUN arch=${TARGETARCH:-amd64} \
&& case $arch in \
amd64) rarch=x86_64 ;; \
arm64) rarch=aarch64 ;; \
*) exit 1 ;; \
esac \
&& ln -rsf /usr/lib/$rarch-linux-gnu/libclang-15.so.15 /usr/lib/$rarch-linux-gnu/libclang-15.so.1
esac

# repo versions doesn't work correctly with C++17
# also we push reports to s3, so we add index.html to subfolder urls
# https://github.com/ClickHouse-Extras/woboq_codebrowser/commit/37e15eaf377b920acb0b48dbe82471be9203f76b
RUN git clone --depth=1 https://github.com/ClickHouse/woboq_codebrowser /woboq_codebrowser \
&& cd /woboq_codebrowser \
&& cmake . -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COMPILER=clang\+\+-${LLVM_VERSION} -DCMAKE_C_COMPILER=clang-${LLVM_VERSION} \
&& cmake . -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COMPILER=clang\+\+-${LLVM_VERSION} -DCMAKE_C_COMPILER=clang-${LLVM_VERSION} -DCLANG_BUILTIN_HEADERS_DIR=/usr/lib/llvm-${LLVM_VERSION}/lib/clang/${LLVM_VERSION}/include \
&& ninja

ENV CODEGEN=/woboq_codebrowser/generator/codebrowser_generator
Expand Down
2 changes: 1 addition & 1 deletion docker/test/fasttest/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ trap 'kill $(jobs -pr) ||:' EXIT
stage=${stage:-}

# Compiler version, normally set by Dockerfile
export LLVM_VERSION=${LLVM_VERSION:-13}
export LLVM_VERSION=${LLVM_VERSION:-16}

# A variable to pass additional flags to CMake.
# Here we explicitly default it to nothing so that bash doesn't complain about
Expand Down
2 changes: 1 addition & 1 deletion docker/test/fuzzer/run-fuzzer.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ stage=${stage:-}
script_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
echo "$script_dir"
repo_dir=ch
BINARY_TO_DOWNLOAD=${BINARY_TO_DOWNLOAD:="clang-15_debug_none_unsplitted_disable_False_binary"}
BINARY_TO_DOWNLOAD=${BINARY_TO_DOWNLOAD:="clang-16_debug_none_unsplitted_disable_False_binary"}
BINARY_URL_TO_DOWNLOAD=${BINARY_URL_TO_DOWNLOAD:="https://clickhouse-builds.s3.amazonaws.com/$PR_TO_TEST/$SHA_TO_TEST/clickhouse_build_check/$BINARY_TO_DOWNLOAD/clickhouse"}

function git_clone_with_retry
Expand Down
2 changes: 1 addition & 1 deletion docker/test/keeper-jepsen/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
set -euo pipefail


CLICKHOUSE_PACKAGE=${CLICKHOUSE_PACKAGE:="https://clickhouse-builds.s3.amazonaws.com/$PR_TO_TEST/$SHA_TO_TEST/clickhouse_build_check/clang-15_relwithdebuginfo_none_unsplitted_disable_False_binary/clickhouse"}
CLICKHOUSE_PACKAGE=${CLICKHOUSE_PACKAGE:="https://clickhouse-builds.s3.amazonaws.com/$PR_TO_TEST/$SHA_TO_TEST/clickhouse_build_check/clang-16_relwithdebuginfo_none_unsplitted_disable_False_binary/clickhouse"}
CLICKHOUSE_REPO_PATH=${CLICKHOUSE_REPO_PATH:=""}


Expand Down
2 changes: 1 addition & 1 deletion docker/test/server-jepsen/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
set -euo pipefail


CLICKHOUSE_PACKAGE=${CLICKHOUSE_PACKAGE:="https://clickhouse-builds.s3.amazonaws.com/$PR_TO_TEST/$SHA_TO_TEST/clickhouse_build_check/clang-15_relwithdebuginfo_none_unsplitted_disable_False_binary/clickhouse"}
CLICKHOUSE_PACKAGE=${CLICKHOUSE_PACKAGE:="https://clickhouse-builds.s3.amazonaws.com/$PR_TO_TEST/$SHA_TO_TEST/clickhouse_build_check/clang-16_relwithdebuginfo_none_unsplitted_disable_False_binary/clickhouse"}
CLICKHOUSE_REPO_PATH=${CLICKHOUSE_REPO_PATH:=""}


Expand Down
46 changes: 21 additions & 25 deletions docker/test/stress/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,31 +20,27 @@ install_packages package_folder

# Thread Fuzzer allows to check more permutations of possible thread scheduling
# and find more potential issues.
# Temporarily disable ThreadFuzzer with tsan because of https://github.com/google/sanitizers/issues/1540
is_tsan_build=$(clickhouse local -q "select value like '% -fsanitize=thread %' from system.build_options where name='CXX_FLAGS'")
if [ "$is_tsan_build" -eq "0" ]; then
export THREAD_FUZZER_CPU_TIME_PERIOD_US=1000
export THREAD_FUZZER_SLEEP_PROBABILITY=0.1
export THREAD_FUZZER_SLEEP_TIME_US=100000

export THREAD_FUZZER_pthread_mutex_lock_BEFORE_MIGRATE_PROBABILITY=1
export THREAD_FUZZER_pthread_mutex_lock_AFTER_MIGRATE_PROBABILITY=1
export THREAD_FUZZER_pthread_mutex_unlock_BEFORE_MIGRATE_PROBABILITY=1
export THREAD_FUZZER_pthread_mutex_unlock_AFTER_MIGRATE_PROBABILITY=1

export THREAD_FUZZER_pthread_mutex_lock_BEFORE_SLEEP_PROBABILITY=0.001
export THREAD_FUZZER_pthread_mutex_lock_AFTER_SLEEP_PROBABILITY=0.001
export THREAD_FUZZER_pthread_mutex_unlock_BEFORE_SLEEP_PROBABILITY=0.001
export THREAD_FUZZER_pthread_mutex_unlock_AFTER_SLEEP_PROBABILITY=0.001
export THREAD_FUZZER_pthread_mutex_lock_BEFORE_SLEEP_TIME_US=10000

export THREAD_FUZZER_pthread_mutex_lock_AFTER_SLEEP_TIME_US=10000
export THREAD_FUZZER_pthread_mutex_unlock_BEFORE_SLEEP_TIME_US=10000
export THREAD_FUZZER_pthread_mutex_unlock_AFTER_SLEEP_TIME_US=10000

export THREAD_FUZZER_EXPLICIT_SLEEP_PROBABILITY=0.01
export THREAD_FUZZER_EXPLICIT_MEMORY_EXCEPTION_PROBABILITY=0.01
fi
export THREAD_FUZZER_CPU_TIME_PERIOD_US=1000
export THREAD_FUZZER_SLEEP_PROBABILITY=0.1
export THREAD_FUZZER_SLEEP_TIME_US=100000

export THREAD_FUZZER_pthread_mutex_lock_BEFORE_MIGRATE_PROBABILITY=1
export THREAD_FUZZER_pthread_mutex_lock_AFTER_MIGRATE_PROBABILITY=1
export THREAD_FUZZER_pthread_mutex_unlock_BEFORE_MIGRATE_PROBABILITY=1
export THREAD_FUZZER_pthread_mutex_unlock_AFTER_MIGRATE_PROBABILITY=1

export THREAD_FUZZER_pthread_mutex_lock_BEFORE_SLEEP_PROBABILITY=0.001
export THREAD_FUZZER_pthread_mutex_lock_AFTER_SLEEP_PROBABILITY=0.001
export THREAD_FUZZER_pthread_mutex_unlock_BEFORE_SLEEP_PROBABILITY=0.001
export THREAD_FUZZER_pthread_mutex_unlock_AFTER_SLEEP_PROBABILITY=0.001
export THREAD_FUZZER_pthread_mutex_lock_BEFORE_SLEEP_TIME_US=10000

export THREAD_FUZZER_pthread_mutex_lock_AFTER_SLEEP_TIME_US=10000
export THREAD_FUZZER_pthread_mutex_unlock_BEFORE_SLEEP_TIME_US=10000
export THREAD_FUZZER_pthread_mutex_unlock_AFTER_SLEEP_TIME_US=10000

export THREAD_FUZZER_EXPLICIT_SLEEP_PROBABILITY=0.01
export THREAD_FUZZER_EXPLICIT_MEMORY_EXCEPTION_PROBABILITY=0.01

export ZOOKEEPER_FAULT_INJECTION=1
# Initial run without S3 to create system.*_log on local file system to make it
Expand Down
3 changes: 2 additions & 1 deletion docker/test/util/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ ARG apt_archive="http://archive.ubuntu.com"
RUN sed -i "s|http://archive.ubuntu.com|$apt_archive|g" /etc/apt/sources.list

# 15.0.2
ENV DEBIAN_FRONTEND=noninteractive LLVM_VERSION=15
ENV DEBIAN_FRONTEND=noninteractive LLVM_VERSION=16

RUN apt-get update \
&& apt-get install \
Expand Down Expand Up @@ -52,6 +52,7 @@ RUN apt-get update \
lld-${LLVM_VERSION} \
llvm-${LLVM_VERSION} \
llvm-${LLVM_VERSION}-dev \
libclang-${LLVM_VERSION}-dev \
moreutils \
nasm \
ninja-build \
Expand Down
8 changes: 4 additions & 4 deletions docs/en/development/build-cross-osx.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,14 @@ This is intended for continuous integration checks that run on Linux servers. If

The cross-build for macOS is based on the [Build instructions](../development/build.md), follow them first.

## Install Clang-15
## Install Clang-16

Follow the instructions from https://apt.llvm.org/ for your Ubuntu or Debian setup.
For example the commands for Bionic are like:

``` bash
sudo echo "deb [trusted=yes] http://apt.llvm.org/bionic/ llvm-toolchain-bionic-15 main" >> /etc/apt/sources.list
sudo apt-get install clang-15
sudo echo "deb [trusted=yes] http://apt.llvm.org/bionic/ llvm-toolchain-bionic-16 main" >> /etc/apt/sources.list
sudo apt-get install clang-16
```

## Install Cross-Compilation Toolset {#install-cross-compilation-toolset}
Expand Down Expand Up @@ -55,7 +55,7 @@ curl -L 'https://github.com/phracker/MacOSX-SDKs/releases/download/10.15/MacOSX1
cd ClickHouse
mkdir build-darwin
cd build-darwin
CC=clang-15 CXX=clang++-15 cmake -DCMAKE_AR:FILEPATH=${CCTOOLS}/bin/x86_64-apple-darwin-ar -DCMAKE_INSTALL_NAME_TOOL=${CCTOOLS}/bin/x86_64-apple-darwin-install_name_tool -DCMAKE_RANLIB:FILEPATH=${CCTOOLS}/bin/x86_64-apple-darwin-ranlib -DLINKER_NAME=${CCTOOLS}/bin/x86_64-apple-darwin-ld -DCMAKE_TOOLCHAIN_FILE=cmake/darwin/toolchain-x86_64.cmake ..
CC=clang-16 CXX=clang++-16 cmake -DCMAKE_AR:FILEPATH=${CCTOOLS}/bin/x86_64-apple-darwin-ar -DCMAKE_INSTALL_NAME_TOOL=${CCTOOLS}/bin/x86_64-apple-darwin-install_name_tool -DCMAKE_RANLIB:FILEPATH=${CCTOOLS}/bin/x86_64-apple-darwin-ranlib -DLINKER_NAME=${CCTOOLS}/bin/x86_64-apple-darwin-ld -DCMAKE_TOOLCHAIN_FILE=cmake/darwin/toolchain-x86_64.cmake ..
ninja
```

Expand Down
4 changes: 2 additions & 2 deletions docs/en/development/build.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,8 @@ GCC as a compiler is not supported
To build with a specific Clang version:

``` bash
export CC=clang-15
export CXX=clang++-15
export CC=clang-16
export CXX=clang++-16
```

### Checkout ClickHouse Sources {#checkout-clickhouse-sources}
Expand Down
2 changes: 1 addition & 1 deletion docs/en/development/continuous-integration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ Builds ClickHouse in various configurations for use in further steps. You have t

### Report Details

- **Compiler**: `clang-15`, optionally with the name of a target platform
- **Compiler**: `clang-16`, optionally with the name of a target platform
- **Build type**: `Debug` or `RelWithDebInfo` (cmake).
- **Sanitizer**: `none` (without sanitizers), `address` (ASan), `memory` (MSan), `undefined` (UBSan), or `thread` (TSan).
- **Status**: `success` or `fail`
Expand Down
2 changes: 1 addition & 1 deletion docs/en/development/developer-instruction.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@ While inside the `build` directory, configure your build by running CMake. Befor
export CC=clang CXX=clang++
cmake ..

If you installed clang using the automatic installation script above, also specify the version of clang installed in the first command, e.g. `export CC=clang-15 CXX=clang++-15`. The clang version will be in the script output.
If you installed clang using the automatic installation script above, also specify the version of clang installed in the first command, e.g. `export CC=clang-16 CXX=clang++-16`. The clang version will be in the script output.

The `CC` variable specifies the compiler for C (short for C Compiler), and `CXX` variable instructs which C++ compiler is to be used for building.

Expand Down
2 changes: 1 addition & 1 deletion src/Common/atomicRename.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ namespace ErrorCodes
#define __NR_renameat2 316
#elif defined(__aarch64__)
#define __NR_renameat2 276
#elif defined(__ppc64__)
#elif defined(__powerpc64__)
#define __NR_renameat2 357
#elif defined(__riscv)
#define __NR_renameat2 276
Expand Down
2 changes: 1 addition & 1 deletion src/Common/waitForPid.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ enum PollPidResult
#define SYS_pidfd_open 434
#elif defined(__aarch64__)
#define SYS_pidfd_open 434
#elif defined(__ppc64__)
#elif defined(__powerpc64__)
#define SYS_pidfd_open 434
#elif defined(__riscv)
#define SYS_pidfd_open 434
Expand Down
4 changes: 2 additions & 2 deletions src/DataTypes/Serializations/SerializationArray.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,9 @@ void SerializationArray::serializeBinary(const Field & field, WriteBuffer & ostr
{
const Array & a = field.get<const Array &>();
writeVarUInt(a.size(), ostr);
for (size_t i = 0; i < a.size(); ++i)
for (const auto & i : a)
{
nested->serializeBinary(a[i], ostr, settings);
nested->serializeBinary(i, ostr, settings);
}
}

Expand Down
2 changes: 1 addition & 1 deletion src/Databases/MySQL/DatabaseMySQL.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ DatabaseMySQL::DatabaseMySQL(
try
{
/// Test that the database is working fine; it will also fetch tables.
empty();
empty(); // NOLINT(bugprone-standalone-empty)
}
catch (...)
{
Expand Down
2 changes: 1 addition & 1 deletion src/Disks/IO/ThreadPoolReader.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@
#define SYS_preadv2 327
#elif defined(__aarch64__)
#define SYS_preadv2 286
#elif defined(__ppc64__)
#elif defined(__powerpc64__)
#define SYS_preadv2 380
#elif defined(__riscv)
#define SYS_preadv2 286
Expand Down
4 changes: 4 additions & 0 deletions src/Functions/FunctionsStringSimilarity.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,10 @@ struct NgramDistanceImpl

if constexpr (case_insensitive)
{
#if defined(MEMORY_SANITIZER)
/// Due to PODArray padding accessing more elements should be OK
Copy link
Copy Markdown
Member

@rschu1ze rschu1ze May 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Speaking of tests/msan_suppressions.txt, it contains another suppression

# We apply std::tolower to uninitialized padding, but don't use the result, so        
# it is OK. Reproduce with "select ngramDistanceCaseInsensitive(materialize(''), '')" 
fun:tolower

Looks like msan16 no longer picks these suppressions up.

__msan_unpoison(code_points + (N - 1), padding_offset * sizeof(CodePoint));
#endif
/// We really need template lambdas with C++20 to do it inline
unrollLowering<N - 1>(code_points, std::make_index_sequence<padding_offset>());
}
Expand Down
Loading