Skip to content

Commit a58cdfa

Browse files
yariks5syariks5s
authored
Merge branch 'ClickHouse:master' into hive_partitioning_fixes
2 parents 489c608 + 660feb5 commit a58cdfa

File tree

253 files changed

+5186
-5763
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

253 files changed

+5186
-5763
lines changed

‎.github/workflows/auto_releases.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,8 @@ concurrency:
77
group: autoreleases
88

99
on:
10-
# schedule:
11-
# - cron: '0 9 * * *'
10+
schedule:
11+
- cron: '45 11 * * 2,4'
1212
workflow_dispatch:
1313
inputs:
1414
dry-run:
@@ -52,7 +52,7 @@ jobs:
5252
echo 'EOF'
5353
} >> "$GITHUB_OUTPUT"
5454
if [[ "${{ github.event_name }}" == "schedule" ]]; then
55-
echo "DRY_RUN=true" >> "$GITHUB_OUTPUT"
55+
echo "DRY_RUN=false" >> "$GITHUB_OUTPUT"
5656
else
5757
echo "DRY_RUN=${{ github.event.inputs.dry-run }}" >> "$GITHUB_OUTPUT"
5858
fi

‎CHANGELOG.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -18,8 +18,8 @@
1818
### <a id="2412"></a> ClickHouse release 24.12, 2024-12-19
1919

2020
#### Backward Incompatible Change
21-
* Functions `greatest` and `least` now ignore NULL input values, whereas they previously returned NULL if one of the arguments was NULL. For example, `SELECT greatest(1, 2, NULL)` now returns 2. This makes the behavior compatible with PostgreSQL, but at the same time it breaks the compatibility with MySQL which returns NULL. To retain the previous behavior, set setting `least_greatest_legacy_null_behavior` (default: `false`) to `true`. [#65519](https://github.com/ClickHouse/ClickHouse/pull/65519) ([kevinyhzou](https://github.com/KevinyhZou)).
22-
* The legacy MongoDB integration based on the Poco driver has been removed. Server setting `use_legacy_mongodb_integration` is obsolete and has no effect anymore. [#71997](https://github.com/ClickHouse/ClickHouse/pull/71997) ([Kirill Nikiforov](https://github.com/allmazz)). The new integration is strictly more capable and powerful.
21+
* Functions `greatest` and `least` now ignore NULL input values, whereas they previously returned NULL if one of the arguments was NULL. For example, `SELECT greatest(1, 2, NULL)` now returns 2. This makes the behavior compatible with PostgreSQL, but at the same time it breaks the compatibility with MySQL which returns NULL. To retain the previous behavior, set setting `least_greatest_legacy_null_behavior` (default: `false`) to `true`. [#65519](https://github.com/ClickHouse/ClickHouse/pull/65519) [#73344](https://github.com/ClickHouse/ClickHouse/pull/73344) ([kevinyhzou](https://github.com/KevinyhZou)).
22+
* A new MongoDB integration is now the default. Users who like to use the legacy MongoDB driver (based on the Poco driver) can enable server setting `use_legacy_mongodb_integration`. [#73359](https://github.com/ClickHouse/ClickHouse/pull/73359) ([Kirill Nikiforov](https://github.com/allmazz).
2323

2424
#### New Feature
2525
* Move `JSON`/`Dynamic`/`Variant` types from experimental features to beta. [#72294](https://github.com/ClickHouse/ClickHouse/pull/72294) ([Pavel Kruglov](https://github.com/Avogar)). We also backported all fixes as well as this change to 24.11.
@@ -63,7 +63,7 @@
6363
* Remove the `allow_experimental_join_condition` setting, allowing non-equi conditions by default. [#69910](https://github.com/ClickHouse/ClickHouse/pull/69910) ([Vladimir Cherkasov](https://github.com/vdimir)).
6464
* Settings from server config (users.xml) now apply on the client too. Useful for format settings, e.g. `date_time_output_format`. [#71178](https://github.com/ClickHouse/ClickHouse/pull/71178) ([Michael Kolupaev](https://github.com/al13n321)).
6565
* Automatic `GROUP BY`/`ORDER BY` to disk based on the server/user memory usage. Controlled with `max_bytes_ratio_before_external_group_by`/`max_bytes_ratio_before_external_sort` query settings. [#71406](https://github.com/ClickHouse/ClickHouse/pull/71406) ([Azat Khuzhin](https://github.com/azat)).
66-
* Adding a new cancellation logic using a singleton object `CancellationChecker` which checks the timeouts for queries. [#69880](https://github.com/ClickHouse/ClickHouse/pull/69880) ([Yarik Briukhovetskyi](https://github.com/yariks5s)).
66+
* Adding a new cancellation logic: `CancellationChecker` checks timeouts for every started query and stops them once the timeout has reached. [#69880](https://github.com/ClickHouse/ClickHouse/pull/69880) ([Yarik Briukhovetskyi](https://github.com/yariks5s)).
6767
* Support ALTER from `Object` to `JSON`, which means you can easily migrate from the deprecated Object type. [#71784](https://github.com/ClickHouse/ClickHouse/pull/71784) ([Pavel Kruglov](https://github.com/Avogar)).
6868
* Allow unknown values in set that are not present in Enum. Fix [#72662](https://github.com/ClickHouse/ClickHouse/issues/72662). [#72686](https://github.com/ClickHouse/ClickHouse/pull/72686) ([zhanglistar](https://github.com/zhanglistar)).
6969
* Support string search operator (eg., LIKE) for `Enum` data type, implements [#72661](https://github.com/ClickHouse/ClickHouse/issues/72661). [#72732](https://github.com/ClickHouse/ClickHouse/pull/72732) ([zhanglistar](https://github.com/zhanglistar)).

‎CMakeLists.txt‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,9 @@ option(FAIL_ON_UNSUPPORTED_OPTIONS_COMBINATION
77
"Stop/Fail CMake configuration if some ENABLE_XXX option is defined (either ON or OFF)
88
but is not possible to satisfy" ON)
99

10+
# This allows to compile some code conditionally in the private build while having byte-identical source files.
11+
set (CLICKHOUSE_CLOUD 0)
12+
1013
if(FAIL_ON_UNSUPPORTED_OPTIONS_COMBINATION)
1114
set(RECONFIGURE_MESSAGE_LEVEL FATAL_ERROR)
1215
else()
@@ -347,10 +350,11 @@ endif()
347350
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fdiagnostics-absolute-paths")
348351
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fdiagnostics-absolute-paths")
349352

350-
if (NOT ENABLE_TESTS AND NOT SANITIZE AND NOT SANITIZE_COVERAGE AND OS_LINUX)
353+
if (NOT ENABLE_TESTS AND NOT ENABLE_CLANG_TIDY AND NOT SANITIZE AND NOT SANITIZE_COVERAGE AND OS_LINUX)
351354
# https://clang.llvm.org/docs/ThinLTO.html
352355
# Applies to clang and linux only.
353356
# Disabled when building with tests or sanitizers.
357+
# Also disabled with clang-tidy where we don't care about linking
354358
option(ENABLE_THINLTO "Clang-specific link time optimization" ON)
355359
endif()
356360

‎README.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -42,8 +42,10 @@ Every month we get together with the community (users, contributors, customers,
4242
Keep an eye out for upcoming meetups and events around the world. Somewhere else you want us to be? Please feel free to reach out to tyler `<at>` clickhouse `<dot>` com. You can also peruse [ClickHouse Events](https://clickhouse.com/company/news-events) for a list of all upcoming trainings, meetups, speaking engagements, etc.
4343

4444
Upcoming meetups
45-
46-
* [Dubai Meetup](https://www.meetup.com/clickhouse-dubai-meetup-group/events/303096989/) - Feb 3
45+
* [Tokyo Meetup](https://www.meetup.com/clickhouse-tokyo-user-group/events/305126993/) - Jan 23
46+
* [FOSDEM 2025](https://www.fosdem.org/2025/schedule/event/fosdem-2025-6351-how-we-built-a-new-powerful-json-data-type-for-clickhouse/) - Feb 1, 2
47+
* [London Meetup](https://www.meetup.com/clickhouse-london-user-group/events/305146729/) - Feb 5
48+
* [Dubai Meetup](https://www.meetup.com/clickhouse-dubai-meetup-group/events/303096989/) - Feb 10
4749

4850
Recently completed meetups
4951

‎ci/docker/binary-builder/Dockerfile‎

Lines changed: 18 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -7,14 +7,30 @@ ENV CXX=clang++-${LLVM_VERSION}
77
# If the cctools is updated, then first build it in the CI, then update here in a different commit
88
COPY --from=clickhouse/cctools:d9e3596e706b /cctools /cctools
99

10+
# A cross-linker for RISC-V 64 (we need it, because LLVM's LLD does not work):
11+
RUN apt-get update \
12+
&& apt-get install software-properties-common --yes --no-install-recommends --verbose-versions
13+
14+
RUN add-apt-repository ppa:ubuntu-toolchain-r/test --yes \
15+
&& apt-get update \
16+
&& apt-get install --yes \
17+
binutils-riscv64-linux-gnu \
18+
build-essential \
19+
python3-boto3 \
20+
yasm \
21+
zstd \
22+
zip \
23+
&& apt-get clean \
24+
&& rm -rf /var/lib/apt/lists/* /var/cache/debconf /tmp/*
25+
1026
# Rust toolchain and libraries
1127
ENV RUSTUP_HOME=/rust/rustup
1228
ENV CARGO_HOME=/rust/cargo
1329
ENV PATH="/rust/cargo/bin:${PATH}"
1430
RUN curl https://sh.rustup.rs -sSf | bash -s -- -y && \
1531
chmod 777 -R /rust && \
16-
rustup toolchain install nightly-2024-04-01 && \
17-
rustup default nightly-2024-04-01 && \
32+
rustup toolchain install nightly-2024-12-01 && \
33+
rustup default nightly-2024-12-01 && \
1834
rustup toolchain remove stable && \
1935
rustup component add rust-src && \
2036
rustup target add x86_64-unknown-linux-gnu && \
@@ -27,22 +43,6 @@ RUN curl https://sh.rustup.rs -sSf | bash -s -- -y && \
2743
rustup target add aarch64-unknown-linux-musl && \
2844
rustup target add riscv64gc-unknown-linux-gnu
2945

30-
# A cross-linker for RISC-V 64 (we need it, because LLVM's LLD does not work):
31-
RUN apt-get update \
32-
&& apt-get install software-properties-common --yes --no-install-recommends --verbose-versions
33-
34-
RUN add-apt-repository ppa:ubuntu-toolchain-r/test --yes \
35-
&& apt-get update \
36-
&& apt-get install --yes \
37-
binutils-riscv64-linux-gnu \
38-
build-essential \
39-
python3-boto3 \
40-
yasm \
41-
zstd \
42-
zip \
43-
&& apt-get clean \
44-
&& rm -rf /var/lib/apt/lists/* /var/cache/debconf /tmp/*
45-
4646
# Download toolchain and SDK for Darwin
4747
RUN curl -sL -O https://github.com/phracker/MacOSX-SDKs/releases/download/11.3/MacOSX11.0.sdk.tar.xz
4848

‎ci/docker/fasttest/Dockerfile‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ FROM ubuntu:22.04
44
# ARG for quick switch to a given ubuntu mirror
55
ARG apt_archive="http://archive.ubuntu.com"
66
RUN sed -i "s|http://archive.ubuntu.com|$apt_archive|g" /etc/apt/sources.list
7-
ARG LLVM_APT_VERSION="1:19.1.4~*"
7+
ARG LLVM_APT_VERSION="1:19.1.4"
88

99
ENV DEBIAN_FRONTEND=noninteractive LLVM_VERSION=19
1010

@@ -27,7 +27,7 @@ RUN apt-get update \
2727
&& echo "deb https://apt.llvm.org/${CODENAME}/ llvm-toolchain-${CODENAME}-${LLVM_VERSION} main" >> \
2828
/etc/apt/sources.list \
2929
&& apt-get update \
30-
&& apt-get install --yes --no-install-recommends --verbose-versions llvm-${LLVM_VERSION}>=${LLVM_APT_VERSION} \
30+
&& apt-get satisfy --yes --no-install-recommends "llvm-${LLVM_VERSION} (>= ${LLVM_APT_VERSION})" \
3131
&& apt-get clean \
3232
&& rm -rf /var/lib/apt/lists/* /var/cache/debconf /tmp/*
3333

‎ci/docker/stateless-test/Dockerfile‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -17,11 +17,16 @@ RUN addgroup --gid 1001 clickhouse2 && adduser --uid 1001 --gid 1001 --disabled-
1717
# moreutils - provides ts fo FT
1818
# expect, bzip2 - requried by FT
1919
# bsdmainutils - provides hexdump for FT
20+
# git - required in one of FT TC
21+
# gnuplot - for perf tests
22+
# imagemagick - for perf tests
23+
# parallel - for perf tests
2024

2125
# golang version 1.13 on Ubuntu 20 is enough for tests
2226
RUN apt-get update -y \
2327
&& env DEBIAN_FRONTEND=noninteractive \
2428
apt-get install --yes --no-install-recommends \
29+
git \
2530
awscli \
2631
brotli \
2732
lz4 \
@@ -60,6 +65,9 @@ RUN apt-get update -y \
6065
wget \
6166
xz-utils \
6267
ripgrep \
68+
gnuplot \
69+
imagemagick \
70+
parallel \
6371
&& apt-get clean \
6472
&& rm -rf /var/lib/apt/lists/* /var/cache/debconf /tmp/*
6573

@@ -107,12 +115,6 @@ ENV HOME="/home"
107115
ENV TEMP_DIR="/tmp/praktika"
108116
ENV PATH="/wd/tests:/tmp/praktika/input:$PATH"
109117

110-
RUN curl -L --no-verbose -O 'https://archive.apache.org/dist/hadoop/common/hadoop-3.3.1/hadoop-3.3.1.tar.gz' \
111-
&& tar -xvf hadoop-3.3.1.tar.gz \
112-
&& rm -rf hadoop-3.3.1.tar.gz \
113-
&& chmod 777 /hadoop-3.3.1
114-
115-
116118
RUN npm install -g [email protected] \
117119
&& npm install -g tslib && npm install -g node
118120

‎ci/docker/stateless-test/requirements.txt‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,3 +5,9 @@ pandas==1.5.3
55
scipy==1.12.0
66
pyarrow==18.0.0
77
grpcio==1.47.0
8+
protobuf==4.25.3
9+
10+
# performance tests >>
11+
clickhouse_driver
12+
scipy==1.12.0
13+
# << performance tests

‎ci/jobs/build_clickhouse.py‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
import argparse
22
import os
33

4-
from praktika.result import Result
4+
from praktika.result import Result, ResultTranslator
55
from praktika.settings import Settings
66
from praktika.utils import MetaClasses, Shell, Utils
77

@@ -52,9 +52,8 @@ def main():
5252
args = parse_args()
5353

5454
# # for sccache
55-
# os.environ["SCCACHE_BUCKET"] = S3_BUILDS_BUCKET
56-
# os.environ["SCCACHE_S3_KEY_PREFIX"] = "ccache/sccache"
57-
# TODO: check with SCCACHE_LOG=debug SCCACHE_NO_DAEMON=1
55+
os.environ["SCCACHE_BUCKET"] = S3_BUILDS_BUCKET
56+
os.environ["SCCACHE_S3_KEY_PREFIX"] = "ccache/sccache"
5857

5958
stop_watch = Utils.Stopwatch()
6059

@@ -221,6 +220,9 @@ def main():
221220
with_log=False,
222221
)
223222
)
223+
if not results[-1].is_ok():
224+
results[-1].set_files(CIFiles.UNIT_TESTS_BIN)
225+
224226
res = results[-1].is_ok()
225227

226228
Result.create_from(results=results, stopwatch=stop_watch).complete_job()

‎ci/jobs/functional_stateful_tests.py‎

Lines changed: 5 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -57,16 +57,11 @@ def main():
5757
if "/" in to:
5858
batch_num, total_batches = map(int, to.split("/"))
5959

60-
# os.environ["AZURE_CONNECTION_STRING"] = Shell.get_output(
61-
# f"aws ssm get-parameter --region us-east-1 --name azure_connection_string --with-decryption --output text --query Parameter.Value",
62-
# verbose=True,
63-
# strict=True
64-
# )
65-
6660
ch_path = args.ch_path
67-
assert Path(
68-
ch_path + "/clickhouse"
69-
).is_file(), f"clickhouse binary not found under [{ch_path}]"
61+
assert (
62+
Path(ch_path + "/clickhouse").is_file()
63+
or Path(ch_path + "/clickhouse").is_symlink()
64+
), f"clickhouse binary not found under [{ch_path}]"
7065

7166
stop_watch = Utils.Stopwatch()
7267

@@ -95,9 +90,9 @@ def main():
9590
f"ln -sf {ch_path}/clickhouse {ch_path}/clickhouse-client",
9691
f"ln -sf {ch_path}/clickhouse {ch_path}/clickhouse-compressor",
9792
f"ln -sf {ch_path}/clickhouse {ch_path}/clickhouse-local",
93+
f"ln -sf {ch_path}/clickhouse {ch_path}/clickhouse-obfuscator",
9894
f"rm -rf {Settings.TEMP_DIR}/etc/ && mkdir -p {Settings.TEMP_DIR}/etc/clickhouse-client {Settings.TEMP_DIR}/etc/clickhouse-server",
9995
f"cp programs/server/config.xml programs/server/users.xml {Settings.TEMP_DIR}/etc/clickhouse-server/",
100-
# TODO: find a way to work with Azure secret so it's ok for local tests as well, for now keep azure disabled
10196
f"./tests/config/install.sh {Settings.TEMP_DIR}/etc/clickhouse-server {Settings.TEMP_DIR}/etc/clickhouse-client --s3-storage --no-azure",
10297
# clickhouse benchmark segfaults with --config-path, so provide client config by its default location
10398
f"cp {Settings.TEMP_DIR}/etc/clickhouse-client/* /etc/clickhouse-client/",

0 commit comments

Comments
 (0)