Bump `llvm-project` to 18.x by nikitamikhaylov · Pull Request #66053 · ClickHouse/ClickHouse

nikitamikhaylov · 2024-07-03T14:42:33Z

Currently llvm-project submodule points to a commit from this branch ClickHouse/llvm-project#44
Previous attempt was here #47783

Changelog category (leave one):

Build/Testing/Packaging Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

The internal dependency LLVM is bumped from 16 to 18.

Documentation entry for user-facing changes

Documentation is written (mandatory for new features)

Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/

CI Settings (Only check the boxes if you know what you are doing):

Allow: All Required Checks
Allow: Stateless tests
Allow: Stateful tests
Allow: Integration Tests
Allow: Performance tests
Allow: All Builds
Allow: batch 1, 2 for multi-batch jobs
Allow: batch 3, 4, 5, 6 for multi-batch jobs

Exclude: Style check
Exclude: Fast test
Exclude: All with ASAN
Exclude: All with TSAN, MSAN, UBSAN, Coverage
Exclude: All with aarch64, release, debug

robot-ch-test-poll3 · 2024-07-03T14:50:00Z

This is an automated comment for commit 5363ee9 with description of existing statuses. It's updated for the latest CI running

❌ Click here to open a full report in a separate page

Check name	Description	Status
Integration tests	The integration tests report. In parenthesis the package type is given, and in square brackets are the optional part/total tests	❌ failure
Stateless tests	Runs stateless functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc	❌ failure

Successful checks

Check name	Description	Status
AST fuzzer	Runs randomly generated queries to catch program errors. The build type is optionally given in parenthesis. If it fails, ask a maintainer for help	✅ success
Builds	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
ClickBench	Runs [ClickBench](https://github.com/ClickHouse/ClickBench/) with instant-attach table	✅ success
Compatibility check	Checks that clickhouse binary runs on distributions with old libc versions. If it fails, ask a maintainer for help	✅ success
Docker keeper image	The check to build and optionally push the mentioned image to docker hub	✅ success
Docker server image	The check to build and optionally push the mentioned image to docker hub	✅ success
Docs check	Builds and tests the documentation	✅ success
Fast test	Normally this is the first check that is ran for a PR. It builds ClickHouse and runs most of stateless functional tests, omitting some. If it fails, further checks are not started until it is fixed. Look at the report to see which tests fail, then reproduce the failure locally as described here	✅ success
Flaky tests	Checks if new added or modified tests are flaky by running them repeatedly, in parallel, with more randomization. Functional tests are run 100 times with address sanitizer, and additional randomization of thread scheduling. Integration tests are run up to 10 times. If at least once a new test has failed, or was too long, this check will be red. We don't allow flaky tests, read the doc	✅ success
Install packages	Checks that the built packages are installable in a clear environment	✅ success
Performance Comparison	Measure changes in query performance. The performance test report is described in detail here. In square brackets are the optional part/total tests	✅ success
Stateful tests	Runs stateful functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc	✅ success
Stress test	Runs stateless functional tests concurrently from several clients to detect concurrency-related errors	✅ success
Style check	Runs a set of checks to keep the code style clean. If some of tests failed, see the related log from the report	✅ success
Unit tests	Runs the unit tests for different release types	✅ success
Upgrade check	Runs stress tests on server version from last release and then tries to upgrade it to the version from the PR. It checks if the new server can successfully startup without any errors, crashes or sanitizer asserts	✅ success

nikitamikhaylov · 2024-07-08T20:32:01Z

Notes:

The llvm::Optional class was removed from LLVM codebase in this commit llvm/llvm-project@397f2e9. I just replaced it with std::optional where it was used in our codebase and looks like everything is Ok.
I've also submitted a PR to the upstream of a pocketfft lib Add missing include mreineck/pocketfft#18

Two things remain unsolved:

We need to patch RocksDB to make in compilable with new libcxx. Hopefully it will be done as a part of Bump RocksDB #66216
We need to rework our JIT implementation. Most of the headers we including don't exist right now CC @kitaisreal

clickhouse-gh · 2024-09-10T13:11:16Z

Dear @nikitamikhaylov, this PR hasn't been updated for a while. Will you continue working on it? If not, please close it. Otherwise, ignore this message.

clickhouse-gh · 2024-11-12T13:12:36Z

Dear @nikitamikhaylov, this PR hasn't been updated for a while. Will you continue working on it? If not, please close it. Otherwise, ignore this message.

nikitamikhaylov · 2024-12-20T20:38:22Z

Depends on: #69654

nikitamikhaylov · 2024-12-21T23:22:11Z

Performance comparision reports: 1, 2, 3, 4
Slowndowns: jit_large_requests - 87%, float_formatting - 21%, jit_small_requests - 49%, schema_inference_text_formats - 15%
Speedups: json_type - 20%, parse_illegal_datetime - 40%.

Results more or less the same as for LLVM-16. Slowdown in float formatting is concerning.

(cherry picked from commit b08310d)

Algunenano · 2024-12-23T11:54:22Z

The JIT slowdown is noticeable but not a blocker. The float_formatting only affects small integers (integers have their own code pass) so we should investigate it at some point, but not a blocker either. Let's update

nikitamikhaylov · 2024-12-23T17:08:51Z

I've checked and there were some other failures of the test_many_mutations in another PRs. I will override the check for now and merge.

…e#66053 since ch bump llvm to 18, which cause building CurrentMemoryTracker.h is failed, I remove two unecessary commits: - Kyligence/ClickHouse#488 - Kyligence/ClickHouse@1922234

* [GLUTEN-1632][CH]Daily Update Clickhouse Version (20241224) * Fix Build due to ClickHouse/ClickHouse#69654 and ClickHouse/ClickHouse#66053 since ch bump llvm to 18, which cause building CurrentMemoryTracker.h is failed, I remove two unecessary commits: - Kyligence/ClickHouse#488 - Kyligence/ClickHouse@1922234 --------- Co-authored-by: kyligence-git <[email protected]> Co-authored-by: Chang Chen <[email protected]>

taiyang-li · 2024-12-26T03:27:11Z

The JIT slowdown is noticeable but not a blocker. The float_formatting only affects small integers (integers have their own code pass) so we should investigate it at some point, but not a blocker either. Let's update

@Algunenano Do you known why performance is slower after upgrade llvm from 18.x. I found similar performance regressions in Gluten with JIT enabled.

Algunenano · 2024-12-26T12:24:54Z

AFAIK we haven't investigated neither of the slowdowns.

rschu1ze · 2024-12-29T19:56:26Z

Let's see if #73957 helps.

robot-clickhouse-ci-1 added pr-build Pull request with build/testing/packaging improvement submodule changed At least one submodule changed in this PR. labels Jul 3, 2024

rschu1ze mentioned this pull request Jul 3, 2024

Clean-up custom LLVM 15 patches #66072

Merged

nikitamikhaylov force-pushed the bump-llvm-18 branch 2 times, most recently from 0c4e7a7 to c896f0b Compare July 8, 2024 16:14

nikitamikhaylov force-pushed the bump-llvm-18 branch 3 times, most recently from efea439 to fd10698 Compare July 11, 2024 13:52

rschu1ze mentioned this pull request Jul 14, 2024

Add string/string_view spaceship operators and add option to enable <regex> ClickHouse/llvm-project#42

Closed

rschu1ze mentioned this pull request Sep 16, 2024

Bump llvm to 16.x #69654

Merged

21 tasks

nikitamikhaylov force-pushed the bump-llvm-18 branch 4 times, most recently from b08310d to 43caaea Compare December 21, 2024 17:31

nikitamikhaylov marked this pull request as ready for review December 21, 2024 23:38

nikitamikhaylov and others added 9 commits December 23, 2024 10:58

Bump

35514c8

Squashed commit

7df46a6

Fix build

14c6940

(cherry picked from commit b08310d)

Bump submodule

748cd1b

Bump submodule [2]

6d010c7

Fix FreeBSD build

3f33597

Fix ASAN failure

95cb582

Misc fixes

c0ff129

Change hardening more to extensive

222c751

nikitamikhaylov and others added 5 commits December 23, 2024 10:58

Fix more container overflows

3ba63fa

Corrections

14ebc48

Style

4911d67

Style [2]

8e47e1a

Bump

92656aa

nikitamikhaylov force-pushed the bump-llvm-18 branch from bb6d733 to 92656aa Compare December 23, 2024 11:01

Cleanup code and cmake

5363ee9

Algunenano self-assigned this Dec 23, 2024

Algunenano approved these changes Dec 23, 2024

View reviewed changes

nikitamikhaylov added this pull request to the merge queue Dec 23, 2024

Merged via the queue into master with commit 11a1a6d Dec 23, 2024

nikitamikhaylov deleted the bump-llvm-18 branch December 23, 2024 17:52

robot-clickhouse-ci-2 added the pr-synced-to-cloud The PR is synced to the cloud repo label Dec 23, 2024

rschu1ze mentioned this pull request Dec 29, 2024

Experiment: Enable SLP vectorizer for faster codegen #73957

Merged

zhanglistar mentioned this pull request Jan 3, 2025

Experiment: Add more pass for faster codegen #74122

Closed

23 tasks

nikitamikhaylov mentioned this pull request Feb 18, 2025

Assertion getOperand(0)->getType() == getOperand(1)->getType() && "Both operands to ICmp instruction are not of the same type!"' failed` #72936

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bump `llvm-project` to 18.x#66053

Bump `llvm-project` to 18.x#66053
nikitamikhaylov merged 15 commits intomasterfrom
bump-llvm-18

nikitamikhaylov commented Jul 3, 2024 •

edited

Loading

Uh oh!

robot-ch-test-poll3 commented Jul 3, 2024 •

edited by robot-clickhouse

Loading

Uh oh!

nikitamikhaylov commented Jul 8, 2024 •

edited

Loading

Uh oh!

clickhouse-gh bot commented Sep 10, 2024

Uh oh!

clickhouse-gh bot commented Nov 12, 2024

Uh oh!

nikitamikhaylov commented Dec 20, 2024

Uh oh!

nikitamikhaylov commented Dec 21, 2024 •

edited by alexey-milovidov

Loading

Uh oh!

Algunenano commented Dec 23, 2024

Uh oh!

nikitamikhaylov commented Dec 23, 2024

Uh oh!

taiyang-li commented Dec 26, 2024

Uh oh!

Algunenano commented Dec 26, 2024

Uh oh!

rschu1ze commented Dec 29, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

nikitamikhaylov commented Jul 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

CI Settings (Only check the boxes if you know what you are doing):

Uh oh!

robot-ch-test-poll3 commented Jul 3, 2024 • edited by robot-clickhouse Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nikitamikhaylov commented Jul 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clickhouse-gh bot commented Sep 10, 2024

Uh oh!

clickhouse-gh bot commented Nov 12, 2024

Uh oh!

nikitamikhaylov commented Dec 20, 2024

Uh oh!

nikitamikhaylov commented Dec 21, 2024 • edited by alexey-milovidov Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Algunenano commented Dec 23, 2024

Uh oh!

nikitamikhaylov commented Dec 23, 2024

Uh oh!

taiyang-li commented Dec 26, 2024

Uh oh!

Algunenano commented Dec 26, 2024

Uh oh!

rschu1ze commented Dec 29, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

nikitamikhaylov commented Jul 3, 2024 •

edited

Loading

robot-ch-test-poll3 commented Jul 3, 2024 •

edited by robot-clickhouse

Loading

nikitamikhaylov commented Jul 8, 2024 •

edited

Loading

nikitamikhaylov commented Dec 21, 2024 •

edited by alexey-milovidov

Loading