Revert "Revert "Enable JIT compilation for more expressions"" and Support JIT for decimal and bigint types by taiyang-li · Pull Request #73509 · ClickHouse/ClickHouse

taiyang-li · 2024-12-18T02:38:38Z

Changelog category (leave one):

Performance Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Revert "Revert "Enable JIT compilation for more expressions""(#73010) and Support JIT for decimal and bigint types

Documentation entry for user-facing changes

Documentation is written (mandatory for new features)

Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/

CI Settings (Only check the boxes if you know what you are doing):

Allow: All Required Checks
Allow: Stateless tests
Allow: Stateful tests
Allow: Integration Tests
Allow: Performance tests
Allow: All Builds
Allow: batch 1, 2 for multi-batch jobs
Allow: batch 3, 4, 5, 6 for multi-batch jobs

Exclude: Style check
Exclude: Fast test
Exclude: All with ASAN
Exclude: All with TSAN, MSAN, UBSAN, Coverage
Exclude: All with aarch64, release, debug

Run only fuzzers related jobs (libFuzzer fuzzers, AST fuzzers, etc.)
Exclude: AST fuzzers

This reverts commit fb02bec.

robot-ch-test-poll · 2024-12-18T02:42:06Z

This is an automated comment for commit a09780f with description of existing statuses. It's updated for the latest CI running

❌ Click here to open a full report in a separate page

Check name	Description	Status
Builds	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	❌ failure

Successful checks

Check name	Description	Status
Docs check	Builds and tests the documentation	✅ success
Fast test	Normally this is the first check that is ran for a PR. It builds ClickHouse and runs most of stateless functional tests, omitting some. If it fails, further checks are not started until it is fixed. Look at the report to see which tests fail, then reproduce the failure locally as described here	✅ success
Style check	Runs a set of checks to keep the code style clean. If some of tests failed, see the related log from the report	✅ success

alexey-milovidov · 2025-06-15T20:52:49Z

@taiyang-li, continue?

taiyang-li · 2025-06-16T02:17:56Z

@taiyang-li, continue?

I had some issues on making the building success. I had splitted FunctionsConversions.{h,cpp}, but it didn't work. Need your help. Thanks!

clickhouse-gh · 2025-07-08T05:12:34Z

Workflow [PR], commit [30d9e9e]

Summary: ❌
15 failures out of 110 shown:

job_name	test_name	status
Style check		failure
	cpp	failure
Docs check		dropped
Fast test		error
Build (amd_debug)		dropped
Build (amd_release)		dropped
Build (amd_asan)		dropped
Build (amd_tsan)		dropped
Build (amd_msan)		dropped
Build (amd_ubsan)		dropped
Build (amd_binary)		dropped
Build (arm_release)		dropped
Build (arm_asan)		dropped
Build (arm_coverage)		dropped
Build (arm_binary)		dropped
Build (amd_darwin)		dropped

taiyang-li · 2025-07-09T08:19:31Z

Failed building seems not related to this PR: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=73509&sha=6a0d47b559681ab8d4e75cbdef7333d15e344593&name_0=PR&name_1=Build%20%28amd_tidy%29

FAILED: contrib/libcxxabi-cmake/CMakeFiles/cxxabi.dir/__/llvm-project/libcxxabi/src/fallback_malloc.cpp.o 
/usr/bin/sccache /usr/bin/clang++-19 --target=x86_64-linux-gnu --sysroot=/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/cmake/linux/../../contrib/sysroot/linux-x86_64/x86_64-linux-gnu/libc -DHAS_THREAD_LOCAL -DSTD_EXCEPTION_HAS_STACK_TRACE=1 -D_LIBCPP_BUILDING_LIBRARY -D_LIBUNWIND_IS_NATIVE_ONLY -isystem /home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/contrib/llvm-project/libcxxabi/../libcxx/src -isystem /home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/contrib/llvm-project/libcxxabi/../libcxx/include -isystem /home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/contrib/llvm-project/libcxxabi/include -isystem /home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/contrib/llvm-project/libunwind/include --gcc-toolchain=/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/cmake/linux/../../contrib/sysroot/linux-x86_64 -fdiagnostics-color=always -Xclang -fuse-ctor-homing -fsized-deallocation  -UNDEBUG -gdwarf-aranges -pipe -mssse3 -msse4.1 -msse4.2 -mpclmul -mpopcnt -fasynchronous-unwind-tables -falign-functions=32 -mbranches-within-32B-boundaries -ffp-contract=off  -fdiagnostics-absolute-paths -fstrict-vtable-pointers -g -Og -g  -D_LIBCPP_HARDENING_MODE=_LIBCPP_HARDENING_MODE_EXTENSIVE -std=c++23   -D OS_LINUX -w -nostdinc++ -fno-sanitize=undefined -Wno-macro-redefined -MD -MT contrib/libcxxabi-cmake/CMakeFiles/cxxabi.dir/__/llvm-project/libcxxabi/src/fallback_malloc.cpp.o -MF contrib/libcxxabi-cmake/CMakeFiles/cxxabi.dir/__/llvm-project/libcxxabi/src/fallback_malloc.cpp.o.d -o contrib/libcxxabi-cmake/CMakeFiles/cxxabi.dir/__/llvm-project/libcxxabi/src/fallback_malloc.cpp.o -c /home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/contrib/llvm-project/libcxxabi/src/fallback_malloc.cpp
sccache: error: Server startup failed: cache storage failed to read: Unexpected (temporary) at read => send http request

Felixoid · 2025-07-10T22:47:27Z

The issue with sccache server will probably be addressed in #83600

alexey-milovidov · 2025-07-12T12:07:19Z

Fuzzer found signal 8 (floating point exception).

taiyang-li · 2025-07-15T01:31:50Z

Fuzzer found signal 8 (floating point exception).

@alexey-milovidov how to process it when division with zero happens in llvm ir ?

Throw exceptions. It behaves the same as without JIT, but I found it is difficult to capture exceptions throw from generated code.
Return zero. It is easy to implement, but its behavior is not consistent with that without JIT.

alexey-milovidov · 2025-07-15T15:47:51Z

Avoid using JIT for integer divisions (intDiv, modulo).

alexey-milovidov · 2025-07-15T15:49:08Z

Note: I also considered another way with sigsetjmp/siglongjmp before a loop where nothing besides computations happens (no memory allocations, no exceptions) and using it to catch divisions by zero.

taiyang-li · 2025-07-17T06:18:55Z

Note: I also considered another way with sigsetjmp/siglongjmp before a loop where nothing besides computations happens (no memory allocations, no exceptions) and using it to catch divisions by zero.

@alexey-milovidov Thanks for your advice. I'll try this.

clickhouse-gh · 2025-08-19T13:17:22Z

Dear @alexey-milovidov, this PR hasn't been updated for a while. You will be unassigned. Will you continue working on it? If so, please feel free to reassign yourself.

alexey-milovidov · 2025-10-09T08:56:13Z

@taiyang-li, let's continue.

alexey-milovidov · 2025-10-18T01:19:03Z

src/Core/Settings.cpp

 Compile some scalar functions and operators to native code.
 )", 0) \
-    DECLARE(UInt64, min_count_to_compile_expression, 3, R"(
+    DECLARE(UInt64, min_count_to_compile_expression, 0, R"(


Change before merge.

alexey-milovidov · 2025-10-18T01:19:20Z

src/Core/SettingsChangesHistory.cpp

            {"enable_vector_similarity_index", false, false, "Added an alias for setting `allow_experimental_vector_similarity_index`"},
            {"distributed_plan_max_rows_to_broadcast", 20000, 20000, "New experimental setting."},
            {"min_joined_block_size_rows", 0, DEFAULT_BLOCK_SIZE, "New setting."},
+            {"min_count_to_compile_expression", 3, 1, "For ci tests"},


Remove before merge.

alexey-milovidov · 2025-10-18T01:21:35Z

src/Functions/DivisionUtils.h

 template <typename A, typename B>
 inline void throwIfDivisionLeadsToFPE(A a, B b)
 {
+    std::cout << "jit stacktrace:" << StackTrace().toString() << std::endl;


Remove before merge.

alexey-milovidov · 2025-10-18T01:21:47Z

src/Functions/DivisionUtils.h

 }
+}
+
+/*


Remove before merge.

alexey-milovidov · 2025-10-18T01:22:28Z

src/Functions/DivisionUtils.h

+                throw Exception(ErrorCodes::LOGICAL_ERROR, "ModuloImpl compilation expected integer types in (U)Int8|16|32|64|128|256");
+
+            auto * mod = b.GetInsertBlock()->getParent()->getParent();
+            // for (const auto & func : mod->functions())


Remove before merge.

alexey-milovidov · 2025-10-18T01:30:37Z

tests/queries/0_stateless/01518_nullable_aggregate_states2.sql

@@ -1,3 +1,8 @@
+-- When compilation is enabled, the query result is more accurate compared to when compilation is disabled.


Should be checked in another way.

taiyang-li · 2025-10-23T06:13:13Z

@alexey-milovidov sorry, I've been a bit busy lately and may not have more time to devote to it. I'm really grateful that you could continue my previous work.

Continuation of #73509.

taiyang-li added 15 commits December 18, 2024 10:20

Reapply "Enable JIT compilation for more expressions"

07d1f3b

This reverts commit fb02bec.

fix jits for isNotNull and isNull

f2c2c1e

support default implementations for nulls returns nullable type in JIT

e2609ce

remove logs

1c0c902

fix crash

64df5d7

fix jits

df1b68c

support cast compilation

8479f47

fix crashes

08e2d8f

remove useless logs

7680d07

resiter for div3/4

91fcbe7

fix failed uts

7255a49

remove some wrong interfacnes

af5cd85

disable jit for i256

bc36217

register i128 to double

2b5be5f

fix style

e70a8d9

robot-ch-test-poll2 added the pr-feature Pull request with new product feature label Dec 18, 2024

alexey-milovidov added the can be tested Allows running workflows for external contributors label Dec 18, 2024

robot-ch-test-poll1 added pr-performance Pull request with some performance improvements and removed pr-feature Pull request with new product feature labels Dec 18, 2024

taiyang-li added 6 commits December 18, 2024 17:05

fix building

1a036ad

fix buildings of built-in i256

30cd201

enable compile expression for testings

1f19f24

Merge branch 'master' into more_jits_new

0eb2941

Update check-large-objects.sh

41a9fd7

Update ExpressionActions.cpp

aad6ec8

taiyang-li marked this pull request as ready for review December 20, 2024 01:28

gix failed uts

a99d278

This was referenced Dec 20, 2024

Revert "Enable JIT compilation for more expressions" #73010

Merged

Enable JIT compilation for more expressions #70598

Merged

alexey-milovidov self-assigned this May 20, 2025

commit again

fc3def2

taiyang-li force-pushed the more_jits_new branch from 4851d2b to fc3def2 Compare July 8, 2025 05:12

taiyang-li added 2 commits July 8, 2025 14:27

commit again

1b99a18

Merge branch 'master' into more_jits_new

6a0d47b

Merge branch 'ClickHouse:master' into more_jits_new

1bc115b

fix fuzzer tests

30d9e9e

clickhouse-gh bot unassigned alexey-milovidov Aug 19, 2025

alexey-milovidov reviewed Oct 18, 2025

View reviewed changes

src/Functions/DivisionUtils.h

}

}

/*

Copy link
Copy Markdown

Member

alexey-milovidov Oct 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove before merge.

alexey-milovidov reviewed Oct 18, 2025

View reviewed changes

alexey-milovidov mentioned this pull request Oct 18, 2025

Continuation of #73509. #88770

Merged

alexey-milovidov closed this in #88770 Dec 26, 2025

alexey-milovidov added a commit that referenced this pull request Dec 26, 2025

Merge pull request #88770 from ClickHouse/more_jits_new

d5a5829

Continuation of #73509.

               }
+              }
+              /*

		@@ -1,3 +1,8 @@
		-- When compilation is enabled, the query result is more accurate compared to when compilation is disabled.

Conversation

taiyang-li commented Dec 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

CI Settings (Only check the boxes if you know what you are doing):

Uh oh!

robot-ch-test-poll commented Dec 18, 2024 • edited by robot-ch-test-poll3 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexey-milovidov commented Jun 15, 2025

Uh oh!

taiyang-li commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clickhouse-gh bot commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

taiyang-li commented Jul 9, 2025

Uh oh!

Felixoid commented Jul 10, 2025

Uh oh!

alexey-milovidov commented Jul 12, 2025

Uh oh!

taiyang-li commented Jul 15, 2025

Uh oh!

alexey-milovidov commented Jul 15, 2025

Uh oh!

alexey-milovidov commented Jul 15, 2025

Uh oh!

taiyang-li commented Jul 17, 2025

Uh oh!

clickhouse-gh bot commented Aug 19, 2025

Uh oh!

alexey-milovidov commented Oct 9, 2025

Uh oh!

alexey-milovidov Oct 18, 2025

Choose a reason for hiding this comment

Uh oh!

alexey-milovidov Oct 18, 2025

Choose a reason for hiding this comment

Uh oh!

alexey-milovidov Oct 18, 2025

Choose a reason for hiding this comment

Uh oh!

alexey-milovidov Oct 18, 2025

Choose a reason for hiding this comment

Uh oh!

alexey-milovidov Oct 18, 2025

Choose a reason for hiding this comment

Uh oh!

alexey-milovidov Oct 18, 2025

Choose a reason for hiding this comment

Uh oh!

taiyang-li commented Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

taiyang-li commented Dec 18, 2024 •

edited

Loading

robot-ch-test-poll commented Dec 18, 2024 •

edited by robot-ch-test-poll3

Loading

taiyang-li commented Jun 16, 2025 •

edited

Loading

clickhouse-gh bot commented Jul 8, 2025 •

edited

Loading

taiyang-li commented Oct 23, 2025 •

edited

Loading