Skip to content

Commit 383be27

Browse files
authored
[fix](build) Fix two arm64 BE build failures: kuromoji_build_dict link and SNII_CRC32C_X86 -Wundef (apache#67451)
### What problem does this PR solve? Issue Number: close apache#67448, close apache#67445 Related PR: apache#64667 (introduced `kuromoji_build_dict`), apache#66052 (introduced the `SNII_CRC32C_X86` test seam), apache#66615 (the macOS arm64 allocator workaround this PR reuses) Problem Summary: Two BE build failures on arm64, both found while building on Apple Silicon. **1. `kuromoji_build_dict` fails to link on macOS arm64 (apache#67448, introduced by apache#64667)** `sh build.sh --be` fails while linking `bin/kuromoji_build_dict`: ``` ld: fixup error (kind=arm64_b26) at '__ZN8tcmallocL14memalign_pagesEmmbb'+0x1DC from libtcmalloc.a[2](libtcmalloc_la-tcmalloc.o), B/BL out of range (displacement=-135695688, max is +/-128MB), from 0x10816FF98 to 0x100007250 ('___clang_call_terminate') __TEXT addr=0x100000000, size=0x008310000 google_malloc addr=0x10816eac0, size=0x0000014fc malloc_hook addr=0x10816ffbc, size=0x0000001dc ``` The offline dictionary converter only calls the kuromoji builder/parser, but those return `Status`, and `Status` reaches `config.cpp` (-> `ExecEnv`), `status.cpp` (-> thrift/protobuf/`BackendOptions`) and `stack_util.cpp`, so the tool's link closure is effectively the whole BE and its `__TEXT` exceeds arm64's +/-128 MB direct-branch reach. Apple's linker lays tcmalloc's custom `google_malloc` / `malloc_hook` sections out after `__text` and cannot insert branch islands there, so the branch from tcmalloc back to `___clang_call_terminate` at the start of `__TEXT` cannot be relaxed. The failed link leaves the four dictionary files ungenerated, the install-time guard fires, and no `output/` is produced. The triage comment on apache#67448 reports the same failure on the `macos-15` runner of the `BE UT (macOS)` workflow. Trimming the tool's link line (the issue's first suggestion) is not possible without refactoring `Status`, so this applies the apache#66615 treatment per target: on macOS arm64 the tool links against the system allocator, and `gperftools_stubs.cpp` supplies the few gperftools symbols that are still referenced unconditionally. `doris_be` keeps tcmalloc, Linux link lines are unchanged, `kuromoji_dict` stays in `ALL` and the install-time file check stays. **2. `SNII_CRC32C_X86` is undefined on non-x86 targets (apache#67445, introduced by apache#66052)** `be/src/storage/index/snii/encoding/crc32c.cpp` tests `SNII_CRC32C_X86` with `#if` but only defines it on x86_64. The BE compiles with `-Wundef -Werror`, so on aarch64 the four `#if` sites fail: ``` crc32c.cpp:105:5: error: 'SNII_CRC32C_X86' is not defined, evaluates to 0 [-Werror,-Wundef] ``` The file is `BE_TEST`-only, so `run-be-ut.sh` cannot build on Apple Silicon or Linux aarch64, while CI never sees it (the only aarch64 workflow builds with `MAKE_TEST=OFF`). Define the flag as `0` on the other branch: the x86 branch is unchanged, and the non-x86 build keeps only the portable slice-by-8 reference path, which the `hw_*` seams already fall back to.
1 parent 224ba5f commit 383be27

3 files changed

Lines changed: 35 additions & 10 deletions

File tree

‎be/CMakeLists.txt‎

Lines changed: 22 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -867,7 +867,8 @@ if ("${CMAKE_BUILD_TYPE}" STREQUAL "DEBUG" OR "${CMAKE_BUILD_TYPE}" STREQUAL "RE
867867
# need a custom allocator; fall back to system malloc here. ASAN
868868
# builds never linked ${MALLOCLIB} anyway, which is why they never
869869
# hit this. gperftools_stubs.cpp supplies the few symbols still
870-
# referenced unconditionally.
870+
# referenced unconditionally. kuromoji_build_dict is a normal-build
871+
# target this branch cannot cover; it opts out per target below.
871872
else()
872873
set(DORIS_LINK_LIBS ${DORIS_LINK_LIBS} ${MALLOCLIB})
873874
endif()
@@ -1100,7 +1101,25 @@ endif()
11001101

11011102
if (NOT MAKE_TEST)
11021103
# Offline generator: compiles the UTF-8 mecab-ipadic source into binary files.
1103-
add_executable(kuromoji_build_dict EXCLUDE_FROM_ALL ${SRC_DIR}/tools/kuromoji_build_dict.cpp)
1104+
set(KUROMOJI_BUILD_DICT_SRCS ${SRC_DIR}/tools/kuromoji_build_dict.cpp)
1105+
set(KUROMOJI_BUILD_DICT_LINK_LIBS ${DORIS_LINK_LIBS})
1106+
if (OS_MACOSX AND ARCH_ARM AND MALLOCLIB STREQUAL "tcmalloc")
1107+
# The tool only calls the dictionary builder/parser, but they return
1108+
# Status, and Status reaches config.cpp (-> ExecEnv), status.cpp
1109+
# (-> thrift/protobuf/BackendOptions) and stack_util.cpp, so the link
1110+
# closure is effectively the whole BE and __TEXT exceeds arm64's
1111+
# +/-128MB direct-branch reach. Apple's linker lays tcmalloc's custom
1112+
# google_malloc/malloc_hook sections out after __text and cannot
1113+
# insert branch islands there ("fixup error (kind=arm64_b26) ... B/BL
1114+
# out of range") -- the failure doris_be_test and benchmark_test hit
1115+
# (see the MALLOCLIB branch above). An offline converter does not need
1116+
# a custom allocator: link it against the system malloc and let
1117+
# gperftools_stubs.cpp supply the few gperftools symbols that are still
1118+
# referenced unconditionally. doris_be keeps tcmalloc.
1119+
list(REMOVE_ITEM KUROMOJI_BUILD_DICT_LINK_LIBS ${MALLOCLIB})
1120+
list(APPEND KUROMOJI_BUILD_DICT_SRCS ${BASE_DIR}/test/testutil/gperftools_stubs.cpp)
1121+
endif()
1122+
add_executable(kuromoji_build_dict EXCLUDE_FROM_ALL ${KUROMOJI_BUILD_DICT_SRCS})
11041123
target_include_directories(kuromoji_build_dict PRIVATE ${PROJECT_SOURCE_DIR}/..)
11051124
pch_reuse(kuromoji_build_dict)
11061125
set_target_properties(kuromoji_build_dict PROPERTIES ENABLE_EXPORTS 1)
@@ -1109,7 +1128,7 @@ if (NOT MAKE_TEST)
11091128
-Wno-implicit-int-conversion
11101129
-Wno-shorten-64-to-32)
11111130
endif()
1112-
target_link_libraries(kuromoji_build_dict ${DORIS_LINK_LIBS})
1131+
target_link_libraries(kuromoji_build_dict ${KUROMOJI_BUILD_DICT_LINK_LIBS})
11131132

11141133
set(KUROMOJI_IPADIC_SRC "${THIRDPARTY_DIR}/share/mecab-ipadic-2.7.0-20250920"
11151134
CACHE PATH "UTF-8 mecab-ipadic source directory used to generate the kuromoji dictionary")

‎be/src/storage/index/snii/encoding/crc32c.cpp‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,10 +34,15 @@
3434
#include <cstddef>
3535
#include <cstring>
3636

37+
// Value-style feature flag: the BE compiles with -Wundef -Werror, so it must be
38+
// defined on every target. Non-x86 builds get 0 and keep only the portable
39+
// slice-by-8 reference path (the hw_* seams then fall back to it).
3740
#if defined(__x86_64__) || defined(_M_X64)
3841
#define SNII_CRC32C_X86 1
3942
#include <cpuid.h> // __get_cpuid, bit_SSE4_2
4043
#include <nmmintrin.h> // _mm_crc32_u8/u32/u64 (SSE4.2)
44+
#else
45+
#define SNII_CRC32C_X86 0
4146
#endif
4247

4348
namespace doris::snii {

‎be/test/testutil/gperftools_stubs.cpp‎

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -15,13 +15,14 @@
1515
// specific language governing permissions and limitations
1616
// under the License.
1717

18-
// doris_be_test on macOS/arm64 links against the system allocator instead of
19-
// tcmalloc: the Debug test binary's .text exceeds arm64's +/-128MB direct
20-
// branch reach and Apple's linker emits no branch islands for the prebuilt
21-
// gperftools archive. These no-op definitions satisfy the few gperftools
22-
// symbols still referenced unconditionally (HeapAction's HTTP handler, which
23-
// no unit test invokes, and brpc's periodic MallocExtension release hint).
24-
// Everywhere else the real libtcmalloc.a provides them and this TU is empty.
18+
// doris_be_test, benchmark_test and the kuromoji_build_dict tool on macOS/arm64
19+
// link against the system allocator instead of tcmalloc: their .text exceeds
20+
// arm64's +/-128MB direct branch reach and Apple's linker emits no branch
21+
// islands for the prebuilt gperftools archive. These no-op definitions satisfy
22+
// the few gperftools symbols still referenced unconditionally (HeapAction's
23+
// HTTP handler, which none of them invokes, and brpc's periodic MallocExtension
24+
// release hint). Everywhere else the real libtcmalloc.a provides them and this
25+
// TU is empty.
2526

2627
#if defined(__APPLE__) && defined(__aarch64__)
2728

0 commit comments

Comments
 (0)