Repository navigation
Commit 383be27
authored
[fix](build) Fix two arm64 BE build failures: kuromoji_build_dict link and SNII_CRC32C_X86 -Wundef (apache#67451)
### What problem does this PR solve?
Issue Number: close apache#67448, close apache#67445
Related PR: apache#64667 (introduced `kuromoji_build_dict`), apache#66052
(introduced the `SNII_CRC32C_X86` test seam), apache#66615 (the macOS arm64
allocator workaround this PR reuses)
Problem Summary:
Two BE build failures on arm64, both found while building on Apple
Silicon.
**1. `kuromoji_build_dict` fails to link on macOS arm64 (apache#67448,
introduced by apache#64667)**
`sh build.sh --be` fails while linking `bin/kuromoji_build_dict`:
```
ld: fixup error (kind=arm64_b26) at '__ZN8tcmallocL14memalign_pagesEmmbb'+0x1DC from libtcmalloc.a[2](libtcmalloc_la-tcmalloc.o),
B/BL out of range (displacement=-135695688, max is +/-128MB), from 0x10816FF98 to 0x100007250 ('___clang_call_terminate')
__TEXT addr=0x100000000, size=0x008310000
google_malloc addr=0x10816eac0, size=0x0000014fc
malloc_hook addr=0x10816ffbc, size=0x0000001dc
```
The offline dictionary converter only calls the kuromoji builder/parser,
but those return `Status`, and `Status` reaches `config.cpp` (->
`ExecEnv`), `status.cpp` (-> thrift/protobuf/`BackendOptions`) and
`stack_util.cpp`, so the tool's link closure is effectively the whole BE
and its `__TEXT` exceeds arm64's +/-128 MB direct-branch reach. Apple's
linker lays tcmalloc's custom `google_malloc` / `malloc_hook` sections
out after `__text` and cannot insert branch islands there, so the branch
from tcmalloc back to `___clang_call_terminate` at the start of `__TEXT`
cannot be relaxed. The failed link leaves the four dictionary files
ungenerated, the install-time guard fires, and no `output/` is produced.
The triage comment on apache#67448 reports the same failure on the `macos-15`
runner of the `BE UT (macOS)` workflow.
Trimming the tool's link line (the issue's first suggestion) is not
possible without refactoring `Status`, so this applies the apache#66615
treatment per target: on macOS arm64 the tool links against the system
allocator, and `gperftools_stubs.cpp` supplies the few gperftools
symbols that are still referenced unconditionally. `doris_be` keeps
tcmalloc, Linux link lines are unchanged, `kuromoji_dict` stays in `ALL`
and the install-time file check stays.
**2. `SNII_CRC32C_X86` is undefined on non-x86 targets (apache#67445,
introduced by apache#66052)**
`be/src/storage/index/snii/encoding/crc32c.cpp` tests `SNII_CRC32C_X86`
with `#if` but only defines it on x86_64. The BE compiles with `-Wundef
-Werror`, so on aarch64 the four `#if` sites fail:
```
crc32c.cpp:105:5: error: 'SNII_CRC32C_X86' is not defined, evaluates to 0 [-Werror,-Wundef]
```
The file is `BE_TEST`-only, so `run-be-ut.sh` cannot build on Apple
Silicon or Linux aarch64, while CI never sees it (the only aarch64
workflow builds with `MAKE_TEST=OFF`). Define the flag as `0` on the
other branch: the x86 branch is unchanged, and the non-x86 build keeps
only the portable slice-by-8 reference path, which the `hw_*` seams
already fall back to.1 parent 224ba5f commit 383be27
3 files changed
Lines changed: 35 additions & 10 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
867 | 867 | | |
868 | 868 | | |
869 | 869 | | |
870 | | - | |
| 870 | + | |
| 871 | + | |
871 | 872 | | |
872 | 873 | | |
873 | 874 | | |
| |||
1100 | 1101 | | |
1101 | 1102 | | |
1102 | 1103 | | |
1103 | | - | |
| 1104 | + | |
| 1105 | + | |
| 1106 | + | |
| 1107 | + | |
| 1108 | + | |
| 1109 | + | |
| 1110 | + | |
| 1111 | + | |
| 1112 | + | |
| 1113 | + | |
| 1114 | + | |
| 1115 | + | |
| 1116 | + | |
| 1117 | + | |
| 1118 | + | |
| 1119 | + | |
| 1120 | + | |
| 1121 | + | |
| 1122 | + | |
1104 | 1123 | | |
1105 | 1124 | | |
1106 | 1125 | | |
| |||
1109 | 1128 | | |
1110 | 1129 | | |
1111 | 1130 | | |
1112 | | - | |
| 1131 | + | |
1113 | 1132 | | |
1114 | 1133 | | |
1115 | 1134 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
37 | 40 | | |
38 | 41 | | |
39 | 42 | | |
40 | 43 | | |
| 44 | + | |
| 45 | + | |
41 | 46 | | |
42 | 47 | | |
43 | 48 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
18 | | - | |
19 | | - | |
20 | | - | |
21 | | - | |
22 | | - | |
23 | | - | |
24 | | - | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
25 | 26 | | |
26 | 27 | | |
27 | 28 | | |
| |||
0 commit comments