Repository navigation
Commit cc86cb3
[fix](build) Fix the BE unit-test and benchmark build on macOS arm64 (#66615)
> Split out of **#66510. That PR
carries the
> whole BE build-time batch and its end-to-end measurements — **please
refer to
> #66510 for the complete benefit numbers**. This PR is the standalone,
> pre-existing engineering-debt part of it: it has no textual or
semantic
> dependency on the rest and can be reviewed and merged on its own.
### What problem does this PR solve?
Related PR: #66510
Problem Summary:
`sh run-be-ut.sh` and `BUILD_BENCHMARK=ON` do not build at all on macOS
arm64,
and digging into it surfaced a bug that affects **every** platform.
#### 1. `-fno-access-control` has never reached `doris_be_test` (all
platforms)
`add_definitions(-D OS_LINUX)` / `(-D OS_MACOSX)` passes `-D` and the
macro name
as two separate argv entries. On the `doris_be_test` target the
flag-injection
order leaves the dangling `-D` immediately before the target's
`COMPILE_FLAGS`,
so clang parses `-D -fno-access-control` as a (rejected) macro
definition and
only warns `macro name must be an identifier`. **The test target has
been
compiled without `-fno-access-control` all along.**
Ten tests compensated with `#define private public`, which breaks any TU
whose
include graph reaches libc++'s `<ranges>`: a macro cannot rewrite a
class's
*default*-private region, so `lazy_split_view`'s member declarations
come out
`redeclared with public access`. simdjson pulls `<ranges>` in via
`segment_iterator.h`, which is why this kept resurfacing.
`e27d10362ac` attributed that failure to `<ranges>`; the actual cause is
the
swallowed flag, and this PR corrects that attribution.
Fix: write `-DOS_LINUX` / `-DOS_MACOSX` so the flag survives, and delete
the
`private`/`protected` `=public` defines from the ten tests — white-box
access
now comes from the (finally effective) `-fno-access-control`, which
coexists
with `<ranges>` fine.
#### 2. Three macOS arm64 link failures in `doris_be_test`
- **tcmalloc branch out of range.** The Debug test binary's `.text`
exceeds
arm64's ±128MB direct-branch reach, and Apple's linker emits no branch
islands
for the prebuilt `libtcmalloc.a`
(`fixup error (kind=arm64_b26) ... B/BL out of range`). `-Og` no longer
keeps
it under the limit. Stop linking `${MALLOCLIB}` into macOS-arm
`MAKE_TEST`
builds — unit tests do not need a custom allocator, and ASAN builds
never
linked it anyway (which is why they never hit this).
- **Unconditionally referenced gperftools symbols.** `HeapAction`'s HTTP
handler
and brpc's `MallocExtension_ReleaseFreeMemory` hint get no-op stubs,
compiled
only under `__APPLE__ && __aarch64__`.
- **Parallel link race.** The APPLE branch links `vector_search_test`
through
`-Wl,-force_load,$<TARGET_FILE:...>`, which creates no target-level
dependency,
so a parallel ninja could reach `doris_be_test`'s link step before the
archive
exists (`library libvector_search_test.a not found`). Added the missing
`add_dependencies`.
`BUILD_BENCHMARK` inflates `.text` past the same ±128MB BL reach, so the
system-malloc fallback and the stub TU are extended to `benchmark_test`
too.
#### 3. Six BE headers drop `<ranges>`
Src-side follow-up to `e27d10362ac`'s test-side cleanup. These headers
only used
range algorithms that `<algorithm>` already provides
(`std::ranges::sort` / `is_sorted` / `find_if` / `any_of` /
`nth_element` /
`max_element`) or trivially rewritable views (`reverse_view` →
`rbegin`/`rend`
loop, `views::values` → structured-binding loop). `object_pool.h` alone
is
included by hundreds of TUs and was paying for the whole `<ranges>`
header for
one reverse loop. **This is the one hunk here with a real compile-time
payoff.**
#### 4. Two test portability fixes, one shell fix
- `file_scanner_v2_test` kept a local anonymous-namespace copy of
`kIcebergPositionDeleteContent` / `kIcebergDeletionVectorContent` after
`iceberg_scan_semantics.h` began exporting the same names into
`namespace
doris`, making unqualified references ambiguous.
- `variant_jsonb_parse_test` constructed `Decimal64` from a bare `long
long`
literal, ambiguous on macOS where `int64_t` is `long long` vs Linux's
`long`
(same family as `e3724df7cb8`).
- `sh run-fe-ut.sh` — the invocation the script's own usage text
documents — died
at parse time with ``syntax error near unexpected token `<' `` before
building
anything: `/bin/sh` is bash, and bash in POSIX mode rejects process
substitution at parse time, so `done < <(find ...)` took the whole
script down.
Regressed in `82646c38c00`. Replaced with a here-string (not a pipe,
which
would put the loop body in a subshell and discard `broken`).
### Release note
None
### Check List (For Author)
- Test
- [x] Unit Test — this PR is what makes `doris_be_test` build and link
on
macOS arm64 in the first place; the existing suites are the test. The
ten
tests that lost `#define private public` still compile and pass, now
relying on the (finally effective) `-fno-access-control`.
- [x] Manual test
- `BUILD_TYPE_UT=Debug sh run-be-ut.sh -j8 --run` builds and links on
macOS 26.5 arm64 / llvm 20 (it did not before this PR).
- `sh run-fe-ut.sh` reaches and passes fe-core's `test-compile`
(1333 test sources, 62 modules, zero errors). The parse fix was
verified under `sh` against the real function: a nested-module report
with failures returns 1 and lists only that module, a clean report
returns 0 with a quoted `failures="7"` inside a stack trace correctly
ignored, and no reports at all returns 0 — the empty input being the
case the here-string could have broken.
- Behavior changed:
- [x] No. Build/test-tooling only; no runtime code path is touched. The
`<ranges>` → `<algorithm>` rewrites are behavior-preserving
(`std::ranges::*` algorithms live in `<algorithm>`; the two view
rewrites
iterate the same elements in the same order).
- Does this need documentation?
- [x] No.
### Proactive disclosure
- The tcmalloc opt-out is deliberately scoped to `OS_MACOSX AND ARCH_ARM
AND
(MAKE_TEST OR BUILD_BENCHMARK)`. Linux CI, and macOS `doris_be` itself,
keep
linking `${MALLOCLIB}` exactly as before — this cannot change allocator
behavior for any shipped binary.
- Verified on macOS arm64 only for the link fixes (they are
macOS-arm-specific by
construction). Items 1, 3 and 4 are cross-platform and want a look from
the
Linux CI lines: item 1 changes how `OS_LINUX` is defined for **every**
target,
and while `-DOS_LINUX` is strictly more correct than `-D OS_LINUX`, it
also
means `-fno-access-control` starts being honoured on Linux
`doris_be_test` too.
---------
Co-authored-by: Claude Fable 5 (1M context) <[email protected]>1 parent 165efa3 commit cc86cb3
21 files changed
Lines changed: 93 additions & 63 deletions
File tree
- be
- src
- common
- exprs/lambda_function
- format/table
- storage/index/inverted/query_v2
- collect
- wand
- test
- exprs
- io/cache
- storage
- iterator
- segment
- testutil
- util/variant
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
45 | 45 | | |
46 | 46 | | |
47 | 47 | | |
48 | | - | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
49 | 52 | | |
50 | 53 | | |
51 | | - | |
| 54 | + | |
52 | 55 | | |
53 | 56 | | |
54 | 57 | | |
| |||
773 | 776 | | |
774 | 777 | | |
775 | 778 | | |
776 | | - | |
| 779 | + | |
| 780 | + | |
| 781 | + | |
| 782 | + | |
| 783 | + | |
| 784 | + | |
| 785 | + | |
| 786 | + | |
| 787 | + | |
| 788 | + | |
| 789 | + | |
| 790 | + | |
777 | 791 | | |
778 | 792 | | |
779 | 793 | | |
| |||
1031 | 1045 | | |
1032 | 1046 | | |
1033 | 1047 | | |
1034 | | - | |
| 1048 | + | |
| 1049 | + | |
| 1050 | + | |
| 1051 | + | |
1035 | 1052 | | |
1036 | 1053 | | |
1037 | 1054 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
21 | | - | |
22 | 21 | | |
23 | 22 | | |
24 | 23 | | |
| |||
53 | 52 | | |
54 | 53 | | |
55 | 54 | | |
56 | | - | |
57 | | - | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
58 | 59 | | |
59 | 60 | | |
60 | 61 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | | - | |
23 | 22 | | |
24 | 23 | | |
25 | 24 | | |
| |||
79 | 78 | | |
80 | 79 | | |
81 | 80 | | |
82 | | - | |
| 81 | + | |
| 82 | + | |
83 | 83 | | |
84 | | - | |
85 | | - | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
86 | 87 | | |
87 | 88 | | |
88 | 89 | | |
| |||
97 | 98 | | |
98 | 99 | | |
99 | 100 | | |
100 | | - | |
101 | | - | |
| 101 | + | |
| 102 | + | |
102 | 103 | | |
103 | 104 | | |
104 | 105 | | |
105 | 106 | | |
106 | | - | |
| 107 | + | |
107 | 108 | | |
108 | 109 | | |
109 | 110 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| 22 | + | |
22 | 23 | | |
23 | 24 | | |
24 | 25 | | |
25 | | - | |
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
25 | | - | |
26 | 25 | | |
27 | 26 | | |
28 | 27 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
36 | | - | |
37 | 36 | | |
38 | 37 | | |
39 | 38 | | |
| |||
66 | 65 | | |
67 | 66 | | |
68 | 67 | | |
69 | | - | |
| 68 | + | |
70 | 69 | | |
71 | 70 | | |
72 | 71 | | |
| |||
76 | 75 | | |
77 | 76 | | |
78 | 77 | | |
79 | | - | |
| 78 | + | |
80 | 79 | | |
81 | 80 | | |
82 | 81 | | |
| |||
85 | 84 | | |
86 | 85 | | |
87 | 86 | | |
88 | | - | |
| 87 | + | |
89 | 88 | | |
90 | 89 | | |
91 | 90 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | | - | |
23 | 22 | | |
24 | 23 | | |
25 | 24 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
164 | 164 | | |
165 | 165 | | |
166 | 166 | | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
167 | 171 | | |
168 | 172 | | |
169 | 173 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
61 | 61 | | |
62 | 62 | | |
63 | 63 | | |
64 | | - | |
65 | 64 | | |
66 | 65 | | |
67 | | - | |
68 | 66 | | |
69 | 67 | | |
70 | 68 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
41 | 41 | | |
42 | 42 | | |
43 | 43 | | |
44 | | - | |
45 | 44 | | |
46 | 45 | | |
47 | | - | |
48 | 46 | | |
49 | 47 | | |
50 | 48 | | |
| |||
0 commit comments