Skip to content

Commit cc86cb3

Browse files
morningmanclaude
andauthored
[fix](build) Fix the BE unit-test and benchmark build on macOS arm64 (#66615)
> Split out of **#66510. That PR carries the > whole BE build-time batch and its end-to-end measurements — **please refer to > #66510 for the complete benefit numbers**. This PR is the standalone, > pre-existing engineering-debt part of it: it has no textual or semantic > dependency on the rest and can be reviewed and merged on its own. ### What problem does this PR solve? Related PR: #66510 Problem Summary: `sh run-be-ut.sh` and `BUILD_BENCHMARK=ON` do not build at all on macOS arm64, and digging into it surfaced a bug that affects **every** platform. #### 1. `-fno-access-control` has never reached `doris_be_test` (all platforms) `add_definitions(-D OS_LINUX)` / `(-D OS_MACOSX)` passes `-D` and the macro name as two separate argv entries. On the `doris_be_test` target the flag-injection order leaves the dangling `-D` immediately before the target's `COMPILE_FLAGS`, so clang parses `-D -fno-access-control` as a (rejected) macro definition and only warns `macro name must be an identifier`. **The test target has been compiled without `-fno-access-control` all along.** Ten tests compensated with `#define private public`, which breaks any TU whose include graph reaches libc++'s `<ranges>`: a macro cannot rewrite a class's *default*-private region, so `lazy_split_view`'s member declarations come out `redeclared with public access`. simdjson pulls `<ranges>` in via `segment_iterator.h`, which is why this kept resurfacing. `e27d10362ac` attributed that failure to `<ranges>`; the actual cause is the swallowed flag, and this PR corrects that attribution. Fix: write `-DOS_LINUX` / `-DOS_MACOSX` so the flag survives, and delete the `private`/`protected` `=public` defines from the ten tests — white-box access now comes from the (finally effective) `-fno-access-control`, which coexists with `<ranges>` fine. #### 2. Three macOS arm64 link failures in `doris_be_test` - **tcmalloc branch out of range.** The Debug test binary's `.text` exceeds arm64's ±128MB direct-branch reach, and Apple's linker emits no branch islands for the prebuilt `libtcmalloc.a` (`fixup error (kind=arm64_b26) ... B/BL out of range`). `-Og` no longer keeps it under the limit. Stop linking `${MALLOCLIB}` into macOS-arm `MAKE_TEST` builds — unit tests do not need a custom allocator, and ASAN builds never linked it anyway (which is why they never hit this). - **Unconditionally referenced gperftools symbols.** `HeapAction`'s HTTP handler and brpc's `MallocExtension_ReleaseFreeMemory` hint get no-op stubs, compiled only under `__APPLE__ && __aarch64__`. - **Parallel link race.** The APPLE branch links `vector_search_test` through `-Wl,-force_load,$<TARGET_FILE:...>`, which creates no target-level dependency, so a parallel ninja could reach `doris_be_test`'s link step before the archive exists (`library libvector_search_test.a not found`). Added the missing `add_dependencies`. `BUILD_BENCHMARK` inflates `.text` past the same ±128MB BL reach, so the system-malloc fallback and the stub TU are extended to `benchmark_test` too. #### 3. Six BE headers drop `<ranges>` Src-side follow-up to `e27d10362ac`'s test-side cleanup. These headers only used range algorithms that `<algorithm>` already provides (`std::ranges::sort` / `is_sorted` / `find_if` / `any_of` / `nth_element` / `max_element`) or trivially rewritable views (`reverse_view` → `rbegin`/`rend` loop, `views::values` → structured-binding loop). `object_pool.h` alone is included by hundreds of TUs and was paying for the whole `<ranges>` header for one reverse loop. **This is the one hunk here with a real compile-time payoff.** #### 4. Two test portability fixes, one shell fix - `file_scanner_v2_test` kept a local anonymous-namespace copy of `kIcebergPositionDeleteContent` / `kIcebergDeletionVectorContent` after `iceberg_scan_semantics.h` began exporting the same names into `namespace doris`, making unqualified references ambiguous. - `variant_jsonb_parse_test` constructed `Decimal64` from a bare `long long` literal, ambiguous on macOS where `int64_t` is `long long` vs Linux's `long` (same family as `e3724df7cb8`). - `sh run-fe-ut.sh` — the invocation the script's own usage text documents — died at parse time with ``syntax error near unexpected token `<' `` before building anything: `/bin/sh` is bash, and bash in POSIX mode rejects process substitution at parse time, so `done < <(find ...)` took the whole script down. Regressed in `82646c38c00`. Replaced with a here-string (not a pipe, which would put the loop body in a subshell and discard `broken`). ### Release note None ### Check List (For Author) - Test - [x] Unit Test — this PR is what makes `doris_be_test` build and link on macOS arm64 in the first place; the existing suites are the test. The ten tests that lost `#define private public` still compile and pass, now relying on the (finally effective) `-fno-access-control`. - [x] Manual test - `BUILD_TYPE_UT=Debug sh run-be-ut.sh -j8 --run` builds and links on macOS 26.5 arm64 / llvm 20 (it did not before this PR). - `sh run-fe-ut.sh` reaches and passes fe-core's `test-compile` (1333 test sources, 62 modules, zero errors). The parse fix was verified under `sh` against the real function: a nested-module report with failures returns 1 and lists only that module, a clean report returns 0 with a quoted `failures="7"` inside a stack trace correctly ignored, and no reports at all returns 0 — the empty input being the case the here-string could have broken. - Behavior changed: - [x] No. Build/test-tooling only; no runtime code path is touched. The `<ranges>` → `<algorithm>` rewrites are behavior-preserving (`std::ranges::*` algorithms live in `<algorithm>`; the two view rewrites iterate the same elements in the same order). - Does this need documentation? - [x] No. ### Proactive disclosure - The tcmalloc opt-out is deliberately scoped to `OS_MACOSX AND ARCH_ARM AND (MAKE_TEST OR BUILD_BENCHMARK)`. Linux CI, and macOS `doris_be` itself, keep linking `${MALLOCLIB}` exactly as before — this cannot change allocator behavior for any shipped binary. - Verified on macOS arm64 only for the link fixes (they are macOS-arm-specific by construction). Items 1, 3 and 4 are cross-platform and want a look from the Linux CI lines: item 1 changes how `OS_LINUX` is defined for **every** target, and while `-DOS_LINUX` is strictly more correct than `-D OS_LINUX`, it also means `-fno-access-control` starts being honoured on Linux `doris_be_test` too. --------- Co-authored-by: Claude Fable 5 (1M context) <[email protected]>
1 parent 165efa3 commit cc86cb3

21 files changed

Lines changed: 93 additions & 63 deletions

‎be/CMakeLists.txt‎

Lines changed: 21 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -45,10 +45,13 @@ endif ()
4545

4646
if (CMAKE_SYSTEM_NAME MATCHES "Linux")
4747
set (OS_LINUX 1)
48-
add_definitions(-D OS_LINUX)
48+
# No space after -D: "-D OS_LINUX" becomes two argv entries in some
49+
# generators, and the dangling "-D" then swallows the next flag
50+
# (doris_be_test lost its -fno-access-control this way).
51+
add_definitions(-DOS_LINUX)
4952
elseif (CMAKE_SYSTEM_NAME MATCHES "Darwin")
5053
set (OS_MACOSX 1)
51-
add_definitions(-D OS_MACOSX)
54+
add_definitions(-DOS_MACOSX)
5255
endif ()
5356

5457
if (OS_MACOSX)
@@ -773,7 +776,18 @@ endif ()
773776

774777
# Add sanitize static link flags
775778
if ("${CMAKE_BUILD_TYPE}" STREQUAL "DEBUG" OR "${CMAKE_BUILD_TYPE}" STREQUAL "RELEASE")
776-
set(DORIS_LINK_LIBS ${DORIS_LINK_LIBS} ${MALLOCLIB})
779+
if (OS_MACOSX AND ARCH_ARM AND (MAKE_TEST OR BUILD_BENCHMARK))
780+
# doris_be_test's and benchmark_test's .text exceeds arm64's +/-128MB
781+
# BL reach (BE_TEST/BE_BENCHMARK macros inflate it further) and Apple's
782+
# linker emits no branch islands for tcmalloc's prebuilt archive
783+
# ("fixup error ... B/BL out of range"). Tests and benchmarks don't
784+
# need a custom allocator; fall back to system malloc here. ASAN
785+
# builds never linked ${MALLOCLIB} anyway, which is why they never
786+
# hit this. gperftools_stubs.cpp supplies the few symbols still
787+
# referenced unconditionally.
788+
else()
789+
set(DORIS_LINK_LIBS ${DORIS_LINK_LIBS} ${MALLOCLIB})
790+
endif()
777791
elseif ("${CMAKE_BUILD_TYPE}" STREQUAL "ASAN" OR "${CMAKE_BUILD_TYPE}" STREQUAL "ASAN_UT")
778792
set(DORIS_LINK_LIBS ${DORIS_LINK_LIBS} ${ASAN_LIBS})
779793
if (OS_LINUX)
@@ -1031,7 +1045,10 @@ if (BUILD_BENCHMARK)
10311045
if (NOT ${CMAKE_BUILD_TYPE} STREQUAL "RELEASE")
10321046
message(FATAL_ERROR "Benchmark should be built with RELEASE build type, current build type is ${CMAKE_BUILD_TYPE}")
10331047
endif()
1034-
add_executable(benchmark_test ${BASE_DIR}/benchmark/benchmark_main.cpp)
1048+
add_executable(benchmark_test ${BASE_DIR}/benchmark/benchmark_main.cpp
1049+
# No-ops everywhere except macOS/arm64, where benchmark_test links
1050+
# without tcmalloc (see the MALLOCLIB branch above).
1051+
${BASE_DIR}/test/testutil/gperftools_stubs.cpp)
10351052
set_target_properties(benchmark_test PROPERTIES COMPILE_FLAGS "-fno-access-control")
10361053
target_link_libraries(benchmark_test ${DORIS_LINK_LIBS})
10371054
message(STATUS "Add benchmark to build")

‎be/src/common/object_pool.h‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,6 @@
1818
#pragma once
1919

2020
#include <mutex>
21-
#include <ranges>
2221
#include <vector>
2322

2423
namespace doris {
@@ -53,8 +52,10 @@ class ObjectPool {
5352
// reverse delete object to make sure the obj can
5453
// safe access the member object construt early by
5554
// object pool
56-
for (auto& _object : std::ranges::reverse_view(_objects)) {
57-
_object.delete_fn(_object.obj);
55+
// NOTE: keep <ranges> out of this widely-included header: doris_be_test
56+
// builds with -fno-access-control, which libc++'s <ranges> rejects.
57+
for (auto it = _objects.rbegin(); it != _objects.rend(); ++it) {
58+
it->delete_fn(it->obj);
5859
}
5960
_objects.clear();
6061
}

‎be/src/exprs/lambda_function/lambda_execution_context.h‎

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,6 @@
1919

2020
#include <glog/logging.h>
2121

22-
#include <ranges>
2322
#include <set>
2423
#include <string>
2524
#include <utility>
@@ -79,10 +78,12 @@ class LambdaExecutionContext {
7978

8079
ResolveResult resolve_column_position(const std::string& name) const {
8180
ResolveResult result;
82-
for (const auto& frame : std::ranges::reverse_view(_frames)) {
81+
for (auto frame_it = _frames.rbegin(); frame_it != _frames.rend(); ++frame_it) {
82+
const auto& frame = *frame_it;
8383
result.searched_named_scope |= frame.bind_by_name;
84-
for (const auto& argument_binding :
85-
std::ranges::reverse_view(frame.argument_bindings)) {
84+
for (auto binding_it = frame.argument_bindings.rbegin();
85+
binding_it != frame.argument_bindings.rend(); ++binding_it) {
86+
const auto& argument_binding = *binding_it;
8687
if (argument_binding.name == name) {
8788
result.found = true;
8889
result.column_position = argument_binding.column_position;
@@ -97,13 +98,13 @@ class LambdaExecutionContext {
9798
}
9899

99100
void collect_visible_binding_column_positions(std::set<int>& column_positions) const {
100-
for (const auto& _frame : std::ranges::reverse_view(_frames)) {
101-
for (const auto& binding : _frame.argument_bindings) {
101+
for (auto frame_it = _frames.rbegin(); frame_it != _frames.rend(); ++frame_it) {
102+
for (const auto& binding : frame_it->argument_bindings) {
102103
if (binding.column_position >= 0) {
103104
column_positions.insert(binding.column_position);
104105
}
105106
}
106-
if (!_frame.parent_bindings_visible) {
107+
if (!frame_it->parent_bindings_visible) {
107108
break;
108109
}
109110
}

‎be/src/format/table/iceberg_reader_mixin.h‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,10 +19,10 @@
1919

2020
#include <gen_cpp/ExternalTableSchema_types.h>
2121

22+
#include <algorithm>
2223
#include <cstddef>
2324
#include <cstdint>
2425
#include <memory>
25-
#include <ranges>
2626
#include <string>
2727
#include <unordered_map>
2828
#include <vector>

‎be/src/storage/index/inverted/query_v2/collect/top_k_collector.h‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,6 @@
2222
#include <algorithm>
2323
#include <cstdint>
2424
#include <limits>
25-
#include <ranges>
2625
#include <roaring/roaring.hh>
2726
#include <string>
2827
#include <vector>

‎be/src/storage/index/inverted/query_v2/composite_reader.h‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,6 @@
3333
#endif
3434

3535
#include <algorithm>
36-
#include <ranges>
3736
#include <unordered_map>
3837
#include <vector>
3938

@@ -66,7 +65,7 @@ class CompositeReader {
6665
}
6766

6867
void close() {
69-
for (auto* reader : std::views::values(_field_readers)) {
68+
for (const auto& [_, reader] : _field_readers) {
7069
reader->close();
7170
}
7271
}
@@ -76,7 +75,7 @@ class CompositeReader {
7675
throw Exception(ErrorCode::INDEX_INVALID_PARAMETERS, "CompositeReader has no readers");
7776
}
7877
uint32_t max_doc = 0;
79-
for (auto* reader : std::views::values(_field_readers)) {
78+
for (const auto& [_, reader] : _field_readers) {
8079
max_doc = std::max(max_doc, static_cast<uint32_t>(reader->maxDoc()));
8180
}
8281
return max_doc;
@@ -85,7 +84,7 @@ class CompositeReader {
8584
[[nodiscard]] std::vector<lucene::index::IndexReader*> readers() const {
8685
std::vector<lucene::index::IndexReader*> readers;
8786
readers.reserve(_field_readers.size());
88-
for (auto* reader : std::views::values(_field_readers)) {
87+
for (const auto& [_, reader] : _field_readers) {
8988
readers.push_back(reader);
9089
}
9190
return readers;

‎be/src/storage/index/inverted/query_v2/wand/block_wand.h‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,6 @@
1919

2020
#include <algorithm>
2121
#include <cassert>
22-
#include <ranges>
2322
#include <vector>
2423

2524
#include "storage/index/inverted/query_v2/term_query/term_scorer.h"

‎be/test/CMakeLists.txt‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -164,6 +164,10 @@ endif()
164164
if (APPLE)
165165
target_link_libraries(doris_be_test ${TEST_LINK_LIBS}
166166
-Wl,-force_load,$<TARGET_FILE:vector_search_test>)
167+
# $<TARGET_FILE:...> inside a link flag does not create a target-level
168+
# dependency, so parallel ninja could link doris_be_test before the
169+
# archive exists.
170+
add_dependencies(doris_be_test vector_search_test)
167171
else()
168172
target_link_libraries(doris_be_test ${TEST_LINK_LIBS}
169173
-Wl,--whole-archive vector_search_test -Wl,--no-whole-archive)

‎be/test/exprs/expr_zonemap_filter_test.cpp‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -61,10 +61,8 @@
6161
#pragma clang diagnostic push
6262
#pragma clang diagnostic ignored "-Wkeyword-macro"
6363
#endif
64-
#define private public
6564
#include "exprs/vdirect_in_predicate.h"
6665
#include "exprs/vin_predicate.h"
67-
#undef private
6866
#if defined(__clang__)
6967
#pragma clang diagnostic pop
7068
#endif

‎be/test/exprs/vsearch_expr_test.cpp‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -41,10 +41,8 @@
4141
#pragma clang diagnostic push
4242
#pragma clang diagnostic ignored "-Wkeyword-macro"
4343
#endif
44-
#define private public
4544
#include "exprs/vslot_ref.h"
4645
#include "storage/segment/segment.h"
47-
#undef private
4846
#if defined(__clang__)
4947
#pragma clang diagnostic pop
5048
#endif

0 commit comments

Comments
 (0)