Skip to content

Commit c2bf234

Browse files
morningmanclaude
andcommitted
[opt](build) Prepare cutting three hot edges: sinks, seeds, suppressions
Pure-additive / behavior-preserving preparation so that three include edges can be cut next: core/pod_array.h -> runtime/thread_context.h (dead include), core/column/column.h -> exec/sort/hybrid_sorter.h (core->exec layering violation reaching 808 TUs) and wide_integer_impl.h's unconditional boost/multiprecision include (4.27MB of preprocessed closure in 1169 TUs on platforms without an 80-bit long double). No include edge is removed here: - core/wide_integer_from_double.cpp (new): compiles the from-double conversion once and explicitly instantiates it for integer<128|256, signed|unsigned>; harmless while the header still defines the members inline - column.h: forward-declare HybridSorter; move the BE_TEST-only get_permutation_default body to column.cpp (it constructs HybridSorter by value); the production get_permutation default keeps its inline throw body - column_decimal.h: move the permutation<U> helper body to column_decimal.cpp (its only caller, get_permutation, is already defined there); member templates are instantiated by that call, not by the class-level explicit instantiations - column_const.h: include <span> directly (rode in via hybrid_sorter.h) - storage/olap_define.h: drop the trailing semicolon inside DISALLOW_COPY_AND_ASSIGN so the call sites' `;` completes the expansion: butil/macros.h defines the same macro without one and wins under #ifndef in TUs that happen to see butil first, so the two definitions must stay call-site compatible - format_v2/native/native_reader.h: include runtime/runtime_profile.h (RuntimeProfile::Counter members need the complete type; it rode in via pod_array -> thread_context) - bm25_similarity.cpp: drop the dead `using namespace inverted_index` (the only declaration of that namespace lives in exec_env.h and rode the same edge) - inverted_index_searcher.h, index_file_writer.h, ann_index_writer.h, query/query.h: wrap their bare <CLucene.h> includes in the -Wconversion suppression (inverted_index_common_impl.h pattern); whether CLucene's first expansion lands inside a suppressed region depends on include order Co-Authored-By: Claude Fable 5 <[email protected]>
1 parent 02cb462 commit c2bf234

13 files changed

Lines changed: 138 additions & 31 deletions

File tree

‎be/src/core/column/column.cpp‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@
2323
#include "core/column/column_const.h"
2424
#include "core/column/column_nullable.h"
2525
#include "core/data_type/data_type.h"
26+
#include "exec/sort/hybrid_sorter.h"
2627
#include "exec/sort/sort_block.h"
2728
#include "util/simd/bits.h"
2829

@@ -237,4 +238,12 @@ void IColumn::check_const_only_in_top_level() const {
237238
for_each_subcolumn(throw_if_const);
238239
}
239240

241+
#ifdef BE_TEST
242+
void IColumn::get_permutation_default(bool reverse, size_t limit, int nan_direction_hint,
243+
Permutation& res) const {
244+
HybridSorter sorter;
245+
get_permutation(reverse, limit, nan_direction_hint, sorter, res);
246+
}
247+
#endif
248+
240249
} // namespace doris

‎be/src/core/column/column.h‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,7 @@ namespace doris {
4747

4848
class Arena;
4949
class ColumnSorter;
50+
class HybridSorter;
5051

5152
using EqualFlags = std::vector<uint8_t>;
5253
using EqualRange = std::pair<int, int>;
@@ -514,11 +515,10 @@ class IColumn : public COW<IColumn> {
514515
}
515516

516517
#ifdef BE_TEST
518+
// Defined in column.cpp: constructs a HybridSorter, which is only
519+
// forward-declared here to keep exec/sort out of column.h's closure.
517520
void get_permutation_default(bool reverse, size_t limit, int nan_direction_hint,
518-
Permutation& res) const {
519-
HybridSorter sorter;
520-
get_permutation(reverse, limit, nan_direction_hint, sorter, res);
521-
}
521+
Permutation& res) const;
522522
#endif
523523

524524
/** Split column to smaller columns. Each value goes to column index, selected by corresponding element of 'selector'.

‎be/src/core/column/column_const.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@
2727
#include <cstdint>
2828
#include <functional>
2929
#include <initializer_list>
30+
#include <span>
3031
#include <string>
3132
#include <type_traits>
3233
#include <utility>

‎be/src/core/column/column_decimal.cpp‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@
3434
#include "core/value/decimalv2_value.h"
3535
#include "exec/common/int_exp.h"
3636
#include "exec/common/sip_hash.h"
37+
#include "exec/sort/hybrid_sorter.h"
3738
#include "exec/sort/sort_block.h"
3839
#include "util/hash_util.hpp"
3940
#include "util/simd/bits.h"
@@ -305,6 +306,33 @@ Field ColumnDecimal<T>::operator[](size_t n) const {
305306
return Field::create_field<T>(*(typename PrimitiveTypeTraits<T>::CppType*)(&data[n]));
306307
}
307308

309+
template <PrimitiveType T>
310+
template <typename U>
311+
void ColumnDecimal<T>::permutation(bool reverse, size_t limit, HybridSorter& sorter,
312+
PaddedPODArray<U>& res) const {
313+
size_t s = data.size();
314+
res.resize(s);
315+
for (U i = 0; i < s; ++i) res[i] = i;
316+
317+
auto sort_end = res.end();
318+
if (limit && static_cast<double>(limit) < static_cast<double>(s) / 8.0) {
319+
sort_end = res.begin() + limit;
320+
if (reverse)
321+
std::partial_sort(res.begin(), sort_end, res.end(),
322+
[this](size_t a, size_t b) { return data[a] > data[b]; });
323+
else
324+
std::partial_sort(res.begin(), sort_end, res.end(),
325+
[this](size_t a, size_t b) { return data[a] < data[b]; });
326+
} else {
327+
if (reverse)
328+
sorter.sort(res.begin(), res.end(),
329+
[this](size_t a, size_t b) { return data[a] > data[b]; });
330+
else
331+
sorter.sort(res.begin(), res.end(),
332+
[this](size_t a, size_t b) { return data[a] < data[b]; });
333+
}
334+
}
335+
308336
template <PrimitiveType T>
309337
void ColumnDecimal<T>::get_permutation(bool reverse, size_t limit, int, HybridSorter& sorter,
310338
IColumn::Permutation& res) const {

‎be/src/core/column/column_decimal.h‎

Lines changed: 3 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -285,31 +285,11 @@ class ColumnDecimal final : public COWHelper<IColumn, ColumnDecimal<T>> {
285285
protected:
286286
Container data;
287287
UInt32 scale;
288+
// Defined in column_decimal.cpp (its only caller is get_permutation there):
289+
// the body dereferences HybridSorter, which is forward-declared in column.h.
288290
template <typename U>
289291
void permutation(bool reverse, size_t limit, HybridSorter& sorter,
290-
PaddedPODArray<U>& res) const {
291-
size_t s = data.size();
292-
res.resize(s);
293-
for (U i = 0; i < s; ++i) res[i] = i;
294-
295-
auto sort_end = res.end();
296-
if (limit && static_cast<double>(limit) < static_cast<double>(s) / 8.0) {
297-
sort_end = res.begin() + limit;
298-
if (reverse)
299-
std::partial_sort(res.begin(), sort_end, res.end(),
300-
[this](size_t a, size_t b) { return data[a] > data[b]; });
301-
else
302-
std::partial_sort(res.begin(), sort_end, res.end(),
303-
[this](size_t a, size_t b) { return data[a] < data[b]; });
304-
} else {
305-
if (reverse)
306-
sorter.sort(res.begin(), res.end(),
307-
[this](size_t a, size_t b) { return data[a] > data[b]; });
308-
else
309-
sorter.sort(res.begin(), res.end(),
310-
[this](size_t a, size_t b) { return data[a] < data[b]; });
311-
}
312-
}
292+
PaddedPODArray<U>& res) const;
313293

314294
void ALWAYS_INLINE decimalv2_do_crc(size_t i, uint32_t& hash) const {
315295
const auto& dec_val = (const DecimalV2Value&)data[i];
Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
// On platforms whose long double lacks an 80-bit mantissa (LDBL_MANT_DIG != 64),
19+
// wide_integer_impl.h emulates the from-double intermediate type with
20+
// boost::multiprecision::cpp_bin_float_double_extended. Making that include part
21+
// of every TU's closure costs ~4.3MB of preprocessed text; instead, ordinary TUs
22+
// only see declarations of set_multiplier / wide_integer_from_builtin(double)
23+
// and this TU compiles the bodies once. The macro below switches
24+
// wide_integer_impl.h into "definitions inline here" mode.
25+
#define DORIS_WIDE_INTEGER_FROM_DOUBLE_IMPL_TU
26+
27+
#include "core/wide_integer.h"
28+
29+
#if !(LDBL_MANT_DIG == 64)
30+
31+
// Every wide::integer specialization used by Doris (see core/extended_types.h)
32+
// gets its from-double conversion emitted here. A new specialization that is
33+
// constructed from double elsewhere will fail at link time — add it below.
34+
template void wide::integer<128, signed>::_impl::wide_integer_from_builtin(
35+
wide::integer<128, signed>&, double) noexcept;
36+
template void wide::integer<128, unsigned>::_impl::wide_integer_from_builtin(
37+
wide::integer<128, unsigned>&, double) noexcept;
38+
template void wide::integer<256, signed>::_impl::wide_integer_from_builtin(
39+
wide::integer<256, signed>&, double) noexcept;
40+
template void wide::integer<256, unsigned>::_impl::wide_integer_from_builtin(
41+
wide::integer<256, unsigned>&, double) noexcept;
42+
43+
#endif // !(LDBL_MANT_DIG == 64)

‎be/src/format_v2/native/native_reader.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@
2525
#include <vector>
2626

2727
#include "format_v2/file_reader.h"
28+
#include "runtime/runtime_profile.h"
2829

2930
namespace doris::format::native {
3031

‎be/src/storage/index/ann/ann_index_writer.h‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,9 +17,20 @@
1717

1818
#pragma once
1919

20+
// CLucene is third-party code and is not clean under -Wconversion (which
21+
// -Wshorten-64-to-32 belongs to). Whether its first expansion lands inside
22+
// someone else's suppressed region depends on include order, so suppress it
23+
// deliberately here (same pattern as inverted_index_common_impl.h).
24+
#ifdef __clang__
25+
#pragma clang diagnostic push
26+
#pragma clang diagnostic ignored "-Wconversion"
27+
#endif
2028
#include <CLucene.h> // IWYU pragma: keep
2129
#include <CLucene/analysis/LanguageBasedAnalyzer.h>
2230
#include <CLucene/util/bkd/bkd_writer.h>
31+
#ifdef __clang__
32+
#pragma clang diagnostic pop
33+
#endif
2334
#include <glog/logging.h>
2435

2536
#include <cstdint>

‎be/src/storage/index/index_file_writer.h‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,8 +17,19 @@
1717

1818
#pragma once
1919

20+
// CLucene is third-party code and is not clean under -Wconversion (which
21+
// -Wshorten-64-to-32 belongs to). Whether its first expansion lands inside
22+
// someone else's suppressed region depends on include order, so suppress it
23+
// deliberately here (same pattern as inverted_index_common_impl.h).
24+
#ifdef __clang__
25+
#pragma clang diagnostic push
26+
#pragma clang diagnostic ignored "-Wconversion"
27+
#endif
2028
#include <CLucene.h> // IWYU pragma: keep
2129
#include <CLucene/store/IndexInput.h>
30+
#ifdef __clang__
31+
#pragma clang diagnostic pop
32+
#endif
2233
#include <gen_cpp/olap_common.pb.h>
2334
#include <gen_cpp/olap_file.pb.h>
2435

‎be/src/storage/index/inverted/inverted_index_searcher.h‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,10 +21,21 @@
2121
#pragma clang diagnostic push
2222
#pragma clang diagnostic ignored "-Wshadow-field"
2323
#endif
24+
// CLucene is third-party code and is not clean under -Wconversion (which
25+
// -Wshorten-64-to-32 belongs to). Whether its first expansion lands inside
26+
// someone else's suppressed region depends on include order, so suppress it
27+
// deliberately here (same pattern as inverted_index_common_impl.h).
28+
#ifdef __clang__
29+
#pragma clang diagnostic push
30+
#pragma clang diagnostic ignored "-Wconversion"
31+
#endif
2432
#include <CLucene.h> // IWYU pragma: keep
2533
#ifdef __clang__
2634
#pragma clang diagnostic pop
2735
#endif
36+
#ifdef __clang__
37+
#pragma clang diagnostic pop
38+
#endif
2839

2940
#include <memory>
3041
#include <optional>

0 commit comments

Comments
 (0)