Skip to content

Commit 7acfd9b

Browse files
authored
[improvement](be) Reduce transitive dependencies in shared headers (apache#69717)
Related PR: apache#66901 ### What problem does this PR solve? Several shared BE headers transitively include protobuf, Thrift, thread-pool, and storage-reader implementations. Translation units that only need declarations must still parse these definitions, increasing compilation work and the rebuild scope when those headers change. Recent CI builds failed during CMake configuration because the reverse-reach counts for threadpool.h and rle_encoding.h exceeded the build-hygiene limits. This PR audits all 32 header-budget entries and reduces unnecessary dependencies without changing the budgets or exception lists. The changes: - Extract `ColumnIteratorOptions` into a dedicated header so `vexpr_context.h` does not require the full `column_reader.h` dependency chain. - Use forward declarations for `thread-pool`, `protobuf`, `cache`, and `WorkloadGroup `types where complete definitions are unnecessary. - Move `UniqueId` conversions, `protobuf` methods in numeric `SerDe`, and `selected construction/destruction` operations into `.cpp` files. - Extract `SyncSizeCallback` into a dedicated header to reduce RPC dependencies propagated through the runtime-filter interface. - Add direct includes at actual use sites and verify independent compilation. Object layouts, move semantics, explicit template instantiations, and per-value decoding implementations are preserved. Moving some definitions into .cpp files changes their visibility for inlining in ordinary separate compilation. Query runtime performance has not been measured in this PR. ### Dependency reductions The following counts come from Ninja dependencies after completed builds, covering 1,531 production translation units and 52 generated translation units: | Header | TUs Before | TUs Now | New Limit | |---|---:|---:|---:| | util/rle_encoding.h | 846 | 231 | 250 | | util/threadpool.h | 1,056 | 585 | 600 | | gen_cpp/types.pb.h | 1,862 | 1,225 | 1,250 | | gen_cpp/segment_v2.pb.h | 1,231 | 1,156 | 1,200 | | gen_cpp/internal_service.pb.h | 777 | 557 | 600 | | gen_cpp/FrontendService_types.h | 593 | 293 | 300 | | gen_cpp/cloud.pb.h | 325 | 30 | 50 | | gen_cpp/BackendService_types.h | 195 | 115 | 150 | | gen_cpp/data.pb.h | 228 | 90 | 100 | | storage/options.h | 502 | 293 | 350 | | io/fs/s3_file_system.h | 65 | 31 | 50 | | gen_cpp/Types_types.h | 2,078 | 2,073 | 2,100 | | gen_cpp/PaloInternalService_types.h | 1,093 | 1,093 | 1,100 | | util/brpc_closure.h | 62 | 35 | 50 | | gen_cpp/Status_types.h | 37 | 37 | 41 | | runtime/workload_group/workload_group.h | 43 | 38 | 47 | Across the 16 monitored heavy headers, measured header-to-TU dependency pairs decrease from 8,108 to 6,617, an 18.39% reduction. These removals affect 742 distinct translation units. The number of production compilation units remains unchanged. The transitive project-header closure of dependency.h decreases from 325 to 246 headers, a 24.31% reduction. This uses the hygiene script’s static include graph and is reported separately from the measured Ninja dependency counts. ### Query runtime performance ### TPC-H SF1000 | Date / Version | Hot-min (s) | PR vs. Master | |---|---:|---:| | master 2026-10-08 | 59.615 | -9.93% | | master 2026-10-13 | 60.776 | -11.65% | | master 2026-10-15 | 57.021 | -5.83% | | master 2026-10-20 | 55.877 | -3.90% | | master 2026-10-27 | 65.987 | -18.62% | | PR apache#69717 | 53.697 | — | ### SSB SF1000 | Date / Version | Hot-min (s) | PR vs. Master | |---|---:|---:| | master 2026-10-08 | 11.418 | +3.71% | | master 2026-10-13 | 11.749 | +0.79% | | master 2026-10-15 | 10.653 | +11.16% | | master 2026-10-20 | 10.773 | +9.92% | | master 2026-10-27 | 11.471 | +3.23% | | PR apache#69717 | 11.842 | — | ### ClickBench Configuration: cluster_type=single, ecs.c9i.4xlarge_16c32g, one 500 GB disk, full analyze, PR apache#69717 release package. | Date / Version | Hot-min (s) | PR vs. Master | |---|---:|---:| | master 2026-10-08 | 24.717 | -1.83% | | master 2026-10-13 | 26.406 | -8.11% | | master 2026-10-15 | 25.079 | -3.25% | | master 2026-10-20 | 24.629 | -1.48% | | master 2026-10-27 | 23.643 | +2.63% | | PR apache#69717 single rerun | 24.265 | — | ### TPC-DS SF1000 (1 TB) Comparisons use the PR result from 2026-10-29. The 2026-10-28 rerun is listed separately. | Date / Version | Hot-min (s) | PR vs. Master | |---|---:|---:| | master 2026-10-08 | 168.926 | -7.40% | | master 2026-10-13 | 170.676 | -8.35% | | master 2026-10-15 | 163.170 | -4.13% | | master 2026-10-20 | 163.926 | -4.58% | | master 2026-10-27 | 195.853 | -20.13% | | PR apache#69717 2026-10-29 main result | 156.423 | — | ### Release note None ### Check List (For Author) - Test: Three complete Release builds; standard ASAN BE/FE build and final incremental build; all BE UT sources compiled and linked; 29 selected unit tests passed with -j50 --coverage; 1,024-row service read, aggregate, shuffle join, sort and cast checks; header hygiene and changed-line format checks passed. Clang-tidy covered all 71 files and retained two existing warnings in relocated protobuf methods (function size and signedness). - Behavior changed: No - Does this need documentation: No
1 parent cb0b56d commit 7acfd9b

72 files changed

Lines changed: 518 additions & 347 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎be/src/cloud/cloud_index_change_compaction.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@
2121

2222
#include "cloud/cloud_storage_engine.h"
2323
#include "cloud/cloud_tablet.h"
24+
#include "gen_cpp/cloud.pb.h"
2425
#include "storage/compaction/compaction.h"
2526

2627
namespace doris {

‎be/src/cloud/cloud_schema_change_job.h‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,10 @@
2727

2828
namespace doris {
2929

30+
namespace cloud {
31+
class TabletJobInfoPB;
32+
}
33+
3034
class CloudSchemaChangeJob {
3135
public:
3236
CloudSchemaChangeJob(CloudStorageEngine& cloud_storage_engine, std::string job_id,

‎be/src/cloud/cloud_tablet.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@
2626
#include "storage/tablet/base_tablet.h"
2727

2828
namespace doris {
29+
class TTabletInfo;
2930

3031
class CloudStorageEngine;
3132

‎be/src/common/signal_handler.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@
3434
#pragma once
3535

3636
#include <butil/macros.h> // ARRAYSIZE_UNSAFE
37+
#include <gen_cpp/types.pb.h>
3738
#include <glog/logging.h>
3839

3940
#include <boost/stacktrace.hpp>

‎be/src/core/data_type_serde/data_type_number_serde.cpp‎

Lines changed: 170 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@
1919

2020
#include <arrow/builder.h>
2121
#include <arrow/util/float16.h>
22+
#include <gen_cpp/types.pb.h>
2223

2324
#include <bit>
2425
#include <cmath>
@@ -2005,6 +2006,175 @@ Status DataTypeNumberSerDe<T>::read_column_from_orc(IColumn& column,
20052006
return DataTypeSerDe::read_column_from_orc(column, view);
20062007
}
20072008

2009+
template <PrimitiveType T>
2010+
Status DataTypeNumberSerDe<T>::read_column_from_pb(IColumn& column, const PValues& arg) const {
2011+
auto old_column_size = column.size();
2012+
if constexpr (T == TYPE_BOOLEAN) {
2013+
column.resize(old_column_size + arg.uint32_value_size());
2014+
auto& data = assert_cast<ColumnType&>(column).get_data();
2015+
for (int i = 0; i < arg.uint32_value_size(); ++i) {
2016+
data[old_column_size + i] =
2017+
cast_set<typename PrimitiveTypeTraits<T>::CppType, uint32_t, false>(
2018+
arg.uint32_value(i));
2019+
}
2020+
} else if constexpr (T == TYPE_DATEV2 || T == TYPE_IPV4) {
2021+
column.resize(old_column_size + arg.uint32_value_size());
2022+
auto& data = assert_cast<ColumnType&>(column).get_data();
2023+
for (int i = 0; i < arg.uint32_value_size(); ++i) {
2024+
data[old_column_size + i] = arg.uint32_value(i);
2025+
}
2026+
} else if constexpr (T == TYPE_TINYINT || T == TYPE_SMALLINT) {
2027+
column.resize(old_column_size + arg.int32_value_size());
2028+
auto& data = reinterpret_cast<ColumnType&>(column).get_data();
2029+
for (int i = 0; i < arg.int32_value_size(); ++i) {
2030+
data[old_column_size + i] =
2031+
cast_set<typename PrimitiveTypeTraits<T>::CppType, int32_t, false>(
2032+
arg.int32_value(i));
2033+
}
2034+
} else if constexpr (T == TYPE_INT) {
2035+
column.resize(old_column_size + arg.int32_value_size());
2036+
auto& data = reinterpret_cast<ColumnType&>(column).get_data();
2037+
for (int i = 0; i < arg.int32_value_size(); ++i) {
2038+
data[old_column_size + i] = arg.int32_value(i);
2039+
}
2040+
} else if constexpr (T == TYPE_DATETIMEV2) {
2041+
column.resize(old_column_size + arg.uint64_value_size());
2042+
auto& data = reinterpret_cast<ColumnType&>(column).get_data();
2043+
for (int i = 0; i < arg.uint64_value_size(); ++i) {
2044+
data[old_column_size + i] =
2045+
binary_cast<UInt64, DateV2Value<DateTimeV2ValueType>>(arg.uint64_value(i));
2046+
}
2047+
} else if constexpr (T == TYPE_TIMESTAMPTZ) {
2048+
column.resize(old_column_size + arg.uint64_value_size());
2049+
auto& data = reinterpret_cast<ColumnType&>(column).get_data();
2050+
for (int i = 0; i < arg.uint64_value_size(); ++i) {
2051+
data[old_column_size + i] = binary_cast<UInt64, TimestampTzValue>(arg.uint64_value(i));
2052+
}
2053+
} else if constexpr (T == TYPE_DATE || T == TYPE_DATETIME) {
2054+
column.resize(old_column_size + arg.int64_value_size());
2055+
auto& data = reinterpret_cast<ColumnType&>(column).get_data();
2056+
for (int i = 0; i < arg.int64_value_size(); ++i) {
2057+
data[old_column_size + i] = binary_cast<Int64, VecDateTimeValue>(arg.int64_value(i));
2058+
}
2059+
} else if constexpr (T == TYPE_BIGINT) {
2060+
column.resize(old_column_size + arg.int64_value_size());
2061+
auto& data = reinterpret_cast<ColumnType&>(column).get_data();
2062+
for (int i = 0; i < arg.int64_value_size(); ++i) {
2063+
data[old_column_size + i] = arg.int64_value(i);
2064+
}
2065+
} else if constexpr (T == TYPE_TIMESTAMP_NS) {
2066+
column.resize(old_column_size + arg.int64_value_size());
2067+
auto& data = reinterpret_cast<ColumnType&>(column).get_data();
2068+
for (int i = 0; i < arg.int64_value_size(); ++i) {
2069+
data[old_column_size + i] = TimeStampNsValue(arg.int64_value(i));
2070+
}
2071+
} else if constexpr (T == TYPE_FLOAT) {
2072+
column.resize(old_column_size + arg.float_value_size());
2073+
auto& data = reinterpret_cast<ColumnType&>(column).get_data();
2074+
for (int i = 0; i < arg.float_value_size(); ++i) {
2075+
data[old_column_size + i] = arg.float_value(i);
2076+
}
2077+
} else if constexpr (T == TYPE_DOUBLE || T == TYPE_TIMEV2) {
2078+
column.resize(old_column_size + arg.double_value_size());
2079+
auto& data = reinterpret_cast<ColumnType&>(column).get_data();
2080+
for (int i = 0; i < arg.double_value_size(); ++i) {
2081+
data[old_column_size + i] = arg.double_value(i);
2082+
}
2083+
} else if constexpr (T == TYPE_LARGEINT) {
2084+
column.resize(old_column_size + arg.bytes_value_size());
2085+
auto& data = reinterpret_cast<ColumnType&>(column).get_data();
2086+
for (int i = 0; i < arg.bytes_value_size(); ++i) {
2087+
data[old_column_size + i] = *(int128_t*)(arg.bytes_value(i).c_str());
2088+
}
2089+
} else {
2090+
return Status::NotSupported("unknown ColumnType for reading from pb");
2091+
}
2092+
return Status::OK();
2093+
}
2094+
2095+
template <PrimitiveType T>
2096+
Status DataTypeNumberSerDe<T>::write_column_to_pb(const IColumn& column, PValues& result,
2097+
int64_t start, int64_t end) const {
2098+
auto row_count = cast_set<int>(end - start);
2099+
auto* ptype = result.mutable_type();
2100+
const auto* col = assert_cast<const ColumnType*>(&column);
2101+
if constexpr (T == TYPE_LARGEINT) {
2102+
ptype->set_id(PGenericType::INT128);
2103+
result.mutable_bytes_value()->Reserve(row_count);
2104+
for (size_t row_num = start; row_num < end; ++row_num) {
2105+
StringRef single_data = col->get_data_at(row_num);
2106+
result.add_bytes_value(single_data.data, single_data.size);
2107+
}
2108+
return Status::OK();
2109+
}
2110+
auto& data = col->get_data();
2111+
if constexpr (T == TYPE_BOOLEAN) {
2112+
ptype->set_id(PGenericType::UINT8);
2113+
auto* values = result.mutable_uint32_value();
2114+
values->Reserve(row_count);
2115+
values->Add(data.begin() + start, data.begin() + end);
2116+
} else if constexpr (T == TYPE_DATEV2) {
2117+
ptype->set_id(PGenericType::UINT32);
2118+
auto* values = result.mutable_uint32_value();
2119+
values->Reserve(row_count);
2120+
values->Add((uint32_t*)data.begin() + start, (uint32_t*)data.begin() + end);
2121+
} else if constexpr (T == TYPE_IPV4) {
2122+
ptype->set_id(PGenericType::UINT32);
2123+
auto* values = result.mutable_uint32_value();
2124+
values->Reserve(row_count);
2125+
values->Add(data.begin() + start, data.begin() + end);
2126+
} else if constexpr (T == TYPE_DATETIMEV2 || T == TYPE_TIMESTAMPTZ) {
2127+
ptype->set_id(PGenericType::UINT64);
2128+
auto* values = result.mutable_uint64_value();
2129+
values->Reserve(row_count);
2130+
values->Add((uint64_t*)data.begin() + start, (uint64_t*)data.begin() + end);
2131+
} else if constexpr (T == TYPE_TINYINT) {
2132+
ptype->set_id(PGenericType::INT8);
2133+
auto* values = result.mutable_int32_value();
2134+
values->Reserve(row_count);
2135+
values->Add(data.begin() + start, data.begin() + end);
2136+
} else if constexpr (T == TYPE_SMALLINT) {
2137+
ptype->set_id(PGenericType::INT16);
2138+
auto* values = result.mutable_int32_value();
2139+
values->Reserve(row_count);
2140+
values->Add(data.begin() + start, data.begin() + end);
2141+
} else if constexpr (T == TYPE_INT) {
2142+
ptype->set_id(PGenericType::INT32);
2143+
auto* values = result.mutable_int32_value();
2144+
values->Reserve(row_count);
2145+
values->Add(data.begin() + start, data.begin() + end);
2146+
} else if constexpr (T == TYPE_DATE || T == TYPE_DATETIME) {
2147+
ptype->set_id(PGenericType::INT64);
2148+
auto* values = result.mutable_int64_value();
2149+
values->Reserve(row_count);
2150+
values->Add((int64_t*)data.begin() + start, (int64_t*)data.begin() + end);
2151+
} else if constexpr (T == TYPE_BIGINT) {
2152+
ptype->set_id(PGenericType::INT64);
2153+
auto* values = result.mutable_int64_value();
2154+
values->Reserve(row_count);
2155+
values->Add(data.begin() + start, data.begin() + end);
2156+
} else if constexpr (T == TYPE_TIMESTAMP_NS) {
2157+
ptype->set_id(PGenericType::INT64);
2158+
auto* values = result.mutable_int64_value();
2159+
values->Reserve(row_count);
2160+
values->Add(reinterpret_cast<const int64_t*>(data.begin()) + start,
2161+
reinterpret_cast<const int64_t*>(data.begin()) + end);
2162+
} else if constexpr (T == TYPE_FLOAT) {
2163+
ptype->set_id(PGenericType::FLOAT);
2164+
auto* values = result.mutable_float_value();
2165+
values->Reserve(row_count);
2166+
values->Add(data.begin() + start, data.begin() + end);
2167+
} else if constexpr (T == TYPE_DOUBLE || T == TYPE_TIMEV2) {
2168+
ptype->set_id(PGenericType::DOUBLE);
2169+
auto* values = result.mutable_double_value();
2170+
values->Reserve(row_count);
2171+
values->Add(data.begin() + start, data.begin() + end);
2172+
} else {
2173+
return Status::NotSupported("unknown ColumnType for writing to pb");
2174+
}
2175+
return Status::OK();
2176+
}
2177+
20082178
/// Explicit template instantiations - to avoid code bloat in headers.
20092179
template class DataTypeNumberSerDe<TYPE_BOOLEAN>;
20102180
template class DataTypeNumberSerDe<TYPE_TINYINT>;

0 commit comments

Comments
 (0)