Skip to content

Commit 6922ab5

Browse files
morningmanclaude
andauthored
[opt](build) 1/4: Speed up BE full build ~22% by cutting hot-header include edges (apache#66400)
### What problem does this PR solve? Problem Summary: A cold full BE build spends most of its wall clock re-parsing a handful of hub headers in nearly every TU. `runtime/exec_env.h`, `runtime/runtime_state.h` and `runtime/thread_context.h` each reached ~1000 of the ~1400 be/src TUs and dragged multi-MB include subtrees (thrift/protobuf type universes, the AWS SDK surface, the workload_management family) into every one of them. This PR adds a measurement harness and then cuts the high-payload include edges from those hub headers, in four prepare+cut commit pairs. Measured with the included bench mode (cold, cache-free, `-j5`, Apple M5 Pro, clang 20): | wave | build wall | delta | |---|---|---| | baseline | 38m54s | | | workload_group isolation | 35m33s | -8.4% | | file cache / frontend info / cluster info out of exec_env.h | 32m34s | -8.3% | | olap_file.pb / threadpool / options out of exec_env.h | 32m40s | neutral (incremental blast-radius + guards) | | s3_file_system / resource_context / data.pb out of runtime_state.h + thread_context.h | 30m21s | -7.1% | **Total: 39m21s -> 30m48s wall (-21.7%).** Per-file compare shows 672-1065 files faster per wave and zero per-file compile-time regressions. Key touch-count reductions (TUs that stop parsing a header): `FrontendService_types.h` 1086->459, `BackendService_types.h` ->225, `io/fs/s3_file_system.h` + AWS SDK 769->58, `gen_cpp/cloud.pb.h` ->192, `workload_management/resource_context.h` 1015->214, `gen_cpp/data.pb.h` ->443, `util/threadpool.h` -515 TUs, `storage/options.h` -837 TUs. What is in the series (10 commits, meant to be read pairwise): 1. `build.sh --compile-bench`: cold cache-free BE bench with per-TU wall/user/maxrss capture, report and A/B compare tooling (`build-support/compile-bench/`) 2. include-edge analysis tools: `cut_impact.py` (blast-radius and reseed estimation from `ninja -t deps` + parsed include graph) and `syntax_sweep.py` (parallel `-fsyntax-only` replay of all TUs) 3. four prepare+cut pairs. Every "prepare" commit is pure-additive (forward declarations, out-of-line sinks of cold method bodies, direct includes for TUs that free-rode the edge about to be cut); every "cut" commit only removes the edges and locks them with new rules in `build-support/check-header-deps.py` (15 rules total), so none of the cut edges can silently return. The only code moved out of line: assigning ThreadPool setters, `ThreadContext::attach_task`, the orphan `ResourceContext` fallback of `resource_ctx()` (hot attached path stays inline), `ScanTask` ctor/dtor and `DataTypeTimeStampTz::to_pb_column_meta` -- all cold paths. Co-authored-by: Claude Fable 5 <[email protected]>
1 parent aa0c49c commit 6922ab5

161 files changed

Lines changed: 2243 additions & 169 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,7 @@ fe/fe-core/src/main/java/org/apache/parquet
9999
be/build*/
100100
be/cmake-build*/
101101
be/ut_build*/
102+
be/compile-bench-results/
102103
be/src/gen_cpp/*.[cc, cpp, h]
103104
be/src/gen_cpp/opcode
104105
be/tags

‎be/src/agent/agent_server.cpp‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@
3737
#include "common/config.h"
3838
#include "common/logging.h"
3939
#include "common/status.h"
40+
#include "runtime/cluster_info.h"
4041
#include "runtime/exec_env.h"
4142
#include "storage/olap_define.h"
4243
#include "storage/options.h"

‎be/src/agent/task_worker_pool.cpp‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,7 @@
6565
#include "io/fs/path.h"
6666
#include "io/fs/remote_file_system.h"
6767
#include "io/fs/s3_file_system.h"
68+
#include "runtime/cluster_info.h"
6869
#include "runtime/exec_env.h"
6970
#include "runtime/fragment_mgr.h"
7071
#include "runtime/index_policy/index_policy_mgr.h"

‎be/src/cloud/cloud_compaction_stop_token.cpp‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222
#include "cloud/cloud_meta_mgr.h"
2323
#include "cloud/config.h"
2424
#include "common/logging.h"
25+
#include "service/backend_options.h"
2526

2627
namespace doris {
2728

‎be/src/cloud/cloud_index_change_compaction.cpp‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@
2121
#include "cloud/config.h"
2222
#include "common/status.h"
2323
#include "cpp/sync_point.h"
24+
#include "service/backend_options.h"
2425

2526
namespace doris {
2627

‎be/src/cloud/cloud_meta_mgr.cpp‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,9 @@
5757
#include "cpp/sync_point.h"
5858
#include "io/fs/obj_storage_client.h"
5959
#include "load/stream_load/stream_load_context.h"
60+
#include "runtime/cluster_info.h"
6061
#include "runtime/exec_env.h"
62+
#include "service/backend_options.h"
6163
#include "storage/olap_common.h"
6264
#include "storage/rowset/rowset.h"
6365
#include "storage/rowset/rowset_factory.h"

‎be/src/cloud/cloud_storage_engine.cpp‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,7 @@
5959
#include "load/memtable/memtable_flush_executor.h"
6060
#include "runtime/exec_env.h"
6161
#include "runtime/memory/cache_manager.h"
62+
#include "service/backend_options.h"
6263
#include "storage/compaction/cumulative_compaction_policy.h"
6364
#include "storage/compaction/cumulative_compaction_time_series_policy.h"
6465
#include "storage/compaction_task_tracker.h"

‎be/src/cloud/cloud_warm_up_manager.cpp‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,7 @@
4343
#include "common/logging.h"
4444
#include "cpp/sync_point.h"
4545
#include "io/cache/block_file_cache_downloader.h"
46+
#include "runtime/cluster_info.h"
4647
#include "runtime/exec_env.h"
4748
#include "service/backend_options.h"
4849
#include "storage/index/inverted/inverted_index_desc.h"

‎be/src/common/signal_handler.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@
3333

3434
#pragma once
3535

36+
#include <butil/macros.h> // ARRAYSIZE_UNSAFE
3637
#include <glog/logging.h>
3738

3839
#include <boost/stacktrace.hpp>

‎be/src/core/block/block.cpp‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020

2121
#include "core/block/block.h"
2222

23+
#include <concurrentqueue.h>
2324
#include <fmt/format.h>
2425
#include <gen_cpp/data.pb.h>
2526
#include <glog/logging.h>

0 commit comments

Comments
 (0)