Repository navigation
Commit 6922ab5
[opt](build) 1/4: Speed up BE full build ~22% by cutting hot-header include edges (apache#66400)
### What problem does this PR solve?
Problem Summary:
A cold full BE build spends most of its wall clock re-parsing a handful
of
hub headers in nearly every TU. `runtime/exec_env.h`,
`runtime/runtime_state.h`
and `runtime/thread_context.h` each reached ~1000 of the ~1400 be/src
TUs and
dragged multi-MB include subtrees (thrift/protobuf type universes, the
AWS SDK
surface, the workload_management family) into every one of them.
This PR adds a measurement harness and then cuts the high-payload
include
edges from those hub headers, in four prepare+cut commit pairs. Measured
with
the included bench mode (cold, cache-free, `-j5`, Apple M5 Pro, clang
20):
| wave | build wall | delta |
|---|---|---|
| baseline | 38m54s | |
| workload_group isolation | 35m33s | -8.4% |
| file cache / frontend info / cluster info out of exec_env.h | 32m34s |
-8.3% |
| olap_file.pb / threadpool / options out of exec_env.h | 32m40s |
neutral (incremental blast-radius + guards) |
| s3_file_system / resource_context / data.pb out of runtime_state.h +
thread_context.h | 30m21s | -7.1% |
**Total: 39m21s -> 30m48s wall (-21.7%).** Per-file compare shows
672-1065
files faster per wave and zero per-file compile-time regressions.
Key touch-count reductions (TUs that stop parsing a header):
`FrontendService_types.h` 1086->459, `BackendService_types.h` ->225,
`io/fs/s3_file_system.h` + AWS SDK 769->58, `gen_cpp/cloud.pb.h` ->192,
`workload_management/resource_context.h` 1015->214, `gen_cpp/data.pb.h`
->443,
`util/threadpool.h` -515 TUs, `storage/options.h` -837 TUs.
What is in the series (10 commits, meant to be read pairwise):
1. `build.sh --compile-bench`: cold cache-free BE bench with per-TU
wall/user/maxrss capture, report and A/B compare tooling
(`build-support/compile-bench/`)
2. include-edge analysis tools: `cut_impact.py` (blast-radius and reseed
estimation from `ninja -t deps` + parsed include graph) and
`syntax_sweep.py` (parallel `-fsyntax-only` replay of all TUs)
3. four prepare+cut pairs. Every "prepare" commit is pure-additive
(forward
declarations, out-of-line sinks of cold method bodies, direct includes
for
TUs that free-rode the edge about to be cut); every "cut" commit only
removes the edges and locks them with new rules in
`build-support/check-header-deps.py` (15 rules total), so none of the
cut edges can silently return.
The only code moved out of line: assigning ThreadPool setters,
`ThreadContext::attach_task`, the orphan `ResourceContext` fallback of
`resource_ctx()` (hot attached path stays inline), `ScanTask` ctor/dtor
and
`DataTypeTimeStampTz::to_pb_column_meta` -- all cold paths.
Co-authored-by: Claude Fable 5 <[email protected]>1 parent aa0c49c commit 6922ab5
161 files changed
Lines changed: 2243 additions & 169 deletions
File tree
- be
- src
- agent
- cloud
- common
- core
- block
- data_type
- exec
- exchange
- operator
- pipeline
- scan
- task_executor/tools/simulator
- sink
- writer
- maxcompute
- exprs
- aggregate
- function
- format
- parquet
- table
- information_schema
- io
- cache
- fs
- load
- channel
- group_commit
- wal
- memtable
- routine_load
- stream_load
- runtime
- memory
- query_cache
- workload_group
- workload_management
- service
- http
- action
- storage
- index
- inverted
- rowset
- segment
- task
- udf/python
- util
- test
- cloud
- exec
- operator
- pipeline
- scan
- format_v2/native
- format
- native
- orc
- parquet
- transformer
- io
- cache
- client
- load/delta_writer
- runtime
- memory
- workload_group
- service/http
- storage
- index/inverted/query_v2
- schema_change
- segment
- tablet
- testutil
- build-support
- compile-bench
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
99 | 99 | | |
100 | 100 | | |
101 | 101 | | |
| 102 | + | |
102 | 103 | | |
103 | 104 | | |
104 | 105 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
37 | 37 | | |
38 | 38 | | |
39 | 39 | | |
| 40 | + | |
40 | 41 | | |
41 | 42 | | |
42 | 43 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
65 | 65 | | |
66 | 66 | | |
67 | 67 | | |
| 68 | + | |
68 | 69 | | |
69 | 70 | | |
70 | 71 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
| 25 | + | |
25 | 26 | | |
26 | 27 | | |
27 | 28 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
| 24 | + | |
24 | 25 | | |
25 | 26 | | |
26 | 27 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
57 | 57 | | |
58 | 58 | | |
59 | 59 | | |
| 60 | + | |
60 | 61 | | |
| 62 | + | |
61 | 63 | | |
62 | 64 | | |
63 | 65 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
59 | 59 | | |
60 | 60 | | |
61 | 61 | | |
| 62 | + | |
62 | 63 | | |
63 | 64 | | |
64 | 65 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
43 | 43 | | |
44 | 44 | | |
45 | 45 | | |
| 46 | + | |
46 | 47 | | |
47 | 48 | | |
48 | 49 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
| 36 | + | |
36 | 37 | | |
37 | 38 | | |
38 | 39 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| 23 | + | |
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
| |||
0 commit comments