proxy build responsible for 10% of the world's carbon production #4676
Description
Okay. Not literally... But the proxy build has started consuming an absurd amount of resources. Building recent main on a 12-core host requires >20GB of memory and takes almost 15 minutes.
:; docker buildx build . --build-arg=PROXY_FEATURES=mock-orig-dst --load --no-cache --progress=plain
b1b6b537 (tag: release/v2.100.0) inbound: Do not cache gateway services (#549)
##14 576.14ust to 1.44.0 (#574)uelease [optimized] target(s) in 9m 36s
#14 576.7 Command being timed: "cargo build --locked --release --features=mock-orig-dst"
#14 576.7 User time (seconds): 2458.59
#14 576.7 System time (seconds): 34.22
#14 576.7 Percent of CPU this job got: 432%
#14 576.7 Elapsed (wall clock) time (h:mm:ss or m:ss): 9:36.17
#14 576.7 Average shared text size (kbytes): 0
#14 576.7 Average unshared data size (kbytes): 0
#14 576.7 Average stack size (kbytes): 0
#14 576.7 Average total size (kbytes): 0
#14 576.7 Maximum resident set size (kbytes): 14764740
#14 576.7 Average resident set size (kbytes): 0
#14 576.7 Major (requiring I/O) page faults: 0
#14 576.7 Minor (reclaiming a frame) page faults: 13084019
#14 576.7 Voluntary context switches: 33178
#14 576.7 Involuntary context switches: 68362
#14 576.7 Swaps: 0
#14 576.7 File system inputs: 1632
#14 576.7 File system outputs: 2099736
#14 576.7 Socket messages sent: 0
#14 576.7 Socket messages received: 0
#14 576.7 Signals delivered: 0
#14 576.7 Page size (bytes): 4096
#14 576.7 Exit status: 0
b72f156e Add a CODEOWNERS (#558)
e23d7a99 Fix shellcheck issues in shell scripts (#554)
c579f42a update the proxy to use std::future and Tokio 0.2 (#568)
#14 687.6 Finished release [optimized] target(s) in 11m 26s
##14 687.6 Command being timed: "cargo build --locked --release --features=mock-orig-dst"
##14 687.6 User time (seconds): 3066.85
##14 687.6 System time (seconds): 37.36
##14 687.6 Percent of CPU this job got: 451%
##14 687.6 Elapsed (wall clock) time (h:mm:ss or m:ss): 11:27.00
##14 687.6 Average shared text size (kbytes): 0
##14 687.6 Average unshared data size (kbytes): 0
##14 687.6 Average stack size (kbytes): 0
##14 687.6 Average total size (kbytes): 0
##14 687.6 Maximum resident set size (kbytes): 13623064
##14 687.6 Average resident set size (kbytes): 0
##14 687.6 Major (requiring I/O) page faults: 32
##14 687.6 Minor (reclaiming a frame) page faults: 12196293
##14 687.6 Voluntary context switches: 35270
##14 687.6 Involuntary context switches: 139919
##14 687.6 Swaps: 0
##14 687.6 File system inputs: 56440
##14 687.6 File system outputs: 1615896
##14 687.6 Socket messages sent: 0
##14 687.6 Socket messages received: 0
##14 687.6 Signals delivered: 0
##14 687.6 Page size (bytes): 4096
##14 687.6 Exit status: 0
#
0070fcf6 Prune unused dependencies (#569)
1b3e899e Support LINKERD_PROXY_LOG_FORMAT=json (#500)
ef7e1258 Change docs references from "master" to "main" (#571)
c2b6d6f0 Upgrade tokio-rustls & webpki. (#570)
7034dc19 Makefile: Add shellcheck recipe (#555)
4535f79d Update proxy-api dependencies (#573)
eb52fd90 integration: fix missing traces (#572)
032f03de Update Rust to 1.44.0 (#574)
#15 565.5 Finished release [optimized] target(s) in 9m 24s
##15 565.5 Command being timed: "cargo build --locked --release --features=mock-orig-dst"
##15 565.5 User time (seconds): 2677.62
##15 565.5 System time (seconds): 32.40
##15 565.5 Percent of CPU this job got: 479%
##15 565.5 Elapsed (wall clock) time (h:mm:ss or m:ss): 9:24.87
##15 565.5 Average shared text size (kbytes): 0
##15 565.5 Average unshared data size (kbytes): 0
##15 565.5 Average stack size (kbytes): 0
##15 565.5 Average total size (kbytes): 0
##15 565.5 Maximum resident set size (kbytes): 12186564
##15 565.5 Average resident set size (kbytes): 0
##15 565.5 Major (requiring I/O) page faults: 1912
##15 565.5 Minor (reclaiming a frame) page faults: 10876911
##15 565.5 Voluntary context switches: 28492
##15 565.5 Involuntary context switches: 66817
##15 565.5 Swaps: 0
##15 565.5 File system inputs: 436280
##15 565.5 File system outputs: 1489936
##15 565.5 Socket messages sent: 0
##15 565.5 Socket messages received: 0
##15 565.5 Signals delivered: 0
##15 565.5 Page size (bytes): 4096
##15 565.5 Exit status: 0
0b0d7277 Use async/await to simplify connection-accept task (#575)
#15 559.1 Finished release [optimized] target(s) in 9m 18s
##15 559.1 Command being timed: "cargo build --locked --release --features=mock-orig-dst"
##15 559.1 User time (seconds): 2684.86
##15 559.1 System time (seconds): 31.55
##15 559.1 Percent of CPU this job got: 486%
##15 559.1 Elapsed (wall clock) time (h:mm:ss or m:ss): 9:18.56
##15 559.1 Average shared text size (kbytes): 0
##15 559.1 Average unshared data size (kbytes): 0
##15 559.1 Average stack size (kbytes): 0
##15 559.1 Average total size (kbytes): 0
##15 559.1 Maximum resident set size (kbytes): 11973076
##15 559.1 Average resident set size (kbytes): 0
##15 559.1 Major (requiring I/O) page faults: 0
##15 559.1 Minor (reclaiming a frame) page faults: 10668082
##15 559.1 Voluntary context switches: 21618
##15 559.1 Involuntary context switches: 68202
##15 559.1 Swaps: 0
##15 559.1 File system inputs: 0
##15 559.1 File system outputs: 1489888
##15 559.1 Socket messages sent: 0
##15 559.1 Socket messages received: 0
##15 559.1 Signals delivered: 0
##15 559.1 Page size (bytes): 4096
##15 559.1 Exit status: 0
2fc01d97 Update Rust to 1.44.1 (#576)
9eb21138 outbound: Split HTTP endpoint builder (#578)
#14 788.1 Finished release [optimized] target(s) in 13m 07s
##14 788.1 Command being timed: "cargo build --locked --release --features=mock-orig-dst"
##14 788.1 User time (seconds): 3573.92
##14 788.1 System time (seconds): 40.86
##14 788.1 Percent of CPU this job got: 458%
##14 788.1 Elapsed (wall clock) time (h:mm:ss or m:ss): 13:07.58
##14 788.1 Average shared text size (kbytes): 0
##14 788.1 Average unshared data size (kbytes): 0
##14 788.1 Average stack size (kbytes): 0
##14 788.1 Average total size (kbytes): 0
##14 788.1 Maximum resident set size (kbytes): 21870872
##14 788.1 Average resident set size (kbytes): 0
##14 788.1 Major (requiring I/O) page faults: 0
##14 788.1 Minor (reclaiming a frame) page faults: 16825292
##14 788.1 Voluntary context switches: 22262
##14 788.1 Involuntary context switches: 73703
##14 788.1 Swaps: 0
##14 788.1 File system inputs: 584
##14 788.1 File system outputs: 1512248
##14 788.1 Socket messages sent: 0
##14 788.1 Socket messages received: 0
##14 788.1 Signals delivered: 0
##14 788.1 Page size (bytes): 4096
##14 788.1 Exit status: 0
7830d244 Simplify protocol detection with async/await (#577)
#14 804.1 Finished release [optimized] target(s) in 13m 23s
##14 804.1 Command being timed: "cargo build --locked --release --features=mock-orig-dst"
##14 804.1 User time (seconds): 3528.62
##14 804.1 System time (seconds): 40.01
##14 804.1 Percent of CPU this job got: 444%
##14 804.1 Elapsed (wall clock) time (h:mm:ss or m:ss): 13:23.57
##14 804.1 Average shared text size (kbytes): 0
##14 804.1 Average unshared data size (kbytes): 0
##14 804.1 Average stack size (kbytes): 0
##14 804.1 Average total size (kbytes): 0
##14 804.1 Maximum resident set size (kbytes): 22364696
##14 804.1 Average resident set size (kbytes): 0
##14 804.1 Major (requiring I/O) page faults: 0
##14 804.1 Minor (reclaiming a frame) page faults: 15947138
##14 804.1 Voluntary context switches: 22096
##14 804.1 Involuntary context switches: 73969
##14 804.1 Swaps: 0
##14 804.1 File system inputs: 0
##14 804.1 File system outputs: 1511912
##14 804.1 Socket messages sent: 0
##14 804.1 Socket messages received: 0
##14 804.1 Signals delivered: 0
##14 804.1 Page size (bytes): 4096
##14 804.1 Exit status: 0
1d9e9b35 (tag: release/v2.101.0) Pin proxy-api at v0.1.13 (#579)
0d28d8f2 (tag: release/v2.102.0) fix InvalidProfileAddr not converting into DiscoveryRejected (#581)
#14 807.2 Finished release [optimized] target(s) in 13m 26s
#14 807.2 Command being timed: "cargo build --locked --release --features=mock-orig-dst"
#14 807.2 User time (seconds): 3557.58
#14 807.2 System time (seconds): 41.20
#14 807.2 Percent of CPU this job got: 446%
#14 807.2 Elapsed (wall clock) time (h:mm:ss or m:ss): 13:26.74
#14 807.2 Average shared text size (kbytes): 0
#14 807.2 Average unshared data size (kbytes): 0
#14 807.2 Average stack size (kbytes): 0
#14 807.2 Average total size (kbytes): 0
#14 807.2 Maximum resident set size (kbytes): 22426472
#14 807.2 Average resident set size (kbytes): 0
#14 807.2 Major (requiring I/O) page faults: 0
#14 807.2 Minor (reclaiming a frame) page faults: 16617187
#14 807.2 Voluntary context switches: 22019
#14 807.2 Involuntary context switches: 74896
#14 807.2 Swaps: 0
#14 807.2 File system inputs: 480
#14 807.2 File system outputs: 1511904
#14 807.2 Socket messages sent: 0
#14 807.2 Socket messages received: 0
#14 807.2 Signals delivered: 0
#14 807.2 Page size (bytes): 4096
#14 807.2 Exit status: 0
We can see that this was made substantially worse in 9eb21138 (#578), where we add a bunch of new type constraint checking... and this comports with what we see when we try to use rustc's self-profile... we spend >8 minutes just trying to do codegen on the binary (after all library crates have been compiled) -- ~5 minutes of monomorphization, and then several minutes of linking (I presume):
I tried setting lto = "off' and opt-level = 2 but these settings did not significantly impact the build.
I then tried commenting out the fallback instances in the outbound proxy and, immediately, the build is much more reasonable:
:; (cd linkerd2-proxy && /usr/bin/time -v cargo build --release )
Compiling linkerd2-app-outbound v0.1.0 (/home/ver/b/l2-proxy/linkerd/app/outbound)
Compiling linkerd2-app-gateway v0.1.0 (/home/ver/b/l2-proxy/linkerd/app/gateway)
Compiling linkerd2-app v0.1.0 (/home/ver/b/l2-proxy/linkerd/app)
Compiling linkerd2-proxy v0.1.0 (/home/ver/b/l2-proxy/linkerd2-proxy)
Finished release [optimized] target(s) in 6m 14s
Command being timed: "cargo build --release"
User time (seconds): 508.65
System time (seconds): 3.70
Percent of CPU this job got: 136%
Elapsed (wall clock) time (h:mm:ss or m:ss): 6:14.97
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 5616764
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 639
Minor (reclaiming a frame) page faults: 2607314
Voluntary context switches: 2126
Involuntary context switches: 27668
Swaps: 0
File system inputs: 183200
File system outputs: 143072
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
"Only" ~500MB...
This makes some sense, as the fallback type contains two stack types, and both of these are huge and complex.
I've tried boxing the HTTP services generated from each stack, and this to maybe have a negative impact on the build... but we may be able to box the stacks under the fallback layer to improve things.
I'd also like to make an attempt to remove anything like fallback from the stack and instead encode this routing logic in a dedicated service that interacts with the underlying stacks without necessarily encoding them in the service's type...