Skip to content

[MOD-10751] build.sh: improve LTO support#6815

Merged
gdesmott merged 8 commits intomasterfrom
gd_10751_lto
Jan 19, 2026
Merged

[MOD-10751] build.sh: improve LTO support#6815
gdesmott merged 8 commits intomasterfrom
gd_10751_lto

Conversation

@gdesmott
Copy link
Collaborator

@gdesmott gdesmott commented Sep 11, 2025

Various improvements to build.sh needed to integrate LTO into the CI.

Mark if applicable

  • This PR introduces API changes
  • This PR introduces serialization changes

Note

Improves cross-language LTO handling and robustness in build.sh.

  • Auto-selects clang/clang++/lld via $CC/$CXX/$LD, falling back to versioned clang-$RUSTC_LLVM_VERSION/lld-$RUSTC_LLVM_VERSION, else unversioned tools
  • Validates Rust and Clang LLVM major versions match; errors with actionable guidance on mismatch
  • Threads chosen compilers/linker into CMake (CMAKE_C_COMPILER, CMAKE_CXX_COMPILER, -fuse-ld) and Rust (RUSTFLAGS with linker-plugin-lto, -C linker=..., -C link-arg=-fuse-ld=...)
  • Simplifies flag parsing (LTO?(=1)) and RUSTFLAGS composition (compact RUST_DYN_CRT handling); removes redundant LTO-specific RUSTFLAGS export block
  • Keeps LTO Linux-only guard and existing test/coverage flag behavior

Written by Cursor Bugbot for commit d2ea0bd. This will update automatically on new commits. Configure here.

@github-actions
Copy link

This pull request is stale because it has been open for 60 days with no activity.

@github-actions github-actions bot added the stale label Nov 11, 2025
@gdesmott gdesmott force-pushed the gd_10751_lto branch 3 times, most recently from 09dbf1c to 49a431a Compare January 6, 2026 12:19
@fcostaoliveira
Copy link
Contributor

fcostaoliveira commented Jan 6, 2026

Automated performance analysis summary

This comment was automatically generated given there is performance data available.

In summary:

  • Detected a total of 25 stable tests between versions.
  • Detected a total of 7 highly unstable benchmarks (7 baseline).
  • Detected a total of 10 improvements above the improvement water line.
  • Detected a total of 1 regressions bellow the regression water line 8.0%.

You can check a comparison in detail via the grafana link

Performance Improvements - Comparison between master and gd_10751_lto.

Time Period from 30 days ago. (environment used: oss-standalone)

Test Case Baseline master (median obs. +- std.dev) Comparison gd_10751_lto (median obs. +- std.dev) % change (higher-better) Note
search-numeric 2432 +- 3.8% (2 datapoints) 3881 59.6% IMPROVEMENT
ftsb-10K-enwiki_abstract-hashes-fulltext-sortby 90 +- 5.3% (7 datapoints) 105 17.3% IMPROVEMENT
ftsb-1M-enwiki_abstract-hashes-load 23356 +- 7.5% (7 datapoints) 27038 15.8% IMPROVEMENT
ftsb-10K-enwiki_pages-hashes-load 65912 +- 4.2% (7 datapoints) 74282 12.7% IMPROVEMENT
vecsim-arxiv-titles-384-angular-filters-m16-ef-128-fulltext-filter 575 +- 4.3% (7 datapoints) 643 12.0% IMPROVEMENT
ftsb-1M-nyc_taxis-hashes-load 30596 +- 3.5% (7 datapoints) 33816 10.5% IMPROVEMENT
search-ftsb-10K-enwiki_abstract-hashes-fulltext-search-sortby-limit-0-100 924 +- 3.6% (2 datapoints) 1020 10.5% IMPROVEMENT
ftsb-1M-nyc_taxis-ftadd-load 29192 +- 3.2% (7 datapoints) 31820 9.0% IMPROVEMENT
search-ftsb-10K-enwiki_abstract-hashes-fulltext-aggregate-sortby-limit-0-100 917 +- 2.5% (7 datapoints) 995 8.6% IMPROVEMENT
ftsb-10K-singlevalue-numeric-json 485 +- 0.1% (2 datapoints) 525 8.3% IMPROVEMENT

Performance Regressions and Issues - Comparison between master and gd_10751_lto.

Time Period from 30 days ago. (environment used: oss-standalone)

Test Case Baseline master (median obs. +- std.dev) Comparison gd_10751_lto (median obs. +- std.dev) % change (higher-better) Note
search-numeric-sortby-desc-optimize 31 +- 5.2% (7 datapoints) 28 -11.1% REGRESSION
search-filtering-tag-numeric 251 +- 14.5% UNSTABLE (7 datapoints) 254 1.4% UNSTABLE (baseline high variance); server: p50 latency stable; client: client latency stable; neither server nor client side confirms regression
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-union-query 2983 +- 10.4% UNSTABLE (7 datapoints) 3172 6.3% UNSTABLE (baseline high variance); server: FT.SEARCH p50 decreased 6.4% (baseline CV=4.4%); client: OverallQuantiles.allCommands.q50 decreased 9.9% (baseline CV=10.2%); neither server nor client side confirms regression
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-intersection-query 376 +- 19.7% UNSTABLE (7 datapoints) 412 9.6% UNSTABLE (baseline high variance); server: FT.SEARCH p50 decreased 6.1% (baseline CV=10.4%); client: client latency stable; neither server nor client side confirms regression
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-union-query-non-sortable 1036 +- 14.1% UNSTABLE (7 datapoints) 1138 9.8% UNSTABLE (baseline high variance); server: FT.SEARCH p50 decreased 5.4% (baseline CV=3.5%); client: OverallQuantiles.allCommands.q50 decreased 9.0% (baseline CV=11.7%); neither server nor client side confirms regression
ftsb-1M-enwiki_abstract-hashes-fulltext-simple-1word-query 911 +- 20.8% UNSTABLE (7 datapoints) 1018 11.7% UNSTABLE (baseline high variance); server: FT.SEARCH p50 decreased 9.8% (baseline CV=17.8%); client: OverallQuantiles.allCommands.q50 decreased 9.6% (baseline CV=19.3%); neither server nor client side confirms regression
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-intersection-query-non-sortable 37 +- 25.0% UNSTABLE (7 datapoints) 48 28.9% UNSTABLE (baseline high variance); server: FT.SEARCH p50 decreased 9.1% (baseline CV=16.0%); client: OverallQuantiles.allCommands.q50 increased 19.8% (baseline CV=8.6%); only client side confirms regression (server side stable) - insufficient evidence
search-numeric-sortby-desc 2307 +- 33.1% UNSTABLE (7 datapoints) 3992 73.0% UNSTABLE (baseline high variance); server: FT.SEARCH p50 decreased 44.0% (baseline CV=26.2%); client: Latency decreased 42.3% (baseline CV=25.3%); neither server nor client side confirms regression
Tests with No Significant Changes (25 tests)

Tests with No Significant Changes

Test Case Baseline master (median obs. +- std.dev) Comparison gd_10751_lto (median obs. +- std.dev) % change (higher-better) Note
ftsb-10K-enwiki_abstract-hashes-term-prefix 6292 +- 5.5% (7 datapoints) 6665.00 5.9% potential IMPROVEMENT
ftsb-10K-enwiki_abstract-hashes-term-suffix 2236 +- 0.8% (7 datapoints) 2275.00 1.8% No Change
ftsb-10K-enwiki_abstract-hashes-term-suffix-withsuffixtrie 16337 +- 1.2% (7 datapoints) 16438.00 0.6% No Change
ftsb-10K-enwiki_abstract-hashes-term-wildcard 8835 +- 3.2% (7 datapoints) 8409.00 -4.8% potential REGRESSION
ftsb-10K-enwiki_pages-hashes-fulltext-mixed_simple-1word-query_write_1_to_read_20.yml 1014 +- 3.9% (7 datapoints) 1012.00 -0.2% No Change
ftsb-10K-multivalue-numeric-json 997 +- 1.5% (7 datapoints) 1037.00 4.0% potential IMPROVEMENT
ftsb-1K-enwiki_abstract-hashes-term-contains 1960 +- 1.8% (7 datapoints) 2000.00 2.1% No Change
search-aggregate-post-filter-simple.yml 17503 +- 1.1% (7 datapoints) 17471.00 -0.2% No Change
search-filtering-tag-numeric-filter-pipeline 11009 +- 1.3% (7 datapoints) 11371.00 3.3% potential IMPROVEMENT
search-ftsb-10K-enwiki_abstract-hashes-term-withoutsuffix-trie 14376 +- 1.1% (7 datapoints) 14482.00 0.7% No Change
search-ftsb-10K-enwiki_abstract-hashes-term-withsuffix-trie 14402 +- 2.4% (2 datapoints) 15057.00 4.5% potential IMPROVEMENT
search-ftsb-1700K-docs-union-iterators-q3 8.1 +- 1.2% (7 datapoints) 8.50 4.7% potential IMPROVEMENT
search-ftsb-1M-enwiki_abstract-hashes-fulltext-simple-1word-query-non-sortable 164 +- 6.9% (2 datapoints) 156.00 -4.6% potential REGRESSION
search-ftsb-1M-enwiki_abstract-hashes-fulltext-simple-1word-query-one-indexed-field 7371 +- 2.9% (2 datapoints) 7332.00 -0.5% No Change
search-ftsb-370K-docs-union-iterators-q4 8.3 +- 1.3% (7 datapoints) 8.80 5.5% potential IMPROVEMENT
search-ftsb-5200K-docs-union-iterators-q1 0.84 +- 2.0% (7 datapoints) 0.86 2.4% No Change
search-ftsb-5500K-docs-union-iterators-q2 1.2 +- 2.3% (7 datapoints) 1.20 -2.5% No Change
search-geo 223 +- 0.3% (2 datapoints) 237.00 6.2% potential IMPROVEMENT
search-high-cardinality-negation-term-baseline 38 +- 2.0% (7 datapoints) 39.00 2.6% No Change
search-high-cardinality-negation-term-comparison_union_all_other_terms 14 +- 2.7% (7 datapoints) 15.00 6.5% potential IMPROVEMENT
search-numeric-optimize 8133 +- 1.6% (7 datapoints) 8596.00 5.7% potential IMPROVEMENT
search-numeric-sortby 2382 +- 2.3% (2 datapoints) 2358.00 -1.0% No Change
search-numeric-sortby-optimize 26 +- 5.9% (7 datapoints) 26.00 -1.3% No Change
vecsim-arxiv-titles-384-angular-filters-m16-ef-128-numeric-filter 155 +- 9.6% (7 datapoints) 157.00 1.3% waterline=9.6%. No Change
vecsim-arxiv-titles-384-angular-filters-m16-ef-128-tag-filter 15690 +- 1.0% (7 datapoints) 16093.00 2.6% No Change

@gdesmott gdesmott force-pushed the gd_10751_lto branch 6 times, most recently from 370e9bf to 606b383 Compare January 6, 2026 19:10
@github-actions github-actions bot removed the stale label Jan 7, 2026
@gdesmott gdesmott force-pushed the gd_10751_lto branch 2 times, most recently from 5e08f3d to 10d61c0 Compare January 8, 2026 08:37
@gdesmott gdesmott changed the title [MOD-10751] C/Rust LTO [MOD-10751] enable C/Rust LTO in benchmarks Jan 8, 2026
@gdesmott gdesmott changed the base branch from master to gd_build_lto January 8, 2026 11:17
@gdesmott gdesmott force-pushed the gd_10751_lto branch 4 times, most recently from e6ed0eb to 390f528 Compare January 13, 2026 10:13
@gdesmott gdesmott changed the base branch from gd_build_lto to master January 13, 2026 10:13
Comment on lines 441 to 443
LTO_PLATFORMS=("ubuntu:noble")
CONTAINER="${{ needs.get-config.outputs.container }}"
if [[ " ${LTO_PLATFORMS[@]} " =~ " ${CONTAINER} " ]]; then
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just want to confirm: is the intention to enable LTO in the merge queue? Because the current conditional won't enable it on PRs.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Since they don't run in containers)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks I missed that.

I assumed the merge flow was a subset of the "Build on Platforms" one.

Adding a note about that then, we still need to update the build artifacts flow as well.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we know how much longer the build takes when LTO is enabled in CI?

Copy link
Collaborator Author

@gdesmott gdesmott force-pushed the gd_10751_lto branch 2 times, most recently from fbf2f8c to 03cd54c Compare January 19, 2026 13:23
Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

This allow users to pick the right clang version manually by defining
CC/CXX/LD without changing the default clang on their system.
Alpine's grep do not support -P so use sed instead.
Fix building tests with LTO on arm64.
Save us from re-checking if LTO is enabled or not.
Use the "append or define" pattern.
We should be good now that we call `make rust-tests LTO`
Linux distributions may ship either clang, clang-21
or both as binaries.
@gdesmott gdesmott changed the title [MOD-10751] enable C/Rust LTO in benchmarks and CI tests [MOD-10751] build.sh: improve LTO support Jan 19, 2026
@gdesmott
Copy link
Collaborator Author

gdesmott commented Jan 19, 2026

@LukeMathWalker : as discussed, this PR now contains only build.sh changes. The CI integration will be part of MOD-13606 : #8092

@gdesmott gdesmott added this pull request to the merge queue Jan 19, 2026
Merged via the queue into master with commit 9382aa6 Jan 19, 2026
14 checks passed
@gdesmott gdesmott deleted the gd_10751_lto branch January 19, 2026 16:30
eyalrund pushed a commit that referenced this pull request Jan 22, 2026
* build: handle env variables when enabling LTO

This allow users to pick the right clang version manually by defining
CC/CXX/LD without changing the default clang on their system.

* build: use new arg parsing syntax for LTO

* build: fix LTO on Alpine

Alpine's grep do not support -P so use sed instead.

* build: enforce LLVM on Rust tests with LTO

Fix building tests with LTO on arm64.

* build: move up RUSTFLAGS lto

Save us from re-checking if LTO is enabled or not.

* build: simplify RUSTFLAGS declaration

Use the "append or define" pattern.

* build: no longer need to re-define RUST_TEST_COMMAND

We should be good now that we call `make rust-tests LTO`

* build: try using llvm versioned binaries if available

Linux distributions may ship either clang, clang-21
or both as binaries.
LukeMathWalker pushed a commit that referenced this pull request Jan 26, 2026
* build: handle env variables when enabling LTO

This allow users to pick the right clang version manually by defining
CC/CXX/LD without changing the default clang on their system.

* build: use new arg parsing syntax for LTO

* build: fix LTO on Alpine

Alpine's grep do not support -P so use sed instead.

* build: enforce LLVM on Rust tests with LTO

Fix building tests with LTO on arm64.

* build: move up RUSTFLAGS lto

Save us from re-checking if LTO is enabled or not.

* build: simplify RUSTFLAGS declaration

Use the "append or define" pattern.

* build: no longer need to re-define RUST_TEST_COMMAND

We should be good now that we call `make rust-tests LTO`

* build: try using llvm versioned binaries if available

Linux distributions may ship either clang, clang-21
or both as binaries.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants