Skip to content

Comments

perf: try reduce max_blocking_threads to save memory and reduce fs contention#9518

Merged
hardfist merged 3 commits intomainfrom
yj/try-reduce-io
Mar 3, 2025
Merged

perf: try reduce max_blocking_threads to save memory and reduce fs contention#9518
hardfist merged 3 commits intomainfrom
yj/try-reduce-io

Conversation

@hardfist
Copy link
Contributor

@hardfist hardfist commented Feb 28, 2025 •
edited by chenjiahan
Loading

Summary

inspired by https://x.com/jarredsumner/status/1893195528764076534
it seems result is not bad, up to 20% faster(bench based on https://github.com/rspack-contrib/build-tools-performance on macOS) with 20% memory reduction, and I think the cause is

  • every thread will allocation 2M memory, even the thread is lazy initialized we still get about hundreds of threads sometimes which bloat memory
  • less thread with less lock contention
    image

Checklist

  • Tests updated (or not required).
  • Documentation updated (or not required).

@github-actions github-actions bot added the team The issue/pr is created by the member of Rspack. label Feb 28, 2025
@netlify
Copy link

netlify bot commented Feb 28, 2025 •
edited
Loading

✅ Deploy Preview for rspack canceled.

Name Link
🔨 Latest commit 7786861
🔍 Latest deploy log https://app.netlify.com/sites/rspack/deploys/67c539c9154ea60008803c44

@hardfist hardfist changed the title chore: try reduce max_blocking_threads to save memory and reduce io c… chore: try reduce max_blocking_threads to save memory and reduce fs contention Feb 28, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Feb 28, 2025 •
edited
Loading

📝 Benchmark detail: Open

Name Base (2025-02-28 2b45815) Current Change
10000_big_production-mode_disable-minimize + exec 36.8 s ± 510 ms 37.6 s ± 1.24 s +2.31 %
10000_development-mode + exec 1.72 s ± 14 ms 1.69 s ± 29 ms -1.55 %
10000_development-mode_hmr + exec 677 ms ± 14 ms 676 ms ± 9.5 ms -0.21 %
10000_production-mode + exec 2.17 s ± 83 ms 2.19 s ± 102 ms +0.95 %
10000_production-mode_persistent-cold + exec 2.32 s ± 61 ms 2.32 s ± 80 ms +0.16 %
10000_production-mode_persistent-hot + exec 1.66 s ± 77 ms 1.63 s ± 43 ms -1.24 %
arco-pro_development-mode + exec 1.71 s ± 83 ms 1.74 s ± 59 ms +1.88 %
arco-pro_development-mode_hmr + exec 376 ms ± 2.2 ms 376 ms ± 2.8 ms +0.04 %
arco-pro_production-mode + exec 3.54 s ± 125 ms 3.5 s ± 63 ms -1.35 %
arco-pro_production-mode_generate-package-json-webpack-plugin + exec 3.62 s ± 290 ms 3.59 s ± 183 ms -0.75 %
arco-pro_production-mode_persistent-cold + exec 3.65 s ± 199 ms 3.59 s ± 115 ms -1.61 %
arco-pro_production-mode_persistent-hot + exec 2.23 s ± 86 ms 2.3 s ± 175 ms +3.07 %
arco-pro_production-mode_traverse-chunk-modules + exec 3.52 s ± 74 ms 3.5 s ± 150 ms -0.40 %
large-dyn-imports_development-mode + exec 1.97 s ± 33 ms 1.96 s ± 65 ms -0.66 %
large-dyn-imports_production-mode + exec 2.04 s ± 50 ms 2.03 s ± 109 ms -0.45 %
threejs_development-mode_10x + exec 1.43 s ± 21 ms 1.44 s ± 18 ms +0.13 %
threejs_development-mode_10x_hmr + exec 775 ms ± 5 ms 787 ms ± 22 ms +1.55 %
threejs_production-mode_10x + exec 5.03 s ± 39 ms 5.06 s ± 142 ms +0.71 %
threejs_production-mode_10x_persistent-cold + exec 5.08 s ± 58 ms 5.11 s ± 25 ms +0.62 %
threejs_production-mode_10x_persistent-hot + exec 4.44 s ± 332 ms 4.4 s ± 103 ms -0.95 %
10000_big_production-mode_disable-minimize + rss memory 8703 MiB ± 150 MiB 8676 MiB ± 53 MiB -0.31 %
10000_development-mode + rss memory 650 MiB ± 17.5 MiB 667 MiB ± 13.4 MiB +2.63 %
10000_development-mode_hmr + rss memory 1420 MiB ± 40.6 MiB 1113 MiB ± 12.4 MiB -21.64 %
10000_production-mode + rss memory 621 MiB ± 19.3 MiB 643 MiB ± 16.7 MiB +3.44 %
10000_production-mode_persistent-cold + rss memory 733 MiB ± 19.3 MiB 756 MiB ± 10.6 MiB +3.16 %
10000_production-mode_persistent-hot + rss memory 705 MiB ± 30.7 MiB 726 MiB ± 21.9 MiB +2.93 %
arco-pro_development-mode + rss memory 565 MiB ± 44.2 MiB 573 MiB ± 23.2 MiB +1.51 %
arco-pro_development-mode_hmr + rss memory 666 MiB ± 85.6 MiB 599 MiB ± 29.1 MiB -10.16 %
arco-pro_production-mode + rss memory 718 MiB ± 15.5 MiB 722 MiB ± 57.3 MiB +0.59 %
arco-pro_production-mode_generate-package-json-webpack-plugin + rss memory 722 MiB ± 46.9 MiB 737 MiB ± 34.3 MiB +2.00 %
arco-pro_production-mode_persistent-cold + rss memory 797 MiB ± 31.3 MiB 789 MiB ± 18 MiB -0.95 %
arco-pro_production-mode_persistent-hot + rss memory 637 MiB ± 23.8 MiB 669 MiB ± 35.1 MiB +5.06 %
arco-pro_production-mode_traverse-chunk-modules + rss memory 719 MiB ± 25.1 MiB 720 MiB ± 45.3 MiB +0.21 %
large-dyn-imports_development-mode + rss memory 654 MiB ± 7.15 MiB 661 MiB ± 5.8 MiB +1.09 %
large-dyn-imports_production-mode + rss memory 534 MiB ± 2.56 MiB 542 MiB ± 8.85 MiB +1.47 %
threejs_development-mode_10x + rss memory 572 MiB ± 13.1 MiB 562 MiB ± 18.6 MiB -1.64 %
threejs_development-mode_10x_hmr + rss memory 1137 MiB ± 129 MiB 1095 MiB ± 43.9 MiB -3.66 %
threejs_production-mode_10x + rss memory 834 MiB ± 48.3 MiB 823 MiB ± 25.5 MiB -1.33 %
threejs_production-mode_10x_persistent-cold + rss memory 947 MiB ± 52.7 MiB 923 MiB ± 39.4 MiB -2.58 %
threejs_production-mode_10x_persistent-hot + rss memory 814 MiB ± 51.9 MiB 788 MiB ± 36.8 MiB -3.11 %

@codspeed-hq
Copy link

codspeed-hq bot commented Feb 28, 2025 •
edited
Loading

CodSpeed Performance Report

Merging #9518 will not alter performance

Comparing yj/try-reduce-io (7786861) with main (3d75d1d)

🎉 Hooray! codspeed-rust just leveled up to 2.7.2!

A heads-up, this is a breaking change and it might affect your current performance baseline a bit. But here's the exciting part - it's packed with new, cool features and promises improved result stability 🥳!
Curious about what's new? Visit our releases page to delve into all the awesome details about this new version.

Summary

✅ 7 untouched benchmarks

@github-actions
Copy link
Contributor

github-actions bot commented Feb 28, 2025 •
edited
Loading

📝 Benchmark detail: Open

Name Base (2025-02-28 2b45815) Current Change
10000_big_production-mode_disable-minimize + exec 36.8 s ± 510 ms 37.8 s ± 498 ms +2.75 %
10000_development-mode + exec 1.72 s ± 14 ms 1.69 s ± 17 ms -1.87 %
10000_development-mode_hmr + exec 677 ms ± 14 ms 675 ms ± 5.9 ms -0.37 %
10000_production-mode + exec 2.17 s ± 83 ms 2.18 s ± 133 ms +0.45 %
10000_production-mode_persistent-cold + exec 2.32 s ± 61 ms 2.31 s ± 86 ms -0.27 %
10000_production-mode_persistent-hot + exec 1.66 s ± 77 ms 1.61 s ± 31 ms -2.55 %
arco-pro_development-mode + exec 1.71 s ± 83 ms 1.78 s ± 70 ms +3.97 %
arco-pro_development-mode_hmr + exec 376 ms ± 2.2 ms 377 ms ± 4.7 ms +0.25 %
arco-pro_production-mode + exec 3.54 s ± 125 ms 3.58 s ± 232 ms +0.98 %
arco-pro_production-mode_generate-package-json-webpack-plugin + exec 3.62 s ± 290 ms 3.56 s ± 262 ms -1.60 %
arco-pro_production-mode_persistent-cold + exec 3.65 s ± 199 ms 3.57 s ± 138 ms -2.26 %
arco-pro_production-mode_persistent-hot + exec 2.23 s ± 86 ms 2.3 s ± 164 ms +3.24 %
arco-pro_production-mode_traverse-chunk-modules + exec 3.52 s ± 74 ms 3.5 s ± 167 ms -0.56 %
large-dyn-imports_development-mode + exec 1.97 s ± 33 ms 1.95 s ± 18 ms -0.96 %
large-dyn-imports_production-mode + exec 2.04 s ± 50 ms 2.03 s ± 146 ms -0.27 %
threejs_development-mode_10x + exec 1.43 s ± 21 ms 1.44 s ± 28 ms +0.39 %
threejs_development-mode_10x_hmr + exec 775 ms ± 5 ms 791 ms ± 32 ms +2.07 %
threejs_production-mode_10x + exec 5.03 s ± 39 ms 5.08 s ± 200 ms +1.06 %
threejs_production-mode_10x_persistent-cold + exec 5.08 s ± 58 ms 5.08 s ± 78 ms +0.04 %
threejs_production-mode_10x_persistent-hot + exec 4.44 s ± 332 ms 4.4 s ± 43 ms -0.98 %
10000_big_production-mode_disable-minimize + rss memory 8703 MiB ± 150 MiB 8682 MiB ± 52 MiB -0.24 %
10000_development-mode + rss memory 650 MiB ± 17.5 MiB 660 MiB ± 25.2 MiB +1.59 %
10000_development-mode_hmr + rss memory 1420 MiB ± 40.6 MiB 1099 MiB ± 53.5 MiB -22.57 %
10000_production-mode + rss memory 621 MiB ± 19.3 MiB 634 MiB ± 35.5 MiB +1.97 %
10000_production-mode_persistent-cold + rss memory 733 MiB ± 19.3 MiB 737 MiB ± 14.9 MiB +0.55 %
10000_production-mode_persistent-hot + rss memory 705 MiB ± 30.7 MiB 734 MiB ± 19.8 MiB +4.17 %
arco-pro_development-mode + rss memory 565 MiB ± 44.2 MiB 579 MiB ± 28.4 MiB +2.44 %
arco-pro_development-mode_hmr + rss memory 666 MiB ± 85.6 MiB 632 MiB ± 45.4 MiB -5.15 %
arco-pro_production-mode + rss memory 718 MiB ± 15.5 MiB 714 MiB ± 14.9 MiB -0.63 %
arco-pro_production-mode_generate-package-json-webpack-plugin + rss memory 722 MiB ± 46.9 MiB 731 MiB ± 27.3 MiB +1.19 %
arco-pro_production-mode_persistent-cold + rss memory 797 MiB ± 31.3 MiB 789 MiB ± 36 MiB -0.97 %
arco-pro_production-mode_persistent-hot + rss memory 637 MiB ± 23.8 MiB 660 MiB ± 41.4 MiB +3.65 %
arco-pro_production-mode_traverse-chunk-modules + rss memory 719 MiB ± 25.1 MiB 720 MiB ± 22.8 MiB +0.12 %
large-dyn-imports_development-mode + rss memory 654 MiB ± 7.15 MiB 658 MiB ± 6.55 MiB +0.61 %
large-dyn-imports_production-mode + rss memory 534 MiB ± 2.56 MiB 541 MiB ± 4.46 MiB +1.31 %
threejs_development-mode_10x + rss memory 572 MiB ± 13.1 MiB 560 MiB ± 11.4 MiB -2.06 %
threejs_development-mode_10x_hmr + rss memory 1137 MiB ± 129 MiB 1096 MiB ± 37.6 MiB -3.57 %
threejs_production-mode_10x + rss memory 834 MiB ± 48.3 MiB 838 MiB ± 41.6 MiB +0.49 %
threejs_production-mode_10x_persistent-cold + rss memory 947 MiB ± 52.7 MiB 908 MiB ± 29.1 MiB -4.11 %
threejs_production-mode_10x_persistent-hot + rss memory 814 MiB ± 51.9 MiB 811 MiB ± 52.2 MiB -0.37 %

@hardfist hardfist marked this pull request as ready for review March 3, 2025 05:38
@hardfist hardfist changed the title chore: try reduce max_blocking_threads to save memory and reduce fs contention perf: try reduce max_blocking_threads to save memory and reduce fs contention Mar 3, 2025
@github-actions github-actions bot added the release: performance release: performance related release(mr only) label Mar 3, 2025
@hardfist hardfist requested a review from h-a-n-a March 3, 2025 05:41
@hardfist hardfist merged commit 832bc69 into main Mar 3, 2025
29 checks passed
@hardfist hardfist deleted the yj/try-reduce-io branch March 3, 2025 05:49
@IWANABETHATGUY IWANABETHATGUY mentioned this pull request Mar 3, 2025
Closed
@Brooooooklyn Brooooooklyn mentioned this pull request Mar 10, 2025
Closed
Copilot AI mentioned this pull request Aug 2, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@h-a-n-a h-a-n-a h-a-n-a approved these changes

Assignees

No one assigned

Labels

release: performance release: performance related release(mr only) team The issue/pr is created by the member of Rspack.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@hardfist @h-a-n-a