[code-infra] Reduce release:build concurrency to mitigate tsgo OOM kills#23139
Merged
Conversation
test_e2e/test_e2e_react_18/test_package build packages on a `medium` resource_class with `pnpm release:build`, which now emits type declarations via tsgo (mui#22826) instead of tsc. When several large packages' tsgo processes (x-charts, x-data-grid, x-date-pickers) overlap under --concurrency 4, combined memory use has been exceeding the container's budget and the OOM killer SIGKILLs one of them -- seen intermittently on master and various PRs, always on x-charts. Lower concurrency to 2 to reduce how many tsgo processes run at once.
LukasTy
added
type: enhancement
Deploy previewBundle size
PerformanceTotal duration: 1,672.55 ms +8.87 ms(+0.5%) | Renders: 63 (+0) | Paint: 2,311.09 ms +47.73 ms(+2.1%)
…and 4 more (+17 within noise) — details Metric alarms
Check out the code infra dashboard for more information about this PR. |
LukasTy
changed the title
Contributor
There was a problem hiding this comment.
Pull request overview
Reduces parallelism in CircleCI package builds to mitigate intermittent OOM SIGKILLs during tsgo declaration emit (notably affecting x-charts) in release:build steps.
Changes:
- Lower
test_e2ebuild step concurrency from4to2(pnpm release:build --concurrency 2). - Add an explicit
--concurrency 2to thetest_packagebuild step (previously relied on the script default / unbounded behavior).
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Contributor
|
In the summary, can you also add how much time increase or decrease was it because of the change ? |
Member
Author
|
For now it looks very close - in the realm of error. |
Member
Author
|
Added a chart with results for now. |
Member
Author
|
@Janpot or @brijeshb42 are you good moving forward with this? |
brijeshb42
approved these changes
Jul 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
test_e2e/test_e2e_react_18/test_packagebuild packages on amedium/medium.gen2CircleCI resource class viapnpm release:build, which since [code-infra] Port codebase to use tsgo #22826 emits type declarations withtsgoinstead oftsc.x-charts,x-data-grid,x-date-pickers) run theirtsgodeclaration-emit concurrently, combined memory use has been exceeding the container's budget and the OOM killer SIGKILLs one of them -- alwaysx-chartsin the failures observed so far.masterand various PRs (e.g. https://circleci.com/gh/mui/mui-x/880430 on master, https://circleci.com/gh/mui/mui-x/880470 on a PR), always with the same signature:Command was killed with SIGKILL (Forced termination)duringtsgodeclaration emit forx-charts.release:buildconcurrency from 4 to 2 intest_e2eand adds an explicit--concurrency 2totest_package(previously unbounded) to reduce how manytsgoprocesses run at once, as a first mitigation to test whether it resolves the flakiness.Test plan
test_e2e,test_e2e_react_18, andtest_packageCI jobs on this PR over several re-runs to confirm theBuild packagesstep no longer gets OOM-killedResults so far
No OOM kill on the retrigger run with concurrency 2 (all three jobs green). Runtime vs. the concurrency-4 baseline is close, not a clear regression in either direction:
Build packagessteptest_e2etest_e2etest_e2e_react_18test_e2e_react_18test_packagetest_packageCaveat: one data point per configuration so far, and the OOM was already intermittent at concurrency 4 (it didn't fail every run), so this isn't conclusive proof yet -- more re-runs would help confirm before merging.