Skip to content

Commit 959b4c8

Browse files
build(ci): retry a setup step that fails on a transient network error (#22212)
* build(ci): retry a setup step that fails on a transient network error Every dependency install and browser download in CI ran once, so a registry 503 or a dropped connection failed the job and only a human re-run recovered it. Both happened today: `yarn --frozen-lockfile` took `503 Service Unavailable` on three tarballs across four jobs, and `install-firefox.js` took `ECONNRESET`. `tooling/retry.js` runs a command again when it exits non-zero, backing off between attempts and propagating the last exit code, so a genuine failure still fails. It uses node builtins only, because every caller runs before `yarn install` has written `node_modules`, and it stays on the Node 10 baseline the legacy jobs run it under. Only network steps go through it; `yarn link` is local and is left alone. * build(ci): refuse an invalid retry setting and retry submodule fetches `RETRY_ATTEMPTS=abc` reached the loop as `NaN`, where every `attempt >= ATTEMPTS` is false, so a failing command retried until the job timed out — the opposite of what the script is for. A non-integer, zero or negative count, and a non-finite or negative delay, are now refused by name before anything runs. The `git submodule update` fetches go through the runner as well, which is the same transient failure the rest of the wiring is about and what AGENTS.md already claimed. `retry.js` exports its pieces and runs only as the entry point, the way `generate-css-data.js` does, so `test/unitCases/Retry.unittest.js` can drive them: the two readers, a success, a retry that then succeeds, an exhausted run keeping its exit code, a single allowed attempt, and a spawn error rejecting rather than retrying. * build(ci): keep the backoff inside the range a timer accepts Node clamps a `setTimeout` above 2**31-1 ms to 1 ms, so a large `RETRY_DELAY`, or enough doublings of a smaller one, turned the longest backoff into an immediate retry — measured: a 2147483648 ms wait fired after 9 ms. `backoffFor` clamps the computed wait instead, and is covered for both the doubling and the ceiling. * build(ci): cover the entry point the retry runner is invoked through The `require.main === module` block held the whole reading of `process.argv` and `process.env`, which a unit test cannot reach and a spawned process does not report, so those lines stood uncovered — 83.67% of the diff against a 90% target. `main` reads the arguments and settings, `runAsScript` turns what it returns or throws into an exit code, and the guard is the one line neither can be asked about. Both are covered, including a refused setting reported rather than thrown, so the file reads 98.3% of statements and every line and function. What the command line does is unchanged for a success, a failure it keeps the code of, an invalid setting and no arguments at all. * build(ci): retry the install the invariants job added The `invariants` job landed on main while this branch was open, and its `yarn --frozen-lockfile` was the one network step not going through the runner — which the guide now states as an invariant.
1 parent b62935a commit 959b4c8

9 files changed

Lines changed: 420 additions & 40 deletions

File tree

‎.github/workflows/benchmarks.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,7 @@ jobs:
4848
node-version: lts/*
4949
cache: yarn
5050

51-
- run: yarn --frozen-lockfile
51+
- run: node tooling/retry.js yarn --frozen-lockfile
5252

5353
- run: yarn link --frozen-lockfile || true
5454

@@ -96,7 +96,7 @@ jobs:
9696
node-version: lts/*
9797
cache: yarn
9898

99-
- run: yarn --frozen-lockfile
99+
- run: node tooling/retry.js yarn --frozen-lockfile
100100

101101
- run: yarn link --frozen-lockfile || true
102102

‎.github/workflows/code-size.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ jobs:
3333
node-version: lts/*
3434
cache: yarn
3535

36-
- run: yarn --frozen-lockfile
36+
- run: node tooling/retry.js yarn --frozen-lockfile
3737

3838
- run: yarn link --frozen-lockfile || true
3939

‎.github/workflows/examples.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ jobs:
2424
node-version: lts/*
2525
cache: yarn
2626

27-
- run: yarn --frozen-lockfile
27+
- run: node tooling/retry.js yarn --frozen-lockfile
2828

2929
- run: yarn link --frozen-lockfile || true
3030

‎.github/workflows/publish-to-pkg-pr-new.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ jobs:
3030
with:
3131
node-version: lts/*
3232
cache: yarn
33-
- run: yarn --frozen-lockfile
33+
- run: node tooling/retry.js yarn --frozen-lockfile
3434
- run: npx pkg-pr-new publish --compact --json output.json --comment=off
3535
- name: Add metadata to output
3636
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0

‎.github/workflows/release.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ jobs:
2929
node-version: lts/*
3030
cache: yarn
3131

32-
- run: yarn --frozen-lockfile
32+
- run: node tooling/retry.js yarn --frozen-lockfile
3333

3434
- name: Create Release Pull Request or Publish to npm
3535
id: changesets

‎.github/workflows/test.yml‎

Lines changed: 33 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ jobs:
2727
node-version: lts/*
2828
cache: yarn
2929

30-
- run: yarn --frozen-lockfile
30+
- run: node tooling/retry.js yarn --frozen-lockfile
3131

3232
- name: Cache prettier result
3333
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
@@ -54,8 +54,8 @@ jobs:
5454

5555
- name: Validate types using old typescript version
5656
run: |
57-
yarn upgrade [email protected] @types/node@20
58-
yarn --frozen-lockfile
57+
node tooling/retry.js yarn upgrade [email protected] @types/node@20
58+
node tooling/retry.js yarn --frozen-lockfile
5959
yarn validate:types
6060
6161
- name: Validate changeset format
@@ -78,7 +78,7 @@ jobs:
7878
node-version: lts/*
7979
cache: yarn
8080

81-
- run: yarn --frozen-lockfile
81+
- run: node tooling/retry.js yarn --frozen-lockfile
8282

8383
- run: yarn link --frozen-lockfile || true
8484

@@ -115,7 +115,7 @@ jobs:
115115
node-version: lts/*
116116
cache: yarn
117117

118-
- run: yarn --frozen-lockfile
118+
- run: node tooling/retry.js yarn --frozen-lockfile
119119

120120
- name: CSS printer invariants
121121
run: yarn test:invariants:css
@@ -143,7 +143,7 @@ jobs:
143143

144144
# `--ignore-engines`: some transitive deps (e.g. node-releases) declare
145145
# `engines.node >= 18` but run fine on old Node; matches the other legacy jobs
146-
- run: yarn install --production --frozen-lockfile --ignore-engines
146+
- run: node tooling/retry.js yarn install --production --frozen-lockfile --ignore-engines
147147

148148
basic:
149149
runs-on: ubuntu-latest
@@ -156,7 +156,7 @@ jobs:
156156
node-version: lts/*
157157
cache: yarn
158158

159-
- run: yarn --frozen-lockfile
159+
- run: node tooling/retry.js yarn --frozen-lockfile
160160

161161
- run: yarn link --frozen-lockfile || true
162162

@@ -175,7 +175,7 @@ jobs:
175175
node-version: lts/*
176176
cache: yarn
177177

178-
- run: yarn --frozen-lockfile
178+
- run: node tooling/retry.js yarn --frozen-lockfile
179179

180180
- run: yarn link --frozen-lockfile || true
181181

@@ -210,7 +210,7 @@ jobs:
210210
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
211211

212212
- name: Check out the test262 corpus
213-
run: git submodule update --init --depth 1 test/external/test262-cases
213+
run: node tooling/retry.js git submodule update --init --depth 1 test/external/test262-cases
214214

215215
- name: Use Node.js
216216
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
@@ -219,7 +219,7 @@ jobs:
219219
node-version: "26.2.0"
220220
cache: yarn
221221

222-
- run: yarn --frozen-lockfile
222+
- run: node tooling/retry.js yarn --frozen-lockfile
223223

224224
- run: yarn link --frozen-lockfile || true
225225

@@ -264,15 +264,15 @@ jobs:
264264
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
265265

266266
- name: Check out the test262 corpus
267-
run: git submodule update --init --depth 1 test/external/test262-cases
267+
run: node tooling/retry.js git submodule update --init --depth 1 test/external/test262-cases
268268

269269
- name: Use Node.js
270270
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
271271
with:
272272
node-version: lts/*
273273
cache: yarn
274274

275-
- run: yarn --frozen-lockfile
275+
- run: node tooling/retry.js yarn --frozen-lockfile
276276

277277
- run: yarn link --frozen-lockfile || true
278278

@@ -298,15 +298,15 @@ jobs:
298298
# WPT is 161k files, so only this job checks it out, and only one commit
299299
# deep — it carries the tree-construction corpus html5lib-tests dropped.
300300
- name: Check out the html5lib and WPT corpora
301-
run: git submodule update --init --depth 1 test/external/html5lib-tests test/external/wpt
301+
run: node tooling/retry.js git submodule update --init --depth 1 test/external/html5lib-tests test/external/wpt
302302

303303
- name: Use Node.js
304304
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
305305
with:
306306
node-version: lts/*
307307
cache: yarn
308308

309-
- run: yarn --frozen-lockfile
309+
- run: node tooling/retry.js yarn --frozen-lockfile
310310

311311
- run: yarn link --frozen-lockfile || true
312312

@@ -364,15 +364,15 @@ jobs:
364364
# Held to both corpora: webpack's own fixtures and the web's. WPT is 161k
365365
# files, so it is checked out one commit deep.
366366
- name: Check out the WPT corpus
367-
run: git submodule update --init --depth 1 test/external/wpt
367+
run: node tooling/retry.js git submodule update --init --depth 1 test/external/wpt
368368

369369
- name: Use Node.js
370370
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
371371
with:
372372
node-version: lts/*
373373
cache: yarn
374374

375-
- run: yarn --frozen-lockfile
375+
- run: node tooling/retry.js yarn --frozen-lockfile
376376

377377
- run: yarn link --frozen-lockfile || true
378378

@@ -408,13 +408,13 @@ jobs:
408408
# key that never moves would pin the engine this suite exists to track.
409409
- name: Install Firefox
410410
if: matrix.browser == 'firefox'
411-
run: yarn setup:firefox
411+
run: node tooling/retry.js yarn setup:firefox
412412

413413
# `--with-deps` because a runner carries none of WebKit's shared
414414
# libraries, where it already carries Chrome's.
415415
- name: Install WebKit
416416
if: matrix.browser == 'webkit'
417-
run: node node_modules/playwright-core/cli.js install --with-deps webkit
417+
run: node tooling/retry.js node node_modules/playwright-core/cli.js install --with-deps webkit
418418

419419
- name: Cache jest result
420420
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
@@ -444,15 +444,15 @@ jobs:
444444
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
445445

446446
- name: Check out the css-parsing corpus
447-
run: git submodule update --init --depth 1 test/external/css-parsing-tests
447+
run: node tooling/retry.js git submodule update --init --depth 1 test/external/css-parsing-tests
448448

449449
- name: Use Node.js
450450
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
451451
with:
452452
node-version: lts/*
453453
cache: yarn
454454

455-
- run: yarn --frozen-lockfile
455+
- run: node tooling/retry.js yarn --frozen-lockfile
456456

457457
- run: yarn link --frozen-lockfile || true
458458

@@ -505,7 +505,7 @@ jobs:
505505
with:
506506
bun-version: latest
507507

508-
- run: yarn --frozen-lockfile
508+
- run: node tooling/retry.js yarn --frozen-lockfile
509509

510510
- run: yarn link --frozen-lockfile || true
511511

@@ -588,36 +588,36 @@ jobs:
588588

589589
# Install old `jest` version and deps for legacy node versions
590590
- run: |
591-
yarn upgrade jest@^27.5.0 jest-circus@^27.5.0 jest-cli@^27.5.0 jest-diff@^27.5.0 jest-environment-node@^27.5.0 jest-snapshot@^27.5.0 jest-junit@^13.0.0 @types/jest@^27.4.0 pretty-format@^27.0.2 husky@^8.0.3 lint-staged@^13.2.1 cspell@^6.31.1 open-cli@^7.2.0 coffee-loader@^1.0.0 babel-loader@^8.1.0 style-loader@^2.0.0 css-loader@^5.0.1 less-loader@^8.1.1 [email protected] mini-css-extract-plugin@^1.6.1 nyc@^15.1.0 [email protected] toml@^4.3.0 --ignore-engines
592-
yarn --frozen-lockfile --ignore-engines
591+
node tooling/retry.js yarn upgrade jest@^27.5.0 jest-circus@^27.5.0 jest-cli@^27.5.0 jest-diff@^27.5.0 jest-environment-node@^27.5.0 jest-snapshot@^27.5.0 jest-junit@^13.0.0 @types/jest@^27.4.0 pretty-format@^27.0.2 husky@^8.0.3 lint-staged@^13.2.1 cspell@^6.31.1 open-cli@^7.2.0 coffee-loader@^1.0.0 babel-loader@^8.1.0 style-loader@^2.0.0 css-loader@^5.0.1 less-loader@^8.1.1 [email protected] mini-css-extract-plugin@^1.6.1 nyc@^15.1.0 [email protected] toml@^4.3.0 --ignore-engines
592+
node tooling/retry.js yarn --frozen-lockfile --ignore-engines
593593
if: matrix.node-version == '10.x' || matrix.node-version == '12.x' || matrix.node-version == '14.x'
594594
595595
- run: |
596-
yarn upgrade jest@^27.5.0 jest-circus@^27.5.0 jest-cli@^27.5.0 jest-diff@^27.5.0 jest-environment-node@^27.5.0 jest-snapshot@^27.5.0 jest-junit@^13.0.0 @types/jest@^27.4.0 pretty-format@^27.0.2 husky@^8.0.3 lint-staged@^13.2.1 nyc@^15.1.0 [email protected] babel-loader@^8.1.0 style-loader@^2.0.0 css-loader@^5.0.1 less-loader@^8.1.1 mini-css-extract-plugin@^1.6.1 toml@^4.3.0 --ignore-engines
597-
yarn --frozen-lockfile
596+
node tooling/retry.js yarn upgrade jest@^27.5.0 jest-circus@^27.5.0 jest-cli@^27.5.0 jest-diff@^27.5.0 jest-environment-node@^27.5.0 jest-snapshot@^27.5.0 jest-junit@^13.0.0 @types/jest@^27.4.0 pretty-format@^27.0.2 husky@^8.0.3 lint-staged@^13.2.1 nyc@^15.1.0 [email protected] babel-loader@^8.1.0 style-loader@^2.0.0 css-loader@^5.0.1 less-loader@^8.1.1 mini-css-extract-plugin@^1.6.1 toml@^4.3.0 --ignore-engines
597+
node tooling/retry.js yarn --frozen-lockfile
598598
if: matrix.node-version == '16.x'
599599
600600
- run: |
601-
yarn upgrade cspell@^8.8.4 lint-staged@^15.2.5 toml@^4.3.0 --ignore-engines
602-
yarn --frozen-lockfile
601+
node tooling/retry.js yarn upgrade cspell@^8.8.4 lint-staged@^15.2.5 toml@^4.3.0 --ignore-engines
602+
node tooling/retry.js yarn --frozen-lockfile
603603
if: matrix.node-version == '18.x'
604604
605605
- run: |
606-
yarn upgrade cspell@^9 --ignore-engines
607-
yarn --frozen-lockfile
606+
node tooling/retry.js yarn upgrade cspell@^9 --ignore-engines
607+
node tooling/retry.js yarn --frozen-lockfile
608608
if: matrix.node-version == '20.x'
609609
610610
- run: |
611-
yarn upgrade [email protected]
612-
yarn --frozen-lockfile
611+
node tooling/retry.js yarn upgrade [email protected]
612+
node tooling/retry.js yarn --frozen-lockfile
613613
if: matrix.node-version == '22.x'
614614
615615
# Install main version of our deps
616-
- run: yarn upgrade enhanced-resolve@webpack/enhanced-resolve#main loader-runner@webpack/loader-runner#main webpack-sources@webpack/webpack-sources#main watchpack@webpack/watchpack#main tapable@webpack/tapable#main
616+
- run: node tooling/retry.js yarn upgrade enhanced-resolve@webpack/enhanced-resolve#main loader-runner@webpack/loader-runner#main webpack-sources@webpack/webpack-sources#main watchpack@webpack/watchpack#main tapable@webpack/tapable#main
617617
if: matrix.use_main_branches == '1'
618618

619619
# Install dependencies for LTS node versions
620-
- run: yarn --frozen-lockfile
620+
- run: node tooling/retry.js yarn --frozen-lockfile
621621
if: matrix.node-version != '10.x' && matrix.node-version != '12.x' && matrix.node-version != '14.x' && matrix.node-version != '16.x' && matrix.node-version != '18.x' && matrix.node-version != '20.x' && matrix.node-version != '22.x'
622622

623623
- run: yarn link --frozen-lockfile || true

‎AGENTS.md‎

Lines changed: 1 addition & 1 deletion
Large diffs are not rendered by default.

0 commit comments

Comments
 (0)