Skip to content

Commit cf1b834

Browse files
test: split minify-corpora by a hash of each source's name, parts only in CI
No timings to keep current: `isInPart(key)` gives each source a part by a hash of its name. `--parts=3` moves from the package.json scripts to the spec (minify-corpora) leg, so a local run stays one part. Claude-Session: https://claude.ai/code/session_01U5pSVQzezanpJSaZ6q2kCi
1 parent 6087b77 commit cf1b834

7 files changed

Lines changed: 28 additions & 433 deletions

File tree

‎.github/workflows/test.yml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -350,10 +350,11 @@ jobs:
350350
flags: test262-parser
351351
# Minifies every source of each JS corpus with the published minifier
352352
# and with webpack's own: each output prints what its input prints.
353-
# About 35 min, so it gets the longer bound below.
353+
# About 35 min of work, run in three parts at once on the runner's
354+
# cores (`--parts`, TESTING_DOCS.md); it keeps the longer bound below.
354355
- language: minify-corpora
355356
submodules: test/external/terser test/external/test262-cases test/external/swc test/external/oxc
356-
script: cover:minify
357+
script: cover:minify --parts=3
357358
flags: minify-corpora
358359
timeout: 90
359360
# Runs webpack's fuzzer on fixed seeds, apart from the corpora: run

‎TESTING_DOCS.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -118,7 +118,7 @@ Directories come first, in alphabetical order, then the individual files worth t
118118
- `syntaxEquivalence.spectest.js` — `yarn test:syntax-equivalence`
119119
- `cssParsing-webpack.spectest.js` — `yarn test:css-parsing`
120120
- `xmlconf.spectest.js` — `yarn test:xmlconf`
121-
- `minify-corpora.spectest.js` — `yarn test:minify-corpora`, in three [parts](#splitting-one-file-across-workers) its corpus groups are dealt to by `minify-corpora-timings.json`; the last part to finish checks no table entry went unmet
121+
- `minify-corpora.spectest.js` — `yarn test:minify-corpora`; CI runs it in three [parts](#splitting-one-file-across-workers), each minifying the sources whose name hashes to it, and the last part to finish checks no table entry went unmet
122122
- `js-fuzz.spectest.js` — `yarn test:js-fuzz`: programs from `test/helpers/jsFuzzProgram.js`, in the sloppy ES5 terser's ufuzz writes, in modern syntax, as modules importing a fixed `dep` and as webpack-shaped bundles, minified under ufuzz's option sets, webpack's and random ones; the parser is held to acorn's tree on each program and output, the printer to the tree it read and to its own output, each output to what its program prints and exports, the default set's source map to its output and source, and the output to what it prints minified again. Needs no submodule; `JS_FUZZ_SEED=<n>` replays one seed, `JS_FUZZ_SEED=random` runs a new one named in the test's title, and `JS_FUZZ_PROGRAMS` sets the programs per seed.
123123

124124
### `statsCases/`
@@ -197,7 +197,7 @@ During the test run, webpack compiles this project and compares the result with
197197

198198
### Splitting one file across workers
199199

200-
Jest runs each file in one worker, so a long suite in one file leaves the runner's other cores idle. `--parts=<n>` (read by `jest.config.js`) runs every matched file in `n` workers at once, as `n` projects whose coverage merges as usual; global setup and teardown run once. A file splits its own work with `test/helpers/testParts.js`: `keysOfPart(keys, weights)` returns the keys this part runs, dealt by measured weight so the parts end together, and `currentPart()` returns `[part, parts]`, e.g. to run a check once in part 1. Without the flag a file is one part and runs everything; a file that doesn't use the helper runs whole in every part, so pass the flag only to a script that runs a file using it.
200+
Jest runs each file in one worker, so a long suite in one file leaves the runner's other cores idle. `--parts=<n>` (read by `jest.config.js`) runs every matched file in `n` workers at once, as `n` projects whose coverage merges as usual; global setup and teardown run once. A file splits its own work with `test/helpers/testParts.js`: `isInPart(key)` tells whether this part runs the work named `key`, by a hash of the name, so nothing to measure or store, and `currentPart()` returns `[part, parts]`, e.g. to run a check once in part 1. Without the flag a file is one part and runs everything; a file that doesn't use the helper runs whole in every part, so pass the flag only where it runs a file using it — e.g. CI's `yarn cover:minify --parts=3`.
201201

202202
### Suites that drive a real browser
203203

‎cspell.json‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -638,7 +638,6 @@
638638
"yarn.lock",
639639
"**/package-lock.json",
640640
"tooling/deep-webpack-imports.json",
641-
"test/specCases/minify-corpora-timings.json",
642641
"tooling/comparison/css-tool-comparison/stylesheets.json",
643642
"types.d.ts",
644643
"**/**/*.snap",

‎package.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -89,7 +89,7 @@
8989
"test:integration:b": "yarn test:base --testMatch \"<rootDir>/test/*.longtest.js\"",
9090
"test:test262": "yarn test:base --testMatch \"<rootDir>/test/specCases/test262.spectest.js\"",
9191
"test:test262-parser": "yarn test:base --testMatch \"<rootDir>/test/specCases/test262-parser.spectest.js\"",
92-
"test:minify-corpora": "yarn test:base --parts=3 --testMatch \"<rootDir>/test/specCases/minify-corpora.spectest.js\"",
92+
"test:minify-corpora": "yarn test:base --testMatch \"<rootDir>/test/specCases/minify-corpora.spectest.js\"",
9393
"test:js-fuzz": "yarn test:base --testMatch \"<rootDir>/test/specCases/js-fuzz.spectest.js\"",
9494
"test:html5lib": "yarn test:base --testMatch \"<rootDir>/test/specCases/html5lib.spectest.js\"",
9595
"test:syntax-equivalence": "yarn test:base --testMatch \"<rootDir>/test/specCases/syntaxEquivalence.spectest.js\"",
@@ -118,7 +118,7 @@
118118
"cover:html5lib": "yarn cover:base --testMatch \"<rootDir>/test/specCases/html5lib.spectest.js\" --coverage",
119119
"cover:syntax-equivalence": "yarn cover:base --testMatch \"<rootDir>/test/specCases/syntaxEquivalence.spectest.js\" --coverage",
120120
"cover:css-parsing": "yarn cover:base --testMatch \"<rootDir>/test/specCases/cssParsing-webpack.spectest.js\" --coverage",
121-
"cover:minify": "yarn cover:base --parts=3 --testMatch \"<rootDir>/test/specCases/minify-corpora.spectest.js\" --coverage",
121+
"cover:minify": "yarn cover:base --testMatch \"<rootDir>/test/specCases/minify-corpora.spectest.js\" --coverage",
122122
"cover:minify-fuzz": "yarn cover:base --testMatch \"<rootDir>/test/specCases/js-fuzz.spectest.js\" --coverage",
123123
"cover:test262-parser": "yarn cover:base --testMatch \"<rootDir>/test/specCases/test262-parser.spectest.js\" --coverage",
124124
"cover:xmlconf": "yarn cover:base --testMatch \"<rootDir>/test/specCases/xmlconf.spectest.js\" --coverage",

‎test/helpers/testParts.js‎

Lines changed: 12 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -10,29 +10,20 @@ const currentPart = () =>
1010
.__TEST_PART__ || [1, 1];
1111

1212
/**
13-
* Picks the keys this part runs, dealing them longest first, each to the part
14-
* with the least weight so far; a key missing from `weights` counts as their median.
15-
* @param {string[]} keys every key, in any order
16-
* @param {Record<string, number>} weights each key's measured cost, e.g. milliseconds
17-
* @returns {Set<string>} the keys of this part
13+
* Tells whether this part runs the work named `key`, by a hash of the name, so
14+
* a key keeps its part however the others around it change.
15+
* @param {string} key a name unique to one piece of work
16+
* @returns {boolean} true when this part runs it
1817
*/
19-
const keysOfPart = (keys, weights) => {
18+
const isInPart = (key) => {
2019
const [part, parts] = currentPart();
21-
if (parts === 1) return new Set(keys);
22-
const known = Object.values(weights).sort((a, b) => a - b);
23-
const fallback = known[Math.floor(known.length / 2)] || 1;
24-
const weighted = keys
25-
.map((key) => ({ key, weight: weights[key] || fallback }))
26-
.sort((a, b) => b.weight - a.weight || (a.key < b.key ? -1 : 1));
27-
const loads = Array.from({ length: parts }, () => 0);
28-
/** @type {Set<string>} */
29-
const mine = new Set();
30-
for (const { key, weight } of weighted) {
31-
const target = loads.indexOf(Math.min(...loads));
32-
loads[target] += weight;
33-
if (target === part - 1) mine.add(key);
20+
if (parts === 1) return true;
21+
// FNV-1a
22+
let hash = 0x811c9dc5;
23+
for (let i = 0; i < key.length; i++) {
24+
hash = Math.imul(hash ^ key.charCodeAt(i), 0x01000193);
3425
}
35-
return mine;
26+
return (hash >>> 0) % parts === part - 1;
3627
};
3728

38-
module.exports = { currentPart, keysOfPart };
29+
module.exports = { currentPart, isInPart };

0 commit comments

Comments
 (0)