Skip to content

Commit f65bead

Browse files
test: stabilize watch, coverage and hotspot runs, name unit benchmarks language-component-unit (#22801)
* test: skip a watch build that started before its step, give coverage runs more time A late event of one watch step can start a build just before the next step is written. That build read the old files, and the harness checked it against the new step (cache/unsafe-cache-managed-paths on macOS): count the builds started and skip one that started before the step. Coverage instrumentation makes a build about 1.5 times slower, more for the case that first loads a large file of lib, and the macOS coverage job timed out random ConfigTestCases builds at 30s. jest.config.js marks a coverage run and every template timeout is three times as long there. A build that ends after its test timed out no longer throws on the cleaned-up suite. Claude-Session: https://claude.ai/code/session_017TkBWTk3wcyUcJYDp5ZhaP * test: mark a coverage run with a Jest global, not a WEBPACK_ variable The environment variable reached the builds, and webpack exposes every `WEBPACK_` variable through `import.meta.env`, so the import-meta-env snapshot gained it in coverage runs. Claude-Session: https://claude.ai/code/session_017TkBWTk3wcyUcJYDp5ZhaP * test: read the coverage flag from global, which Node.js 10 has Claude-Session: https://claude.ai/code/session_017TkBWTk3wcyUcJYDp5ZhaP * test: also treat Jest's collectCoverage flag as a coverage run Claude-Session: https://claude.ai/code/session_017TkBWTk3wcyUcJYDp5ZhaP * test: re-send a dropped file change as a fresh file, and re-send it up to five times Claude-Session: https://claude.ai/code/session_017TkBWTk3wcyUcJYDp5ZhaP * build: update @codspeed/core to 6.1.0 Memory mode now also asks for --no-minor-gc-task and, from Node 24, --no-maglev; the orchestrator checks its own flags, so the benchmark script passes both. * test(benchmark): add JS printer and XML parser and printer benchmarks js-printer-unit prints lodash and three.module.js in print, mangle and minify mode. xml-parser-svg-unit and xml-printer-svg-unit mirror the HTML pair over a generated SVG: parse, skip-text parse, visitor walks and tokenizeXml, then walk, beautify and minify over three tree shapes. * test(benchmark): name syntax benchmarks <language>-<component>-unit * test(benchmark): name the minimizer plugin benchmark js-minimizer-plugin-unit Claude-Session: https://claude.ai/code/session_017TkBWTk3wcyUcJYDp5ZhaP * test: pin only BurnPlugin's own entry in the hotspots stats and error cases Claude-Session: https://claude.ai/code/session_017TkBWTk3wcyUcJYDp5ZhaP
1 parent 7175142 commit f65bead

9 files changed

Lines changed: 87 additions & 26 deletions

File tree

‎jest.config.js‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,15 @@
11
"use strict";
22

3+
// coverage instrumentation slows every build; test/helpers/testTimeout.js
4+
// reads this as a global, not from the environment the builds can expose
5+
const coverage = process.argv.some((arg) =>
6+
/^--(coverage|collectCoverage|collect-coverage)(=true)?$/.test(arg)
7+
);
8+
39
/** @type {import("jest").Config} */
410
const config = {
5-
testTimeout: 30000,
11+
testTimeout: coverage ? 90000 : 30000,
12+
globals: { __WEBPACK_TEST_COVERAGE__: coverage },
613
prettierPath: require.resolve("prettier-2"),
714
forceExit: true,
815
globalSetup: "<rootDir>/test/harness/globalSetup.js",

test/benchmarkCases/minimizer-module-path-unit/index.bench.mjs renamed to test/benchmarkCases/js-minimizer-plugin-unit/index.bench.mjs

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -31,13 +31,13 @@ const pathOptions = {
3131
/** @param {import("tinybench").Bench} bench benchmark suite */
3232
export default (bench) => {
3333
bench.add(
34-
'unit benchmark "minimizer-module-path-unit", minify entry by module path',
34+
'unit benchmark "js-minimizer-plugin-unit", minify entry by module path',
3535
async () => {
3636
await minify(pathOptions);
3737
}
3838
);
3939
bench.add(
40-
'unit benchmark "minimizer-module-path-unit", transform entry by function source',
40+
'unit benchmark "js-minimizer-plugin-unit", transform entry by function source',
4141
async () => {
4242
await transform(serialize(options));
4343
}
Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,10 @@
11
"use strict";
22

3-
module.exports = [[/hotspots: 1 thing holds the main thread/]];
3+
module.exports = [
4+
[
5+
// A slow runner can stall another tap past the threshold, so the count
6+
// and the other entries are not pinned, only BurnPlugin's own.
7+
/hotspots: \d+ things? holds? the main thread long enough to be worth looking at/,
8+
/\n {2}plugin BurnPlugin \(\d+ ms over \d+ runs?\)/
9+
]
10+
];

‎test/configCases/performance/hotspots-stats/index.js‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,12 @@
11
it("should report through the stats channel", () => {
22
expect(__STATS__.hints).toHaveLength(1);
3+
// A slow runner can stall another tap past the threshold, so the count and
4+
// the other entries are not pinned, only BurnPlugin's own.
35
expect(__STATS__.hints[0].message).toMatch(
4-
/hotspots: 1 thing holds the main thread/
6+
/hotspots: \d+ things? holds? the main thread/
7+
);
8+
expect(__STATS__.hints[0].message).toMatch(
9+
/\n {2}plugin BurnPlugin \(\d+ ms over \d+ runs?\)/
510
);
611
expect(__STATS__.warnings).toHaveLength(0);
712
expect(__STATS__.errors).toHaveLength(0);

‎test/helpers/testTimeout.js‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
"use strict";
2+
3+
// Coverage instrumentation slows every build, and a case that first loads a
4+
// large file of `lib` also pays for instrumenting it, so a coverage run gets
5+
// more time. jest.config.js sets the global for a coverage run.
6+
// `global`, not `globalThis`, which Node.js 10 does not have
7+
const FACTOR = /** @type {{ __WEBPACK_TEST_COVERAGE__?: boolean }} */ (
8+
/** @type {unknown} */ (global)
9+
).__WEBPACK_TEST_COVERAGE__
10+
? 3
11+
: 1;
12+
13+
/**
14+
* @param {number} ms timeout of a regular run
15+
* @returns {number} timeout of this run
16+
*/
17+
module.exports = (ms) => ms * FACTOR;

‎test/templates/ConfigTestCases.js‎

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@ const filterInfraStructureErrors = require("../helpers/infrastructureLogErrors")
4646
const prepareOptions = require("../helpers/prepareOptions");
4747
const supportsObjectHasOwn = require("../helpers/supportsObjectHasOwn");
4848
const supportsOptionalChaining = require("../helpers/supportsOptionalChaining");
49+
const testTimeout = require("../helpers/testTimeout");
4950

5051
const testRootDirectory = path.join(__dirname, "..");
5152

@@ -105,7 +106,7 @@ const describeCases = (config) => {
105106
stderr.restore();
106107
});
107108

108-
jest.setTimeout(20000);
109+
jest.setTimeout(testTimeout(20000));
109110

110111
for (const category of categories) {
111112
// eslint-disable-next-line no-loop-func
@@ -271,7 +272,9 @@ const describeCases = (config) => {
271272
} catch (_err) {
272273
// ignored
273274
}
274-
if (testConfig.timeout) setDefaultTimeout(testConfig.timeout);
275+
if (testConfig.timeout) {
276+
setDefaultTimeout(testTimeout(testConfig.timeout));
277+
}
275278
generatedCode = collectGeneratedCode(optionsArr, {
276279
assets: testConfig.ecmaConformance === true
277280
});
@@ -403,7 +406,7 @@ const describeCases = (config) => {
403406
done();
404407
});
405408
});
406-
}, 60000);
409+
}, testTimeout(60000));
407410

408411
it(`${testName} should pre-compile to fill disk cache (2nd)`, (done) => {
409412
rimraf.sync(outputDirectory);
@@ -501,7 +504,7 @@ const describeCases = (config) => {
501504
done();
502505
});
503506
});
504-
}, 40000);
507+
}, testTimeout(40000));
505508
}
506509

507510
it(`${testName} should compile`, (done) => {
@@ -514,6 +517,8 @@ const describeCases = (config) => {
514517
/** @type {import("../../").Stats} */ stats
515518
) => {
516519
const deprecations = deprecationTracker();
520+
// a build that ends after its test timed out finds the suite cleaned up
521+
if (generatedCode === undefined) return;
517522
if (err) return handleFatalError(err, done);
518523
const statOptions = {
519524
preset: "verbose",
@@ -740,15 +745,15 @@ const describeCases = (config) => {
740745
/** @type {EXPECTED_ANY} */ (onCompiled)
741746
);
742747
}
743-
}, 30000);
748+
}, testTimeout(30000));
744749

745750
const {
746751
it: _it,
747752
beforeEach: _beforeEach,
748753
afterEach: _afterEach,
749754
setDefaultTimeout,
750755
getNumberOfTests
751-
} = createLazyTestEnv(10000);
756+
} = createLazyTestEnv(testTimeout(10000));
752757
});
753758
}
754759
});

‎test/templates/HotTestCases.js‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ const createLazyTestEnv = require("../helpers/createLazyTestEnv");
2424
const deprecationTracking = require("../helpers/deprecationTracking");
2525
const supportsObjectHasOwn = require("../helpers/supportsObjectHasOwn");
2626
const supportsOptionalChaining = require("../helpers/supportsOptionalChaining");
27+
const testTimeout = require("../helpers/testTimeout");
2728

2829
const testRootDirectory = path.join(__dirname, "..");
2930

@@ -390,14 +391,14 @@ const describeCases = (config) => {
390391
}
391392
);
392393
compiler.run(/** @type {EXPECTED_ANY} */ (onCompiled));
393-
}, 20000);
394+
}, testTimeout(20000));
394395

395396
const {
396397
it: _it,
397398
beforeEach: _beforeEach,
398399
afterEach: _afterEach,
399400
getNumberOfTests
400-
} = createLazyTestEnv(20000);
401+
} = createLazyTestEnv(testTimeout(20000));
401402
});
402403
}
403404
});

‎test/templates/TestCases.js‎

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@ const deprecationTracking = require("../helpers/deprecationTracking");
3939
const filterInfraStructureErrors = require("../helpers/infrastructureLogErrors");
4040
const supportsObjectHasOwn = require("../helpers/supportsObjectHasOwn");
4141
const supportsOptionalChaining = require("../helpers/supportsOptionalChaining");
42+
const testTimeout = require("../helpers/testTimeout");
4243

4344
const testRootDirectory = path.join(__dirname, "..");
4445

@@ -101,7 +102,7 @@ const describeCases = (config) => {
101102
for (const category of categories) {
102103
// eslint-disable-next-line no-loop-func
103104
describe(category.name, () => {
104-
jest.setTimeout(30000);
105+
jest.setTimeout(testTimeout(30000));
105106

106107
for (const testName of category.tests.filter((test) => {
107108
const testDirectory = path.join(casesPath, category.name, test);
@@ -357,7 +358,7 @@ const describeCases = (config) => {
357358
done();
358359
});
359360
},
360-
testConfig.timeout || 60000
361+
testTimeout(testConfig.timeout || 60000)
361362
);
362363

363364
it(
@@ -403,7 +404,9 @@ const describeCases = (config) => {
403404
done();
404405
});
405406
},
406-
testConfig.cachedTimeout || testConfig.timeout || 10000
407+
testTimeout(
408+
testConfig.cachedTimeout || testConfig.timeout || 10000
409+
)
407410
);
408411
}
409412

@@ -522,8 +525,9 @@ const describeCases = (config) => {
522525
}
523526
},
524527
testConfig.cachedTimeout ||
525-
testConfig.timeout ||
526-
(config.cache ? 20000 : 60000)
528+
testTimeout(
529+
testConfig.timeout || (config.cache ? 20000 : 60000)
530+
)
527531
);
528532

529533
it(`${testName} should load the compiled tests`, (done) => {
@@ -556,10 +560,10 @@ const describeCases = (config) => {
556560
}
557561
done();
558562
}, done);
559-
}, 10000);
563+
}, testTimeout(10000));
560564

561565
const { it: _it, getNumberOfTests } = createLazyTestEnv(
562-
testConfig.timeout || 10000
566+
testTimeout(testConfig.timeout || 10000)
563567
);
564568
});
565569
}

‎test/templates/WatchTestCases.js‎

Lines changed: 21 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -31,12 +31,13 @@ const prepareOptions = require("../helpers/prepareOptions");
3131
const { remove } = require("../helpers/remove");
3232
const supportsObjectHasOwn = require("../helpers/supportsObjectHasOwn");
3333
const supportsOptionalChaining = require("../helpers/supportsOptionalChaining");
34+
const testTimeout = require("../helpers/testTimeout");
3435

3536
const testRootDirectory = path.join(__dirname, "..");
3637
const STEP_DELAY = 200;
3738
const AGGREGATE_TIMEOUT = 50;
3839
const RETOUCH_DELAY = 1000;
39-
const RETOUCH_LIMIT = 3;
40+
const RETOUCH_LIMIT = 5;
4041

4142
/**
4243
* @param {string} src src
@@ -325,6 +326,8 @@ const describeCases = (config) => {
325326
// watched files the current step changed which no build has reported yet
326327
/** @type {Set<string>} */
327328
let unseenChanges = new Set();
329+
// builds started so far, and how many had started when the step was written
330+
const buildCount = { started: 0, beforeStep: 0 };
328331

329332
const currentWatchStepModule = require("../helpers/currentWatchStep");
330333

@@ -391,7 +394,12 @@ const describeCases = (config) => {
391394
fs.renameSync(file, away);
392395
fs.renameSync(away, file);
393396
} else if (fs.existsSync(file)) {
394-
fs.utimesSync(file, now, now);
397+
// a new mtime alone went unreported on macOS once (#22801), so the
398+
// file is replaced by a copy, which arrives as a fresh file
399+
const copy = `${file}.retouch`;
400+
fs.copyFileSync(file, copy);
401+
fs.utimesSync(copy, now, now);
402+
fs.renameSync(copy, file);
395403
} else if (fs.existsSync(path.dirname(file))) {
396404
// a dropped deletion is re-sent by deleting again; no build
397405
// runs between the two calls, so none sees the file
@@ -418,6 +426,9 @@ const describeCases = (config) => {
418426
? compiler.compilers
419427
: [compiler];
420428
for (const child of compilers) {
429+
child.hooks.watchRun.tap("WatchTestCasesTest", () => {
430+
buildCount.started++;
431+
});
421432
child.hooks.finishMake.tap(
422433
{ name: "WatchTestCasesTest", stage: Infinity },
423434
assertModuleGraph
@@ -441,6 +452,9 @@ const describeCases = (config) => {
441452
throw new Error("No stats reported from Compiler");
442453
}
443454
if (waitMode) return;
455+
// a late event of the last step can start a build just before this
456+
// step is written; it read the old files, and the new ones start the next
457+
if (buildCount.started <= buildCount.beforeStep) return;
444458
// macOS can report one step's changes over more than the aggregate
445459
// timeout; a build that missed some reuses the stale cached module,
446460
// and the late change starts the build this step's tests wait for
@@ -650,6 +664,7 @@ const describeCases = (config) => {
650664
false,
651665
changed
652666
);
667+
buildCount.beforeStep = buildCount.started;
653668
unseenChanges = new Set();
654669
if (!restartCompiler && watchedCompiler) {
655670
// only a file the last build watched reports its change
@@ -741,7 +756,7 @@ const describeCases = (config) => {
741756
closeCompiler().then(finish);
742757
}
743758
},
744-
45000
759+
testTimeout(45000)
745760
);
746761

747762
finish();
@@ -755,11 +770,11 @@ const describeCases = (config) => {
755770
}
756771
};
757772
timer = setTimeout(startCompiler, 300);
758-
}, 45000);
773+
}, testTimeout(45000));
759774

760775
for (const run of runs) {
761776
const { it: _it, getNumberOfTests } = createLazyTestEnv(
762-
10000,
777+
testTimeout(10000),
763778
run.name
764779
);
765780
run.it = _it;
@@ -787,7 +802,7 @@ const describeCases = (config) => {
787802
it: _it,
788803
beforeEach: _beforeEach,
789804
afterEach: _afterEach
790-
} = createLazyTestEnv(10000);
805+
} = createLazyTestEnv(testTimeout(10000));
791806
});
792807
}
793808
});

0 commit comments

Comments
 (0)