Skip to content

Commit 4ac3fe0

Browse files
build: declare each comparison corpus as a manifest with a lockfile (#22237)
* build: declare each comparison corpus as a manifest with a lockfile The pinned lists lived in the scripts, so only the top level was pinned: npm still resolved every transitive dependency fresh, and nothing told anyone when a pin went stale. Each corpus is now tooling/comparison/<name>/, a package.json pinning what the script asked for and the package-lock.json freezing what it resolved to, installed with npm ci and skipped while the lockfile it was installed from still matches. Dependabot watches the three directories, so a bump arrives as its own pull request and a version that breaks a printer invariant reds that run rather than an unrelated one. CI keys its cache on the lockfiles, which change exactly when the corpus does. * build(ci): exempt the comparison corpora's unlicensed packages from dependency review The action reviews every manifest a pull request changes, so committing the corpus lockfiles put their packages under webpack's own license policy. Five declare no license in their metadata — coa, dom-walk and three @minify-html platform binaries — rather than one the policy refuses, and none reaches a build. The action offers no way to scope itself to a manifest, so naming them in the list the repository already keeps for this is what there is. * Revert "build(ci): exempt the comparison corpora's unlicensed packages from dependency review" This reverts commit 30b0877. * build: pin the CSS corpus past js-yaml's vulnerable range Dependency review reads every manifest a pull request changes, so committing the lockfiles is what made it see what the corpus had been installing all along: crass asks for svgo ^0.7.1, whose 0.7.2 pins js-yaml ~3.7.0, and seven advisories stand against it. The advisories name <=3.15.1, so an override to ^3.15.2 clears them while staying on the 3.x line svgo's safeLoad call needs. npm audit over the corpus goes from one high and three moderate to one moderate, and crass still parses and optimizes. * build: order the CSS corpus manifest the way npm writes one * build: bump @minify-html/node to a version whose platform packages exist 0.15.0 declares optional dependencies on @minify-html/node-linux-arm64 and -darwin-arm64, which were never published at that version. npm install skips an optional dependency it cannot resolve, so the lockfile recorded three platform packages where the manifest asks for five — and npm ci reads that as a lockfile out of sync and refuses, which is what reds the invariants job while a plain install passed. 0.18.1 publishes all five. Verified with npm 11, the line CI runs: npm ci now succeeds in all three corpora, and minify-html's minify(Buffer, options) API is unchanged. * build: take the Materialize fixture from the fork that is still maintained materialize-css stopped at 1.0.0 in 2018 and carries four moderate XSS advisories with no version that fixes them, so dependency review refuses the corpus now that the lockfile is committed. The comparison reads its stylesheet and never its JavaScript, which is where the advisories are, but the check cannot see that and nothing will ever clear them upstream. @materializecss/materialize is the maintained continuation, ships the same dist/css/materialize.css, and audits clean. The corpus stays 961 stylesheets, FIXTURE=Materialize still matches one, and npm audit over all three corpora is now empty. * build: clear a corpus cache before reinstalling it npm ci refreshes node_modules and nothing else, so a fixture the caller generated from the packages it installed — the three Tailwind stylesheets compare-css-tools.js builds — outlived a lockfile bump and was skipped as already there, then passed for current under the new stamp. That is the staleness the lockfile exists to remove, left alive for the derived half. Everything under the cache is derived from the manifest, so the reinstall now starts from an empty directory. Found by CodeRabbit on this PR. * build(ci): let the corpus's robust-predicates past the license check With the vulnerabilities cleared, dependency review reaches the licenses and refuses one package: robust-predicates, which d3 pulls into the JavaScript corpus. It declares Unlicense, which allow-licenses already permits; GitHub reports the pair LicenseRef-scancode-public-domain AND Unlicense, and the half scancode names is no SPDX identifier, so the conjunction cannot match whatever the list holds. Both halves are as permissive as a license gets, and the action offers no way to scope itself to a manifest, so the list the repository already keeps for this is where it goes. * build: decide a corpus cache on its manifest as well as its lockfile The stamp read the lockfile alone, so a manifest edited without regenerating it left the cache warm — and the install that would have refused the pair never ran, because skipping it is exactly what the stamp does. npm ci is what catches that mismatch, and the shortcut sat in front of it. CI's cache keys read only the lockfiles for the same reason. Both files now decide, hashed with a delimiter between them, and a test moves the manifest alone and asserts the reinstall. Found by CodeRabbit. * build(ci): say what the corpus cache key is actually keyed on The equivalence job's cache comment still described the old scheme, where the key hashed the scripts holding the package lists. Both keys read each corpus's package.json and package-lock.json now, so the comment named an input that no longer exists. Found by CodeRabbit.
1 parent b7757bf commit 4ac3fe0

18 files changed

Lines changed: 14179 additions & 197 deletions

‎.github/dependabot.yml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,28 @@ updates:
2222
- "major"
2323
exclude-patterns:
2424
- "rimraf"
25+
# The corpora the comparison scripts measure against, one manifest each. They
26+
# reach no webpack build, so a bump is only ever a wider corpus to sweep — and
27+
# a bump that breaks a printer invariant reds its own pull request, where it
28+
# is attributable, rather than somebody else's.
29+
- package-ecosystem: npm
30+
directories:
31+
- "/tooling/comparison/css-tool-comparison"
32+
- "/tooling/comparison/html-tool-comparison"
33+
- "/tooling/comparison/js-tool-comparison"
34+
schedule:
35+
interval: "monthly"
36+
open-pull-requests-limit: 3
37+
labels:
38+
- dependencies
39+
- "area: tooling"
40+
# `increase` rather than the root's `widen`: these are pinned exactly, so a
41+
# bump replaces the version rather than loosening the range it sits in.
42+
versioning-strategy: increase
43+
groups:
44+
comparison-corpus:
45+
patterns:
46+
- "*"
2547
- package-ecosystem: "github-actions"
2648
directory: "/"
2749
schedule:

‎.github/workflows/dependency-review.yml‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,10 @@ jobs:
2121
- name: Dependency Review
2222
uses: actions/dependency-review-action@a1d282b36b6f3519aa1f3fc636f609c47dddb294 # v5.0.0
2323
with:
24+
# robust-predicates is the comparison corpus's, reached through d3,
25+
# and declares Unlicense — which this list allows. GitHub reports it
26+
# as `LicenseRef-scancode-public-domain AND Unlicense`, and the half
27+
# scancode names is no SPDX identifier, so the pair can never match.
2428
allow-dependencies-licenses: |
2529
pkg:npm/@cspell/dict-django,
2630
pkg:npm/@cspell/dict-en-common-misspellings,
@@ -35,7 +39,8 @@ jobs:
3539
pkg:npm/ansis,
3640
pkg:npm/strtok3,
3741
pkg:npm/@sinclair/typebox,
38-
pkg:npm/fs-monkey
42+
pkg:npm/fs-monkey,
43+
pkg:npm/robust-predicates
3944
allow-licenses: |
4045
0BSD,
4146
AFL-1.1,

‎.github/workflows/test.yml‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -128,7 +128,7 @@ jobs:
128128
path: |
129129
node_modules/.cache/css-tool-comparison
130130
node_modules/.cache/html-tool-comparison
131-
key: comparison-corpus-${{ runner.os }}-${{ hashFiles('tooling/compare-css-tools.js', 'tooling/compare-html-tools.js') }}
131+
key: comparison-corpus-${{ runner.os }}-${{ hashFiles('tooling/comparison/css-tool-comparison/package.json', 'tooling/comparison/css-tool-comparison/package-lock.json', 'tooling/comparison/html-tool-comparison/package.json', 'tooling/comparison/html-tool-comparison/package-lock.json') }}
132132

133133
# Fatal here, unlike the equivalence job: that one reads whichever
134134
# fixtures are there, and this one is the check that the whole corpus
@@ -399,16 +399,16 @@ jobs:
399399

400400
- run: yarn link webpack --frozen-lockfile
401401

402-
# Keyed on the scripts that declare the package lists; a cache that
403-
# predates a fixture being added is rebuilt by the install below, which
404-
# reinstalls whenever the recorded list differs from the wanted one.
402+
# Keyed on each corpus's manifest and lockfile; a cache predating an edit
403+
# to either is rebuilt by the install below, which reinstalls whenever the
404+
# recorded pair differs from the wanted one.
405405
- name: Cache the comparison corpus
406406
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
407407
with:
408408
path: |
409409
node_modules/.cache/css-tool-comparison
410410
node_modules/.cache/html-tool-comparison
411-
key: comparison-corpus-${{ runner.os }}-${{ hashFiles('tooling/compare-css-tools.js', 'tooling/compare-html-tools.js') }}
411+
key: comparison-corpus-${{ runner.os }}-${{ hashFiles('tooling/comparison/css-tool-comparison/package.json', 'tooling/comparison/css-tool-comparison/package-lock.json', 'tooling/comparison/html-tool-comparison/package.json', 'tooling/comparison/html-tool-comparison/package-lock.json') }}
412412

413413
# The third corpus: the framework stylesheets and documents the tool
414414
# comparisons install. Not fatal — the suite reads whichever fixtures are

‎.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,10 @@
2121
.eslintcache
2222
.cspellcache
2323
package-lock.json
24+
# Except the comparison corpora: each one's lockfile is the record of what
25+
# the benchmarks and the invariant sweeps actually ran against.
26+
!/tooling/comparison/*/package-lock.json
27+
/tooling/comparison/*/node_modules
2428
/events.json
2529
/dist
2630

‎.prettierignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,6 @@
11
package.json
2+
# npm writes these; reformatting one only makes the next `npm ci` rewrite it.
3+
package-lock.json
24

35
# Ignore some test files
46
test/**/*.*

‎AGENTS.md‎

Lines changed: 1 addition & 1 deletion
Large diffs are not rendered by default.

‎cspell.json‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -248,6 +248,7 @@
248248
"lowlink",
249249
"Lucida",
250250
"mangleable",
251+
"materializecss",
251252
"MATHML",
252253
"maxlength",
253254
"MCEP",
@@ -422,6 +423,7 @@
422423
"scalex",
423424
"scaley",
424425
"scalez",
426+
"scancode",
425427
"sccs",
426428
"scrollamount",
427429
"scrolldelay",
@@ -598,6 +600,7 @@
598600
"**/webpack.lock.data/**",
599601
"package.json",
600602
"yarn.lock",
603+
"**/package-lock.json",
601604
"tooling/deep-webpack-imports.json",
602605
"types.d.ts",
603606
"**/**/*.snap",

‎test/unitCases/CompareToolsHarness.unittest.js‎

Lines changed: 123 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
"use strict";
22

3+
const { createHash } = require("crypto");
34
const fs = require("fs");
45
const os = require("os");
56
const path = require("path");
@@ -326,7 +327,10 @@ describe("compare-tools-harness", () => {
326327
const NAMES = [
327328
"compare-tools-harness-unittest-warm",
328329
"compare-tools-harness-unittest-cold",
329-
"compare-tools-harness-unittest-failed"
330+
"compare-tools-harness-unittest-failed",
331+
"compare-tools-harness-unittest-moved",
332+
"compare-tools-harness-unittest-derived",
333+
"compare-tools-harness-unittest-manifest"
330334
];
331335

332336
/**
@@ -344,14 +348,39 @@ describe("compare-tools-harness", () => {
344348
*/
345349
const npmRan = (name) => fs.existsSync(path.join(cacheFor(name), RAN));
346350

351+
/**
352+
* @param {string} name which corpus
353+
* @returns {string} where its committed manifest lives
354+
*/
355+
const corpusFor = (name) =>
356+
path.resolve(__dirname, "../..", "tooling/comparison", name);
357+
358+
/**
359+
* Write a corpus of the shape `installPackages` reads: the manifest and the
360+
* lockfile, whose contents together decide whether an install is owed.
361+
* @param {string} name which corpus
362+
* @param {string} lock what its lockfile says
363+
* @param {string=} manifest what its package.json says
364+
* @returns {string} the hash a matching stamp would carry
365+
*/
366+
const writeCorpus = (name, lock, manifest) => {
367+
const written = manifest || JSON.stringify({ name, private: true });
368+
fs.mkdirSync(corpusFor(name), { recursive: true });
369+
fs.writeFileSync(path.join(corpusFor(name), "package.json"), written);
370+
fs.writeFileSync(path.join(corpusFor(name), "package-lock.json"), lock);
371+
const identity = createHash("sha256");
372+
for (const part of [written, lock]) identity.update(part).update("\0");
373+
return identity.digest("hex");
374+
};
375+
347376
/**
348377
* @param {string} name which cache
349-
* @returns {EXPECTED_ANY} the manifest it holds
378+
* @returns {string | undefined} the lockfile hash it was installed from
350379
*/
351-
const manifestOf = (name) =>
352-
JSON.parse(
353-
fs.readFileSync(path.join(cacheFor(name), "package.json"), "utf8")
354-
);
380+
const stampOf = (name) => {
381+
const stamp = path.join(cacheFor(name), ".corpus-lock");
382+
return fs.existsSync(stamp) ? fs.readFileSync(stamp, "utf8") : undefined;
383+
};
355384

356385
/**
357386
* Run with an `npm` of our own first on `PATH`, so nothing is fetched.
@@ -389,6 +418,7 @@ describe("compare-tools-harness", () => {
389418
const clearCaches = () => {
390419
for (const name of NAMES) {
391420
fs.rmSync(cacheFor(name), { recursive: true, force: true });
421+
fs.rmSync(corpusFor(name), { recursive: true, force: true });
392422
}
393423
};
394424

@@ -398,49 +428,105 @@ describe("compare-tools-harness", () => {
398428

399429
afterAll(clearCaches);
400430

401-
// Nothing is installed here: the manifest already lists what was asked
402-
// for, which is the branch that keeps a re-run from reaching npm.
403-
posixOnly(
404-
"reuses a cache whose manifest lists the same packages",
405-
async () => {
406-
const [name] = NAMES;
407-
fs.mkdirSync(path.join(cacheFor(name), "node_modules"), {
408-
recursive: true
409-
});
410-
fs.writeFileSync(
411-
path.join(cacheFor(name), "package.json"),
412-
JSON.stringify({
413-
name,
414-
comparisonPackages: ["left@1", "right@2"]
415-
})
416-
);
417-
// The stand-in fails, so taking the install path would fail the case
418-
// rather than reach the real npm and the network behind it.
419-
await withStandInNpm(1, () =>
420-
expect(installPackages(name, ["left@1", "right@2"])).resolves.toBe(
421-
cacheFor(name)
422-
)
423-
);
424-
expect(npmRan(name)).toBe(false);
425-
}
426-
);
431+
// Nothing is installed here: the cache was installed from this very
432+
// lockfile, which is the branch a restored CI cache takes.
433+
posixOnly("reuses a cache whose stamp matches the lockfile", async () => {
434+
const [name] = NAMES;
435+
const hash = writeCorpus(name, '{"lockfileVersion":3,"warm":true}');
436+
fs.mkdirSync(path.join(cacheFor(name), "node_modules"), {
437+
recursive: true
438+
});
439+
fs.writeFileSync(path.join(cacheFor(name), ".corpus-lock"), hash);
440+
// The stand-in fails, so taking the install path would fail the case
441+
// rather than reach the real npm and the network behind it.
442+
await withStandInNpm(1, () =>
443+
expect(installPackages(name)).resolves.toBe(cacheFor(name))
444+
);
445+
expect(npmRan(name)).toBe(false);
446+
});
427447

428448
posixOnly("installs into a cache that has none of it yet", async () => {
429449
const name = NAMES[1];
430-
await withStandInNpm(0, () => installPackages(name, ["left@1"]));
450+
const hash = writeCorpus(name, '{"lockfileVersion":3,"cold":true}');
451+
await withStandInNpm(0, () => installPackages(name));
431452
expect(npmRan(name)).toBe(true);
432-
expect(manifestOf(name).comparisonPackages).toEqual(["left@1"]);
453+
expect(stampOf(name)).toBe(hash);
454+
// Both are what `npm ci` reads, so the install ran against the corpus
455+
// as committed rather than against whatever the cache last held.
456+
expect(
457+
fs.existsSync(path.join(cacheFor(name), "package-lock.json"))
458+
).toBe(true);
433459
});
434460

435-
// The list is what a later run compares against, so recording it before
461+
// The stamp is what a later run compares against, so writing it before
436462
// the install succeeded would let a broken cache pass for a warm one.
437463
posixOnly("records nothing when the install fails", async () => {
438464
const name = NAMES[2];
465+
writeCorpus(name, '{"lockfileVersion":3,"failed":true}');
439466
await expect(
440-
withStandInNpm(1, () => installPackages(name, ["right@2"]))
467+
withStandInNpm(1, () => installPackages(name))
441468
).rejects.toThrow("exited with 1");
442469
expect(npmRan(name)).toBe(true);
443-
expect(manifestOf(name).comparisonPackages).toBeUndefined();
470+
expect(stampOf(name)).toBeUndefined();
471+
});
472+
473+
// The manifest decides as much as the lockfile does: one edited without
474+
// regenerating the other is what `npm ci` refuses, and a warm cache that
475+
// read the lockfile alone would skip the install that would refuse it.
476+
posixOnly("reinstalls when only the manifest moved", async () => {
477+
const name = NAMES[5];
478+
const lock = '{"lockfileVersion":3,"pinned":"1.0.0"}';
479+
writeCorpus(name, lock, JSON.stringify({ name, asked: "1.0.0" }));
480+
await withStandInNpm(0, () => installPackages(name));
481+
const before = stampOf(name);
482+
fs.mkdirSync(path.join(cacheFor(name), "node_modules"), {
483+
recursive: true
484+
});
485+
// The lockfile is byte-identical; only what the manifest asks for moved.
486+
const after = writeCorpus(
487+
name,
488+
lock,
489+
JSON.stringify({ name, asked: "1.0.1" })
490+
);
491+
expect(after).not.toBe(before);
492+
fs.rmSync(path.join(cacheFor(name), RAN), { force: true });
493+
await withStandInNpm(0, () => installPackages(name));
494+
expect(npmRan(name)).toBe(true);
495+
expect(stampOf(name)).toBe(after);
496+
});
497+
498+
// A fixture the caller generated with the packages it installed is as
499+
// stale as they are, and nothing else would clear it: `npm ci` rewrites
500+
// `node_modules` alone, and the builder skips an output already there.
501+
posixOnly("clears what the last corpus derived", async () => {
502+
const name = NAMES[4];
503+
writeCorpus(name, '{"lockfileVersion":3,"pinned":"1.0.0"}');
504+
fs.mkdirSync(cacheFor(name), { recursive: true });
505+
const derived = path.join(cacheFor(name), "tailwind-app.css");
506+
fs.writeFileSync(derived, ".from-the-old-tailwind{}");
507+
writeCorpus(name, '{"lockfileVersion":3,"pinned":"1.0.1"}');
508+
await withStandInNpm(0, () => installPackages(name));
509+
expect(fs.existsSync(derived)).toBe(false);
510+
});
511+
512+
// The case the lockfile exists for: Dependabot moves a pin, and a cache
513+
// installed from the version before it is stale rather than warm.
514+
posixOnly("reinstalls when the lockfile has moved", async () => {
515+
const name = NAMES[3];
516+
writeCorpus(name, '{"lockfileVersion":3,"pinned":"1.0.0"}');
517+
await withStandInNpm(0, () => installPackages(name));
518+
const before = stampOf(name);
519+
// The stand-in installs nothing, so the tree it would have written is
520+
// what makes this cache warm — without it the stamp is never consulted.
521+
fs.mkdirSync(path.join(cacheFor(name), "node_modules"), {
522+
recursive: true
523+
});
524+
const after = writeCorpus(name, '{"lockfileVersion":3,"pinned":"1.0.1"}');
525+
expect(after).not.toBe(before);
526+
fs.rmSync(path.join(cacheFor(name), RAN), { force: true });
527+
await withStandInNpm(0, () => installPackages(name));
528+
expect(npmRan(name)).toBe(true);
529+
expect(stampOf(name)).toBe(after);
444530
});
445531
});
446532

0 commit comments

Comments
 (0)