Skip to content

Commit db556f3

Browse files
test: run the printer equivalence suite against Gecko in CI (#22155)
* test(equivalence): keep a condition apart on the features nothing samples Follow-up to #22151. `unsampledBy` treated every feature whose name ends in a viewport dimension as one the viewport bits already tell apart, so `device-width`, `device-height` and `device-aspect-ratio` — the screen's, which `setViewport` does not move — were dropped from the text and from the bits both, and two conditions naming different ones read as one. The media type is now read outside the parentheses as well, so a feature's own words cannot be taken for a type. A filed color rewrite is held exactly, the way every other filed set in the file is: a `FILED_COLOR_REWRITES` entry the engine stopped moving now fails rather than sitting there. It is counted before the within-a-byte tolerance so the tolerance cannot hide it, and the enumerated-fold comparison sorts both sides. Two JSDoc blocks had come apart from what they document: `installHelpers` had gained a second block instead of the `@param`, and `roundedOnly`'s had been left above `withinAByte`. `launchBrowser` and `install-firefox` read `PUPPETEER_CACHE_DIR` and build the path with `path.join`, so they agree on where the browser went; a caller's own `executablePath` is no longer overwritten, a platform puppeteer cannot detect reports no browser rather than throwing, and a failed install prints its error and exits non-zero. `TESTING_DOCS.md` still pointed at the deleted `launchChrome.js`. * ci: run the printer equivalence suite against Gecko as well `syntax-equivalence-firefox` runs the same corpus and the same assertions under `EQUIVALENCE_BROWSER=firefox`, beside the Chrome job rather than replacing it — a rewrite only one engine reads differently is invisible to a suite that asks one, and five of the defects filed today are ones only Gecko sees. It is a job of its own so the existing check keeps its name, installs the browser with `yarn setup:firefox`, and uploads no coverage: the Chrome job already reports this suite's lines under the `syntax-equivalence` flag. `unsampledBy` counted a feature once per occurrence, so `(monochrome) and (monochrome)` and the `(monochrome)` the printer folds it to read as two conditions where the viewport bits agreed. `and`, `or` and `,` are idempotent, so the parts are a set — its own documentation already said "named once". * ci: name the engine on both equivalence jobs The Chrome run was `syntax-equivalence` with nothing saying which browser it asked, while the engine added beside it said `-firefox` — so the pair read as a suite and an afterthought rather than as one suite over two engines. They are one matrix now, `syntax-equivalence (chrome)` and `syntax-equivalence (firefox)`, the shape `runtimes (bun)` and `runtimes (deno)` already use here. That also removes the copy of the job the second engine had brought with it: the corpus checkout, the three caches and both comparison installs were duplicated. Firefox is installed on its own row, Chrome reports the coverage (both cover the same lines, so a second uploader only leaves the report short), and `fail-fast: false` keeps one engine's failure from cancelling the other. * ci: name the equivalence rows for the engine alone A matrix row carries the script and the coverage flag beside the browser, and the generated name prints every key — so the checks read `syntax-equivalence (chrome, cover:syntax-equivalence, true)`. An explicit `name` keeps the engine and drops the rest. * test: ask WebKit the same questions as Blink and Gecko WebKit is the third engine and the one Safari ships, and puppeteer reaches Chrome and Firefox only — so `EQUIVALENCE_BROWSER=webkit` drives it through playwright, under a shim in `launchBrowser.js` presenting the nine calls the suites make. Playwright takes one argument where puppeteer takes several, and offers no `color-gamut`, so it declines media emulation outright and a condition carries what no viewport varies as text, the way Gecko already does. `forEngine` no longer assumes two engines: a reason opening `<engine> only:` is kept where it names the engine being run and dropped everywhere else, so a third filed set needs nothing further. The row is `continue-on-error` until its filed set is written from what it actually sees. Both other engines opened with hundreds of differences that were nearly all the harness rather than the printer, and the same is expected here; filing them before the first run would be inventing them. * test: file what WebKit sees and hold it like the other two engines The first WebKit run reported 24 differences over 5803 checks, and they are five causes. Each is filed with what was measured, so the row blocks like Chrome's and Firefox's rather than reporting past a CI flag. Three are the harness meeting a third engine. WebKit reads a different `document.compatMode` for the two forms of eleven files and the reverse for two, which a direction that flips per file says is how the document is built rather than what the printer wrote. It lays out in sixty-fourths of a pixel, so `112px` reads back as `111.984375px` in five framework stylesheets. And it serializes a computed number to six significant digits where Blink keeps more — which is the printer's own cap, so the whole-element comparison now reads both sides at that precision, Chromium excepted since the fixtures were measured against it. Two are the engine: no `fetchpriority` reflection, so `high` and `low` both read `auto`; no CSS Painting API, so a `paint()` image is dropped; and `shape-image-threshold` read as the number its percentage means. `attribute-tables` and `minimize-values` are filed by two engines for different causes now, so a reason may name more than one.
1 parent d174cdf commit db556f3

10 files changed

Lines changed: 381 additions & 88 deletions

File tree

‎.github/workflows/test.yml‎

Lines changed: 41 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -303,10 +303,30 @@ jobs:
303303
env:
304304
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
305305

306+
# One job per engine, the way `runtimes` is one per runtime: a rewrite only
307+
# one engine reads differently is invisible to a suite that asks one, and
308+
# several of the defects filed today are ones only Gecko sees.
306309
syntax-equivalence:
310+
# Named for the engine alone: a matrix row carries the script and the
311+
# coverage flag too, and the generated name would otherwise print all three.
312+
name: syntax-equivalence (${{ matrix.browser }})
307313
needs: basic
314+
strategy:
315+
fail-fast: false
316+
matrix:
317+
include:
318+
# Chrome's run is the one that reports coverage. Both cover the same
319+
# lines, so a second uploader would only leave every pull request a
320+
# report short until the other engine lands.
321+
- browser: chrome
322+
script: cover:syntax-equivalence
323+
coverage: true
324+
- browser: firefox
325+
script: test:syntax-equivalence
326+
- browser: webkit
327+
script: test:syntax-equivalence
308328
runs-on: ubuntu-latest
309-
# Bounded: a hung Chromium would otherwise hold a runner for the 6h default.
329+
# Bounded: a hung browser would otherwise hold a runner for the 6h default.
310330
timeout-minutes: 60
311331
steps:
312332
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
@@ -356,18 +376,34 @@ jobs:
356376
continue-on-error: true
357377
run: yarn benchmark:html-tools:setup
358378

379+
# Chrome comes with the runner. Firefox is fetched, and deliberately not
380+
# cached: the build id resolves to whatever `stable` is today, so a cache
381+
# key that never moves would pin the engine this suite exists to track.
382+
- name: Install Firefox
383+
if: matrix.browser == 'firefox'
384+
run: yarn setup:firefox
385+
386+
# `--with-deps` because a runner carries none of WebKit's shared
387+
# libraries, where it already carries Chrome's.
388+
- name: Install WebKit
389+
if: matrix.browser == 'webkit'
390+
run: node node_modules/playwright-core/cli.js install --with-deps webkit
391+
359392
- name: Cache jest result
360393
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
361394
with:
362395
path: .jest-cache
363-
key: jest-syntax-equivalence-${{ runner.os }}-${{ hashFiles('**/yarn.lock', '**/jest.config.js') }}-${{ github.sha }}
396+
key: jest-syntax-equivalence-${{ matrix.browser }}-${{ runner.os }}-${{ hashFiles('**/yarn.lock', '**/jest.config.js') }}-${{ github.sha }}
364397
restore-keys: |
365-
jest-syntax-equivalence-${{ runner.os }}-${{ hashFiles('**/yarn.lock', '**/jest.config.js') }}-
366-
jest-syntax-equivalence-${{ runner.os }}-
398+
jest-syntax-equivalence-${{ matrix.browser }}-${{ runner.os }}-${{ hashFiles('**/yarn.lock', '**/jest.config.js') }}-
399+
jest-syntax-equivalence-${{ matrix.browser }}-${{ runner.os }}-
367400
368-
- run: yarn cover:syntax-equivalence --ci --cacheDirectory .jest-cache
401+
- run: yarn ${{ matrix.script }} --ci --cacheDirectory .jest-cache
402+
env:
403+
EQUIVALENCE_BROWSER: ${{ matrix.browser }}
369404

370405
- name: Codecov
406+
if: matrix.coverage
371407
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
372408
with:
373409
flags: syntax-equivalence

‎AGENTS.md‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ All commands are defined in `package.json` `scripts`.
3535
| `yarn test:unit` | Run all `*.unittest.js`. |
3636
| `yarn test:integration` | Run the integration suites (`basictest`/`longtest`/`test`). |
3737
| `yarn test:test262` / `yarn test:html5lib` / `yarn test:css-parsing` | Spec-conformance suites. |
38-
| `yarn test:syntax-equivalence` | Holds the HTML/CSS printers to what Chrome makes of their output, over `configCases` and `test/wpt`. |
38+
| `yarn test:syntax-equivalence` | Holds the HTML/CSS printers to a real browser's reading of their output, over `configCases` and `test/wpt`. |
3939
| `yarn test:base -u` | Update snapshots (eyeball the diff first). |
4040
| `yarn test:size` | Size of the generated code over all `configCases/` (per asset, plus runtime module counts per runtime). |
4141
| `yarn cover:unit` | Unit-test coverage. |
@@ -109,12 +109,12 @@ The directory listings below are the canonical map of the repository. **Whenever
109109

110110
**Git submodules** — the spec-conformance corpora are submodules, checked out on demand: `yarn setup` does not fetch them, and each CI job fetches only the submodules it needs, one commit deep.
111111

112-
| Path | Upstream | Fetched by |
113-
| ------------------------ | ------------------------------------------------------------------------------------- | -------------------------------- |
114-
| `test/test262-cases` | [tc39/test262](https://github.com/tc39/test262) | `test262`, `test262-parser` |
115-
| `test/html5lib-tests` | [html5lib/html5lib-tests](https://github.com/html5lib/html5lib-tests) | `html5lib` |
116-
| `test/wpt` | [web-platform-tests/wpt](https://github.com/web-platform-tests/wpt) | `html5lib`, `syntax-equivalence` |
117-
| `test/css-parsing-tests` | [CourtBouillon/css-parsing-tests](https://github.com/CourtBouillon/css-parsing-tests) | `css-parsing` |
112+
| Path | Upstream | Fetched by |
113+
| ------------------------ | ------------------------------------------------------------------------------------- | ------------------------------------------------ |
114+
| `test/test262-cases` | [tc39/test262](https://github.com/tc39/test262) | `test262`, `test262-parser` |
115+
| `test/html5lib-tests` | [html5lib/html5lib-tests](https://github.com/html5lib/html5lib-tests) | `html5lib` |
116+
| `test/wpt` | [web-platform-tests/wpt](https://github.com/web-platform-tests/wpt) | `html5lib`, `syntax-equivalence` (every browser) |
117+
| `test/css-parsing-tests` | [CourtBouillon/css-parsing-tests](https://github.com/CourtBouillon/css-parsing-tests) | `css-parsing` |
118118

119119
```sh
120120
git submodule update --init --recursive --depth 1 # check out the commits the repo pins

‎TESTING_DOCS.md‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -155,12 +155,25 @@ During the test run, webpack compiles this project and compares the result with
155155
### Suites that drive a real browser
156156

157157
`unitCases/ProfilingPlugin.unittest.js`, `specCases/syntaxEquivalence.spectest.js` and
158-
`WebpackDevServer.longtest.js` launch Chrome through `test/helpers/launchChrome.js`.
158+
`WebpackDevServer.longtest.js` launch a browser through `test/helpers/launchBrowser.js`.
159159
**A browser that will not launch fails the suite — it is never skipped**, so no
160160
environment can report these checks as passing without having run them. The
161161
helper uses the installed Chrome channel; set `PUPPETEER_EXECUTABLE_PATH` to
162162
point at another binary.
163163

164+
`syntaxEquivalence.spectest.js` is the one that reads more than one engine:
165+
`EQUIVALENCE_BROWSER` takes `chrome`, `firefox` or `webkit`, and CI runs all
166+
three as one matrix. Chrome and Firefox go through puppeteer; WebKit goes
167+
through playwright, which is what reaches it, under a shim in
168+
`test/helpers/launchBrowser.js` that answers to the puppeteer names the suites
169+
call. Fetch the browser first — `yarn setup:firefox`, `yarn setup:webkit` —
170+
which installs into puppeteer's own cache (`PUPPETEER_CACHE_DIR`, defaulting to
171+
`~/.cache/puppeteer`) or playwright's; `FIREFOX_EXECUTABLE_PATH` and
172+
`WEBKIT_EXECUTABLE_PATH` point at one already on the machine. Only Chromium
173+
answers the media-emulation calls, so elsewhere a condition carries what no
174+
viewport varies as text. A defect only one engine has is filed against it with a
175+
reason opening `<engine> only:` rather than tolerated in all three.
176+
164177
They are excluded from `test:bun` / `test:deno` (see the `--testPathIgnorePatterns`
165178
in those scripts): under Jest on Bun, loading the ESM-only `puppeteer-core` fails
166179
outright with "Provided module is not an instance of Module", and Jest's

‎cspell.json‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@
2121
"astring",
2222
"asyncloader",
2323
"atlaskit",
24+
"autocapitalize",
2425
"autocrlf",
2526
"autospace",
2627
"avivkeller",
@@ -127,6 +128,7 @@
127128
"embedurl",
128129
"Emscripten",
129130
"endregion",
131+
"enterkeyhint",
130132
"entrypoint",
131133
"entrypoints",
132134
"Eoksni",
@@ -196,6 +198,7 @@
196198
"importmap",
197199
"informations",
198200
"inlines",
201+
"inputmode",
199202
"installurl",
200203
"instanceof",
201204
"inversed",
@@ -284,6 +287,7 @@
284287
"mycompany",
285288
"mylibrary",
286289
"mynamespace",
290+
"mypaint",
287291
"Natsu",
288292
"navigations",
289293
"nesw",

‎package.json‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,7 @@
5959
"benchmark:js-tools": "node tooling/compare-js-tools.js",
6060
"measure:color-agreement": "node tooling/measure-color-agreement.js",
6161
"setup:firefox": "node tooling/install-firefox.js",
62+
"setup:webkit": "node node_modules/playwright-core/cli.js install webkit",
6263
"pretest": "yarn lint",
6364
"test": "yarn test:base",
6465
"test:base": "node --expose-gc --max-old-space-size=4096 --experimental-vm-modules --trace-deprecation node_modules/jest-cli/bin/jest --workerIdleMemoryLimit=512MB --logHeapUsage",
@@ -133,11 +134,11 @@
133134
"@changesets/get-github-info": "^1.0.1",
134135
"@codspeed/core": "^5.5.0",
135136
"@mdn/browser-compat-data": "^8.0.9",
137+
"@puppeteer/browsers": "^3.2.2",
136138
"@types/convert-source-map": "^2.0.3",
137139
"@types/graceful-fs": "^4.1.9",
138140
"@types/jest": "^30.0.0",
139141
"@types/mime-db": "^1.43.6",
140-
"@puppeteer/browsers": "^3.2.2",
141142
"@types/node": "^26.2.0",
142143
"@types/xxhashjs": "^0.2.4",
143144
"@webdiscus/pug-loader": "^2.11.1",
@@ -196,6 +197,7 @@
196197
"open-cli": "^9.0.0",
197198
"oxc-parser": "^0.149.0",
198199
"pkg-pr-new": "^0.0.88",
200+
"playwright-core": "^1.63.0",
199201
"prettier": "^3.8.2",
200202
"prettier-2": "npm:prettier@^2",
201203
"pretty-format": "^30.0.5",

‎test/helpers/launchBrowser.js‎

Lines changed: 104 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,13 @@
11
"use strict";
22

33
const os = require("os");
4+
const path = require("path");
45

5-
const CACHE_DIR = `${os.homedir()}/.cache/puppeteer`;
6+
// The same cache `yarn setup:firefox` installs into, and the one puppeteer's
7+
// own tooling reads, so the two agree wherever it has been pointed elsewhere.
8+
const CACHE_DIR =
9+
process.env.PUPPETEER_CACHE_DIR ||
10+
path.join(os.homedir(), ".cache", "puppeteer");
611

712
/**
813
* Where an already-downloaded Firefox sits, or `undefined` when none is. The
@@ -16,6 +21,7 @@ const installedFirefox = async () => {
1621
await import("@puppeteer/browsers");
1722

1823
const platform = detectBrowserPlatform();
24+
if (platform === undefined) return undefined;
1925
for (const one of await getInstalledBrowsers({ cacheDir: CACHE_DIR })) {
2026
if (one.browser === Browser.FIREFOX && one.platform === platform) {
2127
return one.executablePath;
@@ -24,17 +30,99 @@ const installedFirefox = async () => {
2430
return undefined;
2531
};
2632

33+
/**
34+
* One puppeteer `evaluate` over a playwright page. Playwright takes a single
35+
* argument where puppeteer takes several, so a call with more is rewritten to
36+
* spread one array.
37+
* @param {import("playwright-core").Page} page the playwright page
38+
* @param {EXPECTED_FUNCTION} fn what to run in it
39+
* @param {unknown[]} args its arguments
40+
* @returns {Promise<EXPECTED_ANY>} what it returned
41+
*/
42+
const evaluateOn = (page, fn, args) => {
43+
if (args.length <= 1) {
44+
return page.evaluate(
45+
/** @type {(one: unknown) => EXPECTED_ANY} */ (fn),
46+
args[0]
47+
);
48+
}
49+
// eslint-disable-next-line no-new-func
50+
const spread = new Function(
51+
"passed",
52+
`return (${fn.toString()}).apply(null, passed);`
53+
);
54+
return page.evaluate(
55+
/** @type {(passed: unknown[]) => EXPECTED_ANY} */ (spread),
56+
args
57+
);
58+
};
59+
60+
/**
61+
* A playwright page under the puppeteer names the browser suites call it by,
62+
* so one suite drives either driver.
63+
* @param {import("playwright-core").Page} page the playwright page
64+
* @returns {EXPECTED_ANY} it, answering to puppeteer's names
65+
*/
66+
const asPuppeteerPage = (page) => ({
67+
close: () => page.close(),
68+
setContent: (/** @type {string} */ html) => page.setContent(html),
69+
goto: (/** @type {string} */ url) => page.goto(url),
70+
evaluate: (
71+
/** @type {EXPECTED_FUNCTION} */ fn,
72+
/** @type {unknown[]} */ ...args
73+
) => evaluateOn(page, fn, args),
74+
setViewport: (/** @type {{ width: number, height: number }} */ viewport) =>
75+
page.setViewportSize({ width: viewport.width, height: viewport.height }),
76+
// WHY: playwright emulates the color scheme and reduced motion but not
77+
// `color-gamut`, and a feature the page cannot be put into answers alike for
78+
// both of its values — which reads as one condition where there are two. So
79+
// none is offered, `emulatesMedia` reads the throw, and the suite carries
80+
// what no viewport varies as text, the way it already does for Gecko.
81+
emulateMediaType: () => {
82+
throw new Error("playwright drives the media through no such call");
83+
},
84+
emulateMediaFeatures: () => {
85+
throw new Error("playwright drives the media through no such call");
86+
}
87+
});
88+
89+
/**
90+
* The WebKit the suites compare against, driven by playwright since puppeteer
91+
* reaches Chrome and Firefox only. A browser that is missing throws, the way a
92+
* missing Firefox does.
93+
* @param {{ timeout?: number, executablePath?: string }} options the launch options
94+
* @returns {Promise<EXPECTED_ANY>} the running browser, under puppeteer's names
95+
*/
96+
const launchWebkit = async (options) => {
97+
const { webkit } = await import("playwright-core");
98+
const browser = await webkit.launch({
99+
timeout: options.timeout,
100+
...(options.executablePath || process.env.WEBKIT_EXECUTABLE_PATH
101+
? {
102+
executablePath:
103+
options.executablePath || process.env.WEBKIT_EXECUTABLE_PATH
104+
}
105+
: {})
106+
});
107+
return {
108+
close: () => browser.close(),
109+
newPage: async () => asPuppeteerPage(await browser.newPage())
110+
};
111+
};
112+
27113
/**
28114
* A real browser for the suites that compare against one. A browser that is
29115
* missing or will not launch throws rather than skipping, so no environment can
30116
* report these checks as passing without having run them.
31-
* @param {(import("puppeteer-core").LaunchOptions & { browser?: "chrome" | "firefox" })=} options extra launch options; `browser` picks the engine
117+
* @param {(Omit<import("puppeteer-core").LaunchOptions, "browser"> & { browser?: "chrome" | "firefox" | "webkit" })=} options extra launch options; `browser` picks the engine
32118
* @returns {Promise<import("puppeteer-core").Browser>} the running browser
33119
*/
34120
module.exports = async (options) => {
121+
const { browser: engine = "chrome", ...rest } = options || {};
122+
// WebKit is playwright's, and nothing below reaches it.
123+
if (engine === "webkit") return launchWebkit({ timeout: 100000, ...rest });
35124
// require() of puppeteer-core throws under Jest since it is ESM-only (v25+).
36125
const puppeteer = (await import("puppeteer-core")).default;
37-
const engine = (options && options.browser) || "chrome";
38126
const firefox = engine === "firefox";
39127
/** @type {import("puppeteer-core").LaunchOptions} */
40128
const launchOptions = {
@@ -45,23 +133,29 @@ module.exports = async (options) => {
45133
// Chrome's sandbox and GPU flags mean nothing to Firefox, which reads its
46134
// own settings out of `extraPrefsFirefox` instead.
47135
...(firefox
48-
? {}
136+
? { browser: "firefox" }
49137
: { args: ["--no-sandbox", "--disable-gpu", "--disable-dev-shm-usage"] }),
50-
...options
138+
...rest
51139
};
52140
if (firefox) {
141+
// A caller that named one wins: the environment and the cache are what to
142+
// fall back to, not what to override it with.
53143
const executablePath =
54-
process.env.FIREFOX_EXECUTABLE_PATH || (await installedFirefox());
144+
launchOptions.executablePath ||
145+
process.env.FIREFOX_EXECUTABLE_PATH ||
146+
(await installedFirefox());
55147
if (executablePath === undefined) {
56148
throw new Error(
57149
"no Firefox to launch: run `yarn setup:firefox`, or point FIREFOX_EXECUTABLE_PATH at one"
58150
);
59151
}
60152
launchOptions.executablePath = executablePath;
61-
} else if (process.env.PUPPETEER_EXECUTABLE_PATH) {
62-
launchOptions.executablePath = process.env.PUPPETEER_EXECUTABLE_PATH;
63-
} else {
64-
launchOptions.channel = "chrome";
153+
} else if (!launchOptions.executablePath) {
154+
if (process.env.PUPPETEER_EXECUTABLE_PATH) {
155+
launchOptions.executablePath = process.env.PUPPETEER_EXECUTABLE_PATH;
156+
} else {
157+
launchOptions.channel = "chrome";
158+
}
65159
}
66160
return puppeteer.launch(launchOptions);
67161
};

0 commit comments

Comments
 (0)