Skip to content

Commit 0c02242

Browse files
test: split the syntax unit tests into tokenizer, parser and printer (#22825)
* test: split the syntax unit tests into tokenizer, parser and printer Each language's tests now follow its modules: a tokenizer, a parser and a printer, whose file holds both print modes and the minifier that drives the printer in its minify mode. - css/, html/: syntax splits into tokenizer, parser and printer; the SourceProcessor tests go to the parser when they walk the tree and to the printer when they rewrite what it writes - javascript/: the tokenizer blocks leave parser for tokenizer, minify (jsMinify) joins printer, and webpack-parser becomes module-parser, after `module.parser.javascript`, the layer it tests - xml/, json/: minify becomes printer Every test keeps its name and every snapshot its key; each new file requires only what its tests use. Claude-Session: https://claude.ai/code/session_017TkBWTk3wcyUcJYDp5ZhaP * test: keep JavascriptParser's tests with webpack's classes javascript/ holds the syntax: tokenizer, parser, scope and printer. JavascriptParser is webpack's layer over it, the one plugins tap, so its tests go back to the top level next to NormalModule's and the others. The layout docs give tooling/ its own sentence. Claude-Session: https://claude.ai/code/session_017TkBWTk3wcyUcJYDp5ZhaP * test: cover the CSS tokenizer from its own tests The tokenizer suite now reaches every function of the tokenizer and all of its statements and branches but five that no input can reach: the end-of-input forms of `@`, `#` and numbers, a bad url running to the end, TokenStream's mark/restore, whitespace and prelude skipping, and the name and range helpers. Claude-Session: https://claude.ai/code/session_017TkBWTk3wcyUcJYDp5ZhaP * test: cover the HTML tokenizer from its own tests The tokenizer suite now reaches every function of the tokenizer and all of its statements and branches but sixteen that no input can reach: the content modes a start tag or fragment context switches to, NUL in every text state, out-of-range character references, and tokenizeXml's tags, attributes, comments, CDATA, processing instructions and DOCTYPE. Claude-Session: https://claude.ai/code/session_017TkBWTk3wcyUcJYDp5ZhaP * test: share the HTML tree helpers between the HTML suites The tokenizer, parser and printer suites each carried their own copy of the helpers that materialize the struct-of-arrays tree (childrenOf, materialize, parseHtml, child, html, body) and of the Mat* typedefs. They now live once in test/helpers/materializeHtmlTree.js. Claude-Session: https://claude.ai/code/session_017TkBWTk3wcyUcJYDp5ZhaP * test: cover the JavaScript tokenizer from its own tests Table-driven comparisons with acorn's tokenizer across ecmaVersions and options (punctuators, numbers, strings, templates, regexps, comments, identifiers, keywords and regexp-vs-division contexts), plus the callbacks, error locations and eof behaviour. Every tokenizer function is reached; what is left is a readNumber guard no input reaches and the context-stack fallback of inGeneratorContext. Claude-Session: https://claude.ai/code/session_017TkBWTk3wcyUcJYDp5ZhaP * test: share the JavaScript parse helpers between its suites The tokenizer and parser suites each carried the same `parse` (the options a build hands the parser) and `plain` (a tree, token or error as comparable data). They now live once in test/helpers/javascriptParse.js. Claude-Session: https://claude.ai/code/session_017TkBWTk3wcyUcJYDp5ZhaP * test: keep the XML printing tests with the XML printer The beautify and minify output, the embedded and async renderers and _xmlCharacterData sat in the HTML parser suite; they move, with their snapshots, to xml/printer. Claude-Session: https://claude.ai/code/session_017TkBWTk3wcyUcJYDp5ZhaP
1 parent 660608c commit 0c02242

31 files changed

Lines changed: 20591 additions & 19259 deletions

‎TESTING_DOCS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -142,7 +142,7 @@ Directories come first, in alphabetical order, then the individual files worth t
142142

143143
- **Purpose**: Contains `*.unittest.js` unit tests for various functionalities.
144144
- **Usage**: Ensures individual modules and functions work as expected.
145-
- **Layout**: the syntax tests sit in one folder per language — `javascript/` (`parser`, `webpack-parser`, `scope`, `printer`, `minify`, `no-acorn`), `css/`, `html/`, `xml/` and `json/` — and the tests of the `tooling/` scripts in `tooling/`. The rest stay at the top level.
145+
- **Layout**: the syntax tests sit in one folder per language — `javascript/` (`tokenizer`, `parser`, `scope`, `printer`), `css/` and `html/` (`tokenizer`, `parser`, `printer`), `xml/` (`printer`) and `json/` (`printer`, `web-manifest`). A printer's file holds both of its modes and the language's minifier, which drives the printer in its `minify` mode. The tests of the `tooling/` scripts sit in `tooling/`. The rest, webpack's own classes such as `JavascriptParser`, stay at the top level.
146146

147147
### `watchCases/`
148148

‎cspell.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -642,7 +642,7 @@
642642
"types.d.ts",
643643
"**/**/*.snap",
644644
"test/cases/json/weird-properties/globals.json",
645-
"test/unitCases/javascript/webpack-parser.unittest.js",
645+
"test/unitCases/JavascriptParser.unittest.js",
646646
"lib/javascript/data.js",
647647
"lib/javascript/syntax-printer-data.js",
648648
"tooling/html-entities.json",

‎test/helpers/javascriptParse.js‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
"use strict";
2+
3+
/** @typedef {import("../../lib/javascript/syntax-parser").CollectedComment} CollectedComment */
4+
/** @typedef {import("../../lib/javascript/syntax-parser").ParserOptions} ParserOptions */
5+
/** @typedef {{ ast: import("estree").Program, comments: CollectedComment[] }} ParseResult */
6+
7+
const { parse: webpackParse } = require("../../lib/javascript/syntax").parser;
8+
9+
/**
10+
* Parse with the options a build hands the parser: ranges alone become lazy
11+
* offsets, and comments are collected lazily or eagerly to match.
12+
* @param {string} code source
13+
* @param {object=} options extra parse options, plus `comments` to collect them
14+
* @returns {ParseResult} result
15+
*/
16+
const parse = (code, options) => {
17+
const { comments: collectComments, ...rest } =
18+
/** @type {ParserOptions & { comments?: boolean }} */ ({
19+
sourceType: "script",
20+
ecmaVersion: "latest",
21+
comments: true,
22+
ranges: true,
23+
allowHashBang: true,
24+
...options
25+
});
26+
/** @type {CollectedComment[]} */
27+
const comments = [];
28+
const lazyNodes = rest.ranges === true && rest.locations !== true;
29+
/** @type {ParserOptions} */
30+
const parserOptions = {
31+
allowReturnOutsideFunction: rest.sourceType === "script",
32+
...rest,
33+
ranges: lazyNodes ? false : rest.ranges,
34+
lazyNodes
35+
};
36+
if (collectComments === true) {
37+
if (lazyNodes) parserOptions.lazyComments = comments;
38+
else parserOptions.onComment = comments;
39+
}
40+
return { ast: webpackParse(code, parserOptions), comments };
41+
};
42+
43+
/**
44+
* A tree, token, comment or error as plain data, so acorn's and ours compare by value.
45+
* @param {unknown} value what to convert
46+
* @returns {unknown} the plain data
47+
*/
48+
const plain = (value) =>
49+
JSON.parse(
50+
JSON.stringify(value, (key, v) => {
51+
if (typeof v === "bigint") return `${v}n`;
52+
if (v instanceof RegExp) return { regexp: v.source, flags: v.flags };
53+
if (key === "type" && v && typeof v === "object" && "label" in v) {
54+
return { label: v.label };
55+
}
56+
return v;
57+
})
58+
);
59+
60+
module.exports = { parse, plain };
Lines changed: 156 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,156 @@
1+
"use strict";
2+
3+
const {
4+
A,
5+
Field,
6+
Flag,
7+
NodeType,
8+
Part,
9+
_attributeList,
10+
_namespaceOf,
11+
parseHtml: parseHtmlRefs
12+
} = require("../../lib/html/syntax-parser");
13+
14+
/** @import { HtmlNodeRef, HtmlAttribute } from "../../lib/html/syntax-parser" */
15+
16+
/**
17+
* Materialized plain-object views of the struct-of-arrays AST — the shape
18+
* `parseHtml` used to return, rebuilt through the accessor `A`. Shared by the
19+
* HTML tokenizer, parser and printer unit suites.
20+
* @typedef {object} MatElement
21+
* @property {typeof NodeType.Element} type
22+
* @property {string} tagName
23+
* @property {number} namespace
24+
* @property {HtmlAttribute[]} attributes
25+
* @property {MatNode[]} children
26+
* @property {boolean} selfClosing
27+
* @property {number} start
28+
* @property {number} end
29+
* @property {number} tagEnd
30+
* @property {number} nameEnd
31+
* @property {number} contentEnd
32+
* @property {MatFragment=} templateContent
33+
*/
34+
/** @typedef {{ type: typeof NodeType.Text | typeof NodeType.Comment, data: string, start: number, end: number }} MatText */
35+
/** @typedef {MatText} MatComment */
36+
/** @typedef {{ type: typeof NodeType.ProcessingInstruction, target: string, data: string, start: number, end: number }} MatProcessingInstruction */
37+
/** @typedef {{ type: typeof NodeType.Doctype, name: string, publicId: string | null, systemId: string | null, start: number, end: number }} MatDoctype */
38+
/** @typedef {{ type: typeof NodeType.Document, children: MatNode[] }} MatDocument */
39+
/** @typedef {{ type: typeof NodeType.DocumentFragment, children: MatNode[] }} MatFragment */
40+
/** @typedef {MatElement | MatText | MatComment | MatDoctype | MatProcessingInstruction} MatNode */
41+
42+
/**
43+
* @param {number} ref node
44+
* @returns {number[]} its children, read one by one through the path
45+
*/
46+
const childrenOf = (ref) => {
47+
const out = [];
48+
for (let i = 0, c = A.child(0, ref); c !== 0; c = A.child(++i, ref)) {
49+
out.push(c);
50+
}
51+
return out;
52+
};
53+
54+
/**
55+
* `parseHtml` hands back integer refs into reused module-level columns, valid
56+
* only until the next parse, so each tree is materialized eagerly. Every field
57+
* is read through `A`, which is how this suite reaches the whole accessor
58+
* surface.
59+
* @param {HtmlNodeRef} ref node ref
60+
* @returns {MatNode} plain-object node
61+
*/
62+
const materialize = (ref) => {
63+
const type = A.type(ref);
64+
switch (type) {
65+
case NodeType.Element: {
66+
const tc = A.field(0, Field.content, ref);
67+
return {
68+
type,
69+
tagName: A.name(ref),
70+
namespace: _namespaceOf(ref),
71+
attributes: _attributeList(ref),
72+
children: childrenOf(ref).map(materialize),
73+
selfClosing: A.flag(Flag.selfClosing, ref),
74+
start: A.start(ref),
75+
end: A.end(ref),
76+
tagEnd: A.startOf(Part.content, ref),
77+
nameEnd: A.endOf(Part.name, ref),
78+
contentEnd: A.endOf(Part.content, ref),
79+
templateContent:
80+
tc !== 0
81+
? {
82+
type: NodeType.DocumentFragment,
83+
children: childrenOf(tc).map(materialize)
84+
}
85+
: undefined
86+
};
87+
}
88+
case NodeType.ProcessingInstruction:
89+
return {
90+
type,
91+
target: A.name(ref),
92+
data: A.value(ref),
93+
start: A.range(ref)[0],
94+
end: A.range(ref)[1]
95+
};
96+
case NodeType.Doctype:
97+
return {
98+
type,
99+
name: A.name(ref),
100+
publicId: A.textOf(Part.publicId, ref),
101+
systemId: A.textOf(Part.systemId, ref),
102+
start: A.range(ref)[0],
103+
end: A.range(ref)[1]
104+
};
105+
default:
106+
// Text / Comment
107+
return {
108+
type: /** @type {typeof NodeType.Text | typeof NodeType.Comment} */ (
109+
type
110+
),
111+
data: A.value(ref),
112+
start: A.range(ref)[0],
113+
end: A.range(ref)[1]
114+
};
115+
}
116+
};
117+
118+
/**
119+
* @param {string} src source
120+
* @param {string=} fragmentContext fragment context
121+
* @param {import("../../lib/html/syntax-parser").HtmlAstSkip=} skip skip options
122+
* @returns {MatDocument} materialized document
123+
*/
124+
const parseHtml = (src, fragmentContext, skip) => {
125+
const doc = parseHtmlRefs(src, 0, { fragmentContext, skip });
126+
return {
127+
type: NodeType.Document,
128+
children: childrenOf(doc).map(materialize)
129+
};
130+
};
131+
132+
/**
133+
* @param {MatNode[]} children children
134+
* @param {string} tagName tag name
135+
* @returns {MatElement} the element
136+
*/
137+
const child = (children, tagName) =>
138+
/** @type {MatElement} */ (
139+
children.find((c) => c.type === NodeType.Element && c.tagName === tagName)
140+
);
141+
142+
// The tree builder always produces a full document (html > head, body); these
143+
// helpers reach the interesting subtrees.
144+
/**
145+
* @param {string} src source
146+
* @returns {MatElement} html element
147+
*/
148+
const html = (src) => child(parseHtml(src).children, "html");
149+
/**
150+
* @param {string} src source
151+
* @returns {MatElement[]} body children
152+
*/
153+
const body = (src) =>
154+
/** @type {MatElement[]} */ (child(html(src).children, "body").children);
155+
156+
module.exports = { body, child, childrenOf, html, materialize, parseHtml };

‎test/helpers/syntaxEquivalence.js‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -343,7 +343,7 @@ const installHelpers = (generics) => {
343343
]);
344344

345345
// A generic family is a keyword, so quoting one names a font of that name
346-
// instead, and the quoting is what tells the two apart. `css/syntax.unittest`
346+
// instead, and the quoting is what tells the two apart. `css/printer.unittest`
347347
// holds the printer to the same rule.
348348
const GENERIC_FAMILIES = new Set(generics);
349349

0 commit comments

Comments
 (0)