Run tests in a native Node.js ESM environment #13966

nicolo-ribaudo · 2021-11-14T22:33:00Z

Q	A
Any Dependency Changes?
License	MIT

With this PR I'm trying to run our tests using the native Node.js ESM implementation, rather than Jest's one.

I created a custom jest runner still uses jest-circus/jest-each/jest-mock/jest-expect/jest-snapshot to provide a Jest-like environment, but runs them (and the tests) in the current context rather than spawning a new vm.Context. By doing so, we don't need to wait for Node.js to stabilize the ESM vm api so that Jest can properly use it (it's currently behind an experimental Node.js flag, and Jest's support isn't complete yet also because of some V8 bugs). Also, this approach is faster (2x on my machine).

However, there are some downsides:

We don't have the strong isolation that Jest gives anymore, so one test could set some global state which would be accessible by other tests
We can only mock files manually

(1) doesn't affect us since Babel doesn't rely on global state. I am working on a solution for (2).

This PR contains many changes to align our tests with an ESM-compatible implementation, but I'll extract them to separate PRs:

Use full import specifier path in tests (Use full import specifier path in tests #13938)
Only bundle the release build, and don't import src in tests (Only bundle the release build, and don't import src in tests #13978)
Use Node.js interop when importing lib in test (Use Node.js interop when importing lib in test #13995)
Don't use ESM for required files in @babel/helpers tests (Don't use ESM for required files in @babel/helpers tests #13996)

TODO:

~~Either avoid mocking in @babel/register tests, or make it work~~ Made it work!

codesandbox-ci · 2021-11-14T22:36:57Z

This pull request is automatically built and testable in CodeSandbox.

To see build info of the built libraries, click here or the icon next to each commit SHA.

Latest deployment of this branch, based on commit ac23359:

Sandbox	Source
babel-repl-custom-plugin	Configuration
babel-plugin-multi-config	Configuration

babel-bot · 2021-11-17T18:26:58Z

Build successful! You can test your changes in the REPL here: https://babeljs.io/repl/build/50072/

JLHwung · 2021-11-30T16:00:19Z

test/jest-light-runner/src/index.js

+   * @param {*} onFailure
+   */
+  runTests(tests, watcher, onStart, onResult, onFailure) {
+    return Promise.all(


Should it be Promise.allSettled? If onFailure returns a rejected promise, other pending tests should not end.

onFailure returns a resolved promise when it's called to report a failure. If it rejects, there is a bug in the test runner (not in a test) so we can exit and report the bug.

JLHwung · 2021-11-30T16:00:58Z

test/jest-light-runner/src/index.js

+            {
+              test,
+              port: mc.port1,
+              updateSnapshot: this.#config.updateSnapshot,


Nit: updateSnapshot can be hoisted.

JLHwung · 2021-11-30T16:13:02Z

test/jest-light-runner/src/index.js

+
+const piscina = new Piscina({
+  filename: new URL("./worker-runner.js", import.meta.url).href,
+  maxThreads: os.cpus().length / 2,


Test on my local machine is faster when I use the default maxThreads (cpus().length * 1.5) (29s vs 26s), but

babel/packages/babel-register/test/cache.js

Lines 96 to 106 in 8936501

it("should create the cache after dirty", () => {

load();

setDirty();

return new Promise(resolve => {

process.nextTick(() => {

expect(fs.existsSync(testCacheFilename)).toBe(true);

expect(get()).toEqual({});

resolve();

});

});

});

is randomly failing. Guess process.nextTick has racing conditions?

I don't see that failure 🤔 Also, all the tests in a single file are run serially, so it shouldn't have any race conditions.

Btw, for me it takes way longer 😬 33s with / 2 vs 50s with * 1.5.

Here is an

error output

- Expected - 1 + Received + 45 - Object {} + Object { + "{\"assumptions\":{},\"sourceRoot\":\"/path/to/babel/packages/babel-register/test/fixtures/babelrc/\",\"cwd\":\"/path/to/babel/packages/babel-register/test/fixtures/babelrc\",\"babelrc\":false,\"sourceMaps\":false,\"caller\":{\"name\":\"@babel/register\"},\"filename\":\"/path/to/babel/packages/babel-register/test/fixtures/babelrc/es2015.js\",\"targets\":{},\"cloneInputAst\":true,\"configFile\":false,\"browserslistConfigFile\":false,\"passPerPreset\":false,\"envName\":\"undefined\",\"root\":\"/path/to/babel/packages/babel-register/test/fixtures/babelrc\",\"rootMode\":\"root\",\"plugins\":[],\"presets\":[]}:7.16.0:undefined": Object { + "ast": null, + "code": "const a = 1;", + "map": null, + "metadata": Object {}, + "mtime": 1637340521216, + "options": Object { + "assumptions": Object {}, + "ast": false, + "babelrc": false, + "browserslistConfigFile": false, + "caller": Object { + "name": "@babel/register", + }, + "cloneInputAst": true, + "configFile": false, + "cwd": "/path/to/babel/packages/babel-register/test/fixtures/babelrc", + "envName": "undefined", + "filename": "/path/to/babel/packages/babel-register/test/fixtures/babelrc/es2015.js", + "generatorOpts": Object { + "comments": true, + "compact": "auto", + "filename": "/path/to/babel/packages/babel-register/test/fixtures/babelrc/es2015.js", + "sourceFileName": "es2015.js", + "sourceMaps": false, + "sourceRoot": "/path/to/babel/packages/babel-register/test/fixtures/babelrc/", + }, + "parserOpts": Object { + "plugins": Array [], + "sourceFileName": "/path/to/babel/packages/babel-register/test/fixtures/babelrc/es2015.js", + "sourceType": "module", + }, + "passPerPreset": false, + "plugins": Array [], + "presets": Array [], + "root": "/path/to/babel/packages/babel-register/test/fixtures/babelrc", + "rootMode": "root", + "sourceMaps": false, + "sourceRoot": "/path/to/babel/packages/babel-register/test/fixtures/babelrc/", + "targets": Object {}, + }, + "sourceType": "module", + }, + } at file:/path/to/babel/packages/babel-register/test/cache.js:121:25

The test cache seems to be created by another test test/index.js but then read by test/cache.js, however I can't reproduce it consistently.

33s with / 2 vs 50s with * 1.5.

Hmm, interesting. I think it is related to the Intel hyper-threading, which doubles the available processor count... Anyway the current setting is good enough for me, don't bother change that.

Could you check if c385768 solves the race condition?

JLHwung · 2021-11-30T16:18:25Z

test/jest-light-runner/src/worker-runner.js

+
+import "./global-setup.js";
+
+/** @typedef {import("@jest/test-result").Test} Test */


Is the typedef used?

JLHwung · 2021-11-30T16:21:16Z

test/jest-light-runner/src/worker-runner.js

+
+/**
+ *
+ * @param {*} stats


Can we supplement the shape of stats here?

JLHwung · 2021-11-30T20:35:07Z

test/jest-light-runner/src/worker-runner.js

+    numPassingTests: stats.passes,
+    numPendingTests: stats.pending,
+    perfStats: {
+      end: +new Date(stats.end),


end and start seem to be not yet implemented. They are always undefined on my local machine.

We can implement using the performance API available since Node 8.5.

The perfStats also includes slow and runtime, which can be accordingly implemented

https://github.com/facebook/jest/blob/3a85065fe5604655e1337ffc1631f9999722c821/packages/jest-runner/src/runTest.ts#L293-L299

JLHwung · 2021-11-30T22:08:00Z

eslint/babel-eslint-tests/test/helpers/verifyAndAssertMessages.js

        ),
      },
-      ...overrideConfig?.parserOptions,
+      ...(overrideConfig && overrideConfig.parserOptions),


Now that we skip jest script transformer, should we add https://github.com/mysticatea/eslint-plugin-node/blob/master/docs/rules/no-unsupported-features/es-syntax.md to non-fixture test files?

Oh thanks! I was looking for this rule but didn't remember the plugin name.

I have enabled it, but support for ?? and ?. hasn't been released yet (so it doesn't warn about them).

JLHwung

Awesome work! This PR replaces the default jest-runner by jest-light-runner, which removes the test environment (an isolated VM context) and the script transforms (babel-jest). Although the test runner is backed by worker pools slower than the process-based approach used in jest-worker, we still see significant CI runtime improvement: This branch on Node 16(2m3s) vs main on Node 16(4m57s). Bravo!

nicolo-ribaudo · 2021-12-01T09:51:36Z

I squashed the various review commits into the relative main ones to make it easier for the next reviewer.

nicolo-ribaudo · 2021-12-03T14:32:45Z

🎉

The next steps I'm working on to release an ESM Babel versions are:

Make @babel/register work
Use exports in package.json files

nicolo-ribaudo force-pushed the jest-light branch from 1c17894 to debfaee Compare November 14, 2021 23:10

This was referenced Nov 15, 2021

Transition to vanilla javascript / ES Modules #13964

Closed

Use full import specifier path in tests #13938

Merged

nicolo-ribaudo force-pushed the jest-light branch 8 times, most recently from e09c850 to 73968df Compare November 17, 2021 18:23

nicolo-ribaudo force-pushed the jest-light branch from 73968df to f99c11b Compare November 17, 2021 18:31

This comment has been minimized.

Sign in to view

nicolo-ribaudo force-pushed the jest-light branch from 5e272c3 to fcc5f37 Compare November 18, 2021 21:14

This comment has been minimized.

Sign in to view

nicolo-ribaudo force-pushed the jest-light branch 2 times, most recently from 6b1e33d to d9a4733 Compare November 18, 2021 21:35

This comment has been minimized.

Sign in to view

nicolo-ribaudo force-pushed the jest-light branch from d9a4733 to 67f4458 Compare November 18, 2021 21:48

This comment has been minimized.

Sign in to view

nicolo-ribaudo force-pushed the jest-light branch 3 times, most recently from 7730462 to 6f9abcd Compare November 18, 2021 23:15

nicolo-ribaudo mentioned this pull request Nov 18, 2021

Only bundle the release build, and don't import src in tests #13978

Merged

nicolo-ribaudo force-pushed the jest-light branch 3 times, most recently from 39df24a to db81f8c Compare November 24, 2021 22:06

nicolo-ribaudo added the area: tests label Nov 30, 2021

JLHwung reviewed Nov 30, 2021

View reviewed changes

JLHwung approved these changes Dec 1, 2021

View reviewed changes

nicolo-ribaudo added 8 commits December 1, 2021 10:50

Add necessary package.json files

5c132e4

Implement @babel/jest-light-runner

d15e411

Misc updates

d287134

Don't rely on .chdir() in @babel/node and @babel/cli tests

478bc69

Fix tests on Node.js 12

52222cf

Remove unnecessary ESM Jest resolver

0fb8167

Update jest

c2c4554

Fix @babel/register test mocks

c7bd3a6

nicolo-ribaudo force-pushed the jest-light branch from 70fc2e7 to c7bd3a6 Compare December 1, 2021 09:50

Update .gitignore

5acc4ca

nicolo-ribaudo requested a review from hzoo December 1, 2021 23:54

existentialism approved these changes Dec 3, 2021

View reviewed changes

nicolo-ribaudo merged commit 2d989a9 into babel:main Dec 3, 2021

nicolo-ribaudo deleted the jest-light branch December 3, 2021 14:32

nicolo-ribaudo mentioned this pull request Dec 5, 2021

Bring back deferred require() calls in core and cli #14028

Merged

tolmasky mentioned this pull request Jan 25, 2022

Fuzz test location-related parser options #14201

Merged

snyk-bot mentioned this pull request Feb 22, 2022

[Snyk] Upgrade: @babel/core, @babel/preset-env dfds/tika#39

Closed

github-actions bot added the outdated A closed issue/PR that is archived due to age. Recommended to make a new issue label Mar 5, 2022

github-actions bot locked as resolved and limited conversation to collaborators Mar 5, 2022

	it("should create the cache after dirty", () => {
	load();
	setDirty();
	return new Promise(resolve => {
	process.nextTick(() => {
	expect(fs.existsSync(testCacheFilename)).toBe(true);
	expect(get()).toEqual({});
	resolve();
	});
	});
	});


		import "./global-setup.js";

		/** @typedef {import("@jest/test-result").Test} Test */

Uh oh!

Run tests in a native Node.js ESM environment #13966

Run tests in a native Node.js ESM environment #13966

Uh oh!

Conversation

nicolo-ribaudo commented Nov 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codesandbox-ci bot commented Nov 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

babel-bot commented Nov 17, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JLHwung Nov 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JLHwung left a comment

Choose a reason for hiding this comment

Uh oh!

nicolo-ribaudo commented Dec 1, 2021

Uh oh!

nicolo-ribaudo commented Dec 3, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

nicolo-ribaudo commented Nov 14, 2021 •

edited

Loading

codesandbox-ci bot commented Nov 14, 2021 •

edited

Loading

babel-bot commented Nov 17, 2021 •

edited

Loading

JLHwung Nov 30, 2021 •

edited

Loading