-
Notifications
You must be signed in to change notification settings - Fork 4.6k
Build: fully resolve import paths in transpiled files #73822
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: trunk
Are you sure you want to change the base?
Conversation
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message. To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
|
I'm actually a little bit surprised that we have to write it manually ourselves, I've kind expected esbuild (or some plugin) to do it for us already as it feels like a very common thing for folks building packages these days. |
|
You are not alone who is surprised how tough it is to build fully compliant dual CJS/ESM TypeScript packages and how unprepared the common tools are. There are specialized tools like duel or tsup which are rather complex and whose inner workings are non-trivial to reproduce in a build script like ours. Earlier this year I worked on a dual package in the Jetpack repo. Initially I tried to do my own build script, but eventually gave up and just used |
|
I think ESBuild considers relations between files primarily around bundling, which isn't how we're using it. Although to a some extent, JavaScript / Node.js is shifting toward import paths including file extensions, where this behavior of automatic extension resolution is completely unsupported in ESM (source). So if there was an expectation that the file extensions are included in the source files, we wouldn't need this code ( |
packages/wp-build/src/build.mjs
Outdated
| ); | ||
| if ( isDir ) { | ||
| return { | ||
| path: args.path + '/index.js', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we use path.join ? Not sure about the Windows compatibility of this slash.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, but also Node normalizes all paths so Windows compatibility isn’t an issue.
On Windows, both the forward slash (/) and backward slash () are accepted as path segment separators; however, the path methods only add backward slashes ().
packages/wp-build/src/build.mjs
Outdated
| ! args.path.endsWith( '.js' ) | ||
| ) { | ||
| const baseDir = path.dirname( args.importer ); | ||
| for ( const ext of [ '.js', '.jsx', '.ts', '.tsx' ] ) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we reuse SOURCE_EXTENSIONS ? Maybe what we have currently as SOURCE_EXTENSIONS should be a separate constant that uses the raw array of extension values.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually, I wonder if we should be using some internal property of ESBuild like resolveExtensions.
To that point, is it possible to tap into ESBuild's internal resolution behavior so we don't have to reimplement this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I rewrote the plugin to use esbuild's internal build.resolve function. In the onResolve callback, ask esbuild to resolve the import (preventing recursion with pluginData) and then only modify the returned path. By replacing the source extensions (.tsx and others) with the target extension, which is either .js or .mjs.
That simplifies the plugin a lot. I no longer need to stat physical files, think about caching etc.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we reuse
SOURCE_EXTENSIONS?
Probably yes, but there are complications: the globs want a bracket expansion syntax without dots, {js,jsx}, and for my code it's more convenient to have an array with dots, [ '.js', '.jsx' ]. Etc.
Let's consider this after everything else in this PR is in perfect order 🙂
packages/wp-build/src/build.mjs
Outdated
| ) { | ||
| const baseDir = path.dirname( args.importer ); | ||
| for ( const ext of [ '.js', '.jsx', '.ts', '.tsx' ] ) { | ||
| const isFile = await pathIsFile( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there enough likelihood of reuse that it'd be worth considering to cache this result?
packages/wp-build/src/build.mjs
Outdated
| } | ||
| } | ||
|
|
||
| const isDir = await pathIsDirectory( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we could be more efficient to call stat once for the path and reuse the fs.Stats result for checking if it's a file or directory. With this code, we'll call stat twice for directory imports.
3847d0d to
1f3a7de
Compare
|
Size Change: +746 B (+0.03%) Total Size: 2.58 MB
ℹ️ View Unchanged
|
|
The current state of the PR is quite nice, although CI checks are still failing and attw is not happy 🙂
This should produce very nice dual ESM/CJS packages, although attw is very unhappy with the types for reasons I don't understand: I'll have to debug this further. An alternative approach would be to use |
|
Reasons why things fail:
|
The CI action uses We'll have to add |
95aec17 to
474265f
Compare
|
Some progress I made today:
There are still some wrinkles to iron out with |
| relativePath = | ||
| relativePath.slice( 0, -ext.length ) + newExt; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Feels like a job for path.basename with second argument.
| relativePath = | |
| relativePath.slice( 0, -ext.length ) + newExt; | |
| relativePath = | |
| path.basename( relativePath, ext ) + newExt; |
| // Replace extension: if path ends with one of the extensions, replace it | ||
| const exts = [ '.js', '.jsx', '.ts', '.tsx' ]; | ||
| const newExt = | ||
| build.initialOptions.format === 'cjs' ? '.cjs' : '.js'; | ||
| for ( const ext of exts ) { | ||
| if ( relativePath.endsWith( ext ) ) { | ||
| relativePath = | ||
| relativePath.slice( 0, -ext.length ) + newExt; | ||
| break; | ||
| } | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We could probably do this with RegExp, maybe a bit more efficiently too. Not as future-compatible to if we wanted to actually consistently define extensions somewhere as an array (could probably construct from array using new RegExp constructor).
| // Replace extension: if path ends with one of the extensions, replace it | |
| const exts = [ '.js', '.jsx', '.ts', '.tsx' ]; | |
| const newExt = | |
| build.initialOptions.format === 'cjs' ? '.cjs' : '.js'; | |
| for ( const ext of exts ) { | |
| if ( relativePath.endsWith( ext ) ) { | |
| relativePath = | |
| relativePath.slice( 0, -ext.length ) + newExt; | |
| break; | |
| } | |
| } | |
| // Replace extension: if path ends with one of the extensions, replace it | |
| const exts = [ '.js', '.jsx', '.ts', '.tsx' ]; | |
| const newExt = | |
| build.initialOptions.format === 'cjs' ? '.cjs' : '.js'; | |
| relativePath = relativePath.replace( /\.[jt]sx?$/, '' ) + newExt; |
|
Today I had to solve one more big problem: incompatibility in how esbuild treats default imports from CJS modules. Consider a CJS module that exports this: module.exports = {
__esModule: true,
default: function foo() { /* ... */ }
};What happens when you import this CJS module with an ESM import foo from './foo';The classic webpack/Babel behavior is that you get the function from the But Node.js doesn't respect these tricks and will return the What's worse is that esbuild uses the Node.js behavior when there is the One example is import isShallowEqual from '@wordpress/is-shallow-equal';The module.exports = window.wp.isShallowEqual;But the I fixed this by adding a named Another example is the import TextareaAutosize from 'react-autosize-textarea';Again, the package exports a import { TextareaAutosize } from 'react-autosize-textarea/lib/TextareaAutosize';I'm afraid we'll see many more issues like this once we start using @anomiex @manzoorwanijk Did you run into similar issues when using |
|
https://github.com/arethetypeswrong/arethetypeswrong.github.io/blob/main/docs/problems/CJSOnlyExportsDefault.md has a detailed writeup on the problem. #59087 is about the same thing with respect to |
640c7e9 to
ab6c8b2
Compare
|
Flaky tests detected in ab6c8b2. 🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/20147232455
|
|
|
||
| // `isShallowEqual` is exported also as a named export because esbuild cannot | ||
| // expose the default export from the `window.wp.isShallowEqual` global. | ||
| export { isShallowEqual, isShallowEqualObjects, isShallowEqualArrays }; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should do the same for all the packages mentioned in #59087 (comment)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should do the same for all the packages mentioned in #59087 (comment)
All the other packages should be OK, because they:
- either have a single default export and the
wpScriptDefaultExportflag. Thenmodule.exportsvalue is directly the exported function. - or multiple named exports and no default. Then
module.exportsis an object with a few named field, none of themdefault.
Both types of modules can be easily mapped between CJS and ESM without any problems.
The is-shallow-equal module is a combination of default and named exports. That's the only case that needs fixing.
Maybe I'm wrong, I'll need to look more closely what exactly is the problem with api-fetch as reported in #59087.
This PR implements the same thing as #73422, but as part of esbuild
onResolvecallback, without a separate post-processing step with Babel.Local imports like
import from './proxy'are resolved either to'./proxy.js'or'./proxy/index.js', depending on which file/directory exists. That way the imports become compatible with Node.js ESM import rules, and also with browsers' native ESM import rules. They both require full file paths, without any shortcuts.I originally wanted to eliminate local imports completely in #73516, but that proved too ambitious and risky.
This esbuild approach is very simple and flexible. If we want to, we can easily modify it to make all imports use the
.mjsor.cjsextension. That will be useful for creating fully compliant dual CJS/ESM packages.Before we merge this, I'd like to fine-tune the algorithm a bit. Make sure that special paths like
'.'and'..'are handled correctly, that imports with an existing non-JS extension (likeblock.json) are not handled with this code, etc. But I didn't want to delay the PR further. Let's test and review.