Skip to content

Cache MDX compiler in the Webpack loader#1468

Merged
JounQin merged 1 commit intomdx-js:mainfrom
remcohaszing:cache-mdx-loader
Apr 8, 2021
Merged

Cache MDX compiler in the Webpack loader#1468
JounQin merged 1 commit intomdx-js:mainfrom
remcohaszing:cache-mdx-loader

Conversation

@remcohaszing
Copy link
Copy Markdown
Member

A new MDX compiler was created for every file that’s processed. This means that any initialization logic in remark or rehype plugins was always run for every file.

By reusing the MDX compiler, remark and rehype plugins can run heavy setup logic once, just like when they are called by unified directly.

In practice I use this in remark-mermaidjs. This starts a browser once to process a bunch of files, but it requires the same unified processor to run to be able to do so.

The same principle is used in ts-loader to cache the TypeScript compiler.

A new MDX compiler was created for every file that’s processed. This means that
any initialization logic in remark or rehype plugins was always run for every
file.

By reusing the MDX compiler, remark and rehype plugins can run heavy setup logic
once, just like when they are called by unified directly.
@vercel
Copy link
Copy Markdown

vercel Bot commented Feb 23, 2021

This pull request is being automatically deployed with Vercel (learn more).
To see the status of your deployment, click below or on the icon next to each commit.

🔍 Inspect: https://vercel.com/mdx/mdx/98pc7aCniJSuaojuJsD8ikpgZopW
✅ Preview: Failed

@vercel vercel Bot temporarily deployed to Preview February 23, 2021 10:41 Inactive
Comment thread packages/loader/index.js
Comment thread packages/loader/index.js
@JounQin JounQin merged commit e36caef into mdx-js:main Apr 8, 2021
@slorber
Copy link
Copy Markdown
Contributor

slorber commented Jan 20, 2022

@wooorm
Copy link
Copy Markdown
Member

wooorm commented Jan 20, 2022

I believe xdm was fast enough, it does a lot less work. Do you need it then?

@remcohaszing
Copy link
Copy Markdown
Member Author

The main issue isn’t the MDX compiler, but the plugins that may be slow.

Specifically remark-mermaidjs, which uses puppeteer, is slow. A lot of heavy lifting could be ran only once instead of per file when reusing the same browser instance, but this requires the unified pipeline to be reused.

Personally I haven’t been affected by this issue lately, because I disabled remark-mermaidjs in development (watch) mode. Still this optimization would be nice for those who do use it in development, and it would also significantly speed up production builds for my project.

@wooorm
Copy link
Copy Markdown
Member

wooorm commented Jan 22, 2022

Oh right. This was around the xdm time, I believe the plan was to land it there too but it never did?
xdm has proper caching mechanisms for rollup, esbuild, and node, which have been ported over to mdx-js/mdx. But not for webpack.

wooorm added a commit to wooorm/xdm that referenced this pull request Jan 23, 2022
wooorm added a commit to wooorm/xdm that referenced this pull request Jan 23, 2022
wooorm added a commit that referenced this pull request Jan 25, 2022
wooorm added a commit that referenced this pull request Jan 25, 2022
@wooorm
Copy link
Copy Markdown
Member

wooorm commented Jan 25, 2022

added again!

wooorm added a commit that referenced this pull request Jan 26, 2022
Related-to: GH-1468.
Related-to: GH-1912.

Reviewed-by: Remco Haszing <[email protected]>
Reviewed-by: Christian Murphy <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

5 participants