Skip to content

fix(DST-1904): run docs loaders in worker threads on Vercel - #5988

Merged
aromko merged 2 commits into
mainfrom
fix/DST-1904-docs-build-memory
Oct 22, 2026
Merged

aromko merged 2 commits into
mainfrom
fix/DST-1904-docs-build-memory

Conversation

@sebald

@sebald sebald commented Oct 22, 2026 •

Copy link
Copy Markdown
Member

Description

The marigold-docs build on Vercel is intermittently SIGKILLed with out_of_memory during Turbopack's compile step. It started with the Vite 8.3.2 → 8.3.3 bump (#5980) and blocks the "Version Packages" release PR, because Vercel – marigold-docs is a required status check.

This PR lowers the build's peak memory instead of reverting any dependency. On Vercel preview and production builds only, Turbopack's webpack loaders (mostly the fumadocs-mdx MDX compile) now run in worker threads instead of a pool of separate Node child processes:

turbopackPluginRuntimeStrategy: ['preview', 'production'].includes(process.env.VERCEL_ENV)
  ? 'forceWorkerThreads'
  : undefined,

Local runs (pnpm dev, pnpm build, vercel dev) keep Next's default. The investigation below is kept in full so it can be reused the next time the docs build runs out of memory.

Closes DST-1904

Root cause

The docs build has been running close to the 8 GB limit of Vercel's standard build machine (4 cores, 8 GB), and its peak memory swings by about 1 GB between runs of identical code. The Vite bump didn't break anything; it tipped a borderline build over the edge.

Evidence from Vercel:

  • The OOM always happens inside next build → "Creating an optimized production build" (Turbopack compile), about 50s in. Green builds on the same machine compile in 26s–58s, so the compile phase itself is highly variable.
  • Same tree, different outcome. The chore(deps): bump vite from 8.3.2 to 8.3.3 #5980 PR head e0b2e7e has a tree byte-identical to the failing main commit 62da88a (git diff --stat e0b2e7e 62da88a is empty) and built green.
  • Clear onset. In the ~5 weeks of failed docs builds before the bump there was no OOM (failures were ERR_PNPM_OUTDATED_LOCKFILE, git_info_fail or plain build errors). Since Vite 8.3.3 landed, 5 of 8 docs builds containing it were OOM-killed:
Deployment Branch Commit Result
dpl_6xTpNWTYEKhDZjfs7UwzhqinaRmT renovate/npm-vite-vulnerability 086ef40 OOM
dpl_7mVoemwpYT7XPgr8oVXopNSA4Efc renovate/npm-vite-vulnerability bd783db OOM
dpl_2a4ZRERFRUQUxKHBZekhPeNiSe3H dependabot/npm_and_yarn/vite-8.3.3 15898e6 green (compile 26.4s)
dpl_33nSKAvUnzuQZbgUZ8UTZKGUV3Z7 dependabot/npm_and_yarn/vite-8.3.3 e0b2e7e green (compile 33.1s)
dpl_BDeAi5f1ZkWvhEJev1gjvg4HtTGj main (production) 62da88a OOM
dpl_JBrBfuadtPRRf28hGBQ7WLuh4vTJ changesets-ghcommit-temp/changeset-release/main 96dd25d OOM
dpl_4vesPbs3knC2ztsZebk4oLohL6mY changeset-release/main 96dd25d OOM
dpl_Ef6eF4vcHBJ4owpF5MsJ2XEqQYgr dst-1751_select-all-for-none-table-collections 2db3d3e green (compile 58s)

The last good production deployment is dpl_DWb7szA6ukAdc5pLFs66gHXeEgSL (f8441c2); the live docs stay on it until this is fixed.

What the bump changed for the docs build. The diff only touches vite in the root and packages/cli manifests plus a lockfile dedupe. The one change inside the docs importer is that fumadocs-mdx is re-resolved against new peers (rolldown 1.2.11 → 1.2.13, vite 8.3.2 → 8.3.3). The fumadocs-mdx webpack loader that Next uses does not import Vite or Rolldown; only its dist/vite and rolldown adapters do. So there is no direct mechanism. The bump shifted a noisy distribution just enough.

Where the memory goes

A snapshot of the next build process tree at its compile peak (local, macOS):

Process Count Memory (RSS)
Turbopack webpack-loader workers (pool_entry-[turbopack-node]_transforms_webpack-loaders) 24 1,849 MB
next-build (Rust compiler) 1 1,366 MB
PostCSS loader worker 1 96 MB
Everything else (npm exec, git log for last-modified, esbuild) 7 ~105 MB
Total 33 3,419 MB

More than half the peak is the pool of Node child processes that run JS loaders, each with its own Node runtime and heap. Across runs the pool grew to 35–51 processes on an 8-core machine. That pool is what this PR targets.

What was measured

Method. Local runs on macOS (16 GB, 8 cores, Node 24.11.0) in detached worktrees. Peak = maximum of the summed RSS of the next build process tree, sampled every 0.5s. "Compile peak" stops at Compiled successfully. Summed RSS counts shared pages once per process, so it overstates the child-process setup and the absolute numbers don't map 1:1 to Vercel's Linux container. Treat them as relative comparisons. The run-to-run noise is about ±1 GB, so every variant ran 2–3 times.

1. Cold A/B: last green commit vs. failing commit

Commit Peak run 1 Peak run 2 Compile time
f8441c2 (green on Vercel) 3,823 MB 4,452 MB 37.8s / 41s
62da88a (red on Vercel) 4,105 MB 5,250 MB 38.2s / 43s

Possibly slightly higher, but within noise. This matches "borderline build tipped over", not "regression".

2. Turbopack build cache (experimental.turbopackFileSystemCacheForBuild, default true in 16.4.2): ruled out

Simulates Vercel exactly: f8441c2 was built in place, then 62da88a was checked out at the same path and built on top of that cache.

Variant Compile peak Compile time
default, cache from previous lockfile 3,826 / 5,586 / 5,166 MB 94s / 83s / 86s
turbopackFileSystemCacheForBuild: false 5,498 / 5,664 / 5,561 MB 30.2s / 33.5s / 38.8s

Disabling the cache doesn't lower the peak. Side finding worth a separate look: a cache restored from a build under a different lockfile makes the compile about 2.5× slower than a cold build. Vercel restores the previous deployment's cache on every build, so dependency-bump builds pay this. Not a memory problem, but it costs build minutes.

Pitfall for anyone repeating this: the Turbopack cache stores absolute paths (for example fumadocs-mdx's .source/source.config.mjs). A cache copied to a different directory fails with "webpack loader path … is outside configured roots". Always test cache scenarios at the same path. Vercel always builds at /vercel/path0, so this is not an issue there.

3. Capping Turbopack's Rust thread pools: ruled out

Variant Compile peak Threads in build process
baseline 4,121 / 5,451 / 3,587 MB ~28
TOKIO_WORKER_THREADS=2 5,824 / 5,298 / 4,660 MB ~28
TOKIO_WORKER_THREADS=2 RAYON_NUM_THREADS=2 5,598 / 4,696 / 5,420 MB ~28

Both names are compiled into the @next/swc binary, but Next configures its runtimes explicitly. The thread count doesn't change, so the env vars have no effect.

4. Loader runtime: child processes vs. worker threads (this PR)

experimental.turbopackPluginRuntimeStrategy, added in Next 16.4.0, documented in node_modules/next/dist/docs/01-app/03-api-reference/05-config/01-next-config-js/turbopackPluginRuntimeStrategy.md.

Strategy Compile peak (processes at peak) Avg Compile time Static generation
childProcesses (default) 3,335 (35) / 4,430 (44) / 4,210 (51) MB 3,992 MB 60s / 45s / 68s 11.8s / 12.4s / 11.8s
workerThreads 3,102 (19) / 3,708 (13) / 3,643 (9) MB 3,484 MB 61s / 32.9s / 49s 15.7s / 9.9s / 15.3s

About 0.5 GB (~13%) lower on average, consistently lower in each pair, all builds green, build time unchanged. Because summed RSS overcounts child processes, the real saving on Vercel is probably somewhat smaller. That's why this PR also turns on Vercel's build system report (see Rollout).

Options checked and not available

  • Turbopack memory limit: no such option in Next 16.4.2 (memoryLimit doesn't appear in config-schema.js).
  • experimental.turbopackMemoryEviction: documented as "only effective in dev sessions".
  • experimental.turbopackGc: prunes unreachable tasks from the persistent cache, which doesn't lower the peak.
  • Webpack loader pool size: not configurable.
  • experimental.cpus / staticGenerationMaxConcurrency: these only affect static page generation, which runs after the compile step where the OOM happens.

Why forceWorkerThreads, and why only on Vercel builds

On Node >= 24.13.1, Next ignores 'workerThreads', prints a warning and falls back to child processes (next/dist/server/config.js). That guards against nodejs/node#65100: the process can abort when a worker thread exits while a native addon holds a live Node-API threadsafe function. Vercel's 24.x is newer than 24.13.1, so 'forceWorkerThreads' is the only way to get worker threads there.

The trigger Next names is fsevents, the macOS file watcher. Vercel builds on Linux, where fsevents isn't installed and next build doesn't watch files. On developer Macs, fsevents is installed and .node-version pins only 24, so a teammate on the latest Node 24 would be exposed. Hence the scoping:

VERCEL_ENV Where Strategy
unset pnpm dev / pnpm build locally childProcesses (Next default)
development vercel dev locally childProcesses (Next default)
preview Vercel preview builds workerThreads (forced)
production Vercel production builds workerThreads (forced)

Verified locally by loading the config through next/dist/server/config with each VERCEL_ENV value.

Both Vercel projects build the docs

Project Serves Builds Machine
marigold-docs PR previews, main (marigold-docs-git-main-…) every branch 4 cores, 8 GB
marigold-production www.marigold-ui.io only the docs branch (an ignore step cancels everything else), which release.yml updates after each npm publish 4 cores, 8 GB

Both run pnpm build in docs/ with the same next.config.mjs, so this change covers both: marigold-production builds with VERCEL_ENV=production. It hasn't failed yet only because its last build (dpl_6z9aSsoQwrzPQTdBNhyzdDF91gBd, Oct 12, compile 55s) predates the Vite bump. The first build after the next release would have been the first one at risk, and a failure there would keep the live site on the old version.

Rollout and validation

  • VERCEL_BUILD_SYSTEM_REPORT=1 is now set on both projects: marigold-docs (production + preview, env id lG9sMyZmUQh7oBlx) and marigold-production (production; its previews never build). Vercel's own OOM message recommends it. Every build log now ends with a build system report, so we get the real memory numbers from the Linux container instead of macOS estimates. It's a diagnostic setting and can be removed once things are stable.
  • The failure is intermittent (about 5 in 8), so a single green preview proves nothing. Merge only after 3–5 preview builds in a row succeed.
  • If a preview build aborts instead of running out of memory (a crash at worker teardown rather than SIGKILL), we've hit worker_threads Worker exit aborts (SIGABRT) when a native addon (fsevents) has an in-flight napi_threadsafe_function — napi_release_threadsafe_function / uv_mutex_lock nodejs/node#65100 on Linux after all. Revert this PR.
  • If OOMs continue with worker threads, the remaining option that keeps all dependency updates is Enhanced Builds for marigold-docs (16 GB build machine, paid).

Test Instructions

  1. Open this PR's marigold-docs Vercel build log. In the next build header, the experiments list should show turbopackPluginRuntimeStrategy: "workerThreads" along with Next's warning that 'forceWorkerThreads' bypasses the Node.js guard. The log should end with the "Build system report".
  2. Redeploy the preview 3–5 times (Vercel UI → Redeploy, or vercel redeploy <preview-url> --scope marigold). Every build must finish, and the report shows the peak memory.
  3. Locally: cd docs && pnpm build. The experiments list must not show turbopackPluginRuntimeStrategy, because local builds keep Next's default.
  4. Optional: check the config resolution without building:
    cd docs
    for e in "" development preview production; do
      VERCEL_ENV=$e node -e "require('next/dist/server/config').default('phase-production-build', process.cwd()).then(c => console.log('VERCEL_ENV=${e:-unset} ->', c.experimental.turbopackPluginRuntimeStrategy))"
    done
    Expected: unset and development → childProcesses, preview and production → workerThreads.

Breaking Changes

No

Checklist

  • Storybook preview and Marigold docs preview are available
  • Stories added/updated (with component-test tag where applicable)
  • Unit tests added/updated
  • Component documentation added/updated (if it exists)
  • Accessibility reviewed against ARIA APG (for new/changed interactive components)
  • Visual regression tests updated (for UI changes)
  • Changeset added (pnpm changeset)

N/A for this PR: stories, unit tests, component docs, accessibility, VRT and changeset. It's a build-config change to the private @marigold/docs package, with no published package and no UI change. The docs preview item is left for the reviewer, since it depends on the Vercel builds above.

…ild memory

The docs build on Vercel's 8 GB machine is intermittently OOM-killed during
Turbopack's compile. Its peak swings by about 1 GB between runs of the same
code, and the Vite 8.3.3 bump tipped it over the limit.

About half the compile-peak memory is the pool of Node child processes that
run webpack loaders, mostly the fumadocs-mdx MDX compile. Running them in
worker threads lowered the measured compile peak from 4.0 GB to 3.5 GB on
average, with no change in build time.

Vercel's Node 24.x is >= 24.13.1, where Next falls back to child processes
unless 'forceWorkerThreads' is set. That bypasses a guard against
nodejs/node#65100, whose known trigger is the macOS-only fsevents, so the
setting is scoped to Vercel and local builds keep Next's default.
…builds

VERCEL is also set by `vercel dev` on a developer machine. VERCEL_ENV is
'preview' or 'production' only on Vercel's build machines, so local runs on
macOS, where fsevents can trigger nodejs/node#65100, keep Next's default.
@vercel

vercel Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
marigold-docs Ready Ready Preview Oct 22, 2026 9:13am UTC
marigold-storybook Ready Ready Preview Oct 22, 2026 9:13am UTC
1 Skipped Deployment
Project Deployment Actions Updated
marigold-production Ignored Ignored Oct 22, 2026 9:13am UTC

Request Review

@changeset-bot

changeset-bot Bot commented Oct 22, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 3285a22

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@github-actions github-actions Bot added the type:docs Improvements or additions to documentation label Oct 22, 2026
@marigold-claude

marigold-claude Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Claude finished @sebald's task in 13s —— View job


No issues found against the Marigold review checklist.
· Branch: fix/DST-1904-docs-build-memory

Reviewed 3285a22

@github-actions

Copy link
Copy Markdown
Contributor

Accessibility tests executed. Download the report here.

@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Coverage Report for Marigold Code Coverage

Status Category Percentage Covered / Total
🔵 Lines 99.03% 3384 / 3417
🔵 Statements 98.12% 3556 / 3624
🔵 Functions 98.26% 904 / 920
🔵 Branches 92.07% 2255 / 2449
File CoverageNo changed files found.
Generated in workflow #24689 for commit 33fbb5a by the Vitest Coverage Report Action

@aromko aromko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Review: #5988 — fix(DST-1904): run docs loaders in worker threads on Vercel

Overview

Field Value
Author @sebald
Branch fix/DST-1904-docs-build-memory → main
Files changed 1 (+13 / -0)
Conventions source CLAUDE.md
Visual regression Not needed (build config only, no rendered output)

Linked ticket

DST-1904: 🏗️ Keep the docs build on Vercel from running out of memory, In Review

The ticket asks for three things:

  • Set turbopackPluginRuntimeStrategy: 'forceWorkerThreads'. Done, and limited to Vercel preview and production builds. The ticket didn't ask for that limit, but it makes sense: the trigger in nodejs/node#65100 is fsevents, which is only installed on macOS. The PR description explains the choice well.
  • Set VERCEL_BUILD_SYSTEM_REPORT=1 on the project. The description says it's done (env id lG9sMyZmUQh7oBlx).
  • Redeploy 3–5 times. Still open: the Vercel – marigold-docs check is still deploying.

The ticket counts as done only when repeated preview builds and main builds succeed. That is a rollout condition, not something the code can show.

Verification

I loaded the PR's next.config.mjs through next/dist/server/config (Next 16.4.2) with each VERCEL_ENV:

VERCEL_ENV Resolved strategy
unset childProcesses
development childProcesses
preview workerThreads
production workerThreads

This matches the PR description. Passing undefined lets Next fall back to its own default, so local builds are unaffected. I also read config.js:1086-1103: forceWorkerThreads maps to workerThreads and only logs the warning on Node >=24.13.1, as the code comment says.

Recommendation

Approve once the preview builds are green

It's a small, well-scoped change, and the investigation behind it is thorough: the A/B runs, the options that were ruled out, and a clear rollback signal (abort at worker teardown means revert). The code needs no changes. The one thing left is the condition the PR sets for itself: wait for 3–5 green marigold-docs preview builds in a row before merging. The build failed in 5 of 8 runs, so one green build doesn't prove much. If a build fails by aborting instead of being OOM-killed, revert as the description says.

@aromko
aromko enabled auto-merge (squash) October 22, 2026 09:12
@aromko
aromko merged commit 33fbb5a into main Oct 22, 2026
32 checks passed
@aromko
aromko deleted the fix/DST-1904-docs-build-memory branch October 22, 2026 09:13
@sebald

sebald commented Oct 22, 2026

Copy link
Copy Markdown
Member Author

Post-merge validation on Vercel

All six marigold-docs builds with this change succeeded on the standard 4-core / 8 GB machine. Every log shows turbopackPluginRuntimeStrategy: "forceWorkerThreads" along with Next's warning about bypassing the Node.js guard, and none crashed at worker-thread teardown (nodejs/node#65100).

# Build Commit Build cache Compile Static pages Build system report
1 PR preview (marigold-docs-n2931on6s) 3285a22 restored (dpl_4x9ZCkP3…) 52s 24.5s No memory or disk space problems detected
2 Redeploy (marigold-docs-5bm71kah0) 3285a22 none (cold) 68s 26.2s No memory or disk space problems detected
3 Redeploy (marigold-docs-79mbzi0p4) 3285a22 none (cold) 63s 24.8s No memory or disk space problems detected
4 Redeploy (marigold-docs-jftlj46jf) 3285a22 none (cold) 65s 26.5s No memory or disk space problems detected
5 Redeploy (marigold-docs-lwatyoecj) 3285a22 none (cold) 66s 26.6s No memory or disk space problems detected
6 main after merge (dpl_DPukQyx2…) 33fbb5a restored (dpl_DWb7szA6…) 78s 25.5s No memory or disk space problems detected

Build 6 reproduces the original failure scenario. The failed main build dpl_BDeAi5f1… (62da88a) restored its cache from the same deployment, dpl_DWb7szA6… (the last green main under the old lockfile), and was OOM-killed during the compile. With this change, the same situation compiled and deployed. main is green again for the first time since #5980.

Before the change: 5 of 8 docs builds containing Vite 8.3.3 were OOM-killed. After: 6 of 6 green.

Notes for future readers:

  • VERCEL_BUILD_SYSTEM_REPORT=1 gives no peak memory number. It only reports whether a memory or disk problem was detected, so it confirms "no OOM" but doesn't tell us how much headroom is left. Only the local measurements in the PR description quantify the effect (about 4.0 → 3.5 GB compile peak, macOS).
  • vercel redeploy skips the build cache ("Skipping build cache, deployment was triggered without cache"). Runs 2–5 are therefore cold builds. The runs that restored a cache (1 and 6) are closer to normal pushes.
  • Compile times of 52–78s are within the pre-change range on Vercel (26–58s for green builds, while failures were killed about 50s in). Build 6 is the slowest because it restored a cache from a different lockfile, which matches the side finding about a stale cache making the compile about 2.5× slower.
  • The MaxListenersExceededWarning lines in the logs come from the worker-thread pool sharing one socket. They also appear locally with workerThreads and are harmless log noise.
  • marigold-production (www.marigold-ui.io) uses the same config and machine size and gets the same setting through VERCEL_ENV=production. Its first build with this change will happen after the next release updates the docs branch.

This branch was successfully deployed

2 active deployments
Preview – marigold-docs — 3285a22b Deployed Oct 22, 2026 by vercel[bot]
Preview – marigold-storybook — 3285a22b Deployed Oct 22, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type:docs Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants