Skip to content

oxfmt: random "Failed to read file" / ENOMEM on macOS containers at default --threads #22740

Description

@olegafx

Summary

pnpm oxfmt --check (oxfmt 0.51.0) on a 10k+ file repo inside an OrbStack container on macOS randomly fails with:

× Failed to read file: /app/.../SomeFile.tsx
  help: This may be due to the file being a binary or inaccessible.

A different random 1–4 files fail each run. The failure is reproducible across many consecutive runs.

Workarounds

  • pnpm oxfmt --check --threads=1 — always passes.
  • Running on the host (no container) — always passes.

Environment

  • macOS 26.x on M4 Pro (14 cores → default 14 threads).
  • OrbStack with a host bind-mount of the repo into the container at /app.
  • 10,546 files matched by oxfmt's default walk.
  • Container has no memory limit (/sys/fs/cgroup/memory.max = max) and ulimit -n = 20480, so OS-level resource limits are not the cause.
  • Linux CI on GitLab runners (48 threads, no bind-mount — repo cloned into container's overlayfs) runs the same command without failure.

Comparison point: oxlint does not fail

Same container, same volume, same default thread count (14):

$ for i in 1 2 3 4 5; do pnpm oxlint --quiet; done
# 5/5 clean, ~20s each, 10,533 files

Standalone probe

A minimal Node probe reproduces the same ENOMEM: not enough memory, open failure outside of oxfmt, isolating the failure mode to high-concurrency fs.readFile patterns on the same volume.

Probe results (10,424 files, 5 runs per mode)

Mode UV_THREADPOOL Pass rate Wall time when pass Fail timing
unbounded single-thread (Promise.all(readFile)) 4 (default) 3/5 484–513 ms failures at 21 ms
unbounded single-thread 64 0/5 failures at 43–91 ms
bounded 32 (p-limit, single-thread) 4 5/5 392–416 ms
throttled 530 reads/sec (sequential + sleep) 4 5/5 ~21 s
workers 14 (each Worker Promise.all-reads its chunk, reads only) 4 each 5/5 428–729 ms (14–24k reads/sec)
workers 14 (reads only) 64 each 5/5 266–357 ms (29–39k reads/sec)
master-reads 14 (master Promise.all-reads all 10k, then chunks to Workers for CPU) 4 1/5 945 ms failures at 21–32 ms
workers-format 14 (each Worker reads + calls real oxfmt format() per file) 4 each 3/5 685–746 ms failures at 167–349 ms

Observations

  • Single-process unbounded Promise.all of ~10k readFiles flakes (3/5 pass at default UV_THREADPOOL_SIZE, 0/5 at UV_THREADPOOL_SIZE=64). Failures happen 21–91 ms in — too fast for sustained throughput to matter; the initial submission burst is the trigger.
  • Distributing the same workload across 14 Workers (each Promise.all-reading its chunk independently, without any format work) passes 5/5 even at UV_THREADPOOL_SIZE=64 (39k reads/sec aggregate, well above oxfmt's ~12k reads/sec on the host).
  • Bounded (p-limit(32)) or throttled (rate-capped) submission passes 5/5.
  • Adding format() per file inside Workers reintroduces failures (2/5 fail) — failures happen 167–349 ms in, distinct timing from the master-burst case.

Probe script (probe-bindmount.mjs)

Details
#!/usr/bin/env node
// Probe: does parallel JS fs.readFile flake on macOS bind-mount under high concurrency?
//
// Usage (inside Docker container):
//   node probe-bindmount.mjs unbounded         # bare Promise.all
//   node probe-bindmount.mjs bounded 32        # p-limit(32)
//   node probe-bindmount.mjs throttled 530     # ~oxlint's rate
//   node probe-bindmount.mjs workers 14        # 14 Workers, each Promise.all on its chunk
//   node probe-bindmount.mjs master-reads 14   # master does ALL reads then chunks to N Workers for CPU
//   node probe-bindmount.mjs workers-format 14 # 14 Workers, each reads + calls real oxfmt format() per file
//   UV_THREADPOOL_SIZE=64 node probe-bindmount.mjs unbounded

import { readFile, readdir } from 'node:fs/promises';
import path from 'node:path';
import { performance } from 'node:perf_hooks';
import { Worker, isMainThread, parentPort, workerData } from 'node:worker_threads';
import pLimit from 'p-limit';

const ROOT = '/app/app/javascript'; // adjust to your repo
const EXT_RE = /\.(ts|tsx|js|jsx|mjs|cjs)$/;
const EXCLUDE = /\/(node_modules|__generated__|\.cache|public|vendor)\//;

async function walk(dir) {
  const out = [];
  const entries = await readdir(dir, { withFileTypes: true, recursive: true });
  for (const e of entries) {
    const full = path.join(e.parentPath ?? dir, e.name);
    if (!e.isFile()) continue;
    if (!EXT_RE.test(full)) continue;
    if (EXCLUDE.test(full)) continue;
    out.push(full);
  }
  return out;
}

async function modeUnbounded(files) {
  await Promise.all(files.map((f) => readFile(f)));
}

async function modeBounded(files, n) {
  const limit = pLimit(n);
  await Promise.all(files.map((f) => limit(() => readFile(f))));
}

async function modeThrottled(files, rate) {
  const gapMs = 1000 / rate;
  for (const f of files) {
    await readFile(f);
    if (gapMs > 0) await new Promise((r) => setTimeout(r, gapMs));
  }
}

async function modeWorkers(files, n) {
  return spawnWorkers(files, n, 'read');
}

async function modeMasterReads(files, n) {
  const contents = await Promise.all(files.map((f) => readFile(f, 'utf-8')));
  return spawnWorkers(contents, n, 'cpu-spin');
}

async function modeWorkersFormat(files, n) {
  return spawnWorkers(files, n, 'oxfmt-format');
}

function spawnWorkers(items, n, task) {
  const chunkSize = Math.ceil(items.length / n);
  const chunks = [];
  for (let i = 0; i < items.length; i += chunkSize) chunks.push(items.slice(i, i + chunkSize));
  const workers = chunks.map(
    (chunk) =>
      new Promise((resolve, reject) => {
        const w = new Worker(new URL(import.meta.url), { workerData: { chunk, task } });
        w.on('message', (m) => (m && m.error ? reject(new Error(m.error)) : resolve(m)));
        w.on('error', reject);
        w.on('exit', (code) => {
          if (code !== 0 && code !== null) reject(new Error(`worker exited ${code}`));
        });
      }),
  );
  return Promise.all(workers);
}

if (!isMainThread) {
  (async () => {
    try {
      const { chunk, task } = workerData;
      if (task === 'read') {
        await Promise.all(chunk.map((f) => readFile(f)));
      } else if (task === 'cpu-spin') {
        for (const content of chunk) {
          let h = 0;
          for (let i = 0; i < content.length; i++) h = (h * 31 + content.charCodeAt(i)) | 0;
          if (h === 0xdeadbeef) console.log('unreachable');
        }
      } else if (task === 'oxfmt-format') {
        const { format } = await import('oxfmt');
        const oxfmtConfig = { printWidth: 120, singleQuote: true, sortPackageJson: false };
        await Promise.all(
          chunk.map(async (f) => {
            const code = await readFile(f, 'utf-8');
            await format(f, code, oxfmtConfig);
          }),
        );
      }
      parentPort.postMessage('ok');
    } catch (e) {
      parentPort.postMessage({ error: e.message ?? String(e) });
    }
  })();
} else {
  const [mode, paramRaw] = process.argv.slice(2);
  const param = paramRaw ? Number(paramRaw) : undefined;
  if (!mode) {
    console.error('Usage: node probe-bindmount.mjs <unbounded|bounded N|throttled RATE|workers N|master-reads N|workers-format N>');
    process.exit(2);
  }

  console.log(`Walking ${ROOT}…`);
  const files = await walk(ROOT);
  console.log(`Found ${files.length} files. Running mode=${mode} param=${param ?? '-'}.`);

  const t0 = performance.now();
  try {
    if (mode === 'unbounded') await modeUnbounded(files);
    else if (mode === 'bounded') await modeBounded(files, param ?? 32);
    else if (mode === 'throttled') await modeThrottled(files, param ?? 530);
    else if (mode === 'workers') await modeWorkers(files, param ?? 14);
    else if (mode === 'master-reads') await modeMasterReads(files, param ?? 14);
    else if (mode === 'workers-format') await modeWorkersFormat(files, param ?? 14);
    else {
      console.error(`Unknown mode: ${mode}`);
      process.exit(2);
    }
    const dt = (performance.now() - t0).toFixed(0);
    const rate = ((files.length * 1000) / Number(dt)).toFixed(0);
    console.log(`PASS  ${mode} ${param ?? ''}  ${dt}ms  ${rate} reads/sec  ${files.length} files`);
  } catch (e) {
    const dt = (performance.now() - t0).toFixed(0);
    console.error(`FAIL  ${mode} ${param ?? ''}  ${dt}ms  err=${e.code ?? '?'}  ${e.message}`);
    process.exit(1);
  }
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-formatterArea - FormatterE-Help WantedExperience level - For the experienced collaborators

    Type

    Fields

    Priority

    None yet

    Effort

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions