My benchmarking shows a 10-20% drop in performance due to #13104. Is there a difference between our testing methodologies?

❯ hyperfine 'node ./bench.mjs before 100 10 parallel' 'node ./bench.mjs after 100 10 parallel' 'node ./bench.mjs before 100 10 serial' 'node ./bench.mjs after 100 10 serial' --show-output
Benchmark #1: node ./bench.mjs before 100 10 parallel
  Time (mean ± σ):     25.564 s ±  0.153 s    [User: 34.482 s, System: 0.650 s]
  Range (min … max):   25.387 s … 25.891 s    10 runs
 
Benchmark #2: node ./bench.mjs after 100 10 parallel
  Time (mean ± σ):     29.335 s ±  0.125 s    [User: 41.025 s, System: 0.936 s]
  Range (min … max):   29.153 s … 29.550 s    10 runs
 
Benchmark #3: node ./bench.mjs before 100 10 serial
  Time (mean ± σ):     25.446 s ±  0.179 s    [User: 34.401 s, System: 0.542 s]
  Range (min … max):   25.169 s … 25.828 s    10 runs
 
Benchmark #4: node ./bench.mjs after 100 10 serial
  Time (mean ± σ):     32.375 s ±  0.211 s    [User: 42.611 s, System: 0.776 s]
  Range (min … max):   32.013 s … 32.588 s    10 runs
 
Summary
  'node ./bench.mjs before 100 10 serial' ran
    1.00 ± 0.01 times faster than 'node ./bench.mjs before 100 10 parallel'
    1.15 ± 0.01 times faster than 'node ./bench.mjs after 100 10 parallel'
    1.27 ± 0.01 times faster than 'node ./bench.mjs after 100 10 serial'

benchmark.sh

#!/usr/bin/env bash

doSetup=0

if [ doSetup = 1 ]
then
    git worktree add before
    git worktree add after
    (
        cd before
        git checkout 8dae4abf41e1ccec3515e8acea159505bbd55f33~1
        yarn
        yarn build
    )
    (
        cd after
        git checkout 8dae4abf41e1ccec3515e8acea159505bbd55f33
        yarn
        yarn build
    )
fi

#https://github.com/sharkdp/hyperfine
hyperfine 'node ./bench.mjs before 1000' 'node ./bench.mjs after 1000'

bench.mjs

const [, , version, groupCountString, groupSizeString, method] = process.argv;
const groupCount = +groupCountString;
const groupSize = +groupSizeString;
const prettierApi = await import(`./${version}/index.js`);
import {readFileSync} from 'fs';

const printResults = false;
const sourceText = readFileSync('./before/src/main/comments.js', 'utf8');

for(let i = 0; i < groupCount; i++) {
    if(method === 'serial') {
        for(let i = 0; i < groupSize; i++) {
            const result = await prettierApi.format(sourceText, {
                parser: 'typescript'
            });
            if(printResults) console.log(result);
        }
    }
    if(method === 'parallel') {
        const promises = [];
        for(let i = 0; i < groupSize; i++) {
            promises.push(prettierApi.format(sourceText, {
                parser: 'typescript'
            }));
        }
        await Promise.allSettled(promises);
    }
}

Uh oh!

Performance of async printers (was: Adopt swc parser?) #13158

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Performance of async printers (was: Adopt `swc` parser?) #13158