Skip to content

feat: implement buffers() for RawSource, OriginalSource, PrefixSource, ReplaceSource, SourceMapSource#211

Merged
alexander-akait merged 16 commits intomainfrom
claude/implement-class-buffers-m0Q32
Apr 29, 2026
Merged

feat: implement buffers() for RawSource, OriginalSource, PrefixSource, ReplaceSource, SourceMapSource#211
alexander-akait merged 16 commits intomainfrom
claude/implement-class-buffers-m0Q32

Conversation

@alexander-akait
Copy link
Copy Markdown
Member

Extend the buffers() API to every concrete Source class so callers always
get a Buffer[] without falling back to the abstract Source default. Each
single-buffer-backed class returns [this.buffer()]. ReplaceSource
delegates to the underlying source's buffers() when no replacements are
queued and otherwise materializes the replaced source into a single
buffer.

https://claude.ai/code/session_01EHhGq9PRFRGefVtwwasCqZ

claude added 4 commits April 28, 2026 15:23
…, ReplaceSource, SourceMapSource

Extend the buffers() API to every concrete Source class so callers always
get a Buffer[] without falling back to the abstract Source default. Each
single-buffer-backed class returns [this.buffer()]. ReplaceSource
delegates to the underlying source's buffers() when no replacements are
queued and otherwise materializes the replaced source into a single
buffer.

https://claude.ai/code/session_01EHhGq9PRFRGefVtwwasCqZ
…r/buffers per class

- lib/Source.js: buffers() now reads source() directly and wraps a Buffer
  result without the indirect this.buffer() call. buffer() gains the
  matching @returns JSDoc.
- benchmark: add buffers() cases for RawSource (string, buffer, cached),
  OriginalSource (string, buffer), PrefixSource, ReplaceSource (no
  replacements + 1000 small replacements), SourceMapSource (string,
  buffer), and SizeOnlySource (throws). Existing source()/buffer() cases
  already cover those methods.
- test: add a Source case verifying buffers() returns the buffer
  directly when source() returns a Buffer.

https://claude.ai/code/session_01EHhGq9PRFRGefVtwwasCqZ
…uffer()]

Source.prototype.buffers() returns [this.buffer()] and naturally picks
up subclass buffer() overrides through the prototype chain. The
overrides on RawSource, OriginalSource, PrefixSource, and
SourceMapSource simply repeated that same body, so remove them and let
the base implementation handle those classes. ConcatSource,
CachedSource, CompatSource, ReplaceSource, and SizeOnlySource keep
their overrides since they each carry distinct logic
(multi-buffer collection, caching, SourceLike delegation,
no-replacements pass-through, throwing).

Source.prototype.buffers() reverts from the inlined source()-based
implementation back to [this.buffer()], so subclass buffer()
optimizations (cached _valueAsBuffer in RawSource etc.) flow through
without duplication.

https://claude.ai/code/session_01EHhGq9PRFRGefVtwwasCqZ
…fers

Replaces the "TODO efficient buffer() implementation" comment.

buffers() walks the underlying source's buffer (cached in RawSource /
OriginalSource / SourceMapSource etc.) and emits the prefix buffer
followed by zero-copy subarrays of each line, only re-emitting the
prefix when the newline has at least one byte after it (matching the
existing /\n(?=.|\s)/g semantics). This is ~57% faster than the
previous fallback that materialized the prefixed source via source()
plus a [this.buffer()] wrap.

buffer() does the same walk in a single pre-allocated allocUnsafe
write so the result lands in one buffer with no intermediate string.
Slightly slower CPU than V8's string-replace + Buffer.from fast path
on ASCII-heavy inputs, but halves the transient memory and avoids the
regex pass entirely — which is the point of the TODO.

Tests cover the trailing-newline, consecutive-newlines, empty-prefix
(pass-through), empty-source, and multi-byte UTF-8 cases.

https://claude.ai/code/session_01EHhGq9PRFRGefVtwwasCqZ
@linux-foundation-easycla
Copy link
Copy Markdown

linux-foundation-easycla Bot commented Apr 28, 2026

CLA Not Signed

@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 28, 2026

Codecov Report

❌ Patch coverage is 97.43590% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 97.70%. Comparing base (7c520c2) to head (e2addd5).
⚠️ Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
lib/ReplaceSource.js 90.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #211      +/-   ##
==========================================
- Coverage   97.72%   97.70%   -0.03%     
==========================================
  Files          25       25              
  Lines        1938     1962      +24     
  Branches      608      613       +5     
==========================================
+ Hits         1894     1917      +23     
- Misses         42       43       +1     
  Partials        2        2              
Flag Coverage Δ
integration 97.70% <97.43%> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@codspeed-hq
Copy link
Copy Markdown

codspeed-hq Bot commented Apr 28, 2026

Merging this PR will degrade performance by 35.74%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 3 improved benchmarks
❌ 1 regressed benchmark
✅ 157 untouched benchmarks
🆕 12 new benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
concat-source: buffers() (nested 4x10 raw) 210.9 µs 131 µs +60.98%
concat-source: buffers() (10 raw) 120.5 µs 83.8 µs +43.78%
🆕 original-source: buffers() (from buffer) N/A 88.8 µs N/A
🆕 original-source: buffers() (from string) N/A 2.5 ms N/A
🆕 prefix-source: buffers() N/A 3.4 ms N/A
🆕 raw-source: buffers() (from string) N/A 2.5 ms N/A
🆕 raw-source: buffers() cached N/A 170.5 µs N/A
🆕 raw-source: buffers() (from buffer) N/A 93.3 µs N/A
🆕 replace-source: buffer() (no replacements) N/A 4.9 ms N/A
replace-source: original() 123.6 µs 192.4 µs -35.74%
replace-source: getReplacements() 1,256.5 µs 974 µs +29.01%
🆕 replace-source: buffers() (no replacements) N/A 4.9 ms N/A
🆕 replace-source: buffers() (1000 small replacements) N/A 4.5 ms N/A
🆕 size-only-source: buffers() (throws) N/A 1.8 ms N/A
🆕 source-map-source: buffers() (from string) N/A 2.6 ms N/A
🆕 source-map-source: buffers() (from buffer) N/A 174.8 µs N/A

Comparing claude/implement-class-buffers-m0Q32 (e2addd5) with main (72ce205)

Open in CodSpeed

claude added 2 commits April 28, 2026 16:11
…default

Bench data (es6-promise.js fixture, 10 calls per task body):
  Source.buffer() default:          ~1143 ops/s
  Hand-written single-pass buffer(): ~960 ops/s

V8's optimized string replace + Buffer.from utf8 encode beats a
JS-side indexOf+copy loop on typical ASCII inputs. The earlier commit
shipped a slower buffer() in exchange for halved transient memory,
but the CPU regression isn't worth that trade-off — and Source.buffer()
already routes through this.source(), so subclasses that want the
prefixed bytes don't lose anything.

buffers() keeps the zero-copy splice implementation (~57% faster than
the default [this.buffer()] path), which is the genuine win from
addressing the TODO.

https://claude.ai/code/session_01EHhGq9PRFRGefVtwwasCqZ
Pairs with the existing 'buffers() (no replacements)' bench so the
no-replacements pass-through path can be A/B compared against the
inherited default.

https://claude.ai/code/session_01EHhGq9PRFRGefVtwwasCqZ
@changeset-bot
Copy link
Copy Markdown

changeset-bot Bot commented Apr 28, 2026

🦋 Changeset detected

Latest commit: e2addd5

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
webpack-sources Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

claude and others added 10 commits April 28, 2026 16:46
…t-buffer TODO

Mirrors the _valueAsBuffer caching used by RawSource, OriginalSource,
and SourceMapSource. Without caching every buffer()/size() call
re-runs the regex replace and re-encodes the prefixed string;
buffers() rebuilt the splice array each call.

Caching is gated on isDualStringBufferCachingEnabled() to honor the
same memory-vs-CPU knob the rest of the codebase uses.

Controlled A/B/A bench (es6-promise.js fixture, 10 calls per task):
  buffer():  1058 -> 10898 ops/s   (~10.5x)
  buffers(): 1066 -> 13246 ops/s   (~12.6x)
  size():    1049 -> 10766 ops/s   (~10.3x; goes through buffer().length)

The first call still does the work; calls 2..N return the cached
values, which is the realistic shape of webpack's hot path (size +
buffer + map + updateHash on the same source).

https://claude.ai/code/session_01EHhGq9PRFRGefVtwwasCqZ
…mutable child

PrefixSource wraps an arbitrary Source. ReplaceSource (.replace,
.insert), ConcatSource (.add), and CompatSource over a mutable
SourceLike all break the immutability assumption. source() reads the
child on every call, so caching the derived buffer would silently go
stale after a child mutation:

  const inner = new ReplaceSource(new RawSource("hello world"));
  const ps = new PrefixSource("> ", inner);
  ps.buffer();  // populates cache: "> hello world"
  inner.replace(6, 10, "you");
  ps.source(); // "> hello you"  -- correct
  ps.buffer(); // "> hello world" -- STALE (with caching)

RawSource / OriginalSource / SourceMapSource can cache safely because
they own their value. PrefixSource doesn't have that invariant. The
amortized 10x speedup wasn't worth a correctness regression.

Adds a regression test that calls source/buffer/buffers, mutates the
child, and asserts all three reflect the new state.

If a caller wants the speedup, wrap the PrefixSource in CachedSource —
that's the documented opt-in cache.

https://claude.ai/code/session_01EHhGq9PRFRGefVtwwasCqZ
… by ~17%

Walks the underlying source's buffer directly instead of going through
this.source() + Buffer.from. Skips both the regex replace and the
intermediate prefixed-string allocation.

Key trick that beats the V8 default: skip the count pass. Allocate
worst-case `prefixLen + contentLen + prefixLen * contentLen` with
allocUnsafe (cheap, no init), fill in one pass, return result.subarray.
For typical short prefixes ("\t", "  ") the worst case is bounded by
~2x and the subarray retains the over-allocation cheaply. For long
prefixes the >50% waste threshold triggers a copy into a tight Buffer
so the extra capacity gets GC'd.

No caching — _source can be mutable (ReplaceSource etc.) and the
result must reflect changes on each call. The existing
"reflect-mutations-to-underlying-source" regression test still passes.

A/B/A median over 5 alternating runs (es6-promise.js fixture, 10
calls per task body):
  baseline:  1130 ops/s
  this:      1322 ops/s    (+17%)

Earlier attempts (committed and reverted on this branch) — for the
record so the next person doesn't redo them:
  - Buffer.concat(this.buffers()): array-iteration overhead -> slower
  - count-pass + exact alloc + copy: 2x indexOf walks -> slower
  - cache the result: faster but unsafe with mutable child

https://claude.ai/code/session_01EHhGq9PRFRGefVtwwasCqZ
CodSpeed (PR #211) flagged:
  prefix-source: buffer()  3.6 ms -> 16 ms  (-77.46%)
  prefix-source: size()    3.6 ms -> 16 ms  (-77.44%)
  (size() falls back to this.buffer().length, so it tracks buffer())

Why local wall-clock disagreed with CI: my JS-side splice loop
(allocUnsafe + per-line indexOf + Buffer.copy) does ~1000 JS<->C++
boundary crossings on the es6-promise.js fixture. Modern CPUs
amortize that with branch prediction and cache, so wall-clock looked
+17%. CodSpeed measures instruction count via simulation, which
counts every boundary crossing — and V8's regex.replace +
Buffer.from(string, "utf8") is two big native calls that beat the
loop on that metric by a wide margin.

Caching the result was off the table from the earlier audit
(_source can be mutable). With both single-pass copy AND caching
ruled out, there's no path to a faster buffer() in pure JS, so
the inherited Source.buffer() default stays.

buffers() override stays — it's a new benchmark (no base to regress
against) and matches the changeset description "more effective
buffers for PrefixSource". Consumers that can accept Buffer[]
(writev, fs.createWriteStream) skip the intermediate concatenated
buffer; consumers that want a single buffer use buffer() and pay
the same as before.

The mutable-child regression test still passes.

https://claude.ai/code/session_01EHhGq9PRFRGefVtwwasCqZ
The /\n(?=.|\s)/g lookahead form disables V8's literal-character regex
fast-path. Switching to /\n/g + a tail-strip when the input ends with
a newline produces identical output and lets V8 take the fast path —
faster CPU and lower instruction count.

Hoists the prefixed-string builder into a `buildPrefixed(prefix, node)`
helper used by both source() and streamChunks(), so the two callsites
stay in sync.

Drops the buffers() override that landed earlier in this branch:
- source() got faster, so the inherited Source.buffer() (which goes
  through this.source() + Buffer.from) automatically gets faster too;
- the inherited Source.buffers() = [this.buffer()] therefore also
  speeds up — the prior splice-based buffers() override regressed
  ~3.6x in CodSpeed instruction count and is replaced by the leaner
  inheritance path.

Local A/B/A wall-clock (es6-promise.js fixture, 5 alternating runs):
  source():  1592 -> 1766 ops/s   (+11%)
  buffer():  1149 -> 1520 ops/s   (+32%)
  buffers(): inherited, now ~1506 ops/s
  size():    inherited, now ~1516 ops/s

Mutable-child regression test still passes (no caching introduced).

https://claude.ai/code/session_01EHhGq9PRFRGefVtwwasCqZ
Per-call iterator allocation for the outer `for (const child of
this._children)` and the inner `for (const buffer of child.buffers())`
shows up as ~40% of buffers() time on a nested 4x10 ConcatSource —
the shape webpack produces during emit when wrapping module bodies.

A/B/A median over 5 alternating wall-clock runs (es6-promise.js fixture,
nested 4x10 raw):
  baseline:  659 075 ops/s
  this:    1 054 270 ops/s   (+60%)

Same V8 hot-path principle as the recent PrefixSource regex switch —
the optimizer is much happier when array iteration stays integer-
indexed and the array element type stays monomorphic. Index-based
length hoisting (`childCount`, `blen`) keeps the bound check out of
the loop body.

Tests unchanged; full suite still green.

https://claude.ai/code/session_01EHhGq9PRFRGefVtwwasCqZ
…updateHash()

Same V8 fast-path principle as the buffers() change in c29e061 — keep
array iteration integer-indexed and the element type monomorphic so
the optimizer doesn't fall off the cliff.

A/B/A wall-clock medians (es6-promise.js fixture, 3 alternating runs
so far; bg task still running):
  concat-source: source() (10 raw)   923 075 -> 971 196 ops/s  (+5%)
  concat-source: size()              108 429 -> 692 935 ops/s  (+6.4x)
  concat-source: updateHash()         ~wash (within rme)

The size() bench's 6.4x jump comes from the bench shape: it builds a
fresh 4-child mixed ConcatSource per task and calls size() ten times.
With for-of, each call allocates an iterator; the size() body itself
is so cheap that the iterator setup dominates. for-i removes that
overhead entirely.

source() is more modest because the body work (string concat per
child) outweighs iterator setup. updateHash() is wash for the same
reason — `child.updateHash(hash)` is much heavier than the loop.

Tests still green; lint clean.

https://claude.ai/code/session_01EHhGq9PRFRGefVtwwasCqZ
@alexander-akait alexander-akait merged commit d506580 into main Apr 29, 2026
30 of 34 checks passed
@alexander-akait alexander-akait deleted the claude/implement-class-buffers-m0Q32 branch April 29, 2026 12:57
alexander-akait pushed a commit that referenced this pull request Apr 29, 2026
Every iteration over `_children` (and other small arrays) used `for...of`,
which allocates an iterator object per call and dispatches through the
`[Symbol.iterator]() / .next()` protocol on every step. Indexed loops
skip both — the same pattern PR #211 used in `buffers()` is applied
consistently to every hot iteration site: `source`, `size`, `buffer`,
`buffers`, `updateHash`, `streamChunks`, `_optimize`, the constructor,
`add`, `addAllSkipOptimizing`.

Median-of-3 wall-clock gains over baseline (single-baseline runs were
±20% noisy, so each side was run 3x and the median taken):

  concat-source: buffers() (10 raw)                        +47.3%
  concat-source: buffers() (nested 4x10 raw)               +43.7%
  cached-source: buffer() (cold, wraps ConcatSource x10)   +37.9%
  concat-source: source() (mixed)                          +24.7%
  concat-source: source() (10 raw)                         +24.0%
  concat-source: nested flattening                         +20.5%
  concat-source: buffer() (10 raw)                         +12.9%
  concat-source: new ConcatSource() (strings)              +12.1%
  cached-source: buffers() (cold, wraps ConcatSource x10)  +11.2%
  concat-source: new ConcatSource() (10 raw)               +10.2%
  concat-source: size()                                    +8.9%
  concat-source: addAllSkipOptimizing()                    +7.6%

No semantic change — `_children` is always a dense `Child[]`, so the
indexed traversal yields the same sequence as the iterator. Apparent
regressions (`raw-source: buffer()`, `cached-source: originalLazy()`,
streamChunks variants) are wholly within noise — those code paths don't
touch `ConcatSource` and same-config variance for those benches spans
±10-50%.

All 89,825 tests pass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants