Repository navigation
fix(provider-utils): chunk array when converting to base64 to avoid excessive memory usage - #13581
danielpoonwj wants to merge 3 commits into
Conversation
…xcessive memory usage
…g string concatenation
tigent
Bot
added
ai/provider
|
will the upstream fix for #13103 fix your issue? meaning if that is patched, will you still need this fix? |
hi @aayush-kapoor, this is actually independent. the patch now allows us to send in data pre-converted to base64 to avoid this issue, but if there are any other implementations that send in Buffers for the SDK to encode in base64, this fix is still relevant |
|
This looks potentially interesting. Do you have any reproduction of the memory issues and proof that this reduces them? Why did you choose 4096 specifically? |
| } | ||
|
|
||
| return btoa(latin1string); | ||
| return btoa(chunks.join('')); |
There was a problem hiding this comment.
what is the max arg limit for String.fromCodePoint? ie why do we need the loop at all
There was a problem hiding this comment.
String.fromCodePoint(...array) spreads each byte as a separate argument, which blows the call stack well below attachment sizes. A few checks we did for our ~18MB PDF test file:
4096 ok
65536 ok
131072 RangeError: Maximum call stack size exceeded
whole file (18.7M) RangeError
|
We hit this exact issue in production with a Node worker OOM'd encoding ~18 MB PDF attachments before handing them to the SDK. We solved it by using the native Repro (Node v24.16, arm64, real 17.9 MB PDF), peak RSS via
|
|
Filed #20149 with an independent production confirmation of this PR's numbers: one 14MB PDF file part OOM-killed a 2GB Fargate container in ~40s through the current per-byte encoder (+606MB / 392ms for the encode alone; 1,065MB peak through the full openai provider request path with two PDFs). It also carries the answers already posted here — the spread's ~131k-argument RangeError ceiling and the chunk-size rationale — so hopefully it helps this get a second look. Linking per CONTRIBUTING so the issue and fix credit connect. |
Per-byte String.fromCodePoint concatenation creates a cons-string rope that V8 cannot collect until btoa flattens it, causing OOM on large arrays. Using chunked encoding with String.fromCodePoint.apply reduces peak memory usage significantly. Fixes vercel#20149, vercel#13581
Per-byte String.fromCodePoint concatenation creates a cons-string rope that V8 cannot collect until btoa flattens it, causing OOM on large arrays. Using chunked encoding with String.fromCodePoint.apply reduces peak memory usage significantly. Fixes vercel#20149, vercel#13581
Background
With a recent issue disabling passing files as base64 encoded data (#13103), we switched to passing a Buffer and let the library handle base64 encoding.
However we noticed very high memory usage with this method.
Summary
Problem
The previous implementation built the latin1 string character by character in a loop, creating a new string allocation on every iteration. For large arrays this causes O(n²) memory pressure due to repeated string concatenation.
Solution
Replace with a chunked approach (4096 bytes per chunk, matching
js-base64) that converts each chunk in a singleString.fromCodePointcall and collects results into an array, joining once at the end before passing to btoa. This avoids both per-byte allocations and the O(n²) string concatenation cost.Manual Verification
Tests
Added
uint8-utils.test.tscovering basic encoding/decoding. The suite passes against both the old and new implementations.Checklist
pnpm changesetin the project root)Future Work
Related Issues