fix(openai): bound embedding batch file downloads by sunlit-deng · Pull Request #98554 · openclaw/openclaw

sunlit-deng · 2026-07-01T09:55:32Z

What Problem This Solves

Fixes an issue where OpenAI-compatible embedding batch runs could buffer an unbounded batch output or error file response when downloading /files/{id}/content, without adding a whole-file cap that breaks valid large batch outputs.

Why This Change Was Made

The successful batch output path now streams JSONL one record at a time and caps each output line, so valid large output files can still complete while malformed oversized lines are cancelled early. Batch status and error file reads continue to use bounded response readers.

User Impact

Users running large OpenAI-compatible embedding batches keep working, and a faulty or hostile provider endpoint cannot stream an unbounded single output record into gateway memory.

Evidence

Real loopback HTTP proof: a local node:http server drove the production runOpenAiEmbeddingBatches path through /files/{id}/content. One run streamed a valid 18,876,149-byte JSONL output and completed; the negative-control run streamed a no-newline oversized record and was cancelled after 8 of 1024 MiB chunks.

$ node --import tsx --input-type=module
valid-large-output: passed
  outputLinesSent=18
  outputBytesSent=18876149
  resultCount=18
  firstEmbedding=[1]
  lastEmbedding=[18]
hostile-oversized-line: passed
  error=openai.batch-file-content: JSONL line exceeds 4194304 bytes
  chunksSent=8
  totalChunks=1024

$ node scripts/run-vitest.mjs extensions/openai/embedding-batch.test.ts -- --reporter=verbose
 ✓ |extension-provider-openai| extensions/openai/embedding-batch.test.ts > OpenAI embedding batch output > wraps malformed JSONL output 3ms
 ✓ |extension-provider-openai| extensions/openai/embedding-batch.test.ts > OpenAI embedding batch output > splits provider uploads by serialized JSONL byte cap 9ms
 ✓ |extension-provider-openai| extensions/openai/embedding-batch.test.ts > OpenAI embedding batch output > adapts OpenAI-compatible upload groups after payload-size rejection 2ms
 ✓ |extension-provider-openai| extensions/openai/embedding-batch.test.ts > OpenAI embedding batch output > bounds batch status success body via readProviderJsonResponse 1016ms
 ✓ |extension-provider-openai| extensions/openai/embedding-batch.test.ts > OpenAI embedding batch output > streams valid batch output files larger than the provider text cap 42ms
 ✓ |extension-provider-openai| extensions/openai/embedding-batch.test.ts > OpenAI embedding batch output > bounds batch output file content without buffering the whole response 29ms
 ✓ |extension-provider-openai| extensions/openai/embedding-batch.test.ts > OpenAI embedding batch output > bounds batch resource error bodies without using response.text() 1007ms

 Test Files  1 passed (1)
      Tests  7 passed (7)

[test] passed 1 Vitest shard in 5.58s

$ node scripts/run-oxlint.mjs extensions/openai/embedding-batch.ts extensions/openai/embedding-batch.test.ts
[plugin-sdk boundary dts] fresh; skipping
[plugin-sdk package boundary dts] fresh; skipping
[qa-channel boundary dts] fresh; skipping
[discord boundary dts] fresh; skipping
[slack boundary dts] fresh; skipping
[whatsapp boundary dts] fresh; skipping
[plugin-sdk boundary root shims] fresh; skipping

$ node_modules/.bin/oxfmt --check extensions/openai/embedding-batch.ts extensions/openai/embedding-batch.test.ts
Checking formatting...

All matched files use the correct format.
Finished in 15ms on 2 files using 10 threads.
$ git diff --check

AI-assisted: built with Codex

clawsweeper · 2026-07-01T11:00:25Z

Codex review: needs maintainer review before merge. Reviewed July 1, 2026, 9:54 AM ET / 13:54 UTC.

Summary
The PR changes OpenAI embedding batch file-content reads to stream JSONL output with line and record limits, bounds file/error response reads, and adds regression coverage for large valid output, oversized lines, and early cancellation after expected custom IDs are accounted for.

PR surface: Source +123, Tests +256. Total +379 across 2 files.

Reproducibility: yes. at source/proof level: current main reads OpenAI batch output file content with res.text(), and the PR body includes loopback terminal proof that exercises the production runOpenAiEmbeddingBatches path for a valid large output and an oversized no-newline response. I did not run tests locally because this review was required to keep the checkout read-only.

Review metrics: 1 noteworthy metric.

Output Download Budgets: 2 added: 4 MiB line cap and group-length record stop. These are the concrete bounds that close both oversized-single-record and many-record response growth without imposing a whole-file cap on valid large batch outputs.

Stored data model
Persistent data-model change detected: unknown-data-model-change: extensions/openai/embedding-batch.test.ts, unknown-data-model-change: extensions/openai/embedding-batch.ts, vector/embedding metadata: extensions/openai/embedding-batch.test.ts, vector/embedding metadata: extensions/openai/embedding-batch.ts. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

none.

Risk before merge

[P1] The PR intentionally fails closed when an OpenAI-compatible endpoint emits more output records than the requested group before all requested custom IDs are satisfied; that is the availability-hardening tradeoff, but maintainers should accept that stricter contract before merge.

Maintainer options:

Accept Contract-Bound Output (recommended)
Accept the stricter fail-closed behavior for OpenAI-compatible batch output because it bounds hostile or malformed file-content streams while preserving valid large outputs.
Loosen The Compatibility Budget
If maintainers need to tolerate extra provider records, require an explicit small extra-record budget and tests instead of returning to unbounded output reads.

Next step before merge

No automated repair is needed; the remaining action is maintainer review of the intentional contract-bound fail-closed behavior and normal merge gates.

Security
Cleared: The diff adds bounded handling for untrusted provider response bodies and does not change dependencies, CI, secrets, permissions, packaging, or other supply-chain surfaces.

Review details

Best possible solution:

Keep this PR open until a linked canonical PR proves it covers this PR's unique work, or a maintainer confirms closure.

Do we have a high-confidence way to reproduce the issue?

Yes at source/proof level: current main reads OpenAI batch output file content with res.text(), and the PR body includes loopback terminal proof that exercises the production runOpenAiEmbeddingBatches path for a valid large output and an oversized no-newline response. I did not run tests locally because this review was required to keep the checkout read-only.

Is this the best way to solve the issue?

Yes. Streaming JSONL with a per-line byte cap plus a group-length/custom-id stop is the narrow OpenAI-plugin fix; a shared whole-response cap would break valid large outputs, and broadening this PR to Google or Voyage would mix sibling provider work into an otherwise focused repair.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 634a02e9e6a3.

Label changes

Label changes:

add merge-risk: 🚨 compatibility: The diff changes malformed OpenAI-compatible batch output from potentially ignored-after-reading behavior to fail-closed once the requested group record budget is exceeded.
add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR body includes copied terminal output from a local loopback HTTP proof that drives the production batch path after the fix, plus targeted test/lint/format output.
remove rating: 🧂 unranked krab: Current PR rating is rating: 🐚 platinum hermit, so this older rating label is no longer current.
remove merge-risk: 🚨 availability: Current PR review merge-risk labels are merge-risk: 🚨 compatibility.
remove status: ⏳ waiting on author: Current PR status label is status: 👀 ready for maintainer look.

Label justifications:

P2: This is a normal-priority OpenAI provider availability hardening PR with limited plugin scope and no evidence of a current widespread outage.
merge-risk: 🚨 compatibility: The diff changes malformed OpenAI-compatible batch output from potentially ignored-after-reading behavior to fail-closed once the requested group record budget is exceeded.
rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR body includes copied terminal output from a local loopback HTTP proof that drives the production batch path after the fix, plus targeted test/lint/format output.
proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes copied terminal output from a local loopback HTTP proof that drives the production batch path after the fix, plus targeted test/lint/format output.

Evidence reviewed

PR surface:

Source +123, Tests +256. Total +379 across 2 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	1	139	16	+123
Tests	1	256	0	+256
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	2	395	16	+379

What I checked:

PR close coverage proof: PR close coverage proof kept this PR open against fix(openai): bound embedding-batch and realtime session JSON respons #97533: PR B is related and merged, but the durable PR A report explicitly says current main still has the material unbounded OpenAI batch output-file res.text() behavior after PR B; PR B covers adjacent JSON/status reads, not PR A's core output-file download bounds.
linked superseding PR: fix(openai): bound embedding-batch and realtime session JSON respons #97533 (fix(openai): bound embedding-batch and realtime session JSON respons) is merged at 2026-06-28T18:53:51Z.
cluster evidence: the durable review links that PR in the work cluster or recommended risk path.
no human follow-up: live comments and timeline hydrated by apply contain no non-automation activity after the ClawSweeper review.

Likely related people:

hugenshen: Authored merged OpenAI embedding-batch response-read hardening in PR fix(openai): bound embedding-batch and realtime session JSON respons #97533, which changed the same OpenAI batch module and tests. (role: recent adjacent contributor; confidence: high; commits: 4c477ee6321e, db255d677fcc; files: extensions/openai/embedding-batch.ts, extensions/openai/embedding-batch.test.ts)
ZengWen-DT: PR fix(message-tool): apply messages.responsePrefix to outbound sends #93639 has merge commit 2af2eb2d..., and current-main blame for the unbounded OpenAI file-content read points at that commit despite the PR title covering broader work. (role: recent current-main contributor; confidence: medium; commits: 2af2eb2dfb17; files: extensions/openai/embedding-batch.ts, extensions/openai/embedding-batch.test.ts, extensions/openai/memory-embedding-adapter.ts)
steipete: Git history shows earlier memory embedding batch output helper work and the refactor that moved memory embeddings into provider plugins, which created the plugin-owned OpenAI batch surface. (role: feature/refactor origin; confidence: medium; commits: 77e6e4cf87f7, 7bd073340a5c, 52ee1f697ecc; files: extensions/openai/embedding-batch.ts, packages/memory-host-sdk/src/host/batch-output.ts)
vincentkoc: Recent release/current-main history touched the OpenAI embedding-batch files and shared provider response helper area that this PR relies on. (role: recent area contributor; confidence: low; commits: e085fa1a3ffd; files: extensions/openai/embedding-batch.ts, extensions/openai/embedding-batch.test.ts, src/agents/provider-http-errors.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

sunlit-deng · 2026-07-01T12:09:47Z

@clawsweeper re-review

clawsweeper · 2026-07-01T12:09:50Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

sunlit-deng · 2026-07-01T13:30:45Z

@clawsweeper re-review

clawsweeper · 2026-07-01T13:30:52Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

vincentkoc

Reviewed the refreshed head after updating onto current main. The streaming JSONL approach preserves valid large embedding batch outputs while bounding malformed single-record growth; accepting the per-record cap / fail-closed behavior for OpenAI-compatible batch output.

* fix(openai): bound embedding batch file downloads * fix(openai): bound batch output records --------- Co-authored-by: Vincent Koc <[email protected]>

openclaw-barnacle Bot added extensions: openai size: S labels Jul 1, 2026

fix(openai): bound embedding batch file downloads

eac7a7b

sunlit-deng force-pushed the sunlit/fix/openai-embedding-batch-bound-content branch from f990ade to eac7a7b Compare July 1, 2026 12:09

openclaw-barnacle Bot added size: M and removed size: S labels Jul 1, 2026

fix(openai): bound batch output records

ea854c5

Merge branch 'main' into sunlit/fix/openai-embedding-batch-bound-content

d4254b6

vincentkoc self-assigned this Jul 1, 2026

vincentkoc approved these changes Jul 1, 2026

View reviewed changes

vincentkoc merged commit ce4a259 into openclaw:main Jul 1, 2026
88 checks passed

ly85206559 mentioned this pull request Jul 1, 2026

fix(google): bound embedding batch response reads #98692

Closed

github-actions Bot mentioned this pull request Jul 1, 2026

📡 Upstream Digest — 2026-07-01 16:19 UTC curtismercier/openclaw-mods#1182

Open

This was referenced Jul 1, 2026

fix(memory): stop chunkMarkdown injecting newlines into single-line chunks #96747

Closed

build(deps): bump golang.org/x/net from 0.53.0 to 0.55.0 in /scripts/docs-i18n in the go_modules group across 1 directory #98744

Closed

clawsweeper Bot mentioned this pull request Jul 5, 2026

fix(google): bound Gemini batch embedding file content reads to prevent OOM #97694

Closed

clawsweeper Bot mentioned this pull request Jul 9, 2026

fix(google): bound Gemini batch output file download to prevent OOM #102922

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(openai): bound embedding batch file downloads#98554

fix(openai): bound embedding batch file downloads#98554
vincentkoc merged 3 commits into
openclaw:mainfrom
sunlit-deng:sunlit/fix/openai-embedding-batch-bound-content

sunlit-deng commented Jul 1, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jul 1, 2026 •

edited

Loading

Uh oh!

sunlit-deng commented Jul 1, 2026

Uh oh!

clawsweeper Bot commented Jul 1, 2026 •

edited

Loading

Uh oh!

sunlit-deng commented Jul 1, 2026

Uh oh!

clawsweeper Bot commented Jul 1, 2026 •

edited

Loading

Uh oh!

vincentkoc left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

sunlit-deng commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Problem This Solves

Why This Change Was Made

User Impact

Evidence

Uh oh!

clawsweeper Bot commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sunlit-deng commented Jul 1, 2026

Uh oh!

clawsweeper Bot commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sunlit-deng commented Jul 1, 2026

Uh oh!

clawsweeper Bot commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vincentkoc left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sunlit-deng commented Jul 1, 2026 •

edited

Loading

clawsweeper Bot commented Jul 1, 2026 •

edited

Loading

clawsweeper Bot commented Jul 1, 2026 •

edited

Loading

clawsweeper Bot commented Jul 1, 2026 •

edited

Loading