Skip to content

fix(memory): reject malformed embedding batch JSONL bytes#110502

Merged
steipete merged 2 commits into
openclaw:mainfrom
VectorPeak:fix/embedding-batch-jsonl-utf8
Jul 18, 2026
Merged

fix(memory): reject malformed embedding batch JSONL bytes#110502
steipete merged 2 commits into
openclaw:mainfrom
VectorPeak:fix/embedding-batch-jsonl-utf8

Conversation

@VectorPeak

@VectorPeak VectorPeak commented Jul 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor
Additional instructions

MUST: Keep Allow edits from maintainers enabled for this PR so maintainers
can help update the branch when needed.

AI-assisted: yes. This PR was prepared with Codex assistance.

What Problem This Solves

Fixes an issue where users running memory embedding batch workflows could have malformed provider JSONL output accepted after invalid UTF-8 bytes were rewritten as replacement characters.

The affected path is the shared embedding batch output reader used by the OpenAI, Gemini, and Voyage batch readers. When a provider output line was otherwise valid JSON, a bad byte inside a string could become a U+FFFD replacement character before JSON.parse, which could corrupt fields such as custom_id or provider error text instead of reporting a malformed batch record.

Why This Change Was Made

The batch output reader now decodes each JSONL record with fatal UTF-8 handling and maps decode failures to the existing malformed-record error shape. This keeps the fix at the shared JSONL boundary and leaves the existing record byte cap, physical-record counting, stream cancellation, and provider-specific output-line handling unchanged.

This is not a vector/embedding metadata schema change and does not alter persisted embedding rows, metadata shape, or migration behavior. It only rejects malformed provider output bytes before those bytes can become parsed JSON records.

User Impact

Malformed embedding batch output now fails clearly as a malformed JSONL record instead of silently carrying corrupted replacement characters into memory embedding result processing. Valid UTF-8 JSONL output continues to parse as before across OpenAI, Gemini, and Voyage batch readers.

Evidence

Runtime proof from the embedding-batch provider path:

Command:
node scripts/run-vitest.mjs run --config test/vitest/vitest.extension-provider-openai.config.ts extensions/openai/pr110502-real-proof.test.ts --reporter=verbose

Setup:
Temporary, uncommitted proof test on PR head ec24fc526094.
A local OpenAI-compatible loopback HTTP server handled the same batch flow used by runOpenAiEmbeddingBatches(...): upload file, create completed batch, then download /files/{output_file_id}/content. No live OpenAI API key or production credential was used.

Valid provider output:
[runtime proof] valid server=http://127.0.0.1:62448/v1
[runtime proof] valid path: POST /v1/files -> POST /v1/batches -> GET /v1/files/output-1/content
[runtime proof] valid result: [["req-1",[0.1,0.2]]]

Malformed provider output:
[runtime proof] malformed server=http://127.0.0.1:62452/v1
[runtime proof] malformed path: POST /v1/files -> POST /v1/batches -> GET /v1/files/output-1/content
[runtime proof] malformed bytes: 0xc3 0x28 inside provider JSONL string
[runtime proof] malformed result: openai.batch-file-content: malformed JSONL record

Result:
Test Files  1 passed (1)
Tests       2 passed (2)

This proof exercises runOpenAiEmbeddingBatches(...) and the provider batch-output download path rather than calling readEmbeddingBatchJsonl(...) directly. The OpenAI-compatible batch flow matches the output-file shape used by provider batch APIs: completed batch metadata points to an output file, and the reader downloads /files/{output_file_id}/content before parsing JSONL records.

Focused validation:

pnpm test packages/memory-host-sdk/src/host/batch-output.test.ts extensions/openai/embedding-batch.test.ts extensions/google/embedding-batch.test.ts extensions/voyage/embedding-batch.test.ts

Passed: 4 focused Vitest test files/suites covering the shared JSONL reader plus OpenAI, Google/Gemini, and Voyage embedding batch callers.

Additional checks:

pnpm exec oxfmt --check packages/memory-host-sdk/src/host/batch-output.ts packages/memory-host-sdk/src/host/batch-output.test.ts
git diff --check

Both passed. Full local pnpm build && pnpm check && pnpm test was not run; the focused validation above covers the touched reader and its provider callers, and GitHub CI is running the broader lanes.

The new regression test in packages/memory-host-sdk/src/host/batch-output.test.ts feeds invalid UTF-8 bytes (0xc3, 0x28) inside a JSONL string and verifies the reader rejects it as test.batch-output: malformed JSONL record. It covers both newline-terminated records and an unterminated final record.

User-realistic proof / call chain:

User runs a memory embedding batch workflow
-> provider batch output is downloaded by the OpenAI, Google/Gemini, or Voyage embedding batch reader
-> extensions/openai/embedding-batch.ts, extensions/google/embedding-batch.ts, or extensions/voyage/embedding-batch.ts calls readEmbeddingBatchJsonl(...)
-> packages/memory-host-sdk/src/host/batch-output.ts decodes each provider JSONL record
-> invalid UTF-8 now throws before JSON.parse(...)
-> provider-specific embedding result handling only receives valid decoded JSON objects

The focused test covers the shared reader where provider JSONL bytes first become text, and the provider embedding batch test files were rerun to check the sibling OpenAI, Google/Gemini, and Voyage callers still accept valid JSONL output.

Dependency contract checked: TextDecoder(utf-8, { fatal: true }) throws on malformed data instead of replacing bad bytes with U+FFFD; the PR relies on that built-in decoder behavior to fail closed before JSON.parse.

@VectorPeak @chatgpt-codex-connector
fix(memory): reject malformed embedding batch JSONL bytes
a17e457 
Co-authored-by: chatgpt-codex-connector[bot] <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal backlog priority with limited blast radius. labels Jul 18, 2026
@clawsweeper

clawsweeper Bot commented Jul 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed July 18, 2026, 3:27 AM ET / 07:27 UTC.

Summary
The PR switches shared embedding-batch JSONL record decoding to fatal UTF-8 mode and adds tests for malformed bytes in terminated and final records.

PR surface: Source +6, Tests +24. Total +30 across 2 files.

Reproducibility: yes. in source: permissive TextDecoder() can replace malformed bytes before JSON.parse, and the added regression case directly constructs that byte sequence at the shared reader boundary.

Review metrics: none identified.

Stored data model
Persistent data-model change detected: vector/embedding metadata: packages/memory-host-sdk/src/host/batch-output.test.ts, vector/embedding metadata: packages/memory-host-sdk/src/host/batch-output.ts. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🦪 silver shellfish
Proof: 🦪 silver shellfish
Patch quality: 🐚 platinum hermit
Result: blocked until real behavior proof from a real setup is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • [P1] Add redacted after-fix proof from a real provider embedding-batch run, with private endpoints, identifiers, and credentials removed.
  • After updating the PR body, request a fresh ClawSweeper review if one is not triggered automatically.

Proof guidance:

  • [P1] Needs real behavior proof before merge: The PR provides a useful loopback OpenAI-compatible integration trace, but it explicitly relies on a local mock server and temporary uncommitted test rather than a real provider batch setup; add redacted terminal output or runtime logs from real after-fix behavior. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Risk before merge

  • [P1] A real provider batch-output download has not been shown after the change, so the malformed-byte behavior is supported only by synthetic stream coverage despite the patch being otherwise narrowly scoped.

Maintainer options:

  1. Decide the mitigation before merge
    Keep fatal decoding at the shared JSONL boundary and add redacted after-fix evidence from a real OpenAI, Gemini, or Voyage batch retrieval showing a valid batch completes and the malformed-output handling is exercised through the provider path.
  2. Pause or close
    Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

  • [P1] No mechanical repair is indicated; the contributor must supply real after-fix provider evidence before maintainers can clear the proof gate.

Security
Cleared: The two-file patch changes built-in UTF-8 decoding and tests only; it adds no dependency, permission, secret, network, artifact, or supply-chain surface.

Review details

Best possible solution:

Keep fatal decoding at the shared JSONL boundary and add redacted after-fix evidence from a real OpenAI, Gemini, or Voyage batch retrieval showing a valid batch completes and the malformed-output handling is exercised through the provider path.

Do we have a high-confidence way to reproduce the issue?

Yes in source: permissive TextDecoder() can replace malformed bytes before JSON.parse, and the added regression case directly constructs that byte sequence at the shared reader boundary.

Is this the best way to solve the issue?

Yes for the code path: fatal UTF-8 decoding at the shared record boundary rejects corruption before provider-specific handling and avoids divergent checks in OpenAI, Google/Gemini, and Voyage callers.

AGENTS.md: unclear because the file could not be read completely.

Codex review notes: model internal, reasoning high; reviewed against e9f798ecc424.

Label changes

Label justifications:

  • P2: This is a bounded reliability fix for malformed memory-provider batch output, with no evidence of an urgent broad user regression.
  • rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🐚 platinum hermit.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: The PR provides a useful loopback OpenAI-compatible integration trace, but it explicitly relies on a local mock server and temporary uncommitted test rather than a real provider batch setup; add redacted terminal output or runtime logs from real after-fix behavior. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed

PR surface:

Source +6, Tests +24. Total +30 across 2 files.

View PR surface stats
Area Files Added Removed Net
Source 1 8 2 +6
Tests 1 24 0 +24
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 2 32 2 +30

What I checked:

Likely related people:

  • unresolved: The available PR context identifies the shared reader and its provider callers, but the read-only environment prevented a reliable local blame/log pass; assigning a named owner would be guesswork. (role: shared memory-batch history owner; confidence: low; files: packages/memory-host-sdk/src/host/batch-output.ts, packages/memory-host-sdk/src/host/batch-output.test.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (3 earlier review cycles)
  • reviewed 2026-07-18T07:02:57.905Z sha a17e457 :: needs real behavior proof before merge. :: none
  • reviewed 2026-07-18T07:14:03.340Z sha ec24fc5 :: needs real behavior proof before merge. :: none
  • reviewed 2026-07-18T07:22:06.169Z sha ec24fc5 :: needs real behavior proof before merge. :: none

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. labels Jul 18, 2026
@steipete steipete self-assigned this Jul 18, 2026
@steipete
steipete merged commit badb5ae into openclaw:main Jul 18, 2026
145 of 149 checks passed
@steipete

steipete commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Merged via squash.

@github-actions github-actions Bot mentioned this pull request Jul 18, 2026
Open
This was referenced Jul 18, 2026
Closed
Closed
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 19, 2026
@VectorPeak @chatgpt-codex-connector
fix(memory): reject malformed embedding batch JSONL bytes (openclaw#1…
bfbebff 
…10502)

Co-authored-by: chatgpt-codex-connector[bot] <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@steipete steipete

Labels

P2 Normal backlog priority with limited blast radius. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. size: XS status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@VectorPeak @steipete