fix(media-understanding): parse nested Gemini output JSON by lin-hongkuan · Pull Request #96432 · openclaw/openclaw

lin-hongkuan · 2026-06-24T13:02:18Z

What Problem This Solves

extractGeminiResponse parsed the last JSON object in noisy CLI output by taking lastIndexOf("{"). That fails for ordinary nested JSON payloads because the last { may belong to an inner object such as usage rather than the top-level response object.

In that case the extractor returns null even though the provider output contains a valid response field.

Why This Change Was Made

The extractor now scans candidate { positions from right to left until it finds a full JSON object that parses. This keeps the noisy-output behavior while allowing nested objects in the final provider payload.

User Impact

Gemini-style media-understanding CLI output with nested metadata can still yield the response text instead of dropping the provider result.

Evidence

Live behavior proof from this branch:

{"extracted":"a useful description"}

Focused validation:

RUN  v4.1.8 C:/Users/lin20/Documents/New project 3/openclaw

Test Files  1 passed (1)
     Tests  1 passed (1)
  Start at  21:01:14
  Duration  936ms (transform 647ms, setup 490ms, import 10ms, tests 132ms, environment 0ms)

Checking formatting...

All matched files use the correct format.
Finished in 1200ms on 2 files using 32 threads.

git diff --check produced no output.

AI-assisted: OpenAI Codex helped inspect the media-understanding output extractor and add focused regression coverage.

clawsweeper · 2026-06-24T13:04:13Z

Codex review: needs maintainer review before merge. Reviewed June 25, 2026, 3:43 AM ET / 07:43 UTC.

Summary
The PR replaces media-understanding Gemini stdout suffix parsing with balanced JSON candidate scanning and adds regression tests for nested, quoted, pretty-printed, and malformed output.

PR surface: Source +103, Tests +100. Total +203 across 2 files.

Reproducibility: yes. at source level: current main slices from the last {, and the focused Node check shows that nested usage metadata makes the extractor return null despite a valid top-level response.

Review metrics: none identified.

Stored data model
Persistent data-model change detected: serialized state: packages/media-understanding-common/src/output-extract.test.ts. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🐚 platinum hermit
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

none.

Next step before merge

[P2] No repair lane is needed because there is no actionable review finding; the remaining action is ordinary maintainer review and landing judgment.

Security
Cleared: The diff only changes local stdout parsing and focused tests; it does not touch dependencies, workflows, secrets, permissions, network calls, or package resolution.

Review details

Best possible solution:

Land the shared extractor fix after maintainer review, keeping Gemini JSON recovery in the media-understanding common helper rather than duplicating it in the runner.

Do we have a high-confidence way to reproduce the issue?

Yes, at source level: current main slices from the last {, and the focused Node check shows that nested usage metadata makes the extractor return null despite a valid top-level response.

Is this the best way to solve the issue?

Yes. The media-understanding common extractor is the narrow owner boundary for this runtime path; moving the fix into the runner or reusing the broader agent CLI parser would be a less local ownership fit.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 31e941c3fc32.

Label changes

Label justifications:

P2: This fixes a real but scoped Gemini media-understanding CLI parsing bug with limited blast radius.
rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body includes copied live output from the branch showing the nested JSON response is extracted, plus focused test and formatting output.
proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes copied live output from the branch showing the nested JSON response is extracted, plus focused test and formatting output.

Evidence reviewed

PR surface:

Source +103, Tests +100. Total +203 across 2 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	1	111	8	+103
Tests	1	100	0	+100
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	2	211	8	+203

What I checked:

Current main parser bug: Current main starts parsing at the last {, so nested payloads select an inner object instead of the top-level Gemini response object. (packages/media-understanding-common/src/output-extract.ts:6, 31e941c3fc32)
Source-level reproduction: A read-only Node check of the current-main algorithm returned null for noisy output followed by JSON containing nested usage metadata. (packages/media-understanding-common/src/output-extract.ts:6, 31e941c3fc32)
Runtime entry point: resolveCliOutput delegates Gemini command stdout to extractGeminiResponse, limiting the runtime effect to Gemini media-understanding CLI output normalization. (src/media-understanding/runner.entries.ts:240, 31e941c3fc32)
Documented contract: The CLI backend docs say Gemini JSON output reads reply text from the response field, so dropping a valid top-level response with nested metadata violates documented behavior. Public docs: docs/gateway/cli-backends.md. (docs/gateway/cli-backends.md:293, 31e941c3fc32)
PR implementation: The PR head scans completed JSON object ranges and parses candidates from newest to oldest instead of slicing from the final opening brace. (packages/media-understanding-common/src/output-extract.ts:6, cfc2963d1524)
Regression coverage: The added test covers noisy output followed by nested Gemini JSON plus quoted-noise, pretty-printed, array, and malformed trailing cases. (packages/media-understanding-common/src/output-extract.test.ts:6, cfc2963d1524)

Likely related people:

steipete: Peter Steinberger introduced the media runner helper and later extracted Gemini response parsing into a shared media-understanding helper before the package move. (role: introduced behavior and refactor owner; confidence: high; commits: 2a1f8b2615b1, 3e2f0ca07745; files: src/media-understanding/runner.entries.ts, src/media-understanding/output-extract.ts, packages/media-understanding-common/src/output-extract.ts)
Shakker: Current-main blame for the package helper and media runner points to a broad reintroduction commit that added these files in their current locations. (role: recent area contributor; confidence: medium; commits: 1adc07614888; files: packages/media-understanding-common/src/output-extract.ts, src/media-understanding/runner.entries.ts)
vincentkoc: Vincent Koc authored merged Gemini CLI JSON response and stats parser work for the agent CLI path, which shares the same upstream Gemini output contract. (role: adjacent Gemini JSON parser contributor; confidence: medium; commits: 757573f37417; files: src/agents/cli-output.ts, src/agents/cli-output.test.ts, docs/gateway/cli-backends.md)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

vincentkoc · 2026-06-25T07:50:45Z

Merged via squash.

Prepared head SHA: cfc2963d1524a7585f48bfbfe202c9dd35dd0870
Landed commit: 793b604b2377cd52f465291bfce6ab1cb3c7963a

…6432)

openclaw-barnacle Bot added the size: XS label Jun 24, 2026

vincentkoc force-pushed the codex/media-understanding-nested-json-output branch from 0404471 to cb83c3e Compare June 25, 2026 07:25

openclaw-barnacle Bot added size: M and removed size: XS labels Jun 25, 2026

fix(media-understanding): parse nested Gemini output JSON

cfc2963

vincentkoc force-pushed the codex/media-understanding-nested-json-output branch from cb83c3e to cfc2963 Compare June 25, 2026 07:38

vincentkoc merged commit 793b604 into openclaw:main Jun 25, 2026
77 checks passed

github-actions Bot mentioned this pull request Jun 25, 2026

📡 Upstream Digest — 2026-06-25 10:35 UTC curtismercier/openclaw-mods#1150

Open

github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jun 26, 2026

fix(media-understanding): parse nested Gemini output JSON (openclaw#9…

ebf7f1e

…6432)

QiuYuang pushed a commit to QiuYuang/openclaw that referenced this pull request Jul 1, 2026

fix(media-understanding): parse nested Gemini output JSON (openclaw#9…

288cf75

…6432)

lsr911 mentioned this pull request Jul 8, 2026

fix: don't mark exact-size web_fetch responses as truncated (#101837) #102099

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(media-understanding): parse nested Gemini output JSON#96432

fix(media-understanding): parse nested Gemini output JSON#96432
vincentkoc merged 1 commit into
openclaw:mainfrom
lin-hongkuan:codex/media-understanding-nested-json-output

lin-hongkuan commented Jun 24, 2026

Uh oh!

clawsweeper Bot commented Jun 24, 2026 •

edited

Loading

Uh oh!

Uh oh!

vincentkoc commented Jun 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

lin-hongkuan commented Jun 24, 2026

What Problem This Solves

Why This Change Was Made

User Impact

Evidence

Uh oh!

clawsweeper Bot commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

vincentkoc commented Jun 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

clawsweeper Bot commented Jun 24, 2026 •

edited

Loading