fix(media-understanding): place Qwen/DashScope image prompts in user content (#92688) by xzh-icenter · Pull Request #92770 · openclaw/openclaw

xzh-icenter · 2026-06-13T18:26:19Z

Summary

DashScope vision models (qwen3.7-max, qwen3.7-plus) return 400 "Unexpected item type in content" when the image tool sends the prompt in a system message and only image blocks in the user message. The fix adds DashScope/Qwen endpoint detection to shouldPlaceImagePromptInUserContent so the prompt text is placed in the user content alongside the image.

Root Cause

src/media-understanding/image.ts builds image context with the prompt in systemPrompt by default. Only GitHub Copilot and OpenRouter routes override this to place the prompt in user content. DashScope's OpenAI-compatible endpoint rejects image-only user messages with a 400 schema error.

Changes

src/media-understanding/image.ts: Added isDashScopeEndpoint() helper detecting qwen, qwen-dashscope, dashscope providers and dashscope.aliyuncs.com base URLs. Added early return in shouldPlaceImagePromptInUserContent() for DashScope endpoints.
src/media-understanding/image.test.ts: Added test verifying DashScope/Qwen image prompts are placed in user content with systemPrompt undefined.

Real behavior proof

Behavior or issue addressed: Qwen vision models fail with 400 "Unexpected item type in content" on DashScope when using the image tool.

Real environment tested: Linux 6.6.114.1-microsoft-standard-WSL2 (x64), Node.js v24.16.0, commit 97281d8 on branch fix/qwen-dashscope-vision-prompt-in-user-content.

Exact steps or command run after this patch:

cd /root/.openclaw/workspace/openclaw-repo
grep -A 15 "function isDashScopeEndpoint" src/media-understanding/image.ts
grep -A 5 "DashScope/Qwen" src/media-understanding/image.ts
grep -c "places DashScope" src/media-understanding/image.test.ts
git diff --stat upstream/main

Evidence after fix:

$ grep -A 15 "function isDashScopeEndpoint" src/media-understanding/image.ts
function isDashScopeEndpoint(model: Model): boolean {
  const provider = model.provider?.toLowerCase() ?? "";
  const baseUrl = model.baseUrl?.toLowerCase() ?? "";
  return (
    provider.includes("dashscope") ||
    provider === "qwen" ||
    provider === "qwen-dashscope" ||
    baseUrl.includes("dashscope.aliyuncs.com")
  );
}

$ grep -A 5 "DashScope/Qwen" src/media-understanding/image.ts
  // DashScope/Qwen vision models return 400 "Unexpected item type in content"
  // when the user message contains only image blocks with no text. The prompt
  // must be placed in the user content alongside the image.
  if (isDashScopeEndpoint(model)) {
    return true;
  }

$ grep -c "places DashScope" src/media-understanding/image.test.ts
1

$ git diff --stat upstream/main
 src/media-understanding/image.test.ts | 53 ++++++++++++++++++++++++++++++++
 src/media-understanding/image.ts      | 17 ++++++++++
 2 files changed, 70 insertions(+)

Observed result after fix: The isDashScopeEndpoint function correctly identifies qwen, qwen-dashscope, and dashscope.aliyuncs.com URLs. shouldPlaceImagePromptInUserContent now returns true for these endpoints, causing the prompt to be placed in user content alongside the image blocks instead of in a system message. The test validates that systemPrompt is undefined and user content contains both text and image parts.

What was not tested: Live DashScope API call with credentials (requires DashScope account). Only source-level logic verification was performed. The fix follows the same pattern as the existing GitHub Copilot and OpenRouter detections which are well-tested.

Proof limitations: No live provider 400 reproduction — the fix is based on source analysis of the payload shape and the reporter's error logs. The DashScope endpoint detection covers common provider/URL patterns but may miss exotic custom configurations.

…content DashScope vision models (qwen3.7-max, qwen3.7-plus) return 400 "Unexpected item type in content" when the image request sends the prompt in a system message and only image blocks in the user message. Add DashScope endpoint detection to shouldPlaceImagePromptInUserContent so the prompt text is placed in the user content alongside the image, matching how GitHub Copilot and OpenRouter routes already work. Fixes openclaw#92688

clawsweeper · 2026-06-13T18:28:27Z

Codex review: needs real behavior proof before merge. Reviewed June 15, 2026, 2:01 PM ET / 18:01 UTC.

Summary
The PR adds DashScope/Qwen endpoint detection to image prompt placement and one regression test that expects prompt text in user content before image blocks.

PR surface: Source +17, Tests +53. Total +70 across 2 files.

Reproducibility: yes. at source level: current main can build a system prompt plus image-only user content for Qwen/DashScope image requests. A live DashScope 400 reproduction still needs provider credentials, but the linked logs match this payload path.

Review metrics: 1 noteworthy metric.

Qwen Endpoint Coverage: 7 provider ids declared, 4 ModelStudio base URLs, 1 provider/baseUrl covered by the new test. The prompt-placement rule needs to match the shipped Qwen/DashScope family, not only the single qwen China endpoint path in this PR.

Merge readiness
Overall: 🦪 silver shellfish
Proof: 🦪 silver shellfish
Patch quality: 🦐 gold shrimp
Result: blocked until stronger real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

[P1] Broaden prompt placement through existing ModelStudio provider-family or endpoint-class metadata and add qwencloud/modelstudio plus dashscope-intl coverage.
[P1] Add redacted live DashScope output, OpenClaw runtime logs, terminal output, or a linked artifact showing the image request succeeds after the patch.

Proof guidance:

[P1] Needs stronger real behavior proof before merge: The PR body shows terminal source greps and assertions, but no live DashScope request, redacted OpenClaw runtime log, or linked artifact proving the after-fix image request succeeds. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Risk before merge

[P1] The detector misses supported Qwen/ModelStudio aliases and the global DashScope endpoint, so affected supported configurations can continue sending image-only user messages and hitting the same provider failure.
[P1] The PR changes provider prompt placement without redacted live DashScope or OpenClaw runtime output showing the reported 400 is resolved after the patch.
[P1] Related PRs exist for the same issue, but fix #92688: [Bug]: Qwen vision models fail with 400 "Unexpected item type in content" on DashScope #92782 is still draft and #92688: fix(qwen): use DashScope native image format for Qwen vision models #92704 has a conflicting serializer-direction finding, so neither is a safe supersession close target yet.

Maintainer options:

Broaden Through Provider Metadata (recommended)
Use the existing ModelStudio family or endpoint-class metadata so qwen, qwencloud, modelstudio, dashscope, and CN/global DashScope endpoints share the prompt-in-user-content rule.
Accept Narrow Coverage
Maintainers could intentionally merge the exact qwen CN endpoint repair and track alias/global endpoint coverage separately, but supported configurations would remain broken.
Pause For Provider Proof
Keep the PR paused until redacted DashScope or OpenClaw runtime output proves the changed prompt placement resolves the provider failure.

Next step before merge

[P1] Contributor or maintainer follow-up is needed for broader provider-family coverage and real DashScope/OpenClaw proof; automation cannot supply the contributor's provider environment proof.

Security
Cleared: No concrete security or supply-chain issue found; the diff only changes media-understanding source and tests.

Review findings

[P2] Cover the full Qwen/DashScope provider family — src/media-understanding/image.ts:288-291

Review details

Best possible solution:

Reuse existing ModelStudio provider-family or endpoint-class metadata to cover the full Qwen/DashScope family, add alias/global-endpoint regression tests, and include redacted live DashScope or OpenClaw runtime proof when credentials are available.

Do we have a high-confidence way to reproduce the issue?

Yes, at source level: current main can build a system prompt plus image-only user content for Qwen/DashScope image requests. A live DashScope 400 reproduction still needs provider credentials, but the linked logs match this payload path.

Is this the best way to solve the issue?

No, not as written: moving the prompt into user content is the right direction, but the detector should be driven by existing ModelStudio provider metadata or explicitly cover all shipped aliases and CN/global endpoints.

Full review comments:

[P2] Cover the full Qwen/DashScope provider family — src/media-understanding/image.ts:288-291
The new predicate only matches qwen, qwen-dashscope, provider names containing dashscope, and the CN dashscope.aliyuncs.com host. OpenClaw ships Qwen aliases such as qwencloud and modelstudio, and documents the Global dashscope-intl.aliyuncs.com endpoint, so those supported configurations would still send an image-only user message and can keep hitting the same 400. Please drive this from the existing ModelStudio provider-family/endpoint-class metadata or add explicit alias/global endpoint coverage.
Confidence: 0.88

Overall correctness: patch is incorrect
Overall confidence: 0.86

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against a0b16f37e835.

Label changes

Label justifications:

P2: This is a bounded provider-specific image-understanding bugfix with limited blast radius, but it blocks affected Qwen/DashScope image calls.
merge-risk: 🚨 compatibility: The diff changes provider payload construction with an incomplete Qwen/DashScope predicate, leaving supported alias and global-endpoint configurations on the old failing behavior.
rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🦐 gold shrimp.
status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: The PR body shows terminal source greps and assertions, but no live DashScope request, redacted OpenClaw runtime log, or linked artifact proving the after-fix image request succeeds. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Evidence reviewed

PR surface:

Source +17, Tests +53. Total +70 across 2 files.

View PR surface stats

Area	Files	Added	Net
Source	1	17	+17
Tests	1	53	+53
Docs	0	0	0
Config	0	0	0
Generated	0	0	0
Other	0	0	0
Total	2	70	+70

What I checked:

AGENTS.md policy read: Root AGENTS.md was read fully, along with relevant scoped plugin/extension/agent tool guides; plugin-boundary, provider-routing, compatibility, and real-proof rules shaped this review. (AGENTS.md:1, a0b16f37e835)
Current main prompt-placement behavior: Current main keeps image prompts in systemPrompt unless the provider is GitHub Copilot or OpenRouter-style, so Qwen/DashScope is not already fixed on main. (src/media-understanding/image.ts:258, a0b16f37e835)
PR detector is too narrow: The PR head matches provider ids containing dashscope, exact qwen/qwen-dashscope, and dashscope.aliyuncs.com, but not qwencloud, modelstudio, or the global dashscope-intl endpoint. (src/media-understanding/image.ts:284, 97281d8b73a3)
Qwen manifest family coverage: The bundled Qwen manifest declares providers qwen, qwencloud, modelstudio, dashscope, qwen-oauth, qwen-portal, and qwen-cli, and declares ModelStudio-native base URLs including dashscope-intl.aliyuncs.com. (extensions/qwen/openclaw.plugin.json:7, a0b16f37e835)
Existing provider metadata can cover this: Current provider attribution already resolves manifest endpoint classes and known provider families, including modelstudio-native/modelstudio, which is a better owner boundary than partial string checks. (src/agents/provider-attribution.ts:421, a0b16f37e835)
User-facing Qwen endpoint docs: The Qwen docs list both China and Global DashScope compatible-mode endpoints and state legacy modelstudio refs remain compatibility aliases. Public docs: docs/providers/qwen.md. (docs/providers/qwen.md:167, a0b16f37e835)

Likely related people:

steipete: Authored the bundled Qwen provider and docs history and appears in blame for the current image-context prompt-placement function. (role: feature-history contributor; confidence: high; commits: e3ac0f43df3e, 889ddb5edfaa, 0bbac63d00bc; files: extensions/qwen/openclaw.plugin.json, docs/providers/qwen.md, src/media-understanding/image.ts)
vincentkoc: Authored central provider request capability and endpoint-class history that already exposes the ModelStudio family metadata relevant to this fix. (role: adjacent provider-capability contributor; confidence: high; commits: c405bcfa9836, 8c802aa68351, 90fac509875e; files: src/agents/provider-attribution.ts, src/agents/openai-completions-compat.ts, docs/providers/qwen.md)
neeravmakwana: Earlier history for the shared media-understanding image runtime includes OpenAI/Codex media-understanding support that shaped the current generic image path. (role: media-understanding feature contributor; confidence: medium; commits: 6fd9d2ff3871; files: src/media-understanding/image.ts, src/media-understanding/image.test.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

xzh-icenter · 2026-06-14T02:47:59Z

Closing: cannot provide live Qwen API proof, and the Deno Deploy dependency registry is currently returning 400 errors blocking CI. Issue remains open.

xzh-icenter · 2026-06-14T02:48:28Z

Reopening — CI was passing, closed by mistake.

xzh-icenter · 2026-06-18T15:28:50Z

Closing in favor of the merged fix in #93649.

openclaw-barnacle Bot added size: S proof: supplied External PR includes structured after-fix real behavior proof. labels Jun 13, 2026

xzh-icenter closed this Jun 14, 2026

xzh-icenter reopened this Jun 14, 2026

This was referenced Jun 14, 2026

fix #92688: [Bug]: Qwen vision models fail with 400 "Unexpected item type in content" on DashScope #92782

Closed

#92688: fix(qwen): use DashScope native image format for Qwen vision models #92704

Closed

vincentkoc mentioned this pull request Jun 16, 2026

fix(qwen): place DashScope image prompts in user content #93649

Merged

xzh-icenter closed this Jun 18, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(media-understanding): place Qwen/DashScope image prompts in user content (#92688)#92770

fix(media-understanding): place Qwen/DashScope image prompts in user content (#92688)#92770
xzh-icenter wants to merge 1 commit into
openclaw:mainfrom
xzh-icenter:fix/qwen-dashscope-vision-prompt-in-user-content

xzh-icenter commented Jun 13, 2026

Uh oh!

clawsweeper Bot commented Jun 13, 2026 •

edited

Loading

Uh oh!

xzh-icenter commented Jun 14, 2026

Uh oh!

xzh-icenter commented Jun 14, 2026

Uh oh!

xzh-icenter commented Jun 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

xzh-icenter commented Jun 13, 2026

Summary

Root Cause

Changes

Real behavior proof

Uh oh!

clawsweeper Bot commented Jun 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xzh-icenter commented Jun 14, 2026

Uh oh!

xzh-icenter commented Jun 14, 2026

Uh oh!

xzh-icenter commented Jun 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

clawsweeper Bot commented Jun 13, 2026 •

edited

Loading