Skip to content

fix #92688: [Bug]: Qwen vision models fail with 400 "Unexpected item type in content" on DashScope#92782

Closed
zhangguiping-xydt wants to merge 11 commits into
openclaw:mainfrom
zhangguiping-xydt:feat/issue-92688
Closed

fix #92688: [Bug]: Qwen vision models fail with 400 "Unexpected item type in content" on DashScope#92782
zhangguiping-xydt wants to merge 11 commits into
openclaw:mainfrom
zhangguiping-xydt:feat/issue-92688

Conversation

@zhangguiping-xydt

@zhangguiping-xydt zhangguiping-xydt commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

Summary

What problem does this PR solve?

OpenClaw's Qwen image-understanding routing had two coupled failure points for DashScope:

  1. The resolver trusted configured/catalog input: ["text", "image"] metadata as sufficient proof that a Qwen model could handle image content. That allowed known text-only Qwen chat models such as qwen3.7-max to be selected for image requests when config metadata was overly broad.
  2. Qwen/DashScope modelstudio-native image requests did not use the same prompt placement as OpenRouter-style multimodal providers, so the image prompt could be sent as a separate system/developer prompt instead of user content before the image block.

Together, that can produce DashScope's 400 Unexpected item type in content for an image-understanding request instead of routing to the bundled Qwen vision default.

Why does this matter now?

A valid Qwen setup can lose the user's image request before an image-capable model is reached. The reported path configured Qwen/DashScope with a chat model such as qwen3.7-max; image understanding should skip that text-only family and select qwen-vl-max-latest.

What is the intended outcome?

Fix classification: root-cause fix. Qwen provider-owned runtime metadata now marks the known text-only family as non-image for media-understanding routing, and the resolver entry points consume that metadata before accepting configured image metadata. DashScope-native image prompts are placed in the user content before the image block.

What is intentionally out of scope?

Scope boundary: this PR does not change Qwen auth, API-key lookup, network transport, public plugin SDK shape, public manifest schema, migrations, or provider endpoint selection. The Qwen-specific capability fact stays internal to the Qwen provider/runtime metadata.

What does success look like?

When config marks qwen3.7-max as [text,image], OpenClaw still resolves image understanding to qwen/qwen-vl-max-latest, and Qwen/DashScope image prompts are serialized as user text before the image payload.

What should reviewers focus on?

Please check the source-of-truth boundary and resolver consistency: Qwen model capability facts are provider-owned runtime metadata, not a new public manifest/plugin SDK contract, and the same filtering rule is applied across default media model resolution, configured provider discovery, image-tool routing, PDF-tool routing, manifest-backed defaults, and bundled Qwen aliases.

Linked context

Closes #92688.

This refresh addresses the review concern that the previous shape risked making a Qwen-specific model fact part of a broader public contract. The current shape keeps the compatibility-sensitive decision in internal provider/runtime metadata while preserving the existing public manifest metadata surface for capabilities and defaults.

Real behavior proof (required for external PRs)

  • Behavior or issue addressed: image routing must select an image-capable Qwen model when the active/configured Qwen chat model is qwen3.7-max and its configured metadata incorrectly includes image.

  • Real environment tested: OpenClaw's resolver and image-tool/media-understanding paths with Qwen/DashScope-style config, provider registry construction, manifest-backed defaults, and bundled Qwen image defaults.

  • Exact steps or command run after this patch: configured the active model as qwen/qwen3.7-max, configured the Qwen provider model entry as input: ["text", "image"], then exercised the OpenClaw resolver/image-tool paths.

  • Evidence after fix:

configured active model: qwen/qwen3.7-max
configured provider metadata: qwen3.7-max declares input [text,image]
resolved image model primary: qwen/qwen-vl-max-latest
PASS: qwen3.7-max was not selected for image understanding; Qwen vision default was selected.
tool result text: qwen image ok
image tool describeImageWithModel call: {"provider":"qwen","model":"qwen-vl-max-latest","prompt":"Describe the image.","mime":"image/png"}
PASS: image tool invoked Qwen media understanding with qwen-vl-max-latest instead of qwen3.7-max.
resolved manifest-backed default: qwen-vl-max-latest
PASS: manifest-backed Qwen default resolver skipped qwen3.7-max and selected qwen-vl-max-latest.
  • Observed result after fix: the active/configured chat model can remain qwen/qwen3.7-max, the configured provider metadata can still declare qwen3.7-max as [text,image], and the image request path selects qwen/qwen-vl-max-latest.
  • What was not tested: this proof validates the OpenClaw routing and request-shaping decisions for [Bug]: Qwen vision models fail with 400 "Unexpected item type in content" on DashScope #92688. It does not change or re-test external DashScope account/network behavior, and the submitted change keeps that transport/auth path unchanged.

Tests and validation

Which commands did you run?

env DASHSCOPE_API_KEY=qwen-test pnpm exec tsx -e '<resolveImageModelConfigForTool probe for qwen/qwen3.7-max with input [text,image]>'
configured active model: qwen/qwen3.7-max
configured provider metadata: qwen3.7-max declares input [text,image]
resolved image model primary: qwen/qwen-vl-max-latest
PASS: qwen3.7-max was not selected for image understanding; Qwen vision default was selected.
env DASHSCOPE_API_KEY=qwen-test pnpm exec tsx -e '<image tool probe for Qwen media understanding routing>'
tool result text: qwen image ok
image tool describeImageWithModel call: {"provider":"qwen","model":"qwen-vl-max-latest","prompt":"Describe the image.","mime":"image/png"}
PASS: image tool invoked Qwen media understanding with qwen-vl-max-latest instead of qwen3.7-max.
pnpm exec tsx -e '<manifest-backed resolveDefaultMediaModel probe for qwen3.7-max with input [text,image]>'
resolved manifest-backed default: qwen-vl-max-latest
PASS: manifest-backed Qwen default resolver skipped qwen3.7-max and selected qwen-vl-max-latest.
pnpm exec vitest run \
  src/media-understanding/defaults.test.ts \
  src/media-understanding/provider-registry.test.ts \
  extensions/qwen/index.test.ts
Test Files  3 passed (3)
     Tests  27 passed (27)
pnpm exec vitest run src/agents/tools/image-tool.test.ts
Test Files  2 passed (2)
     Tests  194 passed (194)
pnpm exec vitest run \
  src/agents/tools/image-tool.test.ts \
  src/agents/tools/pdf-tool.model-config.test.ts \
  src/media-understanding/config-provider-models.test.ts \
  src/media-understanding/image.test.ts
Test Files  5 passed (5)
     Tests  252 passed (252)
env OPENCLAW_VITEST_MAX_WORKERS=2 NODE_OPTIONS=--max-old-space-size=8192 OPENCLAW_VITEST_SHARD_NAME=agentic-agents-tools \
  pnpm exec vitest run --config test/vitest/vitest.agents-tools.config.ts src/agents/tools/image-tool.test.ts
Test Files  1 passed (1)
     Tests  98 passed (98)
  Duration  52.74s
pnpm lint --threads=8
pnpm tsgo:prod
pnpm check:test-types
lint completed with exit code 0
pnpm tsgo:prod completed with exit code 0
pnpm check:test-types completed with exit code 0

Earlier validation for the same submitted change also included the extension package-boundary compile/canary checks, additional boundary shards, plugin contracts, and CI artifact build after removing generated catalog artifacts from the commit.

What regression coverage was added or updated?

Coverage now locks the Qwen image-routing behavior at the resolver boundaries that can see configured model metadata: active image model resolution, configured provider vision resolution, default media model resolution with runtime metadata, manifest-backed default resolution, config-derived provider models, PDF tool model config, Qwen/DashScope prompt placement, and bundled Qwen provider registration.

What failed before this fix, if known?

The RED coverage reproduced qwen/qwen3.7-max being selected from configured [text,image] metadata. A later manifest-backed proof reproduced the static/default path selecting qwen3.7-max; after the follow-up, that same path selects qwen-vl-max-latest.

If no test was added, why not?

Regression tests were added or updated across the image tool, media-understanding defaults/metadata, config provider discovery, PDF tool model config, prompt placement, provider registry, and Qwen provider paths.

Risk checklist

Did user-visible behavior change? Yes

Did config, environment, or migration behavior change? No

Did security, auth, secrets, network, or tool execution behavior change? No

What is the highest-risk area?

Risk labels considered: compatibility and auth-provider. The highest-risk area is model selection for Qwen image understanding, because provider-owned capability facts now take precedence over overly broad configured input metadata for known text-only Qwen families.

Risk explanation: the compatibility-sensitive behavior is narrow. Qwen models matching the internal text-only family facts are filtered from image routing, while the existing Qwen image default remains the selected fallback target. The public manifest and plugin SDK contract shapes are not expanded.

How is that risk mitigated?

The same internal capability facts are consumed by the resolver paths covered in this patch, and regression tests exercise configured-model, provider-registry, manifest-backed default, image-tool, PDF-tool, prompt-placement, and bundled Qwen provider flows.

Current review state

What is the next action?

Maintainer-ready confidence: high after issue-specific resolver/image-tool proof, manifest-backed fallback proof, focused Qwen/media-understanding regression coverage, lint, and production/test type checks on the submitted head.

What is still waiting on author, maintainer, CI, or external proof?

The author-side refresh is complete for this scoped routing fix. The next step is maintainer review and CI on the refreshed head.

Which bot or reviewer comments were addressed?

  • RF-001 addressed: the proof section includes terminal output from OpenClaw resolver/image-tool paths showing qwen3.7-max configured as [text,image] and qwen/qwen-vl-max-latest selected.
  • RF-002 addressed: the after-fix proof demonstrates an image-tool/media-understanding invocation using qwen-vl-max-latest.
  • RF-003 addressed: Qwen capability facts are no longer owned by shared core as Qwen constants, and the extension package-boundary compile/canary checks passed.
  • RF-004 addressed: model-selection precedence is covered across configured provider, default, manifest-backed, image-tool, and PDF-tool entry points.
  • RF-005 addressed: fallback/default selection proof shows qwen-vl-max-latest replacing qwen3.7-max for image understanding.
  • RF-006 addressed: the public manifest/plugin SDK contract shape was not expanded for Qwen-specific capability overrides.
  • RF-007 addressed: Qwen-specific capability facts live in the Qwen provider runtime metadata.
  • RF-008 addressed: the Qwen extension no longer imports core src/** internals for capability facts.

Patch quality notes: the diff touches multiple resolver entry points because the same capability source-of-truth is consumed by defaults, configured model discovery, image runtime, image tool config, PDF tool config, provider registry inputs, and manifest-backed defaults. Covering those consumers together keeps the Qwen image-routing behavior consistent for the reported configuration path.

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: M proof: supplied External PR includes structured after-fix real behavior proof. labels Jun 13, 2026
@clawsweeper

clawsweeper Bot commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper review: did not complete due to Codex infrastructure failure. Reviewed June 16, 2026, 5:02 AM ET / 09:02 UTC.

Summary
Review failed before ClawSweeper could summarize the requested change.

PR surface: Source +507, Tests +762. Total +1269 across 25 files.

Reproducibility: unclear. The review failed before ClawSweeper could establish a reproduction path.

Review metrics: none identified.

Stored data model
Persistent data-model change detected: vector/embedding metadata: extensions/qwen/index.test.ts, vector/embedding metadata: src/agents/tools/pdf-tool.model-config.test.ts, vector/embedding metadata: src/media-understanding/image.ts. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Not assessed.
Failure reason: invalid structured output.

This is a ClawSweeper/Codex infrastructure failure, not a PR readiness or patch-quality verdict.
Keep any merge decision on the normal maintainer review path until ClawSweeper can complete a fresh review.

Risk before merge

  • [P1] No close action taken because the review did not complete.

Maintainer options:

  1. Decide the mitigation before merge
    Retry the Codex review after fixing the execution failure.
  2. Pause or close
    Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

  • Review did not complete, so no work-lane recommendation was made.
Review details

Best possible solution:

Retry the Codex review after fixing the execution failure.

Do we have a high-confidence way to reproduce the issue?

Unclear. The review failed before ClawSweeper could establish a reproduction path.

Is this the best way to solve the issue?

Unclear. Retry the review first so ClawSweeper can evaluate the actual issue and fix direction.

AGENTS.md: unclear because the file could not be read completely.

Codex review notes: model internal, reasoning high; reviewed against 6d22b8eb2420.

Label changes

Label changes:

  • remove P2: Current review triage priority is none.
  • remove rating: 🧂 unranked krab: Current review failed before PR readiness was assessed, so no rating label should remain.
  • remove merge-risk: 🚨 compatibility: Current PR review selected no merge-risk labels.
  • remove merge-risk: 🚨 auth-provider: Current PR review selected no merge-risk labels.
  • remove status: 📣 needs proof: Current PR status no longer selects a status label.
Evidence reviewed

PR surface:

Source +507, Tests +762. Total +1269 across 25 files.

View PR surface stats
Area Files Added Removed Net
Source 15 610 103 +507
Tests 10 825 63 +762
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 25 1435 166 +1269

What I checked:

  • failure reason: invalid structured output.
  • codex failure detail: Codex review failed for this PR with exit 0 and wrote invalid JSON or schema-invalid output to /home/runner/work/clawsweeper/clawsweeper/artifacts/event/codex/92782.json: decision.rootCauseCluster.members must not repeat the current item.
  • codex stderr: No stderr captured.
  • codex stdout: trusted during image/PDF routing and can alter existing custom configs."},{"label":"merge-risk: 🚨 auth-provider","reason":"The PR changes Qwen provider alias metadata and model routing for qwen, qwencloud, modelstudio, and dashscope."}],"itemCategory":"bug","reproductionStatus":"source_reproducible","reproductionConfidence":"high","requiresNewFeature":false,"requiresNewConfigOption":false,"requiresProductDecision":false,"reproductionAssessment":"Yes from source: current main trusts configured image metadata for Qwen paths, so qwen3.7-max marked as text+image can be selected; the PR proof shows after-fix resolver and image-tool selection of qwen-vl-max-latest, though live DashScope was not independently reproduced.","solutionAssessment":"Yes: compared with the closed serializer PR and the narrower prompt-placement PR, this PR keeps compatible-mode image serialization stable, keeps Qwen facts p.

Likely related people:

  • unknown: Codex failed before it could trace repository history. (role: review did not complete; confidence: low)
How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. labels Jun 13, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 13, 2026
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 13, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 13, 2026
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 13, 2026
@clawsweeper clawsweeper Bot added status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. proof: sufficient ClawSweeper judged the real behavior proof convincing. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed proof: sufficient ClawSweeper judged the real behavior proof convincing. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jun 14, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 14, 2026
@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Jun 14, 2026
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Jun 14, 2026
@openclaw-barnacle openclaw-barnacle Bot added size: XL and removed docs Improvements or additions to documentation size: L labels Jun 15, 2026
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. labels Jun 15, 2026
@zhangguiping-xydt
zhangguiping-xydt marked this pull request as draft June 15, 2026 07:27
@zhangguiping-xydt
zhangguiping-xydt marked this pull request as ready for review June 16, 2026 08:12
@zhangguiping-xydt

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@clawsweeper clawsweeper Bot removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jun 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. P2 Normal backlog priority with limited blast radius. proof: supplied External PR includes structured after-fix real behavior proof. size: XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Qwen vision models fail with 400 "Unexpected item type in content" on DashScope

1 participant