Skip to content

fix(mcp): preserve native images and embedded text resources - #6250

Merged
jmartisk merged 7 commits into
langchain4j:mainfrom
CryoThrust:fix/mcp-image-content-type
Nov 12, 2026
Merged

jmartisk merged 7 commits into
langchain4j:mainfrom
CryoThrust:fix/mcp-image-content-type

Conversation

@CryoThrust

@CryoThrust CryoThrust commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

Issue

Related to #3364.

Change

MCP image tool results previously failed as unsupported content. The default converter now returns native ImageContent through ToolExecutionResult.resultContents(), preserving mixed text/image order. Images explicitly use DetailLevel.AUTO, as requested in review; the global image default is unchanged.

Text-only results retain newline-joined resultText() behavior, including empty results and embedded text resources. Malformed images and unsupported binary resources fail explicitly; custom converters remain available. Structured-content precedence, error handling and metadata are preserved. No API or dependency is added.

The tutorial explains native tool contents and provider limitations. Anthropic and OpenAI Responses serialization tests preserve image blocks; Chat Completions rejection is tested and documented. The MCP fixture retains inherited exception coverage using unsupported audio.

Merged current main and removed compatibility assertions/documentation for the legacy extractor API that upstream has since removed.

Verification

JDK 17:

mvn clean test

All 121 reactor modules are verified across the initial run and its continuation: 11,355 cases, zero failures/errors, 112 skips. The first run passed 67 modules before a truncated protobuf download from Central; after recovering that dependency and installing the prerequisite reactor artifacts, mvn -rf :langchain4j-milvus-v2 clean test passes the remaining 54 modules. The original interrupted command is not reported as a successful single run. MCP has 469 unit cases with no skips.

JDK 21 with JBang and current reactor dependencies installed:

mvn -pl langchain4j-mcp \
  '-Dtest=DefaultMcpToolResultConverterTest,McpToolResultMetaParsingTest,McpTools*IT#executeToolThatReturnsNativeImage' \
  -DskipITs package

All 33 selected cases pass with no skips, including native-image calls over eight modern/legacy Stdio, Streamable HTTP and WebSocket paths, with JDK, OkHttp and Apache HTTP clients. The local fixture temporarily used a free port because 8080 was occupied; those port-only overrides were reverted. Package/Javadocs and scoped Spotless checks pass.

Provider tests use mocked I/O. No live-model perception or complete provider/MCP integration-suite result is claimed. Checks from older commits are not results for this refreshed head.

General checklist

  • No breaking public API changes introduced by this feature
  • Added unit and integration coverage for the change
  • Tests cover positive and negative cases
  • All reactor unit tests completed successfully, with the skips reported above
  • All unit and integration tests in the changed modules were run and are green
  • All core/main unit and integration tests were run and are green
  • Updated documentation
  • Added an example in the examples repo (not applicable)
  • Updated Spring Boot starters (not applicable)

DefaultMcpToolResultConverter.extractText() threw RuntimeException for
any content type other than 'text'. MCP servers can return 'image'
(base64-encoded) and 'resource' (embedded text) content items.

- Handle 'image' type by returning a data-URI string
- Handle 'resource' type by extracting the text field
- Update tests to cover new content types

Fixes langchain4j#3364
@CryoThrust

Copy link
Copy Markdown
Contributor Author

The JDK 25 job is the only failing check. Its log shows JdkHttpClientErrorBodyTest.should_not_fail_on_successful_response timing out while connecting to a localhost test server (CompletableFuture.get timeout); the changed MCP classes are not involved. I cannot rerun the workflow without repository admin permissions. Could a maintainer please retry the JDK 25 job when convenient?

@CryoThrust CryoThrust left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One edge case is worth clarifying before merge: MCP embedded resources may contain either resource.text or resource.blob. The new branch returns an empty string when the resource map has a blob (or neither field), which silently loses a valid tool result. Could this either preserve the blob in an explicit representation (for example a data URI when the resource MIME type is available) or fail with a descriptive unsupported-resource error? At minimum, a regression test for a blob resource would make the chosen contract explicit. The image path already has an explicit fallback, so silently returning empty for a valid resource is inconsistent.

@CryoThrust CryoThrust changed the title fix(mcp): support image and resource content types in tool results fix(mcp): handle image and embedded text/blob resources Oct 13, 2026
@CryoThrust

Copy link
Copy Markdown
Contributor Author

Synced this branch with current main at 7e617dad without rewriting history and fixed the embedded-resource gap from the earlier review. resource.blob now produces a data URI, with application/octet-stream when MIME is omitted. Empty resources remain valid; a resource without text/blob fails explicitly. Added converter regressions and JSON-response controls for _meta, application errors and structured-content precedence.

The complete MCP unit-test reactor passed on JDK 17: MCP 290 tests, Core 1,608 and main 1,877, with the existing Core/main skips reported in the PR description. Spotless/package checks and tutorial MDX compilation passed. The broader integration check hit an unchanged ONNX test's native-thread limit; no full integration-suite success is claimed. The full-repository unit run is continuing.

I also clarified the text/data-URI contract in the docs and changed #3364 to a related issue rather than auto-closing it: this patch does not add native multimodal tool-result delivery.

@CryoThrust

Copy link
Copy Markdown
Contributor Author

The integration failure was in this patch's test fixture: the inherited no-cause exception test still called getImage, which is now supported. Fixed in e40363e5f by using unsupported audio for that error case and separately asserting the image data URI and successful result. Shared non-MCP tests retain their existing defaults.

All six real local transport suites now complete on JDK 21: 199 tests, zero failures/errors, six existing skips. The shared error-handler suite passes36 cases, and the converter suite passes14. The package/Spotless checks pass. The full-root unit run is restarting after recovery of the truncated Playwright dependency; it is not yet a successful full-suite result.

@CryoThrust

CryoThrust commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor Author

Full-root verification completed on e40363e5f: the JDK 17 clean test reactor builds all 117 modules successfully. Surefire reports 8,911 tests, zero failures/errors and 99 skips (8,812 passed). This follows recovery and checksum verification of the truncated Playwright dependency. The run reused cached embedding assets and limited processor/tokenizer concurrency.

The repaired head's GitHub JDK 17/21/25 and Jackson 3 unit jobs, compliance, split-package and Spotless checks are also green. Update: the independent JDK 21 integration job has now completed successfully. Its Maven reactor reports BUILD SUCCESS; all six MCP tool transport suites report zero failures/errors (199 cases, six credential-gated skips). All eight checks on this head have succeeded. The skipped provider cases remain outside this verification.

@jmartisk

Copy link
Copy Markdown
Collaborator

Is this still an appropriate approach now that there is ToolExecutionResult.resultContents() that can handle non-text content?

ToolExecutionResult already supports multimodal results via Builder#resultContents(List<Content>) (since 1.13.0), and ToolExecutionResultMessage carries List<Content>. So I don't think we need to encode images as data-URI strings inside resultText(), and doing so has a real cost. Before this PR an image result failed with a short Unsupported content type: "image". Now a 1 MB screenshot becomes ~1.3M characters of text in the next LLM request. That can mean a large bill or a context-length error, and the model can't actually see an image passed as base64 text.

@CryoThrust CryoThrust changed the title fix(mcp): handle image and embedded text/blob resources fix(mcp): preserve native images and embedded text resources Oct 14, 2026
@CryoThrust

Copy link
Copy Markdown
Contributor Author

You're right — the base64-text workaround was the wrong default now that typed tool results are available. Revised in 31269d7b5:

  • Images use ImageContent / resultContents(), with mixed-content order preserved.
  • Text-only responses still return one newline-joined text block, so existing resultText() callers keep working.
  • Embedded text resources remain supported; arbitrary binary blobs fail explicitly instead of being expanded into prompt text. The existing custom converter remains available for those types.
  • Malformed image fields fail rather than returning placeholder text.

The five focused classes pass 127 tests; the converter/JSON controls fail seven cases on the preceding implementation. All six real local MCP transport suites pass (199 cases, zero failures/errors, six credential-gated skips), plus 36 shared error-handler cases. Anthropic and OpenAI Responses serialization checks preserve native image blocks; Chat Completions rejection is tested and documented. No live-provider perception result is claimed.

The full-root JDK 17 clean test run also passes on the revision: all 117 modules, 8,916 tests, zero failures/errors and 99 skips. The PR description now reflects the native-image scope and these results; previous-head CI results are not carried forward.

@CryoThrust

Copy link
Copy Markdown
Contributor Author

Merged current main in d12ca451e and resolved the conflict around the removed legacy extractor API. The native-image/AUTO behavior and modern custom-converter coverage are retained; the stale legacy-extractor tutorial sentence is removed.

The JDK 17 unit reactor is complete across the initial run and its resumed segment: 121 modules, 11,355 cases, zero failures/errors and 112 skips. A truncated Central dependency download caused the interruption; the remaining 54 modules pass after recovery. MCP's 469 unit cases have no skips.

On JDK 21, the converter/meta controls and image calls over all eight modern/legacy transport paths pass: 33 cases, no skips. Local HTTP/WebSocket fixtures temporarily used a free port, with those overrides reverted afterward. Package/Javadocs and scoped formatting checks pass. The description now reflects this verification; refreshed-head CI is pending.

@jmartisk
jmartisk merged commit d93368d into langchain4j:main Nov 12, 2026
8 checks passed
@jmartisk

Copy link
Copy Markdown
Collaborator

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants