Repository navigation
fix(mcp): preserve native images and embedded text resources - #6250
Conversation
DefaultMcpToolResultConverter.extractText() threw RuntimeException for any content type other than 'text'. MCP servers can return 'image' (base64-encoded) and 'resource' (embedded text) content items. - Handle 'image' type by returning a data-URI string - Handle 'resource' type by extracting the text field - Update tests to cover new content types Fixes langchain4j#3364
|
The JDK 25 job is the only failing check. Its log shows |
CryoThrust
left a comment
There was a problem hiding this comment.
One edge case is worth clarifying before merge: MCP embedded resources may contain either resource.text or resource.blob. The new branch returns an empty string when the resource map has a blob (or neither field), which silently loses a valid tool result. Could this either preserve the blob in an explicit representation (for example a data URI when the resource MIME type is available) or fail with a descriptive unsupported-resource error? At minimum, a regression test for a blob resource would make the chosen contract explicit. The image path already has an explicit fallback, so silently returning empty for a valid resource is inconsistent.
|
Synced this branch with current main at The complete MCP unit-test reactor passed on JDK 17: MCP 290 tests, Core 1,608 and main 1,877, with the existing Core/main skips reported in the PR description. Spotless/package checks and tutorial MDX compilation passed. The broader integration check hit an unchanged ONNX test's native-thread limit; no full integration-suite success is claimed. The full-repository unit run is continuing. I also clarified the text/data-URI contract in the docs and changed #3364 to a related issue rather than auto-closing it: this patch does not add native multimodal tool-result delivery. |
|
The integration failure was in this patch's test fixture: the inherited no-cause exception test still called All six real local transport suites now complete on JDK 21: 199 tests, zero failures/errors, six existing skips. The shared error-handler suite passes36 cases, and the converter suite passes14. The package/Spotless checks pass. The full-root unit run is restarting after recovery of the truncated Playwright dependency; it is not yet a successful full-suite result. |
|
Full-root verification completed on The repaired head's GitHub JDK 17/21/25 and Jackson 3 unit jobs, compliance, split-package and Spotless checks are also green. Update: the independent JDK 21 integration job has now completed successfully. Its Maven reactor reports BUILD SUCCESS; all six MCP tool transport suites report zero failures/errors (199 cases, six credential-gated skips). All eight checks on this head have succeeded. The skipped provider cases remain outside this verification. |
|
Is this still an appropriate approach now that there is
|
|
You're right — the base64-text workaround was the wrong default now that typed tool results are available. Revised in
The five focused classes pass 127 tests; the converter/JSON controls fail seven cases on the preceding implementation. All six real local MCP transport suites pass (199 cases, zero failures/errors, six credential-gated skips), plus 36 shared error-handler cases. Anthropic and OpenAI Responses serialization checks preserve native image blocks; Chat Completions rejection is tested and documented. No live-provider perception result is claimed. The full-root JDK 17 |
|
Merged current The JDK 17 unit reactor is complete across the initial run and its resumed segment: 121 modules, 11,355 cases, zero failures/errors and 112 skips. A truncated Central dependency download caused the interruption; the remaining 54 modules pass after recovery. MCP's 469 unit cases have no skips. On JDK 21, the converter/meta controls and image calls over all eight modern/legacy transport paths pass: 33 cases, no skips. Local HTTP/WebSocket fixtures temporarily used a free port, with those overrides reverted afterward. Package/Javadocs and scoped formatting checks pass. The description now reflects this verification; refreshed-head CI is pending. |
|
Thanks! |
Issue
Related to #3364.
Change
MCP image tool results previously failed as unsupported content. The default converter now returns native
ImageContentthroughToolExecutionResult.resultContents(), preserving mixed text/image order. Images explicitly useDetailLevel.AUTO, as requested in review; the global image default is unchanged.Text-only results retain newline-joined
resultText()behavior, including empty results and embedded text resources. Malformed images and unsupported binary resources fail explicitly; custom converters remain available. Structured-content precedence, error handling and metadata are preserved. No API or dependency is added.The tutorial explains native tool contents and provider limitations. Anthropic and OpenAI Responses serialization tests preserve image blocks; Chat Completions rejection is tested and documented. The MCP fixture retains inherited exception coverage using unsupported audio.
Merged current
mainand removed compatibility assertions/documentation for the legacy extractor API that upstream has since removed.Verification
JDK 17:
mvn clean testAll 121 reactor modules are verified across the initial run and its continuation: 11,355 cases, zero failures/errors, 112 skips. The first run passed 67 modules before a truncated protobuf download from Central; after recovering that dependency and installing the prerequisite reactor artifacts,
mvn -rf :langchain4j-milvus-v2 clean testpasses the remaining 54 modules. The original interrupted command is not reported as a successful single run. MCP has 469 unit cases with no skips.JDK 21 with JBang and current reactor dependencies installed:
mvn -pl langchain4j-mcp \ '-Dtest=DefaultMcpToolResultConverterTest,McpToolResultMetaParsingTest,McpTools*IT#executeToolThatReturnsNativeImage' \ -DskipITs packageAll 33 selected cases pass with no skips, including native-image calls over eight modern/legacy Stdio, Streamable HTTP and WebSocket paths, with JDK, OkHttp and Apache HTTP clients. The local fixture temporarily used a free port because 8080 was occupied; those port-only overrides were reverted. Package/Javadocs and scoped Spotless checks pass.
Provider tests use mocked I/O. No live-model perception or complete provider/MCP integration-suite result is claimed. Checks from older commits are not results for this refreshed head.
General checklist