Repository navigation
fix(azure-ai): make AzureAIImageAnalysisTool usable - #1302
Merged
Ying Hua (herohua) merged 1 commit intoNov 5, 2026
Merged
Ying Hua (herohua) merged 1 commit into
Ying Hua (herohua) merged 1 commit into
Conversation
AzureAIImageAnalysisTool cannot be constructed: visual_features is annotated
as a single VisualFeatures while the pre_init validator fills in a list, so
pydantic rejects every instance. AzureAIServicesToolkit.get_tools() builds
the tool and fails with it.
Fixing the annotation exposes two more problems. The formatter reads
captionResult["captions"], but the service returns confidence and text, so
any response with a caption raises KeyError. HttpResponseError is also turned
into a dict the formatter ignores, so a failed call returns '{}'.
Annotate the field as a list, return the caption result as is, and add the
dense captions that are requested by default but never returned. Let service
errors reach the RuntimeError wrapper in _run, chained to the original error.
Add unit tests, parsing responses with the real ImageAnalysisResult model.
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are identified for the image-analysis tool changes.
No open issues 🎉
What changed in this PR
This PR restores AzureAIImageAnalysisTool usability in langchain-azure-ai, removing a construction blocker that also affects AzureAIServicesToolkit.get_tools().
Changes:
- Corrects
visual_featuresto accept a list and formats caption and dense-caption responses correctly. - Propagates service failures through the existing
RuntimeErrorwrapper while preserving the original cause. - Adds regression tests for construction, all three image sources, feature formatting, and service errors.
| File | Description |
|---|---|
libs/azure-ai/tests/unit_tests/test_image_analysis_tool.py |
Adds regression coverage using real SDK response models and mocked service calls. |
libs/azure-ai/langchain_azure_ai/tools/services/image_analysis.py |
Fixes feature typing, caption output, and service-error propagation. |
Review effort: Balanced
Ying Hua (herohua)
approved these changes
Nov 5, 2026
Collaborator
|
Thank you Hyeonsang Cho (@hyeonsang010716) for your first-time contribution! This change will be released in next version. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
AzureAIImageAnalysisToolcan't be used at the moment, and becauseAzureAIServicesToolkit.get_tools()builds it, the toolkit fails with it. There are three separate problems, each hidden by the one before it. The tool had no unit tests, so none of them showed up.visual_featuresis annotated asOptional[VisualFeatures], but thepre_initvalidator fills in a list of features and iterates over whatever the caller passes. Pydantic then rejects the list, so evenAzureAIImageAnalysisTool(endpoint=..., credential=...)raisesValidationError._format_image_analysis_resultreadscaptionResult["captions"], but the service returns{"confidence": ..., "text": ...}, so any response with a caption ends inKeyError: 'captions'.denseCaptionsResultis requested by default but never makes it to the output._image_analysisturnsHttpResponseErrorinto a dict, and the formatter finds none of the keys it knows about, so a failed call (a bad key, for example) returns'{}'.Fix
visual_featuresasOptional[List[VisualFeatures]], which is what the validator andImageAnalysisClientalready work with.captionand adddenseCaptionsto the output.HttpResponseErrorso it reaches the existingRuntimeErrorwrapper in_run, and chain it withfrom e. This matches how the Document Intelligence and Content Understanding tools let service errors through.No public behavior that worked before changes, since the tool could not be constructed.
Testing
Added
tests/unit_tests/test_image_analysis_tool.py, with responses parsed by the realImageAnalysisResultmodel so the tests depend on the service's field names rather than on the tool's assumptions. It covers construction, theurl/path/base64sources, formatting of every visual feature, and service errors.main. With only the annotation fixed, 5 still fail: 4 withKeyError: 'captions', and the service error test because nothing is raised.ruff check,ruff format --diffandmypyare clean on the touched files.RuntimeErrorwith the original error as its cause.AzureAIServicesToolkit(...).get_tools()returns all six tools with an API key.Not tested against a live Azure AI Vision resource.
Follow-up
With a
TokenCredential,get_tools()now gets past this tool but still fails onAzureAISpeechToTextTool, becausespeechsdk.SpeechConfigrejectsauth_tokencombined withendpoint. That's a separate bug, so I left it out of this PR.