Skip to content

fix(azure-ai): make AzureAIImageAnalysisTool usable - #1302

Merged
Ying Hua (herohua) merged 1 commit into
langchain-ai:mainfrom
hyeonsang010716:fix/azure-ai-image-analysis-tool
Nov 5, 2026
Merged

Ying Hua (herohua) merged 1 commit into
langchain-ai:mainfrom
hyeonsang010716:fix/azure-ai-image-analysis-tool

Conversation

@hyeonsang010716

Copy link
Copy Markdown
Contributor

Description

AzureAIImageAnalysisTool can't be used at the moment, and because AzureAIServicesToolkit.get_tools() builds it, the toolkit fails with it. There are three separate problems, each hidden by the one before it. The tool had no unit tests, so none of them showed up.

  • Construction always fails. visual_features is annotated as Optional[VisualFeatures], but the pre_init validator fills in a list of features and iterates over whatever the caller passes. Pydantic then rejects the list, so even AzureAIImageAnalysisTool(endpoint=..., credential=...) raises ValidationError.
  • The default features always fail. _format_image_analysis_result reads captionResult["captions"], but the service returns {"confidence": ..., "text": ...}, so any response with a caption ends in KeyError: 'captions'. denseCaptionsResult is requested by default but never makes it to the output.
  • Service errors look like success. _image_analysis turns HttpResponseError into a dict, and the formatter finds none of the keys it knows about, so a failed call (a bad key, for example) returns '{}'.

Fix

  • Annotate visual_features as Optional[List[VisualFeatures]], which is what the validator and ImageAnalysisClient already work with.
  • Return the caption result as caption and add denseCaptions to the output.
  • Stop catching HttpResponseError so it reaches the existing RuntimeError wrapper in _run, and chain it with from e. This matches how the Document Intelligence and Content Understanding tools let service errors through.

No public behavior that worked before changes, since the tool could not be constructed.

Testing

Added tests/unit_tests/test_image_analysis_tool.py, with responses parsed by the real ImageAnalysisResult model so the tests depend on the service's field names rather than on the tool's assumptions. It covers construction, the url/path/base64 sources, formatting of every visual feature, and service errors.

  • 9 of the 10 tests fail on main. With only the annotation fixed, 5 still fail: 4 with KeyError: 'captions', and the service error test because nothing is raised.
  • All 10 pass on Python 3.11, 3.12, 3.13 and 3.14.
  • Full unit suite: 2000 passed, 4 skipped on Python 3.11 and 3.14.
  • ruff check, ruff format --diff and mypy are clean on the touched files.
  • Ran the tool through the real SDK pipeline against a local HTTP server: all three source types send the expected requests, and a 401 response surfaces as RuntimeError with the original error as its cause.
  • AzureAIServicesToolkit(...).get_tools() returns all six tools with an API key.

Not tested against a live Azure AI Vision resource.

Follow-up

With a TokenCredential, get_tools() now gets past this tool but still fails on AzureAISpeechToTextTool, because speechsdk.SpeechConfig rejects auth_token combined with endpoint. That's a separate bug, so I left it out of this PR.

AzureAIImageAnalysisTool cannot be constructed: visual_features is annotated
as a single VisualFeatures while the pre_init validator fills in a list, so
pydantic rejects every instance. AzureAIServicesToolkit.get_tools() builds
the tool and fails with it.

Fixing the annotation exposes two more problems. The formatter reads
captionResult["captions"], but the service returns confidence and text, so
any response with a caption raises KeyError. HttpResponseError is also turned
into a dict the formatter ignores, so a failed call returns '{}'.

Annotate the field as a list, return the caption result as is, and add the
dense captions that are requested by default but never returned. Let service
errors reach the RuntimeError wrapper in _run, chained to the original error.
Add unit tests, parsing responses with the real ImageAnalysisResult model.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are identified for the image-analysis tool changes.

No open issues 🎉

What changed in this PR

This PR restores AzureAIImageAnalysisTool usability in langchain-azure-ai, removing a construction blocker that also affects AzureAIServicesToolkit.get_tools().

Changes:

  • Corrects visual_features to accept a list and formats caption and dense-caption responses correctly.
  • Propagates service failures through the existing RuntimeError wrapper while preserving the original cause.
  • Adds regression tests for construction, all three image sources, feature formatting, and service errors.
File Description
libs/​azure-ai/​tests/​unit_tests/​test_image_analysis_tool.py Adds regression coverage using real SDK response models and mocked service calls.
libs/​azure-ai/​langchain_azure_ai/​tools/​services/​image_analysis.py Fixes feature typing, caption output, and service-error propagation.

Review effort: Balanced

@herohua
Ying Hua (herohua) merged commit 2315de6 into langchain-ai:main Nov 5, 2026
13 checks passed
@herohua

Copy link
Copy Markdown
Collaborator

Thank you Hyeonsang Cho (@hyeonsang010716) for your first-time contribution! This change will be released in next version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

langchain-azure-ai Azure AI integration package

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants