fix(llms): forward instructor mode for anthropic and google/gemini clients - #3022

Issue Link / Problem Description

• Fixes llm_factory: mode= is silently discarded for provider='anthropic' and provider='google'/'gemini' (new google-genai) #3021

llm_factory(..., mode=...) documents mode as the instructor mode for structured output (default Mode.JSON), and _get_instructor_client() normalizes None to Mode.JSON. That normalized value was only forwarded to some branches — instructor.from_openai(client, mode=mode), instructor.from_litellm(client, mode=mode) and _patch_client_for_provider(..., mode=mode) — while the anthropic and google branches called instructor.from_anthropic(client), instructor.from_genai(client) and instructor.from_gemini(client) with no mode argument. The mode was accepted and then discarded, so the patched client stayed on instructor's own per-factory default (Mode.TOOLS for from_anthropic and from_genai, Mode.MD_JSON for from_gemini) instead of the requested — and documented-default — Mode.JSON.

Root cause, unpatched src/ragas/llms/base.py: the effective mode is computed at lines 584-585 but lines 591, 594 and 596 call the three factories without it.

Changes Made

• src/ragas/llms/base.py: forward mode to instructor.from_anthropic, instructor.from_genai and instructor.from_gemini, matching the existing from_openai/from_litellm branches.
• src/ragas/llms/base.py: make the None default provider-aware. google-generativeai (old SDK) has no JSON mode — instructor supports only Mode.TOOLS and Mode.MD_JSON for it and raises ModeError: Invalid mode 'json_mode' for provider 'gemini'. Valid modes: markdown_json_mode, tool_call — so a defaulted mode resolves to Mode.MD_JSON for that client, which is what it already got via instructor's own default. Without this, forwarding the normalized Mode.JSON would turn the working default path into a hard error. An explicitly requested mode is always forwarded, so an unsupported explicit request now fails loudly in instructor instead of being ignored.
• tests/unit/llms/test_instructor_factory.py: add test_llm_factory_mode_reaches_provider_factories, covering both an explicitly requested mode and the default for provider="anthropic", "google" (new google-genai client) and "gemini" (old SDK client).

Not changed: provider="perplexity" also ignores mode, but instructor's perplexity provider supports only Mode.MD_JSON, which is its own default, so forwarding the normalized Mode.JSON there would raise ModeError rather than fix anything.

Testing

How to Test

• Automated tests added/updated
• Manual testing steps:

pytest tests/unit/llms/test_instructor_factory.py

New test on the unpatched tree (src/ragas/llms/base.py at main 298b682, test kept):

>           assert captured_mode[factory_name] == instructor.Mode.TOOLS
E           AssertionError: assert None == <Mode.TOOLS: 'tool_call'>
E            +  where <Mode.TOOLS: 'tool_call'> = <enum 'Mode'>.TOOLS
tests/unit/llms/test_instructor_factory.py:385: AssertionError
=========================== short test summary info ============================
FAILED tests/unit/llms/test_instructor_factory.py::test_llm_factory_mode_reaches_provider_factories
1 failed in 0.30s

On the patched tree:

pytest tests/unit/llms/test_instructor_factory.py   ->  16 passed in 0.21s
pytest tests/unit/llms                              ->  57 passed
pytest tests/unit --ignore=tests/unit/integrations  ->  3 failed, 742 passed, 62 skipped, 8 errors

Unmodified main reports the identical failure/error set (3 failed, 741 passed, 62 skipped, 8 errors); the only difference is the added test. The pre-existing failures are missing optional sacrebleu/haystack dependencies and test_experiment.py git fixtures.

Lint and types: ruff format --check and ruff check pass on both changed files (the repo-wide ruff format --check reports 88 pre-existing reformats in files this PR does not touch); ruff check src tests docs passes repo-wide; pyright -p pyproject.toml src reports its 6 pre-existing Module is not callable errors in src/ragas/integrations/* and 0 errors in src/ragas/llms/base.py.

Local run note (environment only, not part of the diff): import ragas currently fails on main with langchain-community 0.4.x, because src/ragas/llms/base.py:12 imports langchain_community.chat_models.vertexai unconditionally and that module no longer exists. I pinned langchain-community==0.3.31 in a local venv to run the suite; this PR does not touch that separate breakage.

References

• Issue: llm_factory: mode= is silently discarded for provider='anthropic' and provider='google'/'gemini' (new google-genai) #3021
• Instructor's per-provider mode support and defaults, verified against instructor 1.17.0:
  from_anthropic supports TOOLS/JSON/JSON_SCHEMA/PARALLEL_TOOLS (default TOOLS); from_genai supports TOOLS/JSON (default TOOLS); from_gemini supports TOOLS/MD_JSON (default MD_JSON).
• Related: issue Testset generation broken: instructor returns empty dict for Dict-type Pydantic models #2490, the reason Mode.JSON is the documented default for OpenAI.

Screenshots/Examples (if applicable)

N/A — argument forwarding plus a provider-aware default.