fix: honor n in ImageTextPrompt.generate_multiple for LangChain LLMs - #3058

Issue Link / Problem Description

• No existing issue found for this (searched open issues for multi_modal / ImageTextPrompt / IndexError), so describing the problem here.
• ImageTextPrompt.generate_multiple (part of the public BasePrompt API, src/ragas/prompt/base.py:53) raises IndexError when called with n > 1 and a LangChain LLM.

Root cause: with a LangChain LLM the method sends a single prompt — agenerate_prompt([prompt_value]) (src/ragas/prompt/multi_modal_prompt.py:176-180 on main) — so the response holds exactly one generation. The parse loop then runs for i in range(n): resp.generations[0][i].text (multi_modal_prompt.py:194-195 on main), which raises IndexError: list index out of range as soon as n > 1.

Reproducer:

from langchain_core.language_models.fake import FakeListLLM
from pydantic import BaseModel
from ragas.prompt.multi_modal_prompt import ImageTextPrompt

class In(BaseModel):
    user_input: str
    def to_string_list(self):
        return [self.user_input]

class Out(BaseModel):
    answer: str

class P(ImageTextPrompt[In, Out]):
    instruction = "answer the question"
    input_model = In
    output_model = Out
    examples = []

# on main: IndexError: list index out of range
outputs = await P().generate_multiple(
    FakeListLLM(responses=['{"answer": "42"}', '{"answer": "43"}']),
    In(user_input="meaning of life?"),
    n=2,
)

The sibling PydanticPrompt.generate_multiple already handles this case: it batches n prompt copies for LangChain LLMs (src/ragas/prompt/pydantic_prompt.py:243-252) and clamps the parse loop with available_generations / actual_n, warning instead of crashing (pydantic_prompt.py:288-306). ImageTextPrompt was missing the same handling.

Changes Made

• src/ragas/prompt/multi_modal_prompt.py: in generate_multiple, the LangChain branch now sends n copies of the prompt to agenerate_prompt() and each output is read from its own batch result (resp.generations[i][0].text), mirroring PydanticPrompt.generate_multiple. The parse loop is clamped to the number of generations actually returned (actual_n = min(n, available_generations)): a ValueError is raised if the LLM returned none, and a warning is logged when it returned fewer than requested — identical to the sibling implementation.
• tests/unit/prompt/test_multi_modal_prompt.py (new): regression tests for n=1 and n=2 with a LangChain LLM, n > 1 with a Ragas LLM, and graceful degradation when a Ragas LLM returns fewer generations than requested.

Testing

How to Test

• Automated tests added/updated
• Manual testing steps:
  1. The reproducer above raises IndexError on main and returns two outputs with this patch (verified with the same fake-LLM setup as the regression tests, run against both the unpatched and the patched tree).

# new tests (all 4 fail with IndexError on main)
__RAGAS_DEBUG_TRACKING=true RAGAS_DO_NOT_TRACK=true pytest tests/unit/prompt/test_multi_modal_prompt.py -v
# 4 passed

# prompt module
__RAGAS_DEBUG_TRACKING=true RAGAS_DO_NOT_TRACK=true pytest tests/unit/prompt -v
# 64 passed

# full unit suite (same shape as CI run-ci-tests)
__RAGAS_DEBUG_TRACKING=true RAGAS_DO_NOT_TRACK=true pytest tests/unit --dist loadfile -n auto -q
# 3 failed, 772 passed, 105 skipped, 8 errors — the 3 failures and 8 errors are pre-existing on main
# (sacrebleu/haystack ModuleNotFoundError in test_chrf_score/test_import, git-backed test_experiment,
# test_prechunked_generation) and reproduce identically with this patch reverted

# lint / format / types (commands from .github/workflows/ci.yaml)
ruff check src docs tests --exclude src/ragas/_version.py --config pyproject.toml
# All checks passed!
ruff format --check src tests docs --exclude src/ragas/_version.py --config pyproject.toml
# only 4 pre-existing markdown files flagged (identical on main); files in this PR are clean
pyright -p pyproject.toml src
# no new errors; the 5 pre-existing errors on main (cli.py, langfuse.py, mlflow.py) are unchanged

References

• Sibling implementation this mirrors: PydanticPrompt.generate_multiple (src/ragas/prompt/pydantic_prompt.py:243-252, 288-314)
• Public API declaration: BasePrompt.generate_multiple (src/ragas/prompt/base.py:53)

Screenshots/Examples (if applicable)

Reproducer behavior with the patch:

• Before: IndexError: list index out of range
• After: returns [Out(answer='42'), Out(answer='43')]