fix: read component replies in HaystackLLMWrapper.generate_text and pin haystack-ai<3 - #3059

Issue Link / Problem Description

• No existing issue for this; describing the problem here.
• HaystackLLMWrapper.generate_text always returned empty text.

Root cause: generate_text calls the generator component's run() directly, but unwrapped the result as pipeline output — component_output.get("llm", {}).get("replies", []) at src/ragas/llms/haystack_wrapper.py:107. A component-level run() returns replies at the top level of the dict; only Pipeline/AsyncPipeline runs nest component output under the component name. Verified against the installed haystack-ai 2.31.0:

AsyncPipeline().run_async(data={"llm": {...}})  # -> {'llm': {'replies': [...], 'meta': [...]}}
component.run("hi")                               # -> {'replies': [...], 'meta': [...]}

So the sync path always produced "", while agenerate_text (src/ragas/llms/haystack_wrapper.py:127-129), which goes through the pipeline, worked.

Second problem: with the extra unpinned, pip install ragas[ai-frameworks] currently resolves haystack-ai 3.x (3.3.0 today), which removed both haystack.AsyncPipeline and the haystack.components.generators.openai|azure|hugging_face_* modules the wrapper imports — so HaystackLLMWrapper.__init__ raises the misleading ImportError: Haystack is not installed... even though haystack is installed. src/ragas/embeddings/haystack_wrapper.py:58 imports the same removed API.

Changes Made

• src/ragas/llms/haystack_wrapper.py: generate_text now reads replies from the component output directly (component_output.get("replies", [])), mirroring what agenerate_text achieves through the pipeline.
• pyproject.toml: pinned the ai-frameworks extra to haystack-ai<3. Both haystack wrappers are written against the 2.x API (top-level AsyncPipeline, string-prompt generators), which 3.x removed, so the extra otherwise installs a version the wrappers cannot import. A 3.x port (chat generators, message-based API) would be a rewrite of both wrappers and is left as a separate effort.
• tests/unit/llms/test_haystack_wrapper.py: new regression tests for generate_text and agenerate_text, using fakes that mirror the component/pipeline output contracts (CI's minimal dev install does not include haystack-ai, so the tests fake the 2.x import surface and run regardless of the installed haystack version).

Testing

How to Test

• Automated tests added/updated
• Manual testing steps:
  1. uv pip install "haystack-ai<3" (resolves 2.31.0), then construct HaystackLLMWrapper(haystack_generator=OpenAIGenerator(api_key=Secret.from_token("dummy"))) — imports and construction succeed (with the unpinned extra, 3.3.0 installs and construction raises the misleading ImportError)
  2. With OpenAIGenerator.run patched to return {"replies": ["real reply"], "meta": [{}]}: wrapper.generate_text(StringPromptValue(text="hi")) returns "real reply"; before the fix it returned ""

Commands and results (fresh .venv, Python 3.13, uv pip install -e ".[dev-minimal]"; locally also langchain-community==0.4.0, since 0.4.2 removed langchain_community.chat_models.vertexai which ragas/llms/base.py:12 imports — pre-existing and unrelated):

• New test before the fix — pytest tests/unit/llms/test_haystack_wrapper.py -v: FAILED ...::test_generate_text_returns_component_reply - AssertionError: assert '' == 'hello world' (1 failed, 1 passed)
• New test after the fix: 2 passed
• Module suite — pytest tests/unit/llms/ -q: 58 passed
• Import checks — pytest tests/unit/test_import.py -q: 3 passed, 1 failed — the failure (test_wrappers_with_missing_haystack, ModuleNotFoundError: No module named 'llama_index') is pre-existing: llama_index is not part of the dev-minimal install, and the test fails identically on the unpatched tree
• ruff check src docs tests --exclude src/ragas/_version.py --config pyproject.toml: All checks passed!
• ruff format --check on the two changed Python files: 2 files already formatted (the repo-wide check also flags 88 pre-existing markdown formatting issues under docs/, outside this diff)
• pyright -p pyproject.toml src: 5 pre-existing errors in src/ragas/cli.py and src/ragas/integrations/tracing/{langfuse,mlflow}.py (files this PR does not touch); haystack_wrapper.py reports 0 errors

References

• haystack-ai 3.x moved generators to haystack.components.generators.chat.* and removed top-level AsyncPipeline; the string-prompt generators the wrappers use are already deprecated in 2.x ("will be removed in Haystack 3.0")
• docs/howtos/integrations/haystack.md covers the separate ragas-haystack package, not these in-repo wrappers

Screenshots/Examples (if applicable)

With haystack-ai 2.31.0 and a generator whose run() returns {"replies": ["real reply"], ...}:

• before: wrapper.generate_text(prompt) → Generation(text="")
• after: wrapper.generate_text(prompt) → Generation(text="real reply")