refactor: remove unneeded DocumentJoiner in hybrid_rag_pipeline_with_breakpoints - #309

Fixes #275

What this does

Removes the DocumentJoiner from notebooks/hybrid_rag_pipeline_with_breakpoints.ipynb, one of the 5 notebooks in the repo using OutputAdapter/ListJoiner/DocumentJoiner. Since Haystack 2.24.0, Rankers deduplicate documents by id before ranking, and pipelines support connecting multiple list[T] outputs directly to a single list[T] input — together eliminating the need for a DocumentJoiner placed directly before a Ranker (see the 2.24.0 release notes).

This notebook's doc_joiner = DocumentJoiner(sort_by_score=False) is exactly that case: it merges bm25_retriever and embedding_retriever output and feeds straight into a SentenceTransformersSimilarityRanker, with no other role (sort_by_score=False means it wasn't even doing meaningful reordering itself — a pure list-merge). Removed it and connected both retrievers directly to ranker.documents. answer_builder now receives its documents from ranker (the reranked/deduped top-k) instead of doc_joiner (the raw pre-rank union) — this is both the behavior the release notes intend and consistent with what prompt_builder already used for context.

The other 4 notebooks in scope for #275 — surveyed, left unchanged, with reasons

The issue's linked search covers 5 notebooks total. I audited all 5 individually rather than blanket-removing every occurrence, since several use these components for genuine data-transformation logic that Haystack's new automatic type conversions don't cover:

• async_pipeline.ipynb — DocumentJoiner(join_mode="reciprocal_rank_fusion") performs real reciprocal-rank-fusion scoring across two retrievers with no Ranker afterward. This is genuine algorithmic work the release notes' simplification doesn't apply to ("removes the need for... some joiners", not all) — removing it would lose the RRF ranking entirely.
• github_pr_creator_agent.ipynb — its OutputAdapter concatenates issue_messages + [tool_call_result] via a custom Jinja filter into a new List[ChatMessage]. That's genuine list-construction logic, not a type-compatibility adapter Haystack's new auto-conversion (ChatMessage↔str, T↔list[T]) covers.
• information-extraction-gorilla.ipynb — looked like a trivial single-branch passthrough joiner from the .connect() graph alone, but it's actually a dual-source merge point: fetcher→converter→joiner (URL path) and directly-injected Document objects via extract(text=...) both feed joiner at pipeline-run time (see the "joiner":{"documents":documents} line in the extract() helper). Removing it would break the text= code path, which the notebook actually exercises (the "Andean cock of the rock" example).
• using_hyde_for_improved_retrieval.ipynb — its OutputAdapter usages construct new Document objects from generated text (Document(content=d.text) for d in data), genuine data construction rather than type coercion. There's also a prior partial attempt here (refactor: replace OutputAdapter in using_hyde_for_improved_retrieval notebook #279, closed by a maintainer with "let me know when you want to continue" — an invitation, not a rejection) suggesting this one specifically warrants more careful, dedicated treatment than a mechanical sweep.

Happy to take on any of these three as separate, focused PRs if useful — flagging the reasoning here rather than silently skipping them.

How was this tested?

No OpenAI API key available in my environment, so I couldn't execute the notebook's LLM cells end-to-end. Instead, I extracted this notebook's indexing()/hybrid_retrieval() pipeline-building functions into a standalone script with the rewired topology and ran it directly (no API key needed for retrieval + reranking):

• The pipeline connects without a PipelineConnectError (confirming the new topology is type-compatible).
• For the notebook's own example query ("Where does Mark live?"), the top document reaching answer_builder/prompt_builder is still the Berlin document — matching this notebook's existing cached output ("Mark lives in Berlin.").
• Verified 3 documents flow through end-to-end with no duplicates lost.

I did not re-run the notebook itself (no API key), so the notebook's committed cell outputs are unchanged/stale relative to the new code — flagging this explicitly rather than leaving it implicit. The code change itself is a straightforward, verified rewire; happy to re-run and refresh outputs if a reviewer can supply a key, or a maintainer can do so on merge.