You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
perf(core): parallelize recursive retrieval using run_jobs - #23127
Resolve the recursive-retrieval TODO in BaseRetriever by scheduling independent async object lookups through the existing bounded run_jobs helper. Flatten results in input order, then preserve the existing first-node-ID deduplication behavior.
Add three regression tests through the public aretrieve API:
Event-coordinated child retrievers must overlap even when the second finishes first; returned nodes retain input order.
Direct and unresolved index nodes pass through, and duplicate node IDs keep the first result and score.
Nested retrieval errors propagate to the caller.
The follow-up adds tests only; it does not change the production implementation further.
Validation
uv run --no-sync pytest tests/retrievers -q: 8 passed on the updated feature branch.
Counterfactual check: the new concurrency/order test fails against the original sequential BaseRetriever; the other two compatibility tests pass (1 failed, 2 passed).
Ruff, Ruff formatting, Black with Python 3.10 target, and git diff --check pass.
Python-applicable repository pre-commit hooks pass when invoked individually. Full pre-commit bootstrap is not claimed as passed: the unrelated legacy Prettier/nodeenv setup fails parsing an upstream Node prerelease version (0-alpha). Repository hook configuration was not modified or disabled.
Tests use static/mock retrievers and asyncio events, with no live model calls. The five-second timeout only bounds a broken sequential run; no throughput benchmark or full-core-suite result is claimed.
AI assistance: the follow-up regression tests were prepared with Codex assistance and validated with the checks above. Upstream review and merge remain with the maintainers.
Hi maintainers, could someone authorize the current-head CI runs and review this change?
Head f80c99a includes the concurrency/order, pass-through/deduplication, and error-propagation regression tests added on November 9. The retriever suite passed locally (8 tests), and the new concurrency test fails against the original sequential implementation. The PR body records the validation and limitations, including the unrelated pre-commit bootstrap problem.
Unit Testing, Core Typecheck, Build Package, Linting, and CodeQL are still action_required, so I cannot check the upstream results yet. I can address any failures or review feedback once those run. The tests use static retrievers and make no live model calls.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change
Resolve the recursive-retrieval TODO in
BaseRetrieverby scheduling independent async object lookups through the existing boundedrun_jobshelper. Flatten results in input order, then preserve the existing first-node-ID deduplication behavior.Add three regression tests through the public
aretrieveAPI:The follow-up adds tests only; it does not change the production implementation further.
Validation
uv run --no-sync pytest tests/retrievers -q: 8 passed on the updated feature branch.BaseRetriever; the other two compatibility tests pass (1 failed, 2 passed).git diff --checkpass.0-alpha). Repository hook configuration was not modified or disabled.Tests use static/mock retrievers and asyncio events, with no live model calls. The five-second timeout only bounds a broken sequential run; no throughput benchmark or full-core-suite result is claimed.
Upstream workflow authorization (November 11): the current-head Unit Testing, Core Typecheck, Build Package, Linting, and CodeQL workflow runs are awaiting maintainer approval (
action_required). For example: https://github.com/run-llama/llama_index/actions/runs/42699760985. No upstream CI pass is claimed.AI assistance: the follow-up regression tests were prepared with Codex assistance and validated with the checks above. Upstream review and merge remain with the maintainers.