Repository navigation
feat(mcp): forward the routed language and reuse shortlist embeddings - #523
Merged
NandhaKishorM merged 1 commit intoSep 27, 2026
Conversation
Two small gaps the shortlist tool has since it landed, both flagged for a
follow-up when it merged.
**Per-language temperatures.** In auto mode the tool routes once and then
answers with an explicit ``model=`` so the forward pass does not re-route.
That explicit model makes ``Router.predict`` build an explicit-model
decision, which carries no language detection -- so the language the route
had just detected was dropped, and the answering checkpoint's per-language
temperatures did not apply the way they do in ``laya_predict``. The tool now
forwards the routed language alongside ``model=`` when the decision reports
one, and adds nothing when it does not.
**Repeated embedding of a fixed option list.** ``predict_shortlist`` embeds
the query plus every option text on each call, and the tool built a fresh
``embed_fn_from_agent`` embedder every time, so a fixed intent or label list
was re-encoded on every request. The embedder is now wrapped with
``cached_embed_fn`` once per answering checkpoint.
The wrapper is stored on the answering agent itself, so the cache lives
exactly as long as that checkpoint: a different checkpoint, or the same
checkpoint reloaded into a fresh ``Agent``, starts empty and can never reuse
another checkpoint's rows. Nothing is kept in module state, and the cache
stays bounded by ``cached_embed_fn``'s own LRU limit. An injected
``embed_fn`` is still used as given -- that path is the caller's to own.
Routing still happens once, explicit-model and direct-agent modes are
unchanged, lazy loading and ``LAYA_PRELOAD=0`` are untouched, the import of
``laya.shortlist`` stays lazy, and the result and metadata shapes do not
change.
## Tests
``tests/test_mcp.py`` drives the answering checkpoint's own encoder through
a weight-free fake tokenizer/encoder, counting embedded texts:
- an identical repeat call embeds nothing;
- a repeat with a new state embeds the query alone and reuses the option rows;
- a different checkpoint embeds its own six texts and does not reuse another's;
- a reloaded agent starts fresh, and the original agent's cache is unaffected;
- an injected ``embed_fn`` is called on every call and is never wrapped;
- a decision with no detection adds no ``lang``, so ``seen_kwargs`` stays
``{"model": ...}`` exactly as before;
- the reported ``routing`` and result shape are unchanged.
Suites: ``test_mcp.py`` 136, ``test_shortlist.py`` 154,
``test_shortlist_cosine.py`` 3, ``test_lazy_import.py`` 7,
``test_router.py`` 546, all passing, plus ``compileall``, the repository
ruff gate and ``git diff --check``.
Note for review: NandhaKishorM#513 also edits ``laya/mcp/tools.py`` and
``tests/test_mcp.py`` for the batch tools, so the two branches touch
overlapping regions of those files even though the changes are independent.
This was referenced Sep 26, 2026
Owner
|
Thanks @Bruce-Yii, merged. With #567 on top, a |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two gaps in
laya_shortlistsince it landed, both of the kind flagged for a follow-up when it merged.Per-language temperatures. In auto mode the tool routes once, then answers with an explicit
model=so the forward pass does not re-route. That explicit model makesRouter.predictbuild an explicit-model decision, which carries no language detection — so the language the route had just detected was dropped, and the answering checkpoint's per-language temperatures did not apply the way they do inlaya_predict. The tool now forwards the routed language next tomodel=when the decision reports one, and adds nothing when it does not.Repeated embedding of a fixed option list.
predict_shortlistembeds the query plus every option text on each call, and the tool built a freshembed_fn_from_agentembedder every time — so a fixed intent or label list was re-encoded on every request. The embedder is now wrapped withcached_embed_fnonce per answering checkpoint.The wrapper is stored on the answering agent itself, so the cache lives exactly as long as that checkpoint: a different checkpoint, or the same checkpoint reloaded into a fresh
Agent, starts empty and can never reuse another checkpoint's rows. Nothing is kept in module state, and the cache stays bounded bycached_embed_fn's own LRU limit. An injectedembed_fnis still used exactly as given — that path belongs to the caller.Routing still happens once. Explicit-model and direct-agent modes, lazy loading,
LAYA_PRELOAD=0, the lazy import oflaya.shortlist, and the result and metadata shapes are all unchanged.Tests
tests/test_mcp.pydrives the answering checkpoint's own encoder through a weight-free fake tokenizer/encoder and counts embedded texts:embed_fnruns on every call and is never wrapped;lang, leavingseen_kwargsas{"model": ...}exactly as before;routingand the result shape are unchanged.Suites
python tests/test_mcp.py— 136 passed, 0 failedpython tests/test_shortlist.py— 154 passed, 0 failedpython -m pytest tests/test_shortlist_cosine.py— 3 passedpython tests/test_lazy_import.py— 7 passed, 0 failedpython tests/test_router.py— 546 passed, 0 failedpython -m compileall -q laya testsruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120git diff --checkFor the reviewer
#513 also edits
laya/mcp/tools.pyandtests/test_mcp.pyto add the batch tools. The two changes are independent, but they touch overlapping regions of those two files, so one of them will need a rebase.