Skip to content

feat(mcp): forward the routed language and reuse shortlist embeddings - #523

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
Bruce-Yii:feat/mcp-shortlist-lang-embed-cache
Sep 27, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
Bruce-Yii:feat/mcp-shortlist-lang-embed-cache

Conversation

@Bruce-Yii

Copy link
Copy Markdown
Contributor

What

Two gaps in laya_shortlist since it landed, both of the kind flagged for a follow-up when it merged.

Per-language temperatures. In auto mode the tool routes once, then answers with an explicit model= so the forward pass does not re-route. That explicit model makes Router.predict build an explicit-model decision, which carries no language detection — so the language the route had just detected was dropped, and the answering checkpoint's per-language temperatures did not apply the way they do in laya_predict. The tool now forwards the routed language next to model= when the decision reports one, and adds nothing when it does not.

Repeated embedding of a fixed option list. predict_shortlist embeds the query plus every option text on each call, and the tool built a fresh embed_fn_from_agent embedder every time — so a fixed intent or label list was re-encoded on every request. The embedder is now wrapped with cached_embed_fn once per answering checkpoint.

The wrapper is stored on the answering agent itself, so the cache lives exactly as long as that checkpoint: a different checkpoint, or the same checkpoint reloaded into a fresh Agent, starts empty and can never reuse another checkpoint's rows. Nothing is kept in module state, and the cache stays bounded by cached_embed_fn's own LRU limit. An injected embed_fn is still used exactly as given — that path belongs to the caller.

Routing still happens once. Explicit-model and direct-agent modes, lazy loading, LAYA_PRELOAD=0, the lazy import of laya.shortlist, and the result and metadata shapes are all unchanged.

Tests

tests/test_mcp.py drives the answering checkpoint's own encoder through a weight-free fake tokenizer/encoder and counts embedded texts:

  • an identical repeat call embeds nothing;
  • a repeat with a new state embeds the query alone and reuses the cached option rows;
  • a different checkpoint embeds its own six texts and never reuses another's;
  • a reloaded agent starts fresh, and the original agent's cache is unaffected;
  • an injected embed_fn runs on every call and is never wrapped;
  • a route decision without detection adds no lang, leaving seen_kwargs as {"model": ...} exactly as before;
  • the reported routing and the result shape are unchanged.

Suites

  • python tests/test_mcp.py — 136 passed, 0 failed
  • python tests/test_shortlist.py — 154 passed, 0 failed
  • python -m pytest tests/test_shortlist_cosine.py — 3 passed
  • python tests/test_lazy_import.py — 7 passed, 0 failed
  • python tests/test_router.py — 546 passed, 0 failed
  • python -m compileall -q laya tests
  • ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
  • git diff --check

For the reviewer

#513 also edits laya/mcp/tools.py and tests/test_mcp.py to add the batch tools. The two changes are independent, but they touch overlapping regions of those two files, so one of them will need a rebase.

Two small gaps the shortlist tool has since it landed, both flagged for a
follow-up when it merged.

**Per-language temperatures.** In auto mode the tool routes once and then
answers with an explicit ``model=`` so the forward pass does not re-route.
That explicit model makes ``Router.predict`` build an explicit-model
decision, which carries no language detection -- so the language the route
had just detected was dropped, and the answering checkpoint's per-language
temperatures did not apply the way they do in ``laya_predict``. The tool now
forwards the routed language alongside ``model=`` when the decision reports
one, and adds nothing when it does not.

**Repeated embedding of a fixed option list.** ``predict_shortlist`` embeds
the query plus every option text on each call, and the tool built a fresh
``embed_fn_from_agent`` embedder every time, so a fixed intent or label list
was re-encoded on every request. The embedder is now wrapped with
``cached_embed_fn`` once per answering checkpoint.

The wrapper is stored on the answering agent itself, so the cache lives
exactly as long as that checkpoint: a different checkpoint, or the same
checkpoint reloaded into a fresh ``Agent``, starts empty and can never reuse
another checkpoint's rows. Nothing is kept in module state, and the cache
stays bounded by ``cached_embed_fn``'s own LRU limit. An injected
``embed_fn`` is still used as given -- that path is the caller's to own.

Routing still happens once, explicit-model and direct-agent modes are
unchanged, lazy loading and ``LAYA_PRELOAD=0`` are untouched, the import of
``laya.shortlist`` stays lazy, and the result and metadata shapes do not
change.

## Tests

``tests/test_mcp.py`` drives the answering checkpoint's own encoder through
a weight-free fake tokenizer/encoder, counting embedded texts:

- an identical repeat call embeds nothing;
- a repeat with a new state embeds the query alone and reuses the option rows;
- a different checkpoint embeds its own six texts and does not reuse another's;
- a reloaded agent starts fresh, and the original agent's cache is unaffected;
- an injected ``embed_fn`` is called on every call and is never wrapped;
- a decision with no detection adds no ``lang``, so ``seen_kwargs`` stays
  ``{"model": ...}`` exactly as before;
- the reported ``routing`` and result shape are unchanged.

Suites: ``test_mcp.py`` 136, ``test_shortlist.py`` 154,
``test_shortlist_cosine.py`` 3, ``test_lazy_import.py`` 7,
``test_router.py`` 546, all passing, plus ``compileall``, the repository
ruff gate and ``git diff --check``.

Note for review: NandhaKishorM#513 also edits ``laya/mcp/tools.py`` and
``tests/test_mcp.py`` for the batch tools, so the two branches touch
overlapping regions of those files even though the changes are independent.
@NandhaKishorM
NandhaKishorM merged commit b1412db into NandhaKishorM:main Sep 27, 2026
22 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thanks @Bruce-Yii, merged. With #567 on top, a lang the caller passes wins over the detected one; the detected language is forwarded only when the caller did not name one. It ships in 0.3.21.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants