Repository navigation
fix(vectorize): use Cohere's async client for aembed and aembed_many - #785
Merged
Merged
Conversation
vishal-bala
self-requested a review
October 30, 2026 13:05
vishal-bala
requested changes
Oct 30, 2026
CohereTextVectorizer only created a sync Client, so aembed and aembed_many fell back to the blocking sync call (and logged a fallback warning), stalling the event loop in async callers such as SemanticCache.acheck. Create an AsyncClient alongside the sync one, as CohereReranker already does, and implement _aembed and _aembed_many on it. Shared request preparation and response parsing move into helpers so the four paths stay in step.
…client A user-supplied httpx_client is synchronous; handing it to AsyncClient made aembed fail on await where it used to work through the sync fallback. Only build the async client when no sync httpx_client is given, or when async_httpx_client is passed for it.
The async client was created once at init, and its HTTP connection pool is bound to the event loop it first ran on. A vectorizer that outlives a loop, such as one used across two asyncio.run() calls, failed on the next loop with 'Event loop is closed'. The client is now built on first use and rebuilt when the running loop changes; a user-supplied async_httpx_client belongs to the caller's loop and is kept. Also treats httpx_client=None as no sync client rather than one, documents httpx_client and async_httpx_client on __init__ where the API docs render, and trims the unit tests.
Yarmoluk
force-pushed
the
fix/cohere-async-client
branch
from
November 2, 2026 16:35
5bb127a to
289bd03
Compare
vishal-bala
self-requested a review
November 3, 2026 08:34
vishal-bala
approved these changes
Nov 3, 2026
This was referenced Nov 5, 2026
|
🚀 PR was released in |
vishal-bala
pushed a commit
that referenced
this pull request
Nov 6, 2026
Branch: `feat/message-history-async` (fork `Yarmoluk/redis-vl-python`), rebased onto `main` at `506b961` (includes #792 and #785). Commits: `82f91e4` feat, `93657d7` docs, `c1b5d75` test. Label: `auto:minor`. ## Summary Follow-up to #784, which gave `SemanticRouter` `aroute` / `aroute_many`. `MessageHistory` had the same gap: an agent running on asyncio blocked its event loop on every history read or write. This PR adds the async twins -- `aget_recent`, `astore`, `aadd_messages`, `aadd_message`, `acount`, `adrop`, `aclear`, `adisconnect` and an `aindex` property -- using the same pattern #784 landed: - The `AsyncSearchIndex` is built lazily on the first async call, from the sync index's connection settings. A history built from `redis_url` gets its own async connection from that URL and `connection_kwargs`; a history given a sync `redis_client` gets an async client converted from that client's connection settings by `RedisConnectionFactory.sync_to_async_redis`, owned by the async index, so `adisconnect` closes only the converted client and the caller's sync client is never touched or rebuilt. - Every async method goes through the `AsyncSearchIndex`; none reaches into a Redis client directly (`adrop` deletes with `aindex.drop_keys(aindex.key(id))`). - Query building (`_recent_query`, `_count_query`) and message construction (`_chat_messages`, `_exchange`) move into helpers shared by the sync and async paths, so the two cannot drift. - The docstrings say what is and is not supported, in the router's wording: since #792 a sync client over TLS or a unix socket converts; Sentinel and Redis Cluster clients do not (pass `redis_url` instead); the async client is bound to the loop that made the first call, so call `adisconnect` before switching loops. No sync behaviour changes. `SemanticMessageHistory` is not covered here (see Notes). ## Reviewer Brief **Start here:** `redisvl/extensions/message_history/message_history.py`, `_get_async_index()` (line 418). Every async method goes through it, and it decides whether the async index connects from the URL the sync index owns or from a client converted from the caller's sync client; it is the same code as `SemanticRouter._get_async_index`. **Where it could fail** - `message_history.py:432`: a caller-supplied Sentinel or Cluster client, or one with a sync-only setting such as `ssl_validate_ocsp=True`, raises `ValueError` from `sync_to_async_redis` at the first async call; the `aget_recent` docstring names Sentinel and Cluster but, like the router's, not the sync-only-setting case. - `message_history.py:154`: `adrop` removes the key with `UNLINK` (via `drop_keys`) where sync `drop` uses `DEL`; both are `@keyspace @write`, but an ACL that grants `+del` by name and not `+unlink` would allow `drop` and deny `adrop` ([UNLINK](https://redis.io/docs/latest/commands/unlink/), [DEL](https://redis.io/docs/latest/commands/del/)). - `message_history.py:449`: the async client belongs to the loop that made the first call; a second `asyncio.run()` on the same history without `adisconnect` in between fails, the same contract `SemanticRouter` has shipped since #784. **Not verified:** Redis Cluster and Sentinel (no cluster tests run locally or in CI); the TLS and unix-socket conversion paths were not exercised end to end through `MessageHistory` (they are covered by #792's own tests); `connection_kwargs` and `lib_name` forwarding on the URL path is not asserted by a test. **Beyond CI:** each of these mutations makes `test_async_message_history_matches_sync` fail: `astore` ignoring `session_tag`, `adrop` going through the sync client, `adrop(id)` ignoring `id`, the async index rebuilt per call, and `adisconnect` not closing the converted client. The two new integration tests pass locally against `redis:8.4` via testcontainers. **Since the last brief:** rebased onto `506b961` (#792 now converts TLS and unix-socket clients, so the caveat narrows to Sentinel and Cluster); `adrop` deletes through `aindex.drop_keys` instead of the async client; the sync-equivalence test now blocks the sync client's `execute_command` and `pipeline`, seeds a second session with `astore`, and pins `tool_call_id`/metadata, `adrop(id)`, async-index reuse and `adisconnect` closing the converted pool. <sub>Reviewed `c1b5d75` · Claude Code (claude-opus-5-5) · lenses: correctness, security, design, tests, docs, Redis behaviour</sub> ## Test plan Run on the rebased branch at `c1b5d75`, macOS, `uv`, Docker running (testcontainers starts `redis:8.4`): ```bash make format # All done! ✨ 🍰 ✨ # 272 files left unchanged. rm -rf .mypy_cache && make check-types # Success: no issues found in 122 source files uv run pytest tests/integration/test_message_history.py -k async_message_history -q -p no:cacheprovider # 2 passed, 33 deselected uv run pytest -n auto tests/integration/test_message_history.py tests/unit -q -p no:cacheprovider # 3 failed, 1500 passed, 9 skipped, 53 errors ``` The 3 failures and 53 errors are the identical set on `main` at `506b961`: every one is `ModuleNotFoundError: sentence_transformers` (the HuggingFace vectorizer is not installed locally), in `SemanticMessageHistory` tests this PR does not touch. `test_async_message_history_matches_sync` takes its expected values from the sync methods, then patches `execute_command` and `pipeline` on the sync index's Redis client to raise, so any async method that made a blocking call would fail it. `test_async_message_history_with_redis_url` checks that a URL-built history builds its async index from that URL rather than converting the client. ## Notes - `SemanticMessageHistory` gets no async methods in this PR. Its `get_relevant` embeds through the vectorizer, which brings in the `aembed` caveats #784 documented for the router; it is a separate change with its own brief. - No `adelete`: dropping the index is setup/teardown rather than a per-turn call, and #784 drew the same line for the router. No async form of the `messages` property either, since a property cannot be awaited; `aget_recent` with a large `top_k` is the async read. - Loop binding follows #784's `adisconnect` contract rather than the rebuild-on-new-loop #785 added for Cohere's async client. The async index owns a connection pool, and rebuilding it silently on a loop change would leak the previous pool; an explicit `adisconnect` keeps the two extensions consistent and the close visible. - `_get_async_index`, `aindex` and `adisconnect` match `SemanticRouter`'s line for line; they are kept as two copies rather than a shared helper, since extracting one would mean changing the merged router code in this PR. - `count`/`acount` keep the sync method's unannotated signature rather than adding `-> int` to one twin only. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > New async Redis connection lifecycle (lazy index, loop binding, client conversion) could misbehave for Sentinel/Cluster/unix-socket setups; sync API and data paths are unchanged. > > **Overview** > Adds **async counterparts** to `MessageHistory` read/write/clear/drop operations (`aget_recent`, `astore`, `aadd_messages`, `aadd_message`, `acount`, `adrop`, `aclear`, plus `adisconnect` and an `aindex` property) so asyncio agents can use history without blocking the event loop. > > Async work goes through a **lazily created** `AsyncSearchIndex` that reuses the sync index’s schema and connection settings: URL-owned clients open a separate async connection from `redis_url`; caller-supplied sync clients get a derived async client via `RedisConnectionFactory.sync_to_async_redis`, closed only by `adisconnect`. Docstrings document loop binding and that Sentinel/Cluster sync clients need `redis_url` instead. > > Sync behavior is preserved by **extracting shared helpers** (`_recent_query`, `_recent_result`, `_count_query`, `_exchange`, `_chat_messages`) so sync and async paths share query and message construction. Two integration tests assert parity with sync (including a monkeypatch that fails if async calls hit the sync Redis client) and cover the `redis_url` async-index path. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit c1b5d75. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
CohereTextVectorizeronly creates a synccohere.Client, soaembedandaembed_manygo throughBaseVectorizer's fallback (base.py:350-364): they log "Falling back to sync" and make a blocking HTTP call on the event loop. Any async caller using Cohere, such asSemanticCache.acheck/astoreorSemanticRouter's async path, stalls the loop for the length of each embedding request.CohereRerankeralready creates acohere.AsyncClientnext to its sync client (rerank/cohere.py:84,99); this brings the vectorizer in line.Changes
_initialize_clientcreatesself._aclient = AsyncClient(...)alongside the sync client, with the same API key,client_name="redisvl"and kwargs._aembedand_aembed_manycall the async client._aembed_manykeeps the sync method's batching and@retrypolicy.input_typecheck, theembedding_typeswarning, mappingdtypeto Cohere embedding types) and response parsing move into_prepare_requestand_extract_embeddings. The sync and async paths share them, so the four methods can't drift apart.httpx_clientis synchronous and can't be awaited. When one is passed, the async client isn't created, andaembed/aembed_manykeep using the sync fallback exactly as they do today. A new optionalasync_httpx_clientkwarg gives the async client its own transport.tests/unit/test_cohere_vectorizer.py, using a stand-incoheremodule (the same approach astest_ollama_vectorizer.py), so it runs in CI without an API key.Notes
CohereRerankerpasses the same kwargs to both of its clients, so a synchttpx_clientbreaks itsarankthe same way. That's left out of this PR to keep it to the vectorizer; it's a small follow-up if you want it._aembed_manyretries the whole batch loop on any non-TypeErrorerror, re-sending batches that already succeeded. That matches the existing_embed_many, so changing the retry policy is left to a separate change that can cover both.Release Notes
CohereTextVectorizer.aembedandaembed_manynow make non-blocking calls through Cohere's async client instead of falling back to the sync client. If you pass your ownhttpx_client, async calls still use it through the sync fallback; passasync_httpx_clientas well to make them non-blocking.Reviewer Brief
Start here:
redisvl/utils/vectorize/text/cohere.py,_get_aclient. It decides whether anaembed/aembed_manycall gets an async client, and which one;_initialize_clientonly records what to build.Where it could fail
redisvl/utils/vectorize/text/cohere.py,_get_aclient: a replaced client is dropped withoutaclose(), so the old loop's pooled connections are left to garbage collection; check that is acceptable against holding a reference until the vectorizer goes away.redisvl/utils/vectorize/text/cohere.py,_initialize_client: the async client is skipped whenever a synchttpx_clientis passed withoutasync_httpx_client; check this keeps today's behaviour for sync-transport users.redisvl/utils/vectorize/text/cohere.py,_aembed_many_with: the only async method wrapped in@retry; check thatTypeErrorfrom input validation still surfaces at once.Not verified: live Cohere API calls (
tests/integration/test_vectorizers.pyneedsCOHERE_API_KEY); thecohere>=4.44floor, only 7.0.0 was installed;CohereRerankerhas the same once-builtAsyncClientand is left for a follow-up.Beyond CI: against the real cohere 7.0.0 SDK and a local keep-alive HTTP server, two consecutive
asyncio.run(v.aembed(...))calls fail on the second withEvent loop is closedonmainand pass on this branch. Of the 12 unit tests, the rebuild, reuse,httpx_client=Noneand kept-user-client tests each fail when their guard is removed.Since the last brief: 289bd03 builds the async client lazily per event loop, fixes the
httpx_client=Nonecheck, moves the kwarg docs to__init__, and trims the tests.Reviewed
289bd03· Claude Code (claude-fable-5-1) · lenses: correctness, tests, design, docsNote
Medium Risk
Changes async embedding I/O for Cohere callers (e.g. semantic cache/router) with nuanced httpx and event-loop client lifecycle, though sync APIs and explicit sync-transport fallback are preserved and covered by new tests.
Overview
CohereTextVectorizerno longer blocks the event loop onaembed/aembed_many: it calls Cohere'sAsyncClientinstead of inheritingBaseVectorizer's sync fallback.Client setup now lazily builds an async client (with optional
async_httpx_client) and recreates it when the running event loop changes; a custom sync-onlyhttpx_clientstill disables the async client and keeps the previous fallback behavior. Shared_prepare_request,_extract_embeddings, and_validate_contentsdedupe the sync and async embed paths without changing their outward behavior.New unit tests in
test_cohere_vectorizer.pyexercise async batching, dtype/embedding_typeshandling, and the httpx client split using a fakecoheremodule.Reviewed by Cursor Bugbot for commit 289bd03. Bugbot is set up for automated code reviews on this repo. Configure here.