Skip to content

fix(vectorize): use Cohere's async client for aembed and aembed_many - #785

Merged
vishal-bala merged 3 commits into
redis:mainfrom
Yarmoluk:fix/cohere-async-client
Nov 3, 2026
Merged

vishal-bala merged 3 commits into
redis:mainfrom
Yarmoluk:fix/cohere-async-client

Conversation

@Yarmoluk

@Yarmoluk Yarmoluk commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

CohereTextVectorizer only creates a sync cohere.Client, so aembed and aembed_many go through BaseVectorizer's fallback (base.py:350-364): they log "Falling back to sync" and make a blocking HTTP call on the event loop. Any async caller using Cohere, such as SemanticCache.acheck / astore or SemanticRouter's async path, stalls the loop for the length of each embedding request.

CohereReranker already creates a cohere.AsyncClient next to its sync client (rerank/cohere.py:84,99); this brings the vectorizer in line.

Changes

  • _initialize_client creates self._aclient = AsyncClient(...) alongside the sync client, with the same API key, client_name="redisvl" and kwargs.
  • New _aembed and _aembed_many call the async client. _aembed_many keeps the sync method's batching and @retry policy.
  • Request preparation (the input_type check, the embedding_types warning, mapping dtype to Cohere embedding types) and response parsing move into _prepare_request and _extract_embeddings. The sync and async paths share them, so the four methods can't drift apart.
  • A user-supplied httpx_client is synchronous and can't be awaited. When one is passed, the async client isn't created, and aembed / aembed_many keep using the sync fallback exactly as they do today. A new optional async_httpx_client kwarg gives the async client its own transport.
  • New tests/unit/test_cohere_vectorizer.py, using a stand-in cohere module (the same approach as test_ollama_vectorizer.py), so it runs in CI without an API key.

Notes

  • CohereReranker passes the same kwargs to both of its clients, so a sync httpx_client breaks its arank the same way. That's left out of this PR to keep it to the vectorizer; it's a small follow-up if you want it.
  • _aembed_many retries the whole batch loop on any non-TypeError error, re-sending batches that already succeeded. That matches the existing _embed_many, so changing the retry policy is left to a separate change that can cover both.

Release Notes

CohereTextVectorizer.aembed and aembed_many now make non-blocking calls through Cohere's async client instead of falling back to the sync client. If you pass your own httpx_client, async calls still use it through the sync fallback; pass async_httpx_client as well to make them non-blocking.

Reviewer Brief

Start here: redisvl/utils/vectorize/text/cohere.py, _get_aclient. It decides whether an aembed / aembed_many call gets an async client, and which one; _initialize_client only records what to build.

Where it could fail

  • redisvl/utils/vectorize/text/cohere.py, _get_aclient: a replaced client is dropped without aclose(), so the old loop's pooled connections are left to garbage collection; check that is acceptable against holding a reference until the vectorizer goes away.
  • redisvl/utils/vectorize/text/cohere.py, _initialize_client: the async client is skipped whenever a sync httpx_client is passed without async_httpx_client; check this keeps today's behaviour for sync-transport users.
  • redisvl/utils/vectorize/text/cohere.py, _aembed_many_with: the only async method wrapped in @retry; check that TypeError from input validation still surfaces at once.

Not verified: live Cohere API calls (tests/integration/test_vectorizers.py needs COHERE_API_KEY); the cohere>=4.44 floor, only 7.0.0 was installed; CohereReranker has the same once-built AsyncClient and is left for a follow-up.

Beyond CI: against the real cohere 7.0.0 SDK and a local keep-alive HTTP server, two consecutive asyncio.run(v.aembed(...)) calls fail on the second with Event loop is closed on main and pass on this branch. Of the 12 unit tests, the rebuild, reuse, httpx_client=None and kept-user-client tests each fail when their guard is removed.

Since the last brief: 289bd03 builds the async client lazily per event loop, fixes the httpx_client=None check, moves the kwarg docs to __init__, and trims the tests.

Reviewed 289bd03 · Claude Code (claude-fable-5-1) · lenses: correctness, tests, design, docs


Note

Medium Risk
Changes async embedding I/O for Cohere callers (e.g. semantic cache/router) with nuanced httpx and event-loop client lifecycle, though sync APIs and explicit sync-transport fallback are preserved and covered by new tests.

Overview
CohereTextVectorizer no longer blocks the event loop on aembed / aembed_many: it calls Cohere's AsyncClient instead of inheriting BaseVectorizer's sync fallback.

Client setup now lazily builds an async client (with optional async_httpx_client) and recreates it when the running event loop changes; a custom sync-only httpx_client still disables the async client and keeps the previous fallback behavior. Shared _prepare_request, _extract_embeddings, and _validate_contents dedupe the sync and async embed paths without changing their outward behavior.

New unit tests in test_cohere_vectorizer.py exercise async batching, dtype/embedding_types handling, and the httpx client split using a fake cohere module.

Reviewed by Cursor Bugbot for commit 289bd03. Bugbot is set up for automated code reviews on this repo. Configure here.

@vishal-bala
vishal-bala self-requested a review October 30, 2026 13:05
@vishal-bala vishal-bala added the auto:minor Increment the minor version when merged label Oct 30, 2026
Comment thread redisvl/utils/vectorize/text/cohere.py Resolved
Comment thread redisvl/utils/vectorize/text/cohere.py Resolved
Comment thread redisvl/utils/vectorize/text/cohere.py Resolved
Comment thread tests/unit/test_cohere_vectorizer.py Resolved
@vishal-bala vishal-bala self-assigned this Oct 30, 2026
CohereTextVectorizer only created a sync Client, so aembed and
aembed_many fell back to the blocking sync call (and logged a
fallback warning), stalling the event loop in async callers such
as SemanticCache.acheck. Create an AsyncClient alongside the sync
one, as CohereReranker already does, and implement _aembed and
_aembed_many on it. Shared request preparation and response
parsing move into helpers so the four paths stay in step.
…client

A user-supplied httpx_client is synchronous; handing it to AsyncClient
made aembed fail on await where it used to work through the sync
fallback. Only build the async client when no sync httpx_client is
given, or when async_httpx_client is passed for it.
The async client was created once at init, and its HTTP connection pool is
bound to the event loop it first ran on. A vectorizer that outlives a loop,
such as one used across two asyncio.run() calls, failed on the next loop
with 'Event loop is closed'. The client is now built on first use and
rebuilt when the running loop changes; a user-supplied async_httpx_client
belongs to the caller's loop and is kept.

Also treats httpx_client=None as no sync client rather than one, documents
httpx_client and async_httpx_client on __init__ where the API docs render,
and trims the unit tests.
@Yarmoluk
Yarmoluk force-pushed the fix/cohere-async-client branch from 5bb127a to 289bd03 Compare November 2, 2026 16:35
@vishal-bala
vishal-bala self-requested a review November 3, 2026 08:34
@vishal-bala
vishal-bala merged commit de3f7b1 into redis:main Nov 3, 2026
56 checks passed
@Yarmoluk
Yarmoluk deleted the fix/cohere-async-client branch November 3, 2026 12:17
@applied-ai-release-bot

Copy link
Copy Markdown

🚀 PR was released in v0.31.0 🚀

@applied-ai-release-bot applied-ai-release-bot Bot added the released This issue/pull request has been released. label Nov 6, 2026
vishal-bala pushed a commit that referenced this pull request Nov 6, 2026
Branch: `feat/message-history-async` (fork `Yarmoluk/redis-vl-python`),
rebased onto `main` at `506b961` (includes #792 and #785).
Commits: `82f91e4` feat, `93657d7` docs, `c1b5d75` test. Label:
`auto:minor`.

## Summary

Follow-up to #784, which gave `SemanticRouter` `aroute` / `aroute_many`.
`MessageHistory` had the same gap: an agent running on asyncio blocked
its event loop on every history read or write. This PR adds the async
twins -- `aget_recent`, `astore`, `aadd_messages`, `aadd_message`,
`acount`, `adrop`, `aclear`, `adisconnect` and an `aindex` property --
using the same pattern #784 landed:

- The `AsyncSearchIndex` is built lazily on the first async call, from
the sync index's connection settings. A history built from `redis_url`
gets its own async connection from that URL and `connection_kwargs`; a
history given a sync `redis_client` gets an async client converted from
that client's connection settings by
`RedisConnectionFactory.sync_to_async_redis`, owned by the async index,
so `adisconnect` closes only the converted client and the caller's sync
client is never touched or rebuilt.
- Every async method goes through the `AsyncSearchIndex`; none reaches
into a Redis client directly (`adrop` deletes with
`aindex.drop_keys(aindex.key(id))`).
- Query building (`_recent_query`, `_count_query`) and message
construction (`_chat_messages`, `_exchange`) move into helpers shared by
the sync and async paths, so the two cannot drift.
- The docstrings say what is and is not supported, in the router's
wording: since #792 a sync client over TLS or a unix socket converts;
Sentinel and Redis Cluster clients do not (pass `redis_url` instead);
the async client is bound to the loop that made the first call, so call
`adisconnect` before switching loops.

No sync behaviour changes. `SemanticMessageHistory` is not covered here
(see Notes).

## Reviewer Brief

**Start here:** `redisvl/extensions/message_history/message_history.py`,
`_get_async_index()` (line 418). Every async method goes through it, and
it decides whether the async index connects from the URL the sync index
owns or from a client converted from the caller's sync client; it is the
same code as `SemanticRouter._get_async_index`.

**Where it could fail**
- `message_history.py:432`: a caller-supplied Sentinel or Cluster
client, or one with a sync-only setting such as
`ssl_validate_ocsp=True`, raises `ValueError` from `sync_to_async_redis`
at the first async call; the `aget_recent` docstring names Sentinel and
Cluster but, like the router's, not the sync-only-setting case.
- `message_history.py:154`: `adrop` removes the key with `UNLINK` (via
`drop_keys`) where sync `drop` uses `DEL`; both are `@keyspace @write`,
but an ACL that grants `+del` by name and not `+unlink` would allow
`drop` and deny `adrop`
([UNLINK](https://redis.io/docs/latest/commands/unlink/),
[DEL](https://redis.io/docs/latest/commands/del/)).
- `message_history.py:449`: the async client belongs to the loop that
made the first call; a second `asyncio.run()` on the same history
without `adisconnect` in between fails, the same contract
`SemanticRouter` has shipped since #784.

**Not verified:** Redis Cluster and Sentinel (no cluster tests run
locally or in CI); the TLS and unix-socket conversion paths were not
exercised end to end through `MessageHistory` (they are covered by
#792's own tests); `connection_kwargs` and `lib_name` forwarding on the
URL path is not asserted by a test.

**Beyond CI:** each of these mutations makes
`test_async_message_history_matches_sync` fail: `astore` ignoring
`session_tag`, `adrop` going through the sync client, `adrop(id)`
ignoring `id`, the async index rebuilt per call, and `adisconnect` not
closing the converted client. The two new integration tests pass locally
against `redis:8.4` via testcontainers.

**Since the last brief:** rebased onto `506b961` (#792 now converts TLS
and unix-socket clients, so the caveat narrows to Sentinel and Cluster);
`adrop` deletes through `aindex.drop_keys` instead of the async client;
the sync-equivalence test now blocks the sync client's `execute_command`
and `pipeline`, seeds a second session with `astore`, and pins
`tool_call_id`/metadata, `adrop(id)`, async-index reuse and
`adisconnect` closing the converted pool.

<sub>Reviewed `c1b5d75` · Claude Code (claude-opus-5-5) · lenses:
correctness, security, design, tests, docs, Redis behaviour</sub>

## Test plan

Run on the rebased branch at `c1b5d75`, macOS, `uv`, Docker running
(testcontainers starts `redis:8.4`):

```bash
make format
# All done! ✨ 🍰 ✨
# 272 files left unchanged.

rm -rf .mypy_cache && make check-types
# Success: no issues found in 122 source files

uv run pytest tests/integration/test_message_history.py -k async_message_history -q -p no:cacheprovider
# 2 passed, 33 deselected

uv run pytest -n auto tests/integration/test_message_history.py tests/unit -q -p no:cacheprovider
# 3 failed, 1500 passed, 9 skipped, 53 errors
```

The 3 failures and 53 errors are the identical set on `main` at
`506b961`: every one is `ModuleNotFoundError: sentence_transformers`
(the HuggingFace vectorizer is not installed locally), in
`SemanticMessageHistory` tests this PR does not touch.

`test_async_message_history_matches_sync` takes its expected values from
the sync methods, then patches `execute_command` and `pipeline` on the
sync index's Redis client to raise, so any async method that made a
blocking call would fail it. `test_async_message_history_with_redis_url`
checks that a URL-built history builds its async index from that URL
rather than converting the client.

## Notes

- `SemanticMessageHistory` gets no async methods in this PR. Its
`get_relevant` embeds through the vectorizer, which brings in the
`aembed` caveats #784 documented for the router; it is a separate change
with its own brief.
- No `adelete`: dropping the index is setup/teardown rather than a
per-turn call, and #784 drew the same line for the router. No async form
of the `messages` property either, since a property cannot be awaited;
`aget_recent` with a large `top_k` is the async read.
- Loop binding follows #784's `adisconnect` contract rather than the
rebuild-on-new-loop #785 added for Cohere's async client. The async
index owns a connection pool, and rebuilding it silently on a loop
change would leak the previous pool; an explicit `adisconnect` keeps the
two extensions consistent and the close visible.
- `_get_async_index`, `aindex` and `adisconnect` match
`SemanticRouter`'s line for line; they are kept as two copies rather
than a shared helper, since extracting one would mean changing the
merged router code in this PR.
- `count`/`acount` keep the sync method's unannotated signature rather
than adding `-> int` to one twin only.

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> New async Redis connection lifecycle (lazy index, loop binding, client
conversion) could misbehave for Sentinel/Cluster/unix-socket setups;
sync API and data paths are unchanged.
> 
> **Overview**
> Adds **async counterparts** to `MessageHistory` read/write/clear/drop
operations (`aget_recent`, `astore`, `aadd_messages`, `aadd_message`,
`acount`, `adrop`, `aclear`, plus `adisconnect` and an `aindex`
property) so asyncio agents can use history without blocking the event
loop.
> 
> Async work goes through a **lazily created** `AsyncSearchIndex` that
reuses the sync index’s schema and connection settings: URL-owned
clients open a separate async connection from `redis_url`;
caller-supplied sync clients get a derived async client via
`RedisConnectionFactory.sync_to_async_redis`, closed only by
`adisconnect`. Docstrings document loop binding and that
Sentinel/Cluster sync clients need `redis_url` instead.
> 
> Sync behavior is preserved by **extracting shared helpers**
(`_recent_query`, `_recent_result`, `_count_query`, `_exchange`,
`_chat_messages`) so sync and async paths share query and message
construction. Two integration tests assert parity with sync (including a
monkeypatch that fails if async calls hit the sync Redis client) and
cover the `redis_url` async-index path.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
c1b5d75. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto:minor Increment the minor version when merged released This issue/pull request has been released.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants