Relevant issues

Resolves #29121
Docs: BerriAI/litellm-docs#377

Linear ticket

Resolves LIT-3760
Addresses LIT-3860 (semantic cache scope fix; redis/qdrant rollout tracked there)

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Run against a live proxy talking to a real Valkey instance with the valkey-search vector module and real OpenAI APIs.

Valkey (valkey-search bundled):

docker run -d --name litfix-valkey -p 6385:6379 valkey/valkey-bundle:8.1
docker exec litfix-valkey valkey-cli MODULE LIST   # shows the "search" module loaded

Proxy config (litellm_settings.cache_params):

litellm_settings:
  cache: true
  cache_params:
    type: valkey-semantic
    host: "127.0.0.1"
    port: 6385
    similarity_threshold: 0.8
    valkey_semantic_cache_embedding_model: text-embedding-3-small
    valkey_semantic_cache_index_name: litellm_proxy_valkey_demo

Before this change, configuring type: valkey-semantic is unrecognized; the Cache builds no backend and the proxy crashes on startup:

File ".../litellm/proxy/proxy_server.py", line 3655, in _init_cache
    litellm.cache.cache, (RedisCache, RedisClusterCache)
AttributeError: 'Cache' object has no attribute 'cache'

After this change the proxy starts, builds the vector index on valkey-search, and serves a semantic cache hit. Sending the same request twice returns the same response id; the second call is served from Valkey with no new LLM call:

$ curl -s localhost:4000/v1/chat/completions -H "Authorization: Bearer $KEY" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"In exactly one sentence, describe the color of a clear daytime sky."}],"temperature":0}'
id: chatcmpl-DrpVbGn9QkpxyaYNVcIT3M5bqDaR7
answer: The color of a clear daytime sky is a vibrant, serene blue ...

$ curl -s localhost:4000/v1/chat/completions -H "Authorization: Bearer $KEY" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"In exactly one sentence, describe the color of a clear daytime sky."}],"temperature":0}'
id: chatcmpl-DrpVbGn9QkpxyaYNVcIT3M5bqDaR7
answer: The color of a clear daytime sky is a vibrant, serene blue ...

The data landed in valkey-search under the scoped index:

$ docker exec litfix-valkey valkey-cli FT._LIST
litellm_proxy_valkey_demo
$ docker exec litfix-valkey valkey-cli KEYS 'litellm_proxy_valkey_demo:*'
litellm_proxy_valkey_demo:99c85b81...:b5a3f3b5-...

E2E Testing in K8s

Elasticache

Proxy Config

Live semantic scope fix on the managed ElastiCache endpoint, deployed image litellm-pr-30675-c0198c0 (Valkey 9.0 over rediss://): a long prompt, then the same prompt with one word changed (meticulous -> careful). The reworded request returns the same response id in a fraction of the time, and the valkey-search index num_docs is unchanged across it, so it is a cache hit rather than a new store. Pre-fix this same reworded request missed with a new id, full latency, and a new document.

$ curl -s localhost:4000/v1/chat/completions -H "Authorization: Bearer $KEY" \
  -d '{"model":"gpt-5.4-mini","messages":[{"role":"user","content":"You are a meticulous technical writer. ... when running it in production."}]}'
"id":"chatcmpl-DscHT8ujqQjDEiLFepg6Jpg41jK3p"
HTTP 200  total=3.369638s

$ curl -s localhost:4000/v1/chat/completions -H "Authorization: Bearer $KEY" \
  -d '{"model":"gpt-5.4-mini","messages":[{"role":"user","content":"You are a careful technical writer. ... when running it in production."}]}'
"id":"chatcmpl-DscHT8ujqQjDEiLFepg6Jpg41jK3p"
HTTP 200  total=0.387151s

# FT.INFO litellm_proxy_valkey_demo -> num_docs 6 both before and after the reworded request

Semantic scope fix: reworded prompt now hits on all three backends

The scope fix lives in get_cache_key(), which every semantic backend filters on, so the same change fixes all three. It was verified end to end against each backend's real engine and real OpenAI, by sending a long prompt then the same prompt with one word changed (meticulous -> careful). In every case the reworded request returns the same response id in a fraction of the time with no new entry stored, a real cache hit. Pre-fix each of these produced a new id, full latency, and a new stored entry.

Representative local run (valkey-semantic against valkey/valkey-bundle); the managed ElastiCache run is in the E2E Testing in K8s section above:

$ curl -s localhost:4001/v1/chat/completions -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"You are a meticulous technical writer. ... when running it in production."}]}'
"id":"chatcmpl-Dsc22r6wv8srgL4Cnybz9jQwj9GMp"
HTTP 200  total=3.454095s

$ curl -s localhost:4001/v1/chat/completions -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"You are a careful technical writer. ... when running it in production."}]}'
"id":"chatcmpl-Dsc22r6wv8srgL4Cnybz9jQwj9GMp"
HTTP 200  total=0.414303s

Cross-tenant isolation (two keys do not share a bucket) is covered by test_semantic_cache_key_isolates_tenants.

Type

🆕 New Feature

Changes

Adds a valkey-semantic cache type for semantic prompt caching on Valkey clusters running the valkey-search module, which is what AWS ElastiCache for Valkey exposes.

The existing redis-semantic backend cannot be pointed at valkey-search. It is built on RedisVL, which gates the connection on a RediSearch module version that valkey-search does not report, and whose SemanticCache index declares the prompt as a TEXT field that valkey-search does not implement. Both were confirmed against valkey/valkey-bundle: RedisVL raises RedisModuleVersionError on connect, and forcing past it fails index creation with Invalid field type for field 'prompt': Unknown argument 'TEXT'.

ValkeySemanticCache talks to valkey-search directly over redis-py instead. It creates the index from the field types valkey-search supports, a TAG field for caller scope and a VECTOR field (HNSW, cosine) for the prompt embedding, stores each entry as a hash, and retrieves with a KNN query filtered to the caller's scope. Prompt extraction, embedding generation, and cached-response parsing are inherited from RedisSemanticCache. The scope tag is a sha256 of the litellm cache key so it is always a valid, exact-match TAG token regardless of the key contents. The redis dependency is imported lazily in the cache dispatch, so import litellm still works on a base install without redis.

Connections resolve from VALKEY_HOST / VALKEY_PORT / VALKEY_PASSWORD, falling back to REDIS_*, and passwordless clusters are supported for IAM or no-auth setups. A full redis_url (including rediss:// for TLS) can be passed for ElastiCache encryption in transit.

There is no semantic-caching docs page in this repo to extend, so the configuration example lives in this PR for now.

Semantic cache scope fix

This PR also corrects how the semantic cache scope key is built, which previously made semantic caching behave like an exact-match cache. get_cache_key() hashed messages / prompt / input into the litellm_cache_key that all three semantic backends (redis-semantic, qdrant-semantic, valkey-semantic) filter their KNN search on, so a reworded prompt landed in a different bucket and never matched even when vector similarity was well above the threshold. For semantic cache types the prompt-bearing params are now excluded from the scope key and the server-set tenant identity (user_api_key, team, org) is appended instead, so embedding matching works within a tenant while cache entries stay scoped to the authenticated key / team / org. Because the change lives in get_cache_key() and the backends already filter on that key, it fixes redis-semantic and qdrant-semantic as well. Exact caches keep the prompt in their key and are unchanged.

Production configuration (TLS, topology)

Connections support TLS for managed endpoints: pass ssl: true alongside host and port to build a rediss:// URL, or pass a full redis_url. On AWS, vector search requires a node-based ElastiCache for Valkey 8.2+ cluster; a cluster-mode-disabled node group is the supported and recommended target, and a primary with read replicas is fine since only horizontal sharding is unsupported. ElastiCache Serverless does not support vector search. Multi-shard (cluster-mode-enabled) endpoints are rejected with a clear error, since the async client cannot route the FT.* search commands across shards.