Repository navigation
Make semantic cache observability easier out of the box #684
Description
Problem
RedisVL semantic caching can reduce LLM latency and cost, but customers do not currently have an easy, built-in way to understand how effectively a cache is working.
RedisVL exposes useful search index statistics, but those do not answer common semantic cache questions such as:
- What is the cache hit ratio?
- How many checks result in hits, misses, or errors?
- How long do cache lookups take?
- Is cache effectiveness improving or regressing over time?
Today, customers need to wrap cache calls with their own instrumentation. This adds setup work, leads to inconsistent metric definitions, and makes it harder to demonstrate the value of semantic caching. Redis server keyspace hit and miss statistics are not a substitute because they do not represent whether a semantic search produced an acceptable cache match.
Customer value
A more plug-and-play observability experience would help customers:
- Quantify latency and LLM cost savings from semantic caching.
- Tune thresholds, TTLs, and cache strategy using real workload data.
- Detect drops in cache effectiveness or increases in lookup errors and latency.
- Compare behavior across environments and releases.
- Connect semantic cache signals to the monitoring tools they already use with minimal application code.
Desired experience
Customers should be able to opt into semantic cache observability with very little setup and get clearly defined, documented signals for each cache. The experience should be consistent across synchronous and asynchronous usage and, where practical, across RedisVL semantic cache backends.
Useful signals may include cache checks, hits, misses, hit ratio, lookup latency, errors, stores, cache size, and match-distance information. This list is intended to describe customer needs rather than prescribe a particular API, metrics library, or implementation.
Observability should add minimal overhead, avoid exposing prompt or response content, and have metric semantics that customers can rely on in dashboards and alerts.
Activity
vishal-bala commentedon Nov 3, 2026
The OpenTelemetry stack in #755, #776 and #777 covers most of this issue for
SemanticCache, but not forLangCacheSemanticCache, and it doesn't report cache size. #754, the KNN lookup it builds on, has already merged.Opting in takes
pip install "redisvl[otel]"plus an OpenTelemetry SDK configured in your application. Without an SDK, RedisVL emits nothing, and the instrumentation adds about 11 µs to a lookup that embeds (measured on an Apple M4 Pro).REDISVL_OTEL_DISABLED=trueturns it off entirely.redisvl.semantic_cache.operation.durationfor everycheck,store,updateanddrop, sync or async. Each point carries the cache name, the operation, and the result (hitormiss) orerror.type. Its count is the number of operations, so no separate counter is needed.gen_ai.client.operation.durationtimes each embedding call.redisvl.semantic_cache.distance, the nearest candidate's distance on every lookup, hit or miss. Read againstdistance_threshold, it shows how close misses come, which is what threshold tuning needs.operation=store.OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENTisSPAN_ONLYorSPAN_AND_EVENT, and responses never do.gen_ai.*names follow the OpenTelemetry GenAI conventions, which are still in Development status upstream.What remains open:
LangCacheSemanticCachehas no instrumentation, so the "across backends" part of the request isn't met.num_docsfromFT.INFO) give it today, but not as a metric alongside the rest.I'd keep this issue open for those three once the stack merges, or split them into their own issues if that's easier to track.