fix(tests/vcr): make Redis cassette cache replay deterministically (zero VCR misses on consecutive runs) by mateo-berri · Pull Request #28826 · BerriAI/litellm

mateo-berri · 2026-05-26T02:34:34Z

What & why

The Redis-backed VCR cache (see tests/llm_translation/Readme.md) is meant to make every CI run after the first replay HTTP from Redis with no live provider calls within the 24h cassette TTL. In practice a set of cassettes re-recorded live on every run, so the 2nd/3rd run of a day still made (and depended on) live calls. There were four layers to this:

Systematic misses — a dominant import-time network fetch recorded into cassettes, plus volatile request tokens, two genuinely un-cacheable tests, and a CI-only un-provisioned model.
A read-only TTL-expiry floor — load_cassette read with a plain Redis GET, which does not touch the key's TTL. A cassette that is only ever replayed (HIT/NOOP, never re-recorded) therefore expired exactly 24h after its last write, regardless of how often it was read — whichever run crossed that boundary re-recorded it live. Fix: slide the expiry forward on every successful read, so any actively-used cassette never expires.
A telemetry-episode-dropped-on-load bug (the deepest one) — a previous change suppressed recording incidental telemetry leaked from globally-enabled observability callbacks (a Langfuse async flush landing in an unrelated test's VCR window) by returning None from before_record_request. But vcrpy's Cassette._load() replays every stored interaction through Cassette.append(), which also runs before_record_request — so the suppressor was deleting already-recorded telemetry episodes on read whenever a non-telemetry-named test (or the alphabetically-first test in a worker, whose current-nodeid is still empty) loaded them, forcing an endless live re-record. Confirmed by replaying Cassette._load() against the real Redis cassette: empty/non-telemetry nodeid → 0 episodes survive; with the guard → 1 survives. Fix: a thread-local guard set around Cassette._load makes the suppressor inert during load, so it only ever stops new incidental recordings and never filters the cassette on read.
A rotating telemetry async-flush escape — even with (3), the suppressor only fired for non-telemetry-named tests. litellm's observability loggers flush on a background thread, so a Langfuse-ingestion POST scheduled by one telemetry test fires mid-way through a later telemetry-named test (after that test's own httpx mock has exited) and is recorded as a phantom episode — a MISS:RECORDED/PARTIAL that lands on a different telemetry test every run (observed on test_langfuse_logging_completion_with_tags one run, …_with_vertex_llm_response the next). Telemetry export POSTs are fire-and-forget; no test asserts on a recorded export response except the pass-through proxy test (which forwards a client POST to Langfuse ingestion and replays its 207). Fix: _should_drop_telemetry_record now drops incidental export POSTs for every test except that one. Dropping returns None (live fire-and-forget, never stored), so it can only turn a phantom miss into a harmless live call, never the reverse; recorded read-back GETs that telemetry tests assert on are matched by method and left untouched.

This PR makes the cache replay deterministically and proves it with 3 consecutive CI runs: run 1 seeds, runs 2 and 3 show zero VCR MISS verdicts (RECORDED / OVERFLOW / NOT_PERSISTED / PARTIAL) across all VCR-enabled jobs.

All matcher changes live in the shared tests/_vcr_conftest_common.py plumbing and are applied symmetrically at compare-time (to both the incoming request and the stored episode), so they can only change which recorded episode is selected — they can never mask a response-level discrepancy.

Verification

3 consecutive runs on the final tip (380e9361):

Run	Pipeline	VCR misses (all jobs)
1 (seed)	79159	0
2	79161	0
3	79165	0

Logs pulled via the CircleCI API; misses counted from the per-job VCR CACHE CLASSIFICATION SUMMARY and an independent line-level scan of every [VCR MISS:*] / [VCR PARTIAL] verdict.

Summary of every issue addressed

Test / cassette group	Fixed / Ignored	How it was fixed (or extremely defensible reason it was ignored)
Telemetry episode dropped on cassette load — `test_aaapass_through_endpoint_pass_through_keys_langfuse`, `test_async_otel_callback`, `test_add_custom_logger_callback_to_specific_event`, and any test loading a cassette that contains a Langfuse-ingestion episode	✓	vcrpy's `Cassette._load()` replays each stored interaction through `Cassette.append()` → `before_record_request`; a `None` return there silently drops the episode. The telemetry-leak suppressor (`_should_drop_telemetry_record`) returns `None` for telemetry requests, so a non-telemetry-named test (or the alphabetically-first test in a worker, whose nodeid is still empty) deleted the stored Langfuse episode on read, forcing a live re-record → MISS:RECORDED on a cassette that was demonstrably present in Redis. Root-caused with a truncation-proof Redis-list diagnostic (showed `GET` returned the cassette bytes fine — `get=1440B exists=1` — so the loss was downstream in vcrpy) and reproduced offline against the real cassette (`_load()` → 0 episodes survive under an empty nodeid, 1 with the guard). Fix: a thread-local guard around `Cassette._load` makes the suppressor inert during load — it only stops new incidental recordings, never filters the cassette on read. Unit-tested.
Rotating telemetry async-flush escape — `test_langfuse_logging_completion_with_tags`, `test_langfuse_logging_completion_with_vertex_llm_response`, and (run-dependent) any other test in `test_langfuse_e2e_test.py` / the logging suite	✓	litellm's observability loggers flush on a background thread, so a Langfuse-ingestion POST scheduled by one telemetry test fires mid-way through a later telemetry-named test — after that test's own `with patch("httpx.Client.post")` block has exited — and is recorded by VCR as a phantom episode. Because the landing test is telemetry-named, fix (3)'s suppressor exempted it, so it surfaced as a non-deterministic MISS:RECORDED / PARTIAL that rotated onto a different langfuse-e2e test each run (caught on `…_with_tags` in one run, `…_with_vertex_llm_response` in the next). These tests mock the export client and assert on the mock, never on a recorded export response — the only test that asserts on a recorded Langfuse-ingestion response is the pass-through proxy test (deterministic body, replays 207). Fix: `_should_drop_telemetry_record` now drops incidental telemetry export POSTs for every test except the pass-through one; read-back GETs (which `test_alangfuse`-style tests assert on) are matched by method and kept. Dropping returns `None` → live fire-and-forget → classified `UNMARKED:LIVE_CALL`/`NOOP`, so it can only remove a phantom miss, never add one. Unit-tested across export-POST/read-back-GET/non-telemetry/pass-through branches.
Read-only cassettes expiring (TTL not refreshed on read)	✓	`tests/_vcr_redis_persister.py::load_cassette` read with a plain Redis `GET`, which does not refresh the key's TTL. A cassette only ever replayed (HIT/NOOP, never re-recorded) expired exactly 24h after its last write; whichever run crossed that boundary re-recorded it live → spurious MISS:RECORDED. Confirmed by reading Redis TTLs directly and watching an external monitor catch the TTL refresh on load. Fix: slide the expiry forward on every successful load (best-effort `EXPIRE`), so any cassette used at least once per TTL window never expires. Ruled out eviction (`evicted_keys=0`, uncapped `maxmemory`) and real load failures (only a unit test's simulated outages). New unit tests cover the refresh and its best-effort failure path.
Skip-tests recording the model-cost-map fetch — ~39 tests across `llm_responses_api_testing` (openai/azure/anthropic/google delete·get·cancel·context-management skips), `ocr_tests`, and others that skip after one recorded episode	✓	`litellm.__init__` calls `get_model_cost_map()` which does a live `httpx.get` to `raw.githubusercontent.com` unless `LITELLM_LOCAL_MODEL_COST_MAP=True`. Several VCR conftests `importlib.reload(litellm)` in an autouse fixture, so that fetch was recorded as an episode. For a test that then skips it was the only episode, the persister refuses to save skipped tests, so the cassette never seeded and the test re-recorded live every run → MISS:NOT_PERSISTED. Pinned `LITELLM_LOCAL_MODEL_COST_MAP=True` in the shared harness (the established idiom — see `tests/test_litellm/test_cost_calculator.py`). Verified: total misses dropped 77 → 4; these became `NOOP`.
Volatile-token cassettes at the 50-episode cap — Vertex / Gemini OAuth (`ya29.*` access tokens, `oauth2.googleapis.com/token` signed-JWT bodies), Langfuse / OTel / Arize telemetry (fresh span/trace UUIDs, ISO timestamps, durations, build SHA, `trace_id` query param), Bedrock / Azure `time.time()` & `uuid4()` cache-busters	✓	A rotating token in the per-request fingerprint or body meant no stored episode ever matched, so cassettes grew until they hit `MAX_EPISODES_PER_CASSETTE` (50) and overflowed every run. Compare-time symmetric matcher tolerance: skip the API-key fingerprint for Google hosts (the GCP project is already in the matched URL path) and collapse `ya29.`; match credential-exchange / observability hosts on method+scheme+host+port+path (`tolerant_query` matcher + body skip); canonicalize epoch/ISO timestamps and UUIDs in `_safe_body_matcher`. Supporting evidence*: 69 Vertex + 84 Gemini + 11 Langfuse cassettes sit exactly at the 50-cap and now `HIT` cleanly across runs instead of overflowing.
`test_router_text_completion_client` (`local_testing/test_router.py`)	✗	Marked VCR-incompatible. It fires 300 identical requests to verify the async OpenAI client is reused (its own comment: "fails when we create a new client per request"). vcrpy patches the HTTP transport, so replay never opens real connections and can't exercise the pool the test validates; recording instead stores ~300 near-identical episodes → MISS:OVERFLOW every run. The endpoint is a free mock, so the live calls cost nothing. Now `UNMARKED` (no longer a cache miss). Verified: OVERFLOW gone in run 1.
`test_max_parallel_requests_rpm_rate_limiting`	✓	Prompt embedded `random.random()*100`, producing 48 distinct episodes (overflow risk). Replaced with a fixed value — the content is irrelevant to a rate-limit test.
Vertex MaaS Mistral OCR — `test_ocr_vertex_ai.py::TestVertexAIMistralOCR::{test_basic_ocr_with_url[True], test_basic_ocr_with_url[False], test_ocr_response_structure}`	✗	Marked VCR-incompatible. `vertex_ai/mistral-ocr-2505` is a Model-as-a-Service partner model that must be enabled in the GCP project's Model Garden; it is not provisioned in the CI project (`litellm-ci-cd`), confirmed by Redis (no cassette has ever existed for these tests, vs. their sibling direct-Mistral and Azure OCR cassettes which `HIT`). The live `:rawPredict` fails every run and `BaseOCRTest`'s own except-blocks skip, so the recorded attempt never persists → MISS:NOT_PERSISTED every run, forever. Marking incompatible reports them honestly as `UNMARKED:LIVE_CALL` (a live call that isn't cached) rather than a phantom cache miss; behaviour is unchanged (they still run and still skip on the provider error). Remove the opt-out once the model is provisioned.
AWS Secrets Manager lifecycle — `test_write_and_read_simple_secret`, `test_write_and_read_json_secret`, `test_read_nonexistent_secret`, `test_primary_secret_functionality`, `test_write_secret_with_description_and_tags`	✗	Marked VCR-incompatible. Each run creates a uniquely-named AWS secret (a fixed name can't be reused under AWS's ≥7-day deletion-recovery window) and asserts the API response echoes that unique name — deterministic replay returns the previously-recorded name, so the round-trip can't be reproduced offline (they were re-recording 48 episodes → overflow). The config-parsing tests in the same file stay VCR-cached.

Observability fix (enabler)

CircleCI truncates step output to the last ~400 KB. The VCR diagnostic dump (one block per episode comparison, even on an eventual HIT) exceeded that and pushed the VCR CACHE CLASSIFICATION SUMMARY out of the retrievable window, making misses impossible to read in CI. The dump now dedupes identical blocks and caps its size so the summary always survives.

Out of scope — pre-existing failures (not VCR misses, not caused by this PR)

These jobs are red for reasons unrelated to VCR determinism and carry zero VCR misses; documented rather than masked:

batches_testing, e2e_openai_endpoints: AWS Bedrock batch "account is not authorized" for model-invocation-jobs — a consequence of the AWS account migration in chore(tests): migrate Bedrock CI to AWS account 941277531214 #28728.
local_testing_part1: test_vertex_ai_llama_tool_calling raises AuthenticationError against a Vertex MaaS Llama model (NOOP — no recordable traffic).
realtime_translation_testing: test_xai_realtime websocket connect (realtime traffic isn't VCR-intercepted; transient — green in run 1).
Pre-existing VCR opt-out, runs live (flaky): pass_through_unit_testing::test_prompt_caching_streaming_second_call_returns_cache_read and …returns_cache_read_tokens_on_second_call are deliberately marked VCR-incompatible in tests/pass_through_unit_tests/conftest.py (commit 3390fc19722, 2026-05-13 — not part of this PR): they assert that a second identical Bedrock call observes the upstream prompt-cache warm-up (cache_read > 0), which deterministic replay can't model. So VCR is switched off for them and they run live every run (no cassette by design); their verdict is UNMARKED:NO_TRAFFIC (incompatible), which is not a VCR miss. One live call hit a Bedrock 429 in the seed run and passed in runs 2/3. litellm_router_testing::test_usage_based_routing uses mock_response (no HTTP) and flakes on shared-Redis cooldown state ("No deployments available"). Both carry zero VCR misses and are unrelated to this PR.

Appendix — full VCR opt-out inventory

For completeness, here is every test that is not VCR-cached and the mechanism that opts it out. There are five mechanisms in apply_vcr_auto_marker_to_items (priority order): disabled → already_marked → respx_conflict → respx_conflict_module / file_opt_out → incompatible. Three are deliberate, curated opt-outs declared in conftest lists (skip_nodeid_suffixes → incompatible; skip_files → file_opt_out or respx_conflict_module); respx_conflict is auto-detected per test (a @pytest.mark.respx decorator / respx_mock fixture conflicts with vcrpy's transport patch); already_marked/disabled aren't opt-outs.

Counts are the authoritative runtime classification from run #3 (workflow be96e81d, tip 380e9361), cross-checked against the conftest source.

Mechanism	Declared where	# tests (run #3)
`incompatible` (`skip_nodeid_suffixes`)	3 conftests	10 declared / 9 ran
`file_opt_out` (`skip_files`, no real respx)	3 conftests	24 ran (+2 files skipped at runtime)
`respx_conflict_module` (`skip_files`, module uses respx)	1 conftest	23
`respx_conflict` (auto-detected per-item respx)	none — automatic	4 (dynamic, suite-wide)

`incompatible` — `skip_nodeid_suffixes` (the deliberate, named opt-outs)

Test file :: test name	Way
`pass_through_unit_tests/test_anthropic_messages_prompt_caching.py::TestBedrockConversePromptCaching::test_prompt_caching_returns_cache_read_tokens_on_second_call`	`incompatible`
`…::TestBedrockConversePromptCaching::test_prompt_caching_streaming_second_call_returns_cache_read`	`incompatible`
`…::TestBedrockInvokePromptCaching::test_prompt_caching_returns_cache_read_tokens_on_second_call`	`incompatible`
`…::TestBedrockInvokePromptCaching::test_prompt_caching_streaming_second_call_returns_cache_read`	`incompatible`
`litellm_utils_tests/test_aws_secret_manager.py::test_write_and_read_simple_secret`	`incompatible`
`litellm_utils_tests/test_aws_secret_manager.py::test_write_and_read_json_secret`	`incompatible`
`litellm_utils_tests/test_aws_secret_manager.py::test_read_nonexistent_secret`	`incompatible`
`litellm_utils_tests/test_aws_secret_manager.py::test_primary_secret_functionality`	`incompatible`
`litellm_utils_tests/test_aws_secret_manager.py::test_write_secret_with_description_and_tags`	`incompatible`
`local_testing/test_router.py::test_router_text_completion_client`	`incompatible` ¹

`file_opt_out` — whole file in `skip_files`, module has no live respx wiring

Test file (parametrized items)	Way
`litellm_utils_tests/test_litellm_overhead.py` (10 items: `test_litellm_overhead_stream[...]`, `…_non_streaming[...]`, `…_cache_hit`)	`file_opt_out`
`logging_callback_tests/test_amazing_s3_logs.py` (6 items: `test_basic_s3_logging[...]`×4, `test_basic_s3_v2_logging[****]`, `test_basic_s3_v2_logging_failure`)	`file_opt_out`
`logging_callback_tests/test_assemble_streaming_responses.py` (8 items: `test_assemble_complete_response_from_streaming_chunks_1..4[False/****]`)	`file_opt_out` ²
`local_testing/test_assistants.py` (whole file)	`file_opt_out` ¹
`local_testing/test_router_caching.py` (whole file)	`file_opt_out` ¹

`respx_conflict_module` — file in `skip_files`, module does use respx

Test file (parametrized items)	Way
`logging_callback_tests/test_langfuse_unit_tests.py` (23 items: all `test_*` in the file)	`respx_conflict_module`

`respx_conflict` — auto-detected per test (not in any conftest list)

Test file :: test name	Way
`local_testing/test_azure_openai.py::test_aaaaazure_tenant_id_auth`	`respx_conflict`
`local_testing/test_amazing_vertex_completion.py::test_vertexai_embedding_finetuned`	`respx_conflict`
`local_testing/test_amazing_vertex_completion.py::test_vertexai_model_garden_model_completion[None]`	`respx_conflict`
`local_testing/test_amazing_vertex_completion.py::test_vertexai_model_garden_model_completion[3]`	`respx_conflict`

(This category is dynamic — any test suite-wide that wires up respx is auto-skipped; the 4 above are what surfaced in run #3's jobs.)

Footnotes:

¹ The 3 local_testing opt-outs are declared in local_testing/conftest.py but were skipped at runtime in run #3 (no credentials / test-split timing), so they never reached the VCR outcome recorder and produced no runtime verdict — opted out by configuration, but uncounted. Hence "10 declared / 9 ran" for incompatible.

² test_assemble_streaming_responses.py is listed in _RESPX_CONFLICTING_FILES (intended as a respx conflict) but the AST visitor finds no real respx wiring, so it classifies as file_opt_out, not respx_conflict_module — i.e. a stale/dead respx import. Cleanup candidate (drop it from _RESPX_CONFLICTING_FILES); out of scope for this PR.

Note

Medium Risk
Changes only test/VCR infrastructure (matchers, Redis TTL, recording filters), but broad compare-time relaxations and telemetry dropping could mask real request mismatches if misclassified; production code is unaffected.

Overview
Makes the Redis-backed VCR test cache replay reliably across CI runs by fixing matcher drift, cassette lifecycle bugs, telemetry leakage, and a handful of tests that can never be cached offline.

Shared harness (tests/_vcr_conftest_common.py) pins LITELLM_LOCAL_MODEL_COST_MAP so importlib.reload(litellm) does not record a live GitHub model-cost fetch. Compare-time matchers now tolerate rotating Google OAuth (ya29.*), OAuth token bodies, telemetry hosts (skip body/query; drop incidental export POSTs except pass-through Langfuse), and UUID/timestamp cache-busters in bodies. A Cassette._load thread-local guard stops telemetry drop logic from deleting stored episodes when cassettes load. CI diagnostic logs are deduplicated and capped so the VCR classification summary stays visible.

Redis persister refreshes cassette TTL on every successful load so replay-only keys do not expire 24h after last write.

Conftest opt-outs: AWS Secrets Manager unique-name tests, router client-pool concurrency test, Vertex MaaS Mistral OCR (unprovisioned in CI). test_max_parallel_requests uses a fixed prompt instead of random.random().

Tests cover TTL refresh, matchers, telemetry suppression, load guard, and diagnostic emission.

^{Reviewed by Cursor Bugbot for commit 2348334. Bugbot is set up for automated code reviews on this repo. Configure here.}

…s runs - Pin LITELLM_LOCAL_MODEL_COST_MAP=True in the shared VCR harness so the per-test importlib.reload(litellm) no longer fetches the model cost map from raw.githubusercontent.com. That live fetch was being recorded into cassettes; for tests that subsequently skip it was the only recorded episode, so the persister refused to save it (skipped tests don't persist) and the test re-recorded it live every run (MISS:NOT_PERSISTED). - Compare-time symmetric matcher tolerance for Google OAuth (ya29.*) tokens, observability/telemetry payloads, credential-exchange bodies, and volatile UUID/timestamp tokens, so existing cassettes select a recorded episode instead of growing past the 50-episode cap and re-recording live. - Don't record fire-and-forget telemetry (langfuse/arize/otel/...) into non-telemetry tests' cassettes. Several modules set litellm.success_callback at import time, so observability logging is globally enabled and an async flush from the background logging worker lands in an unrelated test's VCR window, saved as a spurious MISS:RECORDED (observed: a Langfuse batch from another completion landing on test_lowest_latency_routing_buffer). Such a request now passes through live (telemetry hosts aren't real-spend hosts); tests that actually assert on telemetry keep recording it. - Dedupe + cap the VCR diagnostic dump so the classification summary survives CircleCI's ~400KB step-output truncation. - Stabilize a non-deterministic rate-limit test body; mark AWS Secrets Manager lifecycle tests VCR-incompatible (uniquely-named secrets can't be replayed). - Mark test_router_text_completion_client VCR-incompatible: it fires 300 identical requests to verify async-client reuse, but vcrpy patches the HTTP transport so replay never exercises the real connection pool the test validates, and recording 300 near-identical episodes overflows the 50-episode cap (MISS:OVERFLOW every run). It hits a free mock endpoint. - Mark the Vertex AI MaaS Mistral OCR tests (vertex_ai/mistral-ocr-2505) VCR-incompatible: the MaaS model is not provisioned in the CI GCP project, so the live :rawPredict call fails and the test skips every run, leaving no cassette to record (MISS:NOT_PERSISTED every run). Sibling direct-Mistral and Azure OCR tests are unaffected and still replay from cache.

…n't expire The Redis VCR persister loaded cassettes with a plain GET, which does not touch the key's TTL. A cassette that is only ever replayed (HIT/NOOP, never re-recorded) therefore expired exactly 24h after its last *write*, no matter how often it was read. Whichever CI run happened to cross that boundary re-recorded the cassette live and surfaced a spurious VCR MISS on otherwise deterministic cassettes — the residual per-run flakiness floor (a different random subset of read-only cassettes expiring each run). Slide the expiry forward on every successful load (best-effort EXPIRE), so any cassette used at least once per TTL window stays alive indefinitely and the 2nd/3rd run of a day replays cleanly.

…keys Under concurrent CI load, the persister's load GET was observed returning None for a cassette key that demonstrably existed on the (single, non- clustered) Redis master — an external monitor saw the key present with a healthy TTL at the same instant the in-process client read None. Because None is a valid GET result (not a RedisError), the retry-on-error client config never engaged, so the cassette re-recorded live (a phantom MISS:RECORDED); for flaky/networked tests the failed live call then triggered a pytest rerun, which is why a rotating subset of otherwise deterministic tests missed each run. On a None result, re-check EXISTS and re-read once. If the key really exists, use the recovered value and log [vcr-transient-miss-recovered] (also counted in cassette_cache_health). A genuinely absent key (a new cassette) still falls through to CassetteNotFoundError.

Logs GET/EXISTS at load time for the three cassettes that re-record every run despite being present in Redis, to capture what the in-process client sees. To be reverted before merge.

CI stdout truncates to the last ~400KB, dropping the early loaddbg lines for the alphabetically-first failing test. Push the load probe to a Redis list instead so it survives. To be reverted before merge.

…load Root cause of the residual per-run misses on present cassettes: vcrpy's Cassette._load() replays each *stored* interaction through Cassette.append(), which runs before_record_request on it — and a None return there silently drops that episode. The telemetry-leak suppressor (_should_drop_telemetry_record) returns None for telemetry requests, so when a non-telemetry-named test (or the alphabetically-first test in a worker, whose _current_test_nodeid is still empty) loaded a cassette containing a Langfuse ingestion episode, the episode was dropped on read — forcing an endless live re-record (a phantom MISS:RECORDED on a cassette that was demonstrably present in Redis). Verified by reproducing Cassette._load() against the real cassette: empty/non-telemetry nodeid -> 0 episodes survive; with the guard -> 1 survives. Fix: guard the suppressor with a thread-local set around Cassette._load (via a small idempotent monkeypatch), so the drop only ever stops *new* incidental telemetry from being recorded and never filters the existing cassette on read. Also drops the speculative GET-None recovery + its diagnostics from the previous commits: the load diagnostic showed GET returns the cassette bytes fine (get=1440B), so the persister never returned a spurious None — the loss happened later in vcrpy's append. The proven TTL-refresh-on-read fix is retained.

…ng async-flush misses litellm's observability loggers flush on a background thread, so a Langfuse ingestion POST scheduled by one telemetry test can fire mid-way through a *later* telemetry-named test (after that test's own httpx mock has exited) and be recorded by VCR as a phantom episode — a non-deterministic MISS:RECORDED / PARTIAL that rotates onto a different telemetry test from run to run. Telemetry export POSTs are fire-and-forget; no test asserts on a *recorded* export response except the pass-through proxy test (which forwards a client POST to Langfuse ingestion and replays its 207). So _should_drop_telemetry_record now drops incidental export POSTs for every test except that one. Dropping returns None (live fire-and-forget, never stored), so it can only turn a phantom miss into a harmless live call, never the reverse; recorded read-back GETs that telemetry tests assert on are matched by method and left untouched.

codecov · 2026-05-26T02:35:30Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

greptile-apps · 2026-05-26T18:09:52Z

Greptile Summary

This PR makes the Redis-backed VCR test cache replay deterministically across consecutive CI runs by fixing four distinct root causes of cache misses in the shared test harness. All changes are confined to test infrastructure; production LiteLLM runtime is unaffected.

Thread-local load guard (patch_vcrpy_cassette_load_guard) prevents vcrpy's Cassette._load from invoking the telemetry-drop suppressor on already-recorded episodes, which was silently deleting stored interactions on read and forcing endless live re-records.
TTL refresh on load (_vcr_redis_persister.py) slides the cassette expiry forward on every successful GET so replay-only keys never expire 24h after their last write.
Compare-time normalization makes volatile per-run tokens (UUIDs, timestamps, ya29.* access tokens, fresh trace_id query params) match their recorded counterparts symmetrically at compare time.
Deterministic opt-outs mark tests that genuinely cannot be replayed and fix one non-deterministic prompt.

Confidence Score: 5/5

Safe to merge — all changes are scoped to test infrastructure and CI observability; no production code paths are touched.

Every changed file is a test helper, conftest, or unit test. The four root causes are each well-documented, the fixes are symmetric (compare-time normalization can only change which recorded episode is selected, never mask a response discrepancy), and the thread-local load guard correctly scopes the telemetry suppressor to new recordings only. Previous reviewer comments are confirmed resolved. No new blocking issues were identified.

No files require special attention.

Important Files Changed

Filename	Overview
tests/_vcr_conftest_common.py	Major overhaul of VCR harness: adds volatile-token normalization, telemetry-leak suppression with load guard, tolerant matchers for Google and observability hosts, and a deduped/capped diagnostic log emitter.
tests/_vcr_redis_persister.py	Adds best-effort TTL refresh on every successful cassette load so replay-only keys never silently expire 24h after the last write.
tests/llm_translation/test_vcr_conftest_common_banner.py	New unit tests cover the diagnostic-log dedup/cap, telemetry drop suppressor, and load-guard patch idempotency; previous assertions remain intact.
tests/llm_translation/test_vcr_redis_persister.py	Adds regression tests for TTL refresh on load and the failure-is-non-fatal path.
tests/test_litellm/test_vcr_safe_body_matcher.py	Extends unit tests to cover Google OAuth ya29 fingerprint collapsing, volatile-token normalization, credential-exchange body skipping, and the tolerant-query matcher.

_{Reviews (2): Last reviewed commit: "fix(tests/vcr): restore assertion in tes..." | Re-trigger Greptile}

cursor

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Bugbot Autofix prepared a fix for the issue found in the latest run.

✅ Fixed: Displaced assertion leaves test verifying nothing
- Moved the assert reporter.output == "" back into test_banner_silent_when_vcr_disabled and removed the duplicate assertion from test_diagnostic_log_silent_when_no_dir.

_{You can send follow-ups to the cloud agent here.}

^{Reviewed by Cursor Bugbot for commit 380e936. Configure here.}

…bled The assertion that the banner is suppressed when VCR is disabled was inadvertently moved into test_diagnostic_log_silent_when_no_dir when the diagnostic-log tests were added, leaving the disabled-VCR test verifying nothing. Co-authored-by: Yassin Kortam <[email protected]>

mateo-berri · 2026-05-26T18:11:18Z

@greptileai

mateo-berri · 2026-05-26T18:15:15Z

Replying to the greptile summary findings:

Give me consistent exceptions #1 (P1) — test_banner_silent_when_vcr_disabled missing assertion: fixed on HEAD in commit 2348334 (re-adds assert reporter.output == "" at the end of the test).
Enable model / call timeouts #2 (P2) — duplicate assert reporter.output == "" in test_diagnostic_log_silent_when_no_dir: fixed in the same commit (drops the orphaned duplicate).
Guarantee format of exceptions #3 (P2) — module global read from a background thread: not actionable. Greptile itself notes "The PR's design intentionally accepts this (the worst case is a phantom miss turned into a harmless live call or vice-versa, never a response-level discrepancy)" — i.e. the suggestion is to add a comment, not to change behavior. The deliberate race acceptance is already documented in two places: (a) the module-global declaration at tests/_vcr_conftest_common.py:503-508 describes when the nodeid is set and what reads it, and (b) the load-guard block at tests/_vcr_conftest_common.py:561-572 explicitly contrasts the load-time path (where mis-classification would lose stored episodes — hence the thread-local guard) against the record-time path (where mis-classification is benign — drop returns None → live fire-and-forget, classified UNMARKED:LIVE_CALL/NOOP, never stored). The PR description's "Rotating telemetry async-flush escape" section walks through the exact failure mode greptile is describing and proves it can only convert a phantom miss into a harmless live call, never the reverse. Adding a third block of prose restating the same invariant would be duplicative without changing the code's correctness, and the repo's code-style preference is to minimize inline commentary on logic already captured nearby. Leaving as-is.

…replay (#29229) The Redis-backed VCR layer was recording and replaying the Google OAuth2/STS token-mint call. The replayed ya29.* access token is long-expired, but its recorded expires_in keeps credentials.expired False, so litellm never refreshes it and sends the stale token to a live Vertex/Gemini endpoint, which returns 401 ACCESS_TOKEN_EXPIRED. This broke live partner-model tests whose completion call is not itself cassette-backed (e.g. test_vertex_ai_llama_tool_calling). Force credential-exchange hosts to pass through live (never recorded, never replayed) by returning None from before_record_request, mirroring the existing telemetry passthrough, so a fresh token is minted each run. Regression from #28826, which added OAuth-token matcher tolerance plus TTL-refresh-on-read so a stale token episode matched and never expired.

* feat: add support for claude code goal mode for bedrock opus output config (BerriAI#28898) * feat: support goal mode for claude on bedrock * fix failing lint test * addressing greptile comments * fixing failed test * address greptile: copy output_config and warn on dropped converse format * fix(bedrock): skip redundant output_config normalization on Converse reasoning_effort path When reasoning_effort is mapped via _handle_reasoning_effort_parameter, the resulting output_config is already normalized via normalize_bedrock_opus_output_config_effort. Mark it as normalized so _prepare_request_params can skip the redundant call (and the associated get_model_info lookup) on every request. Co-authored-by: Yassin Kortam <[email protected]> * test(reasoning-effort-grid): reflect Bedrock opus-4-6 xhigh→max clamping * fix(bedrock): stop leaking output_config marker and message-content mutation * fix(bedrock): guard effort key access in normalize_bedrock_opus_output_config_effort Defensively check that 'effort' is a valid key in _BEDROCK_OUTPUT_CONFIG_EFFORT_ORDER before indexing, to prevent a KeyError if the hardcoded guard tuple ever drifts from the order dict's keys. Co-authored-by: Yassin Kortam <[email protected]> * fix(bedrock): drop dead second clause in effort normalization guard The 'effort not in _BEDROCK_OUTPUT_CONFIG_EFFORT_ORDER' check is unreachable once 'effort not in ("xhigh", "max")' has been ruled out, since both literals are present in the order dict. Keep the literal membership check and let the dict lookups below speak for themselves. * fix(bedrock): clamp output_config.effort against ceiling for any known value The early return when effort was not 'xhigh'/'max' meant a ceiling of 'low' or 'medium' would silently forward an out-of-range value. Gate on the known effort ordering instead so the ceiling comparison runs for every recognized effort. * test(grid_spec): use _CAPS_OPUS_4_7 for non-Bedrock opus-4-6 entries claude-opus-4-6 now declares supports_xhigh_reasoning_effort in the model map, so production accepts xhigh on Azure AI and Vertex AI routes. Update those grid_spec entries to match production capabilities so expected() predicts 200 for xhigh instead of 400. Co-authored-by: Yassin Kortam <[email protected]> * test(grid_spec): revert xhigh caps for non-Bedrock opus-4-6 azure_ai/claude-opus-4-6 and vertex_ai/claude-opus-4-6 do not declare supports_xhigh_reasoning_effort in model_prices_and_context_window.json. Azure AI upstream rejects xhigh with HTTP 400 ("Supported levels: high, low, max, medium"). Restore _CAPS_4_6 so the grid predicts 400 for xhigh, matching production capabilities. * fix: stop advertising xhigh effort on Opus 4.5/4.6 Only Opus 4.7 supports the xhigh reasoning effort level. Remove the supports_xhigh_reasoning_effort flag from every Opus 4.5 and Opus 4.6 entry (direct Anthropic, Bedrock, and regional variants) in both model catalog files. On the direct Anthropic path there is no effort clamp, so flagging 4.5/4.6 as xhigh-capable caused litellm to forward xhigh to a model that rejects it (and made get_model_info misreport the capability). xhigh now correctly degrades to high / raises on those models. Bedrock graceful degradation for Claude Code goal mode is unaffected: it relies solely on the bedrock_output_config_effort_ceiling clamp (4.5->high, 4.6->max, 4.7->xhigh), which runs before validation, so xhigh requests to older Bedrock Opus models are still silently lowered rather than rejected. Update effort-gating tests to reflect that 4.5/4.6 no longer accept xhigh. * fix: clamp xhigh effort on Bedrock Invoke /v1/messages instead of rejecting Claude Code "goal mode" sends output_config.effort=xhigh over the Anthropic /v1/messages API, which routes Bedrock models through AmazonAnthropicClaudeMessagesConfig. That path validated effort against the model's native capability and raised 400 for xhigh on Opus 4.6, while the chat-completions paths (Converse + Invoke) already clamp xhigh to the model's bedrock_output_config_effort_ceiling. That asymmetry broke goal mode on the exact API surface Claude Code uses. Apply the same ceiling clamp on the messages path before the shared effort gate runs, so xhigh degrades to max on Opus 4.6 (and stays xhigh on 4.7). Scoped to adaptive-thinking models and to models that declare a ceiling, so Sonnet 4.6 (no ceiling) and Opus 4.5 (budget mode) are unaffected and still reject xhigh. * fix(bedrock): preserve user output_config when applying reasoning_effort - Converse path: merge mapped effort into existing output_config via setdefault instead of overwriting it, matching the Anthropic Messages path. Prevents user-supplied output_config.format from being silently dropped when reasoning_effort is also provided. - tests: clear _get_local_model_cost_map lru_cache in the autouse fixture alongside get_bedrock_response_stream_shape to avoid stale cache leakage between tests. Co-authored-by: Yassin Kortam <[email protected]> * fix(bedrock): pre-clamp reasoning_effort for chat invoke; correct test caps - Add _clamp_adaptive_reasoning_effort_for_bedrock to AmazonAnthropicClaudeConfig so raw reasoning_effort=xhigh degrades to the model's bedrock effort ceiling before AnthropicConfig.map_openai_params converts it to output_config. Mirrors converse path (_handle_reasoning_effort_parameter) and messages path (_clamp_adaptive_reasoning_effort_for_bedrock) so the three Bedrock paths are consistent. - grid_spec: restore caps=_CAPS_4_6 for Bedrock converse/invoke Opus 4.6 entries so the test reflects the model's actual JSON capabilities. Teach expected() to bypass the xhigh/max cap check when bedrock_effort_ceiling will clamp the wire effort, so the test still passes for Bedrock's graceful degradation contract without lying about native model caps. Co-authored-by: Yassin Kortam <[email protected]> --------- Co-authored-by: Dennis Henry <[email protected]> Co-authored-by: Cursor Agent <[email protected]> Co-authored-by: Yassin Kortam <[email protected]> * feat(guardrails): wire apply_guardrail into proxy logging callbacks (BerriAI#28970) * feat(guardrails): wire apply_guardrail into proxy logging callbacks Route /apply_guardrail through pre/post proxy hooks and LiteLLM success/failure handlers so Langfuse and OTEL integrations receive input/output on guardrail-only requests. Co-authored-by: Cursor <[email protected]> * fix(guardrails): fix Greptile review comments on apply_guardrail logging Co-authored-by: Cursor <[email protected]> * fix(apply_guardrail): preserve original exception and capture modified response - Capture return value from post_call_success_hook so callback-modified responses propagate to the caller. - Wrap success/failure logging calls in defensive try/except so logging infrastructure failures don't replace the user-visible response or mask the original guardrail exception. Co-authored-by: Yassin Kortam <[email protected]> * Fix mypy * fix(apply_guardrail): isolate failure logging and use post-hook response for logging - Split async_failure_handler and post_call_failure_hook into independent try/except blocks so a callback bug in one does not silently skip the other. - Build response_for_logging inside _emit_guardrail_success_logs after post_call_success_hook runs, so logged data matches the response the caller actually receives when the hook modifies the response. Co-authored-by: Yassin Kortam <[email protected]> * fix(apply_guardrail): fix black formatting and update tests for fastapi_request param - Run black on guardrail_endpoints.py to fix CI formatting check - Add _mock_proxy_logging() helper to enterprise guardrail tests to patch proxy-server globals imported at call time - Pass fastapi_request=Mock() in all direct apply_guardrail test calls to match updated function signature Co-authored-by: Cursor <[email protected]> * fix(guardrails): use transformed exception from post_call_failure_hook in apply_guardrail Co-authored-by: Yassin Kortam <[email protected]> * fix(guardrails): isolate sync/async logging handlers in apply_guardrail Separate each logging handler call into its own try/except so a failure in the async handler does not silently skip the sync handler submission (and vice versa). Matches the docstring's defensive intent. Co-authored-by: Yassin Kortam <[email protected]> * fix(apply_guardrail): guard transformed_exception with isinstance check Co-authored-by: Cursor <[email protected]> * test(guardrails): mock proxy globals in not_found test and share apply_guardrail logging fixture - Add proxy-server global mocks to test_apply_guardrail_not_found so the failure-path post_call_failure_hook call doesn't touch the real proxy logging singleton. - Extract the duplicated _mock_proxy_logging context manager out of the two enterprise apply_guardrail test files into a shared conftest fixture so the helper stays in one place. * fix(guardrails): use update_messages to keep logging obj in sync Co-authored-by: Yassin Kortam <[email protected]> --------- Co-authored-by: Cursor <[email protected]> Co-authored-by: Yassin Kortam <[email protected]> Co-authored-by: mateo-berri <[email protected]> * chore(ci): merge dev brach (BerriAI#29192) * build(deps): bump next from 16.2.4 to 16.2.6 in /ui/litellm-dashboard (BerriAI#27665) Bumps [next](https://github.com/vercel/next.js) from 16.2.4 to 16.2.6. - [Release notes](https://github.com/vercel/next.js/releases) - [Changelog](https://github.com/vercel/next.js/blob/canary/release.js) - [Commits](vercel/next.js@v16.2.4...v16.2.6) --- updated-dependencies: - dependency-name: next dependency-version: 16.2.6 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump protobufjs in /tests/pass_through_tests (BerriAI#28296) Bumps [protobufjs](https://github.com/protobufjs/protobuf.js) from 7.5.6 to 7.6.0. - [Release notes](https://github.com/protobufjs/protobuf.js/releases) - [Changelog](https://github.com/protobufjs/protobuf.js/blob/protobufjs-v7.6.0/CHANGELOG.md) - [Commits](protobufjs/protobuf.js@protobufjs-v7.5.6...protobufjs-v7.6.0) --- updated-dependencies: - dependency-name: protobufjs dependency-version: 7.6.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump ws from 8.20.0 to 8.20.1 in /tests/pass_through_tests (BerriAI#28303) Bumps [ws](https://github.com/websockets/ws) from 8.20.0 to 8.20.1. - [Release notes](https://github.com/websockets/ws/releases) - [Commits](websockets/ws@8.20.0...8.20.1) --- updated-dependencies: - dependency-name: ws dependency-version: 8.20.1 dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: improve bedrock streaming hot path perf (BerriAI#28720) * fix(proxy): enforce tag budgets for key-level tags (BerriAI#29108) * fix(proxy): enforce tag budgets for key-level tags Merge API key metadata.tags into request_data before _tag_max_budget_check so per-tag budgets apply when tags are set on the key at creation time. Co-authored-by: Cursor <[email protected]> * fix(auth): avoid false reject for key-inherited tags Run reject_clientside_metadata_tags before key-tag injection, then inject key metadata tags immediately before tag budget checks so key tags still enforce budgets without being treated as client-supplied tags. Co-authored-by: Cursor <[email protected]> --------- Co-authored-by: Cursor <[email protected]> * fix(vertex-ai): use DB credentials in video handlers + implement Veo video edit (BerriAI#29098) * fix(vertex-ai): pass litellm_params to validate_environment in video handlers and implement video edit for Veo - Pass litellm_params to validate_environment in 11 video handler call sites (remix, create_character, get_character, edit, extension, delete) so DB-stored Vertex AI credentials are used instead of falling back to ADC - Implement transform_video_edit_request/response for VertexAI: fetches source video via fetchPredictOperation then submits a new predictLongRunning request with the video bytes/gcsUri + edit prompt Co-authored-by: Cursor <[email protected]> * fix(vertex-ai): hoist fetchPredictOperation into handlers to avoid blocking event loop - Add get_video_edit_prefetch_params() to BaseVideoConfig (returns None) - VertexAI overrides it to return the fetchPredictOperation URL/body - Both sync and async video_edit handlers call this and use their shared httpx client for the fetch, passing the result as prefetched_source_data - transform_video_edit_request is now a pure transform with no HTTP calls - Fix extra_body.pop() mutation by working on a shallow copy Co-authored-by: Cursor <[email protected]> * fix(vertex-ai): include prefetch call inside _handle_error try/except block Co-authored-by: Cursor <[email protected]> * fix(videos): add prefetched_source_data param to all transform_video_edit_request overrides Co-authored-by: Cursor <[email protected]> * fix(video_edit): keep transform/pre_call outside try so validation errors propagate Move transform_video_edit_request and logging_obj.pre_call outside the try/except that wraps HTTP calls in (async_)video_edit_handler so that ValueError validation errors (e.g. 'source video not complete yet') are not silently wrapped as 500s by _handle_error. The prefetch HTTP call keeps its own try/except so its errors are still mapped through the provider's error handler. Matches the pattern used by video_extension_handler and video_remix_handler. Co-authored-by: Yassin Kortam <[email protected]> * refactor(vertex_ai): delegate get_video_edit_prefetch_params to status retrieve Co-authored-by: Yassin Kortam <[email protected]> * Fix varia review * fix(video_edit): route transform errors through _handle_error Wrap transform_video_edit_request and pre_call in the same try/except as the HTTP call in sync and async handlers so validation failures (e.g. source video not complete) return typed LiteLLM exceptions. Co-authored-by: Cursor <[email protected]> --------- Co-authored-by: Cursor <[email protected]> Co-authored-by: Yassin Kortam <[email protected]> * fix(datadog): drain cost-management queue + opt-in FinOps tag allowlist (BerriAI#28487) * fix(datadog): drain cost-management queue + opt-in FinOps tag allowlist * fix(datadog): guard non-dict callback_specific_params + log empty aggregation * fix(datadog): block user-controlled tags from overwriting reserved cost-attribution dimensions * fix(datadog): cast metadata to dict[str, Any] to satisfy mypy * feat(helm): split per-component ServiceAccounts for gateway, backend, and UI (BerriAI#28712) * feat(helm): split per-component ServiceAccounts for gateway, backend, and UI Replace the single shared serviceAccount with three separate serviceAccounts (gateway, backend, ui) so operators can attach different IRSA / Workload Identity annotations per component without granting data-plane credentials to the UI pod. Key changes: - values.yaml: rename serviceAccount → serviceAccounts with gateway/backend/ui sub-keys; UI defaults to automount: false - _helpers.tpl: replace litellm.serviceAccountName with three component-scoped helpers (litellm.gateway/backend/ui.serviceAccountName) - serviceaccount.yaml: create up to three separate ServiceAccount objects with component labels and per-SA automountServiceAccountToken - gateway/backend deployments: use their respective SA helpers - ui deployment: use litellm.ui.serviceAccountName + explicit automountServiceAccountToken: false on the pod spec so the projected token is absent even when the SA itself allows it - migrations-job: share the backend SA (both need DB write access) Resolves LIT-3171 https://claude.ai/code/session_01QPy362WnjmEpeNuJaPUqmF * fix(helm): enforce automountServiceAccountToken on all pod specs; fix leading --- in serviceaccount.yaml - gateway/backend deployments: add explicit automountServiceAccountToken on the pod spec so serviceAccounts.*.automount is honoured regardless of whether the SA is chart-created or operator-supplied (previously the flag only took effect on the SA object when create: true, creating an asymmetry with the UI which already enforced it at pod-spec level) - serviceaccount.yaml: use a $prev sentinel to emit --- only between documents, preventing a leading --- when gateway SA is skipped but backend or ui SA is created (avoids lint/GitOps warnings from strict YAML parsers and tools like ArgoCD) https://claude.ai/code/session_01QPy362WnjmEpeNuJaPUqmF --------- Co-authored-by: Claude <[email protected]> * bump deps (BerriAI#29208) (BerriAI#29226) * fix(deps): bump vulnerable proxy dependencies (starlette/fastapi, granian, pyarrow, semantic-router) Resolve known CVEs flagged by osv-scanner/grype against uv.lock. All bumped versions verified to resolve, install, and pass the proxy auth/route/middleware unit suites (717 tests) plus an import smoke on the new stack. - starlette 0.50.0 -> 1.1.0 (CVE-2026-48710 "BadHost", GHSA-86qp-5c8j-p5mr): versions <1.0.1 reconstruct request.url from the unvalidated Host header, poisoning request.url.path. Required raising fastapi 0.124.4 -> 0.136.3, which dropped fastapi's starlette<0.51.0 cap; an explicit starlette>=1.0.1 floor blocks regression to a vulnerable transitive resolution. The proxy's own auth already reads scope["path"] via get_request_route, but the locked starlette still flagged in container scanners and left other request.url consumers exposed. - granian 2.5.7 -> 2.7.4 (CVE-2026-42544, unauthenticated DoS via WebSocket subprotocol header panic; CVE-2026-42545, WSGI response-header-panic DoS). granian is a selectable proxy server (proxy_cli). - pyarrow 22.0.0 -> 23.0.1 (CVE-2026-25087 / PYSEC-2026-113). - semantic-router 0.1.12 -> 0.1.15: 0.1.12 was yanked (CVE-2026-42208 — its unbounded litellm pin could resolve a credential-exfiltrating litellm==1.82.8 wheel). Not fixable by bump: diskcache 5.6.3 (CVE-2025-69872, unsafe pickle deserialization) has no upstream fix and is left pinned; exploiting it requires write access to the local cache directory. Relock side effect: sse-starlette 3.4.2 -> 3.4.4. * deps: relax exact pins in optional extras to compatible ranges The proxy/optional extras exact-pinned every dependency, which (1) forces downstream `pip install litellm[proxy]` consumers into version lockstep and (2) blocks them from pulling transitive security patches without forking — the structural cause behind needing a litellm release to clear the starlette CVE in the previous commit. Convert the ordinary extras deps to `>=current,<next_major` ranges, mirroring the core [project].dependencies style. Reproducibility for litellm's own Docker/CI is unaffected: images install via `uv sync --frozen`, and the lock re-resolves to the identical versions (no locked version changed). Kept exact-pinned: - litellm-proxy-extras, litellm-enterprise — litellm's own sub-packages, versioned in lockstep with the release. - opentelemetry-api/sdk/exporter-otlp — must resolve to matching versions. - grpcio — supply-chain-pinned to a vetted, aged release. Also corrects the stale comment claiming the extras are exact-pinned for Docker reproducibility (the images use the lock, not these pins). * fix(ci): resolve license-check lookup version from the floor for ranged deps check_licenses.py derived the PyPI lookup version with `next(iter(req.specifier))`, which returns an arbitrary specifier clause. For a range like `>=0.12.1,<1.0` it picked the upper bound (`1.0`) — a version that doesn't exist on PyPI — so the license lookup 404'd and the package was flagged as having an unknown license. The previous commit's switch from exact pins to ranges exposed this for soundfile, pyroscope-io, redisvl, diskcache, and mlflow (the ranged deps not already in liccheck.ini's allowlist). Prefer a lower-bound/exact version (a real released version) for the lookup. * fix(proxy): set strict_content_type=False on the FastAPI app Starlette 1.0 / FastAPI 0.13x flipped the default to strict_content_type=True, which refuses to parse a JSON request body when the client omits the Content-Type header. The proxy previously accepted those requests, so the fastapi/starlette bump in this PR would silently break clients that don't send a Content-Type. Restore the prior lenient behavior explicitly. Co-authored-by: stuxf <[email protected]> * fix(tests/vcr): mint Google OAuth tokens live to prevent stale-token replay (BerriAI#29229) The Redis-backed VCR layer was recording and replaying the Google OAuth2/STS token-mint call. The replayed ya29.* access token is long-expired, but its recorded expires_in keeps credentials.expired False, so litellm never refreshes it and sends the stale token to a live Vertex/Gemini endpoint, which returns 401 ACCESS_TOKEN_EXPIRED. This broke live partner-model tests whose completion call is not itself cassette-backed (e.g. test_vertex_ai_llama_tool_calling). Force credential-exchange hosts to pass through live (never recorded, never replayed) by returning None from before_record_request, mirroring the existing telemetry passthrough, so a fresh token is minted each run. Regression from BerriAI#28826, which added OAuth-token matcher tolerance plus TTL-refresh-on-read so a stale token episode matched and never expired. * chore(cookbook): bump Go directive to 1.26.3 in gollem example (BerriAI#29234) Updates the gollem_go_agent_framework example to the current Go release. Clears stale Go stdlib advisories reported by osv-scanner against the older 1.25.1 directive. No source changes; the single pinned dependency (gollem v0.1.0) is backward compatible. * chore(ci): bump version (BerriAI#29242) * bump: version 1.87.0 → 1.88.0 * uv lock * feat(anthropic): add Claude Opus 4.8 and prune reasoning-effort flags (BerriAI#29238) * feat(anthropic): add Claude Opus 4.8 and prune reasoning-effort flags Register claude-opus-4-8 across the anthropic/bedrock/vertex/azure cost-map entries, BEDROCK_CONVERSE_MODELS, and the setup-wizard provider list. Prune two reasoning-effort fields from the cost map: - Drop supports_minimal_reasoning_effort from the Claude fleet (58 entries). "minimal" is not a real Anthropic effort level (the API accepts only low/medium/high/xhigh/max), so LiteLLM degrades it to "low" regardless; the flag was inert and misleading on Anthropic. - Remove tool_use_system_prompt_tokens everywhere (103 entries). It is not in the ModelInfo type and is read by no production code. Update the affected config/schema tests; the reasoning-effort registry tests now assert the Claude fleet omits supports_minimal. * fix(anthropic): recognize output_config effort after minimal-flag prune Pruning supports_minimal_reasoning_effort from the Claude fleet removed the only "supports effort param" marker from 11 Opus 4.5 / mythos-preview map entries that lack supports_output_config. _model_supports_effort_param then returned False for them, so output_config was wrongly dropped under drop_params=True -- regressing test_anthropic_model_supports_effort_param_recognizes_supporting_models for claude-opus-4-5-20251101 and the mythos preview. - _model_supports_effort_param now treats supports_output_config as a sufficient signal, matching the bedrock-invoke call sites that already check supports_output_config OR a reasoning-effort flag. Shared map lookup extracted into _supports_model_capability. - Add supports_output_config: true to the 11 Opus 4.5 / mythos entries that lost their only marker, restoring prior effort-forwarding behavior without re-adding the inert minimal flag. * fix(ci): restore real Bedrock batch S3 bucket and role in oai_misc_config (BerriAI#29245) The OSS-staging sync (d52fbfb) overwrote the Bedrock batch model's s3_bucket_name and aws_batch_role_arn with public-safe placeholders (account 123456789012 / *_EXAMPLE role). The e2e_openai_endpoints CI job runs the proxy with AWS account 941277531214 credentials, so on file upload test_bedrock_batches_api failed with: NoSuchBucket: The specified bucket does not exist <BucketName>litellm-proxy-123456789012</BucketName> Restore the real resources that live in account 941277531214 (verified to exist) — the same values tests/batches_tests/test_bedrock_files_and_batches.py already references. Co-authored-by: Claude Opus 4.8 <[email protected]> * fix(guardrails): persist disable_global_guardrails on keys (BerriAI#29233) * fix(guardrails): restore disable_global_guardrails persistence for keys The per-key/team "Disable Global Guardrails" toggle silently stopped working after BerriAI#17042, which removed `disable_global_guardrails` from the key/team request models and from the premium metadata allowlist. Without those, the UI's top-level field was dropped by pydantic and never folded into key `metadata`, so the runtime gate always read False and global default_on guardrails kept running. Restore the request-model fields (KeyRequestBase, NewTeamRequest, UpdateTeamRequest) and the `LiteLLM_ManagementEndpoint_MetadataFields_Premium` entry so the flag is promoted into metadata again. Because the key edit form always submits the flag (false by default), guard the UI so it is only sent when it actually changed (edit) or is enabled (create) — this keeps the premium gate on enabling intact while not 403-ing non-premium users who edit unrelated key fields, mirroring how guardrails/tags are already stripped. * test(guardrails): cover disable_global_guardrails toggle-off + clarify premium field comment Add a prepare_metadata_fields case asserting `disable_global_guardrails: False` overwrites an existing `True`, and rewrite the PREMIUM_METADATA_FIELDS comment to explain why boolean premium fields are excluded from the empty-value strip loop. * test(e2e): cover Team Admin view + member + key flows (BerriAI#29072) * test(e2e): cover Team Admin view + member + key flows Adds a new spec exercising the previously-uncovered team-admin manual-QA items: viewing all team keys (including other members'), adding a member, removing a member, and creating a team key with All Team Models. Also seeds a dedicated invitee user so the add-member test can run in parallel with the proxy-admin invite test without colliding on the team roster. * test(e2e): harden team-admin member specs per review feedback Address Greptile feedback on the Team Admin spec: - locate the delete action via getByTestId("delete-member") instead of the fragile svg/img .last() selector - match the seeded removable member by user_id (members_with_roles stores no email, so the roster renders user_id) - assert exact success-toast strings rather than broad regexes that could match unrelated "success" text * docs: hand-written CLAUDE.md; point GEMINI.md and AGENTS.md at it (BerriAI#29252) * docs: replace generated CLAUDE.md with hand-written guidance, remove AGENTS.md Swap the auto-generated CLAUDE.md for a concise hand-written version that captures how we actually want agents to work in this repo: minimal comments, simplicity first, meaningful tests with a high mutation kill rate, PRs based off litellm_internal_staging rather than main, and curl against a live proxy as proof of fix instead of pasted pytest output. Remove AGENTS.md so there is one source of truth for agent guidance. The customer and company name confidentiality policy, along with the MCP available_on_public_internet note, are carried over from the previous CLAUDE.md. * fix: further clarify communication guidelines * docs: point GEMINI.md at CLAUDE.md instead of duplicating guidance Replace the standalone GEMINI.md copy, which had already drifted from the new CLAUDE.md, with a one-line pointer so Gemini reads the same single source of truth. * docs: simplify PR template test checklist item Replace the rigid "at least 1 test is a hard requirement" checklist line with "I have added meaningful tests", which matches the testing guidance in CLAUDE.md, and tidy a comma into a semicolon in the scope-isolation item. * docs: point AGENTS.md at CLAUDE.md instead of deleting it Keep AGENTS.md so tools that read it still resolve guidance, but collapse it to the same one-line pointer to CLAUDE.md used by GEMINI.md, keeping a single source of truth. * fix: make AI-generated rules more concise * fix: spelling Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix: make the .env usage more careful * docs: restore MCP available_on_public_internet note to CLAUDE.md The PR description states this note was carried over verbatim from the previous CLAUDE.md, but it was dropped in the rewrite. Restore it so the file matches the description and the team guidance is not lost. * docs: restore browser storage and CI supply-chain safety notes to CLAUDE.md These security-relevant rules were dropped in the rewrite. Restore the sessionStorage-over-localStorage (XSS) guidance and the CI supply-chain rules (no curl|bash, pin versions, verify checksums) so agents editing UI or CI code are still steered away from those pitfalls. * docs: move area-specific guidance into nested CLAUDE.md files The MCP, browser-storage, and CI supply-chain notes are scoped to particular parts of the tree, so move each into a nested CLAUDE.md that Claude Code loads on demand when those files are touched: the MCP note under the mcp_server gateway, the browser-storage rule under the UI dashboard, and the CI supply-chain rules under .circleci. Keeps the root CLAUDE.md focused on general guidance while the area notes surface where they are relevant. * docs: keep CI supply-chain note in root CLAUDE.md CI guidance applies beyond .circleci (it also covers downloads in GitHub workflows and any CI script), and CI work does not reliably touch a single subtree, so a nested file under .circleci would not surface it dependably. Keep it in the always-loaded root instead. The MCP and browser-storage notes stay nested where they map cleanly to one area of the tree. * fix: make it clear we prefer httpOnly * chore: make ci rule more concise * chore: make concise Fix formatting and punctuation in MCP note. * fix: don't include Claude attribution --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix: regenerate uv.lock to sync with pyproject.toml Co-Authored-By: Claude Sonnet 4.6 <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Mateo Wang <[email protected]> Co-authored-by: Dennis Henry <[email protected]> Co-authored-by: Cursor Agent <[email protected]> Co-authored-by: Yassin Kortam <[email protected]> Co-authored-by: Sameer Kankute <[email protected]> Co-authored-by: yuneng-jiang <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: michelligabriele <[email protected]> Co-authored-by: Claude <[email protected]> Co-authored-by: stuxf <[email protected]> Co-authored-by: ryan-crabbe-berri <[email protected]> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

…ero VCR misses on consecutive runs) (BerriAI#28826) * test(vcr): make Redis-backed cassettes replay deterministically across runs - Pin LITELLM_LOCAL_MODEL_COST_MAP=True in the shared VCR harness so the per-test importlib.reload(litellm) no longer fetches the model cost map from raw.githubusercontent.com. That live fetch was being recorded into cassettes; for tests that subsequently skip it was the only recorded episode, so the persister refused to save it (skipped tests don't persist) and the test re-recorded it live every run (MISS:NOT_PERSISTED). - Compare-time symmetric matcher tolerance for Google OAuth (ya29.*) tokens, observability/telemetry payloads, credential-exchange bodies, and volatile UUID/timestamp tokens, so existing cassettes select a recorded episode instead of growing past the 50-episode cap and re-recording live. - Don't record fire-and-forget telemetry (langfuse/arize/otel/...) into non-telemetry tests' cassettes. Several modules set litellm.success_callback at import time, so observability logging is globally enabled and an async flush from the background logging worker lands in an unrelated test's VCR window, saved as a spurious MISS:RECORDED (observed: a Langfuse batch from another completion landing on test_lowest_latency_routing_buffer). Such a request now passes through live (telemetry hosts aren't real-spend hosts); tests that actually assert on telemetry keep recording it. - Dedupe + cap the VCR diagnostic dump so the classification summary survives CircleCI's ~400KB step-output truncation. - Stabilize a non-deterministic rate-limit test body; mark AWS Secrets Manager lifecycle tests VCR-incompatible (uniquely-named secrets can't be replayed). - Mark test_router_text_completion_client VCR-incompatible: it fires 300 identical requests to verify async-client reuse, but vcrpy patches the HTTP transport so replay never exercises the real connection pool the test validates, and recording 300 near-identical episodes overflows the 50-episode cap (MISS:OVERFLOW every run). It hits a free mock endpoint. - Mark the Vertex AI MaaS Mistral OCR tests (vertex_ai/mistral-ocr-2505) VCR-incompatible: the MaaS model is not provisioned in the CI GCP project, so the live :rawPredict call fails and the test skips every run, leaving no cassette to record (MISS:NOT_PERSISTED every run). Sibling direct-Mistral and Azure OCR tests are unaffected and still replay from cache. * fix(tests/vcr): refresh cassette TTL on read so replayed cassettes don't expire The Redis VCR persister loaded cassettes with a plain GET, which does not touch the key's TTL. A cassette that is only ever replayed (HIT/NOOP, never re-recorded) therefore expired exactly 24h after its last *write*, no matter how often it was read. Whichever CI run happened to cross that boundary re-recorded the cassette live and surfaced a spurious VCR MISS on otherwise deterministic cassettes — the residual per-run flakiness floor (a different random subset of read-only cassettes expiring each run). Slide the expiry forward on every successful load (best-effort EXPIRE), so any cassette used at least once per TTL window stays alive indefinitely and the 2nd/3rd run of a day replays cleanly. * fix(tests/vcr): recover from spurious GET-None for existing cassette keys Under concurrent CI load, the persister's load GET was observed returning None for a cassette key that demonstrably existed on the (single, non- clustered) Redis master — an external monitor saw the key present with a healthy TTL at the same instant the in-process client read None. Because None is a valid GET result (not a RedisError), the retry-on-error client config never engaged, so the cassette re-recorded live (a phantom MISS:RECORDED); for flaky/networked tests the failed live call then triggered a pytest rerun, which is why a rotating subset of otherwise deterministic tests missed each run. On a None result, re-check EXISTS and re-read once. If the key really exists, use the recovered value and log [vcr-transient-miss-recovered] (also counted in cassette_cache_health). A genuinely absent key (a new cassette) still falls through to CassetteNotFoundError. * chore(tests/vcr): TEMP diagnostic for persistent-miss cassette load path Logs GET/EXISTS at load time for the three cassettes that re-record every run despite being present in Redis, to capture what the in-process client sees. To be reverted before merge. * chore(tests/vcr): write load diagnostic to Redis (truncation-proof) CI stdout truncates to the last ~400KB, dropping the early loaddbg lines for the alphabetically-first failing test. Push the load probe to a Redis list instead so it survives. To be reverted before merge. * fix(tests/vcr): don't drop stored telemetry episodes during cassette load Root cause of the residual per-run misses on present cassettes: vcrpy's Cassette._load() replays each *stored* interaction through Cassette.append(), which runs before_record_request on it — and a None return there silently drops that episode. The telemetry-leak suppressor (_should_drop_telemetry_record) returns None for telemetry requests, so when a non-telemetry-named test (or the alphabetically-first test in a worker, whose _current_test_nodeid is still empty) loaded a cassette containing a Langfuse ingestion episode, the episode was dropped on read — forcing an endless live re-record (a phantom MISS:RECORDED on a cassette that was demonstrably present in Redis). Verified by reproducing Cassette._load() against the real cassette: empty/non-telemetry nodeid -> 0 episodes survive; with the guard -> 1 survives. Fix: guard the suppressor with a thread-local set around Cassette._load (via a small idempotent monkeypatch), so the drop only ever stops *new* incidental telemetry from being recorded and never filters the existing cassette on read. Also drops the speculative GET-None recovery + its diagnostics from the previous commits: the load diagnostic showed GET returns the cassette bytes fine (get=1440B), so the persister never returned a spurious None — the loss happened later in vcrpy's append. The proven TTL-refresh-on-read fix is retained. * fix(tests/vcr): drop incidental telemetry export POSTs to stop rotating async-flush misses litellm's observability loggers flush on a background thread, so a Langfuse ingestion POST scheduled by one telemetry test can fire mid-way through a *later* telemetry-named test (after that test's own httpx mock has exited) and be recorded by VCR as a phantom episode — a non-deterministic MISS:RECORDED / PARTIAL that rotates onto a different telemetry test from run to run. Telemetry export POSTs are fire-and-forget; no test asserts on a *recorded* export response except the pass-through proxy test (which forwards a client POST to Langfuse ingestion and replays its 207). So _should_drop_telemetry_record now drops incidental export POSTs for every test except that one. Dropping returns None (live fire-and-forget, never stored), so it can only turn a phantom miss into a harmless live call, never the reverse; recorded read-back GETs that telemetry tests assert on are matched by method and left untouched. * fix(tests/vcr): restore assertion in test_banner_silent_when_vcr_disabled The assertion that the banner is suppressed when VCR is disabled was inadvertently moved into test_diagnostic_log_silent_when_no_dir when the diagnostic-log tests were added, leaving the disabled-VCR test verifying nothing. Co-authored-by: Yassin Kortam <[email protected]> --------- Co-authored-by: Cursor Agent <[email protected]> Co-authored-by: Yassin Kortam <[email protected]>

…replay (BerriAI#29229) The Redis-backed VCR layer was recording and replaying the Google OAuth2/STS token-mint call. The replayed ya29.* access token is long-expired, but its recorded expires_in keeps credentials.expired False, so litellm never refreshes it and sends the stale token to a live Vertex/Gemini endpoint, which returns 401 ACCESS_TOKEN_EXPIRED. This broke live partner-model tests whose completion call is not itself cassette-backed (e.g. test_vertex_ai_llama_tool_calling). Force credential-exchange hosts to pass through live (never recorded, never replayed) by returning None from before_record_request, mirroring the existing telemetry passthrough, so a fresh token is minted each run. Regression from BerriAI#28826, which added OAuth-token matcher tolerance plus TTL-refresh-on-read so a stale token episode matched and never expired.

mateo-berri added 7 commits May 25, 2026 23:12

chore(tests/vcr): TEMP diagnostic for persistent-miss cassette load path

ec4ebc8

Logs GET/EXISTS at load time for the three cassettes that re-record every run despite being present in Redis, to capture what the in-process client sees. To be reverted before merge.

chore(tests/vcr): write load diagnostic to Redis (truncation-proof)

9219601

CI stdout truncates to the last ~400KB, dropping the early loaddbg lines for the alphabetically-first failing test. Push the load probe to a Redis list instead so it survives. To be reverted before merge.

mateo-berri marked this pull request as ready for review May 26, 2026 18:03

cursor Bot reviewed May 26, 2026

View reviewed changes

Comment thread tests/llm_translation/test_vcr_conftest_common_banner.py

mateo-berri requested a review from yuneng-berri May 26, 2026 18:27

yuneng-berri approved these changes May 26, 2026

View reviewed changes

mateo-berri merged commit 533eab4 into litellm_internal_staging May 26, 2026
116 of 119 checks passed

yuneng-berri mentioned this pull request May 28, 2026

fix(tests/vcr): mint Google OAuth tokens live to prevent stale-token replay #29229

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(tests/vcr): make Redis cassette cache replay deterministically (zero VCR misses on consecutive runs)#28826

fix(tests/vcr): make Redis cassette cache replay deterministically (zero VCR misses on consecutive runs)#28826
mateo-berri merged 8 commits into
litellm_internal_stagingfrom
litellm_vcr_replay_determinism

mateo-berri commented May 26, 2026 •

edited by cursor Bot

Loading

Uh oh!

codecov Bot commented May 26, 2026

Uh oh!

greptile-apps Bot commented May 26, 2026 •

edited

Loading

Important Files Changed

Uh oh!

cursor Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

mateo-berri commented May 26, 2026

Uh oh!

mateo-berri commented May 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

mateo-berri commented May 26, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What & why

Verification

Summary of every issue addressed

Observability fix (enabler)

Out of scope — pre-existing failures (not VCR misses, not caused by this PR)

Appendix — full VCR opt-out inventory

incompatible — skip_nodeid_suffixes (the deliberate, named opt-outs)

file_opt_out — whole file in skip_files, module has no live respx wiring

respx_conflict_module — file in skip_files, module does use respx

respx_conflict — auto-detected per test (not in any conftest list)

Uh oh!

codecov Bot commented May 26, 2026

Codecov Report

Uh oh!

greptile-apps Bot commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mateo-berri commented May 26, 2026

Uh oh!

mateo-berri commented May 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mateo-berri commented May 26, 2026 •

edited by cursor Bot

Loading

`incompatible` — `skip_nodeid_suffixes` (the deliberate, named opt-outs)

`file_opt_out` — whole file in `skip_files`, module has no live respx wiring

`respx_conflict_module` — file in `skip_files`, module does use respx

`respx_conflict` — auto-detected per test (not in any conftest list)

greptile-apps Bot commented May 26, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading