Skip to content

Commit 1132a54

Browse files
gglucassclaudeJerrettDavis
authored
fix(proxy): do not cache error replies delivered as http 200 (#3930)
## Description The response cache stores any non-streaming reply whose upstream status is 200. An Anthropic- or OpenAI-compatible gateway can answer 200 with an error object (`{"type": "error", ...}` / `{"error": {...}}`) or an empty object, and that body was then replayed from the cache to every matching request for the full TTL (1h by default) without the upstream being asked again. Only a proxy restart or `POST /cache/clear` recovered. `SemanticCache.set` now refuses an empty body and any JSON body that is not a non-empty object or that carries an error. Every store goes through `set` (the Anthropic and OpenAI chat handlers are its only callers), so one check covers both instead of a guard at each call site. Closes #3929 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/semantic_cache.py`: new `_is_cacheable_reply(response_body)`; `SemanticCache.set` returns without storing when it is false. Refused: empty bytes, JSON that is not a dict, `{}`, a dict with `"type": "error"`, a dict with a non-null `"error"`. A reply with `"error": null` is still cached. Bytes that are not JSON are stored as before: both handlers only store a reply they already decoded, and existing cache tests store opaque bytes. - `tests/test_proxy_response_cache_store_guard.py`: parametrized unit test of `_is_cacheable_reply`, plus an end-to-end test per handler (`/v1/messages`, `/v1/chat/completions`) with the cache on and an upstream that answers 200 + error body once, then a good body. It asserts the second request reaches the upstream and the good reply is then served from cache. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed Lint, format and type checks ran on the changed files only. Pytest ran the new file plus the existing response-cache test files, not the whole suite. ### Test Output Before the fix (`semantic_cache.py` at upstream/main d5318ac, plus a pass-through `_is_cacheable_reply` stub so the new test module imports): ```text $ python -m pytest tests/test_proxy_response_cache_store_guard.py -q -p no:cacheprovider -k replayed E assert 1 == 2 E assert 1 == 2 FAILED tests/test_proxy_response_cache_store_guard.py::test_error_200_is_not_replayed_from_cache[/v1/messages-headers0-claude-haiku-4-5-error_body0-ok_body0] FAILED tests/test_proxy_response_cache_store_guard.py::test_error_200_is_not_replayed_from_cache[/v1/chat/completions-headers1-gpt-4o-mini-error_body1-ok_body1] ================= 2 failed, 10 deselected, 1 warning in 9.28s ================== ``` After the fix: ```text $ python -m pytest tests/test_proxy_response_cache_store_guard.py tests/test_proxy_semantic_cache_key.py \ tests/test_proxy_semantic_cache_key_integration.py tests/test_proxy_openai_cache_key_integration.py \ tests/test_openai_response_cache_key.py tests/test_proxy_semantic_cache_key_policy.py \ tests/test_semantic_cache_key_policy.py -q -x -p no:cacheprovider ======================== 61 passed, 1 warning in 16.55s ======================== $ python -m pytest tests/test_proxy_response_cache_replay.py tests/test_ccr_buffered_stream_signed_thinking.py \ tests/test_handler_outcome_tag_invariant.py tests/test_read_maturation_handler_nobust.py \ tests/test_observability_metrics.py -q -p no:cacheprovider =================== 45 passed, 1 warning in 69.84s (0:01:09) =================== $ ruff check headroom/proxy/semantic_cache.py tests/test_proxy_response_cache_store_guard.py All checks passed! $ ruff format --check headroom/proxy/semantic_cache.py tests/test_proxy_response_cache_store_guard.py 2 files already formatted $ mypy headroom/proxy/semantic_cache.py Success: no issues found in 1 source file ``` The warning is the existing `StarletteDeprecationWarning` from `fastapi.testclient`. ## Real Behavior Proof - Environment: macOS, Python 3.11.13. The real `headroom proxy` CLI, booted from the source tree with the default `ProxyConfig` (cache on, TTL 3600s, `Mode: cache`), pointed at a local stub upstream on 127.0.0.1 via `--anthropic-api-url` / `--openai-api-url`. The stub answers the first POST per path with HTTP 200 + an error body (`{"type": "error", "error": {"type": "overloaded_error", ...}}` for `/v1/messages`, `{"error": {"message": "Upstream overloaded", "type": "server_error", "code": null}}` for `/v1/chat/completions`) and every later POST with a normal reply. The scratch venv lacks the `mcp`, `magika` and `onnxruntime` extras, so the CLI was started through a three-line launcher that skips `ensure_proxy_dependencies()` and then calls the stock `headroom.cli.main.main(["proxy", ...])`. None of those modules is on the response-cache path. - Exact command / steps: `python launch.py --port <p> --anthropic-api-url http://127.0.0.1:<s> --openai-api-url http://127.0.0.1:<s>/v1`, then three identical non-streaming requests per endpoint, then `GET /stats`. Run once with upstream/main d5318ac and once with this branch, same venv and stub. - Observed result: on main the stub is hit once per path and requests 2 and 3 replay the stored error from cache; on this branch request 2 reaches the stub and gets the good reply, and request 3 is served that reply from cache (output below). upstream/main (d5318ac): the stub is hit once per path; requests 2 and 3 get the stored error from cache. ```text stub: /v1/messages hit #1 -> 200 error body client: /v1/messages request 1 -> 200 in 668ms: {"type": "error", "error": {"type": "overloaded_error", "message": "Overloaded"}} client: /v1/messages request 2 -> 200 in 26ms: {"type": "error", "error": {"type": "overloaded_error", "message": "Overloaded"}} client: /v1/messages request 3 -> 200 in 22ms: {"type": "error", "error": {"type": "overloaded_error", "message": "Overloaded"}} stub: /v1/chat/completions hit #1 -> 200 error body client: /v1/chat/completions request 1 -> 200 in 61ms: {"error": {"message": "Upstream overloaded", "type": "server_error", "code": null}} client: /v1/chat/completions request 2 -> 200 in 12ms: {"error": {"message": "Upstream overloaded", "type": "server_error", "code": null}} client: /v1/chat/completions request 3 -> 200 in 24ms: {"error": {"message": "Upstream overloaded", "type": "server_error", "code": null}} proxy /stats cache: {"entries": 2, "max_entries": 1000, "total_hits": 4, "ttl_seconds": 3600} ``` This branch: request 2 reaches the stub and gets the good reply; request 3 is served that good reply from cache. ```text stub: /v1/messages hit #1 -> 200 error body client: /v1/messages request 1 -> 200 in 1010ms: {"type": "error", "error": {"type": "overloaded_error", "message": "Overloaded"}} stub: /v1/messages hit #2 -> 200 good body client: /v1/messages request 2 -> 200 in 304ms: {"id": "msg_1", "type": "message", "role": "assistant", "model": "claude-haiku-4-5", "content": [...] client: /v1/messages request 3 -> 200 in 115ms: {"id": "msg_1", "type": "message", "role": "assistant", "model": "claude-haiku-4-5", "content": [...] stub: /v1/chat/completions hit #1 -> 200 error body client: /v1/chat/completions request 1 -> 200 in 51ms: {"error": {"message": "Upstream overloaded", "type": "server_error", "code": null}} stub: /v1/chat/completions hit #2 -> 200 good body client: /v1/chat/completions request 2 -> 200 in 21ms: {"id": "chatcmpl_1", "object": "chat.completion", "model": "gpt-4o-mini", "choices": [...] client: /v1/chat/completions request 3 -> 200 in 20ms: {"id": "chatcmpl_1", "object": "chat.completion", "model": "gpt-4o-mini", "choices": [...] proxy /stats cache: {"entries": 2, "max_entries": 1000, "total_hits": 2, "ttl_seconds": 3600} ``` Proxy `PERF` lines match: on main requests 2-3 log `tok_before=0` (cache hit) on both models; on this branch request 2 logs `tok_before=10` (forwarded) and only request 3 logs `tok_before=0`. - Not tested: real Anthropic/OpenAI or a real third-party gateway (stub only, no credentials used); streaming requests (they never read or write this cache); the full test suite (targeted files only); the empty / unparseable 200 seen in production, because on main neither handler can pass such a body to `set` any more (Anthropic requires `resp_json is not None`, OpenAI raises to its 502 path), so that case is covered by the unit test only. ## Runtime Rollout Safety - Rollout-managed feature(s): none. This is a correctness guard in `SemanticCache.set`, not a rollout feature. - Minimum rollout channel: n/a (applies on every channel). - Stable/default behavior changed: yes, narrowly. With the cache on (the default), an HTTP 200 reply that is empty, an empty JSON object, a non-object JSON value, or a JSON object carrying `"type": "error"` or a non-null `"error"` is no longer stored, so the next identical request goes upstream instead of replaying it. The reply itself is still returned to the client unchanged. Successful replies are cached exactly as before. - Kill switch / disable path: none added; there is no case where replaying a stored error is wanted. `--no-cache` still disables the cache as a whole. - Unsafe override required: no. - Qualification impact: none expected. Compression, prompt-cache prefix handling and token accounting are untouched; only which bodies the response cache keeps changes. Cache-hit counts drop only where the hits were replayed errors. - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` - it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A. ## Additional Notes - Documentation: N/A. No user-facing option or documented behavior changes. - Overlap with #3862 (partition the response cache by credential and principal): both touch `SemanticCache.set`, but this change only adds lines after the key computation and does not touch the signature, `_compute_key` or `get`. A three-way merge of `semantic_cache.py` against #3862's head is clean. The new tests call `_is_cacheable_reply` directly and go through the handlers for the end-to-end case, so they do not depend on whether `set`/`get` take `partition=`. - History: the production case that surfaced this was the malformed buffered-CCR 200 that #2997 turned into a real 502. That source is gone on main, but a 200 with a JSON error payload from a compatible gateway still reaches `set`, as the proof above shows. - The check parses the body once per store. Stores only happen on non-streaming 200 replies, so the cost is one `json.loads` per such reply. - Headroom Desktop has shipped an equivalent downstream patch since 2026-08-13. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3930" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> Co-authored-by: JD Davis <[email protected]>
1 parent 4ba231a commit 1132a54

2 files changed

Lines changed: 151 additions & 0 deletions

File tree

‎headroom/proxy/semantic_cache.py‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
from __future__ import annotations
99

1010
import asyncio
11+
import json
1112
import sys
1213
from collections import OrderedDict
1314
from datetime import datetime
@@ -22,6 +23,31 @@
2223
_strip_cache_control = strip_cache_control
2324

2425

26+
def _is_cacheable_reply(response_body: bytes) -> bool:
27+
"""Whether a 200 body is a reply the cache may replay.
28+
29+
The status code alone does not say so: an Anthropic- or OpenAI-compatible
30+
gateway can answer 200 with an error object, or with nothing. Stored, that
31+
body is replayed to every matching request for the full TTL and the
32+
upstream is never asked again. Refuse an empty body, and JSON that is not a
33+
non-empty object or that carries an error (Anthropic ``"type": "error"``,
34+
OpenAI ``"error": {...}``). Bytes that are not JSON pass: the cache is
35+
format-agnostic, and both handlers only store a reply they have decoded.
36+
"""
37+
if not response_body:
38+
return False
39+
try:
40+
reply = json.loads(response_body)
41+
except ValueError:
42+
return True
43+
return (
44+
isinstance(reply, dict)
45+
and bool(reply)
46+
and reply.get("type") != "error"
47+
and reply.get("error") is None
48+
)
49+
50+
2551
class SemanticCache:
2652
"""Simple semantic cache based on message content hash.
2753
@@ -84,6 +110,9 @@ async def set(
84110
"""Cache a response."""
85111
key = self._compute_key(messages, model, **key_fields)
86112

113+
if not _is_cacheable_reply(response_body):
114+
return
115+
87116
async with self._lock:
88117
# If key already exists, remove it first to update position
89118
if key in self._cache:
Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
"""The response cache must not store an error delivered as HTTP 200.
2+
3+
Both handlers gate ``SemanticCache.set`` on ``status_code == 200``. Anthropic- and
4+
OpenAI-compatible gateways can answer 200 with an error object (or an empty
5+
object), and once stored that body is replayed to every matching non-streaming
6+
request for the full TTL without the upstream ever being asked again.
7+
"""
8+
9+
from __future__ import annotations
10+
11+
import json
12+
13+
import httpx
14+
import pytest
15+
16+
pytest.importorskip("fastapi")
17+
18+
from fastapi.testclient import TestClient
19+
20+
from headroom.proxy.semantic_cache import _is_cacheable_reply
21+
from headroom.proxy.server import ProxyConfig, create_app
22+
23+
MESSAGES = [{"role": "user", "content": "Say hi."}]
24+
25+
ANTHROPIC_ERROR = {"type": "error", "error": {"type": "api_error", "message": "boom"}}
26+
OPENAI_ERROR = {"error": {"message": "boom", "type": "server_error", "code": None}}
27+
28+
ANTHROPIC_OK = {
29+
"id": "msg_1",
30+
"type": "message",
31+
"role": "assistant",
32+
"content": [{"type": "text", "text": "Hello"}],
33+
"usage": {"input_tokens": 10, "output_tokens": 3},
34+
}
35+
OPENAI_OK = {
36+
"id": "chatcmpl_1",
37+
"object": "chat.completion",
38+
"choices": [
39+
{"index": 0, "message": {"role": "assistant", "content": "Hello"}, "finish_reason": "stop"}
40+
],
41+
"usage": {"prompt_tokens": 10, "completion_tokens": 3, "total_tokens": 13},
42+
}
43+
44+
45+
@pytest.mark.parametrize(
46+
"body,cacheable",
47+
[
48+
(json.dumps(ANTHROPIC_OK).encode(), True),
49+
(json.dumps(OPENAI_OK).encode(), True),
50+
(json.dumps({**OPENAI_OK, "error": None}).encode(), True),
51+
(json.dumps(ANTHROPIC_ERROR).encode(), False),
52+
(json.dumps(OPENAI_ERROR).encode(), False),
53+
(b"", False),
54+
(b"opaque-bytes", True),
55+
(b"{}", False),
56+
(b"null", False),
57+
(b"[]", False),
58+
],
59+
)
60+
def test_is_cacheable_reply(body: bytes, cacheable: bool) -> None:
61+
assert _is_cacheable_reply(body) is cacheable
62+
63+
64+
def _client() -> TestClient:
65+
config = ProxyConfig(
66+
optimize=False,
67+
cache_enabled=True,
68+
rate_limit_enabled=False,
69+
cost_tracking_enabled=False,
70+
log_requests=False,
71+
ccr_inject_tool=False,
72+
ccr_handle_responses=False,
73+
ccr_context_tracking=False,
74+
image_optimize=False,
75+
)
76+
return TestClient(create_app(config))
77+
78+
79+
@pytest.mark.parametrize(
80+
"path,headers,model,error_body,ok_body",
81+
[
82+
(
83+
"/v1/messages",
84+
{"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
85+
"claude-haiku-4-5",
86+
ANTHROPIC_ERROR,
87+
ANTHROPIC_OK,
88+
),
89+
(
90+
"/v1/chat/completions",
91+
{"authorization": "Bearer test-key"},
92+
"gpt-4o-mini",
93+
OPENAI_ERROR,
94+
OPENAI_OK,
95+
),
96+
],
97+
)
98+
def test_error_200_is_not_replayed_from_cache(path, headers, model, error_body, ok_body) -> None:
99+
calls = {"n": 0}
100+
101+
with _client() as client:
102+
103+
async def _fake_retry(method, url, req_headers, body, stream=False, **kwargs): # noqa: ANN001
104+
calls["n"] += 1
105+
return httpx.Response(200, json=error_body if calls["n"] == 1 else ok_body)
106+
107+
client.app.state.proxy._retry_request = _fake_retry
108+
request = {"model": model, "max_tokens": 64, "messages": MESSAGES, "stream": False}
109+
110+
first = client.post(path, headers=headers, json=request)
111+
assert first.json() == error_body
112+
113+
# Before the fix the error body was cached: this came back from the
114+
# cache with the upstream never called a second time.
115+
second = client.post(path, headers=headers, json=request)
116+
assert calls["n"] == 2
117+
assert second.json() == ok_body
118+
119+
# A real reply is still cached.
120+
third = client.post(path, headers=headers, json=request)
121+
assert calls["n"] == 2
122+
assert third.json() == ok_body

0 commit comments

Comments
 (0)