Repository navigation
Commit 1132a54
fix(proxy): do not cache error replies delivered as http 200 (#3930)
## Description
The response cache stores any non-streaming reply whose upstream status
is 200. An Anthropic- or OpenAI-compatible gateway can answer 200 with
an error object (`{"type": "error", ...}` / `{"error": {...}}`) or an
empty object, and that body was then replayed from the cache to every
matching request for the full TTL (1h by default) without the upstream
being asked again. Only a proxy restart or `POST /cache/clear`
recovered.
`SemanticCache.set` now refuses an empty body and any JSON body that is
not a non-empty object or that carries an error. Every store goes
through `set` (the Anthropic and OpenAI chat handlers are its only
callers), so one check covers both instead of a guard at each call site.
Closes #3929
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/semantic_cache.py`: new
`_is_cacheable_reply(response_body)`; `SemanticCache.set` returns
without storing when it is false. Refused: empty bytes, JSON that is not
a dict, `{}`, a dict with `"type": "error"`, a dict with a non-null
`"error"`. A reply with `"error": null` is still cached. Bytes that are
not JSON are stored as before: both handlers only store a reply they
already decoded, and existing cache tests store opaque bytes.
- `tests/test_proxy_response_cache_store_guard.py`: parametrized unit
test of `_is_cacheable_reply`, plus an end-to-end test per handler
(`/v1/messages`, `/v1/chat/completions`) with the cache on and an
upstream that answers 200 + error body once, then a good body. It
asserts the second request reaches the upstream and the good reply is
then served from cache.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
Lint, format and type checks ran on the changed files only. Pytest ran
the new file plus the existing response-cache test files, not the whole
suite.
### Test Output
Before the fix (`semantic_cache.py` at upstream/main d5318ac, plus a
pass-through `_is_cacheable_reply` stub so the new test module imports):
```text
$ python -m pytest tests/test_proxy_response_cache_store_guard.py -q -p no:cacheprovider -k replayed
E assert 1 == 2
E assert 1 == 2
FAILED tests/test_proxy_response_cache_store_guard.py::test_error_200_is_not_replayed_from_cache[/v1/messages-headers0-claude-haiku-4-5-error_body0-ok_body0]
FAILED tests/test_proxy_response_cache_store_guard.py::test_error_200_is_not_replayed_from_cache[/v1/chat/completions-headers1-gpt-4o-mini-error_body1-ok_body1]
================= 2 failed, 10 deselected, 1 warning in 9.28s ==================
```
After the fix:
```text
$ python -m pytest tests/test_proxy_response_cache_store_guard.py tests/test_proxy_semantic_cache_key.py \
tests/test_proxy_semantic_cache_key_integration.py tests/test_proxy_openai_cache_key_integration.py \
tests/test_openai_response_cache_key.py tests/test_proxy_semantic_cache_key_policy.py \
tests/test_semantic_cache_key_policy.py -q -x -p no:cacheprovider
======================== 61 passed, 1 warning in 16.55s ========================
$ python -m pytest tests/test_proxy_response_cache_replay.py tests/test_ccr_buffered_stream_signed_thinking.py \
tests/test_handler_outcome_tag_invariant.py tests/test_read_maturation_handler_nobust.py \
tests/test_observability_metrics.py -q -p no:cacheprovider
=================== 45 passed, 1 warning in 69.84s (0:01:09) ===================
$ ruff check headroom/proxy/semantic_cache.py tests/test_proxy_response_cache_store_guard.py
All checks passed!
$ ruff format --check headroom/proxy/semantic_cache.py tests/test_proxy_response_cache_store_guard.py
2 files already formatted
$ mypy headroom/proxy/semantic_cache.py
Success: no issues found in 1 source file
```
The warning is the existing `StarletteDeprecationWarning` from
`fastapi.testclient`.
## Real Behavior Proof
- Environment: macOS, Python 3.11.13. The real `headroom proxy` CLI,
booted from the source tree with the default `ProxyConfig` (cache on,
TTL 3600s, `Mode: cache`), pointed at a local stub upstream on 127.0.0.1
via `--anthropic-api-url` / `--openai-api-url`. The stub answers the
first POST per path with HTTP 200 + an error body (`{"type": "error",
"error": {"type": "overloaded_error", ...}}` for `/v1/messages`,
`{"error": {"message": "Upstream overloaded", "type": "server_error",
"code": null}}` for `/v1/chat/completions`) and every later POST with a
normal reply. The scratch venv lacks the `mcp`, `magika` and
`onnxruntime` extras, so the CLI was started through a three-line
launcher that skips `ensure_proxy_dependencies()` and then calls the
stock `headroom.cli.main.main(["proxy", ...])`. None of those modules is
on the response-cache path.
- Exact command / steps: `python launch.py --port <p>
--anthropic-api-url http://127.0.0.1:<s> --openai-api-url
http://127.0.0.1:<s>/v1`, then three identical non-streaming requests
per endpoint, then `GET /stats`. Run once with upstream/main d5318ac
and once with this branch, same venv and stub.
- Observed result: on main the stub is hit once per path and requests 2
and 3 replay the stored error from cache; on this branch request 2
reaches the stub and gets the good reply, and request 3 is served that
reply from cache (output below).
upstream/main (d5318ac): the stub is hit once per path; requests 2 and
3 get the stored error from cache.
```text
stub: /v1/messages hit #1 -> 200 error body
client: /v1/messages request 1 -> 200 in 668ms: {"type": "error",
"error": {"type": "overloaded_error", "message": "Overloaded"}}
client: /v1/messages request 2 -> 200 in 26ms: {"type": "error",
"error": {"type": "overloaded_error", "message": "Overloaded"}}
client: /v1/messages request 3 -> 200 in 22ms: {"type": "error",
"error": {"type": "overloaded_error", "message": "Overloaded"}}
stub: /v1/chat/completions hit #1 -> 200 error body
client: /v1/chat/completions request 1 -> 200 in 61ms: {"error":
{"message": "Upstream overloaded", "type": "server_error", "code":
null}}
client: /v1/chat/completions request 2 -> 200 in 12ms: {"error":
{"message": "Upstream overloaded", "type": "server_error", "code":
null}}
client: /v1/chat/completions request 3 -> 200 in 24ms: {"error":
{"message": "Upstream overloaded", "type": "server_error", "code":
null}}
proxy /stats cache: {"entries": 2, "max_entries": 1000, "total_hits": 4,
"ttl_seconds": 3600}
```
This branch: request 2 reaches the stub and gets the good reply; request
3 is served that good reply from cache.
```text
stub: /v1/messages hit #1 -> 200 error body
client: /v1/messages request 1 -> 200 in 1010ms: {"type": "error",
"error": {"type": "overloaded_error", "message": "Overloaded"}}
stub: /v1/messages hit #2 -> 200 good body
client: /v1/messages request 2 -> 200 in 304ms: {"id": "msg_1", "type":
"message", "role": "assistant", "model": "claude-haiku-4-5", "content":
[...]
client: /v1/messages request 3 -> 200 in 115ms: {"id": "msg_1", "type":
"message", "role": "assistant", "model": "claude-haiku-4-5", "content":
[...]
stub: /v1/chat/completions hit #1 -> 200 error body
client: /v1/chat/completions request 1 -> 200 in 51ms: {"error":
{"message": "Upstream overloaded", "type": "server_error", "code":
null}}
stub: /v1/chat/completions hit #2 -> 200 good body
client: /v1/chat/completions request 2 -> 200 in 21ms: {"id":
"chatcmpl_1", "object": "chat.completion", "model": "gpt-4o-mini",
"choices": [...]
client: /v1/chat/completions request 3 -> 200 in 20ms: {"id":
"chatcmpl_1", "object": "chat.completion", "model": "gpt-4o-mini",
"choices": [...]
proxy /stats cache: {"entries": 2, "max_entries": 1000, "total_hits": 2,
"ttl_seconds": 3600}
```
Proxy `PERF` lines match: on main requests 2-3 log `tok_before=0` (cache
hit) on both models; on this branch request 2 logs `tok_before=10`
(forwarded) and only request 3 logs `tok_before=0`.
- Not tested: real Anthropic/OpenAI or a real third-party gateway (stub
only, no credentials used); streaming requests (they never read or write
this cache); the full test suite (targeted files only); the empty /
unparseable 200 seen in production, because on main neither handler can
pass such a body to `set` any more (Anthropic requires `resp_json is not
None`, OpenAI raises to its 502 path), so that case is covered by the
unit test only.
## Runtime Rollout Safety
- Rollout-managed feature(s): none. This is a correctness guard in
`SemanticCache.set`, not a rollout feature.
- Minimum rollout channel: n/a (applies on every channel).
- Stable/default behavior changed: yes, narrowly. With the cache on (the
default), an HTTP 200 reply that is empty, an empty JSON object, a
non-object JSON value, or a JSON object carrying `"type": "error"` or a
non-null `"error"` is no longer stored, so the next identical request
goes upstream instead of replaying it. The reply itself is still
returned to the client unchanged. Successful replies are cached exactly
as before.
- Kill switch / disable path: none added; there is no case where
replaying a stored error is wanted. `--no-cache` still disables the
cache as a whole.
- Unsafe override required: no.
- Qualification impact: none expected. Compression, prompt-cache prefix
handling and token accounting are untouched; only which bodies the
response cache keeps changes. Cache-hit counts drop only where the hits
were replayed errors.
- Rollback path: revert this commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` - it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A.
## Additional Notes
- Documentation: N/A. No user-facing option or documented behavior
changes.
- Overlap with #3862 (partition the response cache by credential and
principal): both touch `SemanticCache.set`, but this change only adds
lines after the key computation and does not touch the signature,
`_compute_key` or `get`. A three-way merge of `semantic_cache.py`
against #3862's head is clean. The new tests call `_is_cacheable_reply`
directly and go through the handlers for the end-to-end case, so they do
not depend on whether `set`/`get` take `partition=`.
- History: the production case that surfaced this was the malformed
buffered-CCR 200 that #2997 turned into a real 502. That source is gone
on main, but a 200 with a JSON error payload from a compatible gateway
still reaches `set`, as the proof above shows.
- The check parses the body once per store. Stores only happen on
non-streaming 200 replies, so the cost is one `json.loads` per such
reply.
- Headroom Desktop has shipped an equivalent downstream patch since
2026-08-13.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3930"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
Co-authored-by: JD Davis <[email protected]>1 parent 4ba231a commit 1132a54
2 files changed
Lines changed: 151 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| 11 | + | |
11 | 12 | | |
12 | 13 | | |
13 | 14 | | |
| |||
22 | 23 | | |
23 | 24 | | |
24 | 25 | | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
25 | 51 | | |
26 | 52 | | |
27 | 53 | | |
| |||
84 | 110 | | |
85 | 111 | | |
86 | 112 | | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
87 | 116 | | |
88 | 117 | | |
89 | 118 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
0 commit comments