Skip to content

fix(proxy): recognise mixed and CR-only SSE event terminators - #4307

Open
kevin9327 wants to merge 11 commits into
headroomlabs-ai:mainfrom
kevin9327:fix/sse-mixed-line-endings
Open

kevin9327 wants to merge 11 commits into
headroomlabs-ai:mainfrom
kevin9327:fix/sse-mixed-line-endings

Conversation

@kevin9327

@kevin9327 kevin9327 commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Description

In the SSE spec an event ends at a blank line, and every line may end in CRLF, LF or CR independently (WHATWG §9.2.5). find_sse_event_terminator only looked for \n\n and \r\n\r\n, so events framed with any other mix (\n\r\n, \n\r, \r\n\r, \r\r\n, \r\r) were never seen as complete. They stayed in the buffer until the next \n\n or the buffer cap, which held the stream back in StreamUsageGuard (context_guard.py), the memory-tool stream filter and the streaming handler's byte-buffer parser.

No linked issue; found by reading the parser against the spec.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update
  • Performance improvement
  • Code refactoring (no functional changes)

Changes Made

  • sse_byte_buffer_policy.find_sse_event_terminator now matches any pair of line endings with one regex, (?:\r\n|\n|\r(?!\n))(?:\r\n|\n|\r). A CR directly followed by LF is always one CRLF ending. When the earliest terminator ends in a CR that is the last byte of the buffer, it returns None and waits for the next byte, because that byte may be the LF of a CRLF that belongs to the same terminator. SSE_EVENT_TERMINATORS is left as it was on main.
  • context_guard.StreamUsageGuard splits frames on CRLF, LF and CR (_split_sse_lines), and the message_start / message_delta rewrites put each line's original ending back. Before this, a CR-framed message_start was seen as one line, so the rewrite dropped its \r\r terminator and the client got malformed SSE (this is the issue Devin raised on the first version of this PR).
  • Tests in tests/test_sse_byte_buffer_policy.py and tests/test_proxy/test_context_guard.py.

Testing

  • Unit tests pass (pytest)
  • Linting passes (ruff check .)
  • Type checking passes (mypy headroom)
  • New tests added for new functionality
  • Manual testing performed

ruff check / ruff format --check are clean on the four touched files. mypy headroom/proxy/sse_byte_buffer_policy.py headroom/proxy/context_guard.py reports one error, in headroom/providers/model_metadata.py:79, which is the same on main.

New tests, run against main's two source files (tests from this PR): 9 failures (test_parse_sse_events_accepts_mixed_and_cr_only_line_endings x3, test_find_sse_event_terminator_waits_for_the_byte_after_a_trailing_cr x3, test_cr_framed_events_are_released_and_rewritten, test_armed_guard_rewrites_a_cr_framed_message_delta, plus one unrelated environment failure noted below). With the fix:

Test Output

$ pytest -q tests/test_sse_byte_buffer_policy.py tests/test_proxy/test_context_guard.py
======================== 59 passed, 1 warning in 0.89s =========================
$ pytest -q tests/test_sse_byte_buffer_policy.py tests/test_proxy/test_context_guard.py tests/test_memory_tool_stream.py tests/test_streaming_sse_malformed_events.py tests/test_sse_utf8_split.py
1 failed, 472 passed
# the one failure is TestHandlerPassesServerMemoryTools::test_injected_memory_tools_reach_the_stream,
# which also fails on main in my environment (the Rust extension headroom._core is not built here)

Real Behavior Proof

  • Environment: Linux 6.12, Python 3.13.5, headroom at this branch (de405d9) imported from source, no provider calls.
  • Exact command / steps: a script that builds an Anthropic-style stream mixing CR, LF and CRLF framing (message_start ending in \r\r, a ping ending in \n\r, a content delta ending in \r\n\r, message_stop ending in \r\r\n), runs it through headroom.proxy.helpers.parse_sse_events_from_byte_buffer, then feeds it in 7-byte chunks through StreamUsageGuard(believed_limit=1_000_000, effective_limit=200_000) and flushes.
  • Observed result:
parse_sse_events_from_byte_buffer ->
   message_start {"type": "message_start", "message": {"id": "msg_1", "type":
   content_block_delta {"type":"content_block_delta"}
   message_stop {"type":"message_stop"}
left in buffer: b''
StreamUsageGuard output:
   b'event: message_start\rdata: {"type":"mess' ... b'lta"}\r\n\revent: message_stop\rdata: {"type":"message_stop"}\r\r\n'
   message_start input_tokens -> 950000

On main the same script prints no parsed events, leaves all 354 bytes in the buffer, and the guard output still reports input_tokens -> 190000 (nothing was framed, so nothing was nudged until the flush).

  • Not tested: a live upstream that actually sends CR-only framing (Anthropic and OpenAI send LF), and the full proxy end to end.

Runtime Rollout Safety

  • Rollout-managed feature(s): none
  • Minimum rollout channel: n/a
  • Stable/default behavior changed: only for streams using mixed or CR-only line endings, which are now framed at their blank lines; LF and CRLF streams are framed exactly as before (existing tests unchanged and passing)
  • Kill switch / disable path: HEADROOM_CONTEXT_GUARD=0 still disables the guard; the parser change has none
  • Unsafe override required: no
  • Qualification impact: none expected
  • Rollback path: revert this PR

Review Readiness

  • I have performed a self-review
  • This PR is ready for human review

Checklist

  • My code follows the project's style guidelines
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation (n/a, internal parser)
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I did not edit CHANGELOG.md — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this)

Additional Notes

AI-assistance note: I found and wrote this change with help from an AI coding assistant. I reviewed it and ran the tests and the script above myself.

Maintainer current-main requalification

Exact head b145b4101df21e1761a57cf1b716c44c286204a6, upstream main c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4.

Current-main integration with upstream c01cf07. The sole conflict was adjacent tests: retained the contributor's CR/mixed-ending regressions and main's Unicode-line-separator/one-leading-space regressions. No parser or context-guard logic rewritten. Fresh focused byte-buffer/context-guard/memory-stream tests: 127 passed, one test-client deprecation warning. Four changed files pass Ruff lint/format and main-relative whitespace checks. Current-main DLL used for Python startup, without a branch-specific native-build claim.

The prior clean-EOF framing finding still reproduces over a real local HTTP SSE response delivered one byte at a time: LF control parses one event, leaves no residual bytes, and nudges 185000 to 190000 input tokens. CR-only clean EOF parses zero events, leaves 102 buffered bytes, and forwards 185000 unchanged. Please drain/process complete trailing-CR blank-line events at EOF while preserving split-CRLF and incomplete-UTF-8 behavior. Cover CR+CR, LF+CR and CRLF+CR under full and bytewise delivery. This is the remaining framing-correctness request, not an inherited pipeline issue. Maintainers handle hosted checks; no contributor CI action requested. Nothing merged.

Exact-head hosted checks are being handled by maintainers. Existing correctness/qualification holds remain; nothing merged.

Maintainer current-main requalification

Exact head 70710b18abea32f74424facdc3d0ef78b0dd0c77, upstream main 0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c.

Clean integration against main 0ddaa56; contributor parsing policy unchanged. 222 focused context-guard/SSE and incoming-main tests passed, 11 optional skips, 1 warning. Incoming-main patches match exactly; lint/format, offline lock and whitespace checks pass. Existing native DLL reused only after complete input-set/hash and artifact checksum verification; no fresh native build claimed.

Fresh loopback TCP/httpx byte-at-a-time SSE probe still reproduces the EOF gap: LF control releases one event, leaves zero buffered bytes and nudges input_tokens from185000 to190000. CR-only frame ending in CR CR at connection close releases zero events, retains102bytes and guard.flush forwards unchanged185000. Please support finalizing a valid trailing CR blank-line at EOF without sacrificing split-CRLF handling, and cover parser plus guard EOF behavior. This is local streamed transport evidence, not a live provider/client proof.

Read author discussion and all three Devin threads/replies. CR-line splitting and mixed-terminator fixes are present; resolved flags do not eliminate the separate EOF gap. Author's dependency constraints/audit response is acknowledged; no unrelated dependency upgrade or contributor CI request is made. Source/test-only delta excludes workflow-validation installer and Init/bootstrap paths; no workflow/setup/dependency/CLI changes. Nothing merged.

Exact-head hosted checks are being handled by maintainers. Existing correctness/qualification holds remain; nothing merged.

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

PR governance

This PR does not yet satisfy the required template fields:

  • Fill in Real Behavior Proof → Observed result.

Please update the PR body, or move the PR back to draft while it is still in progress.

@github-actions github-actions Bot added the status: needs author action Pull request body or readiness checklist still needs author updates label Oct 20, 2026

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 3 potential issues.

Devin Review

Comment thread tests/test_sse_byte_buffer_policy.py Resolved
@ghost

ghost commented Oct 20, 2026

Copy link
Copy Markdown

Excelente gracias

Match every blank line the SSE spec allows (CRLF, LF and CR endings mixed
freely) instead of a fixed list, wait for the next byte when the terminator
ends in a trailing CR, and let StreamUsageGuard split and rewrite frames on
CR and CRLF line endings too, keeping each line's original ending.

@JerrettDavis JerrettDavis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After enabling the waiting workflow, Security confirms three vulnerabilities in aiohttp 3.14.3, fixed in 3.14.4 (CVE-2026-111064, GHSA-2g87-pp6c-29x5, CVE-2026-111063): https://github.com/headroomlabs-ai/headroom/actions/runs/39378698202. Please ensure the branch receives the vetted dependency fix before merge; this is a confirmed readiness blocker, not a request for you to run CI.

The targeted SSE byte-buffer/context-guard suites pass locally (59 tests).

@codecov-commenter

codecov-commenter commented Oct 20, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 97.29730% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
headroom/proxy/context_guard.py 96.29% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@JerrettDavis JerrettDavis added area: proxy Touches proxy files; triage routing, not a CODEOWNERS assignment. bug Something isn't working impact: security Security-sensitive paths or a documented security dependency blocker. python Pull requests that update python code risk: high Scope estimate: security, dependencies, release, broad or critical runtime change. size: medium 101–500 added/deleted lines. status: blocked Unresolved checks, review findings, integration or dependency gate; see discussion. status: ci failing Required or reported CI checks are failing status: code changes requested status: waiting on dependency Explicit stacked parent or vetted dependency update still needed. labels Oct 20, 2026
@JerrettDavis

Copy link
Copy Markdown
Collaborator

Shared dependency blocker: the current-head security/audit check is failing. Existing review discussion identifies aiohttp 3.14.3 vulnerabilities fixed in 3.14.4; #4310 is the current remediation candidate. Coordinate a vetted dependency resolution and require a passing audit on this PR’s updated head; this relationship does not require choosing or merging another PR, and it does not clear separate review findings.

@JerrettDavis JerrettDavis added the status: awaiting author Author work on code, checks, integration or PR-body requirements remains. label Oct 20, 2026
@github-actions github-actions Bot removed the status: ci failing Required or reported CI checks are failing label Oct 21, 2026

@JerrettDavis JerrettDavis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed updated head cec6d57 and its existing discussion. It already contains current main 4c7acba and the aiohttp 3.14.4 remediation, so no base-sync commit was needed. All 59 focused SSE byte-buffer/context-guard tests pass locally; Ruff and whitespace checks pass. The mixed/CR line-ending handling and split trailing-CR guard are covered.

I authorized the waiting current-head workflows. The new Security run still fails its dependency audit: Pillow 12.3.0 has advisories fixed in 13.0.0, including CVE-2026-112895, CVE-2026-112400, and CVE-2026-112397. Evidence: https://github.com/headroomlabs-ai/headroom/actions/runs/39596761971 . This is a different dependency blocker from the now-remediated aiohttp findings. Please use a compatible, vetted dependency resolution before merge; do not force a Pillow version that violates the embedder dependency constraints. No contributor CI run is requested; maintainer-authorized validation supplied this result. The existing change request remains necessary until the security gate is resolved. No contributor logic was modified.

@kevin9327

Copy link
Copy Markdown
Contributor Author

Thanks for checking the head. I tried the Pillow fix: raising the floor to pillow>=13.0.0 and running uv lock --upgrade-package pillow has no solution, because every fastembed release in the [all] extra pins pillow<13.0 (fastembed>=0.8.0 requires pillow>=12.0.0,<13.0). So 13.0.0 can't be used here without breaking that constraint, which you asked me not to do. The same audit run also flags certifi 2026.1.4 and the Rust crates rustls-webpki (RUSTSEC-2026-0382) and webpki-roots (RUSTSEC-2026-0381); those come from main's lockfiles and don't touch this change. I've left the dependencies alone in this PR. If you'd like the certifi and Rust lock bumps done in a separate PR against main, I can open one. I also renamed the proof line to Observed result: so the governance check can find it.

@github-actions github-actions Bot added the status: ci failing Required or reported CI checks are failing label Oct 22, 2026
@github-actions github-actions Bot removed the status: ci failing Required or reported CI checks are failing label Nov 4, 2026

@JerrettDavis JerrettDavis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the new head, author runtime evidence, and all three Devin findings, then cleanly synced current main without changing contributor logic. Exact head 6aba22f passes all 475 focused byte-buffer/context-guard/memory-stream/malformed/UTF-8 tests plus changed-file Ruff, formatting, and whitespace checks. The earlier CR-line parsing and mixed-terminator cases are addressed, and the body now includes observed behavior.

One uncovered EOF case remains in the new trailing-CR handling. I served a real HTTP SSE response containing event: message_start\rdata: {"type":"message_start","message":{"usage":{"input_tokens":185000}}}\r\r, with Content-Length and a clean EOF, and delivered it through httpx one byte at a time. The CR stream produced zero parsed events with 102 bytes retained; StreamUsageGuard(believed_limit=200000, effective_limit=200000).flush() returned the frame with 185000 unchanged. An otherwise identical LF control produced one event, no residual bytes, and the expected 190000 usage nudge. find_sse_event_terminator deliberately waits after a final CR, but neither the parser nor the guard resolves that ambiguity at EOF; the streaming finalizer clears the residual parser buffer and guard.flush forwards it without processing.

Please add EOF-aware draining/processing for complete CR-terminated blank lines while preserving protection for CRLF split across chunks and retaining fail-open handling of genuinely incomplete events/UTF-8. Cover final message_start and usage/message_delta frames ending with CR+CR, LF+CR, and CRLF+CR, with whole-buffer and byte-at-a-time delivery and no synthetic extra event. This is a remaining correctness case for the newly supported framing, not an inherited dependency-audit hold. Maintainers handle hosted qualification; no contributor CI run is requested.

@github-actions github-actions Bot added status: needs rebase Pull request branch is behind the base branch on files it also changes status: has conflicts Pull request has merge conflicts with the base branch labels Nov 5, 2026
@github-actions github-actions Bot removed status: needs rebase Pull request branch is behind the base branch on files it also changes status: has conflicts Pull request has merge conflicts with the base branch labels Nov 6, 2026

@JerrettDavis JerrettDavis left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current-main requalification of exact published head 70710b1.

Clean integration against main 0ddaa56; contributor parsing policy unchanged. 222 focused context-guard/SSE and incoming-main tests passed, 11 optional skips, 1 warning. Incoming-main patches match exactly; lint/format, offline lock and whitespace checks pass. Existing native DLL reused only after complete input-set/hash and artifact checksum verification; no fresh native build claimed.

Fresh loopback TCP/httpx byte-at-a-time SSE probe still reproduces the EOF gap: LF control releases one event, leaves zero buffered bytes and nudges input_tokens from185000 to190000. CR-only frame ending in CR CR at connection close releases zero events, retains102bytes and guard.flush forwards unchanged185000. Please support finalizing a valid trailing CR blank-line at EOF without sacrificing split-CRLF handling, and cover parser plus guard EOF behavior. This is local streamed transport evidence, not a live provider/client proof.

Read author discussion and all three Devin threads/replies. CR-line splitting and mixed-terminator fixes are present; resolved flags do not eliminate the separate EOF gap. Author's dependency constraints/audit response is acknowledged; no unrelated dependency upgrade or contributor CI request is made. Source/test-only delta excludes workflow-validation installer and Init/bootstrap paths; no workflow/setup/dependency/CLI changes. Nothing merged.

Original review commit metadata is historical; this body records refreshed evidence. Hosted exact-head checks must still be revisited.

JerrettDavis and others added 3 commits November 6, 2026 14:42
find_sse_event_terminator holds back an event whose blank line ends in the
last byte of the buffer when that byte is a CR, because a LF may follow and
belong to the same CRLF. At end of stream nothing follows, so such an event
was never released: StreamUsageGuard.flush() forwarded it unread and the
parser left it in the buffer.

Add at_eof to find_sse_event_terminator and parse_sse_events_from_byte_buffer
(and the helpers wrapper), and have StreamUsageGuard.flush() process complete
events with at_eof=True before returning the rest verbatim. Split-CRLF
handling during feed() and unterminated / incomplete-UTF-8 tails are
unchanged.
@kevin9327

Copy link
Copy Markdown
Contributor Author

Thanks for the byte-at-a-time probe, and sorry this sat for a few days. Pushed c417e66 for the EOF gap.

  • find_sse_event_terminator and parse_sse_events_from_byte_buffer (and the helpers wrapper) take at_eof=False. With at_eof=True a blank line whose last byte is a CR counts as complete, since no LF can follow any more. Without it nothing changes, so a CR at the end of a chunk still waits for a possible LF.
  • StreamUsageGuard.flush() now runs the complete events left in the buffer through _process_event with at_eof=True, then returns whatever is left unchanged. A CR-only message_start ending in CR CR that closes the connection now gets nudged (185000 -> 190000 in the new test) and keeps its CR CR ending.
  • Tests: CR+CR, LF+CR and CRLF+CR, each with full and bytewise delivery, for both the parser (held back before EOF, drained at EOF) and the guard (feed returns nothing, flush returns the rewritten frame, and the guard is inert afterwards). Also: a tail with no terminator, or cut inside a UTF-8 character, stays in the buffer at EOF, and flush() returns a partial CRLF event ending in a lone CR exactly as it came in.

pytest tests/test_sse_byte_buffer_policy.py tests/test_proxy/test_context_guard.py: 81 passed. With the memory-stream, malformed-event and UTF-8-split suites added it's 497 passed. The 3 failures in TestHandlerPassesServerMemoryTools fail the same way without this commit, because uvicorn isn't installed in my env. Ruff check and format pass on the five touched files.

I didn't wire at_eof=True into the streaming handler's own end-of-stream path. Let me know if you want that in this PR too.

@JerrettDavis JerrettDavis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes required before this PR can be approved.

Reviewed head c417e6660b3b4ac731d2c4cedd131707e6f7c274 against main c05fbdddc873078dd97cb9ede841e75ef7ed0bef.

Important — headroom/proxy/helpers.py:1472

Complete the existing EOF fix in the streaming handler. The new at_eof option is exposed but no production parser caller enables it. A final valid CR-terminated blank-line event therefore remains unavailable to usage accounting and completion detection, even though the client-bound guard now processes it correctly. Preserve incomplete-event and UTF-8 handling while draining complete events at EOF.

Qualification still required:

  • Unresolved findings require follow-up
  • Exact-source provenance review is incomplete

@JerrettDavis JerrettDavis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The existing EOF integration request remains unresolved at ca2b231. A final valid event such as event: message_stop\rdata: {"type":"message_stop"}\r\r stays buffered because the streaming handler calls the parser with its default at_eof=False; finalization then clears the buffer. Expected: process the complete terminal event at clean EOF. Actual: terminal detection and final usage accounting miss trailing-CR events, although the client-bound guard now processes them. Please wire EOF-aware draining through the production handler and cover final usage/terminal events using CR+CR, LF+CR and CRLF+CR under full and bytewise delivery, preserving partial-event and UTF-8 safety. This is the previously documented finding, not a new defect. Fresh focused offline policy/guard checks pass; they do not exercise handler finalization. Inherited workflow provenance issues are separate and no contributor CI action is requested.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: proxy Touches proxy files; triage routing, not a CODEOWNERS assignment. bug Something isn't working impact: security Security-sensitive paths or a documented security dependency blocker. python Pull requests that update python code risk: high Scope estimate: security, dependencies, release, broad or critical runtime change. size: medium 101–500 added/deleted lines. status: awaiting author Author work on code, checks, integration or PR-body requirements remains. status: blocked Unresolved checks, review findings, integration or dependency gate; see discussion. status: code changes requested status: needs author action Pull request body or readiness checklist still needs author updates status: waiting on dependency Explicit stacked parent or vetted dependency update still needed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants