Repository navigation
Conversation
PR governanceThis PR does not yet satisfy the required template fields:
Please update the PR body, or move the PR back to draft while it is still in progress. |
|
Excelente gracias |
Match every blank line the SSE spec allows (CRLF, LF and CR endings mixed freely) instead of a fixed list, wait for the next byte when the terminator ends in a trailing CR, and let StreamUsageGuard split and rewrite frames on CR and CRLF line endings too, keeping each line's original ending.
JerrettDavis
left a comment
There was a problem hiding this comment.
After enabling the waiting workflow, Security confirms three vulnerabilities in aiohttp 3.14.3, fixed in 3.14.4 (CVE-2026-111064, GHSA-2g87-pp6c-29x5, CVE-2026-111063): https://github.com/headroomlabs-ai/headroom/actions/runs/39378698202. Please ensure the branch receives the vetted dependency fix before merge; this is a confirmed readiness blocker, not a request for you to run CI.
The targeted SSE byte-buffer/context-guard suites pass locally (59 tests).
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
|
Shared dependency blocker: the current-head security/audit check is failing. Existing review discussion identifies aiohttp 3.14.3 vulnerabilities fixed in 3.14.4; #4310 is the current remediation candidate. Coordinate a vetted dependency resolution and require a passing audit on this PR’s updated head; this relationship does not require choosing or merging another PR, and it does not clear separate review findings. |
JerrettDavis
left a comment
There was a problem hiding this comment.
Reviewed updated head cec6d57 and its existing discussion. It already contains current main 4c7acba and the aiohttp 3.14.4 remediation, so no base-sync commit was needed. All 59 focused SSE byte-buffer/context-guard tests pass locally; Ruff and whitespace checks pass. The mixed/CR line-ending handling and split trailing-CR guard are covered.
I authorized the waiting current-head workflows. The new Security run still fails its dependency audit: Pillow 12.3.0 has advisories fixed in 13.0.0, including CVE-2026-112895, CVE-2026-112400, and CVE-2026-112397. Evidence: https://github.com/headroomlabs-ai/headroom/actions/runs/39596761971 . This is a different dependency blocker from the now-remediated aiohttp findings. Please use a compatible, vetted dependency resolution before merge; do not force a Pillow version that violates the embedder dependency constraints. No contributor CI run is requested; maintainer-authorized validation supplied this result. The existing change request remains necessary until the security gate is resolved. No contributor logic was modified.
|
Thanks for checking the head. I tried the Pillow fix: raising the floor to |
JerrettDavis
left a comment
There was a problem hiding this comment.
Reviewed the new head, author runtime evidence, and all three Devin findings, then cleanly synced current main without changing contributor logic. Exact head 6aba22f passes all 475 focused byte-buffer/context-guard/memory-stream/malformed/UTF-8 tests plus changed-file Ruff, formatting, and whitespace checks. The earlier CR-line parsing and mixed-terminator cases are addressed, and the body now includes observed behavior.
One uncovered EOF case remains in the new trailing-CR handling. I served a real HTTP SSE response containing event: message_start\rdata: {"type":"message_start","message":{"usage":{"input_tokens":185000}}}\r\r, with Content-Length and a clean EOF, and delivered it through httpx one byte at a time. The CR stream produced zero parsed events with 102 bytes retained; StreamUsageGuard(believed_limit=200000, effective_limit=200000).flush() returned the frame with 185000 unchanged. An otherwise identical LF control produced one event, no residual bytes, and the expected 190000 usage nudge. find_sse_event_terminator deliberately waits after a final CR, but neither the parser nor the guard resolves that ambiguity at EOF; the streaming finalizer clears the residual parser buffer and guard.flush forwards it without processing.
Please add EOF-aware draining/processing for complete CR-terminated blank lines while preserving protection for CRLF split across chunks and retaining fail-open handling of genuinely incomplete events/UTF-8. Cover final message_start and usage/message_delta frames ending with CR+CR, LF+CR, and CRLF+CR, with whole-buffer and byte-at-a-time delivery and no synthetic extra event. This is a remaining correctness case for the newly supported framing, not an inherited dependency-audit hold. Maintainers handle hosted qualification; no contributor CI run is requested.
There was a problem hiding this comment.
Current-main requalification of exact published head 70710b1.
Clean integration against main 0ddaa56; contributor parsing policy unchanged. 222 focused context-guard/SSE and incoming-main tests passed, 11 optional skips, 1 warning. Incoming-main patches match exactly; lint/format, offline lock and whitespace checks pass. Existing native DLL reused only after complete input-set/hash and artifact checksum verification; no fresh native build claimed.
Fresh loopback TCP/httpx byte-at-a-time SSE probe still reproduces the EOF gap: LF control releases one event, leaves zero buffered bytes and nudges input_tokens from185000 to190000. CR-only frame ending in CR CR at connection close releases zero events, retains102bytes and guard.flush forwards unchanged185000. Please support finalizing a valid trailing CR blank-line at EOF without sacrificing split-CRLF handling, and cover parser plus guard EOF behavior. This is local streamed transport evidence, not a live provider/client proof.
Read author discussion and all three Devin threads/replies. CR-line splitting and mixed-terminator fixes are present; resolved flags do not eliminate the separate EOF gap. Author's dependency constraints/audit response is acknowledged; no unrelated dependency upgrade or contributor CI request is made. Source/test-only delta excludes workflow-validation installer and Init/bootstrap paths; no workflow/setup/dependency/CLI changes. Nothing merged.
Original review commit metadata is historical; this body records refreshed evidence. Hosted exact-head checks must still be revisited.
find_sse_event_terminator holds back an event whose blank line ends in the last byte of the buffer when that byte is a CR, because a LF may follow and belong to the same CRLF. At end of stream nothing follows, so such an event was never released: StreamUsageGuard.flush() forwarded it unread and the parser left it in the buffer. Add at_eof to find_sse_event_terminator and parse_sse_events_from_byte_buffer (and the helpers wrapper), and have StreamUsageGuard.flush() process complete events with at_eof=True before returning the rest verbatim. Split-CRLF handling during feed() and unterminated / incomplete-UTF-8 tails are unchanged.
|
Thanks for the byte-at-a-time probe, and sorry this sat for a few days. Pushed c417e66 for the EOF gap.
I didn't wire |
JerrettDavis
left a comment
There was a problem hiding this comment.
Changes required before this PR can be approved.
Reviewed head c417e6660b3b4ac731d2c4cedd131707e6f7c274 against main c05fbdddc873078dd97cb9ede841e75ef7ed0bef.
Important — headroom/proxy/helpers.py:1472
Complete the existing EOF fix in the streaming handler. The new at_eof option is exposed but no production parser caller enables it. A final valid CR-terminated blank-line event therefore remains unavailable to usage accounting and completion detection, even though the client-bound guard now processes it correctly. Preserve incomplete-event and UTF-8 handling while draining complete events at EOF.
Qualification still required:
- Unresolved findings require follow-up
- Exact-source provenance review is incomplete
Sync PR headroomlabs-ai#4307 with latest main.
JerrettDavis
left a comment
There was a problem hiding this comment.
The existing EOF integration request remains unresolved at ca2b231. A final valid event such as event: message_stop\rdata: {"type":"message_stop"}\r\r stays buffered because the streaming handler calls the parser with its default at_eof=False; finalization then clears the buffer. Expected: process the complete terminal event at clean EOF. Actual: terminal detection and final usage accounting miss trailing-CR events, although the client-bound guard now processes them. Please wire EOF-aware draining through the production handler and cover final usage/terminal events using CR+CR, LF+CR and CRLF+CR under full and bytewise delivery, preserving partial-event and UTF-8 safety. This is the previously documented finding, not a new defect. Fresh focused offline policy/guard checks pass; they do not exercise handler finalization. Inherited workflow provenance issues are separate and no contributor CI action is requested.
Description
In the SSE spec an event ends at a blank line, and every line may end in CRLF, LF or CR independently (WHATWG §9.2.5).
find_sse_event_terminatoronly looked for\n\nand\r\n\r\n, so events framed with any other mix (\n\r\n,\n\r,\r\n\r,\r\r\n,\r\r) were never seen as complete. They stayed in the buffer until the next\n\nor the buffer cap, which held the stream back inStreamUsageGuard(context_guard.py), the memory-tool stream filter and the streaming handler's byte-buffer parser.No linked issue; found by reading the parser against the spec.
Type of Change
Changes Made
sse_byte_buffer_policy.find_sse_event_terminatornow matches any pair of line endings with one regex,(?:\r\n|\n|\r(?!\n))(?:\r\n|\n|\r). A CR directly followed by LF is always one CRLF ending. When the earliest terminator ends in a CR that is the last byte of the buffer, it returnsNoneand waits for the next byte, because that byte may be the LF of a CRLF that belongs to the same terminator.SSE_EVENT_TERMINATORSis left as it was onmain.context_guard.StreamUsageGuardsplits frames on CRLF, LF and CR (_split_sse_lines), and themessage_start/message_deltarewrites put each line's original ending back. Before this, a CR-framedmessage_startwas seen as one line, so the rewrite dropped its\r\rterminator and the client got malformed SSE (this is the issue Devin raised on the first version of this PR).tests/test_sse_byte_buffer_policy.pyandtests/test_proxy/test_context_guard.py.Testing
pytest)ruff check .)mypy headroom)ruff check/ruff format --checkare clean on the four touched files.mypy headroom/proxy/sse_byte_buffer_policy.py headroom/proxy/context_guard.pyreports one error, inheadroom/providers/model_metadata.py:79, which is the same onmain.New tests, run against
main's two source files (tests from this PR): 9 failures (test_parse_sse_events_accepts_mixed_and_cr_only_line_endingsx3,test_find_sse_event_terminator_waits_for_the_byte_after_a_trailing_crx3,test_cr_framed_events_are_released_and_rewritten,test_armed_guard_rewrites_a_cr_framed_message_delta, plus one unrelated environment failure noted below). With the fix:Test Output
Real Behavior Proof
de405d9) imported from source, no provider calls.message_startending in\r\r, a ping ending in\n\r, a content delta ending in\r\n\r,message_stopending in\r\r\n), runs it throughheadroom.proxy.helpers.parse_sse_events_from_byte_buffer, then feeds it in 7-byte chunks throughStreamUsageGuard(believed_limit=1_000_000, effective_limit=200_000)and flushes.On
mainthe same script prints no parsed events, leaves all 354 bytes in the buffer, and the guard output still reportsinput_tokens -> 190000(nothing was framed, so nothing was nudged until the flush).Runtime Rollout Safety
HEADROOM_CONTEXT_GUARD=0still disables the guard; the parser change has noneReview Readiness
Checklist
CHANGELOG.md— it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this)Additional Notes
AI-assistance note: I found and wrote this change with help from an AI coding assistant. I reviewed it and ran the tests and the script above myself.
Maintainer current-main requalification
Exact head
b145b4101df21e1761a57cf1b716c44c286204a6, upstream mainc01cf07d943d816cfd9d97e6838ed4e38f1ff9a4.Current-main integration with upstream c01cf07. The sole conflict was adjacent tests: retained the contributor's CR/mixed-ending regressions and main's Unicode-line-separator/one-leading-space regressions. No parser or context-guard logic rewritten. Fresh focused byte-buffer/context-guard/memory-stream tests: 127 passed, one test-client deprecation warning. Four changed files pass Ruff lint/format and main-relative whitespace checks. Current-main DLL used for Python startup, without a branch-specific native-build claim.
The prior clean-EOF framing finding still reproduces over a real local HTTP SSE response delivered one byte at a time: LF control parses one event, leaves no residual bytes, and nudges 185000 to 190000 input tokens. CR-only clean EOF parses zero events, leaves 102 buffered bytes, and forwards 185000 unchanged. Please drain/process complete trailing-CR blank-line events at EOF while preserving split-CRLF and incomplete-UTF-8 behavior. Cover CR+CR, LF+CR and CRLF+CR under full and bytewise delivery. This is the remaining framing-correctness request, not an inherited pipeline issue. Maintainers handle hosted checks; no contributor CI action requested. Nothing merged.
Exact-head hosted checks are being handled by maintainers. Existing correctness/qualification holds remain; nothing merged.
Maintainer current-main requalification
Exact head
70710b18abea32f74424facdc3d0ef78b0dd0c77, upstream main0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c.Clean integration against main 0ddaa56; contributor parsing policy unchanged. 222 focused context-guard/SSE and incoming-main tests passed, 11 optional skips, 1 warning. Incoming-main patches match exactly; lint/format, offline lock and whitespace checks pass. Existing native DLL reused only after complete input-set/hash and artifact checksum verification; no fresh native build claimed.
Fresh loopback TCP/httpx byte-at-a-time SSE probe still reproduces the EOF gap: LF control releases one event, leaves zero buffered bytes and nudges input_tokens from185000 to190000. CR-only frame ending in CR CR at connection close releases zero events, retains102bytes and guard.flush forwards unchanged185000. Please support finalizing a valid trailing CR blank-line at EOF without sacrificing split-CRLF handling, and cover parser plus guard EOF behavior. This is local streamed transport evidence, not a live provider/client proof.
Read author discussion and all three Devin threads/replies. CR-line splitting and mixed-terminator fixes are present; resolved flags do not eliminate the separate EOF gap. Author's dependency constraints/audit response is acknowledged; no unrelated dependency upgrade or contributor CI request is made. Source/test-only delta excludes workflow-validation installer and Init/bootstrap paths; no workflow/setup/dependency/CLI changes. Nothing merged.
Exact-head hosted checks are being handled by maintainers. Existing correctness/qualification holds remain; nothing merged.