Repository navigation
Cut streamed /v1/messages text at stop_sequences - #2486
Merged
lucasnewman merged 3 commits intoNov 3, 2026
Merged
lucasnewman merged 3 commits into
lucasnewman merged 3 commits into
Conversation
The streaming Anthropic endpoint emitted every text delta and only checked stop_sequences after generation, to report stop_reason, so the stream contained the matched sequence and everything after it while claiming it had stopped there. Non-streaming already truncated. Add StopSequenceStreamState, which holds back text that could still start a stop sequence, emits only the text before a match, and records the matched sequence; the stream loop stops reading tokens on a match.
Defer completed stop matches while an earlier or higher-priority partial match can still win. Exclude tool calls whose visible position follows the matched stop, preserving calls before the cutoff and ignoring stop text in thinking and arguments. Add regression coverage for chunk boundaries, prefix fallback, tie order, stream finalization, buffered whitespace, and response-template tool calls. Validation: 527 server tests passed, 6 skipped; Black, isort, and autoflake checks passed.
Preserve prompt-aware response parsing and reasoning-aware tool extraction alongside the streamed stop cutoff. Ignore unclosed tool examples inside reasoning when locating calls beyond the cutoff, with regression coverage. Validation: 1,285 server and tool-streaming tests passed, 23 skipped, using the main-required mlx-audio 0.6.2 in an isolated overlay. Black, isort, autoflake, and diff checks passed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix #2485 - a new
StopSequenceStreamStateafter the tool-call filter in the/v1/messagesstream loop; only text before a match is emitted, it records which sequence matched, and generation actually stops. New tests in test_server.py.AI usage disclosure: Claude Code was used to help locate the bug, write the fix and the tests, and run the test suite. I reviewed and verified the changes.