Skip to content

Cut streamed /v1/messages text at stop_sequences - #2486

Merged
lucasnewman merged 3 commits into
Neywa-Labs:mainfrom
AkshayShah03:fix-anthropic-stream-stop-sequences
Nov 3, 2026
Merged

lucasnewman merged 3 commits into
Neywa-Labs:mainfrom
AkshayShah03:fix-anthropic-stream-stop-sequences

Conversation

@AkshayShah03

Copy link
Copy Markdown
Contributor

Fix #2485 - a new StopSequenceStreamState after the tool-call filter in the /v1/messages stream loop; only text before a match is emitted, it records which sequence matched, and generation actually stops. New tests in test_server.py.

AI usage disclosure: Claude Code was used to help locate the bug, write the fix and the tests, and run the test suite. I reviewed and verified the changes.

The streaming Anthropic endpoint emitted every text delta and only
checked stop_sequences after generation, to report stop_reason, so the
stream contained the matched sequence and everything after it while
claiming it had stopped there. Non-streaming already truncated.

Add StopSequenceStreamState, which holds back text that could still
start a stop sequence, emits only the text before a match, and records
the matched sequence; the stream loop stops reading tokens on a match.
Defer completed stop matches while an earlier or higher-priority partial match can still win. Exclude tool calls whose visible position follows the matched stop, preserving calls before the cutoff and ignoring stop text in thinking and arguments.

Add regression coverage for chunk boundaries, prefix fallback, tie order, stream finalization, buffered whitespace, and response-template tool calls. Validation: 527 server tests passed, 6 skipped; Black, isort, and autoflake checks passed.
Preserve prompt-aware response parsing and reasoning-aware tool extraction alongside the streamed stop cutoff. Ignore unclosed tool examples inside reasoning when locating calls beyond the cutoff, with regression coverage.

Validation: 1,285 server and tool-streaming tests passed, 23 skipped, using the main-required mlx-audio 0.6.2 in an isolated overlay. Black, isort, autoflake, and diff checks passed.

@lucasnewman lucasnewman left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lucasnewman
lucasnewman merged commit 12c4d7d into Neywa-Labs:main Nov 3, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Streaming /v1/messages sends text past stop_sequences

2 participants