fix(parsers): re-jail a complete start marker in trailing content - #408

Fixes ai-dynamo/dynamo#16383 (for the gemma4 tool-call jail; see Scope below).

Problem

With parallel tool calls and multi-token deltas (for example under speculative decoding), one delta can close call A and carry the full start marker of call B:

"<|tool_call>call", ":f{", "a:<|\"|>1<|\"|>",
"}<tool_call|><|tool_call>ca",
"ll:g{", "b:", "<|\"|>2<|\"|>}<tool_call|>"

After call A completes, the rest of that delta (<|tool_call>ca) goes to handle_trailing_content. That function only handled MatchResult::Partial. A MatchResult::Complete fell through to split_partial_tool_call_start, and the gemma4 call-prefix split took ca as a partial call: prefix. It emitted <|tool_call> as content and held only ca. On main the stream above returns both calls but content is "<|tool_call>". With other chunkings, a whole call ends up in content and is missing from tool_calls. For example, ["<", "|tool_call>call:f{…}<tool_call|><|tool_call>c", "all:g{…}<tool_call|><|tool_call>call:h{…}<tool_call|>"] returns only f and h.

Fix

handle_trailing_content now handles MatchResult::Complete the same way process_content does. Prose before the marker is emitted as trailing content, unless it carries harmony protocol, in which case it is jailed with the marker, as in process_content. The jail then restarts from the marker. The Partial and None paths are unchanged.

Tests

• test_gemma4_next_call_start_marker_in_trailing_content_is_jailed in parsers/v1/tests/jail.rs. It runs the exact delta sequence from the issue, and then every 3-chunk split of a 3-call gemma4 stream where the middle chunk spans the end of call g and the start of call h. Each run must return f, g, h with empty content. It fails on main (left: "<|tool_call>") and passes with the fix.
• cargo test -p dynamo-parsers: all suites pass (974 lib tests, 84 in jail).
• cargo clippy -p dynamo-parsers --all-targets -- -D warnings and cargo fmt --all -- --check are clean.

I also ran a local sweep over every 3-chunk split of that 3-call gemma4 stream (8,911 chunkings), comparing each against the single-chunk result. 1,263 differed on main and 0 differ with this change. That sweep is not in the PR because it is slow.

Scope

The issue also lists llama3_json and pythonic. The same sweep shows chunking-dependent differences for those parsers, but this change doesn't affect them: the counts are the same before and after. They come from other paths (the ; separator between llama3_json calls and back-to-back pythonic bracket groups), so I left them out of this PR rather than guess at the intended output.

This needs a dynamo-parsers release and a bump in ai-dynamo/dynamo to reach the frontend.