fix(litellm): surface content-filter refusals instead of an empty turn - #3769

Thanks @seratch — added the streaming path in da268a9.

In ChatCmplStreamHandler.handle_stream, when choice 0 terminates with finish_reason == "content_filter" and no real text / refusal / tool call was emitted, it now synthesizes the refusal through the normal event sequence (output_item.added → content_part.added → refusal.delta, finalized by the existing done/completed path). Previously-emitted text or tool calls are left intact.

One real-world wrinkle I verified against a live Bedrock claude-* refusal stream: LiteLLM opens an empty-string ("") text content part before the terminal content_filter chunk, so the guard checks the accumulated text is empty (not that the text part is absent) and drops that empty part so the completed message carries only the ResponseOutputRefusal.

Added two streaming regression tests: the synthesize case (using the real empty-""-then-content_filter chunk shape) and a does-not-clobber case (real text before a late content_filter stands). The synthesize test fails against a naive "part absent" guard, so it pins that regression. Full tests/models litellm suite green; ruff/mypy clean.

@codex good catch — fixed in 7b56953.

The empty-"" text part is now closed (content_part.done) and its index reused for the refusal, and the assistant message is not re-announced when a text part already opened it (output_item.added only fires when no prior part exists). So the stream stays coherent: one message open, every content_part.added balanced by a content_part.done.

Proof of fix (live Bedrock claude-fable-5 refusal, end-to-end stream_response → LiteLLM → Converse)

Real content-filtered stream, full event sequence emitted:

response.created
response.output_item.added          # message opened once
response.content_part.added         # (empty) text part
response.output_text.delta          # ""
response.content_part.done          # text part CLOSED
response.content_part.added         # refusal part (same index)
response.refusal.delta              # "Response withheld by the provider's content filter."
response.content_part.done          # refusal part closed
response.output_item.done
response.completed

• output_item.added count: 1 (not re-announced)
• content_part.added == content_part.done: 2 == 2 (no dangling part)
• final response.output: a single ResponseOutputRefusal

Tests

Two streaming cases in test_litellm_chatcompletions_stream.py, using the real empty-""-then-content_filter chunk shape: the synthesize case now also asserts the coherence invariants (one message open; balanced part add/done), and the does-not-clobber case verifies real streamed text before a late content_filter is left intact. Both fail against the pre-fix behaviour. Full tests/models litellm suite (140) green; ruff/mypy clean.

@seratch — a scoping question for you on @codex's follow-up (it's suggesting the empty text part be prevented from opening at all, rather than closed/reused).

The current revision (7b56953) keeps the stream coherent: when the leading content="" delta has opened an (empty) text part, the synthesized refusal closes it (content_part.done), reuses its index, and does not re-announce the assistant message. Verified live against Bedrock claude-fable-5 — exactly one output_item.added, content_part.added count == content_part.done count, final output is a single ResponseOutputRefusal. Codex's remaining point is that content index 0 is reused across the (closed) text part and the refusal, which a consumer replaying raw events could find surprising even though the sequence is balanced.

The alternative is to suppress the leading empty-"" content delta from opening a text part in the first place (in the shared delta.content is not None branch). That's cleaner and sidesteps the index reuse, but it touches the general Chat Completions path used by all providers, not just the LiteLLM/Bedrock content-filter case — so I didn't want to make that change unilaterally on a bugfix PR.

Which would you prefer — keep the current content_filter-scoped fix, or suppress the empty leading delta in the shared content branch? Happy to do either; just flagging the blast-radius trade-off before touching shared code.

@seratch — resolved this without touching the shared content branch, using @codex's own alternative suggestion ("ensure each opened content part gets a distinct index").

In 728b951: when the leading empty-"" delta has opened a text part, close it at its index and open the synthesized refusal at the next index (not the text part's). So no (item_id, content_index) slot ever carries two part types, the empty text part is still dropped from response.completed, and the general Chat Completions path is untouched.

Live Bedrock claude-fable-5 refusal, per-index events:

index 0: content_part.added(text)    -> content_part.done(text)      # empty, dropped from final
index 1: content_part.added(refusal) -> refusal.delta -> content_part.done(refusal)
output_item.added: 1   |   final response.output: [ResponseOutputRefusal]

Also fixed the typecheck CI failure (my earlier local run checked the source file but not the test): the terminal event is now isinstance-narrowed to ResponseCompletedEvent before reading .response. make mypy/make pyright clean on the changed files, full tests/models litellm suite (140) green. So my earlier scope question is moot — no shared-path change needed.

Thanks for the updates. The underlying fix is valid, but the current streaming sequence is still inconsistent: it closes and removes the empty text part at content index 0, emits the refusal at index 1, and then returns a completed message whose only content part is the refusal at position 0.

Please suppress empty-string content deltas before they open a text part in ChatCmplStreamHandler, then emit the synthesized refusal at the contiguous index used by the completed response. Please also update the regression test to assert that no text part or response.output_text.delta is emitted and that the refusal event indexes match the refusal’s position in response.completed. This will keep raw-event replay and the final response aligned.

Done — implemented exactly as you described in 52c3a4e.

The leading empty-"" content delta is now suppressed before it opens a text part (scoped to a leading empty delta when no text part has opened yet; empty deltas after real text are unchanged). With no empty part to reconcile, the synthesized refusal is emitted at the first content index (0) — the same index it occupies in response.completed — so raw-event replay and the final response stay aligned. The old close/distinct-index logic is now dead and removed.

Tests updated per your ask: assert no text part / no response.output_text.delta is emitted, and that the refusal's streamed content_index (0) matches its position in response.completed.

Live proof — Bedrock claude-fable-5, real content-filtered stream_response

output_item.added count : 1
output_text.delta count : 0
content_part.added      : idx=0  part=ResponseOutputRefusal
refusal.delta           : idx=0  delta="Response withheld by the provider's content filter."
completed .output len   : 1
completed content[0]    : ResponseOutputRefusal  (position 0)

Streamed index == completed position == 0. ruff/mypy clean, full tests/models litellm suite (140) green, and the synthesize test still fails against the pre-fix handler (non-vacuous).

Thanks, the empty leading content-part issue is fixed in 52c3a4e. One index mismatch remains when the stream contains reasoning before the terminal content filter.

Reasoning is a separate output item, so it must affect output_index, not the assistant message's content_index. The synthesized refusal is the only content part in that message and must therefore use content_index=0, even when a reasoning item exists. Please remove the reasoning-based increment from this path and add a regression test covering reasoning followed by an empty content delta and terminal content_filter. The test should verify that refusal added/delta/done events use the assistant message's output index and content_index=0, matching response.completed.

Good catch — fixed in a2f7e4b.

You're right: a reasoning item is a separate output item, so it shifts the assistant message's output_index (via assistant_message_output_index), not its content_index, and is never appended to the message's content. The synthesized refusal is the sole content part of its message, so I removed the reasoning-based refusal_index += 1 — it's now unconditionally content_index=0.

Added the regression test you asked for: reasoning delta → empty "" content delta → terminal content_filter. It asserts:

• output[0] is the ResponseReasoningItem, output[1] is the assistant message
• the refusal is the message's sole content part
• every refusal added/delta event uses output_index=1 and content_index=0, matching the refusal's position in response.completed
• the empty "" delta still opens no text part

It's non-vacuous — fails against the pre-fix increment (refusal streamed at content_index=1 while response.completed has it at 0). ruff/mypy clean, full tests/models litellm suite now 141 green.