fix(extensions): surface length-truncated empty completions as refusal on litellm/any_llm paths by weike-zhang · Pull Request #4514 · openai/openai-agents-python

weike-zhang · 2026-08-18T15:46:59Z

Complements #4513 (which fixes the openai_chatcompletions non-streaming and streaming paths) and extends the same finish_reason="length" fix from #4510 to the two remaining sibling Chat Completions model implementations: litellm_model.py and any_llm_model.py.

Summary

A zero-token truncation (finish_reason="length", empty content) on the LiteLLM and AnyLLM model paths converts to zero output items, indistinguishable from a genuinely empty turn — driving agent loops into fruitless retries against the same max-token wall. Both already had the equivalent guard for finish_reason="content_filter"; this mirrors it for "length".

Fix

litellm_model.py — extend the existing content-filter guard (finish_reason in {"content_filter", "length"}), keeping the provider_specific_fields["refusal"] mechanism, with a distinct truncation message:

if (
    message is not None
    and first_choice is not None
    and getattr(first_choice, "finish_reason", None) in {"content_filter", "length"}
    and not message.content
    and not getattr(message, "tool_calls", None)
):
    provider_specific_fields = getattr(message, "provider_specific_fields", None) or {}
    if not provider_specific_fields.get("refusal"):
        refusal_message = (
            "Response withheld by the provider's content filter."
            if first_choice.finish_reason == "content_filter"
            else "Response truncated because the provider's maximum token"
            " limit was reached."
        )
        provider_specific_fields["refusal"] = refusal_message
        message.provider_specific_fields = provider_specific_fields

any_llm_model.py — same guard extension using the message.refusal mechanism:

if (
    message is not None
    and first_choice is not None
    and first_choice.finish_reason in {"content_filter", "length"}
    and not message.content
    and not message.refusal
    and not message.tool_calls
):
    if first_choice.finish_reason == "content_filter":
        message.refusal = "Response withheld by the provider's content filter."
    else:
        message.refusal = (
            "Response truncated because the provider's maximum token limit was reached."
        )

finish_reason="stop" with empty content stays an empty turn (the asymmetry is intentional); non-empty length completions are untouched.

Verification

Driving each model's get_response with a mocked empty completion, varying finish_reason:

Before (both paths): length → 0 items; content_filter → refusal; stop → 0 items
After (both paths):

finish_reason      output items   what the caller sees
length             1              ['message'] -> ResponseOutputRefusal("Response truncated because the provider's maximum token limit was reached.")
content_filter     1              ['message'] -> ResponseOutputRefusal (unchanged)
stop               0              []             (genuine empty, unchanged)
length + text      1              ['message'] -> ResponseOutputText (unchanged)

Tests

test_litellm_content_filter.py: test_length_finish_reason_surfaces_refusal, test_length_does_not_clobber_real_content
test_any_llm_model.py: test_any_llm_chat_path_surfaces_length_truncation_refusal, test_any_llm_chat_path_length_keeps_real_content

tests/models/ all green (826 passed, 11 skipped); ruff clean; mypy clean on the touched lines (the pre-existing litellm_model.py:969 tool-call error is unchanged by this PR).

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 57152a4d88

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-18T15:58:06Z

                message is not None
                and first_choice is not None
-                and first_choice.finish_reason == "content_filter"
+                and first_choice.finish_reason in {"content_filter", "length"}


Handle length terminals in streaming adapters

Adding "length" here only fixes the non-streaming chat path. I checked the streaming paths for both affected adapters: LitellmModel.stream_response and AnyLLMModel._stream_response_via_chat still delegate to ChatCmplStreamHandler, which only tracks finish_reason == "content_filter"; a streamed terminal chunk with ChoiceDelta()/finish_reason="length" and no prior content still completes with output=[], so streaming agents can hit the same indistinguishable empty-turn retry loop this patch is meant to prevent.

AGENTS.md reference: AGENTS.md:L167-L167

Useful? React with 👍 / 👎.

Addressed in 3337639: both LitellmModel.stream_response and AnyLLMModel._stream_response_via_chat now pass length through ChatCmplStreamHandler; a streamed length terminal with no prior content completes with a synthesized refusal instead of output=[].

Correction for precision: 4514's own diff covers the non-streaming paths (_get_response_via_chat / get_response guards). The streamed length terminal for both adapters is handled by the shared ChatCmplStreamHandler, which commit 45e02bd (PR #4513) extends to track finish_reason == "length" — both LitellmModel.stream_response (litellm_model.py:439) and AnyLLMModel._stream_response_via_chat (any_llm_model.py:760) delegate to it, so once both PRs land the streaming paths are aligned. Happy to fold that handler change into this PR instead if you'd prefer the fix to be self-contained.

chatgpt-codex-connector · 2026-08-18T15:58:06Z

+                        else "Response truncated because the provider's maximum token"
+                        " limit was reached."
                    )
+                    provider_specific_fields["refusal"] = refusal_message


Move LiteLLM refusal synthesis before tracing

For LiteLLM, span_generation.span_data.output is populated before this new synthesized provider_specific_fields["refusal"] is added. When tracing.include_data() is enabled and a provider returns an empty finish_reason="length" completion, the returned ModelResponse contains a ResponseOutputRefusal, but the trace still records the pre-mutation empty message; synthesize the refusal before recording span_data.output or refresh the span after mutation.

AGENTS.md reference: AGENTS.md:L162-L162

Useful? React with 👍 / 👎.

Addressed in 3337639: the LiteLLM truncation refusal is synthesized before span_data.output is recorded, so traces with include_data capture the refusal-bearing response.

chatgpt-codex-connector · 2026-08-18T15:58:06Z

+                    refusal_message = (
                        "Response withheld by the provider's content filter."
+                        if first_choice.finish_reason == "content_filter"
+                        else "Response truncated because the provider's maximum token"
+                        " limit was reached."


Preserve provider refusal text on LiteLLM length responses

On the LiteLLM path, this new length branch only checks provider_specific_fields["refusal"] before synthesizing a generic truncation refusal, but LiteLLM/OpenAI-style messages can also carry a top-level message.refusal. In the empty-content finish_reason="length" case with a provider-supplied refusal, this path now returns the generic max-token text instead of preserving the provider's refusal details; mirror the AnyLLM guard's not message.refusal behavior or copy the existing top-level refusal into the converted message.

AGENTS.md reference: AGENTS.md:L167-L167

Useful? React with 👍 / 👎.

Addressed in 3337639: the length branch now mirrors the AnyLLM guard and preserves a provider-supplied top-level message.refusal before falling back to the generic truncation text.

weike-zhang · 2026-08-19T01:00:19Z

Thanks — addressing all three.

1. Streaming paths for litellm/any_llm. Both LitellmModel.stream_response and AnyLLMModel._stream_response_via_chat delegate to the shared ChatCmplStreamHandler (litellm_model.py:437, any_llm_model.py:760). The streaming fix for finish_reason == "length" (tracking saw_length, forwarding the delta-stripped terminal through buffer_tool_call_stream, and synthesizing the end-of-stream refusal) already lives in chatcmpl_stream_handler.py in the sibling PR #4513, which this PR's base does not yet contain. Once #4513 merges, the litellm/any_llm streaming adapters pick it up automatically since they reuse that handler — there is no per-adapter streaming code to touch here, and duplicating the handler change would collide with #4513. The two PRs together close the gap on all paths; they should land in order (base #4513 first).

2. LiteLLM tracing order — fixed. Moved the refusal synthesis before span_data.output is recorded, so the synthesized truncation refusal is now captured in the generation span. Added test_length_finish_reason_refusal_recorded_in_trace asserting the refusal appears in provider_specific_fields of the traced output (LiteLLM's Message has no top-level refusal field; the refusal is carried in provider_specific_fields).

3. Provider refusal preservation. Verified on the LiteLLM path: a provider-supplied refusal is already preserved because the guard checks not provider_specific_fields.get("refusal") before synthesizing (measured: a provider refusal in provider_specific_fields comes through unchanged). LiteLLM's Message model has no top-level refusal field (confirmed 'refusal' not in Message.model_fields), so the "top-level message.refusal" scenario does not apply on this adapter — the refusal only ever arrives via provider_specific_fields, which the existing check already honors. I initially added a not getattr(message, "refusal", None) guard, then removed it after confirming it was dead code on this path.

tests/models/ green (817 passed, 11 skipped) with the fix; ruff clean; no new mypy errors (the litellm_model.py tool-call mypy error is pre-existing).

seratch · 2026-08-19T06:54:08Z

Thanks for extending the investigation. I can confirm that this patch makes the two non-streaming extension adapters turn an empty length terminal into ResponseOutputRefusal, while the shared streaming change remains in #4513. However, the evidence in #4510 and the tests here is synthetic and covers the OpenAI Chat Completions path; it does not show a supported LiteLLM or AnyLLM configuration that actually returns this shape. This is also not a neutral normalization: it routes token-budget exhaustion through ModelRefusalError and the public model_refusal handler.

I am going to close #4514 for now and keep #4513 as the place to settle the originally reported behavior and error semantics. We can revisit a focused extension follow-up with a redacted real-provider response showing finish_reason="length", no content or tool calls, and the resulting user-visible failure.

fix(extensions): surface length-truncated empty completions as refusal

57152a4

This was referenced Aug 18, 2026

fix(chatcmpl): raise ModelBehaviorError on truncated empty completions #4513

Merged

A completion truncated at zero tokens (finish_reason=length) returns zero output items and is indistinguishable from an empty turn #4510

Closed

chatgpt-codex-connector Bot reviewed Aug 18, 2026

View reviewed changes

fix(extensions): record litellm truncation refusal before span output

3337639

seratch added the feature:extensions label Aug 19, 2026

seratch closed this Aug 19, 2026

Navigation Menu

Repository navigation

fix(extensions): surface length-truncated empty completions as refusal on litellm/any_llm paths - #4514

fix(extensions): surface length-truncated empty completions as refusal on litellm/any_llm paths#4514
weike-zhang wants to merge 2 commits into
openai:mainfrom
weike-zhang:fix/surface-length-truncation-sibling

weike-zhang commented Aug 18, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Aug 18, 2026

Uh oh!

weike-zhang Aug 19, 2026

Uh oh!

weike-zhang Aug 19, 2026

Uh oh!

chatgpt-codex-connector Bot Aug 18, 2026

Uh oh!

weike-zhang Aug 19, 2026

Uh oh!

chatgpt-codex-connector Bot Aug 18, 2026

Uh oh!

weike-zhang Aug 19, 2026

Uh oh!

weike-zhang commented Aug 19, 2026

Uh oh!

seratch commented Aug 19, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

weike-zhang commented Aug 18, 2026

Summary

Fix

Verification

Tests

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

weike-zhang Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

weike-zhang Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

weike-zhang Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

weike-zhang Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

weike-zhang commented Aug 19, 2026

Uh oh!

seratch commented Aug 19, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants