Skip to content

fix(openai): handle assistant messages without content key - #3440

Open
BlueX888 wants to merge 1 commit into
confident-ai:mainfrom
BlueX888:fix/prep-openai-input-extractor-assistant-message-without-content
Open

BlueX888 wants to merge 1 commit into
confident-ai:mainfrom
BlueX888:fix/prep-openai-input-extractor-assistant-message-without-content

Conversation

@BlueX888

Copy link
Copy Markdown

Summary

extract_input_parameters_from_completion raised KeyError: 'content' when a Chat Completions payload contained an assistant message without a content key. Because safe_extract_input_parameters swallows all exceptions, this degraded the whole payload to InputParameters(model="NA"), so the LLM span lost its input, messages, tools and tool descriptions.

Why

An assistant history message that only carries tool_calls may legally omit content: in the openai SDK, ChatCompletionAssistantMessageParam.__required_keys__ is empty and only role is marked Required (deepeval pins openai = "*", so any current SDK version applies).

extract_input_parameters_from_completion read the key unconditionally with message["content"] (deepeval/openai/extractors.py:49). The KeyError propagated to the bare except in safe_extract_input_parameters (deepeval/openai/extractors.py:25-26), which returned the placeholder InputParameters(model="NA"). The sibling render_messages (deepeval/openai/utils.py:118-119) reads the same messages with message.get(...), and stringify_multimodal_content (deepeval/openai/utils.py:61-62) already returns "" for None content, so a .get() here changes nothing downstream.

Changes

  • deepeval/openai/extractors.py: read content with message.get("content") instead of message["content"] in extract_input_parameters_from_completion, mirroring render_messages.
  • tests/test_core/test_models/test_openai_extractors.py: two regression tests — one against extract_input_parameters_from_completion (asserts model, input and rendered messages are preserved), one against safe_extract_input_parameters (asserts the payload no longer degrades to model="NA").

Testing

Both new tests fail on main:

  • test_extract_input_parameters_with_assistant_message_without_content → KeyError: 'content' at deepeval/openai/extractors.py:49
  • test_safe_extract_input_parameters_with_assistant_message_without_content → AssertionError: assert 'NA' == 'gpt-4o'

With the fix:

pytest tests/test_core/test_models/test_openai_extractors.py -q
→ 3 passed

pytest tests/test_core/test_models/test_openai_extractors.py tests/test_core/test_models/test_openai_model.py tests/test_core/test_imports.py -q
→ 62 passed

black --check deepeval/openai/extractors.py tests/test_core/test_models/test_openai_extractors.py   # black 25.12.0, as pinned in .github/workflows/black.yml
→ 2 files would be left unchanged.

@vercel

vercel Bot commented Oct 11, 2026

Copy link
Copy Markdown

@BlueX888 is attempting to deploy a commit to the Confident AI Team on Vercel.

A member of the Team first needs to authorize it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant