Skip to content

GrokModel.generate/a_generate pass an OpenAI-style list-of-dicts to xai_sdk.chat.user(), so every call crashes with a protobuf TypeError before any network I/O #3436

Description

@BlueX888

Describe the bug

GrokModel.generate() and GrokModel.a_generate() build the prompt as an OpenAI-style content list — [{"type": "text", "text": prompt}] — and pass that list to xai_sdk.chat.user(). user() only accepts str or a protobuf chat_pb2.Content, so every call raises a TypeError from protobuf message construction before any network I/O happens. The multimodal path (generate_content) emits the same OpenAI-style dicts with image_url blocks, so every prompt shape breaks the same way. As far as I can tell this makes GrokModel entirely non-functional for both sync and async calls.

To Reproduce

import asyncio
from deepeval.models import GrokModel

class FakeChat:
    def create(self, **kw): return self
    def append(self, msg): return self
    def sample(self): raise AssertionError("network sample() must not be reached")
    async def asample(self): raise AssertionError("network asample() must not be reached")
class FakeClient:
    chat = FakeChat()

m = GrokModel(model="grok-3", api_key="test")
m.load_model = lambda async_mode=False: FakeClient()  # stubbed only to show the crash pre-dates any API call

out = m.generate("What is 2+2?")          # TypeError before chat.sample() is reached
out = asyncio.run(m.a_generate("What is 2+2?"))

load_model is stubbed purely to demonstrate that the failure happens during message construction, not on the wire — with a real client and API key the identical TypeError is raised at the same line, before a request is sent.

Observed output (verbatim):

generate CRASHED: TypeError: Parameter to initialize message field must be dict or instance of same class: expected <class 'xai.api.v1.chat_pb2.Content'> got <class 'list'>.
a_generate CRASHED: TypeError: Parameter to initialize message field must be dict or instance of same class: expected <class 'xai.api.v1.chat_pb2.Content'> got <class 'list'>.

Expected behavior

xai_sdk.chat.user() accepts only str | chat_pb2.Content:

  • xai_sdk/chat.py (v1.20.0): def user(*args: Content) -> chat_pb2.Message, where each item goes through _process_content, which returns text(content) for a str and otherwise passes the object straight through as a proto Content — a Python list of dicts cannot initialize a repeated Content field, which is exactly the TypeError above.
  • xai_sdk/types/chat.py (v1.20.0): Content: TypeAlias = Union[str, chat_pb2.Content].
  • The SDK's own docstring example passes a plain string: chat.append(user("Tell me a joke")).
  • The same contract holds on upstream xai-org/xai-sdk-python and at the oldest installable tag (v1.0.0), so it covers everything deepeval's xai-sdk (>=0.2.0) pin can resolve to.

So for a text prompt user(prompt) (or user(text(prompt))) should be passed, and multimodal input should be expressed with the SDK's native xai_sdk.chat.text(...) / xai_sdk.chat.image(...) Content objects. The call should then reach the API and return a completion instead of crashing.

Screenshots

N/A.

Desktop (please complete the following information):

  • OS: macOS (arm64, darwin 25.6.0)
  • Browser: N/A (server-side Python)
  • Version: deepeval 4.2.6 at main @ 7e94626702befc558c718e1edb18408e094c4bed, Python 3.12.14, xai-sdk 1.20.0

Smartphone (please complete the following information):

N/A.

Additional context

Root cause: deepeval/models/llms/grok_model.py:120 (and :166 for a_generate) build content = [{"type": "text", "text": prompt}], and deepeval/models/llms/grok_model.py:128 / :174 pass that list to chat.append(user(content)). user() requires str | chat_pb2.Content, so protobuf construction fails immediately. The multimodal helper generate_content() (deepeval/models/llms/grok_model.py:196-222) emits the same OpenAI wire-format dicts ({"type": "image_url", "image_url": {"url": ...}}) instead of xai-sdk Content objects, so that path can't work either.

Reachability: anything that uses GrokModel as its judge/metric LLM goes through generate()/a_generate(). This isn't caught in CI because tests/test_core/test_models/test_grok_model.py stubs load_model (its own comment: "Avoid importing xai_sdk in tests by stubbing load_model.") and never exercises generate() against the real user() helper.

I searched existing issues (GrokModel, grok, and the exact TypeError message) and found nothing covering this.

Happy to open a PR that passes the prompt string for text prompts and rewrites generate_content() to emit xai_sdk.chat.text(...) / xai_sdk.chat.image(...) Content objects, plus a generate() test that runs against the real user() helper.

Activity

  1. Christian-Sidak commented on Oct 11, 2026

    @Christian-Sidak
    Contributor

    Opened a fix in #3437: converts the messages list to a plain string before passing it to xai_sdk.chat.user(), which is what the SDK expects, so the protobuf TypeError no longer fires before any network I/O happens.

  2. added a commit that references this issue on Nov 2, 2026
    2469335
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions