Repository navigation
Description
Describe the bug
GrokModel.generate() and GrokModel.a_generate() build the prompt as an OpenAI-style content list — [{"type": "text", "text": prompt}] — and pass that list to xai_sdk.chat.user(). user() only accepts str or a protobuf chat_pb2.Content, so every call raises a TypeError from protobuf message construction before any network I/O happens. The multimodal path (generate_content) emits the same OpenAI-style dicts with image_url blocks, so every prompt shape breaks the same way. As far as I can tell this makes GrokModel entirely non-functional for both sync and async calls.
To Reproduce
import asyncio
from deepeval.models import GrokModel
class FakeChat:
def create(self, **kw): return self
def append(self, msg): return self
def sample(self): raise AssertionError("network sample() must not be reached")
async def asample(self): raise AssertionError("network asample() must not be reached")
class FakeClient:
chat = FakeChat()
m = GrokModel(model="grok-3", api_key="test")
m.load_model = lambda async_mode=False: FakeClient() # stubbed only to show the crash pre-dates any API call
out = m.generate("What is 2+2?") # TypeError before chat.sample() is reached
out = asyncio.run(m.a_generate("What is 2+2?"))load_model is stubbed purely to demonstrate that the failure happens during message construction, not on the wire — with a real client and API key the identical TypeError is raised at the same line, before a request is sent.
Observed output (verbatim):
generate CRASHED: TypeError: Parameter to initialize message field must be dict or instance of same class: expected <class 'xai.api.v1.chat_pb2.Content'> got <class 'list'>.
a_generate CRASHED: TypeError: Parameter to initialize message field must be dict or instance of same class: expected <class 'xai.api.v1.chat_pb2.Content'> got <class 'list'>.
Expected behavior
xai_sdk.chat.user() accepts only str | chat_pb2.Content:
xai_sdk/chat.py(v1.20.0):def user(*args: Content) -> chat_pb2.Message, where each item goes through_process_content, which returnstext(content)for astrand otherwise passes the object straight through as a protoContent— a Pythonlistof dicts cannot initialize a repeatedContentfield, which is exactly theTypeErrorabove.xai_sdk/types/chat.py(v1.20.0):Content: TypeAlias = Union[str, chat_pb2.Content].- The SDK's own docstring example passes a plain string:
chat.append(user("Tell me a joke")). - The same contract holds on upstream
xai-org/xai-sdk-pythonand at the oldest installable tag (v1.0.0), so it covers everything deepeval'sxai-sdk (>=0.2.0)pin can resolve to.
So for a text prompt user(prompt) (or user(text(prompt))) should be passed, and multimodal input should be expressed with the SDK's native xai_sdk.chat.text(...) / xai_sdk.chat.image(...) Content objects. The call should then reach the API and return a completion instead of crashing.
Screenshots
N/A.
Desktop (please complete the following information):
- OS: macOS (arm64, darwin 25.6.0)
- Browser: N/A (server-side Python)
- Version: deepeval 4.2.6 at
main@7e94626702befc558c718e1edb18408e094c4bed, Python 3.12.14, xai-sdk 1.20.0
Smartphone (please complete the following information):
N/A.
Additional context
Root cause: deepeval/models/llms/grok_model.py:120 (and :166 for a_generate) build content = [{"type": "text", "text": prompt}], and deepeval/models/llms/grok_model.py:128 / :174 pass that list to chat.append(user(content)). user() requires str | chat_pb2.Content, so protobuf construction fails immediately. The multimodal helper generate_content() (deepeval/models/llms/grok_model.py:196-222) emits the same OpenAI wire-format dicts ({"type": "image_url", "image_url": {"url": ...}}) instead of xai-sdk Content objects, so that path can't work either.
Reachability: anything that uses GrokModel as its judge/metric LLM goes through generate()/a_generate(). This isn't caught in CI because tests/test_core/test_models/test_grok_model.py stubs load_model (its own comment: "Avoid importing xai_sdk in tests by stubbing load_model.") and never exercises generate() against the real user() helper.
I searched existing issues (GrokModel, grok, and the exact TypeError message) and found nothing covering this.
Happy to open a PR that passes the prompt string for text prompts and rewrites generate_content() to emit xai_sdk.chat.text(...) / xai_sdk.chat.image(...) Content objects, plus a generate() test that runs against the real user() helper.
Activity
Christian-Sidak commentedon Oct 11, 2026
Opened a fix in #3437: converts the messages list to a plain string before passing it to xai_sdk.chat.user(), which is what the SDK expects, so the protobuf TypeError no longer fires before any network I/O happens.
fix(grok): pass str to xai_sdk.chat.user(), use SDK Content for multi…