Skip to content

fix(openai_agents): run base Agent __post_init__ in DeepEvalAgent - #3323

Open
BlueX888 wants to merge 1 commit into
confident-ai:mainfrom
BlueX888:fix/prep-deepeval-agent-post-init-skips-super
Open

BlueX888 wants to merge 1 commit into
confident-ai:mainfrom
BlueX888:fix/prep-deepeval-agent-post-init-skips-super

Conversation

@BlueX888

Copy link
Copy Markdown

Summary

DeepEvalAgent now runs the base agents.Agent.__post_init__ normalization when constructed.

Why

DeepEvalAgent.__post_init__ (deepeval/openai_agents/agent.py:35) overrode the SDK's Agent.__post_init__ but only called patch_default_agent_runner_get_model(), never super().__post_init__(). Since DeepEvalAgent is a dataclass, its __post_init__ fully replaces the base one, so the base normalization never ran:

  • model_settings kept the dataclass default. The SDK's default model is a GPT-5 model whose default settings carry reasoning=Reasoning(effort="none"), so every DeepEvalAgent used GPT-5 reasoning/verbosity defaults even for non-GPT-5 models such as gpt-4o, for which the base class swaps in generic ModelSettings().
  • A dict passed as model_settings was never coerced by the SDK's _coerce_model_settings, leaving a raw dict on the field.
  • Base field validation was skipped, so a non-list tools was silently accepted instead of raising TypeError.

Changes

  • deepeval/openai_agents/agent.py: call super().__post_init__() before the runner patch.
  • tests/test_integrations/test_openai_agents/test_deepeval_agent.py: deterministic regression tests (no API key, no network): model_settings parity with a base Agent for gpt-5 and gpt-4o, dict model_settings coercion, and TypeError on a non-list tools.

Testing

Verified with openai-agents 0.22.3 (the openai-agents CI job installs the latest SDK via pip install -U openai-agents, .github/workflows/test_integrations.yml:219).

With the fix:

pytest tests/test_integrations/test_openai_agents/test_deepeval_agent.py -q
4 passed in 0.03s

Without the one-line fix, the same file fails:

FAILED ...::test_model_settings_match_base_agent[gpt-5]
FAILED ...::test_model_settings_match_base_agent[gpt-4o]
FAILED ...::test_dict_model_settings_are_coerced  # AttributeError: 'dict' object has no attribute 'temperature'
FAILED ...::test_invalid_tools_are_rejected       # Failed: DID NOT RAISE TypeError

Full module suite (pytest tests/test_integrations/test_openai_agents/ -q): 15 failed, 4 passed, 1 skipped; the 15 failures are pre-existing missing-credential failures (identical 15 failed, 1 skipped on the unpatched tree) and CI runs this job only when OPENAI_API_KEY is set.

DeepEvalAgent.__post_init__ only called the runner patch, so the base
agents.Agent normalization never ran. model_settings kept the dataclass
default (reasoning effort 'none' for every model), a dict model_settings
was never coerced by _coerce_model_settings, and the field validation
that rejects e.g. a non-list `tools` was skipped.
@vercel

vercel Bot commented Sep 20, 2026

Copy link
Copy Markdown

@BlueX888 is attempting to deploy a commit to the Confident AI Team on Vercel.

A member of the Team first needs to authorize it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant