Repository navigation
test(openai-agents): exercise the real Runner offline - #14
Conversation
|
@FU-max-boop Thanks for the clear PR and for disclosing your connection to AgentRunProof. I agree with the coverage gap you’re targeting: our mocked streaming result exercises the adapter, but it can't catch changes in the real Runner, tool-event, or cancellation path. I ran the branch independently and reproduced the results: 208 passed and 5 skipped, including all four OpenAI Agents integration tests. Before merging, I’d like to avoid adding AgentRunProof to LoopGain’s general test extra. Version 0.1.2 constrains openai-agents to <0.21, which caused pip to select 0.20.0 instead of the current 0.21.0 release. It would also make every normal test installation pull the full OpenAI Agents dependency tree. The SDK now exposes the same offline path through agents.testing.ScriptedModel, function_call, and assistant_message. I verified this test using those official fixtures against OpenAI Agents 0.21.0, and the full suite still passes. Would you be open to revising the PR to use agents.testing directly and leaving the general test extra as pytest-only? I’d be happy to merge the real-Runner test in that form. Thanks again for identifying the gap! |
|
Thanks for the independent verification and the clear direction. I updated the PR in
I revalidated in a fresh Python 3.12 environment with exact
|
|
@FU-max-boop Thanks for turning this around so quickly. The dependency concern is resolved, and the updated suite passes cleanly. I added one small assertion to verify that the SDK cancellation method is actually called, and updated the PR description to match the revised implementation. I’ve merged it. Thanks again for the contribution! |
Summary
agents.testing.ScriptedModelRunner.run_streamedand function-tool pathagents.testingis unavailable, preserving compatibility with older SDK installationsWhy
The existing smoke tests intentionally monkeypatch
Runner.run_streamedwith a small lookalike. That gives precise adapter-level coverage, but it cannot catch drift in the SDK's real Runner, stream events, function tools, or cancellation behavior.The official scripted model exercises that path deterministically without an API key, network call, model budget, or additional test dependency.
Validation
openai-agents==0.21.0python -m pip check— cleanpython -m pytest -q tests/integration/test_openai_agents_smoke.py -m integration— 4 passedpython -m pytest -q— 208 passed, 5 skipped