Skip to content

test(openai-agents): exercise the real Runner offline - #14

Merged
fitz2882 merged 3 commits into
loopgain-ai:mainfrom
FU-max-boop:agent/test-real-openai-runner
Aug 15, 2026
Merged

fitz2882 merged 3 commits into
loopgain-ai:mainfrom
FU-max-boop:agent/test-real-openai-runner

Conversation

@FU-max-boop

@FU-max-boop FU-max-boop commented Aug 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • add a no-network OpenAI Agents integration test using the SDK's official agents.testing.ScriptedModel
  • exercise the real Runner.run_streamed and function-tool path
  • verify LoopGain observes a real tool output, converges after one observation, executes the tool exactly once, and calls the SDK cancellation method before the next scripted response
  • skip only the real-Runner test when agents.testing is unavailable, preserving compatibility with older SDK installations

Why

The existing smoke tests intentionally monkeypatch Runner.run_streamed with a small lookalike. That gives precise adapter-level coverage, but it cannot catch drift in the SDK's real Runner, stream events, function tools, or cancellation behavior.

The official scripted model exercises that path deterministically without an API key, network call, model budget, or additional test dependency.

Validation

  • Python 3.12 with openai-agents==0.21.0
  • python -m pip check — clean
  • python -m pytest -q tests/integration/test_openai_agents_smoke.py -m integration — 4 passed
  • python -m pytest -q — 208 passed, 5 skipped

@FU-max-boop
FU-max-boop marked this pull request as ready for review August 15, 2026 06:39
@fitz2882

Copy link
Copy Markdown
Contributor

@FU-max-boop Thanks for the clear PR and for disclosing your connection to AgentRunProof. I agree with the coverage gap you’re targeting: our mocked streaming result exercises the adapter, but it can't catch changes in the real Runner, tool-event, or cancellation path.

I ran the branch independently and reproduced the results: 208 passed and 5 skipped, including all four OpenAI Agents integration tests.

Before merging, I’d like to avoid adding AgentRunProof to LoopGain’s general test extra. Version 0.1.2 constrains openai-agents to <0.21, which caused pip to select 0.20.0 instead of the current 0.21.0 release. It would also make every normal test installation pull the full OpenAI Agents dependency tree.

The SDK now exposes the same offline path through agents.testing.ScriptedModel, function_call, and assistant_message. I verified this test using those official fixtures against OpenAI Agents 0.21.0, and the full suite still passes.

Would you be open to revising the PR to use agents.testing directly and leaving the general test extra as pytest-only? I’d be happy to merge the real-Runner test in that form. Thanks again for identifying the gap!

@FU-max-boop

Copy link
Copy Markdown
Contributor Author

Thanks for the independent verification and the clear direction. I updated the PR in bade7c1:

  • removed AgentRunProof from the general test extra;
  • switched the real-Runner test to agents.testing.ScriptedModel, function_call, and assistant_message; and
  • kept the test guarded with pytest.importorskip("agents.testing"), so older SDK installations skip only this 0.21+ path.

I revalidated in a fresh Python 3.12 environment with exact openai-agents==0.21.0:

  • pip check — clean
  • focused OpenAI Agents integration tests — 4 passed
  • full suite — 208 passed, 5 skipped

@fitz2882
fitz2882 merged commit 1b06ab1 into loopgain-ai:main Aug 15, 2026
@fitz2882

Copy link
Copy Markdown
Contributor

@FU-max-boop Thanks for turning this around so quickly. The dependency concern is resolved, and the updated suite passes cleanly. I added one small assertion to verify that the SDK cancellation method is actually called, and updated the PR description to match the revised implementation. I’ve merged it. Thanks again for the contribution!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants