Skip to content

Python: fail assert_score_at_least when evaluator filter matches no scores - #10180

Merged
Eduard van Valkenburg (eavanvalkenburg) merged 1 commit into
microsoft:mainfrom
Dhruv-Sh-git:fix/issue-10166-assert-score-missing-evaluator
Nov 9, 2026
Merged

Eduard van Valkenburg (eavanvalkenburg) merged 1 commit into
microsoft:mainfrom
Dhruv-Sh-git:fix/issue-10166-assert-score-missing-evaluator

Conversation

@Dhruv-Sh-git

@Dhruv-Sh-git Dhruv Shaswat (Dhruv-Sh-git) commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #10166.

Motivation & Context

Previously, assert_score_at_least() could silently pass when an evaluator filter didn't match any scores. This could happen because of a typo in the evaluator name or a missing evaluation check, making it appear that the score validation had succeeded even though nothing was actually checked.

Changes Made

  • Updated EvalResults.assert_score_at_least() to keep track of matched scores and evaluator names across evaluation items and sub-results.
  • Added a check to raise EvalNotPassedError when a specified evaluator doesn't match any scores.
  • The error message identifies the requested evaluator and lists the available evaluator names to make debugging easier.
  • Preserved the existing behavior when no evaluator filter is provided.
  • Added regression tests covering unmatched evaluators, empty scores, custom error messages, and nested sub-results.

Validation

  • uv run pytest packages/core/tests/core/test_local_eval.py — 97 passed.
  • uv run pytest packages/core/tests/core/test_evaluation.py — 91 passed.
  • uv run poe syntax -P core — Passed.
  • uv run poe pyright -P core — 0 errors.

AI Assistance

AI assistance was used during issue investigation, implementation, and writing regression tests. All changes were reviewed and tested locally.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Merged via the queue into microsoft:main with commit 1297a5f Nov 9, 2026
56 checks passed
@Dhruv-Sh-git
Dhruv Shaswat (Dhruv-Sh-git) deleted the fix/issue-10166-assert-score-missing-evaluator branch November 9, 2026 12:52

This branch was previously deployed

1 inactive deployment
github-app-auth — 1285b021 Deployed Nov 9, 2026 by Dhruv-Sh-git via add_label #27369
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python Usage: [Issues, PRs], Target: Python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python: [Bug]: assert_score_at_least(evaluator=...) passes silently when no score matches the evaluator name

3 participants