Fix genai.evaluate() dropping dataset expectations and tags with scorers=[]#23957
Merged
B-Step62 merged 3 commits intoJun 16, 2026
Merged
Conversation
PR #20940 added a short-circuit in _run_pipeline that skips the score pool when there are no single-turn scorers, returning EvalResult(assessments=[]) directly. This bypassed _run_score, which besides scoring also logs the dataset expectations as Expectation assessments and applies eval_item.tags to the trace - so with scorers=[] both expectations and dataset tags were silently dropped (regression from 3.11.1). Remove the short-circuit and always route completed predict items through _run_score. Scoring itself is already a no-op for empty scorers (_compute_eval_scores returns early), and _get_scorer_rate_config falls back to the predict rate when num_scorers is zero, so the score pool path is safe in this case. Since every item now produces a score-pool task, count all eval items in the progress bar total (previously 0 when no scorers were given). Signed-off-by: Mingi Jeong <[email protected]>
…y (#23746) When scorers=[], dataset expectations must still be persisted to the trace as Expectation assessments, and the dataset tags column must still be applied to the traces. Add regression tests that fail on the pre-fix harness (which set assessments=[] and skipped both) and pass with the fix. Signed-off-by: Mingi Jeong <[email protected]>
Contributor
Install mlflow from this PR
Install mlflow from this PR# mlflow
pip install git+https://github.com/mlflow/mlflow.git@refs/pull/23957/merge
# mlflow-skinny
pip install git+https://github.com/mlflow/mlflow.git@refs/pull/23957/merge#subdirectory=libs/skinnyFor Databricks, use the following command: %sh curl -LsSf https://raw.githubusercontent.com/mlflow/mlflow/HEAD/dev/install-skinny.sh | sh -s pull/23957/mergePR author's recent activityIn the last 14 days, @Incheonkirin opened 17 PRs across 11 repos (showing top 10):
|
Contributor
|
@Incheonkirin Thank you for the contribution! Could you fix the following issue(s)? Otherwise, this PR may be automatically closed. ❌ Invalid PR templateThe PR description is missing required sections. Please use the PR template. |
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds regression coverage and fixes a pipeline short-circuit so that mlflow.genai.evaluate(..., scorers=[]) still persists dataset expectations and dataset tags on traces (regression #23746).
Changes:
- Add tests asserting dataset expectations are logged as
Expectationassessments even whenscorers=[]. - Add a test asserting dataset tags are persisted to traces when
scorers=[]. - Remove the “no single-turn scorers” short-circuit so
_run_scorestill runs (no-op scoring, but logs expectations/tags), and update progress total tasks accordingly.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| tests/genai/evaluate/test_evaluation.py | Adds regression tests verifying expectations and tags are persisted when scorers=[]. |
| mlflow/genai/evaluation/harness.py | Always queues _run_score per item to ensure expectations/tags are logged even without scorers; adjusts progress accounting. |
github-actions
Bot
added
area/evaluation
Contributor
|
Documentation preview for 3cdba82 is available at: More info
|
Signed-off-by: Yuki Watanabe <[email protected]>
Contributor
Author
|
Thanks @B-Step62! Appreciate you taking the time to get this one in. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issues/PRs
Closes #23746
What changes are proposed in this pull request?
mlflow.genai.evaluate(..., scorers=[])stopped logging the dataset'sexpectations(andtags) onto the generated traces — a regression from #20940. In_run_pipeline, the no-single-turn-scorer case short-circuited intoEvalResult(assessments=[])instead of going through_run_score, which is what persists dataset expectations (_get_new_expectations→_log_assessments) and applieseval_item.tagsviaset_trace_tag.This removes that special-case branch so every eval item goes through
_run_score. Scoring is a no-op when there are no single-turn scorers, but expectations and tags are persisted again.total_tasksis also corrected so the progress bar counts the items that now go through the score pool.Credit to @maxime2476 for independently reproducing and confirming the root cause; per the maintainer's request this is the single consolidated PR.
How is this PR tested?
Added regression tests in
tests/genai/evaluate/test_evaluation.pythat callevaluate(scorers=[])and assert, viasearch_traces, that the datasetexpectationsland asExpectationassessments and the datasettagsland on the traces. Both fail on the pre-fix harness and pass with the fix.Does this PR require documentation update?
Does this PR require updating the MLflow Skills repository?
Release Notes
Is this a user-facing change?
Fixed
mlflow.genai.evaluate()so datasetexpectationsandtagsare again logged onto the generated traces when called withscorers=[](regression since 3.11).What component(s), interfaces, languages, and integrations does this PR affect?
Components
area/tracking: Tracking Service, tracking client APIs, autologgingarea/models: MLmodel format, model serialization/deserialization, flavorsarea/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registryarea/scoring: MLflow Model server, model deployment tools, Spark UDFsarea/evaluation: MLflow model evaluation features, evaluation metrics, and evaluation workflowsarea/gateway: MLflow AI Gateway client APIs, server, and third-party integrationsarea/prompts: MLflow prompt engineering features, prompt templates, and prompt managementarea/tracing: MLflow Tracing features, tracing APIs, and LLM tracing functionalityarea/projects: MLproject format, project running backendsarea/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev serverarea/build: Build and test infrastructure for MLflowarea/docs: MLflow documentation pagesHow should the PR be classified in the release notes? Choose one:
rn/none- No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" sectionrn/breaking-change- The PR will be mentioned in the "Breaking Changes" sectionrn/feature- A new user-facing feature worth mentioning in the release notesrn/bug-fix- A user-facing bug fix worth mentioning in the release notesrn/documentation- A user-facing documentation change worth mentioning in the release notesIs this PR a critical bugfix or security fix that should go into the next patch release?
What is a minor/patch release?
Minor releases are expected to contain larger changes, such as new features and improvements. Non-critical bug fixes and doc updates can be included as well. By default, your PR should target the next minor release.
Patch releases are typically only performed when there has been a major regression or bug in the latest release. For the sake of stability, your PR should not be included in a patch release unless it is a critical fix, or if the risk level of your PR is exceedingly low.