Repository navigation
fix(metrics): report effective denominator for value-dropping aggregations - #4320
Open
barlowa124 wants to merge 1 commit into
Open
barlowa124 wants to merge 1 commit into
barlowa124 wants to merge 1 commit into
Conversation
nanmean drops NaN values inside the aggregation call, so the metric is averaged over fewer values than the count reported alongside it (EleutherAI#4319). Mark excluding aggregations with an excludes_nan attribute, count the values actually averaged in _compute_task_aggregations, publish the per-metric counts in the task result's sample_count field (previously populated by groups only), and warn when the count is smaller than the number of produced values.
This was referenced Oct 6, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #4319.
Problem
nanmeandiscards NaN inside aggregation, but the result entry pairsthat mean with a count over every produced value. The reported number
describes a smaller population than the adjoining count implies. The
medtexttask hits this because it storesnp.NANon scorer errors.Fix
excludes_nan._compute_task_aggregationscounts the values each excludingaggregation consumed and returns them alongside the metrics.
sample_counton the task result (NotRequiredon
_TaskMetrics), keeping the effective denominator next to thenumber it describes. A warning names the metric and both counts.
sample_lenstill counts produced values. Existing warnings and thegroup.pyweighting TODO are unchanged.Checks
The reporter's shape now reports the mean over the surviving values
with the matching count beside the full
sample_len. New tests coverthe effective denominator and the counts emitted when nothing is
excluded, plus absence on non-excluding aggregations and the result
field.
tests/test_evaluator_utils.py,tests/test_metrics.pyandtests/test_aggregation_pipeline.pyare green. Ruff is clean.Generated by Devin