Skip to content

fix(metrics): report effective denominator for value-dropping aggregations - #4320

Open
barlowa124 wants to merge 1 commit into
EleutherAI:mainfrom
barlowa124:nanmean-denominator-4319
Open

barlowa124 wants to merge 1 commit into
EleutherAI:mainfrom
barlowa124:nanmean-denominator-4319

Conversation

@barlowa124

@barlowa124 barlowa124 commented Oct 5, 2026 •

Copy link
Copy Markdown

Resolves #4319.

Problem

nanmean discards NaN inside aggregation, but the result entry pairs
that mean with a count over every produced value. The reported number
describes a smaller population than the adjoining count implies. The
medtext task hits this because it stores np.NAN on scorer errors.

Fix

  • Aggregations that drop values are marked via excludes_nan.
  • _compute_task_aggregations counts the values each excluding
    aggregation consumed and returns them alongside the metrics.
  • Counts publish in sample_count on the task result (NotRequired
    on _TaskMetrics), keeping the effective denominator next to the
    number it describes. A warning names the metric and both counts.

sample_len still counts produced values. Existing warnings and the
group.py weighting TODO are unchanged.

Checks

The reporter's shape now reports the mean over the surviving values
with the matching count beside the full sample_len. New tests cover
the effective denominator and the counts emitted when nothing is
excluded, plus absence on non-excluding aggregations and the result
field.
tests/test_evaluator_utils.py, tests/test_metrics.py and
tests/test_aggregation_pipeline.py are green. Ruff is clean.

Generated by Devin

nanmean drops NaN values inside the aggregation call, so the metric is
averaged over fewer values than the count reported alongside it (EleutherAI#4319).
Mark excluding aggregations with an excludes_nan attribute, count the
values actually averaged in _compute_task_aggregations, publish the
per-metric counts in the task result's sample_count field (previously
populated by groups only), and warn when the count is smaller than the
number of produced values.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] nanmean in metrics.py silently shrinks denominator without reporting excluded count

1 participant