FEA Adds `decision_threshold_curve` function #31338

lucyleeow · 2025-05-08T08:50:50Z

Reference Issues/PRs

closes #25639 (supercedes)
closes #21391

Thought it would be easier to open a new PR and there were no long discussions on the old PR (#25639).

What does this implement/fix? Explain your changes.

Adds a function that takes a scoring function, y_true, y_score, thresholds and outputs score per threshold.

The intention is to later add a new display class using this function, allowing us to plot metric score per threshold e.g.

Uses the _CurveScorer as suggested by @glemaitre . Refactors out a new _scores_from_predictions static method, that takes the predictions. The old _score takes estimator, calculates y_score and passes to the new _scores_from_prediction.

Notes:

_scores_from_predictions - name is inspired by the display class methods 'from_predictions' , but happy to change
it did not make sense to use from_scorer because we are not using a scorer (which has the signature callable(estimator, X, y)) we are using a 'scoring_functionwith signaturescore_func(y_true, y_pred, **kwargs)`
- we instantiate _CurveScorer directly instead, and then call _scores_from_predictions in decision_threshold_curve
- decided to make _scores_from_predictions a static method, but I also could have made it a method, and in decision_threshold_curve instantiate _CurveScorer directly first (not via from_scorer). Only went with staticmethod path because I initially didn't register from_scorer instantiates differently than directly via _CurveScorer. Not 100% on what is best here. [realised that method is nicer to avoid having too many params in decision_threshold_curve and avoids some lines of code (use self.xx directly, instead of passing self.xx to _scores_from_predictions)]

Any other comments?

cc @glemaitre

Lots more to do, but should get implementation right first.

To do:

Add tests
Add example
Review _decision_threshold.py module docstring

…/scikit-learn into metric_threshold_curve

lucyleeow · 2025-05-15T02:51:10Z

doc/modules/classification_threshold.rst

 .. currentmodule:: sklearn.model_selection

-.. _TunedThresholdClassifierCV:
+.. _threshold_tunning:


I referenced this page starting here for decision_threshold_curve because I thought this first paragraph was appropriate, for context, not just the section I added. Not 100% on this though and happy to change

lucyleeow · 2025-05-15T02:52:11Z

doc/modules/classification_threshold.rst

+.. _threshold_tunning:

 ==================================================
 Tuning the decision threshold for class prediction


@glemaitre Can't comment where this is relevant (after L49), but I wonder if it would be interesting to add another scenario where threshold tunning may be of interest - imbalanced datasets?

lucyleeow · 2025-05-15T02:54:31Z

sklearn/metrics/_decision_threshold.py

+        good, or a loss function, meaning low is good. In the latter case, the
+        the output of `score_func` will be sign-flipped.
+
+    labels : array-like, default=None


_CurveScorer uses the term "classes" but "labels" is consistent with what is used for other classification metrics, so chose this.

lucyleeow · 2025-05-15T02:55:01Z

sklearn/metrics/_decision_threshold.py

+        between the minimum and maximum of `y_score`. If an array-like, it will be
+        used as the thresholds.
+
+    greater_is_better : bool, default=True


Again, this is consistent, with term used in other metrics, so avoided using "sign".

lucyleeow · 2025-05-15T02:56:13Z

sklearn/metrics/_scorer.py

+        # This could also be done in `decision_threshold_curve`, not sure which
+        # is better
+        y_true_unique = cached_unique(y_true)
+        if classes is None:
+            classes = y_true_unique
+        # not sure if this separate error msg needed.
+        # there is the possibility that set(classes) != set(y_true_unique) fails
+        # because `y_true` only contains one class.
+        if len(y_true_unique) == 1:
+            raise ValueError("`y_true` only contains one class label.")
+        if set(classes) != set(y_true_unique):
+            raise ValueError(
+                f"`classes` ({classes}) is not equal to the unique values found in "
+                f"`y_true` ({y_true_unique})."
+            )


These checks could be done in decision_threshold_curve instead, not sure which is better.

lucyleeow · 2025-05-15T02:57:19Z

sklearn/metrics/_scorer.py

-            for th in potential_thresholds
-        ]
-        return np.array(score_thresholds), potential_thresholds
+        # why 'potential' ?


Just for my education, why use the term "potential", in "potential_thresholds". Is it because there a possibility that a threshold is redundant because the predicted labels are the same for adjacent thresholds?

lucyleeow · 2025-05-15T03:01:06Z

@glemaitre I've highlighted questions in the code.

Tests still to be added for decision_threshold_curve. Depending on where the validation checks are done, I don't think we need many value checks, as the function just uses _CurveScorer methods, which should be tested elsewhere.

jeremiedbb

Thanks for the PR @lucyleeow. I haven't looked at the whole PR yet. I'm wondering if there's a way to not have threshold as a parameter.

jeremiedbb · 2025-07-28T10:25:58Z

sklearn/metrics/_decision_threshold.py

+    thresholds : int or array-like, default=20
+        Specifies number of decision thresholds to compute score for. If an integer,
+        it will be used to generate `thresholds` thresholds uniformly distributed
+        between the minimum and maximum of `y_score`. If an array-like, it will be
+        used as the thresholds.


I believe we can compute the thresholds automatically, like in the other curve functions.
score_func is called on y_true (independent of the threshold) and y_pred where y_pred is computed from y_score and the threshold.

If we sort y_score in ascending order, it's only where the value of sorted_y_score changes that we need to set a threshold. That way we compute the exact number of thresholds. We can probably leverage some code from _binary_clf_curve

scikit-learn/sklearn/metrics/_ranking.py

Lines 891 to 906 in 18e89a4

# sort scores and corresponding truth values

desc_score_indices = xp.argsort(y_score, stable=True, descending=True)

y_score = y_score[desc_score_indices]

y_true = y_true[desc_score_indices]

if sample_weight is not None:

weight = sample_weight[desc_score_indices]

else:

weight = 1.0

# y_score typically has many tied values. Here we extract

# the indices associated with the distinct values. We also

# concatenate a value for the end of the curve.

distinct_value_indices = xp.nonzero(xp.diff(y_score))[0]

threshold_idxs = xp.concat(

[distinct_value_indices, xp.asarray([size(y_true) - 1], device=device)]

)

This is a good point, I have added a threshold=None option that does the above.

Would you care to take a quick look at the overall PR before I add tests, as I am not sure about a few things, thanks 🙏

jeremiedbb · 2025-10-08T08:49:28Z

Sorry for the delay, there was something puzzling me with the approach of the problem in this PR and I had a hard time figuring out precisely what it was. In particular, the need for thresholds and greater_is_better to be part of the API didn't feel right.

To me the issue is the use of the CurveScorer to tackle this. We don't need to transform the metric into a scorer to compute the curve. What we need is to a lot closer to the _binary_clf_curve

scikit-learn/sklearn/metrics/_ranking.py

Line 830 in ea9e824

def _binary_clf_curve(y_true, y_score, pos_label=None, sample_weight=None):

The difference is that instead of computing the true positives and false positives, we'd compute any (threshold dependent) metric. Briefly, this is done in 3 steps:

sort the arrays by y_score
compute the indices in y_score that will be the thresholds, I.e. at each index where sorted y_score changes.
compute the metric for these thresholds

I think that we can extract most of the _binary_clf_curve code inside a new, more generic, function. Then make _binary_clf_curve call this new function and decision_threshold_curve call this new function as well.

This is related to #30134 where the goal is to make _binary_clf_curve public and return the tns and fns as well.

lucyleeow · 2025-10-28T05:24:44Z

Interesting, I've reviewed #30134, hopefully that will be merged soon and I can look at your suggested changes after!

lucyleeow · 2025-11-18T01:35:53Z

@jeremiedbb I've tried your approach in a separate PR #32732, as I thought it was nice to be able to see the two implementations side by side, and also it would have been complicated history/changes wise to do it all in this PR.

I think your suggested approach is better, please share your thoughts!

vitaliset and others added 30 commits February 18, 2023 01:47

initial proposal with preliminary tests

172ac47

removing check that validate_params already does

d038e11

changelog and linting from CI

322eccf

trying to resolve doc related ci

7dbbec5

duplicate label

2a0c6b3

docstring example import error

fbb9b9b

docstring typo

acb94be

docstring typo

a5cd201

docstring typo

253b3e2

docstring typo

cb5fee1

change in doc order and typos

9e45e2e

removing example

ad901a2

Merge branch 'main' into metric_threshold_curve

1a4ce1b

Update import of _check_pos_label_consistency

9b4febb

codecov

119db53

Merge branch 'metric_threshold_curve' of https://github.com/vitaliset…

347f524

…/scikit-learn into metric_threshold_curve

linting

be893c8

correcting typo

bd1e64f

test typo

0318950

add example again to check pytest

efd6d72

Merge branch 'main' into metric_threshold_curve

1e500c0

Merge remote-tracking branch 'origin/main' into pr/vitaliset/25639

10ebc90

fixing imports

dfa66a5

towards glemaitre suggestions

1fb1c13

applying black suggestions

e7bb2a7

update extra stuff for consistency

5a8f0c5

removing doc files for now as we need to adapt to pr 29038

4fab2a3

Merge branch 'main' into metric_threshold_curve

48a0055

Merge branch 'main' into metric_threshold_curve

fbf1d2e

Merge branch 'main' into metric_threshold_curve

98873e6

lucyleeow added 3 commits May 8, 2025 19:18

revert from scorer order

8efaaca

amend to method

6e0b5e0

fix param valid, use greater_is_better

0ac8d1d

glemaitre self-requested a review May 9, 2025 19:06

lucyleeow added 9 commits May 12, 2025 12:02

fix example

8a8e240

Merge branch 'main' into metric_threshold_curve

6715d6d

fix examples

bf887aa

rm pos label as param

edf99f0

typo

9c624c5

pos label fixes

7643942

fix kwargs

13c8545

add user guide section

b589505

change ref label

577ea24

lucyleeow commented May 15, 2025

View reviewed changes

jeremiedbb reviewed Jul 28, 2025

View reviewed changes

lucyleeow added 3 commits September 6, 2025 21:27

merge main

38068b2

fix imports

71fe4f1

auto calculate thresholds

5b1aba1

lucyleeow added 6 commits October 28, 2025 16:27

merge main

6cb8c66

fix bad merge, and param constraint

ac1ab8d

fix bad merge

4b4dd30

fix test;

929003e

fix docstring

5389765

fix docstring

11badf0

glemaitre assigned lucyleeow Nov 14, 2025

lucyleeow mentioned this pull request Nov 18, 2025

FEA Add decision_threshold_curve (approach 2) #32732

Open

	# sort scores and corresponding truth values
	desc_score_indices = xp.argsort(y_score, stable=True, descending=True)
	y_score = y_score[desc_score_indices]
	y_true = y_true[desc_score_indices]
	if sample_weight is not None:
	weight = sample_weight[desc_score_indices]
	else:
	weight = 1.0

	# y_score typically has many tied values. Here we extract
	# the indices associated with the distinct values. We also
	# concatenate a value for the end of the curve.
	distinct_value_indices = xp.nonzero(xp.diff(y_score))[0]
	threshold_idxs = xp.concat(
	[distinct_value_indices, xp.asarray([size(y_true) - 1], device=device)]
	)

Uh oh!

FEA Adds decision_threshold_curve function #31338

Are you sure you want to change the base?

FEA Adds decision_threshold_curve function #31338

Conversation

lucyleeow commented May 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reference Issues/PRs

What does this implement/fix? Explain your changes.

Any other comments?

Uh oh!

lucyleeow May 15, 2025

Choose a reason for hiding this comment

Uh oh!

lucyleeow May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lucyleeow May 15, 2025

Choose a reason for hiding this comment

Uh oh!

lucyleeow May 15, 2025

Choose a reason for hiding this comment

Uh oh!

lucyleeow May 15, 2025

Choose a reason for hiding this comment

Uh oh!

lucyleeow May 15, 2025

Choose a reason for hiding this comment

Uh oh!

lucyleeow commented May 15, 2025

Uh oh!

jeremiedbb left a comment

Choose a reason for hiding this comment

Uh oh!

jeremiedbb Jul 28, 2025

Choose a reason for hiding this comment

Uh oh!

lucyleeow Sep 8, 2025

Choose a reason for hiding this comment

Uh oh!

jeremiedbb commented Oct 8, 2025

Uh oh!

lucyleeow commented Oct 28, 2025

Uh oh!

lucyleeow commented Nov 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

FEA Adds `decision_threshold_curve` function #31338

FEA Adds `decision_threshold_curve` function #31338

lucyleeow commented May 8, 2025 •

edited

Loading

lucyleeow May 15, 2025 •

edited

Loading