Can you please do that as part of the new KMeans pull request (#11950) to actually check that this is actually working as expected to protect against oversubscription issues.

done

If I understand correctly, we are still waiting for issues in threadpoolctl to be fixed, and good part of it is addressed in joblib 0.14 ?

Maybe let's mark this as WIP then?

and good part of it is addressed in joblib 0.14

joblib does not include threadpoolctl

I mean for handling oversubscritpion issues in process based parallelism.

If I understand correctly, we are still waiting for issues in threadpoolctl to be fixed,

It's done now, we released threadpoolctl 2.0 and I updated the PR accordingly.

I mean for handling oversubscritpion issues in process based parallelism.

joblib does not handle oversubscription in thread based parallelism which is needed for #11950

I think there is no point in merging this PR alone. I prefer to do it as part of the K-Means PR as for now, only the new K-Means implementation actually need threadpoolctl.

Why vendor threadpoolctl given that we recently unvendored joblib?

Why vendor threadpoolctl given that we recently unvendored joblib?

threadpoolctl is not included in joblib. We need it for KMeans to prevent oversubscription.

I understand that, this isn't my question.

My question is about vendoring vs having a dependency.

We can't introduce a new dependency without warning 2 versions in advance, right ?
I guess we could do both vendor and warn that it will be a dependency in 2 versions. What do you think @ogrisel ?

We can't introduce a new dependency without warning 2 versions in advance, right ?

There is no such rule. But unvendoring something public will require a deprecation cycle for backward compat but here threadpoolctl is vendored as a private module to avoid this problem.

I am ok to add a dependency instead of vendoring. This is a pure python project with a rather simple API that is unlikely to change. WDYT?

We probably need to discuss this at the next dev meeting.

I am ok to add a dependency instead of vendoring. This is a pure python project with a rather simple API that is unlikely to change. WDYT?

I'm ok with that. We need to be careful to make sure it will be installed properly when people will upgrade sklearn.

We probably need to discuss this at the next dev meeting.

agreed

I changed from vendoring to dependency in kmeans PR. I keep this one open until we discuss it in dev meeting.

I keep this one open until we discuss it in dev meeting.

So the decision was to use a dependency at the meeting (that I missed)? In which case should we close this PR?

Edit: OK, I see pending decision in #16242

There's a consensus on adding a dependency. Closing.