You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hmm, the 1.41 library is five years old, maybe too early.
OTOH, VS2017 (which comes with VC141) is EOL now; as long as things like matthew-brett/dll_investigation#1 can be solved (which looks like it can be done with an extra flag), I think the cost/benefit of dropping VC141 should be considered, especially vis-à-vis how much dev time gets eaten by bugs like this one.
Conda-forge is on 14.1, as are our wheel builds. So we should just fix this right? Now that the problem is identified it shouldn't be too hard I'd think.
A little surprising that we didn't notice this elsewhere yet. Maybe we're not using 14.1 in regular CI anywhere yet, and didn't see it in the weekly numpy-wheels run yet (or ignored it there).
The 1.22.3+ wheels are built with vs2019 using the 1.41 toolchain
Right, so either 2017 or the 1.41 toolchain, but not the 1.42 toolchain. PR 151 goes to quite a bit of effort to make sure numpy-wheels produces wheels using the 1.41 toolchain.
Yes we are, two of them I believe
I see the prerelease_deps_coverage_64bit_blas azure job, which is running scipy + numpy weekly on ubuntu. I couldn't find one doing the same for windows
I think the cost/benefit of dropping VC141 should be considered
I agree with the sentiment, the question seems to be how to manage that for conda-forge.
My (personal) take is that conda-forge could easily make the move to vc142 - not least because we have complete control over the conda-forge-wide toolchain. The discussion already came up in the context of Azure deprecating VS2017, and all that's missing is someone to champion that change.
I've been holding off on doing that because it's not a fun discussion to have, but it's IMO overdue (e.g. LLVM 14 already moved to require VS2019+, and we have followed suit there as well.) - the only concern was for people using conda-environments to develop on windows, but having VS2017 installed locally. I claim (and would like to be challenged on this) that the set of affected users (who develop on windows using conda, aren't already directly or indirectly** affected by CI providers dropping VS2017, and cannot upgrade to a 100% ABI-compatible, 1:1 drop-in & free compiler) is empty, or very very close to it.
** Based on the aggressive removal of VS2017, many open source projects have already moved away (at the very least unintentionally, as numpy did originally for 1.22) from vc141, and that means bitrot & breakage for those using any such project as a dependency while trying to keep on living with an EOL compiler.
PS. I continue to think that we should have a toolchain-equivalent of NEP29. It's IMO not a productive use of everyone's time to keep rehashing many of the same points when it comes to toolchain upgrades. It think it would be worth the effort to come up with a reasonable policy for the various platforms / scenarios, and then stick to that.
I might even be convinced to write such a NEP, but last time I proposed this, the appetite for it was low.
If we choose to require MSVC 142 or greater as a solution to this issue, we should document it in the release notes. The code was introduced in PR #20133 and will be part of the upcoming 1.23 release. This means we require conda-forge to update before the release.
I started to try to work around the missing intrinsic in #21361, but the same CI commands that worked to set the toolchain in numpy-wheels did not work for me in that PR. I will see if I can set up a 141 toolchain locally.
The changes in the PATH environment variable are not carried over to the next azure job step, even though I copied the step from the numpy-wheels azure yaml file.
You can also use $env: as long as it is in the same powershell execution block. There are several levels of environment variables in powershell, so it gets complicated. See azure-steps-windows.yml for examples.
EDIT: That is for azure-pipelines, but perhaps for github actions as well.
@charris: I agree with the sentiment, the question seems to be how to manage that for conda-forge.
@h-vetinari: My (personal) take is that conda-forge could easily make the move to vc142 - not least because we have complete control over the conda-forge-wide toolchain. The discussion already came up in the context of Azure deprecating VS2017, and all that's missing is someone to champion that change.
Activity
charris commentedon Apr 15, 2022
@serge-sans-paille Thoughts?
serge-sans-paille commentedon Apr 15, 2022
This intrinsic comes with
<immintrin.h>(see https://www.intel.com/content/www/us/en/docs/intrinsics-guide/index.html#expand=4295&text=_knot_mask16&ig_expand=4166) which is unconditionally included innumpy/core/src/npysort/x86-qsort.dispatch.cppLooks like an MSVC issue: https://godbolt.org/z/nz63xTv8d
charris commentedon Apr 15, 2022
Hmm, the 1.41 library is five years old, maybe too early.
h-vetinari commentedon Apr 18, 2022
OTOH, VS2017 (which comes with VC141) is EOL now; as long as things like matthew-brett/dll_investigation#1 can be solved (which looks like it can be done with an extra flag), I think the cost/benefit of dropping VC141 should be considered, especially vis-à-vis how much dev time gets eaten by bugs like this one.
rgommers commentedon Apr 18, 2022
Conda-forge is on 14.1, as are our wheel builds. So we should just fix this right? Now that the problem is identified it shouldn't be too hard I'd think.
A little surprising that we didn't notice this elsewhere yet. Maybe we're not using 14.1 in regular CI anywhere yet, and didn't see it in the weekly
numpy-wheelsrun yet (or ignored it there).charris commentedon Apr 18, 2022
I checked that they are all being built and uploaded.
I agree with the sentiment, the question seems to be how to manage that for conda-forge.
mattip commentedon Apr 18, 2022
I don't think scipy is running a CI build against the weekly numpy wheels. The PyPI wheels are still built with visual 2017.
rgommers commentedon Apr 18, 2022
Yes we are, two of them I believe. Although one of them may be structurally timing out. But we've done this in SciPy for years.
charris commentedon Apr 18, 2022
The 1.22.3+ wheels are built with vs2019 using the 1.41 toolchain. I'm not clear at this point what that all means :)
mattip commentedon Apr 18, 2022
Right, so either 2017 or the 1.41 toolchain, but not the 1.42 toolchain. PR 151 goes to quite a bit of effort to make sure numpy-wheels produces wheels using the 1.41 toolchain.
I see the
prerelease_deps_coverage_64bit_blasazure job, which is running scipy + numpy weekly on ubuntu. I couldn't find one doing the same for windowsh-vetinari commentedon Apr 19, 2022
My (personal) take is that conda-forge could easily make the move to vc142 - not least because we have complete control over the conda-forge-wide toolchain. The discussion already came up in the context of Azure deprecating VS2017, and all that's missing is someone to champion that change.
I've been holding off on doing that because it's not a fun discussion to have, but it's IMO overdue (e.g. LLVM 14 already moved to require VS2019+, and we have followed suit there as well.) - the only concern was for people using conda-environments to develop on windows, but having VS2017 installed locally. I claim (and would like to be challenged on this) that the set of affected users (who develop on windows using conda, aren't already directly or indirectly** affected by CI providers dropping VS2017, and cannot upgrade to a 100% ABI-compatible, 1:1 drop-in & free compiler) is empty, or very very close to it.
** Based on the aggressive removal of VS2017, many open source projects have already moved away (at the very least unintentionally, as numpy did originally for 1.22) from vc141, and that means bitrot & breakage for those using any such project as a dependency while trying to keep on living with an EOL compiler.
h-vetinari commentedon Apr 19, 2022
PS. I continue to think that we should have a toolchain-equivalent of NEP29. It's IMO not a productive use of everyone's time to keep rehashing many of the same points when it comes to toolchain upgrades. It think it would be worth the effort to come up with a reasonable policy for the various platforms / scenarios, and then stick to that.
I might even be convinced to write such a NEP, but last time I proposed this, the appetite for it was low.
mattip commentedon Apr 19, 2022
If we choose to require MSVC 142 or greater as a solution to this issue, we should document it in the release notes. The code was introduced in PR #20133 and will be part of the upcoming 1.23 release. This means we require conda-forge to update before the release.
I started to try to work around the missing intrinsic in #21361, but the same CI commands that worked to set the toolchain in numpy-wheels did not work for me in that PR. I will see if I can set up a 141 toolchain locally.
matthew-brett commentedon Apr 19, 2022
@mattip - what didn't work - I'd like to know to make sure we're doing it right. Here's what I use locally:
mattip commentedon Apr 19, 2022
The changes in the
PATHenvironment variable are not carried over to the next azure job step, even though I copied the step from the numpy-wheels azure yaml file.matthew-brett commentedon Apr 19, 2022
@mattip - does the step lack a
#vso[task.prependpath]set?charris commentedon Apr 20, 2022
You can also use
$env:as long as it is in the same powershell execution block. There are several levels of environment variables in powershell, so it gets complicated. Seeazure-steps-windows.ymlfor examples.EDIT: That is for azure-pipelines, but perhaps for github actions as well.
matthew-brett commentedon Apr 20, 2022
Maybe this is the contrast between Azure workflows and Github actions? Setting paths in Github actions is via environment files - e.g. https://github.com/actions/toolkit/blob/main/docs/commands.md#powershell
mattip commentedon Apr 20, 2022
Let's continue the CI disussion in #21361, and leave this issue for the compilation error itself
h-vetinari commentedon Apr 22, 2022
I opened conda-forge/conda-forge.github.io#1732 to discuss this on the conda-forge side.