BUG: numpy 1.26.0 no longer finds MKL when building from source #24808
Description
Describe the issue:
Hi there! I'm not sure if this is a bug or a question, and I couldn't work out how to submit a question, so sorry in advance if this is the wrong format/place.
We use poetry to manage our project dependencies, and poetry builds numpy from source in order to compile it with MKL rather than openblas (using poetry.toml). Until version 1.26.0, this has worked fine. With version 1.26.0, during the source build of numpy, meson fails with an error claiming that it cannot find BLAS despite MKL being installed as before. Is that expected or should it still find it?
I found documentation surrounding meson and BLAS difficult to navigate and potentially incomplete, so I'm not sure if there is something I should be doing now to ensure the numpy can find MKL. The scipy documentation which is referenced suggests that we'd need to pass command line arguments to a build script, and I don't know whether that is possible when building from source using poetry.
To reproduce the issue, extract this project archive: numpy-install-test.tar.gz
It contains a Dockerfile which installs the dependencies that we've been successfully using up to v1.25.2, including mkl. The is test_environment.py script shows that mkl is found and used. There is a bunch of poetry setup stuff but that can probably be ignore for this issue, the apt-get install command is the most relevant.
The Makefile target test-mkl will build the docker container and run test_environment.py.
Run NUMPY_VERSION=1.25.2 make test-mkl to show that it installs and finds MKL with version 1.25.2
Run NUMPY_VERSION=1.26.0 make test-mkl to show that it fails to find MKL with version 1.26.0
A bit of extra context which may be relevant: this is actually a simpler version of the issue we've encountered. In reality, we have a dependency asammdf which depends on an untagged version of numpy as a build dependency, and we also build asammdf package from source, which as a result builds the latest release of numpy from source. Prior to v1.26.0 being released we have no had any issues with this build. Unfortunately, since this build dependency is not pinned by asammdf, this issue was sprung upon us without us making any changes.
Reproduce the code example:
N/A see description for reproduction instructionsError message:
#19 0.845 [urllib3.connectionpool] https://files.pythonhosted.org:443 "GET /packages/55/b3/b13bce39ba82b7398c06d10446f5ffd5c07db39b09bd37370dc720c7951c/numpy-1.26.0.tar.gz HTTP/1.1" 200 15633455
#19 12.65
#19 12.65 Stack trace:
#19 12.65
#19 12.65 8 /opt/poetry-venv/lib/python3.11/site-packages/poetry/installation/executor.py:286 in _execute_operation
#19 12.67 284│
#19 12.67 285│ try:
#19 12.67 → 286│ result = self._do_execute_operation(operation)
#19 12.67 287│ except EnvCommandError as e:
#19 12.67 288│ if e.e.returncode == -2:
#19 12.67
#19 12.67 7 /opt/poetry-venv/lib/python3.11/site-packages/poetry/installation/executor.py:388 in _do_execute_operation
#19 12.70 386│ return 0
#19 12.70 387│
#19 12.70 → 388│ result: int = getattr(self, f"_execute_{method}")(operation)
#19 12.70 389│
#19 12.70 390│ if result != 0:
#19 12.70
#19 12.70 6 /opt/poetry-venv/lib/python3.11/site-packages/poetry/installation/executor.py:508 in _execute_install
#19 12.73 506│
#19 12.73 507│ def _execute_install(self, operation: Install | Update) -> int:
#19 12.73 → 508│ status_code = self._install(operation)
#19 12.73 509│
#19 12.73 510│ self._save_url_reference(operation)
#19 12.73
#19 12.73 5 /opt/poetry-venv/lib/python3.11/site-packages/poetry/installation/executor.py:546 in _install
#19 12.75 544│ archive = self._download_link(operation, Link(package.source_url))
#19 12.75 545│ else:
#19 12.75 → 546│ archive = self._download(operation)
#19 12.75 547│
#19 12.75 548│ operation_message = self.get_operation_message(operation)
#19 12.75
#19 12.75 4 /opt/poetry-venv/lib/python3.11/site-packages/poetry/installation/executor.py:750 in _download
#19 12.77 748│ self._yanked_warnings.append(message)
#19 12.77 749│
#19 12.77 → 750│ return self._download_link(operation, link)
#19 12.77 751│
#19 12.77 752│ def _download_link(self, operation: Install | Update, link: Link) -> Path:
#19 12.77
#19 12.77 3 /opt/poetry-venv/lib/python3.11/site-packages/poetry/installation/executor.py:800 in _download_link
#19 12.79 798│ self._write(operation, message)
#19 12.79 799│
#19 12.79 → 800│ archive = self._chef.prepare(archive, output_dir=output_dir)
#19 12.79 801│
#19 12.79 802│ # Use the original archive to provide the correct hash.
#19 12.79
#19 12.79 2 /opt/poetry-venv/lib/python3.11/site-packages/poetry/installation/chef.py:101 in prepare
#19 12.79 99│ return self._prepare(archive, destination=destination, editable=editable)
#19 12.79 100│
#19 12.79 → 101│ return self._prepare_sdist(archive, destination=output_dir)
#19 12.79 102│
#19 12.79 103│ def _prepare(
#19 12.79
#19 12.79 1 /opt/poetry-venv/lib/python3.11/site-packages/poetry/installation/chef.py:185 in _prepare_sdist
#19 12.80 183│ destination.mkdir(parents=True, exist_ok=True)
#19 12.80 184│
#19 12.80 → 185│ return self._prepare(
#19 12.80 186│ sdist_dir,
#19 12.80 187│ destination,
#19 12.80
#19 12.80 ChefBuildError
#19 12.80
#19 12.80 Backend subprocess exited when trying to invoke build_wheel
#19 12.80
#19 12.80 + /tmp/tmpui_ig9n8/.venv/bin/python /tmp/tmpuvtebmnp/numpy-1.26.0/vendored-meson/meson/meson.py setup /tmp/tmpuvtebmnp/numpy-1.26.0 /tmp/tmpuvtebmnp/numpy-1.26.0/.mesonpy-7a2l1x5j/build -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/tmp/tmpuvtebmnp/numpy-1.26.0/.mesonpy-7a2l1x5j/build/meson-python-native-file.ini
#19 12.80 The Meson build system
#19 12.80 Version: 1.2.99
#19 12.80 Source dir: /tmp/tmpuvtebmnp/numpy-1.26.0
#19 12.80 Build dir: /tmp/tmpuvtebmnp/numpy-1.26.0/.mesonpy-7a2l1x5j/build
#19 12.80 Build type: native build
#19 12.80 Project name: NumPy
#19 12.80 Project version: 1.26.0
#19 12.80 C compiler for the host machine: cc (gcc 10.2.1 "cc (Debian 10.2.1-6) 10.2.1 20210110")
#19 12.80 C linker for the host machine: cc ld.bfd 2.35.2
#19 12.80 C++ compiler for the host machine: c++ (gcc 10.2.1 "c++ (Debian 10.2.1-6) 10.2.1 20210110")
#19 12.80 C++ linker for the host machine: c++ ld.bfd 2.35.2
#19 12.80 Cython compiler for the host machine: cython (cython 3.0.2)
#19 12.80 Host machine cpu family: x86_64
#19 12.80 Host machine cpu: x86_64
#19 12.80 Program python found: YES (/tmp/tmpui_ig9n8/.venv/bin/python)
#19 12.80 Found pkg-config: /usr/bin/pkg-config (0.29.2)
#19 12.80 Run-time dependency python found: YES 3.11
#19 12.80 Has header "Python.h" with dependency python-3.11: YES
#19 12.80 Compiler for C supports arguments -fno-strict-aliasing: YES
#19 12.80 Test features "SSE SSE2 SSE3" : Supported
#19 12.80 Test features "SSSE3" : Supported
#19 12.80 Test features "SSE41" : Supported
#19 12.80 Test features "POPCNT" : Supported
#19 12.80 Test features "SSE42" : Supported
#19 12.80 Test features "AVX" : Supported
#19 12.80 Test features "F16C" : Supported
#19 12.80 Test features "FMA3" : Supported
#19 12.80 Test features "AVX2" : Supported
#19 12.80 Test features "AVX512F" : Supported
#19 12.80 Test features "AVX512CD" : Supported
#19 12.80 Test features "AVX512_KNL" : Supported
#19 12.80 Test features "AVX512_KNM" : Supported
#19 12.80 Test features "AVX512_SKX" : Supported
#19 12.80 Test features "AVX512_CLX" : Supported
#19 12.80 Test features "AVX512_CNL" : Supported
#19 12.80 Test features "AVX512_ICL" : Supported
#19 12.80 Test features "AVX512_SPR" : Unsupported due to Arguments "-msse, -msse2, -msse3, -mssse3, -msse4.1, -mpopcnt, -msse4.2, -mavx, -mf16c, -mfma, -mavx2, -mno-mmx, -mavx512f, -mavx512cd, -mavx512vl, -mavx512bw, -mavx512dq, -mavx512vnni, -mavx512ifma, -mavx512vbmi, -mavx512vbmi2, -mavx512bitalg, -mavx512vpopcntdq, -mavx512fp16" are not supported
#19 12.80 Configuring npy_cpu_dispatch_config.h using configuration
#19 12.80 Message:
#19 12.80 CPU Optimization Options
#19 12.80 baseline:
#19 12.80 Requested : min
#19 12.80 Enabled : SSE SSE2 SSE3
#19 12.80 dispatch:
#19 12.80 Requested : max -xop -fma4
#19 12.80 Enabled : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_KNL AVX512_KNM AVX512_SKX AVX512_CLX AVX512_CNL AVX512_ICL
#19 12.80
#19 12.80 Library m found: YES
#19 12.80 Found CMake: /usr/bin/cmake (3.18.4)
#19 12.80 WARNING: CMake Toolchain: Failed to determine CMake compilers state
#19 12.80 Run-time dependency openblas found: NO (tried pkgconfig and cmake)
#19 12.80 Run-time dependency openblas found: NO (tried pkgconfig and cmake)
#19 12.80
#19 12.80 ../../numpy/meson.build:207:4: ERROR: Problem encountered: No BLAS library detected! Install one, or use the `allow-noblas` build option (note, this may be up to 100x slower for some linear algebra operations).
#19 12.80
#19 12.80 A full log can be found at /tmp/tmpuvtebmnp/numpy-1.26.0/.mesonpy-7a2l1x5j/build/meson-logs/meson-log.txt
#19 12.80
#19 12.80
#19 12.80 at /opt/poetry-venv/lib/python3.11/site-packages/poetry/installation/chef.py:147 in _prepare
#19 12.80 143│
#19 12.80 144│ error = ChefBuildError("\n\n".join(message_parts))
#19 12.80 145│
#19 12.80 146│ if error is not None:
#19 12.80 → 147│ raise error from None
#19 12.80 148│
#19 12.80 149│ return path
#19 12.80 150│
#19 12.80 151│ def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:
#19 12.80
#19 12.80 Note: This error originates from the build backend, and is likely not a problem with poetry but with numpy (1.26.0) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "numpy (==1.26.0)"'.
#19 12.80
#19 ERROR: process "/bin/sh -c poetry --no-cache install -vvv" did not complete successfully: exit code: 1
------
> [11/13] RUN poetry --no-cache install -vvv:
12.80 145│
12.80 146│ if error is not None:
12.80 → 147│ raise error from None
12.80 148│
12.80 149│ return path
12.80 150│
12.80 151│ def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:
12.80
12.80 Note: This error originates from the build backend, and is likely not a problem with poetry but with numpy (1.26.0) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "numpy (==1.26.0)"'.
12.80
------
Dockerfile:45
--------------------
43 | WORKDIR /code/
44 |
45 | >>> RUN poetry --no-cache install -vvv
46 |
47 | RUN cat pyproject.toml
--------------------
ERROR: failed to solve: process "/bin/sh -c poetry --no-cache install -vvv" did not complete successfully: exit code: 1
make: *** [Makefile:13: build-base] Error 1Runtime information:
sys.version:
3.11.4 (main, Jul 28 2023, 05:28:12) [GCC 10.2.1 20210110]
cannot import numpy because this is an installation issue
Context for the issue:
No response