Python: fix clingo bootstrapping on Apple M1 by adamjstewart · Pull Request #30834 · spack/spack

adamjstewart · 2022-05-25T01:29:41Z

This PR fixes several issues I noticed while trying to get Spack working on Apple M1.

build_environment.py attempts to add spec['foo'].libs and spec['foo'].headers to our compiler wrappers for all dependencies using a try-except that ignores NoLibrariesError and NoHeadersError respectively. However, The libs and headers attributes of the Python package were erroneously using RuntimeError instead.
spack external find python (used during bootstrapping) currently has no way to determine whether or not an installation is +shared, so previously we would only search for static Python libs. However, most distributions including XCode/Conda/Intel ship shared Python libs. I updated libs to search for both shared and static (order based on variant) as a fallback.
The headers attribute was recursively searching in prefix.include for pyconfig.h, but this could lead to non-deterministic behavior if multiple versions of Python are installed and pyconfig.h files exist in multiple <prefix>/include/pythonX.Y locations. It's safer to search in sysconfig.get_path('include') instead.
The Python installation that comes with XCode is broken, and sysconfig.get_paths is hard-coded to return specific directories. This meant that our logic for platlib/purelib/include where we replace platbase/base/installed_base with prefix wasn't working and the mkdirp in setup_dependent_package was trying to create a directory in root, giving permissions issues. Even if you commented out those mkdirp calls, Spack would add the wrong directories to PYTHONPATH. Added a fallback hard-coded to lib/pythonX.Y/site-packages if sysconfig is broken (this is what distutils always did).

The result of all these changes? I still can't bootstrap Spack on M1! It seems like the version of Python distributed with XCode doesn't include the development libraries? Maybe @alalazo @tgamblin know more about the following CMake error:

==> clingo-bootstrap: Executing phase: 'cmake'
==> [2022-05-24-15:47:00.404703] 'cmake' '-G' 'Unix Makefiles' '-DCMAKE_INSTALL_PREFIX:STRING=/Users/ajstewart/.spack/bootstrap/store/darwin-monterey-aarch64/apple-clang-13.1.6/clingo-bootstrap-spack-suuyj2i6n5dhctps6ftpnrfkz7uborhi' '-DCMAKE_BUILD_TYPE:STRING=Release' '-DBUILD_TESTING:BOOL=OFF' '-DCMAKE_INTERPROCEDURAL_OPTIMIZATION:BOOL=OFF' '-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON' '-DCMAKE_FIND_FRAMEWORK:STRING=LAST' '-DCMAKE_FIND_APPBUNDLE:STRING=LAST' '-DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON' '-DCMAKE_INSTALL_RPATH:STRING=/Users/ajstewart/.spack/bootstrap/store/darwin-monterey-aarch64/apple-clang-13.1.6/clingo-bootstrap-spack-suuyj2i6n5dhctps6ftpnrfkz7uborhi/lib;/Users/ajstewart/.spack/bootstrap/store/darwin-monterey-aarch64/apple-clang-13.1.6/clingo-bootstrap-spack-suuyj2i6n5dhctps6ftpnrfkz7uborhi/lib64;/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib' '-DCMAKE_PREFIX_PATH:STRING=/Users/ajstewart/.spack/bootstrap/store/darwin-monterey-aarch64/apple-clang-13.1.6/re2c-2.2-kx6h3wjm64tizph7wtdamnoyeijtpgcd;/Users/ajstewart/.spack/bootstrap/store/darwin-monterey-aarch64/apple-clang-13.1.6/cmake-3.23.1-cphj53473emsulziljrcdcsj6zyaihvv;/Users/ajstewart/.spack/bootstrap/store/darwin-monterey-aarch64/apple-clang-13.1.6/bison-3.8.2-b3rsh4rcums6uj3j3edsyvpunfxhwjqt;/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8' '-DCLINGO_REQUIRE_PYTHON=ON' '-DCLINGO_BUILD_WITH_PYTHON=ON' '-DPYCLINGO_USER_INSTALL=OFF' '-DPYCLINGO_USE_INSTALL_PREFIX=ON' '-DCLINGO_BUILD_WITH_LUA=OFF' '-DCLINGO_BUILD_PY_SHARED:STRING=OFF' '-DPython_EXECUTABLE:STRING=/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/bin/python3.8' '-DPython_INCLUDE_DIR:STRING=/Library/Python/3.8/include' '-DCLINGO_BUILD_APPS:STRING=OFF' '/var/folders/j1/68dlgpr91vlgs26vty2c8xk80000gn/T/ajstewart/spack-stage/spack-stage-clingo-bootstrap-spack-suuyj2i6n5dhctps6ftpnrfkz7uborhi/spack-src'
-- The C compiler identification is AppleClang 13.1.6.13160021
-- The CXX compiler identification is AppleClang 13.1.6.13160021
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Users/ajstewart/spack/lib/spack/env/clang/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Users/ajstewart/spack/lib/spack/env/clang/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Error at /Users/ajstewart/.spack/bootstrap/store/darwin-monterey-aarch64/apple-clang-13.1.6/cmake-3.23.1-cphj53473emsulziljrcdcsj6zyaihvv/share/cmake-3.23/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Python (missing: Python_LIBRARIES Interpreter Development
  Development.Module Development.Embed)
Call Stack (most recent call first):
  /Users/ajstewart/.spack/bootstrap/store/darwin-monterey-aarch64/apple-clang-13.1.6/cmake-3.23.1-cphj53473emsulziljrcdcsj6zyaihvv/share/cmake-3.23/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  /Users/ajstewart/.spack/bootstrap/store/darwin-monterey-aarch64/apple-clang-13.1.6/cmake-3.23.1-cphj53473emsulziljrcdcsj6zyaihvv/share/cmake-3.23/Modules/FindPython.cmake:561 (find_package_handle_standard_args)
  CMakeLists.txt:130 (find_package)

Anyway, we really need bootstrapping binaries for clingo on M1. I hope that this PR is a step in the right direction for native building, and may work for most users, especially if you have Python provided by Homebrew/Conda/source install.

Fixes #28183 @rchoudhary @mathstuf @kgerheiser @tmenari @healther
Fixes #28190 @certik @sethrj
Fixes #29255 @kgerheiser
Fixes #30513 @bryank-cs
Fixes #30587 @adrienbernede

certik · 2022-05-25T01:54:32Z

Does this PR disable clingo bootstrapping? I tested it with a fresh clone of Spack, and it seems to work (with Conda), but I don't see any clingo being bootstrapped:

$ conda create create -n spack python=3.10
$ conda activate spack
$ ./bin/spack spec zlib
Input spec
--------------------------------
zlib

Concretized
--------------------------------
[email protected]%[email protected]+optimize+pic+shared patches=0d38234 arch=darwin-monterey-m1
$ ./bin/spack install zlib
==> Installing zlib-1.2.12-kv442a3m27eyisk3msyutprlor6xfqrc
==> No binary for zlib-1.2.12-kv442a3m27eyisk3msyutprlor6xfqrc found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/91/91844808532e5ce316b3c010929493c0244f3d37593afd6de04f71821d5136d9.tar.gz
==> Applied patch /private/tmp/spack/var/spack/repos/builtin/packages/zlib/configure-cc.patch
==> zlib: Executing phase: 'install'
==> zlib: Successfully installed zlib-1.2.12-kv442a3m27eyisk3msyutprlor6xfqrc
  Fetch: 1.73s.  Build: 1.28s.  Total: 3.01s.
[+] /private/tmp/spack/opt/spack/darwin-monterey-m1/apple-clang-13.0.0/zlib-1.2.12-kv442a3m27eyisk3msyutprlor6xfqrc

With master, this is how you get Spack bootstrapped on M1 (just command line tools needed, I don't even have Xcode): #28190 (comment)

adamjstewart · 2022-05-25T02:23:59Z

This PR does not disable bootstrapping. I would first check to see if you already have things bootstrapped in ~/.spack. You can run spack clean -b to clear this out. If that doesn't exist, try running spack config get config and make sure concretizer is set to clingo instead of original.

adamjstewart · 2022-05-25T16:48:26Z

@spackbot run pipeline

spackbot-app · 2022-05-25T16:48:29Z

I've started that pipeline for you!

kgerheiser · 2022-05-25T16:50:38Z

Nice, this might also fix #29255

adamjstewart · 2022-05-25T16:53:05Z

Yes, this PR should def fix that issue, I wonder if there are any others.

kgerheiser · 2022-05-25T17:55:26Z

So, this won't fix #29255 because sysconfig for LDLIBRARY and LIBRARY both say libpython3.9.a, but the installation actually has shared libraries. Intel builds the interpreter with static libraries and installs the interpreter. Then, builds shared libraries and install them in /lib.

Would it be possible to fallback to the default config vars in config_vars, so that it'll search for

spack/var/spack/repos/builtin/packages/python/package.py

Lines 945 to 959 in 7d3e5c9

    
           if dag_hash not in self._config_vars: 
        
               # Default config vars 
        
               version = self.version.up_to(2) 
        
               config = { 
        
                   # get_config_vars 
        
                   'BINDIR': self.prefix.bin, 
        
                   'CC': 'cc', 
        
                   'CXX': 'c++', 
        
                   'INCLUDEPY': self.prefix.include.join('python{}').format(version), 
        
                   'LIBDEST': self.prefix.lib.join('python{}').format(version), 
        
                   'LIBDIR': self.prefix.lib, 
        
                   'LIBPL': self.prefix.lib.join('python{0}').join( 
        
                       'config-{0}-{1}').format(version, sys.platform), 
        
                   'LDLIBRARY': 'libpython{}.{}'.format(version, dso_suffix), 
        
                   'LIBRARY': 'libpython{}.a'.format(version),

adamjstewart · 2022-05-25T18:01:53Z

Hmm, we could search LDLIBRARY, LIBRARY, libpythonX.Y.so, libpythonX.Y.dylib, libpythonX.Y.a, etc. It gets more complicated because historical versions of Python used to append m or d to malloc and debug builds, so the library name was really hard to guess. It seems to me like the Intel Python installation is broken and that this is a bug that the Intel folks should fix. But maybe we can add a workaround...

kgerheiser · 2022-05-25T18:08:54Z

Ok, maybe not necessary. Intel accepted my bug report and says they're going to fix it in the next release.

https://community.intel.com/t5/Intel-Distribution-for-Python/libpython3-9-a-does-not-exist/td-p/1363865?emcs_t=S2h8ZW1haWx8c29sdXRpb25fdG9fcXVlc3Rpb258TDFISjJPVUM5TktVTEl8MTM3Mzc2NnxBQ0NFUFRFRF9TT0xVVElPTlN8aEs

adamjstewart · 2022-05-25T18:13:59Z

Awesome! I fixed it anyway, try the latest commit.

adamjstewart · 2022-05-26T04:19:32Z

@spackbot run pipeline

spackbot-app · 2022-05-26T04:19:35Z

I've started that pipeline for you!

tgamblin · 2022-05-27T10:17:20Z

The failing rdma-core build here was already fixed on develop, so I'm merging this without re-running the pipeline.

This PR fixes several issues I noticed while trying to get Spack working on Apple M1. - [x] `build_environment.py` attempts to add `spec['foo'].libs` and `spec['foo'].headers` to our compiler wrappers for all dependencies using a try-except that ignores `NoLibrariesError` and `NoHeadersError` respectively. However, The `libs` and `headers` attributes of the Python package were erroneously using `RuntimeError` instead. - [x] `spack external find python` (used during bootstrapping) currently has no way to determine whether or not an installation is `+shared`, so previously we would only search for static Python libs. However, most distributions including XCode/Conda/Intel ship shared Python libs. I updated `libs` to search for both shared and static (order based on variant) as a fallback. - [x] The `headers` attribute was recursively searching in `prefix.include` for `pyconfig.h`, but this could lead to non-deterministic behavior if multiple versions of Python are installed and `pyconfig.h` files exist in multiple `<prefix>/include/pythonX.Y` locations. It's safer to search in `sysconfig.get_path('include')` instead. - [x] The Python installation that comes with XCode is broken, and `sysconfig.get_paths` is hard-coded to return specific directories. This meant that our logic for `platlib`/`purelib`/`include` where we replace `platbase`/`base`/`installed_base` with `prefix` wasn't working and the `mkdirp` in `setup_dependent_package` was trying to create a directory in root, giving permissions issues. Even if you commented out those `mkdirp` calls, Spack would add the wrong directories to `PYTHONPATH`. Added a fallback hard-coded to `lib/pythonX.Y/site-packages` if sysconfig is broken (this is what distutils always did).

adrienbernede · 2022-05-31T14:45:23Z

Thank you @adamjstewart. I can confirm that I get the same error you mentioned. So still not able to bootstrap from source.

adamjstewart · 2022-05-31T14:49:34Z

Try #30894

This PR fixes several issues I noticed while trying to get Spack working on Apple M1. - [x] `build_environment.py` attempts to add `spec['foo'].libs` and `spec['foo'].headers` to our compiler wrappers for all dependencies using a try-except that ignores `NoLibrariesError` and `NoHeadersError` respectively. However, The `libs` and `headers` attributes of the Python package were erroneously using `RuntimeError` instead. - [x] `spack external find python` (used during bootstrapping) currently has no way to determine whether or not an installation is `+shared`, so previously we would only search for static Python libs. However, most distributions including XCode/Conda/Intel ship shared Python libs. I updated `libs` to search for both shared and static (order based on variant) as a fallback. - [x] The `headers` attribute was recursively searching in `prefix.include` for `pyconfig.h`, but this could lead to non-deterministic behavior if multiple versions of Python are installed and `pyconfig.h` files exist in multiple `<prefix>/include/pythonX.Y` locations. It's safer to search in `sysconfig.get_path('include')` instead. - [x] The Python installation that comes with XCode is broken, and `sysconfig.get_paths` is hard-coded to return specific directories. This meant that our logic for `platlib`/`purelib`/`include` where we replace `platbase`/`base`/`installed_base` with `prefix` wasn't working and the `mkdirp` in `setup_dependent_package` was trying to create a directory in root, giving permissions issues. Even if you commented out those `mkdirp` calls, Spack would add the wrong directories to `PYTHONPATH`. Added a fallback hard-coded to `lib/pythonX.Y/site-packages` if sysconfig is broken (this is what distutils always did).

Python: fix clingo bootstrapping on Apple M1

57c5fd7

adamjstewart added python bootstrap Anything that has to do with Spack building its own dependencies. labels May 25, 2022

spackbot-app bot added libraries update-package labels May 25, 2022

spackbot-app bot requested review from scheibelp, skosukhin and varioustoxins May 25, 2022 01:29

Style fix

7d3e5c9

Fix Intel Python lib detection

618e777

tgamblin approved these changes May 27, 2022

View reviewed changes

tgamblin merged commit a3a8710 into spack:develop May 27, 2022

tgamblin mentioned this pull request May 27, 2022

Fix clingo source bootstrapping on M1 macs #30894

Closed

adamjstewart deleted the packages/python branch May 27, 2022 14:53

alalazo mentioned this pull request Jun 8, 2022

Backports v0.18.1 #31049

Merged

33 tasks

This was referenced Jul 28, 2022

Bootstrap issue: cannot find Python (new apple M2) #31735

Closed

Python: fix clingo bootstrapping on Apple M1/M2 #31792

Merged

Conversation

adamjstewart commented May 25, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

certik commented May 25, 2022

Uh oh!

adamjstewart commented May 25, 2022

Uh oh!

adamjstewart commented May 25, 2022

Uh oh!

spackbot-app bot commented May 25, 2022

Uh oh!

kgerheiser commented May 25, 2022

Uh oh!

adamjstewart commented May 25, 2022

Uh oh!

kgerheiser commented May 25, 2022

Uh oh!

adamjstewart commented May 25, 2022

Uh oh!

kgerheiser commented May 25, 2022

Uh oh!

adamjstewart commented May 25, 2022

Uh oh!

adamjstewart commented May 26, 2022

Uh oh!

spackbot-app bot commented May 26, 2022

Uh oh!

tgamblin commented May 27, 2022

Uh oh!

adrienbernede commented May 31, 2022

Uh oh!

adamjstewart commented May 31, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

adamjstewart commented May 25, 2022 •

edited

Loading