I think my comment #22475 (comment) still applies.

This should not be useful for any Python package; if they need this change then they're doing dependency discovery incorrectly in their build config files.

Packages typically provide a get_include function in the top-level namespace (at least NumPy, Pybind11, mpi4py and Pythran do exactly this; mpi4py also has get_config), but of course that's slightly cumbersome for end users. So this may be useful for Spack end users. It's a little hacky though, and it's hard to tell if there's a risk of false positives. So I'm a little on the fence on this one.

@scottwittenburg this PR is a pretty disruptive change that may possibly break many Python packages in Spack. However, it doesn't change the spec hash, so it looks like GitLab CI is skipped entirely. Is there any way to force run CI? I guess I could add a temp commit to change the default versions of Python and see what breaks...

We still don't have a good way to force rebuilding all the stacks from source :-( But what you suggest is one way to achieve it.

However, is it possible that some sub-dags among all the tested stacks involve specific versions of python (rather than the default one), and thus the hashes in those sub-dags still wouldn't change? If that's a possibility, a way to force rebuilding everything is a temporary commit that changes the mirror urls in any stacks you want to be sure to rebuild completely from source. So, e.g.:

diff --git a/share/spack/gitlab/cloud_pipelines/stacks/e4s/spack.yaml b/share/spack/gitlab/cloud_pipelines/stacks/e4s/spack.yaml
index 0fd946a0ab..59ef42051e 100644
--- a/share/spack/gitlab/cloud_pipelines/stacks/e4s/spack.yaml
+++ b/share/spack/gitlab/cloud_pipelines/stacks/e4s/spack.yaml
@@ -219,7 +219,7 @@ spack:
     - - $cuda_specs
     - - $arch
 
-  mirrors: { "mirror": "s3://spack-binaries-develop/e4s" }
+  mirrors: { "mirror": "s3://spack-binaries-develop/force-rebuild/e4s" }
 
   gitlab-ci:

Far from ideal, I know, but if you're making a temporary commit anyway, something like that (repeat for whatever stacks you want to force rebuild) should do the trick. Note however that after the first pipeline with that temp change, the PR-specific mirror created for your PR will contain cached binaries for any builds that succeeded.

GitLab CI passed! I'll revert the last commit now.

So now the only remaining question is whether or not this is the right thing to do. I think @rgommers raises a good point. Unlike most Autotools/CMake packages where libs/headers get installed directly to /usr/lib or /usr/include, these libs/headers are normally tucked away in <prefix>/lib/pythonX.Y/site-packages/<package>. These paths don't normally end up being searched by the compiler unless the build system explicitly tells it to. It's possible that we could see a lot of namespace pollution, especially for header files with generic names. Fortunately we didn't see any of that in the CI, so maybe I'm just paranoid.

I can't speak for the rationale in #14513 and #22475, but I added the libs/headers attributes to PyTorch in #18093. I'm guessing I encountered some kind of issue with torchvision finding the PyTorch libs and needed to modify the package, but I wonder if that's really necessary. Our mpi4py package also has headers (added in #17295).

Maybe @mdorier @RemiLacroix-IDRIS @sebasgo can chime in with their motivation for #14513 #22475 #17295?

In my original issue, I had developed a python library with C headers and noticed that C/C++ programs could not include it due to the include path having pythonX.Y. Adding a get_include function to the library did not solve the problem because C/C++ codes used cmake or autotools as build system. I don't remember how I solved this problem. I probably hard-coded a cmake function that tries to figure out the header path.

Maybe @mdorier @RemiLacroix-IDRIS @sebasgo can chime in with their motivation for #14513 #22475 #17295?

I don't remember exactly but I think I had a package that depended on Numpy's headers and they could not be found without overriding the headers property.

Pinging other Python maintainers for review: @scheibelp @skosukhin @varioustoxins

Hmm, this doesn't seem to be working. When I try to build py-torchvision, which uses:

        for dep in self.spec.dependencies(deptype='link'):                               
            query = self.spec[dep.name]                                                  
            include.extend(query.headers.directories)                                    
            library.extend(query.libs.directories)

it crashes when trying to find py-torch headers:

Error: NoHeadersError: Unable to locate py-torch headers in /Users/ajstewart/spack/opt/spack/darwin-monterey-m1/apple-clang-13.1.6/py-torch-1.12.0-ci7hcwrdnhozwr2ecwo2kk7ifi5gqhfr

/Users/ajstewart/spack/var/spack/repos/builtin/packages/py-torchvision/package.py:116, in setup_build_environment:
        113            query = self.spec[dep.name]
        114            print('DEBUG: In py-torchvision.setup_build_environment')
        115            print('DEBUG:', dep.name)
  >>    116            include.extend(query.headers.directories)
        117            library.extend(query.libs.directories)
        118
        119        # README says to use TORCHVISION_INCLUDE and TORCHVISION_LIBRARY,


Traceback (most recent call last):
  File "/Users/ajstewart/spack/lib/spack/spack/build_environment.py", line 1058, in _setup_pkg_and_run
    kwargs['env_modifications'] = setup_package(
  File "/Users/ajstewart/spack/lib/spack/spack/build_environment.py", line 812, in setup_package
    pkg.setup_build_environment(env_mods)
  File "/Users/ajstewart/spack/var/spack/repos/builtin/packages/py-torchvision/package.py", line 117, in setup_build_environment
    library.extend(query.libs.directories)
  File "/Users/ajstewart/spack/lib/spack/spack/spec.py", line 1071, in __get__
    value = f()
  File "/Users/ajstewart/spack/lib/spack/spack/spec.py", line 1063, in <lambda>
    callbacks_chain.append(lambda: self.default(self, instance, cls))
  File "/Users/ajstewart/spack/lib/spack/spack/spec.py", line 949, in _headers_default_handler
    raise spack.error.NoHeadersError(
spack.error.NoHeadersError: Unable to locate py-torch headers in /Users/ajstewart/spack/opt/spack/darwin-monterey-m1/apple-clang-13.1.6/py-torch-1.12.0-ci7hcwrdnhozwr2ecwo2kk7ifi5gqhfr

Interestingly, it's crashing in spack.spec._headers_default_handler instead of in PythonPackage.headers. If I put print statements in both, it seems like both are being called. @alalazo any idea why this would be the case? Can headers not be defined in a base class?

Adding the exact same code from the base class directly to the py-torch class fixes my issue. Why? Will open a PR to fix torchvision in the meantime but we should figure out why this PR didn't work.

@alalazo any ideas on the above bug?

Fixed in #32970