py-cdat-lite: ensure that RPATH includes netcdf by mjwoods · Pull Request #4521 · spack/spack

mjwoods · 2017-06-16T00:42:33Z

In py-cdat-lite, the netcdf library was not being found during import cdms2 (unless a suitable library was installed in a system directory). I now set the RPATH explicitly in the install method of the package.

adamjstewart

You should also consider using import_tests. That way, when you run spack install --run-tests py-cdat-lite, it will try to import cdms2 and crash if RPATH wasn't working properly. There's a bug where if you override phases you have to redefine tests, but I can help you with that.

adamjstewart · 2017-06-16T13:52:43Z

var/spack/repos/builtin/packages/py-cdat-lite/package.py

+        """Install everything from build directory."""
+        install_args = self.install_args(spec, prefix)
+        # Combine all phases into a single setup.py command,
+        # otherwise extensions are rebuilt without rpath by install phase:


Wait, really? I wouldn't expect that behavior. Maybe I'll have to test this one.

Btw, could this routine theoretically be used to install any Python package? Like in PythonPackage can we always run this sequence of commands, and is --rpath always available? If that is the case, we could solve the problem you've been seeing by always adding link dependencies to the --rpath.

Hi @adamjstewart , I think the need to combine phases is limited to this package. I didn't have the same trouble with py-pillow.

I think the --rpath option is part of py-setuptools, but it is only available for the build_ext phase. So to use it generally, we would have to include the build_ext phase by default.

Perhaps the more general solution would be to rely on the spack compiler wrappers to set rpath. But the wrappers are deliberately replaced by references to the real compilers during the python installation procedure. The reason seems to be to allow users to install their own packages without spack.

I am wondering if we need a way to run actual compiler versions outside of spack while still making use of the spack wrappers to define rpath correctly. For example, when a package like python is installed, the associated compiler could be represented by a wrapper script inside the python package. Then when python extensions are installed, the python compiler would invoke the correct compiler via the spack compiler wrappers, so that rpath and other search paths would always be set correctly. Does that idea make any sense?

otherwise extensions are rebuilt without rpath by install phase:

I figured out how to prevent this:

$ python setup.py install --skip-build

I'll add this to the default install phase.

Hmm, I don't know if --skip-build does what I think it does. According to the help message:

$ python setup.py install --help ... --skip-build skip rebuilding everything (for testing/debugging)

I figured that this meant I could run:

$ python setup.py build $ python setup.py install --skip-build

and separate out the logic, but this doesn't work at all for py-numpy. The build phase doesn't actually do much of anything, and the install phase crashes when I use --skip-build. So I won't be adding --skip-build. Guess I'll have to investigate this a bit more.

mjwoods · 2017-06-19T04:23:55Z

Hi again @adamjstewart , one other issue I came across was that the PythonPackage class defines methods like install_lib and install_headers without setting the package prefix (in contrast to the install method that most packages use). I think this is confusing and should either be fixed or mentioned in documentation somewhere.

adamjstewart · 2017-06-19T11:57:46Z

@mjwoods I'll fix the missing prefix for install_lib, etc. I'll have to do some major testing, but I think running the build_ext phase by default is a good idea. Another idea would be to unfilter the Python compiler wrappers every time a package is installed and re-filter them after, but I don't know how to do this within the current confines of Spack.

adamjstewart · 2017-06-19T17:02:20Z

@mjwoods It looks like all Python packages accept build_ext --rpath=... --include-dirs=... --library-dirs=... so I'll add those to the default PythonPackage phases and see if I can install every Python package with them.

I looked at install_lib and install_headers and it looks like those methods don't accept --prefix. Out of curiosity, do you have to use them? I don't think I've seen a package yet where install didn't work.

adamjstewart · 2017-06-19T20:20:05Z

I was looking for a way to specify the compiler to use but I think distutils chooses the compiler Python was built with no matter what you do. The closest thing I could find was:

$ python setup.py build --compiler

but that doesn't seem to accept paths, only the name of the compiler family:

$ python setup.py build --help-compiler
Running from numpy source directory.
List of available compilers:
  --compiler=bcpp      Borland C++ Compiler
  --compiler=cygwin    Cygwin port of GNU C Compiler for Win32
  --compiler=emx       EMX port of GNU C Compiler for OS/2
  --compiler=intel     Intel C Compiler for 32-bit applications
  --compiler=intele    Intel C Itanium Compiler for Itanium-based applications
  --compiler=intelem   Intel C Compiler for 64-bit applications
  --compiler=intelemw  Intel C Compiler for 64-bit applications on Windows
  --compiler=intelw    Intel C Compiler for 32-bit applications on Windows
  --compiler=mingw32   Mingw32 port of GNU C Compiler for Win32
  --compiler=msvc      Microsoft Visual C++
  --compiler=pathcc    PathScale Compiler for SiCortex-based applications
  --compiler=unix      standard UNIX-style compiler

mjwoods · 2017-06-20T06:01:03Z

Re: #4521 (comment)

I found lists of options for each install method by running commands like python setup.py install_dir --help. I notice that there is an option --install-dir that is common to all methods. My experiments suggest that --install-dir is not the same as --prefix, and each method would need a different --install-dir to create a directory tree with the usual sub-directories (e.g. bin, lib, include, share).

I considered using the separate installation methods when I found that install was rebuilding my package without --rpath. I worked around the problem by merging all of the build and install methods into a single phase. But I prefer your suggestion of using install --skip-build.

mjwoods · 2017-06-20T06:04:25Z

Re: #4521 (comment)

I think that refiltering the python compiler wrappers would be difficult to do reliably. For example, multiple packages could be installed simultaneously, leading to a race-condition. And we would need to restore the wrappers after various interruptions too.

adamjstewart · 2017-07-20T19:13:19Z

#4545 is taking longer than I would like so I'm going to merge this PR in the meantime.

py-cdat-lite: ensure that RPATH includes netcdf

ed5b102

adamjstewart requested changes Jun 16, 2017

View reviewed changes

adamjstewart added python rpath labels Jun 16, 2017

adamjstewart mentioned this pull request Jun 19, 2017

[WIP] PythonPackage: RPATH link dependencies #4545

Closed

adamjstewart approved these changes Jul 20, 2017

View reviewed changes

adamjstewart merged commit 7876262 into spack:develop Jul 20, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

py-cdat-lite: ensure that RPATH includes netcdf#4521

py-cdat-lite: ensure that RPATH includes netcdf#4521
adamjstewart merged 1 commit intospack:developfrom
mjwoods:py-cdat-lite-rpath

mjwoods commented Jun 16, 2017

Uh oh!

adamjstewart left a comment

Uh oh!

adamjstewart Jun 16, 2017

Uh oh!

mjwoods Jun 19, 2017

Uh oh!

adamjstewart Jun 19, 2017

Uh oh!

adamjstewart Jun 22, 2017

Uh oh!

mjwoods commented Jun 19, 2017

Uh oh!

adamjstewart commented Jun 19, 2017

Uh oh!

adamjstewart commented Jun 19, 2017

Uh oh!

adamjstewart commented Jun 19, 2017

Uh oh!

mjwoods commented Jun 20, 2017

Uh oh!

mjwoods commented Jun 20, 2017

Uh oh!

adamjstewart commented Jul 20, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

mjwoods commented Jun 16, 2017

Uh oh!

adamjstewart left a comment

Choose a reason for hiding this comment

Uh oh!

adamjstewart Jun 16, 2017

Choose a reason for hiding this comment

Uh oh!

mjwoods Jun 19, 2017

Choose a reason for hiding this comment

Uh oh!

adamjstewart Jun 19, 2017

Choose a reason for hiding this comment

Uh oh!

adamjstewart Jun 22, 2017

Choose a reason for hiding this comment

Uh oh!

mjwoods commented Jun 19, 2017

Uh oh!

adamjstewart commented Jun 19, 2017

Uh oh!

adamjstewart commented Jun 19, 2017

Uh oh!

adamjstewart commented Jun 19, 2017

Uh oh!

mjwoods commented Jun 20, 2017

Uh oh!

mjwoods commented Jun 20, 2017

Uh oh!

adamjstewart commented Jul 20, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants