[RtM] spack setup, Developer Support for CMake Projects#543
[RtM] spack setup, Developer Support for CMake Projects#543tgamblin merged 22 commits intospack:developfrom
Conversation
not change the behavior for existing packages that subclass from
spack.Package.
If a package subclasses spack.StagedPackage instead of spack.Package,
the install() phase (run inside a forked process) is now separated
into sub-stages:
a) spconfig: Generate a script spconfig.py that will configure the
package (eg by running CMake or ./configure) This is for use if
the user wishes to build externally from Spack. Therefore, the
Spack compiler wrappers are NOT used here. Currently, that
means that RPATH support is up to the user.
b) configure: Configure the project (eg: runs configure, CMake,
etc). This will configure it for use within Spack, using the
Spack wrapper.
c) build: eg: "make"
d) install: eg: "install"
If one chooses to use StagedPackage instead of Package, then one must
implement each of the install sub-stages as a separate method.
StagedPackage.install() then calls each of the sub-stages as
appropriate.
StagedPackage can be configured to only run certain sub-stages. This
is done by setting the optional kwarg install_phases when calling
do_install(). Setting install_phases() ONLY has an effect on
StagedPackage, not on any existing packages. By default,
install_phases is set to make StagedPackage run the same stages that
are normally run for any package: configure, build, install (and NOT
spconfig).
The ability for Spack to run stages selectively for StagedPackage
instances will enable new functionality. For example, explicit
CMake/Autotools helpers that allow Spack to help configure a user's
project without fetching, building or installing it.
-------------
One implementation of StagedPackage is provided, CMakePackage. This
has the following advantage for CMake-based projects over using the
standard Package class:
a) By separating out the phases, it enables future new functionality
for packages that use it.
b) It provides an implementation of intall_spconfig(), which will
help users configure their CMake-based projects.
CMakePackage expects users to implement configure_args() and
configure_env(). These methods provide the package-specific arguments
and environment needed to properly configure the package. They are
placed in separated functions because they are used in both the
spconfig and configure stages.
TODO:
1. Generate spconfig.py scripts that are more readable. This allows
the users to understand how their project is configured.
2. Provide a practical way for the user to ACCESS the spconfig stage
without building the project through Spack.
3. The CMAKE_TRANSITIVE_INCLUDE_PATH stuff needs to be reworked; it
should be considered provisional for now.
4. User of Autotools might wish to make a similar ConfigurePackage
subclass of StagedPackage.
---------------
One package using CMakePackage is introduced. See ibmisc/package.py.
(2) Neatened up the spconfig.py auto-generated file.
|
OK, I just committed "part II" of the design: the "spack spconfig" command. This completes the basic stuff I need to get my work done. Going forward, I'm hoping for a code review --- of the design and implementation details. I will add docs to this PR in the near future. This PR is useful for my work, and should have no effect on existing packages. |
…StagedPackage # Conflicts: # lib/spack/spack/package.py
|
This PR is now ready for review. |
|
TODO: It should be possible to run |
|
citibeth
changed the title
|
@tgamblin This PR broke when I merged from develop. For some reason, the module variable make was no longer defined inside my CMakePackage class. I just added a patch that makes it work; but I suspect the core problem is somewhere else that I don't understand. |
…is important to avoid breaking things that require module loads to work; for example, non-activate Python packages.
|
OK... this PR should be ready for a merge. Recent changes:
(There has been discussion that projects SHOULDN'T need this variable at all. That may be true. But my projects so far do need it; I tried without. No one forces projects to use this variable, so it shouldn't hurt anyone). I will try out this last set of changes for a few days and make sure they don't break anything. If they don't, I'm hoping this PR can be merged within a week. I have a growing list of additional PRs, mostly new Spack packages, that require |
citibeth
changed the title
|
@citibeth: can you please merge the latest develop into this? Merging it as-is would remove parts of the build logic in I like most everything in here except the |
|
So to be clear: Imerge into origin/develop, resolve On Thu, Jun 30, 2016 at 5:22 AM, Todd Gamblin [email protected]
|
|
Or do I merge origin/develop into ? On Thu, Jun 30, 2016 at 7:24 AM, Elizabeth A. Fischer <
|
|
I think the latter. I usually do |
…StagedPackage # Conflicts: # lib/spack/docs/packaging_guide.rst # lib/spack/spack/package.py Mostly minor/formatting issues in lib/spack/spack/package.py (the heavyweight merge was already done recently). Only one serious issue: it looks like the feature branch had accidentally deleted the line `spack.hooks.post_install(self)`. This got added back in from develop.
|
I recommend this is merged, as long as it passes all (non-Flake8) tests -- i.e. as long as it doesn't break old features. I would like to write unit tests for the new features, but need help on that (or pointers on how to get started). I will test the merged version; and if all things are a go, will add more to the user manual showing how to use this in a typical use case. Then I will announce it more broadly to the user base. |
| with open(os.path.join(prefix, 'dummy'), 'w') as fout: | ||
| pass | ||
|
|
||
| # stackoverflow.com/questions/12791997/how-do-you-do-a-simple-chmod-x-from-within-python |
There was a problem hiding this comment.
See llnl.util.filesystem.set_executable
|
This looks good to me. Thanks! I think there are obvious directions to extend this to further generify the stages and maybe to support autotools. But it looks pretty good as-is. One thing that can be fixed by a future PR would be the name of the |
| def install_configure(self): | ||
| cmake = which('cmake') | ||
| with working_dir(self.build_directory, create=True): | ||
| os.environ.update(self.configure_env()) |
There was a problem hiding this comment.
Probably no good reason. I used standard python routines because that is
what I knew.
On Jul 8, 2016 6:55 AM, "Massimiliano Culpo" [email protected]
wrote:
In lib/spack/spack/package.py:
fout.write('"""))\n')
fout.write("env['CMAKE_TRANSITIVE_INCLUDE_PATH'] = env['SPACK_TRANSITIVE_INCLUDE_PATH'] # Deprecated\n")fout.write('\ncmd = cmdlist("""\n')fout.write('%s\n' % cmd[0])for arg in cmd[1:]:fout.write(' %s\n' % arg)fout.write('""") + sys.argv[1:]\n')fout.write('\nproc = subprocess.Popen(cmd, env=env)\nproc.wait()\n')make_executable(setup_fname)
- def install_configure(self):
cmake = which('cmake')with working_dir(self.build_directory, create=True):os.environ.update(self.configure_env())@citibeth https://github.com/citibeth @tgamblin
https://github.com/tgamblin Is there any reason not to use
Package.setup_environment and set spack_env there in subclasses of
CMakePackage ?—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/LLNL/spack/pull/543/files/6327877a6f9d2c30e4d644ec145a6f3f323ead81#r70057615,
or mute the thread
https://github.com/notifications/unsubscribe/AB1cd-hzRHpRBXTqGNT2VUBF7mvo6Jcwks5qTiyXgaJpZM4HvOFN
.
…o libirc.so) (spack#543) * Add variant shared-intel for oneapi compilers to met to avoid linking to libirc.so * Add variant shared-intel for oneapi compilers to parallel-netcdf to avoid linking to libirc.so (don't do this by default, and remove legacy Intel) * Fix style errors in var/spack/repos/builtin/packages/met/package.py and var/spack/repos/builtin/packages/parallel-netcdf/package.py * Bug fix in var/spack/repos/builtin/packages/met/package.py, remove invalid version from variant definition
4 participants
@mathstuf @tgamblin:
Here's a first shot at separating the install() phase into sub-phases.
This is core functionality that will support the goals described in PR #474, but in a cleaner way than originally envisioned.
I'm hoping we can iterate on this...
A new subclass StagedPackage(Package) is introduced. This PR should
not change the behavior for existing packages that subclass from
spack.Package.
If a package subclasses spack.StagedPackage instead of spack.Package,
the install() phase (run inside a forked process) is now separated
into sub-stages:
If one chooses to use StagedPackage instead of Package, then one must
implement each of the install sub-stages as a separate method.
StagedPackage.install() then calls each of the sub-stages as
appropriate.
StagedPackage can be configured to only run certain sub-stages. This
is done by setting the optional kwarg install_phases when calling
do_install(). Setting install_phases() ONLY has an effect on
StagedPackage, not on any existing packages. By default,
install_phases is set to make StagedPackage run the same stages that
are normally run for any package: configure, build, install (and NOT
spconfig).
The ability for Spack to run stages selectively for StagedPackage
instances will enable new functionality. For example, explicit
CMake/Autotools helpers that allow Spack to help configure a user's
project without fetching, building or installing it.
One implementation of StagedPackage is provided, CMakePackage. This
has the following advantage for CMake-based projects over using the
standard Package class:
a) By separating out the phases, it enables future new functionality
for packages that use it.
b) It provides an implementation of intall_spconfig(), which will
help users configure their CMake-based projects.
CMakePackage expects users to implement configure_args() and
configure_env(). These methods provide the package-specific arguments
and environment needed to properly configure the package. They are
placed in separated functions because they are used in both the
spconfig and configure stages.
TODO:
the users to understand how their project is configured.
without building the project through Spack.
should be considered provisional for now.
subclass of StagedPackage.
One package using CMakePackage is introduced. See ibmisc/package.py.