Skip to content

Intel software packages and license enhancements#946

Merged
tgamblin merged 5 commits intodevelopfrom
features/intel2
Jun 17, 2016
Merged

Intel software packages and license enhancements#946
tgamblin merged 5 commits intodevelopfrom
features/intel2

Conversation

@lee218llnl
Copy link
Copy Markdown
Contributor

@lee218llnl lee218llnl commented May 13, 2016

This PR adds various Intel packages (compilers, parallel studio, mkl, ipp, and daal. It also enhances Spack's license support. This is to clean up my rebase attempt from #796, which addresses #761.

This was referenced May 13, 2016
Closed
Closed
adamjstewart
adamjstewart reviewed May 13, 2016
View reviewed changes
lib/spack/spack/hooks/licensing.py Outdated
target = pkg.global_license_file
for filename in pkg.license_files:
link_name = join_path(pkg.prefix, filename)
license_dir = os.path.split(link_name)[0]
Copy link
Copy Markdown
Member

@adamjstewart adamjstewart May 13, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

license_dir = os.path.dirname(link_name)

@adamjstewart
Copy link
Copy Markdown
Member

adamjstewart commented May 13, 2016

I tried installing Intel Parallel Studio - Professional Edition on our cluster with your package and I'm getting the following error message:

==> './install.sh' '--silent' 'silent.cfg'                                      
error: cannot open Packages index using db3 - No such file or directory (2)     
error: cannot open Packages database in /var/lib/rpm                            
error: cannot open Packages database in /var/lib/rpm                            
/scratch/install.twHG35/sysreq/plugin.sh: line 60: [: too many arguments        
error: cannot open Packages index using db3 - No such file or directory (2)     
error: cannot open Packages database in /var/lib/rpm                            
error: cannot open Packages database in /var/lib/rpm                            
error: cannot open Packages index using db3 - No such file or directory (2)     
error: cannot open Packages database in /var/lib/rpm                            
error: cannot open Packages database in /var/lib/rpm                            
Missing optional prerequisites                                                  
-- Unsupported OS                                                               
-- Cannot detect gtk library                                                    
--------------------------------------------------------------------------------
There are some missing optional prerequisites. If you still want to continue,   
please correct the CONTINUE_WITH_OPTIONAL_ERROR variable in silent configuration
file.                                                                           
--------------------------------------------------------------------------------

Any idea what's up?

@lee218llnl
Copy link
Copy Markdown
Contributor Author

lee218llnl commented May 13, 2016

Any idea what's up?
@adamjstewart in intel/package.py, add "CONTINUE_WITH_OPTIONAL_ERROR=yes" to the silent config file that is generated and see if that does the trick. I don't know if it will get past the "Unsupported OS" error and all the database errors as I have not seen those messages before and don't know if they're fatal to the install.

@adamjstewart
Copy link
Copy Markdown
Member

adamjstewart commented May 13, 2016

Adding that line did allow the installation to complete successfully. I'll play around with it more to see whether or not it compiles things. I'll get back to you next week.

@adamjstewart
Copy link
Copy Markdown
Member

adamjstewart commented May 16, 2016

@lee218llnl: I installed the intel-parallel-studio package and tried building OpenMPI with it, but found the {icc,icpc,ifort}.cfg files inadequate. I was unable to install OpenMPI unless my config files contained the following paths:

-Xlinker -rpath -Xlinker /blues/gpfs/home/software/spack/opt/spack/linux-x86_64/gcc-5.3.0/intel-parallelstudio-professional.2016.3-sz7z7dmdxcdkuhakijzapfj3b5ysuhte/compilers_and_libraries_2016.3.210/linux/compiler/lib/intel64 -Xlinker -rpath -Xlinker /blues/gpfs/home/software/spack/opt/spack/linux-x86_64/gcc-5.3.0/intel-parallelstudio-professional.2016.3-sz7z7dmdxcdkuhakijzapfj3b5ysuhte/compilers_and_libraries_2016.3.210/linux/compiler/lib/intel64_lin -Xlinker -rpath -Xlinker /blues/gpfs/home/software/spack/opt/spack/linux-x86_64/gcc-5.3.0/intel-parallelstudio-professional.2016.3-sz7z7dmdxcdkuhakijzapfj3b5ysuhte/compilers_and_libraries_2016.3.210/linux/mpi/intel64/lib -Xlinker -rpath -Xlinker /blues/gpfs/home/software/spack/opt/spack/linux-x86_64/gcc-5.3.0/intel-parallelstudio-professional.2016.3-sz7z7dmdxcdkuhakijzapfj3b5ysuhte/compilers_and_libraries_2016.3.210/linux/mpi/mic/lib -Xlinker -rpath -Xlinker /blues/gpfs/home/software/spack/opt/spack/linux-x86_64/gcc-5.3.0/intel-parallelstudio-professional.2016.3-sz7z7dmdxcdkuhakijzapfj3b5ysuhte/compilers_and_libraries_2016.3.210/linux/ipp/lib/intel64 -Xlinker -rpath -Xlinker /blues/gpfs/home/software/spack/opt/spack/linux-x86_64/gcc-5.3.0/intel-parallelstudio-professional.2016.3-sz7z7dmdxcdkuhakijzapfj3b5ysuhte/compilers_and_libraries_2016.3.210/linux/compiler/lib/intel64 -Xlinker -rpath -Xlinker /blues/gpfs/home/software/spack/opt/spack/linux-x86_64/gcc-5.3.0/intel-parallelstudio-professional.2016.3-sz7z7dmdxcdkuhakijzapfj3b5ysuhte/compilers_and_libraries_2016.3.210/linux/mkl/lib/intel64 -Xlinker -rpath -Xlinker /blues/gpfs/home/software/spack/opt/spack/linux-x86_64/gcc-5.3.0/intel-parallelstudio-professional.2016.3-sz7z7dmdxcdkuhakijzapfj3b5ysuhte/compilers_and_libraries_2016.3.210/linux/tbb/lib/intel64/gcc4.4 -Xlinker -rpath -Xlinker /blues/gpfs/home/software/spack/opt/spack/linux-x86_64/gcc-5.3.0/intel-parallelstudio-professional.2016.3-sz7z7dmdxcdkuhakijzapfj3b5ysuhte/debugger_2016/libipt/intel64/lib -Xlinker -rpath -Xlinker /blues/gpfs/home/software/spack/opt/spack/linux-x86_64/gcc-5.3.0/intel-parallelstudio-professional.2016.3-sz7z7dmdxcdkuhakijzapfj3b5ysuhte/compilers_and_libraries_2016.3.210/linux/daal/lib/intel64_lin -Xlinker -rpath -Xlinker /blues/gpfs/home/software/spack/opt/spack/linux-x86_64/gcc-5.3.0/intel-parallelstudio-professional.2016.3-sz7z7dmdxcdkuhakijzapfj3b5ysuhte/compilers_and_libraries_2016.3.210/linux/daal/../compiler/lib/intel64_lin -L/blues/gpfs/home/software/spack/opt/spack/linux-x86_64/gcc-5.3.0/intel-parallelstudio-professional.2016.3-sz7z7dmdxcdkuhakijzapfj3b5ysuhte/compilers_and_libraries_2016.3.210/linux/ipp/lib/intel64 -L/blues/gpfs/home/software/spack/opt/spack/linux-x86_64/gcc-5.3.0/intel-parallelstudio-professional.2016.3-sz7z7dmdxcdkuhakijzapfj3b5ysuhte/compilers_and_libraries_2016.3.210/linux/compiler/lib/intel64 -L/blues/gpfs/home/software/spack/opt/spack/linux-x86_64/gcc-5.3.0/intel-parallelstudio-professional.2016.3-sz7z7dmdxcdkuhakijzapfj3b5ysuhte/compilers_and_libraries_2016.3.210/linux/mkl/lib/intel64 -L/blues/gpfs/home/software/spack/opt/spack/linux-x86_64/gcc-5.3.0/intel-parallelstudio-professional.2016.3-sz7z7dmdxcdkuhakijzapfj3b5ysuhte/compilers_and_libraries_2016.3.210/linux/tbb/lib/intel64/gcc4.4 -L/blues/gpfs/home/software/spack/opt/spack/linux-x86_64/gcc-5.3.0/intel-parallelstudio-professional.2016.3-sz7z7dmdxcdkuhakijzapfj3b5ysuhte/compilers_and_libraries_2016.3.210/linux/daal/lib/intel64_lin -L/blues/gpfs/home/software/spack/opt/spack/linux-x86_64/gcc-5.3.0/intel-parallelstudio-professional.2016.3-sz7z7dmdxcdkuhakijzapfj3b5ysuhte/compilers_and_libraries_2016.3.210/linux/daal/../compiler/lib/intel64_lin

These paths come from sourcing compilervars.sh, although hopefully there is an easier way for your install package to recognize to find these paths. The -Xlinker paths were from LD_LIBRARY_PATH and the -L paths were from LIBRARY_PATH.

@adamjstewart
Copy link
Copy Markdown
Member

adamjstewart commented May 17, 2016

Also, I noticed an intel directory in my home directory. Among other things, it contains intel/licenses/license.lic. I'm not sure how your package works, but isn't it supposed to link to the global license directory?

@lee218llnl
Copy link
Copy Markdown
Contributor Author

lee218llnl commented May 17, 2016

@adamjstewart I'm not sure why you would need all of the other library paths in the .cfg file. Do you know which one in particular it needed (I would have just thought the lib/intel64 would be sufficient)? Note that one of the paths that you added is the Intel MPI lib path, which might not be the best thing to include.

Regarding the license, the $HOME/intel directory is something that the installer creates, not spack.

@adamjstewart
Copy link
Copy Markdown
Member

adamjstewart commented May 17, 2016

I can try narrowing it down to a smaller number of paths. Those were just the ones compilervars.sh added for me. The initial icc.cfg that your script generated wasn't enough for me to build anything. I wonder if the Intel MPI lib path is causing problems for me in #784.

@adamjstewart adamjstewart mentioned this pull request May 17, 2016
Closed
@adamjstewart
Copy link
Copy Markdown
Member

adamjstewart commented May 17, 2016

@lee218llnl: I mentioned this in #784, but the magic ticket is:

-Xlinker -rpath -Xlinker /soft/spack/opt/spack/linux-x86_64/gcc-5.3.0/intel-parallelstudio-professional.2016.3-sz7z7dmdxcdkuhakijzapfj3b5ysuhte/compilers_and_libraries_2016.3.210/linux/compiler/lib/intel64 -L/soft/spack/opt/spack/linux-x86_64/gcc-5.3.0/intel-parallelstudio-professional.2016.3-sz7z7dmdxcdkuhakijzapfj3b5ysuhte/compilers_and_libraries_2016.3.210/linux/compiler/lib/intel64

@lee218llnl
Copy link
Copy Markdown
Contributor Author

lee218llnl commented May 17, 2016

Moving discussion from #784:

@adamjstewart I'm flattered! One question, though, do you really need to add the -L option to the .cfg file? The Intel compiler should add this to the link line by default. If you remove it from your .cfg files, you can do an icc/icpc/ifort with the -v option to confirm what it passes to ld.

I do. I'm not sure why, but with only the -Xlinker -rpath -Xlinker option, I'm unable to build anything. Simple packages like libsigsegv or jpeg crash during configure and claim that the compiler cannot create C executables. If you want I can recreate this and post some config.log files.

@lee218llnl
Copy link
Copy Markdown
Contributor Author

lee218llnl commented May 17, 2016

@adamjstewart Yes, it would be helpful to see some log files and get to the bottom of this.

@adamjstewart
Copy link
Copy Markdown
Member

adamjstewart commented May 17, 2016

With -Xlinker only, libsigsegv fails during configure:
config.log
spack-build.out

With -L only, jpeg fails during make test:
config.log
spack-build.out

When using both -Xlinker and -L, both packages can be installed.

@tgamblin
Copy link
Copy Markdown
Member

tgamblin commented May 17, 2016

@adamjstewart: do all compilers like -Xlinker? It's the same as -Wl,-rpath but I am not sure it's universally supported.

@adamjstewart
Copy link
Copy Markdown
Member

adamjstewart commented May 17, 2016

shrugs

It's Intel. If Intel doesn't like it, that's not a good sign. Anyway, it seems to be doing something since I can't compile jpeg without it. Perhaps it works but can't handle multiple Xlinker args like I was previously using?

@lee218llnl
Copy link
Copy Markdown
Contributor Author

lee218llnl commented May 17, 2016

@adamjstewart It looks like you are picking up some older compiler bits in the ld line:

-L/soft/intel/13.1.3/lib/intel64/

@adamjstewart
Copy link
Copy Markdown
Member

adamjstewart commented May 18, 2016

Huh. It must be picking that up from my LD_LIBRARY_PATH during installation. Hopefully not during runtime, or else Spack is broken. Anyway, adding that line to the .cfg files solves it, so it's probably safe to add.

@lee218llnl
Copy link
Copy Markdown
Contributor Author

lee218llnl commented Jun 10, 2016

This hasn't been looked at in a while. Can someone give a final review and merge if it looks good?

@luigi-calori luigi-calori mentioned this pull request Jun 15, 2016
Closed
@tgamblin tgamblin merged commit 0bbbfc2 into develop Jun 17, 2016
@tgamblin
Copy link
Copy Markdown
Member

tgamblin commented Jun 17, 2016

@lee218llnl: Looks good to me! Thanks!

@tgamblin tgamblin deleted the features/intel2 branch August 29, 2016 16:58
@adamjstewart adamjstewart mentioned this pull request May 23, 2017
Merged
12 tasks
olupton pushed a commit to olupton/spack that referenced this pull request Feb 7, 2022
@ppodhajski
adapt brion test to a new python module name (spack#946)
0944b6d
olupton pushed a commit to olupton/spack that referenced this pull request Feb 7, 2022
@AurelienJaquier @matz-e
@anilbey @sergiorg-hpc @ppodhajski @ferdonline @asanin-epfl @NadirRoGue @pramodk
py-currentscape: deploy and generate module (spack#936)
967c78d 
* Update bbp-packages.yaml

Adding my currentscape module to bbp-packages.

* Update modules.yaml

Adding my currentscape package to the whitelist.

* Create package.py

Added my currentscape package.

* Update var/spack/repos/builtin/packages/py-currentscape/package.py

Co-authored-by: Matthias Wolf <[email protected]>

* Update var/spack/repos/builtin/packages/py-currentscape/package.py

Co-authored-by: Matthias Wolf <[email protected]>

* Update var/spack/repos/builtin/packages/py-currentscape/package.py

Co-authored-by: Matthias Wolf <[email protected]>

* Update package.py

Modified the version. Is it better now?

* update bglibpy to 4.4.6 (spack#937)

* Update package.py

Changing tag to match the one from source on gerrit.

* Bglibpy 4.4.10 (spack#938)

* update bglibpy to 4.4.6

* updated to bglpy 4.4.10

v 4.4.10 does not have the pandas, pyrsistent or the bluepy-configfile dependencies

* Updates Libsonata package to include readers improvements (spack#940)

* update glm to 0.9.9.3 (spack#943)

* Bump neurodamus for most recent patches (spack#942)

Bump neurodamus-py 2.0.0 to 2.0.2:
 - Fixing replay to work with multiple populations
 - Ensure data dir when skipping model build
 - Fix skipping synapse creation when weight is 0 (BBPBGLIB-673)
 - Fix deadlock when an exception is thrown from NEURON (BBPBGLIB-678)
 - Logging colors only for tty

Bump neurodamus-core 3.0.0 to 3.0.1:
 - Avoid getting nilSecRef from objects (HPCTM-1381)

* Update py-sonata-network-reduction dependencies: [email protected], [email protected] (spack#930)

* Update bbp-packages.yaml

Updated currentscape version.

* Update package.py

Updated version & tag.

* Brion and Brayns are dependent on GLM (spack#944)

* Steps updates (spack#941)

* gmsh: add version 4.6.0
* omega-h: new version 9.32.5.dev3
* steps: new test requirements

* libsonata-report: Improves initialization performance (spack#945)

* adapt brion test to a new python module name (spack#946)

* Adding nvidia-hpc-sdk based on upstream PR (spack#935)

* New compiler: nvhpc (NVIDIA HPC SDK) (spack#19294)
* Add nvhpc compiler definition: "spack compiler add" will now look
  for instances of the NVIDIA HPC SDK compiler executables
  (nvc, nvc++, nvfortran) in supplied paths
* Add the nvhpc package which installs the nvhpc compiler
* Add testing for nvhpc detection and C++-standard/pic flags

Based on spack#19294

* Add CUDA@11 required for latest NVIDIA-HPC-SDK
* Fix legacy apis : setup_environment to setup_run_environment

* NEURON and CoreNEURON should use legacy units for BBP/HBP deployment (spack#947)

* Update bbp-packages.yaml

Adding my currentscape module to bbp-packages.

* Update modules.yaml

Adding my currentscape package to the whitelist.

* Create package.py

Added my currentscape package.

* Update var/spack/repos/builtin/packages/py-currentscape/package.py

Co-authored-by: Matthias Wolf <[email protected]>

* Update var/spack/repos/builtin/packages/py-currentscape/package.py

Co-authored-by: Matthias Wolf <[email protected]>

* Update var/spack/repos/builtin/packages/py-currentscape/package.py

Co-authored-by: Matthias Wolf <[email protected]>

* Update package.py

Modified the version. Is it better now?

* Update package.py

Changing tag to match the one from source on gerrit.

* Update bbp-packages.yaml

Updated currentscape version.

* Update package.py

Updated version & tag.

Co-authored-by: Matthias Wolf <[email protected]>
Co-authored-by: anilbey <[email protected]>
Co-authored-by: Sergio <[email protected]>
Co-authored-by: ppodhajski <[email protected]>
Co-authored-by: Fernando Pereira <[email protected]>
Co-authored-by: asanin-epfl <[email protected]>
Co-authored-by: Nadir Román Guerrero <[email protected]>
Co-authored-by: Tristan Carel <[email protected]>
Co-authored-by: Pramod Kumbhar <[email protected]>
Co-authored-by: Jaquier Aurélien Tristan <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@lee218llnl @adamjstewart @tgamblin