Skip to content

Move compiler detection logic to packages#3

Closed
alalazo wants to merge 211 commits intopackages/gcc_detectionfrom
features/move_compiler_detection_logic
Closed

Move compiler detection logic to packages#3
alalazo wants to merge 211 commits intopackages/gcc_detectionfrom
features/move_compiler_detection_logic

Conversation

@alalazo
Copy link
Copy Markdown
Owner

@alalazo alalazo commented Jun 9, 2020

Modifications:

  • Compiler detection is moved to packages and uses external detection logic
  • Added custom detection logic for Cray Programming Environments (requires module inspection)
  • Added packages for each compiler known
  • XL compilers split into xlc and xlf
  • CCE compilers split into cce and cce-classic
  • Added tests for compiler detection
  • Support for BG/Q
  • Updated documentation where needed

@alalazo alalazo force-pushed the packages/gcc_detection branch from 89025d8 to 42307e6 Compare June 10, 2020 08:55
@alalazo alalazo force-pushed the features/move_compiler_detection_logic branch from 5951760 to 83ee688 Compare June 10, 2020 10:13
@alalazo alalazo force-pushed the packages/gcc_detection branch from 3e8ca9e to b47e28d Compare June 10, 2020 12:59
@alalazo alalazo force-pushed the features/move_compiler_detection_logic branch from 83ee688 to dedece3 Compare June 10, 2020 13:19
alalazo pushed a commit that referenced this pull request Jun 16, 2020
Whenever attempting to use any ncurses functionality within cscope, a
page fault would result within the ncurses library.

    Program received signal SIGSEGV, Segmentation fault.
    0x00007ffff7fad3cf in termattrs_sp () from .../lib/libncursesw.so.6
    (gdb) bt
    #0  0x00007ffff7fad3cf in termattrs_sp () from .../lib/libncursesw.so.6
    #1  0x00007ffff7faa794 in _nc_setupscreen_sp () from .../lib/libncursesw.so.6
    #2  0x00007ffff7fa614c in newterm_sp () from .../lib/libncursesw.so.6
    #3  0x00007ffff7fa65b9 in newterm () from .../lib/libncursesw.so.6
    spack#4  0x00007ffff7fa2970 in initscr () from .../lib/libncursesw.so.6
    spack#5  0x0000000000403dc2 in main (argc=<optimized out>, argv=0x7fffffffcea8) at main.c:574

This is due to a conflict between libtinfo.so and libtinfow.so. Both are
linked into cscope:

    $ ldd $(which cscope)
    /bin/bash: .../lib/libtinfo.so.6: no version information available (required by /bin/bash)
        linux-vdso.so.1 (0x00007fff5dbcb000)
        libncursesw.so.6 => .../lib/libncursesw.so.6 (0x00007f435cc69000)
        libtinfo.so.6 => .../lib/libtinfo.so.6 (0x00007f435cc2c000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f435ca29000)
        libtinfow.so.6 => .../lib/libtinfow.so.6 (0x00007f435c9e8000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f435cca7000)

Specifically linking libtinfow.so instead of libtinfo.so resolves the
issue.

All instances of '...' above represent the path to the installed ncurses
for Spack.
@alalazo alalazo force-pushed the packages/gcc_detection branch from b47e28d to 1560d66 Compare June 17, 2020 11:59
@alalazo alalazo force-pushed the features/move_compiler_detection_logic branch 2 times, most recently from 58e7093 to 513d705 Compare June 19, 2020 13:34
@alalazo alalazo force-pushed the packages/gcc_detection branch 2 times, most recently from 3fdcf8c to 85fefe4 Compare June 26, 2020 15:25
@alalazo alalazo force-pushed the features/move_compiler_detection_logic branch 2 times, most recently from 2ae68fc to b0dc452 Compare June 30, 2020 12:17
@alalazo alalazo force-pushed the packages/gcc_detection branch from 85fefe4 to f291890 Compare July 1, 2020 15:10
@alalazo alalazo force-pushed the features/move_compiler_detection_logic branch 2 times, most recently from 6db3ecc to ccc046c Compare July 2, 2020 09:49
@alalazo alalazo force-pushed the packages/gcc_detection branch 2 times, most recently from b8cdcd2 to 4492fdd Compare July 8, 2020 09:58
@alalazo alalazo force-pushed the features/move_compiler_detection_logic branch 7 times, most recently from 4546831 to 39fe3f7 Compare July 14, 2020 16:32
@alalazo alalazo force-pushed the packages/gcc_detection branch 2 times, most recently from 0ade6e8 to d97b0ea Compare July 15, 2020 10:29
@alalazo alalazo force-pushed the packages/gcc_detection branch from a824d4f to 14d066f Compare July 23, 2020 12:49
@alalazo alalazo force-pushed the features/move_compiler_detection_logic branch from 39fe3f7 to f9697a8 Compare July 27, 2020 15:40
@alalazo alalazo force-pushed the packages/gcc_detection branch from 956c191 to dbce632 Compare July 28, 2020 07:06
@alalazo alalazo force-pushed the features/move_compiler_detection_logic branch from f9697a8 to c1eb804 Compare July 28, 2020 08:13
adamjstewart and others added 25 commits August 11, 2020 10:23
* Don't detect Apple's clang as [email protected]
* Avoid inspecting links except for Cray platforms
* Always return string paths from compiler properties
Moved data for compiler version detection in YAML file

The test is unique and parametrized on different compilers.
Each package might define a YAML file in the same
directory where the package.py resides. The file
should be called "detection_test.yaml" and contain
specifications to setup different detection scenarios.

Currently only specs are checked, but we can extend
the test to do a finer check on extra attributes.
The code in spack.cmd.external related to package detection
has been moved to the spack.detection package. Added
docstrings to functions.
The ExternalPackageEntry class was used to carry
information on detected specs. Now this information
is attached to the spec directly, so only specs
are manipulated during the detection process.
@alalazo alalazo force-pushed the features/move_compiler_detection_logic branch from 7c639e3 to 400e143 Compare August 11, 2020 13:53
@alalazo alalazo closed this Aug 12, 2020
alalazo pushed a commit that referenced this pull request Aug 18, 2020
* Add Collier and SysCalc recipes

* Remove extra syscalc version

* Build collier with -j1 for @:1.2.4

* Add recipe for gosam-contrib

* Update gosam-contrib recipe with 'provides'

* Madgraph recipe, first version

* Finalize madgraph recipe + flake8

* Make py2 version of madgraph default; fix hash for syscalc; fix patch

* Handle virtual packages (#3)

* Update package.py

* Update packages.yaml

* Remove virtual packages - pt. 1

* Remove virtual packages - pt. 2

* Changes from review - pt. 1

* Changes from code review - pt. 2

* Update var/spack/repos/builtin/packages/collier/package.py

Co-authored-by: Adam J. Stewart <[email protected]>

* Update var/spack/repos/builtin/packages/madgraph5amc/package.py

Co-authored-by: Adam J. Stewart <[email protected]>

* Add hash for version 2.7.2 (available in our private mirror)

* Fixes for 2.7.3 family

* Patches for 2.7.3{.py3,}{.atlas,}

* Fix hash of syscalc

* Hack to fix concretization (2.7.3 matches 2.7.3.py3)

* Add conflict statement (reported to devs)

* Apply suggestions from code review

Co-authored-by: Adam J. Stewart <[email protected]>

* Update package.py

* Delete madgraph5amc-2.7.2.atlas.patch

* Delete madgraph5amc-2.7.2.patch

* Update package.py

* Apply suggestions from code review

Co-authored-by: Adam J. Stewart <[email protected]>

Co-authored-by: iarspider <[email protected]>
Co-authored-by: Adam J. Stewart <[email protected]>
alalazo pushed a commit that referenced this pull request Aug 3, 2021
* Set LIBVER and LIBMAJOR

* Typo fix

* Fix 2

* Fix #3
alalazo pushed a commit that referenced this pull request Apr 24, 2023
1. support version 3.1.3, which now depends on sundials@6

2. support version 3.1.2:, which broke the two patch files and
   therefore the two patch files have been replaced by more flexible
   filter_file() commands inside a patch() function.

3. rename the variant for python extension from using the package name
   "+pyuqtk" to the more standard "+python"

4. add maintainers @omsai and the upstream developer @bjdebus who
   offered to help with the spack packaging.

5. swig should only be a build-time dependency.  swig is only
   necessary until @:3.1.0

6. confirmed python dependencies are correct by inspecting imports,
   subset python dependencies type to build, run, and confirmed all
   31 build-time tests pass including the 9 python tests:

```console
$ spack env create uqtk-dev
$ spack add [email protected]
$ spack install --test root && cat $(spack location -i uqtk)/.spack/install-time-test-log.txt
==> Testing package uqtk-3.1.3-nok6fut
==> [2023-04-19-14:56:25.005361] Running build-time tests
==> [2023-04-19-14:56:25.005536] RUN-TESTS: build-time tests [check]
==> [2023-04-19-14:56:25.009543] '/home/omsai/src/spack/opt/spack/linux-pureos10-skylake/gcc-10.2.1/gmake-4.4.1-b6g4apmfvxz3bn4eabh37dehcrg65fj7/bin/make' '-j4' '-n' 'test'
==> [2023-04-19-14:56:25.014903] '/home/omsai/src/spack/opt/spack/linux-pureos10-skylake/gcc-10.2.1/gmake-4.4.1-b6g4apmfvxz3bn4eabh37dehcrg65fj7/bin/make' '-j4' 'test'
Running tests...
/home/omsai/src/spack/opt/spack/linux-pureos10-skylake/gcc-10.2.1/cmake-3.26.3-zjmsfz23j5l4ytniz26uzvxonlu5qebr/bin/ctest --force-new-ctest-process
Test project /tmp/omsai/spack-stage/spack-stage-uqtk-3.1.3-nok6fut47h42cnaau7wkoohgqy5f2qqa/spack-build-nok6fut
      Start  1: ArrayReadAndWrite
      Start  2: ArrayDelColumn
      Start  3: Array1DMiscTest
      Start  4: Array2DMiscTest
 1/31 Test  #1: ArrayReadAndWrite ................   Passed    0.01 sec
      Start  5: ArraySortTest
 2/31 Test  #2: ArrayDelColumn ...................   Passed    0.01 sec
      Start  6: MultiIndexTest
 3/31 Test  #3: Array1DMiscTest ..................   Passed    0.01 sec
      Start  7: CorrTest
 4/31 Test  spack#4: Array2DMiscTest ..................   Passed    0.01 sec
      Start  8: QuadLUTest
 5/31 Test  spack#5: ArraySortTest ....................   Passed    0.02 sec
      Start  9: MCMC2dTest
 6/31 Test  spack#6: MultiIndexTest ...................   Passed    0.01 sec
      Start 10: MCMCRandomTest
 7/31 Test  spack#8: QuadLUTest .......................   Passed    0.02 sec
      Start 11: MCMCNestedTest
 8/31 Test spack#10: MCMCRandomTest ...................   Passed    0.02 sec
      Start 12: Deriv1dTest
 9/31 Test spack#12: Deriv1dTest ......................   Passed    0.01 sec
      Start 13: SecondDeriv1dTest
10/31 Test spack#13: SecondDeriv1dTest ................   Passed    0.01 sec
      Start 14: GradHessianTest
11/31 Test spack#11: MCMCNestedTest ...................   Passed    0.03 sec
      Start 15: GradientPCETest
12/31 Test spack#14: GradHessianTest ..................   Passed    0.01 sec
      Start 16: PCE1dTest
13/31 Test spack#15: GradientPCETest ..................   Passed    0.01 sec
      Start 17: PCEImplTest
14/31 Test spack#16: PCE1dTest ........................   Passed    0.01 sec
      Start 18: PCELogTest
15/31 Test spack#18: PCELogTest .......................   Passed    0.01 sec
      Start 19: Hessian2dTest
16/31 Test spack#19: Hessian2dTest ....................   Passed    0.01 sec
      Start 20: BCS1dTest
17/31 Test spack#20: BCS1dTest ........................   Passed    0.01 sec
      Start 21: BCS2dTest
18/31 Test spack#21: BCS2dTest ........................   Passed    0.01 sec
      Start 22: LowRankRegrTest
19/31 Test spack#22: LowRankRegrTest ..................   Passed    0.01 sec
      Start 23: PyModTest
20/31 Test spack#17: PCEImplTest ......................   Passed    0.07 sec
      Start 24: PyArrayTest
21/31 Test spack#23: PyModTest ........................   Passed    0.08 sec
      Start 25: PyArrayTest2
22/31 Test spack#25: PyArrayTest2 .....................   Passed    0.30 sec
      Start 26: PyQuadTest
23/31 Test spack#24: PyArrayTest ......................   Passed    1.44 sec
      Start 27: PyBCSTest1D
24/31 Test spack#26: PyQuadTest .......................   Passed    1.68 sec
      Start 28: PyBCSTest2D
25/31 Test spack#27: PyBCSTest1D ......................   Passed    1.66 sec
      Start 29: PyBADPTest
26/31 Test  spack#7: CorrTest .........................   Passed    3.43 sec
      Start 30: PyRegressionTest
27/31 Test spack#28: PyBCSTest2D ......................   Passed    1.50 sec
      Start 31: PyGalerkinTest
28/31 Test  spack#9: MCMC2dTest .......................   Passed    3.90 sec
29/31 Test spack#29: PyBADPTest .......................   Passed    1.66 sec
30/31 Test spack#30: PyRegressionTest .................   Passed    1.72 sec
31/31 Test spack#31: PyGalerkinTest ...................   Passed    1.63 sec

100% tests passed, 0 tests failed out of 31

Total Test time (real) =   5.35 sec
==> [2023-04-19-14:56:30.382797] '/home/omsai/src/spack/opt/spack/linux-pureos10-skylake/gcc-10.2.1/gmake-4.4.1-b6g4apmfvxz3bn4eabh37dehcrg65fj7/bin/make' '-j4' '-n' 'check'
==> [2023-04-19-14:56:30.385983] Target 'check' not found in Makefile
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.