CEED v4.0 release by v-dobrev · Pull Request #22735 · spack/spack

v-dobrev · 2021-04-01T21:26:05Z

Version 4.0 of the CEED package collection.

Disabling OCCA because backend updates did not make this release and there are some known bugs so most users won't have reason to use OCCA. CEED/libCEED#688

@MPhysXDev

* MFEM package updates * mfem: flake8 * [mfem] Various fixes and tweaks. [arpack-ng] Add a patch to fix building with IBM XL Fortran. [libceed] Fix building with IBM XL C/C++. [pumi] Add C++11 flag for version 2.2.3. * [mfem] Fix the shared CUDA build. Reported by: @MPhysXDev * [mfem] Fix a TODO item * [mfem] Tweak the AmgX dependencies * [suite-sparse] Fix the version of the mpfr dependency * MFEM: add initial HIP support using the ROCmPackage. * MFEM: add 'slepc' variant. * MFEM: update the patch for v4.2 for SLEPc. * mfem: apply 'mfem-4.2-slepc.patch' just to v4.2.

[laghos] Add laghos version 3.1 based on the latest commit in the repository; this version works with mfem v4.2. [ceed] For ceed v4.0 use laghos v3.1.

var/spack/repos/builtin/packages/laghos/package.py

compiler.

[ceed] Use pumi v2.2.5 with ceed v4.0.0.

Use [email protected]+rocm with [email protected]+mfem+hip.

v-dobrev · 2021-04-03T04:10:02Z

Latest commit worked fine on Lassen with gcc v8.3.1 with

./bin/spack install ceed+cuda ^magma cuda_arch=70 ^mfem cuda_arch=sm_70

v-dobrev · 2021-04-07T21:36:48Z

The NekRS package seems to have both hip (explicitly defined but not used) and rocm (inherited from ROCmPackage) variants which is redundant. Also, depends_on('hip', when='+rocm') is unnecessary since it is inherited from ROCmPackage, see https://github.com/spack/spack/blob/develop/lib/spack/spack/build_systems/rocm.py.

Ping: @thilinarmtb.

On a similar topic: I think wee should add a hip (or rocm?) variant to occa. I also added relevant TODO comments about this in the ceed and libceed packages.

Ping: @dmed256.

v-dobrev · 2021-05-03T08:42:54Z

TODO:

In petsc, libceed, and ceed: replace the +hip variant with +rocm by inheriting from ROCmPackage.
In mfem: replace the dependency 'libceed+hip' with 'libceed+rocm'.

jedbrown · 2021-05-20T15:25:48Z

var/spack/repos/builtin/packages/ceed/package.py

+    depends_on('[email protected]~rocm', when='@4.0.0~rocm')
+    for target in ROCmPackage.amdgpu_targets:
+        depends_on('[email protected]+rocm amdgpu_target={0}'.format(target),
+                   when='@4.0.0+rocm amdgpu_target={0}'.format(target))


@becker33 Is this what you meant when you said (on Slack) that the variant needs to be forwarded to each dependent package? The spec makes sense with this change, though it's tedious going and all the BundlePackages would need this treatment.

'cuda_arch' values should not include the 'sm_' prefix. Also, propagate 'cuda_arch' and 'amdgpu_target' variants to enabled dependencies.

jedbrown · 2021-05-20T19:26:39Z

var/spack/repos/builtin/packages/mfem/package.py

            if '+cuda' in spec:
                cxxflags += [
-                    '-x=cu --expt-extended-lambda -arch=%s' % cuda_arch,
+                    '-x=cu --expt-extended-lambda -arch=sm_%s' % cuda_arch,


I think Todd describes this flag as being handled automatically by the compiler wrappers when using CudaPackage.

This is a flag for nvcc -- I don't think spack has a wrapper for that -- unless that is a recent addition.

v-dobrev · 2021-05-20T21:24:21Z

The following install worked fine on Lassen (with gcc-8.3.1 and various other customizations in compilers.yaml and packages.yaml):

./bin/spack install ceed+cuda cuda_arch=70

jedbrown

LGTM. @balay may want to review the PETSc changes.

tzanio

Thank you @v-dobrev, @jedbrown and @thilinarmtb !

tzanio · 2021-05-31T20:04:46Z

We plan to merge this on Wednesday (06/02/21). Please speak up if you have feedback or concerns.

tzanio · 2021-06-03T15:37:37Z

@jedbrown, can you resolve the conflict in petsc so we can merge?

Resolve petsc conflict on new hwloc and openmp variants.

jedbrown

This should be good to merge once tests pass.

@MPhysXDev

* petsc: add hip variant * libceed: add 0.8, disable occa by default, and let autodetect AVX Disabling OCCA because backend updates did not make this release and there are some known bugs so most users won't have reason to use OCCA. CEED/libCEED#688 * WIP: ceed: 4.0 release * MFEM package updates (spack#19748) * MFEM package updates * mfem: flake8 * [mfem] Various fixes and tweaks. [arpack-ng] Add a patch to fix building with IBM XL Fortran. [libceed] Fix building with IBM XL C/C++. [pumi] Add C++11 flag for version 2.2.3. * [mfem] Fix the shared CUDA build. Reported by: @MPhysXDev * [mfem] Fix a TODO item * [mfem] Tweak the AmgX dependencies * [suite-sparse] Fix the version of the mpfr dependency * MFEM: add initial HIP support using the ROCmPackage. * MFEM: add 'slepc' variant. * MFEM: update the patch for v4.2 for SLEPc. * mfem: apply 'mfem-4.2-slepc.patch' just to v4.2. * ceed: apply 'spack style' * [mfem] Add a patch for mfem v4.2 to work with petsc v3.15.0. [laghos] Add laghos version 3.1 based on the latest commit in the repository; this version works with mfem v4.2. [ceed] For ceed v4.0 use laghos v3.1. * [libceed] Explicitly set 'CC_VENDOR=icc' when using 'intel' compiler. * [mfem] Allow pumi >= 2.2.3 with mfem >= 4.2.0. [ceed] Use pumi v2.2.5 with ceed v4.0.0. * [ceed] Explicitly use occa v1.1.0 with ceed v4.0.0. Use [email protected]+rocm with [email protected]+mfem+hip. * [ceed] Add NekRS v21 as a dependency for ceed v4.0.0. * [ceed] Fix NekRS version: 21 --> 21.0 * [ceed] Propagate +cuda variant to petsc for ceed v4.0. * [mfem] Propagate '+rocm' variant to some other packages. * [ceed] Use +rocm variant of nekrs instead of +hip. * [ceed] Do not enable magma with [email protected]+hip. * [libceed] Fix hip build with [email protected]. * [laghos] For v3.1, use the release .tar.gz file instead of commit. * Remove cuda & hip variants as they are inherited * [ceed] Remove comments and FIXMEs about 'magma+hip'. * [ceed] [libceed] Remove TODOs about occa + hip. * libceed: use ROCmPackage and +rocm * petsc: use ROCmPackage for HIP * libceed, petsc: use CudaPackage * ceed: forward cuda_arch and amdgpu_target * [mfem] Use Spack's CudaPackage as a base class; as a result, 'cuda_arch' values should not include the 'sm_' prefix. Also, propagate 'cuda_arch' and 'amdgpu_target' variants to enabled dependencies. * petsc: variant is +rocm, package name is hip Co-authored-by: Jed Brown <[email protected]> Co-authored-by: Thilina Rathnayake <[email protected]>

@MPhysXDev

* petsc: add hip variant * libceed: add 0.8, disable occa by default, and let autodetect AVX Disabling OCCA because backend updates did not make this release and there are some known bugs so most users won't have reason to use OCCA. CEED/libCEED#688 * WIP: ceed: 4.0 release * MFEM package updates (spack#19748) * MFEM package updates * mfem: flake8 * [mfem] Various fixes and tweaks. [arpack-ng] Add a patch to fix building with IBM XL Fortran. [libceed] Fix building with IBM XL C/C++. [pumi] Add C++11 flag for version 2.2.3. * [mfem] Fix the shared CUDA build. Reported by: @MPhysXDev * [mfem] Fix a TODO item * [mfem] Tweak the AmgX dependencies * [suite-sparse] Fix the version of the mpfr dependency * MFEM: add initial HIP support using the ROCmPackage. * MFEM: add 'slepc' variant. * MFEM: update the patch for v4.2 for SLEPc. * mfem: apply 'mfem-4.2-slepc.patch' just to v4.2. * ceed: apply 'spack style' * [mfem] Add a patch for mfem v4.2 to work with petsc v3.15.0. [laghos] Add laghos version 3.1 based on the latest commit in the repository; this version works with mfem v4.2. [ceed] For ceed v4.0 use laghos v3.1. * [libceed] Explicitly set 'CC_VENDOR=icc' when using 'intel' compiler. * [mfem] Allow pumi >= 2.2.3 with mfem >= 4.2.0. [ceed] Use pumi v2.2.5 with ceed v4.0.0. * [ceed] Explicitly use occa v1.1.0 with ceed v4.0.0. Use [email protected]+rocm with [email protected]+mfem+hip. * [ceed] Add NekRS v21 as a dependency for ceed v4.0.0. * [ceed] Fix NekRS version: 21 --> 21.0 * [ceed] Propagate +cuda variant to petsc for ceed v4.0. * [mfem] Propagate '+rocm' variant to some other packages. * [ceed] Use +rocm variant of nekrs instead of +hip. * [ceed] Do not enable magma with [email protected]+hip. * [libceed] Fix hip build with [email protected]. * [laghos] For v3.1, use the release .tar.gz file instead of commit. * Remove cuda & hip variants as they are inherited * [ceed] Remove comments and FIXMEs about 'magma+hip'. * [ceed] [libceed] Remove TODOs about occa + hip. * libceed: use ROCmPackage and +rocm * petsc: use ROCmPackage for HIP * libceed, petsc: use CudaPackage * ceed: forward cuda_arch and amdgpu_target * [mfem] Use Spack's CudaPackage as a base class; as a result, 'cuda_arch' values should not include the 'sm_' prefix. Also, propagate 'cuda_arch' and 'amdgpu_target' variants to enabled dependencies. * petsc: variant is +rocm, package name is hip Co-authored-by: Jed Brown <[email protected]> Co-authored-by: Thilina Rathnayake <[email protected]>

jedbrown added 3 commits March 31, 2021 22:35

petsc: add hip variant

cf3692f

libceed: add 0.8, disable occa by default, and let autodetect AVX

62bd12f

Disabling OCCA because backend updates did not make this release and there are some known bugs so most users won't have reason to use OCCA. CEED/libCEED#688

WIP: ceed: 4.0 release

d6772aa

v-dobrev added new-version WIP labels Apr 1, 2021

v-dobrev assigned jedbrown and v-dobrev Apr 1, 2021

v-dobrev mentioned this pull request Apr 1, 2021

MFEM package updates #19748

Merged

v-dobrev and others added 2 commits April 1, 2021 14:58

ceed: apply 'spack style'

b3827ec

tldahlgren added conflicts new-variant patch update-package variants AMD cuda labels Apr 2, 2021

[mfem] Add a patch for mfem v4.2 to work with petsc v3.15.0.

3d81d23

[laghos] Add laghos version 3.1 based on the latest commit in the repository; this version works with mfem v4.2. [ceed] For ceed v4.0 use laghos v3.1.

v-dobrev commented Apr 2, 2021

View reviewed changes

var/spack/repos/builtin/packages/laghos/package.py Outdated Show resolved Hide resolved

v-dobrev mentioned this pull request Apr 2, 2021

fix build with petsc-3.15 mfem/mfem#2143

Closed

v-dobrev added 6 commits April 2, 2021 18:03

[libceed] Explicitly set 'CC_VENDOR=icc' when using 'intel'

63735af

compiler.

[mfem] Allow pumi >= 2.2.3 with mfem >= 4.2.0.

d64dd6b

[ceed] Use pumi v2.2.5 with ceed v4.0.0.

[ceed] Explicitly use occa v1.1.0 with ceed v4.0.0.

66010cc

Use [email protected]+rocm with [email protected]+mfem+hip.

Merge branch 'develop' into packages/ceed-4.0

9c16781

[ceed] Add NekRS v21 as a dependency for ceed v4.0.0.

a8963b9

[ceed] Fix NekRS version: 21 --> 21.0

3726dee

[ceed] Propagate +cuda variant to petsc for ceed v4.0.

f74b1b4

[mfem] Propagate '+rocm' variant to some other packages.

2a28784

v-dobrev added 2 commits April 19, 2021 13:09

[ceed] Remove comments and FIXMEs about 'magma+hip'.

f1890a9

[ceed] [libceed] Remove TODOs about occa + hip.

b42079a

jedbrown added 5 commits May 14, 2021 19:39

Merge remote-tracking branch 'origin/develop' into packages/ceed-4.0

9f70d5d

libceed: use ROCmPackage and +rocm

2fd0ae5

petsc: use ROCmPackage for HIP

8781351

libceed, petsc: use CudaPackage

e5b9b3a

ceed: forward cuda_arch and amdgpu_target

0128eed

jedbrown reviewed May 20, 2021

View reviewed changes

[mfem] Use Spack's CudaPackage as a base class; as a result,

2127349

'cuda_arch' values should not include the 'sm_' prefix. Also, propagate 'cuda_arch' and 'amdgpu_target' variants to enabled dependencies.

jedbrown reviewed May 20, 2021

View reviewed changes

petsc: variant is +rocm, package name is hip

85658d7

v-dobrev removed the WIP label May 23, 2021

v-dobrev requested review from jedbrown and tzanio May 23, 2021 05:32

jedbrown previously approved these changes May 23, 2021

View reviewed changes

jedbrown requested a review from balay May 23, 2021 05:43

tzanio previously approved these changes May 23, 2021

View reviewed changes

Merge remote-tracking branch 'origin/develop' into packages/ceed-4.0

ee65885

Resolve petsc conflict on new hwloc and openmp variants.

jedbrown dismissed stale reviews from tzanio and themself via ee65885 June 3, 2021 17:17

jedbrown approved these changes Jun 3, 2021

View reviewed changes

tzanio merged commit e4a559a into develop Jun 3, 2021

tzanio deleted the packages/ceed-4.0 branch June 3, 2021 18:32

tldahlgren mentioned this pull request Jun 4, 2021

libceed: bugfix: Update the hip-related variant #24142

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CEED v4.0 release#22735

CEED v4.0 release#22735
tzanio merged 29 commits intodevelopfrom
packages/ceed-4.0

v-dobrev commented Apr 1, 2021

Uh oh!

Uh oh!

v-dobrev commented Apr 3, 2021

Uh oh!

v-dobrev commented Apr 7, 2021

Uh oh!

v-dobrev commented May 3, 2021 •

edited

Loading

Uh oh!

jedbrown May 20, 2021

Uh oh!

jedbrown May 20, 2021

Uh oh!

v-dobrev May 20, 2021

Uh oh!

v-dobrev commented May 20, 2021

Uh oh!

jedbrown left a comment

Uh oh!

tzanio left a comment

Uh oh!

tzanio commented May 31, 2021

Uh oh!

tzanio commented Jun 3, 2021

Uh oh!

jedbrown left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

v-dobrev commented Apr 1, 2021

Uh oh!

Uh oh!

v-dobrev commented Apr 3, 2021

Uh oh!

v-dobrev commented Apr 7, 2021

Uh oh!

v-dobrev commented May 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jedbrown May 20, 2021

Choose a reason for hiding this comment

Uh oh!

jedbrown May 20, 2021

Choose a reason for hiding this comment

Uh oh!

v-dobrev May 20, 2021

Choose a reason for hiding this comment

Uh oh!

v-dobrev commented May 20, 2021

Uh oh!

jedbrown left a comment

Choose a reason for hiding this comment

Uh oh!

tzanio left a comment

Choose a reason for hiding this comment

Uh oh!

tzanio commented May 31, 2021

Uh oh!

tzanio commented Jun 3, 2021

Uh oh!

jedbrown left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

v-dobrev commented May 3, 2021 •

edited

Loading