ECP-SDK: enable hdf5 VOL adapters by svenevs · Pull Request #33393 · spack/spack

svenevs · 2022-10-18T15:50:46Z

When +hdf5, enable VOL adapters suitable for the SDK.
Each VOL package must prepend to the HDF5_PLUGIN_PATH.
hdf5: 1.13.3 will break existing VOL packages, constrain
VOLs related to SDK and add note to keep 1.13.2 available.
hdf5-vol-async:
- Do not set HDF5_VOL_CONNECTOR, consumers must opt-in.
- Enforce DAG constraints on MPI to require threaded version.
- Depend on an explicit version of argbots to relax
  concretization issues in other spack environments.
paraview: fix compiler flag usage for the 110 ABI (followup to ParaView: ParaView needs to set the HDF5 API #33617).
Confirm builds.
- e4s container build
- vanilla build bare metal
- e4s site build incl. site yamls
- gitlab dav-sdk pipeline keeps failing in paraview, cannot reproduce locally
Confirm with HDF5 group about vol-async environment variables.

root@79bbd49cf2a8:/# spack load ecp-data-vis-sdk                                                                                                        
root@79bbd49cf2a8:/# echo $HDF5_PLUGIN_PATH | tr : $'\n' | xargs ls                                                                                     
/build/src/opt/spack/linux-ubuntu20.04-zen2/gcc-11.1.0/hdf5-vol-async-1.3-qdnz5mqjb2puxdcwoy7muk63udmw7uwz/lib:
libasynchdf5.a  libh5async.so

/build/src/opt/spack/linux-ubuntu20.04-zen2/gcc-11.1.0/hdf5-vol-cache-v1.0-oevenlhfpnlhbnykhwudmpovgg4rihtu/lib:
libhdf5_vol_cache.so

/build/src/opt/spack/linux-ubuntu20.04-zen2/gcc-11.1.0/hdf5-vol-log-1.3.0-2dl7s3whgwz6hs2ftzrolzfjd54me4hs/lib:
libH5VL_log.a  libH5VL_log.so  libH5VL_log.so.0  libH5VL_log.so.0.0.0

spackbot-app · 2022-10-18T15:50:57Z

@lrknox can you review this PR?

This PR modifies the following package(s), for which you are listed as a maintainer:

ecp-data-vis-sdk
hdf5-vol-async
hdf5-vol-cache
hdf5-vol-log

var/spack/repos/builtin/packages/ecp-data-vis-sdk/package.py

var/spack/repos/builtin/packages/hdf5-vol-async/package.py

kwryankrattiger

LGTM

var/spack/repos/builtin/packages/hdf5-vol-async/package.py

houjun · 2022-10-25T18:18:35Z

@derobins Any comment on setting the environment variables for VOL connectors in spack?

var/spack/repos/builtin/packages/hdf5-vol-async/package.py

brtnfld · 2022-10-25T19:43:46Z

I agree that Each VOL package should append to HDF5_PLUGIN_PATH.
As for HDF5_VOL_CONNECTOR, Spack should not be setting this environment variable. Setting this causes an HDF5 application, by default, to use the vol connector, which might not be what the user intended. The default for an HDF5 install should be the native connector unless specified otherwise. It is common practice to set HDF5_VOL_CONNECTOR when using a vol connector since this is how one enables/disables using the vol connectors. Enabling a vol connector should be explicit.

derobins · 2022-10-26T19:04:03Z

I agree that Each VOL package should append to HDF5_PLUGIN_PATH. As for HDF5_VOL_CONNECTOR, Spack should not be setting this environment variable. Setting this causes an HDF5 application, by default, to use the vol connector, which might not be what the user intended. The default for an HDF5 install should be the native connector unless specified otherwise. It is common practice to set HDF5_VOL_CONNECTOR when using a vol connector since this is how one enables/disables using the vol connectors. Enabling a vol connector should be explicit.

Agree with Scot - append to HDF5_PLUGIN_PATH (order shouldn't be important) and don't set HDF5_VOL_CONNECTOR.

var/spack/repos/builtin/packages/hdf5-vol-async/package.py

hyoklee · 2022-11-04T20:04:38Z

@spackbot run pipeline

spackbot-app · 2022-11-04T20:04:42Z

I've started that pipeline for you!

- When +hdf5, enable VOL adapters suitable for the SDK. - Each VOL package must prepend to the HDF5_PLUGIN_PATH. - hdf5: 1.13.3 will break existing VOL packages, constrain VOLs related to SDK and add note to keep 1.13.2 available. - hdf5-vol-async: - Do not set HDF5_VOL_CONNECTOR, consumers must opt-in. - Enforce DAG constraints on MPI to require threaded version. - Depend on an explicit version of argbots to relax concretization issues in other spack environments. - paraview: fix compiler flag usage for the 110 ABI (followup to spack#33617).

xdelaruelle · 2023-06-21T04:49:44Z

Maybe this PR could be closed as #35195 has been merged.

spackbot-app bot added the update-package label Oct 18, 2022

spackbot-app bot requested review from houjun, hyoklee, jeanbez and kwryankrattiger October 18, 2022 15:50

svenevs marked this pull request as draft October 18, 2022 15:51

svenevs commented Oct 18, 2022

View reviewed changes

var/spack/repos/builtin/packages/ecp-data-vis-sdk/package.py Show resolved Hide resolved

var/spack/repos/builtin/packages/ecp-data-vis-sdk/package.py Show resolved Hide resolved

var/spack/repos/builtin/packages/hdf5-vol-async/package.py Show resolved Hide resolved

jeanbez suggested changes Oct 18, 2022

View reviewed changes

var/spack/repos/builtin/packages/hdf5-vol-async/package.py Show resolved Hide resolved

tldahlgren assigned jeanbez Oct 18, 2022

kwryankrattiger previously approved these changes Oct 24, 2022

View reviewed changes

svenevs dismissed kwryankrattiger’s stale review via 3e9e4fc October 25, 2022 18:10

svenevs force-pushed the ecp-hdf5-vol branch from 5510d05 to 3e9e4fc Compare October 25, 2022 18:10

spackbot-app bot added the conflicts label Oct 25, 2022

svenevs commented Oct 25, 2022

View reviewed changes

var/spack/repos/builtin/packages/hdf5-vol-async/package.py Outdated Show resolved Hide resolved

svenevs commented Oct 25, 2022

View reviewed changes

var/spack/repos/builtin/packages/hdf5-vol-async/package.py Outdated Show resolved Hide resolved

svenevs force-pushed the ecp-hdf5-vol branch from 3e9e4fc to ed2ea96 Compare October 26, 2022 19:46

spackbot-app bot added the dependencies label Oct 26, 2022

svenevs force-pushed the ecp-hdf5-vol branch from ed2ea96 to db99bc9 Compare October 26, 2022 20:49

kwryankrattiger previously approved these changes Oct 26, 2022

View reviewed changes

hyoklee previously approved these changes Oct 26, 2022

View reviewed changes

svenevs marked this pull request as ready for review October 26, 2022 22:01

kwryankrattiger reviewed Oct 27, 2022

View reviewed changes

var/spack/repos/builtin/packages/hdf5-vol-async/package.py Outdated Show resolved Hide resolved

jeanbez previously approved these changes Oct 27, 2022

View reviewed changes

houjun previously approved these changes Oct 27, 2022

View reviewed changes

svenevs dismissed stale reviews from houjun and jeanbez via c4cf98f October 27, 2022 19:35

svenevs dismissed stale reviews from hyoklee and kwryankrattiger via c4cf98f October 27, 2022 19:35

svenevs force-pushed the ecp-hdf5-vol branch 3 times, most recently from ef75c42 to 4864df4 Compare November 2, 2022 16:41

svenevs force-pushed the ecp-hdf5-vol branch 2 times, most recently from ef08ad7 to 59da04d Compare November 3, 2022 20:59

kwryankrattiger previously approved these changes Nov 3, 2022

View reviewed changes

hyoklee previously approved these changes Nov 3, 2022

View reviewed changes

jeanbez previously approved these changes Nov 4, 2022

View reviewed changes

houjun previously approved these changes Nov 4, 2022

View reviewed changes

svenevs dismissed stale reviews from houjun, jeanbez, hyoklee, and kwryankrattiger via b3bb8e3 November 8, 2022 20:07

svenevs force-pushed the ecp-hdf5-vol branch from 59da04d to b3bb8e3 Compare November 8, 2022 20:07

kwryankrattiger mentioned this pull request Nov 16, 2022

ParaView/VTK: Patch xdmf2 for HDF5 1.13 #33930

Merged

svenevs force-pushed the ecp-hdf5-vol branch from b3bb8e3 to e5c51d7 Compare November 29, 2022 00:03

kwryankrattiger mentioned this pull request Jan 27, 2023

Ecp hdf5 vol #35195

Merged

hyoklee mentioned this pull request Jun 20, 2023

Add verbatim attribute to env.set command #38465

Merged

kwryankrattiger closed this Jun 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ECP-SDK: enable hdf5 VOL adapters#33393

ECP-SDK: enable hdf5 VOL adapters#33393
svenevs wants to merge 1 commit intospack:developfrom
svenevs:ecp-hdf5-vol

svenevs commented Oct 18, 2022 •

edited

Loading

Uh oh!

spackbot-app bot commented Oct 18, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kwryankrattiger left a comment

Uh oh!

Uh oh!

houjun commented Oct 25, 2022

Uh oh!

Uh oh!

brtnfld commented Oct 25, 2022

Uh oh!

derobins commented Oct 26, 2022

Uh oh!

Uh oh!

hyoklee commented Nov 4, 2022

Uh oh!

spackbot-app bot commented Nov 4, 2022

Uh oh!

xdelaruelle commented Jun 21, 2023 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Conversation

svenevs commented Oct 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

spackbot-app bot commented Oct 18, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kwryankrattiger left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

houjun commented Oct 25, 2022

Uh oh!

Uh oh!

brtnfld commented Oct 25, 2022

Uh oh!

derobins commented Oct 26, 2022

Uh oh!

Uh oh!

hyoklee commented Nov 4, 2022

Uh oh!

spackbot-app bot commented Nov 4, 2022

Uh oh!

xdelaruelle commented Jun 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

svenevs commented Oct 18, 2022 •

edited

Loading

xdelaruelle commented Jun 21, 2023 •

edited

Loading