Skip to content

netcdf-cxx4: use mpi compile wrapper if needed.#17051

Closed
tkameyama wants to merge 5 commits intospack:developfrom
tkameyama:bugfix/netcdf-cxx4
Closed

netcdf-cxx4: use mpi compile wrapper if needed.#17051
tkameyama wants to merge 5 commits intospack:developfrom
tkameyama:bugfix/netcdf-cxx4

Conversation

@tkameyama
Copy link
Copy Markdown
Contributor

If netcdf-c was compiled with MPI, netcdf-cxx4 is compiled with mpicc and mpicxx.

@adamjstewart
Copy link
Copy Markdown
Member

@WardF @skosukhin can you review this PR?

@tkameyama
Copy link
Copy Markdown
Contributor Author

tkameyama commented Jun 22, 2020

ping @adamjstewart @WardF @skosukhin

@whardier

This comment has been minimized.

@tkameyama

This comment has been minimized.

def configure_args(self):
config_args = []

if self.spec.satisfies('^mpi'):
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would specify the condition like this (like in #16719):

netcdf_c_spec = self.spec['netcdf-c']
if '+mpi' in netcdf_c_spec or '+parallel-netcdf' in netcdf_c_spec:

However, the current solution is also fine in my opinion.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When netcdf-c is updated (For example new variant is added and the variant is needed MPI),
#16719 solution is needed to change netcdf-cxx.
But thi PR is not needed to change.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fact that mpi is somewhere in the dependency tree does not necesserily mean that we need to use MPI wrappers. For example, in theory (not in Spack yet), it is possible to build netcdf-c~mpi ^hdf5+mpi and I am not sure that we need to compile netcdf-cxx with mpi wrappers in this case.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see.
I Changed to look at variant of netcdf-c.

@skosukhin
Copy link
Copy Markdown
Member

@tkameyama thank you. A few more comments:

  1. I am not sure that we need to build with MPI wrappers at all. netcdf-fortran, for example, uses mpi at least in the test suite and in the examples. Therefore, it makes sense to build it with MPI wrappers. netcdf-cxx4, on the other hand, does not seem to use mpi directly, the dependencies on MPI libraries are transitive and are supposed to be resolved implicitly via rpaths. The case of static linking does not seem to be properly covered anyway (at least I got linking errors when tried to build netcdf-cxx4 ^netcdf-c~mpi~shared). Could you, please, provide an example when we need to use MPI wrappers?

  2. There is a problem with NEEDED entries in libnetcdf_c++4.so: underlinked libnetcdf_c++4.so.1: undefined symbol: nc_* Unidata/netcdf-cxx4#86. Basically, we can't simply link to the library: g++ main.c -L/some/path -lnetcdf_c++4. Instead, we always have to append -lnetcdf to the command. We can solve this in Spack. We can either:

    config_args.append('LIBS=-lnetcdf')

    or wait until autotools: set 'ldlibs' as 'LIBS' #17254 is merged and extend flag_handler with:

    elif name == 'ldlibs':
        flags.append('-lnetcdf')

    Could you, please, take care of this? In any case, it makes sense to add the link to the issue as a comment: the issue might get resolved upstream in the future and the workaround might become redundant.

  3. Currently, the package unconditionally depends on automake, autoconf, libtool, and m4. This is because we have force_autoreconf = True, which was introduced ages ago. Could you, please, check whether we need this for the newer version (I would simply remove it).

Of course, it's fine if you say no to any of this. I just thought that since you seem to be interested in the package, you might want to fix some other issues as well.

@tkameyama tkameyama force-pushed the bugfix/netcdf-cxx4 branch from 4ff111c to 5f00b16 Compare June 26, 2020 06:40
@tkameyama
Copy link
Copy Markdown
Contributor Author

  1. I am not sure that we need to build with MPI wrappers at all. netcdf-fortran, for example,
    uses mpi at least in the test suite and in the examples. Therefore, it makes sense to build it with
    MPI wrappers. netcdf-cxx4, on the other hand, does not seem to use mpi directly, the dependencies
    on MPI libraries are transitive and are supposed to be resolved implicitly via rpaths. The case of static
    linking does not seem to be properly covered anyway (at least I got linking errors when tried to build netcdf-cxx4 ^netcdf-c~mpi~shared). Could you, please, provide an example when we need to use MPI wrappers?

I faild to compile plugins/H5Zbzip2.c.
H5Zbzip2.c include hdf5.h, and hdf5.h include mpi.h when hdf5+mpi.
(And I use Fujitsu MPI, and this package's header information is incorrect. #17253)
I restore mpi wrapper condition to dependency baseed.

  1. There is a problem with NEEDED entries in libnetcdf_c++4.so:
    Unidata/netcdf-cxx4#86.
    Basically, we can't simply link to the library: g++ main.c -L/some/path -lnetcdf_c++4.
    Instead, we always have to append -lnetcdf to the command. We can solve this in Spack.

I think it's better to solve this problem by adding -lnetcdf to LIBS in the configure script.
I add patch for this.

3. Currently, the package unconditionally depends on `automake`, `autoconf`, `libtool`, and `m4`. This is because we have `force_autoreconf = True`, which was introduced ages ago. Could you, please, check whether we need this for the newer version (I would simply remove it).

Maybe we still need it.

@tkameyama
Copy link
Copy Markdown
Contributor Author

@skosukhin

  1. There is a problem with NEEDED entries in libnetcdf_c++4.so: Unidata/netcdf-cxx4#86. Basically, we can't simply link to the library: g++ main.c -L/some/path -lnetcdf_c++4. Instead, we always have to append -lnetcdf to the command. We can solve this in Spack.

I understand this problem.

Nomally, If we want too use libnetcdf, we use AC_SEARCH_LIBS in configure.ac.
AC_SEARCH_LIBS([nc_create], [netcdf], [], [])
AC_SEARCH_LIBS search libnetcdf and add -lnetcdf to LIBS variable.

But currently when nc-config is found, configure.ac skip AC_SEARCH_LIBS, and LIBS variable is not updated.

libnetcdf_so.4.3.1.patch append output of nc-config --libs in LIBS variable.

@alalazo
Copy link
Copy Markdown
Member

alalazo commented Sep 16, 2021

@skosukhin @tkameyama Is this something we still want or can we close the PR?

@alalazo alalazo self-assigned this Sep 16, 2021
@alalazo
Copy link
Copy Markdown
Member

alalazo commented Jan 25, 2022

Closing this PR as stale, but feel free to reopen if you want to continue working on it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants