Skip to content

py-netcdf4: enable non-MPI build per variant, even if netcdf-c was built with MPI#48694

Merged
tldahlgren merged 13 commits intospack:developfrom
climbfuji:bugfix/py-netcdf4-nompi
Apr 29, 2025
Merged

py-netcdf4: enable non-MPI build per variant, even if netcdf-c was built with MPI#48694
tldahlgren merged 13 commits intospack:developfrom
climbfuji:bugfix/py-netcdf4-nompi

Conversation

@climbfuji
Copy link
Copy Markdown
Contributor

@climbfuji climbfuji commented Jan 23, 2025

Description

Allow building py-netcdf4 ~mpi when netcdf-c was built with +mpi. The patch overrides the auto-detect feature (has_parallel_support) in setup.py. The logic in setup.py changed between 1.6.5 and 1.7.1, therefore this patch
only works for versions 1.7.1 and later.

Unidata/netcdf4-python#1389

@Chrismarsh FYI

Copy link
Copy Markdown
Member

@skosukhin skosukhin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine with moving all @:1.6 versions under with default_args(deprecated=True):.

@skosukhin
Copy link
Copy Markdown
Member

skosukhin commented Jan 24, 2025

This is the full patch that seems to allow for py-netcdf4~mpi ^hdf5+mpi (at least, the Debian-provided GNU patch 2.7.6 does not have problems with it):

--- a/include/netCDF4.pxi
+++ b/include/netCDF4.pxi
@@ -7,2 +7,2 @@
-cdef extern from "H5public.h":
-    ctypedef int herr_t
+ctypedef int herr_t
+cdef extern from *:
--- a/setup.py
+++ b/setup.py
@@ -395 +395 @@
-    has_parallel_support = check_has_parallel_support(inc_dirs)
+    has_parallel_support = False
Here is the patch with more context.
--- a/include/netCDF4.pxi
+++ b/include/netCDF4.pxi
@@ -4,8 +4,8 @@
   ctypedef long ptrdiff_t

# hdf5 version info.
-cdef extern from "H5public.h":
-    ctypedef int herr_t
+ctypedef int herr_t
+cdef extern from *:
   int H5get_libversion( unsigned int *majnum, unsigned int *minnum, unsigned int *relnum ) nogil

cdef extern from *:
--- a/setup.py
+++ b/setup.py
@@ -392,7 +392,7 @@
      (netcdf_lib_version > "4.4" and netcdf_lib_version < "4.5"):
       has_cdf5_format = True

-    has_parallel_support = check_has_parallel_support(inc_dirs)
+    has_parallel_support = False
   has_has_not = "has" if has_parallel_support else "does not have"
   print(f"netcdf lib {has_has_not} parallel functions")

We apply the patch as follows:

    with when("@1.7:~mpi"):
        patch("disable_parallel_support.patch", when="^netcdf-c+mpi")
        patch("disable_parallel_support.patch", when="^hdf5+mpi")

It might be enough to apply the patch only when ^hdf5+mpi (because ^netcdf-c+mpi implies ^hdf5+mpi) but netcdf-c~mpi might be an external package built against hdf5+mpi. I don't know how Spack handles that at the moment. Some (long) time ago it used to drop the dependencies of external packages from the specs.

And Wow! It looks like all that is just to emit an obsolete warning under a broken condition.

@climbfuji
Copy link
Copy Markdown
Contributor Author

I updated the patch and package as you suggested, and I built py-netcdf4~mpi ^netcdf-c~mpi ^hdf5+mpi successfully.

@climbfuji climbfuji requested a review from skosukhin January 24, 2025 20:02
@Chrismarsh
Copy link
Copy Markdown
Contributor

It would be good to not let this stagnant, is there anything I can test to help move this along?

@skosukhin
Copy link
Copy Markdown
Member

Friday evening is for sure not the best time for a reminder like this :) But thank you, I will try my best to take a look at this on Monday.

@climbfuji climbfuji requested a review from skosukhin March 12, 2025 16:07
@climbfuji
Copy link
Copy Markdown
Contributor Author

@skosukhin Sorry that I apparently missed your request for changes three weeks ago. I applied the suggestions and built py-netcdf4 ~mpi with netcdf-c +mpi successfully on my laptop (gcc@13).

@Chrismarsh
Copy link
Copy Markdown
Contributor

@climbfuji are you just waiting on a review for this?

@alalazo
Copy link
Copy Markdown
Member

alalazo commented Apr 14, 2025

Some (long) time ago it used to drop the dependencies of external packages from the specs.

For the record, Spack still trims the dependencies of external packages

@climbfuji climbfuji requested a review from tldahlgren April 16, 2025 20:52
climbfuji added a commit to climbfuji/spack that referenced this pull request Apr 17, 2025
…-pick changes from spack#48694, change numpy@2 dependency to @1.26
@climbfuji
Copy link
Copy Markdown
Contributor Author

@skosukhin I had to revert part of the patch in commit 8976169 that you contributed a while ago. With that patch, I got the errors reported in #48694 (comment). Would you be able to review again and merge if it looks ok?

@climbfuji climbfuji requested a review from skosukhin April 22, 2025 15:26
@skosukhin
Copy link
Copy Markdown
Member

@climbfuji what is the value of this MR then? What is the spec that you couldn't install and that can be installed with these changes?

@climbfuji
Copy link
Copy Markdown
Contributor Author

climbfuji commented Apr 22, 2025

@climbfuji what is the value of this MR then? What is the spec that you couldn't install and that can be installed with these changes?

py-netcdf4 ~mpi ^netcdf-c+mpi

@skosukhin
Copy link
Copy Markdown
Member

It turns out that we cannot even spack install py-netcdf4 (see #50184).

@climbfuji
Copy link
Copy Markdown
Contributor Author

It turns out that we cannot even spack install py-netcdf4 (see #50184).

I am still on the last commit before the compilers-as-nodes changes were merged (three weeks-ish ago) and py-netcdf4 builds fine for me. This PR started months ago when that code wasn't in develop yet. We should merge this PR and then add the updates for the new compiler dependencies features in a separate PR, since it is a separate issue.

@Chrismarsh
Copy link
Copy Markdown
Contributor

@skosukhin I have also had issues with mpi on the recent py-netcdf

#50075

As it ingests a header that has mpi (when +mpi)

I think this should be merged and then I can rebase #50075 on top of this PR.

Copy link
Copy Markdown
Member

@skosukhin skosukhin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please, fix the PR description.

and when netCDF was built with +mpi but hdf5 was built with ~mpi

  1. It's not possible to have netcdf-c+mpi ^hdf5~mpi.
  2. We can't have netcdf-c~mpi ^hdf5+mpi after you removed the respective part of the patch.

@climbfuji
Copy link
Copy Markdown
Contributor Author

Please, fix the PR description.

and when netCDF was built with +mpi but hdf5 was built with ~mpi

  1. It's not possible to have netcdf-c+mpi ^hdf5~mpi.
  2. We can't have netcdf-c~mpi ^hdf5+mpi after you removed the respective part of the patch.

Done, thanks.

@skosukhin
Copy link
Copy Markdown
Member

@alalazo could you merge this, please?

@tldahlgren tldahlgren enabled auto-merge (squash) April 29, 2025 17:29
@tldahlgren tldahlgren merged commit eaabde6 into spack:develop Apr 29, 2025
16 checks passed
teaguesterling pushed a commit to teaguesterling/spack that referenced this pull request May 20, 2025
…ilt with MPI (spack#48694)

* Add patch for py-netcdf4 so that we can build py-netcdf4 with ~mpi when netCDF was built with +mpi
* Address reviewer requests for py-netcdf4. Add conflict for 'pynetcdf4~mpi ^netcdf-c~mpi ^hdf5+mpi'
* Make py-netcdf4~mpi ^netcdf-c~mpi ^hdf5+mpi work
* Apply suggestions from code review
  Co-authored-by: Sergey Kosukhin <[email protected]>
* Update var/spack/repos/builtin/packages/py-netcdf4/disable_parallel_support.patch
* Apply suggestions from code review
  Co-authored-by: Sergey Kosukhin <[email protected]>

---------

Co-authored-by: Sergey Kosukhin <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants