zlib-api: new virtual with zlib/zlib-ng as providers by haampie · Pull Request #37372 · spack/spack

haampie · 2023-05-02T09:18:31Z

Create a new virtual zlib-api and have zlib / zlib-ng as providers.

zlib remains the default provider, maybe in the future we can swap it for zlib-ng to save some CPU cycles across the globe...

My PR zlib-ng/zlib-ng#1546 to improve the zlib-ng build system was merged upstream and included here as a patch, that's why this PR has more + than -; the package changes are mostly one-liners replacing zlib with zlib-api.

haampie · 2023-05-02T13:56:41Z

Comparing python +opimizations ^zlib-ng against system Python on macOS Ventura to create a compressed tarball of a folder with 250MB of stuff:

zlib-ng version: 2.92s
zlib version: 5.89s

So, 2x faster 🙃

And Spack's tar (based on pigz w/ zlib-ng) vs system tar (gzip w/ zlib) on a macbook air m1:

0.649s
8.661s

13x faster... not bad. (this is both threaded and has better single-threaded perf thanks to zlib-ng)

haampie · 2023-07-23T19:54:23Z

gdal is the only package that needed something extra to fix the build: ENABLE_DEFLATE64=OFF This is because it uses a vendored libdeflate that seems to use zlib-specific stuff (which they also vendor). Solution is probably to use external libdeflate, also note that they use deflate because it's faster than zlib, but maybe it's slower than zlib-ng?

zlib-ng v2.1.3 has some weird rpath issues (zlib-ng/zlib-ng#1523) should be fixable in a next release

haampie · 2023-07-24T10:53:26Z

Created a patch upstream for darwin and included it here.
Added a clarifying comment to the GDAL package.

haampie · 2023-07-24T13:11:40Z

CI is green after building 1000s of packages, so I think this PR is good to go after switching back to zlib as the default zlib-api.

https://gitlab.spack.io/spack/spack/-/pipelines/452853

haampie · 2023-07-24T16:14:39Z

Sigh. It's going to be building 1000s of packages again, because the zlib package hash has changed now that it provides zlib-api 😆

wdconinc · 2023-08-02T13:38:35Z

Are we potentially adding regressions by replacing depends_on("[email protected]:") with depends_on("zlib-api")? Is it potentially safer, or at least more conservative, to leave those explicit versions in place, or to use an additional depends_on("[email protected]:", when="^zlib")?

var/spack/repos/tutorial/packages/hdf5/package.py

var/spack/repos/builtin/packages/tandem/package.py

var/spack/repos/builtin/packages/r/package.py

var/spack/repos/builtin/packages/py-imagecodecs/package.py

var/spack/repos/builtin/packages/orc/package.py

var/spack/repos/builtin/packages/netcdf-c/package.py

var/spack/repos/builtin/packages/hdf5/package.py

var/spack/repos/builtin/packages/hdf/package.py

haampie · 2023-08-03T09:19:36Z

Thanks @wdconinc! I've revived the zlib lowerbounds

var/spack/repos/builtin/packages/libreproc/package.py

var/spack/repos/builtin/packages/libsolv/package.py

adamjstewart · 2023-08-08T14:45:46Z

Only concern is the inability to specify which version of zlib is required. Does zlib-ng have a known compatibility mapping? Would be nice if we could specify which versions of zlib-api are provided by each version of zlib/zlib-ng.

haampie · 2023-08-08T14:51:55Z

I agree with that, but I think zlib follows https://www.hyrumslaw.com/; it doesn't have a clear public API, and some people mess with accidentally exposed internals :p I think the easiest way to fix those issues is on a per package basis, so e.g. depends_on("[email protected]:", when="^zlib-ng") instead of adding provides("zlib-api@...") to zlib and zlib-ng.

var/spack/repos/builtin/packages/gdal/package.py

haampie · 2023-08-08T21:18:20Z

@spackbot rerun pipeline

spackbot-app · 2023-08-08T21:18:25Z

I've started that pipeline for you!

Co-Authored-By: Wouter Deconinck <[email protected]>

alalazo

I checked all the commits, though in the first one I just checked the core files and zlib + zlib-ng packages. This LGTM. We can make zlib-ng the default in a following PR.

In the HPC package manager, we want the fastest `zlib` implementation by default. `zlib-ng` is up to 4x faster than stock `zlib`, and it can do things like take advantage of AVX-512 instructions. This PR makes `zlib-ng` the default `zlib-api` provider (`zlib-api` was introduced earlier, in #37372). As far as I can see, the only issues you can encounter are: 1. Build issues with packages that heavily rely on `zlib` internals. In Gitlab CI only one out of hundreds of packages had that issue (it extended zlib with deflate stuff, and used its own copy of zlib sources). 2. Packages that like to detect `zlib-ng` separately and rely on `zlib-ng` internals. The only issue I've found with this among the hundreds of packages built in CI is `perl` trying to report more specific zlib-ng version details, and relied on some internals that got refactored. But yeah... that warrants a patch / conflict and is nothing special. At runtime, you cannot really have any issues, given that zlib and zlib-ng export the exact same symbols (and zlib-ng tests this in their CI). You can't really have issues with externals when using zlib-ng either. The only type of issue is when system zlib is rather new, and not marked as external; if another external uses new symbols, and Spack builds an older zlib/zlib-ng, then the external might not find the new symbols. But this is a configuration issue, and it's not an issue caused by zlib-ng, as the same would happen with older Spack zlib. * zlib-api: use zlib-ng +compat by default * make a trivial change to zlib-ng to trigger a rebuild * add `haampie` as maintainer

Introduces a new virtual zlib-api, which replaces zlib in most packages. This allows users to switch to zlib-ng by default for better performance.

In the HPC package manager, we want the fastest `zlib` implementation by default. `zlib-ng` is up to 4x faster than stock `zlib`, and it can do things like take advantage of AVX-512 instructions. This PR makes `zlib-ng` the default `zlib-api` provider (`zlib-api` was introduced earlier, in spack#37372). As far as I can see, the only issues you can encounter are: 1. Build issues with packages that heavily rely on `zlib` internals. In Gitlab CI only one out of hundreds of packages had that issue (it extended zlib with deflate stuff, and used its own copy of zlib sources). 2. Packages that like to detect `zlib-ng` separately and rely on `zlib-ng` internals. The only issue I've found with this among the hundreds of packages built in CI is `perl` trying to report more specific zlib-ng version details, and relied on some internals that got refactored. But yeah... that warrants a patch / conflict and is nothing special. At runtime, you cannot really have any issues, given that zlib and zlib-ng export the exact same symbols (and zlib-ng tests this in their CI). You can't really have issues with externals when using zlib-ng either. The only type of issue is when system zlib is rather new, and not marked as external; if another external uses new symbols, and Spack builds an older zlib/zlib-ng, then the external might not find the new symbols. But this is a configuration issue, and it's not an issue caused by zlib-ng, as the same would happen with older Spack zlib. * zlib-api: use zlib-ng +compat by default * make a trivial change to zlib-ng to trigger a rebuild * add `haampie` as maintainer

After zlib -> zlib-api change (spack#37372) petsc is not including the zlib. This updates the spack-lib -> petsc-lib mapping.

spackbot-app bot added conflicts core PR affects Spack core functionality defaults dependencies documentation Improvements or additions to documentation new-variant new-version patch python R tests General test capability(ies) update-package virtual-dependencies libraries labels May 2, 2023

haampie mentioned this pull request May 5, 2023

python allow builds with zlib-ng #35174

Closed

haampie force-pushed the fix/zlib-api-virtual branch from cb4b2b7 to 9221271 Compare May 16, 2023 09:11

haampie force-pushed the fix/zlib-api-virtual branch 2 times, most recently from 1fd7a01 to c7f4779 Compare July 23, 2023 07:14

wdconinc reviewed Aug 2, 2023

View reviewed changes

var/spack/repos/tutorial/packages/hdf5/package.py Show resolved Hide resolved

wdconinc reviewed Aug 2, 2023

View reviewed changes

wdconinc reviewed Aug 3, 2023

View reviewed changes

var/spack/repos/builtin/packages/libreproc/package.py Show resolved Hide resolved

var/spack/repos/builtin/packages/libsolv/package.py Show resolved Hide resolved

wdconinc approved these changes Aug 6, 2023

View reviewed changes

haampie force-pushed the fix/zlib-api-virtual branch from 1e5d00f to 3052fbe Compare August 8, 2023 13:24

adamjstewart reviewed Aug 8, 2023

View reviewed changes

var/spack/repos/builtin/packages/gdal/package.py Show resolved Hide resolved

haampie and others added 10 commits August 9, 2023 12:29

zlib-api: support zlib-ng as alternative

7da1dac

add 2.1.3

519686f

fix perl build with zlib-ng

263e498

fix gdal build with zlib-ng

924ac90

try version 2.1.2 on darwin

75fb4d5

zlib-ng: add a patch for macos rpath

33d2b61

gdal: clarifying comment

0a14502

make zlib the default provider again

c33871e

Bring back zlib lowerbounds

2596b36

Co-Authored-By: Wouter Deconinck <[email protected]>

fix a few new packages

4b29372

haampie force-pushed the fix/zlib-api-virtual branch from 5ddfa19 to 4b29372 Compare August 9, 2023 10:29

alalazo approved these changes Aug 9, 2023

View reviewed changes

haampie enabled auto-merge (squash) August 9, 2023 12:49

haampie merged commit e51748e into spack:develop Aug 9, 2023

G-071 mentioned this pull request Aug 10, 2023

sgpp: fix build issues #39086

Merged

alalazo mentioned this pull request Aug 11, 2023

dd4hep: new version 1.26 #39230

Merged

tgamblin mentioned this pull request Aug 17, 2023

zlib-api: use zlib-ng +compat by default #39358

Merged

Thomas-Ulrich mentioned this pull request Aug 25, 2023

add memalign option to petsc to be used with tandem #37282

Merged

haampie deleted the fix/zlib-api-virtual branch November 1, 2023 15:41

JaroslavHron added a commit to JaroslavHron/spack that referenced this pull request Aug 1, 2024

fix petsc zlib-api dependence

47396c6

After zlib -> zlib-api change (spack#37372) petsc is not including the zlib. This updates the spack-lib -> petsc-lib mapping.

JaroslavHron mentioned this pull request Aug 1, 2024

fix petsc zlib-api processing #45545

Merged

JaroslavHron added a commit to JaroslavHron/spack that referenced this pull request Aug 1, 2024

fix petsc zlib-api dependence

13d5e48

After zlib -> zlib-api change (spack#37372) petsc is not including the zlib. This updates the spack-lib -> petsc-lib mapping.

Conversation

haampie commented May 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

haampie commented May 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

haampie commented Jul 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

haampie commented Jul 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

haampie commented Jul 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

haampie commented Jul 24, 2023

Uh oh!

wdconinc commented Aug 2, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

haampie commented Aug 3, 2023

Uh oh!

Uh oh!

Uh oh!

adamjstewart commented Aug 8, 2023

Uh oh!

haampie commented Aug 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

haampie commented Aug 8, 2023

Uh oh!

spackbot-app bot commented Aug 8, 2023

Uh oh!

alalazo left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

haampie commented May 2, 2023 •

edited

Loading

haampie commented May 2, 2023 •

edited

Loading

haampie commented Jul 23, 2023 •

edited

Loading

haampie commented Jul 24, 2023 •

edited

Loading

haampie commented Jul 24, 2023 •

edited

Loading

haampie commented Aug 8, 2023 •

edited

Loading