Repository navigation
Support external lz4, zstd, blosc, zlib #464
Description
Problem description
This package seems to unconditionally build vendored copies of lz4, zstd and blosc. This makes it hard to
- integrate this package with other libraries that pull in any of these libraries at a different version -- if it were external, one version could be used for all;
- update or patch binary dependencies in case of CVEs;
- deal with alternative providers, e.g. zlib-ng instead of zlib.
Also, the "build system" seems to be effectively native only (cpuinfo based), and very much x86 centered. That also wouldn't be a big deal if I could point numcodes to pre-compiled external versions of lz4, zstd and blosc.
Can you please make this package work with an external/system lz4, zstd, and blosc? That'd be very helpful, thanks.
Probably this means using a better build system (maybe meson?) instead of a handwritten setup.py.
Reacted by Adam J. Stewart, ♘, Christoph Gohlke, Benjamin Leff and Agriya Khetarpal
Activity
[-]Support external lz4, zstd, blosc[/-][+]Support external lz4, zstd, blosc, zlib[/+]martindurant commentedon Sep 25, 2023
I have mentioned elsehwere, but the package
cramjamincludes several useful and important compressors in one install, and ther tend to have good preformance compared to other versions. It does not support blosc, but that could be requested and would make numcodecs a much much simpler package to build and distribute.joshmoore commentedon Sep 25, 2023
Hi @haampie. I don't think anyone will disagree with your thesis here. It's more a matter of making it happen. Helping hands and suggestions like @martindurant's on getting us there are very welcome!
(e.g. see #274)
BwL1289 commentedon Aug 23, 2024
Linking #254. I know this is quite old. I'm experiencing this as well.
milesgranger commentedon Sep 11, 2024
I learned about this issue thru one on cramjam (milesgranger/cramjam#110); I'm fixing to release 2.8.4 w/ blosc2 support; it's available now for testing w/ 2.8.4rc3 on PyPI. It already supports lz4, zstd and others in a small and easy install.
It would be interesting to know if this helps at all here.
martindurant commentedon Sep 11, 2024
Absolutely, thanks @milesgranger . The code here still references blosc1, so some compat work would be needed I think.
I really think that there is no need for numcodecs to be repeating the build process for these standard compressors and/or relying on system libraries. Some things will still need cython for custom algorithms, but the less the better.