Repository navigation
Problems importing blosc on HPC cluster #69
Description
I am using zarr on an HPC cluster with a rather complex configuration (multiple conda envs, dask distributed, etc.)
Minimal, reproducible code sample, a copy-pastable example if possible
# What should happen (and works on cluster head node)
>>> import zarr
>>> zarr.Blosc
numcodecs.blosc.Blosc
# What happens on a compute node
>>> import zarr
>>> zarr.Blosc
AttributeError: module 'zarr' has no attribute 'Blosc
>>> import numcodecs.blosc
ImportError: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/numcodecs/blosc.cpython-36m-x86_64-linux-gnu.so)Problem description
Both examples use the same underlying conda environment and have identical zarr / numcodecs version. However, there is some deep library error which makes Blosc unusable on the compute nodes.
Version and installation information
Please provide the following:
- Value of
zarr.__version__: 2.2.0rc4.dev1 - Value of
numcodecs.__version__: 0.5.3 - Version of Python interpreter: 3.6.4
- Operating system (Linux/Windows/Mac) Linux
- How Zarr was installed (e.g., "using pip into virtual environment", or "using conda"): pip
Activity
jakirkham commentedon Feb 28, 2018
The error means that the build (i.e.
pip install), as we ship sdists on PyPI only, occurred on a machine that used glibc 2.14 or greater, was run on a machine that had a glibc version older than 2.14. IOW it was built on a newer version of Linux than you are running it on. The solution would be to build on the version of Linux you intend to run it on, but for various reasons that may be more complicated. One alternative might be to try the conda-forge binary fornumcodecs, which works with glibc 2.12 or newer. If that doesn't work, we can discuss other options.jakirkham commentedon Mar 1, 2018
Any luck @rabernat?
jakirkham commentedon Mar 3, 2018
Going to close as this isn't a problem with numcodecs per se, but an issue with the build (i.e.
pip installstep) occurring on a newer OS than the target machine's OS. Of course happy to talk with you more about this @rabernat if you would like and discuss other options if the one's suggested above don't work for your use case.jakirkham commentedon Mar 3, 2018
FWIW we could probably build binary wheels too as outlined in issue ( #70 ).