Can't install numcodecs on OSX Big Sur

Minimal, reproducible code sample, a copy-pastable example if possible

pip install numcodecs

Problem description

Installation fails due to Blosc/python-blosc#229. Updating the c-blosc submodule to v1.21.0 fixes the issue.

Version and installation information

Please provide the following:

Value of numcodecs.__version__: current master branch
Version of Python interpreter: 3.9
Operating system (Linux/Windows/Mac): Mac

Thanks for the data point! 😄

I'm also seeing this problem on macOS Catalina. The issue is with Apple Clang 12.0.0, not with the specific macOS version. See Blosc/c-blosc#316 for my error msg. It would be great if c-blosc could be installed externally at any version we need instead of using the git submodule.

@d70-t

I'm getting the same issue on a M1 MacBook Pro MacOS Big Sur 11.5 when trying to install numcodec v0.8.0 in a virtualenv using poetry and pyenv. Trying to install for python 3.8.11.

The error message is

c-blosc/internal-complibs/zlib-1.2.8/gzread.c:30:15: error: implicit declaration of function 'read' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        ret = read(state->fd, buf + *have, len - *have);
              ^
c-blosc/internal-complibs/zlib-1.2.8/gzread.c:30:15: note: did you mean 'fread'?
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/stdio.h:158:9: note: 'fread' declared here
size_t   fread(void * __restrict __ptr, size_t __size, size_t __nitems, FILE * __restrict __stream);
         ^
c-blosc/internal-complibs/zlib-1.2.8/gzread.c:591:11: error: implicit declaration of function 'close' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
    ret = close(state->fd);
          ^
2 errors generated.
[numcodecs] command 'clang' failed with exit status 1

@d70-t - how did you force the numcodecs install to use a newer version of c-blosc? Did you manage to figure out a workaround?

The issue is slightly different when installing in a Python 3.9.6 virtualenv. Instead I get

 c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:256:24: error: implicit declaration of function 'lseek' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
              state->start = LSEEK(state->fd, 0, SEEK_CUR);
                             ^
      c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:14:17: note: expanded from macro 'LSEEK'
      #  define LSEEK lseek
                      ^
      c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:256:24: note: did you mean 'fseek'?
      c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:14:17: note: expanded from macro 'LSEEK'
      #  define LSEEK lseek
                      ^
      /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/stdio.h:162:6: note: 'fseek' declared here
      int      fseek(FILE *, long, int);
               ^
      c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:355:9: error: implicit declaration of function 'lseek' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
          if (LSEEK(state->fd, state->start, SEEK_SET) == -1)
              ^
      c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:14:17: note: expanded from macro 'LSEEK'
      #  define LSEEK lseek
                      ^
      c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:396:15: error: implicit declaration of function 'lseek' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
              ret = LSEEK(state->fd, offset - state->x.have, SEEK_CUR);
                    ^
      c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:14:17: note: expanded from macro 'LSEEK'
      #  define LSEEK lseek
                      ^
      c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:492:14: error: implicit declaration of function 'lseek' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
          offset = LSEEK(state->fd, 0, SEEK_CUR);
                   ^
      c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:14:17: note: expanded from macro 'LSEEK'
      #  define LSEEK lseek
                      ^
      4 errors generated.
      [numcodecs] command '/usr/bin/clang' failed with exit code 1

Probably we just need to update the vendored Blosc. In particular commit ( Blosc/c-blosc@406f596 ) is referenced as having fixed that issue. Could someone please open a PR updating the Blosc submodule in Numcodecs to the latest version?

It seems like it should be installing the latest version of blosc https://github.com/zarr-developers/numcodecs/blob/master/.gitmodules

more details - the blosc community seem to believe this is a numcodecs issue Blosc/c-blosc#316

It seems like it should be installing the latest version of blosc https://github.com/zarr-developers/numcodecs/blob/master/.gitmodules

Exactly can you please send a PR to update it?

I haven't worked with the numcodecs module so I could be wrong. My reading of the code is that the .gitmodules import uses the latest blosc package so I don't think the issue is the blosc package version. I believe the issue is to do with building blosc as a vendor submodule Blosc/c-blosc#316 (comment)

I'll have a look into alternative methods for building blosc. Maybe this is the answer long term? https://github.com/Blosc/python-blosc

I haven't worked with the numcodecs module so I could be wrong. My reading of the code is that the .gitmodules import uses the latest blosc package so I don't think the issue is the blosc package version. I believe the issue is to do with building blosc as a vendor submodule Blosc/c-blosc#316 (comment)

I don't think that is correct. We are using v1.18.1. The latest tag is v1.21.0. The fix alluded to earlier is v1.20.1.

Anyways if you have a moment please send a PR, would be happy to review 🙂

I'll have a look into alternative methods for building blosc. Maybe this is the answer long term? https://github.com/Blosc/python-blosc

Yeah this is something we have discussed. Agree it would be beneficial. Previously this was not possible as there were not Blosc wheels. That has since changed ( #262 ). More generally there are other compiled bits that would be nice to replace with 3rd party libraries ( #264 ). This would cutdown (maybe even eliminate) compilation on the Numcodecs side, which seems to be an endless source of pain for some users (as you have experienced first hand). There's PR ( #274 ) that starts down this path. Would be good if someone picked that up and pushed it across the finish line

I haven't worked with the numcodecs module so I could be wrong. My reading of the code is that the .gitmodules import uses the latest blosc package so I don't think the issue is the blosc package version. I believe the issue is to do with building blosc as a vendor submodule Blosc/c-blosc#316 (comment)

I don't think that is correct. We are using v1.18.1. The latest tag is v1.21.0. The fix alluded to earlier is v1.20.1.

Anyways if you have a moment please send a PR, would be happy to review 🙂

Ahh yes I understand now. Here's the PR #283

I now have numcodecs and zarr working with this PR #283

Navigation Menu

Repository navigation

Can't install numcodecs on OSX Big Sur #269

Minimal, reproducible code sample, a copy-pastable example if possible

Problem description

Version and installation information

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Can't install numcodecs on OSX Big Sur #269

Description

Minimal, reproducible code sample, a copy-pastable example if possible

Problem description

Version and installation information

Activity

jakirkham commented on Jan 22, 2021

adamjstewart commented on Mar 5, 2021

olly-writes-code commented on Jul 27, 2021

olly-writes-code commented on Jul 28, 2021

jakirkham commented on Jul 28, 2021

olly-writes-code commented on Jul 28, 2021

olly-writes-code commented on Jul 28, 2021

jakirkham commented on Jul 28, 2021

olly-writes-code commented on Jul 28, 2021

olly-writes-code commented on Jul 28, 2021

jakirkham commented on Jul 28, 2021

jakirkham commented on Jul 28, 2021

olly-writes-code commented on Jul 28, 2021

olly-writes-code commented on Jul 29, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions