Repository navigation
[ENH]: musllinux wheels for Alpine #26438
Description
Given a wide usage of Alpine within Docker containers, it would be nice if matplotlib had pre-built wheels for musl-based distributions.
Apologies if this is a duplicate or already done, but I cannot find anything in the issue tracker and there is no musl-tagged matplotlib wheel on PyPI.
For contetxt: Alpine is used for Docker images because of its small footprint. It uses a different implementation of Libc, "musl", which necessitates recompilation step for projects that don't provide wheels with platform tag musllinux_x_y_<arch> (see PEP-656 for details). This recompilation step is a significant burden to many projects: each compilation requires identification of build-time dependencies, and adding them to the docker image one's using for testing.
cibuildwheel used in GH Actions has support for building these wheels since v2.2.0.
Are there any roadblocks for this to happen in matplotlib?
Activity
tacaswell commentedon Aug 3, 2023
I have a general concern about the ever escalating number of wheels that we produce (see https://pypackaging-native.github.io/meta-topics/user_expectations_wheels/ for longer comments on this). However, it looks like numpy is building an uploading musl wheels so there is no good reason not to enable it.
Two soap box style comments:
First, it appears that alpine comes with an up-to-date build of Matplotlib in its packaging system (https://pkgs.alpinelinux.org/packages?branch=edge&repo=community&name=py3-matplotlib) so using the native package manager may be a better option than using
pipas (tautologically) you are sure that there will be a compatible artifact.Second, if the reason you are using alpine is to keep the footprint small, then using pip where the wheels obligatorily vendor everything (so if you install both pytables and h5py you get two copies of libhdf5) seems to be at cross purposes. I would expect you to be better off using the native package manager or something like conda that properly tracks all of the dependencies.
BLD: stop skipping musl wheel builds
BLD: stop skipping musl wheel builds
MrMino commentedon Aug 4, 2023
Thanks!
The system package manager doesn't take into account Pip's dependency resolver, and mixing the two is generally considered bad practice. It's also difficult to do this right if Matplotlib is a transitive dependency of the app that is being installed.
For CI it doesn't matter, as the installation is going to be redone on every run. The unpacked wheel is not embedded in the image.
This is then sped up by Pip's cache, which would store the downloaded artiffacts in a docker volume. Since sdists are normally larger than wheels (for Matplotlib it's 38MB vs 7.5MB), the footprint is actually smaller when using wheels. The wheel resulting from an sdist build gets stored in the cache too, so unless the image already contains required binaries and the build step skips vendoring the shared objects into a wheel because of that - the footprint resulting from the build is always going to be larger.
With wheels, the image can get away with not installing
gcc,musl-dev, etc., so it's far easier to manage. Additionally, a standardpython:alpineimage can be used, without a need to build your own.tacaswell commentedon Aug 4, 2023
I very much agree; I am suggesting to use the alpine package manager for all of your dependencies.
I expect the size of installing from the native package manager to be smaller vs wheels (that is what I meant to compare not wheels vs sdist). I would also expect the native packages to always self-consistent with the other packages and to always be available for what ever version of Python you are getting from alpine. Further, given that when building the packages like numpy or pandas they are able to make better optimization choices. I also assume that you can cache those artifacts just like you cache wheels.
Once you have the base of everything you need (presumable everything but your application?) you can do a final
pip install --no-deps my_package.Sorry for being cranky about this, I have significant concerns about the social and technical sustainability of the pip/wheel packaging ecosystem. https://pypackaging-native.github.io/ is a good summary of the problems, but can also point you at (long) discussions in other forums.
MrMino commentedon Aug 4, 2023
You have nothing to apologize for, the issue is there and I very much agree with the people who point it out. I get why you are concerned. The maintenance burden of delivering bdists for the plethora of platforms is too big.
That said, generally, I still doubt that moving into system package managers is the way to go. For Alpine, the index keeps only one package version in its index per Alpine version. This is already a no-go for apps with larger dependency trees. Sooner or later you hit a set of version bounds that no Alpine index can satisfy cleanly.
Then there are issues with dependency pinning, requirement files translation, package naming being different on every distro, optional dependencies no longer being optional... there is a lot. In my view the cost of solving these issues is not balanced by the optimizations.
I don't believe that FOSS maintainers should bare the cost of adding wheels just so that I don't have to compile them myself or deal with system package managers. I'm asking because I know that building for
musllinuxis a ~relatively simple thing withcibuildwheel, and the benefits to the wider ecosystem far outweigh the costs.If I needed an
s390xwheel - I wouldn't even dare to think about asking about it here. I'd just make it myself. Alpine on the other hand... 🤷There is also - arguably - a profit side to this, in that there is now less people posting issues about build problems or bugs stemming from their own misconfiguration of the enviroment on Alpine. Whether that's worthy of creating a bdist is debatable, I'll admit.
Side note: I always wondered how many GWhs of electricity world-wide is conserved by releasing a pre-built of something like
matplotlibornumpy. Some of these libraries do heat my laptop up significantly when installing from source.sanmai-NL commentedon Oct 26, 2023
@MrMino, I recognize all your issues and points. Thanks for explaining!
@tacaswell Let me share a use case that explains how pressing this issue is.
I develop Alpine Linux container images on a modern macBook. A pretty common scenario I suppose.
This means the PEP 656
musllinuxplatform tag becomesmusllinux_1_1_aarch64. Matplotlib currently doesn't publish a wheel with such a tag, only forx86_64.Common, popular data science packages like
ultralyticsrequire a very modern version of some packages, likematplotlib(perhaps unwarranted, but normally this doesn't cause them issues). A compatible version isn't available from the Alpine Linux packages (edge).The solution would be to build from source a set of dependencies, determined through trial and error, that happen to not have such a wheel available. The waste this causes, e.g. in time and energy consumption, is significant.
tacaswell commentedon Oct 26, 2023
I'm a bit cranky this morning so I apologize in advance.
Things are always easier when you push the externalities of your choices off to someone else 🤣
Packaging is a Hard Problem, packaging systems that try to actually solve the problem (like any of the linux distributions and conda) are going to look "worse" but the problems are still there with pip/wheel, just swept under the rug to get you later 💣 .
If Alpine lacks up-to-date version of libraries, then the best solution is to get those packages updated / built.
The second best would be the people who are pushing the pip/wheel ecosystem as a thing to take responsibility for building a consistent set of packages for their packaging ecosystem. It is cheap and easy to add another platform tag, but doing so definitely feeds the user expectation that projects will build wheels with that tag and inflicts cost on the projects.
If apps and thier dependencies are over pinning then that should be solved by those projects.
As a side note,
ultralyticsappears to require newer than 3.3.0 (https://github.com/ultralytics/ultralytics/blob/a05edfbc27d74e6dce9d0f169036282042aadb97/requirements.txt#L5) which is from July 2020.sanmai-NL commentedon Oct 26, 2023
@tacaswell, on the other hand, a choice for
matplotlibis not mine but an externality of a direct choice for, e.g., YOLOv8 (ultralytics). That externality has the upside that your packagematplotlibgains popularity. If I'm correct, the only upside for you to even invest in offering it to the public.I have studied your references about the sorry state of Python packaging. With that I cannot but agree. I would expect that that those who host a package index, also build and host its contents. The same is done for e.g., OS package indices.
Indeed,
ultralyticsdoesn't depend onmatplotlib3.8.0, but the dependency resolution algorithm of my Python package manager did pin the required version to that, even when it found 3.7.1 installed as Alpine Linux package. This presumably is be due a constraint put by some other machine learning type of package.My trouble is in supporting data scientists in the Python ecosystem who use all these packages.