Repository navigation
Dependency TODOs: enablements & clean-ups #944
Description
From looking at the list of third-party deps, we could still do better with:
- azure bindings
- unix (lib): Add Dependencies for Arrow's Azure Filesystem #1431
- unix (py): Build PyArrow with AzureFS Support pyarrow-feedstock#126
- win (lib): Rebuild for Azure Core 1.16.3; drop CUDA 11.8; enable azure on windows #1919 (+ backports back to v21)
- previously, Support azure bindings also on windows #1452 was blocked on [C++] Error when building Arrow-CPP w/ Azure Filesystem for Conda-Forge apache/arrow#41990 for a long time
- win (py): add azure on win; rebuild for py3.14t; fix substrait; drop CUDA 11.8 pyarrow-feedstock#169 (+ backports back to v21)
-
jemalloc(not possible?) - gcp: supported for a long time, but version updates currently stuck
- someone needs to go fix google-cloud-cpp-core v2.43.0 google-cloud-cpp-core-feedstock#46 etc.
- opentelemetry
- orc for windows
- Build for windows orc-feedstock#54
- enabled for libarrow (as of db46b2c)
- [Python] BUG: Reading ORC segfaults on windows (if TZDIR isn't set) apache/arrow#36026
- ORC-1684: [C++] Find tzdb without TZDIR when in conda-environments apache/orc#1882
- [15.x] Rebuild for orc 2.0.1; enable orc-on-windows also for pyarrow #1086
- don't fail tzdb availability check prematurely #1424
- enable orc-on-windows also for pyarrow pyarrow-feedstock#122
- GH-41755: [C++][ORC] Ensure setting detected ORC version apache/arrow#41767
- nlohmann_json (as of c548fc2)
- substrait (not yet in conda-forge)
- ucx (as of c548fc2)
Switch to shared builds for (if/once available):
- google-cloud-cpp (win; not yet available in cf)
- gflags (win; available, but apparently symbols aren't needed; clarified as of bda7682)
- glog (win; available, but apparently symbols aren't needed; clarified as of bda7682)
- grpc (win; available as of 1.55, which requires protobuf >=4.23; fixed as of 025b77a)
- thrift (needed fixes for CMake metadata; fully cleaned up as of 2af3d60)
Bindings:
- skyhook (needs librados)
- tensorflow (though we should use
libtensorflow, if at all, because it's a very heavy dependency to pull in)
Activity
xhochy commentedon Jan 27, 2023
We cannot unvendor jemalloc as the build is with special options. Unverdoring it would have an impact on the default allocator.
jakirkham commentedon Jan 27, 2023
Would it be possible to just build a variant of the
jemallocpackage with options needed?jakirkham commentedon Jan 27, 2023
Would not count on
ucxbeing supported on otherarchitecturesOSes. Would also like to understand why that is relevanth-vetinari commentedon Jan 27, 2023
I have a maximalist approach to packaging the bindings offered by a given library -- if the library offers integration for it, why shouldn't we package it?1
Users rely on conda-forge because a lot of these packages are really hard to build from source, so if a binding is not supported by us, it's effectively unusuable (even moreso if the distributed wheels also turn it off).
Footnotes
whether to break a package into separate outputs is a separate question ↩
jakirkham commentedon Jan 27, 2023
Sorry I should clarify, what I mean is why are other OSes relevant? Asking since
ucxlikely will remain Linux only (possibly macOS at some point and basically no chance of Windows support)h-vetinari commentedon Jan 27, 2023
Not relevant indeed, unless they become: a) available for the respective platform & b) are supported by arrow on that platform.
[-]Unvendor more libs[/-][+]Dependency TODOs: enablements & clean-ups[/+]RAMitchell commentedon Apr 25, 2025
This may be more or a problem for upstream arrow, but this list of dependencies seems huge. Do all users need every cloud integration?
raulcd commentedon Apr 25, 2025
There are some efforts to try and move filesystems outside of
libarrowon upstream Arrow but this is quite a big task with limited resources at the moment. Currently, when built, filesystems are part of the corelibarrowand can't be split.Some of the efforts already being done to achieve it:
The vision is to have separate modules
libarrow_s3.so,libarrow_gcs.so, etcetera. This is something currently on the project roadmap.h-vetinari commentedon Feb 25, 2026
@raulcd, is there a newer issue for this to track somewhere? AFAIU, the various cloud provider bindings have not moved out of
libarrowyet.raulcd commentedon Feb 26, 2026
As far as I know there's currently no issue tracking the filesystems split work. The only slightly related issue open I can find is:
h-vetinari commentedon Feb 26, 2026
Thanks for the response! I think such an issue should be opened. Could you perhaps open one? I can do it as well if you'd like, but I'm probably not going to do the best job at reflecting all the previous efforts towards this.
raulcd commentedon Feb 26, 2026
I agree with you that a new issue should be opened. I don't think the original issue should have been closed and we probably should have created a sub-issue for the FileSystem registry work but that's history. I've created the following one. I am unsure if this covers the whole story but we can expand with more details. The idea would be to create sub-issues and keep that one as a parent issue to track the different pieces.