Repository navigation
Python wheel should export DuckDB C API symbols (or provide a path to them) #404
Description
What happens?
Prior to 1.4.1, the DuckDB Python wheel exported all C API symbols on every platform. Starting with 1.4.1, duckdb-python#81 restricted exports to only PyInit__duckdb and duckdb_adbc_init via linker flags in CMakeLists.txt L83-L110.
On macOS, -exported_symbol is a hard allowlist — all C API symbols are gone. On Linux, --export-dynamic-symbol is additive, so all symbols remain visible as a side effect. This means the C API works on Linux today but is broken on macOS — and could break on Linux too if a version script is added in the future.
| Version | macOS C API symbols | Linux C API symbols |
|---|---|---|
| 1.3.x | ~6,300 ✅ | ~6,300 ✅ |
| 1.4.0 | ~6,300 ✅ | ~5,900 ✅ |
| 1.4.1+ | 0 ❌ | ~5,900 ✅ (by accident) |
| 1.5.x | 0 ❌ | ~6,300 ✅ (by accident) |
This is a breaking change for projects that load the C API from the Python wheel.
Use case
numbduck wraps 160 DuckDB C API functions for use inside numba @njit compiled code. It loads the shared library from the installed duckdb Python package and resolves C API symbols at runtime. This worked on all platforms through duckdb 1.4.0, but breaks on macOS starting with 1.4.1.
The standalone libduckdb downloads have all symbols, but requiring a separate install (brew install duckdb) alongside pip install duckdb is a friction point — users expect the Python package to be self-contained.
Possible solutions
- Export all
duckdb_*C API symbols from the Python wheel on all platforms. The symbols are already compiled into the binary — they're just hidden by the linker flags. - Add a
ctypes-friendly entry point — e.g. a function that returns a pointer to a struct of C API function pointers (similar to howduckdb_adbc_initworks for ADBC). - Bundle
libduckdb.dylib/.soalongside the Python extension in the wheel, soctypes.CDLLcan load it directly. - Document the limitation — if the C API is intentionally not part of the Python wheel's public interface, document that C API consumers should use the standalone
libduckdbdownloads.
Option 1 is the smallest change — just remove the linker flags. The PR #81 commit message says these are "the only two [symbols] that are ever needed," but that assumes all consumers go through Python or ADBC.
Environment
- duckdb 1.4.1 through 1.5.1 (macOS broken; Linux works by accident)
- macOS ARM64 and x86_64
- Verified with
nm -gD/nm -gon installed wheel.sofiles
Activity
nelson2005 commentedon Apr 12, 2026
Update: we shipped a workaround in Goykhman/numbduck#17 —
find_duckdb_shared_lib()now falls back to a standalonelibduckdb.dylib(Homebrew or manual download) on macOS when the Python wheel lacks C API symbols. This gets macOS users unblocked but adds an extra install step (brew install duckdb) that isn't needed on any other platform.The underlying issue remains: the C API symbols are compiled into the wheel binary on all platforms — they're just hidden by linker flags on macOS. Any project loading DuckDB's C API from the Python package hits this on macOS, and Linux is one version script away from the same breakage.
evertlammerts commentedon Apr 12, 2026
Hey @nelson2005, thanks for the heads up. I tried to clean up the exported symbols last year because from the two symbols that we knew had to be exported, only one actually was.
I'll put this on my list and try to get this in before 1.5.3.
nelson2005 commentedon Apr 13, 2026
Thanks, much appreciated!
nelson2005 commentedon Apr 15, 2026
@evertlammerts do you think there'd be any interest in an article about jitted duckdb? It's a new capability, me and goykhman worked on it because we do a lot of numerical work using numba and there was no way to call duckdb from jitted code.
It's a little specific, but this technique permits calling to/from duckdb without involving the Python interpreter and the GIL, creating some significant benefits in various dimensions over plain-python and pyarrow alternative implementations. I hadn't seen anyone implementing this technique for duckdb, though it's quite possible I missed it
Export all symbols (#445)
evertlammerts commentedon May 8, 2026
@nelson2005, I forwarded your question to our devrels.
I just landed #445. Do you have a way to confirm that this actually fixes your issue?
nelson2005 commentedon May 10, 2026
Awesome, thanks! @goykhman is traveling, should be back Saturday May 16 or so and can check then.
Goykhman commentedon May 16, 2026
Thanks @evertlammerts and @nelson2005.
To reproduce the issue, I first verified against the current 1.5.2 release that the DuckDB Python wheel on macOS does not export the DuckDB C API symbols needed by numbduck.
I then checked out v1.5-variegata (commit 5e63f4d, May 15) and built duckdb-python from source. I confirmed that with this build on my mac the required DuckDB C API symbols are successfully resolved from the compiled shared library during the numbduck tests.
For completeness, visibility is also confirmed with nm.
evertlammerts commentedon May 18, 2026
Perfect, thanks @Goykhman and @nelson2005! v1.5.3 (will be released this week, likely) will have all symbols available again.