Add support for duckdb 1.3.x, 1.4.x, and 1.5.x - #17

The skip condition only checks for duckdb_create_varint, but the test also calls duckdb_get_varint. If a DuckDB build exposes one symbol but not the other (or if the Python bindings end up conditionally defining only one wrapper), this test will still run and fail. Update the skipif condition to require both functions to be present (or skip if either is missing).

From the fake Slim Shady:

Good catch — updated to check both symbols. See test_ducklib.py:1332-1335.

Do we want to pin this to a version, like https://duckdb.org/docs/1.3/clients/c/api#duckdb_get_varint ?

From the fake Slim Shady:

Good call — pinned both varint docstrings to /docs/1.3/ since they were removed in 1.4. See ducklib.py:870 and ducklib.py:1266.

Can we tuck this away in some sort of a decorator or something? :)

From the fake Slim Shady:

Looked into this. A decorator above or below @cres doesn't work because @cres eagerly JIT-compiles via njit at decoration time — a decorator above it can't prevent @cres from firing (Python applies decorators bottom-up), and a decorator below it can't return anything useful that @cres won't choke on or silently miscompile.

What does work is a decorator that replaces @cres with conditional compilation:

def _optional_cres(symbol_name, sig):
    def decorator(func):
        if _has_symbol(symbol_name):
            return cres(sig)(func)
        return None
    return decorator

Then the @intrinsic stays unconditional (it just registers codegen, no symbol resolution until JIT compile time) and only the wrapper is gated:

@intrinsic
def _duckdb_create_varint(typingctx, varint_tup_ty):
    ...

@_optional_cres("duckdb_create_varint", intp(duckdb_varint_ty))
def duckdb_create_varint(val):
    return _duckdb_create_varint(val)

This removes the if blocks and extra indentation. Is this what you had in mind?

Yes. You can just recycle the name to get the signature, no need to provide it as the second argument, right?

Alternatively, I was also thinking about adding an optional keyword argument "skip" to maintain consistency in cres. You can probably then keep the "cres" name inact everywhere, and use old one (aliased _cres) within the decorator.

From the fake Slim Shady:

Here are three options:

A. Separate _optional_cres decorator — infers symbol name from func.__name__:

def _optional_cres(sig):
    def decorator(func):
        if _has_symbol(func.__name__):
            return cres(sig)(func)
        return None
    return decorator

@_optional_cres(intp(duckdb_varint_ty))
def duckdb_create_varint(val):
    return _duckdb_create_varint(val)

B. Shadow cres with skip kwarg — consistent name, but requires passing symbol name as a string (evaluated before func exists):

from numbox.utils.highlevel import cres as _cres

def cres(sig, skip=False, **kwargs):
    if skip:
        return lambda func: None
    return _cres(sig, **kwargs)

@cres(intp(duckdb_varint_ty), skip=not _has_symbol("duckdb_create_varint"))
def duckdb_create_varint(val):
    return _duckdb_create_varint(val)

C. Shadow cres with optional kwarg — consistent name + infers symbol name from func.__name__:

from numbox.utils.highlevel import cres as _cres

def cres(sig, optional=False, **kwargs):
    if not optional:
        return _cres(sig, **kwargs)
    def decorator(func):
        if _has_symbol(func.__name__):
            return _cres(sig, **kwargs)(func)
        return None
    return decorator

@cres(intp(duckdb_varint_ty), optional=True)
def duckdb_create_varint(val):
    return _duckdb_create_varint(val)

I'd lean C — keeps the @cres name everywhere, no string duplication, and optional=True is self-documenting. The only trade-off is shadowing the numbox import, but it's backwards compatible.

C looks good. Thanks. Couple of comments - instead of return None, how about returning a function that raises NotImplementedError when called? Also, a better name than 'optional'?

From the fake Slim Shady:

Implemented Option C with if_available. The cres wrapper at ducklib.py:33-41 shadows the numbox import and returns a NotImplementedError stub when the symbol is missing. Usage at ducklib.py:869 and ducklib.py:1265.

Note: the stub is a plain Python function, not JIT-compiled — NotImplementedError fires for Python-level calls; @njit calls would fail at numba compile time instead. Test skip updated to use _has_symbol directly.

For the name — other options considered were guard and symbol_check, but if_available reads most naturally: @cres(sig, if_available=True). Open to alternatives if you have a preference.

Just hasattr is sufficient, right?

From the fake Slim Shady:

Yes — verified locally that hasattr on a CDLL correctly catches the AttributeError from a failed dlsym. Simplified in ducklib.py:23.

Has this been tested for duckdb==1.5.1?

From the fake Slim Shady:

Yes — tested across duckdb 1.3.2, 1.4.0, and 1.5.1. Fork CI run with all three versions in the matrix: 44 jobs, all passing.

Thanks. On darwin arm64, I do

(venv) mikhailgoykhman@MacBookPro numbduck % nm venv/lib/python3.12/site-packages/_duckdb.cpython-312-darwin.so | grep duckdb_array_type_array_size

which on 1.4.0 returns

0000000000e61dd8 T _duckdb_array_type_array_size

but nothing on 1.5.0. The latter in particular manifests in the shared lib not adding any of the stuff we need into the llvm symbol table.

From the fake Slim Shady:

Confirmed — this is worse than hidden visibility. The C API symbols are completely absent from the duckdb 1.5.0 macOS ARM64 wheel.

We ran a diagnostic probe on macOS ARM64 (macos-15-arm64, Python 3.12, duckdb 1.5.0) checking multiple angles:

Probe source: numbduck_ci.yml on the macos-arm64-verify branch.

This looks like a DuckDB packaging regression — the C API object files weren't linked into the 1.5.0 macOS ARM64 Python extension. Only duckdb_adbc_init survived (likely a separate compilation unit). No workaround from our side since the code isn't in the binary at all.

Options:

1. File a DuckDB issue requesting C API symbols be exported in macOS ARM64 wheels
2. Cap the version range at <1.5 until it's fixed
3. Both — cap now, widen when DuckDB ships a fix

From the fake Slim Shady:

Did some deeper investigation. The C API symbols aren't just hidden — they were never compiled into the 1.5.0 macOS ARM64 wheel.

What we checked (probe 2, probe 3, probe 4):

• file reports Mach-O bundle (not dylib), built with STRIP_STATIC_SYMS
• nm -m, nm -a (all symbols including debug): only 1 duckdb_ symbol (duckdb_adbc_init)
• strings on the 38 MB binary: duckdb_open, duckdb_query, etc. not found as strings at all — the C API shim code wasn't linked in
• dlsym(RTLD_DEFAULT) after import _duckdb: nothing resolves
• otool -L: only links libc++ and libSystem — no bundled libduckdb.dylib

What IS there: The Python extension is a pybind11 C++ binding that talks directly to DuckDB's C++ internals, bypassing the C API entirely. The DuckDBPyConnection class wraps a ConnectionGuard containing shared_ptr<DuckDB> + unique_ptr<Connection>. pybind11's _pybind11_conduit_v1_ mechanism can extract the raw DuckDBPyConnection* from Python objects.

Options:

1. File a DuckDB issue requesting the C API symbols be included in macOS ARM64 wheels (they're in Linux wheels). Cleanest fix but depends on upstream.

2. Cap at <1.5 until upstream fixes it. Simple, but limits users.

3. Tiny C extension shim — ship a small .so with numbduck that:

   • Uses pybind11 conduit to extract the DuckDBPyConnection* from a Python connection object
   • Navigates from ConnectionGuard → shared_ptr<DuckDB> → raw duckdb::DuckDB*
   • Re-exports the C API functions (they're just thin wrappers around C++ calls) as proper exported symbols for numba/LLVM to resolve

   Trade-off: couples us to DuckDB's internal C++ layout, which could break across versions. Would need version-specific offsets or build against DuckDB headers.

4. Hybrid: options 1 + 2 — cap now, file upstream, widen when fixed. Pragmatic.

I'd lean option 4. The shim (option 3) is clever but fragile — we'd be reimplementing part of the C API against unstable C++ internals. Thoughts?

From the fake Slim Shady:

Follow-up: found a way in. The one exported symbol — duckdb_adbc_init — is a gateway to a full set of C function pointers via the ADBC (Arrow Database Connectivity) driver interface.

How it works: Calling duckdb_adbc_init fills an AdbcDriver struct with ~42 C function pointers — DatabaseNew, ConnectionInit, StatementExecuteQuery, etc. These are real, callable C functions inside the 38 MB binary. No symbol lookup needed — we get raw addresses.

Proof of concept — full round-trip on macOS ARM64, duckdb 1.5.0:

DatabaseNew → DatabaseSetOption(":memory:") → DatabaseInit     ✅
ConnectionNew → ConnectionInit                                  ✅
StatementNew → SetSqlQuery("SELECT 42 AS answer") → Execute     ✅
Arrow get_schema → 2 columns (answer:int32, pi:decimal128)      ✅
Arrow get_next → batch.length=1, answer=42                      ✅
StatementRelease → ConnectionRelease → DatabaseRelease          ✅

All via pure C function pointers extracted from the ADBC driver, callable from numba JIT.

What this means for numbduck: Rather than needing the DuckDB C API symbols, we could use ADBC as the backend on platforms where C API symbols are missing. Trade-offs:

Probe source: numbduck_ci.yml on macos-arm64-verify.

This could be the path forward — ADBC backend for macOS ARM64, DuckDB C API where available. Thoughts?