abi3 compatibility - #672

If we have one abi3 wheel per platform for CPython 3.12 and up, instead of one wheel per Python version, a new CPython release can use it without a new duckdb release. Now that the bindings are on nanobind, which supports the limited API from 3.12, the remaining obstacle is our own code. I checked every binding source against the Python 3.12 headers with Py_LIMITED_API=0x030C0000: 41 of 43 files failed, from five places. This PR removes all of them except the GIL check, which comes in its own PR. The build is unchanged; the wheels still use the full API. Behaviour stays the same except where noted below.

The five places and what replaced them:

The pyutil.hpp helpers used PyTuple_SET_ITEM, PyByteArray_GET_SIZE and PyMemoryView_GET_BUFFER. They now use PyTuple_SetItem with its result checked, PyByteArray_Size, and PyObject_GetBuffer with a release guard. A memoryview that skips bytes, like memoryview(b"abcdef")[::2], used to be stored as the wrong bytes (abc instead of ace); it now raises BufferError, since the buffer request asks for one contiguous run. Contiguous views, slices and casts convert as before.

The object column string scan read CPython's internal string layout, and decoded non-ASCII strings codepoint by codepoint. It now makes one PyUnicode_AsUTF8AndSize call per value and points DuckDB's string at the result without copying: for ASCII that is the string's own buffer, for anything else the UTF-8 copy CPython caches on the string object for its lifetime. That cache is the memory trade: a frame with many non-ASCII strings grows by their UTF-8 size after the first scan. A lone surrogate used to trip an INTERNAL error in the UTF-8 encoder that invalidated the database; it now raises InvalidInputException and the connection stays usable. The scan holds the GIL for each chunk of 2048 rows, like every other object column scan already did; the old path read the string internals without it.

The datetime C API has no stable ABI counterpart at all. Fields of Python datetime, date, time and timedelta values are now read as attributes, through interned names, and the object is held for the whole conversion because an attribute read runs Python code. Since a subclass's property overrides now apply, the fields are validated before they reach the engine: Date::FromDate already rejects an invalid date, and times go through Time::IsValidTime and raise InvalidInputException. In the other direction, DuckDB values become Python objects by calling the datetime, date and time classes with the fields packed into the 10, 4 or 6 bytes their pickle state uses, the form __reduce__ produces. The classes accept that single argument without parsing keyword arguments, which is roughly twice as fast as calling them with the integer fields. Every CPython from 2.7 to the main branch and the pure-Python datetime module accept it, but it is undocumented, and it does no range checking of its own, so we check: years outside 1 to 9999 and times outside Python's range fall back to the string, as they did before. No other stable-ABI library I found uses this form; nanobind, PyO3 and through it polars and cryptography call the classes with integers.

The pyarrow RecordBatchReader check (which makes a second scan of a reader raise instead of returning nothing) read tp_mro and tp_name. It now compares against pyarrow's real class with PyType_IsSubtype whenever pyarrow is in sys.modules. When it is not, it falls back to the type's own data: an extension type's PyType_GetName and __module__ come from tp_name, which a Python class cannot forge, and the bases walk uses Py_tp_bases, which a metaclass cannot replace, visiting each ancestor once. The three existing properties stay pinned: it works with pyarrow out of sys.modules, a Python class named and homed like the reader is not it, and a metaclass that replaces __mro__ does not confuse it. The new Appender code read tp_name for an error message; it uses nb::type_name.

Performance was measured with the benchmark suite under Callgrind as CI runs it, two sweeps per side, against the merge base, plus wall clock where Callgrind proved unreliable. Strings: the mixed ASCII and non-ASCII scan takes 46% fewer instructions, the ASCII-only scan 17% more. Values into DuckDB through values(): temporal values +1 to +2%, binary unchanged. Tuple building through PyTuple_SetItem: +3 to +4% on plain integer fetches. The one real regression is building Python temporal objects, where the stable ABI has nothing as cheap as the C API: per 1M values fetched, TIMESTAMP 365 ms against 267 ms, DATE 313 ms against 198 ms, TIME 318 ms against 185 ms (the integer-field form would have been 496, 386 and 364 ms). Parallel scans of object string columns no longer scale past 4 threads, because the conversion now happens under the GIL: on 2M mixed strings, 1 thread 44 ms against 109 ms before, 8 threads 31 ms against 18 ms, 16 threads 32 ms against 13 ms.

Known limitations, both pre-existing: a str() of a non-string object in a scanned column runs Python code, and if that code removes an earlier string from the frame, the pointer the scan already took dangles; and the loop converting a Python list crashes if Python code shrinks the list mid-conversion. The temporal attribute reads join the callbacks that could do either. A TIME from Arrow with 100 hours or more fails in the engine's two-digit formatting with an internal error, before and after.