Skip to content

Make GIL checks abi3 compatible - #673

Merged
evertlammerts merged 3 commits into
v2.0-cyanopterafrom
abi3-gil
Oct 16, 2026
Merged

evertlammerts merged 3 commits into
v2.0-cyanopterafrom
abi3-gil

Conversation

@evertlammerts

Copy link
Copy Markdown
Member

Stacked on #672. That PR made the bindings compile under the limited API everywhere except the GIL check. This one removes that last obstacle, switches the release wheels to the stable ABI, and makes CI test the promise. After it, each platform ships two wheels instead of four: one for 3.11 and one tagged cp312-abi3 that serves 3.12, 3.13, 3.14 and later releases without a new duckdb build.

The GIL check is the one thing the stable ABI cannot express: there is no function in it that answers "does this thread hold the GIL". We asked that question in three places. The Arrow stream callbacks and DuckDBPyResult::Close release the GIL only when the thread holds it, because ending a query waits for its tasks and a task inside a Python UDF needs the GIL to finish, while pyarrow calls get_schema without the GIL. And 49 debug asserts plus three runtime diagnostics checked the GIL state outright. For the two release sites I replaced the check with a scope guard, PyUtil::GilReleaseIfHeld. From 3.12 it calls PyThreadState_Swap(nullptr), which is in the stable ABI and does exactly what we need: it detaches the thread's state and releases the GIL when the thread holds it, and returns null and does nothing otherwise; the destructor swaps the state back. I probed that on 3.12, 3.13 and 3.14. On 3.11 the same call only swaps a process-wide pointer and is unsafe, so there the guard keeps today's check, which the 3.11 wheel can use because it is not abi3. The asserts go through two macros that are empty in stable-ABI builds, and the three runtime diagnostics (the two GilAssert calls and the pandas scan's "GIL was already held" error) became debug asserts: they guarded against our own mistakes, and only full-API builds can check them.

The build switch is small. wheel.py-api = "cp312" makes scikit-build-core tag the wheel abi3 on 3.12 and later and hand CMake the Development.SABIModule component; on 3.11 the same setting yields a version-specific wheel. CMake passes that component to find_package(Python) and, under the same condition nanobind applies in linked mode (CPython 3.12 or later, not free-threaded), builds nanobind-static-abi3 instead of nanobind-static and passes STABLE_ABI to the module. The choice is made once and used for both the module and our object libraries, because nanobind puts Py_LIMITED_API on its abi3 library as a public define and the two sides must compile against the same Python headers. Editable builds clear py-api through an override, so local builds keep the full API and the debug asserts.

CI changes shape. The seed jobs build cp312, the abi3 wheel, on every platform (it was cp314); the fan-out builds only cp311. The build action runs abi3audit --strict on every cp312 wheel, which checks the extension's imported symbols against the stable ABI list, something compiling with Py_LIMITED_API does not prove for symbols pulled in through inline functions or dependencies. A new job downloads each cp312 wheel and runs the suite on 3.13 and 3.14; cibuildwheel tests a wheel only on the interpreter that built it, so this job is what tests the promise. It skips the universal2 wheel, as CIBW_TEST_SKIP already does, because its slices are the arm64 and x86_64 wheels, tested as those. In PR runs it tests on Linux only, like the sanity job; the Windows and macOS legs run in full packaging runs.

Two things the review caught that are worth knowing: uv refuses to install a lockfile and a wheel in one command, and uv export needs the project's version, which the build backend takes from the DuckDB submodule and the git history. The test job therefore installs in two steps and checks out with the submodule and full history.

Verification on this machine (Linux aarch64): the limited-API syntax check over all 44 binding sources reports zero errors. The fast suite passes on a full-API debug build (6128 tests). A cp312 wheel built here is tagged cp312-abi3, passes abi3audit --strict, has Py_LIMITED_API in every one of its 44 binding compile commands and all 11 nanobind units, and passes the fast suite installed into fresh 3.12, 3.13 and 3.14 environments with the locked test dependencies, 5996 tests each. A 3.11 build is tagged cp311-cp311. The editable build has no Py_LIMITED_API in its compile commands, and its GIL-specific tests (the subinterpreter cases, the stream deadlock tests, fsspec) pass. What I could not test here and leave to CI: the Windows and macOS legs of the new test job, and abi3audit on the universal2 wheel.

Base automatically changed from abi3-wheels to v2.0-cyanoptera October 15, 2026 18:49
@evertlammerts
evertlammerts merged commit 4763c1e into v2.0-cyanoptera Oct 16, 2026
3 of 4 checks passed
@evertlammerts
evertlammerts deleted the abi3-gil branch October 16, 2026 06:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant