Repository navigation
Make GIL checks abi3 compatible - #673
Merged
Merged
Conversation
evertlammerts
enabled auto-merge
October 15, 2026 20:52
evertlammerts
force-pushed
the
abi3-gil
branch
from
October 16, 2026 06:41
cb2762b to
5d3d871
Compare
evertlammerts
disabled auto-merge
October 16, 2026 06:41
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #672. That PR made the bindings compile under the limited API everywhere except the GIL check. This one removes that last obstacle, switches the release wheels to the stable ABI, and makes CI test the promise. After it, each platform ships two wheels instead of four: one for 3.11 and one tagged
cp312-abi3that serves 3.12, 3.13, 3.14 and later releases without a new duckdb build.The GIL check is the one thing the stable ABI cannot express: there is no function in it that answers "does this thread hold the GIL". We asked that question in three places. The Arrow stream callbacks and
DuckDBPyResult::Closerelease the GIL only when the thread holds it, because ending a query waits for its tasks and a task inside a Python UDF needs the GIL to finish, while pyarrow callsget_schemawithout the GIL. And 49 debug asserts plus three runtime diagnostics checked the GIL state outright. For the two release sites I replaced the check with a scope guard,PyUtil::GilReleaseIfHeld. From 3.12 it callsPyThreadState_Swap(nullptr), which is in the stable ABI and does exactly what we need: it detaches the thread's state and releases the GIL when the thread holds it, and returns null and does nothing otherwise; the destructor swaps the state back. I probed that on 3.12, 3.13 and 3.14. On 3.11 the same call only swaps a process-wide pointer and is unsafe, so there the guard keeps today's check, which the 3.11 wheel can use because it is not abi3. The asserts go through two macros that are empty in stable-ABI builds, and the three runtime diagnostics (the twoGilAssertcalls and the pandas scan's "GIL was already held" error) became debug asserts: they guarded against our own mistakes, and only full-API builds can check them.The build switch is small.
wheel.py-api = "cp312"makes scikit-build-core tag the wheel abi3 on 3.12 and later and hand CMake theDevelopment.SABIModulecomponent; on 3.11 the same setting yields a version-specific wheel. CMake passes that component tofind_package(Python)and, under the same condition nanobind applies in linked mode (CPython 3.12 or later, not free-threaded), buildsnanobind-static-abi3instead ofnanobind-staticand passesSTABLE_ABIto the module. The choice is made once and used for both the module and our object libraries, because nanobind putsPy_LIMITED_APIon its abi3 library as a public define and the two sides must compile against the same Python headers. Editable builds clearpy-apithrough an override, so local builds keep the full API and the debug asserts.CI changes shape. The seed jobs build cp312, the abi3 wheel, on every platform (it was cp314); the fan-out builds only cp311. The build action runs
abi3audit --stricton every cp312 wheel, which checks the extension's imported symbols against the stable ABI list, something compiling withPy_LIMITED_APIdoes not prove for symbols pulled in through inline functions or dependencies. A new job downloads each cp312 wheel and runs the suite on 3.13 and 3.14; cibuildwheel tests a wheel only on the interpreter that built it, so this job is what tests the promise. It skips the universal2 wheel, asCIBW_TEST_SKIPalready does, because its slices are the arm64 and x86_64 wheels, tested as those. In PR runs it tests on Linux only, like the sanity job; the Windows and macOS legs run in full packaging runs.Two things the review caught that are worth knowing: uv refuses to install a lockfile and a wheel in one command, and
uv exportneeds the project's version, which the build backend takes from the DuckDB submodule and the git history. The test job therefore installs in two steps and checks out with the submodule and full history.Verification on this machine (Linux aarch64): the limited-API syntax check over all 44 binding sources reports zero errors. The fast suite passes on a full-API debug build (6128 tests). A cp312 wheel built here is tagged
cp312-abi3, passesabi3audit --strict, hasPy_LIMITED_APIin every one of its 44 binding compile commands and all 11 nanobind units, and passes the fast suite installed into fresh 3.12, 3.13 and 3.14 environments with the locked test dependencies, 5996 tests each. A 3.11 build is taggedcp311-cp311. The editable build has noPy_LIMITED_APIin its compile commands, and its GIL-specific tests (the subinterpreter cases, the stream deadlock tests, fsspec) pass. What I could not test here and leave to CI: the Windows and macOS legs of the new test job, andabi3auditon the universal2 wheel.