Expose Py_SETREF and Py_XSETREF in the limited API

See python/cpython#99574 for the last time this came up.

Victor told me on Discord that he originally planned to re-propose this in 2023 but simply forgot.

I noticed these macros are missing while reviewing a NumPy PR that is part of the effort to make it possible to build NumPy under the limited API. The contributor replaced Py_SETREF calls with the "deadly" incorrect version that SETREF exists to avoid.

https://github.com/numpy/numpy/pull/32256/changes#diff-22d279e0a87948dce92784ff43557067e20213f6f4c06669a0a6315f80478f85L157-R161

I'm +1 on adding Py_SETREF() and Py_XSETREF(). They help to write correct code managing reference counting.

There are documented at:

Py_CLEAR() is already part of the limited C API.

Py_CLEAR(), Py_SETREF() and Py_XSETREF() are only macros, they are not implemented as function, so they would not be added to the stable ABI in practice.

@encukou

As for other limited API macros, to guide non-C/C++ reimplementations the docs should include an implementation (so, a C23-only version with typeof is fine -- users of older C can just use the macro itself).

With that, +1 :)

Also, let's formally add all of them to stable_abi.toml, so they get the "Part of the Limited API" line in docs. That is, add the following to limited API 3.2+:

Py_INCREF
Py_XINCREF
Py_DECREF
Py_XDECREF
Py_CLEAR

Hopefully that's not controversial.

As for other limited API macros, to guide non-C/C++ reimplementations the docs should include an implementation (so, a C23-only version with typeof is fine -- users of older C can just use the macro itself).

These macros are special and I'm not sure that it's a good idea to provide similar functions doing the same. There were multiple attempts to implement such functions and they all failed so far.

@davidhewitt

For PyO3/Rust, I think we'd probably implement Py_SETREF as:

#[inline]
pub unsafe fn Py_SETREF(op: *mut *mut PyObject, new: *mut PyObject) {
    let old = *op;
    *op = new;
    Py_DECREF(old);   // Py_XDECREF for Py_XSETREF
}

This is a function, so no need to worry about type safety or use __typeof__ or worry about C aliasing rules: the operands are PyObject pointers. Rust doesn't have the type punning concerns that make the C API want to abstract all the type punning into the macro.

Another neat aspect of this is that using *mut *mut instead of &mut *mut for the first argument avoids exactly the Rust-side re-entrancy concern that the SETREF macro exists in C to avoid.

So anyway, I personally think it's fine to just document the C23-ism macros and don't provide function variants at all. The functions just force callers to do the type punning that the macro exists partially to avoid.

Ping @davidhewitt in case you're interested.

To clarify, for the docs I want something along the lines of:

Py_SETREF(dst, src) expands roughly to the following, except it only evaluates each argument once and it doesn't need the C23 keyword typeof:
do {
    typeof(dst) old = dst;
    dst = src;
    Py_DECREF(old);
} while (0)

That's enough code to show how to use it and how to implement it in a non-C/C++ language. (I'm not too happy about the do/while(0), but it's easier to keep it than explain in prose.)

For the stable ABI, Py_DECREF is going to be a function call anyway, so why wouldn't we make a real function Py_SetRef(PyObject **obj, PyObject *new_ref)?

I wouldn't expect anyone to actually use it, apart from the fact that someone is asking for it. It's such a simple function with language-specific complexities that anyone who cares enough to care about the performance is going to want to reimplement it anyway.

why wouldn't we make a real function Py_SetRef(PyObject **obj, PyObject *new_ref)?

That's indeed the systematic way of doing things, but seems hypercorrect. FWIW, the draft guidelines have an exception for “functionality that can be accomplished trivially, but perhaps tediously, without macros”.

Either way, we should expose Py_SETREF for API compatibility when switching to stable ABI.

The only risk I see is that our implementation of these macros is now a constraint on all changes we might make in the future - we have to ensure that the 3.16 version of the macro continues to work against our PyObject * for ~all time, since we'll have people executing the old code against modern objects, as well as new versions of the macros having to either keep a copy of the older implementation in the headers (for limited API targeting earlier versions). Basically just more work for us, rather than a real technical concern, but let's not gloss over the maintenance burden or forget to actually do it.

why wouldn't we make a real function Py_SetRef(PyObject **obj, PyObject *new_ref)?

I tried and failed multiple times to implement Py_SETREF() as a function call. There is a subtle issue involving type punning (and strict aliasing). When I modified Py_SETREF() to blindly cast the arguments to PyObject*, the compiler "miscompiled" code causing crashes. If the argument is a PyListObject* but the macro writes into PyObject*, the compiler may not understand that the PyListObject* variable is modified as well.

That's why the macro now uses typeof(), or has to use memcpy() when typeof() is not available (MSVC).

@erlend-aasland

ping @erlend-aasland @serhiy-storchaka

has to use memcpy() when typeof() is not available (MSVC).

We should have __typeof__ in all the versions we care about (it can be detected with _MSC_FULL_VER>=193933428 if needed - typeof is only available when compiling for a specific standard, but __typeof__ is the same and is available without that specific target).

@serhiy-storchaka

ping @serhiy-storchaka

@serhiy-storchaka

ping @serhiy-storchaka

👍

We should have __typeof__ in all the versions we care about

For C - yes, but for C++ see python/cpython#157649.

We should have __typeof__ in all the versions we care about

For C - yes, but for C++ see python/cpython#157649.

Oof... fun... (aside: you know about godbolt.org, yes? That should make it much easier for you to dig into these kinds of issues)

you know about godbolt.org

Sure I do and in the issue linked above I've posted a few godbolt links ...

Okay, good (it was the "I don't have a setup for ..." comment that made me wonder)

@vstinner

@vstinner is it now safe to add Py_SETREF to the 3.16 limited API? Given python/cpython#157649 and fallout there. Unless that's already happened and I missed it.

@ngoldbaum

@ngoldbaum: Oh, I just saw that all C API WG members voted in favor of adding this function to the limited C API! Do you want to propose a PR for that? I can do it if you prefer.

@vstinner is it now safe to add Py_SETREF to the 3.16 limited API? Given python/cpython#157649 and fallout there. Unless that's already happened and I missed it.

The C++ issue affected not only Py_SETREF(), but also Py_CLEAR() (which is already part of the limited C API). python/cpython@05b6061 introduced the regression in the main branch. Previously, Py_CLEAR() and Py_SETREF() were not tested in C++.

Py_CLEAR(), Py_SETREF() and Py_XSETREF() have been fixed in C++ and I made sure that these macros are now well tested on C and C++ in the different flavors of the C API and with different compiler flags.

Writing a correct implementation for these macros (which only evaluates arguments once) is tedious. Hopefully, the macros are now well tested.

I can do it but I'll be busy for the next week for a Quansight all-hands meeting. So why don't you do it if you have time and feel free to ping me for a review pass.

I can do it but I'll be busy for the next week for a Quansight all-hands meeting. So why don't you do it if you have time and feel free to ping me for a review pass.

Ok, I went ahead and I created PR python/cpython#158843. I close this issue.

Navigation Menu

Repository navigation

Expose Py_SETREF and Py_XSETREF in the limited API #110

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Expose Py_SETREF and Py_XSETREF in the limited API #110

Description

Activity

vstinner commented on Aug 12, 2026

encukou commented on Aug 13, 2026

vstinner commented on Aug 13, 2026

ngoldbaum commented on Aug 13, 2026

encukou commented on Aug 14, 2026

zooba commented on Aug 17, 2026

encukou commented on Aug 18, 2026

zooba commented on Aug 18, 2026

vstinner commented on Aug 31, 2026

encukou commented on Aug 31, 2026

zooba commented on Aug 31, 2026

encukou commented on Sep 7, 2026

encukou commented on Sep 14, 2026

serhiy-storchaka commented on Sep 18, 2026

chris-eibl commented on Sep 18, 2026

zooba commented on Sep 21, 2026

chris-eibl commented on Sep 21, 2026

zooba commented on Sep 21, 2026

ngoldbaum commented on Oct 3, 2026

vstinner commented on Oct 3, 2026

ngoldbaum commented on Oct 3, 2026

vstinner commented on Oct 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions