Repository navigation
Releases
7.2.1
Bug fixes
-
Fixed a regression in 7.2.0 where a subclass of
MultiDictProxyorCIMultiDictProxy
that converted its argument in its own__init__()raised
TypeErrorbefore that__init__()ran; such a subclass got an
empty multidict until its__init__()set the proxied one
-- by @asvetlov.
7.2.0
7.2.0 drops Python 3.10, fixes a crash in a proxy built without
__init__() and a MemoryError from a huge __length_hint__(),
and is otherwise a performance release for the C extension: lookups,
adds, updates and deletes each needed a few percent fewer instructions.
Bug fixes
-
Stopped a huge
__length_hint__()from raisingMemoryErrorin
extend(),update()
andmerge(); the C extension capped the room
it reserved for the hint at the largest table the constructor allocates
up front -- by @asvetlov.Related issues and pull requests on GitHub:
#1767. -
Fixed a segmentation fault when using a
MultiDictProxy
orCIMultiDictProxybuilt by__new__()without
__init__();__new__()started requiring the proxied multidict, the
same argument as the constructor -- by @asvetlov.Related issues and pull requests on GitHub:
#1782.
Features
-
Sped up reading a multidict in the C extension.
md[key],
get()andgetone()
became up to 7% faster by returning the lookup's result by value instead
of through a pointer to the caller's local, which had made
-fstack-protector-strongadd a stack canary to every call. On the GIL
build aget()miss became 5% faster and
key in mdinlined its hash table lookup. On the free-threaded build
getall()stopped taking the multidict's lock
for a multidict without repeated keys, the way
get()already avoided it.len(md.keys())
and iterating over the keys of a multidict that holds a key more than
once became 4% to 13% faster -- by @asvetlov.Related issues and pull requests on GitHub:
#1775, #1776, #1781, #1800, #1817. -
Sped up building and updating a multidict in the C extension. On the GIL
build, constructing a multidict from a sequence of pairs needed 8% to 10%
fewer instructions andupdate()3.5% to 4.5%
fewer. On the free-threaded build, building a
MultiDictbecame up to 14% faster, since a table that
is both full and unable to hold the new key in its compact form was grown
and moved in one rebuild, andadd(),
md[key] = valuefor a new key and
setdefault()needed 2% to 3% fewer
instructions, about 1% fewer on the GIL build. Updating a multidict with
existing keys needed 1.5% to 2.9% fewer instructions, while adding a new
key needed about 0.5% more on the GIL build -- by @asvetlov.Related issues and pull requests on GitHub:
#1763, #1774, #1780, #1785, #1802. -
Sped up replacing and removing items in the C extension.
add(),getall(),
md[key] = valueanddel md[key]stopped computing the identity
of a key that, or whoseistrcanonical form, already
serves as one, asmd[key]did, and checked a
CIMultiDictkey holding uppercase letters only once.
On the GIL builddel md[key]needed 4% to 5% fewer instructions,
md[key] = valueabout 2% fewer andpop()
1% fewer. Replacing a key that had duplicates needed 0.8% to 1.7% fewer
instructions, andclear()became 6% faster on
the free-threaded build -- by @asvetlov.Related issues and pull requests on GitHub:
#1769, #1770, #1779, #1796, #1809.
Removals and backward incompatible breaking changes
-
Dropped support for Python 3.10 as it has reached end of life
-- by @asvetlov.Related issues and pull requests on GitHub:
#1777.
Improved documentation
-
Refreshed the instruction-count tables in the benchmarking reference
against the current tree -- by @asvetlov.
Packaging updates and notes for downstreams
-
Added the CPython 3.15 trove classifier and started testing the
GIL and free-threaded 3.15 builds in CI -- by @asvetlov.Related issues and pull requests on GitHub:
#1766.
Contributor-facing changes
-
Split the release publishing and signing across four parallel jobs, so
that each one finishes well within the lifetime of its short-lived PyPI
upload token -- by @asvetlov. -
Changed the
clear()benchmarks to clear a thousand prebuilt
multidicts of 0 to 99 items each, copied outside the measured region,
since the copy inside the timed loop and the single small table it
measured made the result swing by up to 19% between runs of the same
commit -- by @asvetlov.Related issues and pull requests on GitHub:
#1784. -
Pinned Hypothesis in the MyPy pre-commit hooks and dropped a
type: ignorecomment from the Hypothesis tests that became unused
once Hypothesis annotatedrun_state_machine_as_test(), which failed
the MyPy lint job -- by @asvetlov.Related issues and pull requests on GitHub:
#1820. -
Fixed a test of a huge
__length_hint__()failing on 32-bit builds,
where its2**40hint did not fit aPy_ssize_t
-- by @asvetlov.Related issues and pull requests on GitHub:
#1821.
Miscellaneous internal changes
-
Split the C extension's helpers per table kind, so that the lookup
loops ofpop(),popall(), item replacement andupdate(), the
table rebuild and the deferred delete of an entry call the helper for
their own kind instead of testing it again, and moved the choice between
overwriting and dropping each entry of a repeated key into the per-kind
loops ofupdate(). Folded helpers with a single caller into that
caller: choosing the kind of a new table, reading a compact entry's
identity and hash, filling a new entry, replacing a key's values,
deleting a key and looking up the first value of a key that needs a
computed identity. Merged the two per-kind checks for a deleted entry
into one, and removed the separate watched copies of delete, pop and
replace, folding the add that follows a missed replace into the replace
itself -- by @asvetlov.Related issues and pull requests on GitHub:
#1771, #1772, #1773, #1786, #1788,
#1789, #1790, #1792, #1804, #1805,
#1810, #1811. -
Removed duplicated code from the C extension: the
(key, default)
methods shared their argument parsing and missing-key result, and one
macro each replaced the copies of the lock-free compact entry match on
the free-threaded build and of the per-kind dispatch of a single key
lookup. Turned the watcher record, flush and pending check helpers and
_growth_rate()into macros, so that a multidict without watchers
skipped evaluating the event arguments, renamed the lowercase
function-like macros to upper case, and dropped the leading underscore
from the watcher functions that the macros call from other headers
-- by @asvetlov.Related issues and pull requests on GitHub:
#1793, #1794, #1795, #1806, #1807,
#1808, #1812. -
Adjusted how the C extension spends GCC's inlining budget: marked most
checks that lead to a raised exception as unlikely, so the compiler
moves the error paths out of the hot code, kept the event name lookup of
a failing watcher callback report and the cleanup of a dropped watcher
event log out of line, and dropped theinlinehint from the helpers
that compute a table's size and that rebuild, build or free a table's
keys -- by @asvetlov.Related issues and pull requests on GitHub:
#1783, #1791, #1801, #1815. -
Moved the C extension's
repr()of multidicts, proxies and views and
its equality comparison out of the hash table header, each into a header
of its own -- by @asvetlov.
7.1.0
7.1.0 is a correctness and housekeeping release. It changes what
keys() yields, fixes a long list of crashes and
inconsistencies that Python code run in the middle of an operation could
trigger, stores the common all-str multidicts in half the space, and
deprecates multidict.upstr.
Behaviour changes. keys() and iteration over a
multidict yield each key once, as spelled in its first item, so
len(d.keys()) counts distinct keys; items(),
values() and len(d) still see every item. A
str subclass's own lower() is no longer called to compute the
case-insensitive identity of a CIMultiDict key.
Robustness. Finalizers, __eq__(), __repr__(), lower() and
argument iterators that read or mutate the multidict in the middle of
__init__(), update(), merge(), del d[key],
popall(), popitem(),
to_dict() or repr() no longer crash the C
extension, lose or duplicate pairs, or make the two backends disagree. Two
data races against lock-free readers on the free-threaded build are gone
too.
Memory. A MultiDict whose keys are all exact
str, and a CIMultiDict whose keys are all exact
istr, are stored in a compact table without a separate
identity and hash per entry, which cuts their size by about 45%.
Bug fixes
-
Fixed building the C extension on platforms where
charis unsigned,
such as ARM, PowerPC, s390x and RISC-V, and with GCC 9, which flags
CPython's own headers from 3.12 on; both failed on a sign conversion
warning since-Wconversiontook effect. GCC 9 builds the extension
without-Wconversion-- by @asvetlov. -
Fixed
getversion()returning only the low 32 bits of
the version on Windows and on 32-bit platforms, where two different
versions could compare equal, and fixed a race in the pure-Python
implementation on GIL builds that could give two multidicts mutated at
the same time from different threads the same version
-- by @asvetlov. -
Fixed
inon an items view truncating the length of a non-tuple,
non-list operand to 32 bits in the C extension, which let an object
reporting2**32 + 2items be indexed as a pair and raise
TypeErrorinstead of returningFalse-- by @asvetlov.Related issues and pull requests on GitHub:
#1611. -
Fixed adding to a full table of the C extension, which compacted it in
place whenever it held deleted entries and so regained only as many
slots as were deleted: a delete-then-add loop on a full table rebuilt
it on every add. The table was resized by the number of live entries
instead, as the pure-Python implementation did, which also let it
shrink after deletions. Fixedextend(),
update()andmerge()
reserving room by comparing table sizes, which ignored the room taken by
deleted entries; the reservation checked the free room instead, in both
implementations -- by @asvetlov. -
Fixed calling
__init__()again on a live multidict losing and leaking
whatever a finalizer of an old value added to it in the C extension, and
miscountinglen()in the pure-Python implementation, by installing
the new contents and size before releasing the old pairs
-- by @asvetlov. -
Fixed two data races on free-threaded builds against lock-free readers:
copy()and__init__()from a multidict of the same kind read the
source's reader count while its readers updated it, and__init__()
on a live multidict rewrote the case-insensitivity flag its lookups
read -- by @asvetlov.Related issues and pull requests on GitHub:
#1621. -
Fixed
MultiDict.__init__()called on aCIMultiDictmaking it
case-sensitive in the C extension, andCIMultiDict.__init__()
rejecting aMultiDictthere, by taking the mode from the
instance rather than from the class whose__init__()was called, as
the pure-Python implementation already did -- by @asvetlov.Related issues and pull requests on GitHub:
#1622. -
Fixed the C extension dropping the existing items when a multidict was
re-initialized from itself, or from a proxy of itself, together with keyword
arguments, as ind.__init__(d, key=value); the pure-Python
implementation was not affected -- by @asvetlov.Related issues and pull requests on GitHub:
#1666. -
Fixed a use-after-free in
popitem()when
building the result ran Python code that mutated the multidict: a
strsubclass key's__str__()on
CIMultiDict, or on Python 3.10 and 3.11 a garbage
collection's finalizers, by removing the pair first, and fixed the
pure-Pythonpopitem()miscountinglen()in the same case
-- by @asvetlov.Related issues and pull requests on GitHub:
#1625. -
Fixed
del md[key]andpopall()also
removing, andpopall()also returning, the pairs a finalizer of a
removed key or value added while the call was still running, by
releasing the removed pairs only once every match is gone, and fixed the
pure-Python implementation raisingIndexErroror miscounting
len()in the same case, formd[key] = valuetoo
-- by @asvetlov.Related issues and pull requests on GitHub:
#1627. -
Fixed a crash when Python code run between the items of
update()(the argument's own iterator, a key's
lower()or a finalizer) read the multidict, where the pure-Python
implementation returnedNonefor the pairs the call was about to
remove, and fixedupdate()andmerge()
overwriting or losing pairs when that code changed the multidict. An
update from another multidict raisedRuntimeErrorwhen a key's
lower()mutated the source -- by @asvetlov. -
Fixed a use-after-free in
to_dict()on Python
3.10 and 3.11, where allocating a value list can run a garbage collection
whose finalizers mutate the multidict, by refusing that mutation with
RuntimeErroras it is refused from a key's__hash__()
-- by @asvetlov.Related issues and pull requests on GitHub:
#1630. -
Fixed
repr()of a multidict reading freed memory on the GIL build
when a key's or value's__repr__called anextend(),update()
ormerge()that grew the table and then failed, by bumping the version
on every resize -- by @asvetlov.Related issues and pull requests on GitHub:
#1658. -
Fixed
CIMultiDictreturning a key added as a
strsubclass with the spelling of the subclass's own
__str__()instead of the key's, and stopped the C extension from
replacing such a key with itsistrwhile reading it, which
could run the key's finalizer inside the read. Documented that
CIMultiDictconverts keys toistr
-- by @asvetlov.Related issues and pull requests on GitHub:
#1640. -
Fixed the
istrkeys that
extend(),update(),
merge()and theMultiDict
constructor took from aCIMultiDict: they carried the
key's own spelling as their case-insensitive identity instead of its
lower-cased form, so aCIMultiDictkeyed by one of them
found it under no spelling of the key. Changed the pure-Python
implementation to hand over those keys asistrtoo, as
the C extension did -- by @asvetlov.Related issues and pull requests on GitHub:
#1647. -
Fixed a lock-free lookup on the free-threaded build reading the hash
through a key of a compact table that a concurrent delete had already
freed, by taking a reference to such a key first unless it is the very
object being looked up -- by @asvetlov.Related issues and pull requests on GitHub:
#1653. -
Changed
CIMultiDictProxy.__init__()called without an argument to
report the sameTypeErrormessage asMultiDictProxyand the
pure-Python implementation -- by @asvetlov.Related issues and pull requests on GitHub:
#1677.
Features
-
Made
MultiDictin the C extension store a mapping whose
keys are all exactstrin half the space, keeping no separate
identity or hash per entry, and did the same for a
CIMultiDictwhose keys are all exact
istr, taking each identity from the key's canonical
form, which cut its size by about 45%. Copying such a mapping became
about 55% cheaper in instructions andclear()
about 27%; any other key moves the mapping to the full layout, and
clear()lets it start compact again. A lookup
also stopped taking a reference to the identity of an exactstr
oristrkey, which maded[key]andkey in dup
to 7% faster on the GIL build and up to 20% cheaper in instructions on the
free-threaded one -- by @asvetlov.Related issues and pull requests on GitHub:
#1644, #1648, #1651. -
Sped up
getall(),del d[key], replacing a
key withd[key] = valueandupdate()by 5
to 14 percent, and up to 26 percent in the pure-Python implementation,
when no key repeats, which each hash table started to track; the
tracking madeadd()up to 10 percent slower,
and building a small multidict from items up to 18 percent, both up to
16 percent in the pure-Python implementation -- by [@asvetlov](https://github.com/asvetlo...
7.0.0
7.0.0 is a major release. It makes istr final, adds a
public C API for third-party extensions, and makes the C extension
substantially faster across the board.
Breaking change. istr can no longer be subclassed,
on either backend. Code that subclassed it has to wrap or convert instead.
The C extension also started rejecting an argument passed both
positionally and by name with TypeError, where it used to silently
misbehave.
Public C and Cython API. Other C extensions and Cython modules can now
create, read and mutate multidicts through a capsule, without going through
the Python-level API. The capsule comes with multidict_capi.h, a
cimport-able multidict/__init__.pxd and
multidict.get_include(). The API includes MultiDict_ForEach() and
a watchers API modeled on CPython's dict watchers. See the C API and
Cython API references.
Performance. On CodSpeed's GIL-build benchmarks against 6.9.1, 82
C-extension benchmarks got faster and none got slower. Rough figures:
CIMultiDictkeyed by plainstr: construction,
add(),extend(),update()and item assignment got 2 to 3.8
times faster, and lookups about 2 times faster.CIMultiDictkeyed byistr, and
case-sensitiveMultiDict: lookups, insertion and
deletion got 10% to 70% faster.getall()and iterating
items()got about 35% faster, and the view set
operations 10% to 85% faster.popitem()on a whole mapping went from
quadratic to linear time.- On the free-threaded build, measured in instructions against 6.9.1,
lookups got 24% to 33% cheaper onMultiDictand 4 to 5
times cheaper onCIMultiDictwith lowercasestr
keys, and item assignment 15% to 19% cheaper. Construction and deletion on
MultiDictcost 2% to 7% more. With several threads
mutating at once, ad[key] = valuegot about four times faster.
The pure-Python backend is unchanged in speed.
Robustness. This release fixed several crashes and leaks in the C
extension, including use-after-free bugs when a key's lower() or a
value's __eq__ mutated a multidict, a crash at interpreter shutdown,
and table and reference leaks on the free-threaded build.
Bug fixes
-
Fixed a segmentation fault in the C extension when a finalizer let the
same multidict be resized whiledel md[key]or
popall()was still scanning the table, and a
spuriousRuntimeErrorfromgetall()
when a garbage collection triggered while building its result mutated
the multidict -- by @asvetlov. -
Fixed a reference leak of the key and value in the C extension when
growing the hash table fails withMemoryErrorduring
add(),update(),
merge(),setdefault()
or item assignment -- by @asvetlov.Related issues and pull requests on GitHub:
#1507. -
Fixed a data race on the free-threaded build of the C extension where a
MultiDict's own version counter was read by
getversion()and written on every mutation with a plain,
non-atomic load and store. Both sides switched to a relaxed atomic
load/store, matching the pattern already used for the object's item
count -- by @asvetlov.Related issues and pull requests on GitHub:
#1514. -
Fixed the C extension discarding a
MemoryErroror
KeyboardInterruptraised by a positional argument's__len__
while the size of an update was estimated.extend(),update(),
merge()and subclass construction started propagating it, matching the
pure-Python backend; only theTypeErrorfrom an unusable
__length_hint__stayed ignored -- by @asvetlov.Related issues and pull requests on GitHub:
#1517. -
Ensured that canonical keys in pure-Python
MultiDictand
CIMultiDictare converted to exactstrinstances
without invokingobject.__str__()overrides, matching the C extension --
by @agustin18.Related issues and pull requests on GitHub:
#1535. -
Fixed the argument counts reported by the C extension when a method is
called with too many positional arguments; methods with an optional
second argument claimed to take exactly one, and methods requiring two
claimed to take a range. A method called with none of its two required
arguments now reports both as missing -- by @asvetlov.Related issues and pull requests on GitHub:
#1545. -
Fixed the C extension accepting an argument passed both positionally and
by name instead of raisingTypeError.md.add("a", key="k", value="v")stored a wrong value, andmd.get("a", "b", key="c")
silently ignored the surplus arguments -- by @asvetlov.Related issues and pull requests on GitHub:
#1545. -
Fixed a memory leak in the free-threaded build of the C extension.
clear()hands its hash table to a drain that
frees it once no lock-free reader is in flight, and a drain that finds
one leaves the table for the next drain to free. Deallocation ran no
drain of its own, so a multidict dropped after such a clear, with no
operation in between, never freed that table nor released the references
its entries still held. Builds with the GIL enabled emit byte-identical
code and are unaffected -- by @asvetlov. -
Fixed a crash during interpreter shutdown in the C extension: a
multidict started keeping a reference to its own module, so the module
state it reads while being torn down could not be freed first
-- by @asvetlov.Related issues and pull requests on GitHub:
#1563. -
Fixed retired hash tables lingering in the free-threaded build of the C
extension. A drain that finds a lock-free reader in flight puts the tables
it holds back for the next drain to run, and the reader it saw could
already have looked, so nothing was left to free them: they stayed until
the multidict was next used, and if it was dropped instead, the references
their entries held were never released. Builds with the GIL enabled emit
byte-identical code and are unaffected -- by @asvetlov.Related issues and pull requests on GitHub:
#1565. -
Fixed reference cycles through a multidict going uncollected in the
free-threaded build of the C extension. A hash table waiting to be freed
still owns the references of the entries left in it, and those were not
reported to the garbage collector, which then read the values as reachable
from outside the cycle and kept it alive. Builds with the GIL enabled emit
byte-identical code and are unaffected -- by @asvetlov.Related issues and pull requests on GitHub:
#1566. -
Fixed
extend(),update()andmerge()of the C extension
raisingSystemErrorwhen given an object whose
__length_hint__()reported close tosys.maxsizetogether
with keyword arguments, and overflowing the table size computation for
such a hint; a hint too large to reserve for is ignored, as
list.extend()does -- by @asvetlov.Related issues and pull requests on GitHub:
#1577. -
Fixed a use-after-free in the C extension's
MultiDict.__eq__
when a value's__eq__mutated either multidict; comparing
exactstrvalues also became faster -- by @asvetlov.Related issues and pull requests on GitHub:
#1579. -
Fixed a use-after-free in the C extension when a
strsubclass
key'slower()mutated the source while a
CIMultiDictwas built from, extended, updated or
merged with adictor a multidict of the other case
sensitivity. The extra references this takes are limited to the keys
whoselower()can run Python code, so other updates do not pay for
them -- by @asvetlov.
Features
-
Added a public C and Cython API for third-party extensions. Its
MultiDict_ForEach()family hands a visitor each entry's key, value,
identity (the canonical key the mapping looks entries up by, lower-cased
on aCIMultiDict) and the identity's hash
-- by @Vizonex and @asvetlov.Related issues and pull requests on GitHub:
#1478. -
Changed the C extension to track the entries that
getall(), item assignment,
update(),merge()
andto_dict()have visited in a bitmap private
to the call, instead of temporarily marking entry hashes in the table
itself. Concurrent readers on free-threaded builds no longer have to
tolerate marked entries. Item assignment,
to_dict()and lookups of keys with many values
got faster, whileupdate()with many repeated
keys got somewhat slower
-- by @asvetlov.Related issues and pull requests on GitHub:
#1505. -
Replaced the C extension's per-key finder object with a single
callback-driven walk in a new_multilib/walk.hheader. The entries a
walk has already reported were tracked by the walk itself rather than
by every caller, and the walk kept its position in registers instead of
reloading it from a struct on each step, so
getall()and the items and keys view set
operations got faster on keys with many values -- by @asvetlov.Related issues and pull requests on GitHub:
#1522. -
Sped up
CIMultiDictoperations keyed by a plainstrin the C
extension. An all-ASCII key was lowered ...
6.9.1
Bug fixes
-
Fixed the C extension reading freed memory on free-threaded builds when a
list handed to :py:meth:~multidict.MultiDict.update,
:py:meth:~multidict.MultiDict.extend, :py:meth:~multidict.MultiDict.merge
or the :py:class:~multidict.MultiDictand :py:class:~multidict.CIMultiDict
constructors, a[key, value]item inside any iterable handed to them, or a
list tested withinagainst :py:meth:~multidict.MultiDict.items, is
changed by another thread; a call that catches the list shrinking under it
now raises :py:exc:RuntimeError-- by :user:rodrigobnogueira.Related issues and pull requests on GitHub:
#1437. -
Fixed a data race on the free-threaded build where a retired hash table's
reader count used relaxed atomics, letting a lock-freeget()/getone()/
__getitem__()read race a concurrent free of that table. The reader-exit
decrement and the drain's free check now use release/acquire ordering
instead -- by :user:asvetlov.Related issues and pull requests on GitHub:
#1481. -
Fixed a free-threaded build bug where two threads calling
update(),
merge(), or__setitem__()on the same key at the same time could lose
the key entirely instead of just racing on which value wins. A decref of the
replaced value could transiently suspend the writer's critical section,
letting a second writer for the same key observe the first writer's
in-progress entry as absent and, once both settled, mistake it for a stale
duplicate and delete it. Every such decref is now deferred until the writer
has released its critical section, so the window can no longer open.
setdefault()had an unrelated instance of the same blind spot (it could
insert a duplicate rather than recognizing an in-flight key), fixed alongside
it -- by :user:asvetlov.Related issues and pull requests on GitHub:
#1483. -
Fixed a free-threaded build bug where
getall()and theitems()/
keys()/values()equality path could raiseKeyErroror report a
present, never-deleted key as missing. A concurrentupdate()/extend()/
__setitem__()call can have its critical section transiently suspended
(a decref triggering a blocking allocator call) while an entry is marked as
part of its own bookkeeping; a reader landing in that window used to treat
the mark as "not found" instead of "still there, in flight" -- by
:user:asvetlov.Related issues and pull requests on GitHub:
#1484. -
Fixed a reference leak in the C extension where
operand | md.items()
andmd.items() - operandleaked one key and one value reference per
element ofoperand, letting a large operand grow memory without bound
(:gh:GHSA-54p9-h82j-f925 <aio-libs/multidict/security/advisories/GHSA-54p9-h82j-f925>)
-- by :user:asvetlov.The issue was reported by :user:
waydeshi.Related commits on GitHub:
:commit:350b4a0. -
Fixed a segmentation fault on the standard (non-free-threaded) C extension
build when a value type's__del__released the GIL (for example by
callingtime.sleep()) whileupdate(),merge(),__setitem__(),
__delitem__(),pop(),popone(), orpopall()was dropping a
replaced or removed value.Py_BEGIN_CRITICAL_SECTIONcompiles to a no-op
on this build, so nothing else was stopping a second thread from mutating the
very sameMultiDictconcurrently once the GIL was released mid-mutation.
Every such decref is now deferred until the mutation has fully finished, the
same technique already used to close the analogous free-threaded-build race
-- by :user:asvetlov.Related issues and pull requests on GitHub:
#1489. -
Fixed the C extension reading freed memory while iterating a
:py:class:~multidict.CIMultiDictwhose keys are plain :py:class:str.
Converting such a key to :py:class:~multidict.istrcould run Python code
(a :py:class:strsubclass's__str__or__del__) or, on free-threaded
builds, suspend the iterator's critical section, after which the iterator read
the entry again even though a concurrent mutation could already have freed it.
As part of the fix, :py:meth:~multidict.MultiDict.copyand re-initializing
from another multidict now assign a new version in the C extension instead of
reusing the source's, matching the pure Python implementation
-- by :user:asvetlov.Related issues and pull requests on GitHub:
#1496. -
Fixed a use-after-free on the free-threaded build where a lock-free
get(),[]orincould read a hash table that a concurrent
resize had just retired and another reader was freeing
-- by :user:asvetlov.Related issues and pull requests on GitHub:
#1497.
Contributor-facing changes
-
Removed a redundant
includeand a duplicatedexcludeline from
MANIFEST.in; sdist contents are unchanged -- by :user:asvetlov.Related issues and pull requests on GitHub:
#1478. -
Added
.claudeignorefile -- by :user:asvetlovRelated issues and pull requests on GitHub:
#1479. -
Scaled up the pure-Python
pop(),popitem(),__delitem__(),
add()and item-insertion benchmarks to do more work per measurement.
Repeated CodSpeed runs on the same commit showed these particular
benchmarks flagged as dominated by syscalls, understating their real
cost and adding noise to the reported values; a larger working set
amortizes that overhead -- by :user:asvetlov.Related issues and pull requests on GitHub:
#1485. -
Replaced deprecated instrumentation codspeed mode with simulation -- by :user:
asvetlovRelated issues and pull requests on GitHub:
#1493. -
Reorganized the mutating benchmarks (item insertion,
update(),
add()of the same key,pop(),popitem(),clear(),
__delitem__()and__setitem__()) to copy a fresh multidict and
apply the operation in a loop, like theadd()andextend()
benchmarks already do. The insertion,update()andclear()
benchmarks previously mutated a single multidict shared across
rounds, so only the first round measured the intended operation; the
rest did a single copy per round, letting per-round overhead dominate
-- by :user:asvetlov.Related issues and pull requests on GitHub:
#1494. -
The
repr()and view inequality benchmarks were updated to repeat their
operation in a loop, like the other benchmarks, and the CodSpeed benchmark
job was moved to Python 3.14 -- by :user:asvetlov.Related issues and pull requests on GitHub:
#1498. -
Dropped
-Ifrom the AddressSanitizer test command inAGENTS.md
and in the CI job. It implies-E, which made Python ignore
PYTHONMALLOC=malloc, so small hash tables were still served from
pymallocarenas where use-after-free went undetected
-- by :user:asvetlov.Related issues and pull requests on GitHub:
#1499. -
The CI/CD workflow was updated to stop superseded runs of the same pull request
when a new commit is pushed; runs onmaster, release branches, tags,
the merge queue, and the daily schedule are never interrupted
-- by :user:asvetlov.Related issues and pull requests on GitHub:
#1500. -
The release job was changed to upload distributions and their signatures
to the GitHub Release one file at a time, skipping assets that were
already attached and retrying after a pause, so that a parallel upload
burst no longer tripped the GitHub secondary rate limit
-- by :user:asvetlov.Related issues and pull requests on GitHub:
#1503.
Miscellaneous internal changes
-
Corrected several comments in the free-threaded C extension that
attributed critical-section suspension to a blockingPyMem_Malloc()
call; allocation alone never suspends an acquired critical section, and
the real risk at those sites is a decref running
a finalizer or weakref callback. Also dropped a retry loop in
md_clone_from_ht()that guarded against the same, non-existent
allocation-triggered suspension -- by :user:asvetlov.Related issues and pull requests on GitHub:
#1486. -
Changed the free-threaded build's deferred decref buffer, used by
update()and__setitem__(), to a chain of fixed-size blocks
with a large inline first block instead of a small inline array that
was reallocated on growth -- by :user:asvetlov.Related issues and pull requests on GitHub:
#1501.
6.9.0
Bug fixes
-
Protected
repr()ofMultiDict,MultiDictProxy, and their views
in the C extension with a critical section, avoiding data races on the
free-threaded build of CPython -- by :user:asvetlov.Related issues and pull requests on GitHub:
#1431. -
Guarded
repr()ofMultiDictProxyin the C extension and of
KeysViewin both the C extension and the pure-Python
implementation against infinite recursion on self-referential
containers, matching the existing guard onMultiDict,
ItemsView, andValuesView-- by :user:asvetlov.Related issues and pull requests on GitHub:
#1431. -
Protected
MultiDict.update(),.extend(),.merge(),.clear(),
.copy(), and theMultiDict/CIMultiDictconstructors in the C
extension with a critical section, using the two-object form when a
second multidict, multidict proxy, or plaindictinstance is
involved, avoiding data races and a segmentation fault on the
free-threaded build of CPython..clear()now also publishes the
empty table before releasing any entry's references, so a concurrent
caller can never observe a partially-cleared multidict even if releasing
a value runs arbitrary Python code that suspends the held critical
section -- by :user:asvetlov.Related issues and pull requests on GitHub:
#1433. -
Protected
MultiDict.add(),__setitem__/__delitem__,
get()/getone()/__getitem__,__contains__,getall(),
setdefault(),pop()/popone()/popall()/popitem(),
__eq__, iteration, and the&/|/-/^/in/
isdisjoint()operations on.keys()and.items()views in the
C extension with a critical section, avoiding data races and
use-after-free crashes on the free-threaded build of CPython
-- by :user:asvetlov.Related issues and pull requests on GitHub:
#1438. -
Fixed a data race on the free-threaded build where
MultiDictObject.used
was written non-atomically whilelen()read it with a relaxed atomic
load -- by :user:asvetlov.Related issues and pull requests on GitHub:
#1441. -
Fixed several pre-existing use-after-free races on the free-threaded build:
_md_resize()
andmd_clone_from_ht()could allocate a new hash table, have their
critical section transiently suspended during that allocation, and then use
a keys-table pointer or size a concurrent operation had already invalidated;
_md_del_at()(plaindel/pop()) could likewise leaveselfin
an inconsistent state across a suspending decref -- by :user:asvetlov.Related issues and pull requests on GitHub:
#1441. -
Fixed a data race on the free-threaded build where
md->keyswas
published with a plain store while lock-free readers loaded it
atomically, and a false-negative race inget()/__contains__
where a hash temporarily marked by a concurrent replace could make a
present key look absent -- by :user:asvetlov.Related issues and pull requests on GitHub:
#1441. -
Fixed several more pre-existing use-after-free and data-corruption races on
the free-threaded build, this time in the__setitem__()/update()/
extend()/merge()replace path: a decref suspending the critical
section mid-replace could leave a stale table pointer in use, misplace a
temporarily-marked entry during a concurrent resize, or let one thread's
duplicate-tracking mark get overwritten by an unrelated key's entry sharing
the same hash bucket -- by :user:asvetlov.Related issues and pull requests on GitHub:
#1441. -
Fixed an unbounded memory leak on the free-threaded build where a retired
hash table could sit onmd->retiredfor the rest of the object's
lifetime under sustained concurrent read traffic instead of being freed
-- by :user:asvetlov.Related issues and pull requests on GitHub:
#1443. -
Fixed a data race on the free-threaded build of the C extension where
state->global_version, the version counter shared by every
multidict instance and used to derivegetversion(), was bumped with
a plain increment instead of an atomic op. Two threads mutating
different instances at the same time could step on each other's update
and hand out a duplicate (or non-monotonic) version number
-- by :user:asvetlov.Related issues and pull requests on GitHub:
#1444. -
Fixed a data race on the free-threaded build of the C extension where
getall()/popall()andupdate()/extend()/merge()
temporarily marked and unmarked a matching entry's hash with a plain,
non-atomic store, while a lock-free__contains__/get()on another
thread could load that same field with an atomic operation and no lock at
all -- by :user:asvetlov.Related issues and pull requests on GitHub:
#1448. -
Fixed a data race in the pure-Python
MultiDict/CIMultiDict
fallback where concurrent mutating calls (add(),__setitem__,
__delitem__,setdefault(),pop()/popone(),
popall(),popitem(),update(),extend(),merge(),
clear(), and re-__init__) could corrupt the shared hash table,
or silently drop a concurrent write, on a regular, GIL-enabled
interpreter. Pure-Python bytecode is not atomic under the GIL, so two
threads could interleave mid hash-table insert or deletion, leaving
the index table pointing at stale or out-of-range entries and causing
an infinite probe loop, anAttributeError, or a lost mutation.
EachMultiDict/CIMultiDictinstance now holds its own lock for
the duration of these operations, with two-object operations locking
both instances in a fixed order to avoid deadlock. The C extension
already serializes these operations with a critical section and was
not affected
-- by :user:asvetlov.Related issues and pull requests on GitHub:
#1449. -
Fixed a data race on the free-threaded build of the C extension where
to_dict()cleared every entry's temporary "seen" mark with a plain,
non-atomic store, while a lock-free__contains__/get()on another
thread could load that same field with an atomic operation and no lock at
all -- by :user:asvetlov.Related issues and pull requests on GitHub:
#1450. -
Fixed a data race on the free-threaded build of the C extension where
update()/extend()/merge()readself's module state without
holdingself's lock, while a concurrent__init__()on the same,
already-published multidict could rewrite that same field under lock.
Rather than routing every read through a lock-free lookup, the field is
now written only once, at object construction, and never touched again by
__init__(), so it is safe to read unlocked anywhere -- by
:user:asvetlov.Related issues and pull requests on GitHub:
#1452. -
Fixed
items()set algebra (&,|,-,in,isdisjoint()) so it no
longer corrupts or hangs when a compared value's__eq__()re-enters the
same :class:~multidict.MultiDict, for example by callinggetall()on
it. The comparison now runs only after the internal hash-chain walk has
been fully materialized and restored, so the callback can no longer
observe entries the walk still has marked -- by :user:asvetlov.Related issues and pull requests on GitHub:
#1453. -
Fixed a data race on the free-threaded build of the C extension where
popall()/popone()/__delitem__/add()rewrote a hash table
index slot with a plain, non-atomic store, while a lock-free
get()/getone()/__getitem__/__contains__on another thread
walked that same index array with a plain, non-atomic load and no lock at
all -- by :user:asvetlov.Related issues and pull requests on GitHub:
#1454. -
Stopped folding the case of :class:
~multidict.istrkeys in the
case-sensitive :class:~multidict.MultiDict, so both the C and the pure
Python implementations now keep such a key exactly as given
-- by :user:youdie006.Related issues and pull requests on GitHub:
#1457. -
Fixed the C extension's
update()(andmerge(), which shares the same
code path) silently failing to invalidate an in-progresskeys()/
items()/values()iterator when the call only overwrote an
already-present key's value in place. Adding a new key already invalidated
iterators correctly; only the in-place overwrite branch was missing the
version bump -- by :user:asvetlov.Related issues and pull requests on GitHub:
#1458. -
Fixed the pure-Python
MultiDict/CIMultiDictiterator guard so that
mutating the mapping before the firstnext()call on anitems(),
keys(), orvalues()iterator (or oniter(md)) now reliably raises
RuntimeError, matching the C extension. Previously a mutation that
happened before the iterator was ever advanced, such asclear()or a
del/popone()that removed the only remaining entry, could make the
iterator silently raiseStopIterationinstead -- by :user:asvetlov.Related issues and pull requests on GitHub:
#1459. -
Fixed a use-after-free race on the free-threaded build of the C extension
where a retired hash table could be freed while a lock-free
get()/getone()/__getitem__/__contains__reader on another
thread was still walking it: the coarse "no reader in flight" gate
reaching zero did not reliably mean every such reader had also reached
its own per-table exit, so a table whose own reader count is still
nonzero is now deferred for a later retry instead of freed
-- by :user:asvetlov.Related issues and pull requests on GitHub:
#1461.
Features
- Added a :meth:
~multidict.MultiDict.to_dictmethod returning a plain
:class:dictthat maps eve...
6.8.0
Bug fixes
-
A segmentation fault that could be triggered when getting an item is now fixed
-- by :user:Vizonex.Related issues and pull requests on GitHub:
#1310. -
Fixed reference leak in iterators, views and
istr
-- by :user:Vizonex.Related issues and pull requests on GitHub:
#1311. -
Fixed the pure-Python :class:
~multidict.MultiDictconstructor and
:py:meth:~multidict.MultiDict.extend,
:py:meth:~multidict.MultiDict.update, and
:py:meth:~multidict.MultiDict.mergemethods over-allocating their
internal hash table when called with both a positional argument and
keyword arguments, because keyword arguments were counted twice in the
size estimate -- by :user:aiolibsbot.Related issues and pull requests on GitHub:
#1338. -
Fixed
__repr__of :class:~multidict.MultiDict,
:class:~multidict.CIMultiDict, their proxies, and the keys/items views
producing invalid output when keys contained quote characters --
keys are now formatted with :func:reprso the result is a valid Python
string literal -- by :user:aiolibsbot.Related issues and pull requests on GitHub:
#1342. -
Fixed a segfault when calling :py:meth:
~multidict.MultiDict.addwith only one of its two required arguments supplied by keyword, e.g.d.add(key="k"). Extra keyword arguments passed to the lookup and removal methods are also now rejected with :exc:TypeErrorinstead of being silently ignored.-- by :user:
devdanzinRelated issues and pull requests on GitHub:
#1376. -
Fixed a segfault when constructing a multidict iterator type directly, e.g.
type(iter(md.keys())).__new__(...). Such an iterator had a NULL internal pointer thatnext()dereferenced. The iterator types now forbid direct instantiation, the same way the view types were fixed in #1163.-- by :user:
devdanzinRelated issues and pull requests on GitHub:
#1377. -
Fixed a segfault when using a :py:class:
~multidict.MultiDictor :py:class:~multidict.CIMultiDictcreated via__new__without calling__init__(for example a subclass that does not callsuper().__init__()). The internal state was left as NULL pointers that the first method call dereferenced.tp_newnow initializes the object to a valid empty mapping.-- by :user:
devdanzinRelated issues and pull requests on GitHub:
#1378. -
Fixed two crashes in the C extension caused by holding a raw pointer into a hash table across an operation that could reshape it. Updating a multidict from itself (e.g.
d.extend(d)) freed the very table being iterated -- a use-after-free;extend(self)now doubles the contents andupdate(self)/merge(self)are no-ops. Building a multidict from a list of pairs whose case-insensitive key.lower()shrinks that list read past the end of the list; the length is now re-checked on every iteration.-- by :user:
devdanzinRelated issues and pull requests on GitHub:
#1379. -
Fixed three reference/resource leaks on error paths in the C extension: the items-view
__contains__leaked the first element of a candidate pair when reading the second one raised;__repr__leaked itsPyUnicodeWriterwhen the multidict was mutated during iteration; and the internal iteration helper leaked the identity reference when key materialization failed under low memory.-- by :user:
devdanzinRelated issues and pull requests on GitHub:
#1381. -
Stopped several feature-detection fallbacks in the C extension from swallowing every exception. When probing an argument (
arg.items()/arg.keys()) or measuring it (len(other)) fails, the code now clears only the expected :exc:TypeError/:exc:AttributeErrorand lets everything else -- notably :exc:MemoryErrorand :exc:KeyboardInterrupt-- propagate, matching the already-correct sites elsewhere in the module.-- by :user:
devdanzinRelated issues and pull requests on GitHub:
#1382. -
Fixed a segmentation fault when extending a multidict with itself
-- by :user:cananooand :user:asvetlov.Related issues and pull requests on GitHub:
#1398. -
Fixed a memory leak when
MultiDictandCIMultiDictinstances are
initialized more than once -- by :user:asvetlov.Related issues and pull requests on GitHub:
#1412. -
Fixed a reference leak in items-view set operations
-- by :user:asvetlov.Related issues and pull requests on GitHub:
#1413. -
Fixed a memory leak when destroying C-extension multidict instances
and proxies -- by :user:asvetlov.Related issues and pull requests on GitHub:
#1415. -
Fixed a reference leak from repeated
__init__calls on
MultiDictProxyandCIMultiDictProxyinstances -- by :user:asvetlov.Related issues and pull requests on GitHub:
#1419. -
Fixed missing decref for :data:
Noneon error in :meth:~multidict.MultiDict.setdefault
-- by :user:asvetlovRelated issues and pull requests on GitHub:
#1421.
Features
-
Added :py:func:
reversedsupport to the keys, values, and items views of
:class:~multidict.MultiDict, :class:~multidict.CIMultiDict, and their
proxies in both the C-extension and pure-Python implementations
-- by :user:aiolibsbot.Related issues and pull requests on GitHub:
#448. -
Optimized key identity comparison for C Extension.
Now it uses fast path that is equal to :c:func:
PyUnicode_Equalfrom
Python 3.14+ but without redundant type checks. It gives ~15% speed-up on benchmarks with many key comparisons.-- by :user:
asvetlovRelated issues and pull requests on GitHub:
#1406. -
Changed both the pure-Python and C implementations to mark a temporarily
removed hash table entry by setting the high bit of its hash instead of
overwriting it with a sentinel value, so restoring the entry no longer
recomputes the hash; :meth:~multidict.MultiDict.getallis faster ~10% now in C version
-- by :user:asvetlovRelated issues and pull requests on GitHub:
#1426.
Removals and backward incompatible breaking changes
-
Dropped support for Python 3.9 as it has reached end of life.
Related issues and pull requests on GitHub:
#1316. -
Dropped support for free-threaded Python 3.13 -- by :user:
ngoldbaum.Related issues and pull requests on GitHub:
#1326.
Improved documentation
-
Fixed
CIMultiDictProxydocumentation to state it inherits from
MultiDictProxy(notMultiDict), and fixed missing word in
istrsection -- by :user:veeceey.Related issues and pull requests on GitHub:
#1298. -
Clarified that
istrpreserves the original casing and
compares as a regularstr; case-insensitive matching is
handled byCIMultiDict-- by :user:gyanu2507.Related issues and pull requests on GitHub:
#1397. -
Fixed broken RST markup in
items()docstrings
-- by :user:veeceey.Related issues and pull requests on GitHub:
#1299.
Packaging updates and notes for downstreams
-
Dropped support for free-threaded Python 3.13 -- by :user:
ngoldbaum.Related issues and pull requests on GitHub:
#1326. -
Wheels for iOS and Android (CPython 3.13+) are now published on
release-tag builds, so downstreams on those platforms no longer need a
local compiler to installmultidict; these wheels are build-verified
only, since running the test suite on mobile needs a simulator or an
emulator -- by :user:aiolibsbot.Related issues and pull requests on GitHub:
#1337. -
Added support for building and shipping riscv64 wheels
-- by :user:justeph.Related issues and pull requests on GitHub:
#1293. -
The
setuptoolsbuild dependency lower bound has been restored to be
>= 47after an incorrect automated increase in :pr:1315
-- by :user:aiolibsbot.Related issues and pull requests on GitHub:
#1400.
Contributor-facing changes
-
Added support for collecting code coverage of isolated multidict tests
-- by :user:Vizonex.Related issues and pull requests on GitHub:
#1314. -
Switched to
mirrors-clang-formatand enabled clang-format inpre-commit.ci
-- by :user:bdraco.Related issues and pull requests on GitHub:
#1318. -
Added a release-tag-gated CI job that cross-compiles the C extension
for iOS and Android throughcibuildwheel-- by :user:aiolibsbot.Related issues and pull requests on GitHub:
#1337. -
Added an
AGENTS.mdorientation file at the repository root, covering the
pull request template, theCHANGES/news fragment conventions, the
draft-PR / human-review workflow, and the dual pure-Python and C-extension
parity rule, so LLM contributors land changes that match project style
-- by :user:bdraco.Related issues and pull requests on GitHub:
#1339. -
Added
init,installable,instantiation,parametrization,
parametrized,postfix, andunparseableto the docs spelling list
so news fragments and docs can use these words without failing the spell
check build -- by :user:pctablet505.Related issues and pull requests on GitHub:
#1343. -
Documented three common agent failure modes in
:file:AGENTS.md: the docs spell check
(make doc-spelling) catches unknown words in news fragments
before CI does; coverage runs over the test tree too, so
unreachable defensiveraiseguards and one-sided cleanup
branches in tests will surface as uncovered on the codecov
patch report; and branch creation is restricted on
aio-libs/multidict, so PRs must be pushed from a fork
-- by :user:bdraco. -
Trimmed the CI test matrix to drop redundant
Py_DEBUGjobs on
macOS and Wi...
6.7.1
Bug fixes
-
Fixed slow memory leak caused by identity by adding
Py_DECREFto identity value before leavingmd_pop_oneon success
-- by :user:Vizonex.Related issues and pull requests on GitHub:
#1284.
6.7.0
Contributor-facing changes
-
Updated tests and added CI for CPython 3.14 -- by :user:
kumaraditya303.Related issues and pull requests on GitHub:
#1235.
6.6.4
Bug fixes
-
Fixed
MutliDict&CIMultiDictmemory leak when deleting values or clearing them
-- by :user:VizonexRelated issues and pull requests on GitHub:
#1233.
Contributor-facing changes
-
The type preciseness coverage report generated by
MyPy <https://mypy-lang.org>__ is now uploaded toCoveralls <https://coveralls.io/github/aio-libs/multidict>__ and
will not be included in theCodecov views <https://app.codecov.io/gh/aio-libs/multidict>__ going forward
-- by :user:webknjaz. -
Added memory leak test for popping or deleting attributes from a multidict to prevent future issues or bogus claims.
-- by :user:VizonexRelated issues and pull requests on GitHub:
#1233.