You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 050f1c6
Browse filesBrowse the repository at this point in the historyBrowse files
* fix(py): pin mimalloc v2 — v3 corrupts a co-resident pyarrow heap
`import pyarrow` followed by `import laterite` was enough to corrupt
pyarrow's buffers. No laterite call is required, no Arrow FFI is
involved, and the damage is silent until some later native call trips
over it — which is why the crash kept surfacing under an innocent test
name.
pyarrow bundles its own mimalloc as its default memory pool, and
libmimalloc-sys 0.1.49 defaults to v3. Two co-resident mimalloc v3
instances hand out overlapping memory (microsoft/mimalloc#1287, fixed
only on `dev3`, not in a stable release). apache/datafusion-python#1607
is the same fault and took the same pin.
Measured on the faulty build: corruption needs a string buffer over 64
bytes with 2+ rows, in the first pyarrow allocations after our module
loads — the first 64 bytes come back zeroed. Reproduced 100% of the
time; both orders and every consumer are clean after the pin.
Keeping the allocator matters, so the cheap fix was the wrong one.
Against the system allocator on the 25 MB fixture, v2 holds read ~18-19%
and validate ~14-15% faster; dropping mimalloc entirely gives that back
(read 122ms -> 150ms). v2 costs ~6% read against v3.
Scope: pyarrow only. polars and duckdb are unaffected, so the base
install was never exposed — only `[pyarrow]` / `[compat,pyarrow]` /
`[all]`, and dev/CI, which installs all extras.
The regression test runs in a subprocess: the fault depends on which
allocator initialises first, so it is a property of process startup that
no in-process assertion can express. That is why the existing #122
regression passed on every build that crashed; its docstring now says so
rather than claiming a guarantee it cannot check.
Also corrects the arrow-rs attribution in the `build_ags4` guard, which
sent readers to apache/arrow-rs#10439. The guard stays: it is no longer
what makes the pandas path memory-safe, but it is still what keeps that
path working on a pyarrow-free `[compat]` install, where pandas'
`__arrow_c_stream__` raises ImportError.
* fix(node,cli): pin mimalloc v2 on the remaining surfaces
Both carried a bare `mimalloc = "0.1"`, so both built the v3 that
corrupts a co-resident heap (microsoft/mimalloc#1287). Neither is a
proven exposure and neither is a fix:
- the Node addon loads no pyarrow, but it is dlopen'd into a process
whose other native addons we do not choose — the same shape that bit
the wheel;
- `lat` owns its whole process and loads no second allocator, so the
fault cannot reach it at all. Pinned for version hygiene: one
workspace, one mimalloc major, so a future surface cannot inherit v3
by default.
`lat` builds and validates clean; laterite-cli's 47 tests pass.
* docs: correct the upstream citation — mimalloc#1327, not #1287
#1287 is "mimalloc >= 3.3.0 causes segmentation faults when used from
multiple threads", a threading bug. It is not this fault, and it was
carried into every comment the previous two commits touched.
The co-residency report is microsoft/mimalloc#1327, and Arrow has the
same fault from its own side as apache/arrow GH-50428: a pyarrow 24.0.0
regression, milestone 25.0.1, whose accepted workaround is switching the
extension's mimalloc copy to v2 — the pin these commits make.
Two things that changes:
- pyarrow >=25.0.1 fixes it upstream, so the regression test only goes
red on pyarrow 24.0.0-25.0.0. Measured: 44 of 50 shapes corrupt under
v3 on 25.0.0, 0 of 50 on 25.0.1, 0 of 50 under v2 on both. Noted in
the test, because a green run on a fixed pyarrow proves nothing.
- The pin stays regardless. We do not control which pyarrow a user
installs, and CPython 3.14 vendors its own mimalloc v3, so pyarrow is
not the only co-residency partner.
* docs(changelog): declare the co-resident allocator fix
Copy file name to clipboardExpand all lines: CHANGELOG.md
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -40,6 +40,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
40
40
41
41
### Fixed
42
42
43
+
- **The wheel could corrupt pyarrow's memory just by being imported.** `import pyarrow` followed by `import laterite` was enough to leave a subsequently-built `pyarrow.Table` holding zeroed bytes — no laterite call on the path, and no Arrow FFI involved. The damage was silent until some later native call tripped over it, so for months it surfaced as an intermittent SIGSEGV blamed on whichever innocent test ran next, never on the thing that caused it. pyarrow bundles its own mimalloc as its default memory pool and the `mimalloc` crate defaults to v3, and two co-resident v3 instances hand out overlapping memory (microsoft/mimalloc#1327; apache/arrow GH-50428 is the same fault from Arrow's side — a pyarrow 24.0.0 regression fixed in 25.0.1, whose own recommended workaround is the pin taken here). All three native surfaces — the wheel, the Node addon and `lat` — now pin mimalloc v2. Only pyarrow-bearing installs were ever affected (`[pyarrow]`, `[compat,pyarrow]`, `[all]`); polars and duckdb are unaffected, so the base install was never exposed. Keeping the allocator was the point: v2 holds ~18-19% read and ~14-15% validate over the system allocator, which dropping mimalloc would have given back.
43
44
-**`COMPAT.md` documented a `check_file` keyword that has never existed, and a version frozen nine releases back.** Both blocks of the error-handling section passed `dictionary='not-a-dict.txt'`; the parameter is `standard_AGS4_dictionary` in laterite *and* in python-ags4, so the two snippets illustrating the divergence raised `TypeError` on both sides and showed the reader nothing — the prose was correct, only the code was uncopyable. The `laterite.compat.__version__` doctest, the `Metadata.Checker` banner a user sees in their own validation report, and the phase-1 illustration all still said `0.1.0`. `bump-my-version` stamps those two strings in `compat.py` and had no `COMPAT.md` entry, so code and page provably diverged on every release; neither superseded-version gate could catch it, because both match versions listed in `changelog.json` and its history starts at 0.6.0. The page is now stamped, and `tests/test_docs_snippets.py` asserts it against the wheel's real values.
44
45
-**`gen_doc_outputs.py` silently under-counted a page that included an example SECTION.** The include→example map registered the literal `:cmd` spelling beside the bare filename, so the first page to use any other section name fell through the `not in wanted` branch as "a surface this run didn't ask for" — pointing three pages at `…py:code` dropped the matched-block count from 38 to 35 and still reported success. The key is now the include with its section suffix stripped, which covers every section name there will ever be.
45
46
-**The PEP 723 pin gate could not see the file that motivated it.** It matched `laterite==(version)` and compared the version, which passes anything that is not an exact pin at all — so `examples/laterite_tour.py` asking for `laterite>=0.5.0` at product 0.10.1 (five minors of compatibility it claimed and nobody had tested) produced no match and therefore no finding, and a bare `"laterite"` escaped identically. The gate now asserts the whole specifier, discovers across both example trees rather than the docs corpus alone, and admits the `[extras]` group that the pandas examples need — `bump-version.sh` was equally blind to `laterite[compat]==`, which would have frozen those two pins at the old version while every sibling moved.
Copy file name to clipboardExpand all lines: changelog.json
+3Lines changed: 3 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -73,6 +73,9 @@
73
73
}
74
74
],
75
75
"fixed": [
76
+
{
77
+
"text": "**The wheel could corrupt pyarrow's memory just by being imported.** `import pyarrow` followed by `import laterite` was enough to leave a subsequently-built `pyarrow.Table` holding zeroed bytes \u2014 no laterite call on the path, and no Arrow FFI involved. The damage was silent until some later native call tripped over it, so for months it surfaced as an intermittent SIGSEGV blamed on whichever innocent test ran next, never on the thing that caused it. pyarrow bundles its own mimalloc as its default memory pool and the `mimalloc` crate defaults to v3, and two co-resident v3 instances hand out overlapping memory (microsoft/mimalloc#1327; apache/arrow GH-50428 is the same fault from Arrow's side \u2014 a pyarrow 24.0.0 regression fixed in 25.0.1, whose own recommended workaround is the pin taken here). All three native surfaces \u2014 the wheel, the Node addon and `lat` \u2014 now pin mimalloc v2. Only pyarrow-bearing installs were ever affected (`[pyarrow]`, `[compat,pyarrow]`, `[all]`); polars and duckdb are unaffected, so the base install was never exposed. Keeping the allocator was the point: v2 holds ~18-19% read and ~14-15% validate over the system allocator, which dropping mimalloc would have given back."
78
+
},
76
79
{
77
80
"text": "**`COMPAT.md` documented a `check_file` keyword that has never existed, and a version frozen nine releases back.** Both blocks of the error-handling section passed `dictionary='not-a-dict.txt'`; the parameter is `standard_AGS4_dictionary` in laterite *and* in python-ags4, so the two snippets illustrating the divergence raised `TypeError` on both sides and showed the reader nothing — the prose was correct, only the code was uncopyable. The `laterite.compat.__version__` doctest, the `Metadata.Checker` banner a user sees in their own validation report, and the phase-1 illustration all still said `0.1.0`. `bump-my-version` stamps those two strings in `compat.py` and had no `COMPAT.md` entry, so code and page provably diverged on every release; neither superseded-version gate could catch it, because both match versions listed in `changelog.json` and its history starts at 0.6.0. The page is now stamped, and `tests/test_docs_snippets.py` asserts it against the wheel's real values."
0 commit comments