Skip to content

Commit 050f1c6

Browse files
authored
fix(py): pin mimalloc v2 — v3 corrupts a co-resident pyarrow heap (#301)
* fix(py): pin mimalloc v2 — v3 corrupts a co-resident pyarrow heap `import pyarrow` followed by `import laterite` was enough to corrupt pyarrow's buffers. No laterite call is required, no Arrow FFI is involved, and the damage is silent until some later native call trips over it — which is why the crash kept surfacing under an innocent test name. pyarrow bundles its own mimalloc as its default memory pool, and libmimalloc-sys 0.1.49 defaults to v3. Two co-resident mimalloc v3 instances hand out overlapping memory (microsoft/mimalloc#1287, fixed only on `dev3`, not in a stable release). apache/datafusion-python#1607 is the same fault and took the same pin. Measured on the faulty build: corruption needs a string buffer over 64 bytes with 2+ rows, in the first pyarrow allocations after our module loads — the first 64 bytes come back zeroed. Reproduced 100% of the time; both orders and every consumer are clean after the pin. Keeping the allocator matters, so the cheap fix was the wrong one. Against the system allocator on the 25 MB fixture, v2 holds read ~18-19% and validate ~14-15% faster; dropping mimalloc entirely gives that back (read 122ms -> 150ms). v2 costs ~6% read against v3. Scope: pyarrow only. polars and duckdb are unaffected, so the base install was never exposed — only `[pyarrow]` / `[compat,pyarrow]` / `[all]`, and dev/CI, which installs all extras. The regression test runs in a subprocess: the fault depends on which allocator initialises first, so it is a property of process startup that no in-process assertion can express. That is why the existing #122 regression passed on every build that crashed; its docstring now says so rather than claiming a guarantee it cannot check. Also corrects the arrow-rs attribution in the `build_ags4` guard, which sent readers to apache/arrow-rs#10439. The guard stays: it is no longer what makes the pandas path memory-safe, but it is still what keeps that path working on a pyarrow-free `[compat]` install, where pandas' `__arrow_c_stream__` raises ImportError. * fix(node,cli): pin mimalloc v2 on the remaining surfaces Both carried a bare `mimalloc = "0.1"`, so both built the v3 that corrupts a co-resident heap (microsoft/mimalloc#1287). Neither is a proven exposure and neither is a fix: - the Node addon loads no pyarrow, but it is dlopen'd into a process whose other native addons we do not choose — the same shape that bit the wheel; - `lat` owns its whole process and loads no second allocator, so the fault cannot reach it at all. Pinned for version hygiene: one workspace, one mimalloc major, so a future surface cannot inherit v3 by default. `lat` builds and validates clean; laterite-cli's 47 tests pass. * docs: correct the upstream citation — mimalloc#1327, not #1287 #1287 is "mimalloc >= 3.3.0 causes segmentation faults when used from multiple threads", a threading bug. It is not this fault, and it was carried into every comment the previous two commits touched. The co-residency report is microsoft/mimalloc#1327, and Arrow has the same fault from its own side as apache/arrow GH-50428: a pyarrow 24.0.0 regression, milestone 25.0.1, whose accepted workaround is switching the extension's mimalloc copy to v2 — the pin these commits make. Two things that changes: - pyarrow >=25.0.1 fixes it upstream, so the regression test only goes red on pyarrow 24.0.0-25.0.0. Measured: 44 of 50 shapes corrupt under v3 on 25.0.0, 0 of 50 on 25.0.1, 0 of 50 under v2 on both. Noted in the test, because a green run on a fixed pyarrow proves nothing. - The pin stays regardless. We do not control which pyarrow a user installs, and CPython 3.14 vendors its own mimalloc v3, so pyarrow is not the only co-residency partner. * docs(changelog): declare the co-resident allocator fix
1 parent ec66197 commit 050f1c6

8 files changed

Lines changed: 125 additions & 17 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
4040

4141
### Fixed
4242

43+
- **The wheel could corrupt pyarrow's memory just by being imported.** `import pyarrow` followed by `import laterite` was enough to leave a subsequently-built `pyarrow.Table` holding zeroed bytes — no laterite call on the path, and no Arrow FFI involved. The damage was silent until some later native call tripped over it, so for months it surfaced as an intermittent SIGSEGV blamed on whichever innocent test ran next, never on the thing that caused it. pyarrow bundles its own mimalloc as its default memory pool and the `mimalloc` crate defaults to v3, and two co-resident v3 instances hand out overlapping memory (microsoft/mimalloc#1327; apache/arrow GH-50428 is the same fault from Arrow's side — a pyarrow 24.0.0 regression fixed in 25.0.1, whose own recommended workaround is the pin taken here). All three native surfaces — the wheel, the Node addon and `lat` — now pin mimalloc v2. Only pyarrow-bearing installs were ever affected (`[pyarrow]`, `[compat,pyarrow]`, `[all]`); polars and duckdb are unaffected, so the base install was never exposed. Keeping the allocator was the point: v2 holds ~18-19% read and ~14-15% validate over the system allocator, which dropping mimalloc would have given back.
4344
- **`COMPAT.md` documented a `check_file` keyword that has never existed, and a version frozen nine releases back.** Both blocks of the error-handling section passed `dictionary='not-a-dict.txt'`; the parameter is `standard_AGS4_dictionary` in laterite *and* in python-ags4, so the two snippets illustrating the divergence raised `TypeError` on both sides and showed the reader nothing — the prose was correct, only the code was uncopyable. The `laterite.compat.__version__` doctest, the `Metadata.Checker` banner a user sees in their own validation report, and the phase-1 illustration all still said `0.1.0`. `bump-my-version` stamps those two strings in `compat.py` and had no `COMPAT.md` entry, so code and page provably diverged on every release; neither superseded-version gate could catch it, because both match versions listed in `changelog.json` and its history starts at 0.6.0. The page is now stamped, and `tests/test_docs_snippets.py` asserts it against the wheel's real values.
4445
- **`gen_doc_outputs.py` silently under-counted a page that included an example SECTION.** The include→example map registered the literal `:cmd` spelling beside the bare filename, so the first page to use any other section name fell through the `not in wanted` branch as "a surface this run didn't ask for" — pointing three pages at `…py:code` dropped the matched-block count from 38 to 35 and still reported success. The key is now the include with its section suffix stripped, which covers every section name there will ever be.
4546
- **The PEP 723 pin gate could not see the file that motivated it.** It matched `laterite==(version)` and compared the version, which passes anything that is not an exact pin at all — so `examples/laterite_tour.py` asking for `laterite>=0.5.0` at product 0.10.1 (five minors of compatibility it claimed and nobody had tested) produced no match and therefore no finding, and a bare `"laterite"` escaped identically. The gate now asserts the whole specifier, discovers across both example trees rather than the docs corpus alone, and admits the `[extras]` group that the pandas examples need — `bump-version.sh` was equally blind to `laterite[compat]==`, which would have frozen those two pins at the old version while every sibling moved.

‎changelog.json‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,9 @@
7373
}
7474
],
7575
"fixed": [
76+
{
77+
"text": "**The wheel could corrupt pyarrow's memory just by being imported.** `import pyarrow` followed by `import laterite` was enough to leave a subsequently-built `pyarrow.Table` holding zeroed bytes \u2014 no laterite call on the path, and no Arrow FFI involved. The damage was silent until some later native call tripped over it, so for months it surfaced as an intermittent SIGSEGV blamed on whichever innocent test ran next, never on the thing that caused it. pyarrow bundles its own mimalloc as its default memory pool and the `mimalloc` crate defaults to v3, and two co-resident v3 instances hand out overlapping memory (microsoft/mimalloc#1327; apache/arrow GH-50428 is the same fault from Arrow's side \u2014 a pyarrow 24.0.0 regression fixed in 25.0.1, whose own recommended workaround is the pin taken here). All three native surfaces \u2014 the wheel, the Node addon and `lat` \u2014 now pin mimalloc v2. Only pyarrow-bearing installs were ever affected (`[pyarrow]`, `[compat,pyarrow]`, `[all]`); polars and duckdb are unaffected, so the base install was never exposed. Keeping the allocator was the point: v2 holds ~18-19% read and ~14-15% validate over the system allocator, which dropping mimalloc would have given back."
78+
},
7679
{
7780
"text": "**`COMPAT.md` documented a `check_file` keyword that has never existed, and a version frozen nine releases back.** Both blocks of the error-handling section passed `dictionary='not-a-dict.txt'`; the parameter is `standard_AGS4_dictionary` in laterite *and* in python-ags4, so the two snippets illustrating the divergence raised `TypeError` on both sides and showed the reader nothing — the prose was correct, only the code was uncopyable. The `laterite.compat.__version__` doctest, the `Metadata.Checker` banner a user sees in their own validation report, and the phase-1 illustration all still said `0.1.0`. `bump-my-version` stamps those two strings in `compat.py` and had no `COMPAT.md` entry, so code and page provably diverged on every release; neither superseded-version gate could catch it, because both match versions listed in `changelog.json` and its history starts at 0.6.0. The page is now stamped, and `tests/test_docs_snippets.py` asserts it against the wheel's real values."
7881
},

‎packages/laterite/python/laterite/__init__.py‎

Lines changed: 12 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2246,16 +2246,21 @@ def _check_meta(meta: Mapping[str, Mapping[str, str]] | None, name: str) -> None
22462246
con = None
22472247
for i, (code, frame) in enumerate(items):
22482248
frame = _drop_synth_keys(frame) # never emit a read(keys=True) _id/_parent_id
2249-
# A pandas >=2.2 frame exposes a *pyarrow-backed* Arrow C-stream capsule;
2250-
# handing it straight to the native emit corrupts the process heap and
2251-
# hard-crashes a later in-process native call — a use-after-free in the
2252-
# pyo3-arrow/arrow-rs consumer of a pyarrow-produced stream (polars' native
2253-
# stream is unaffected; reproduces from `build_ags4({g: pandas_frame})`).
2249+
# A pandas >=2.2 frame exposes a *pyarrow-backed* Arrow C-stream capsule.
22542250
# Normalise those to a native polars capsule first — the guard compat's
22552251
# writer (`_to_polars`) also applies. A pandas <2.2 frame has no capsule and
22562252
# falls through to the DuckDB bridge below (its existing pyarrow-free path),
2257-
# so that branch stays reachable. A pure Rust fix would need a workspace
2258-
# arrow/pyo3-arrow major bump across every binding; tracked separately.
2253+
# so that branch stays reachable.
2254+
#
2255+
# This was added for what was diagnosed as an arrow-rs use-after-free
2256+
# consuming a pyarrow-produced stream (#122, apache/arrow-rs#10439). #294
2257+
# established that diagnosis was wrong: the Arrow FFI is not involved, and
2258+
# the crash needed no laterite call at all — it was mimalloc v3 in our
2259+
# cdylib colliding with the mimalloc pyarrow bundles, now pinned to v2 in
2260+
# rust-packages/laterite-py/Cargo.toml. So this guard is NOT what makes the
2261+
# pandas path safe, and removing it would not reintroduce that crash. It is
2262+
# kept because it is also what keeps the pandas branch pyarrow-free; retire
2263+
# it deliberately, with measurements, not as a side effect of #294.
22592264
if type(frame).__module__.partition(".")[0] == "pandas" and hasattr(
22602265
frame, "__arrow_c_stream__"
22612266
):

‎packages/laterite/tests/test_public_api_surface.py‎

Lines changed: 71 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,9 @@
1111

1212
from __future__ import annotations
1313

14+
import subprocess
15+
import sys
16+
import textwrap
1417
from pathlib import Path
1518

1619
import laterite as L
@@ -121,8 +124,10 @@ def test_register_user_frame() -> None:
121124
# polars anyway (the DuckDB fallback at build_ags4's 2136-2142 is pandas<2.2
122125
# only, hence unreachable on the CI/dev pandas — a documented gap, not a test
123126
# target). Building from pandas here ALSO tripped a native crash when a later
124-
# native write ran in the same process — an arrow-rs use-after-free consuming
125-
# pandas' pyarrow-backed stream, now guarded (niko86/laterite#122, apache/arrow-rs#10439).
127+
# native write ran in the same process. That was long attributed to an arrow-rs
128+
# use-after-free consuming pandas' pyarrow-backed stream (niko86/laterite#122,
129+
# apache/arrow-rs#10439); #294 showed the Arrow FFI is not involved at all — see
130+
# the co-resident-allocator note below.
126131

127132

128133
def test_build_result_repr() -> None:
@@ -144,11 +149,16 @@ def test_build_drops_synth_key_columns_polars() -> None:
144149

145150

146151
def test_pandas_build_then_native_write_is_safe() -> None:
147-
"""Regression for niko86/laterite#122 (arrow-rs UAF, apache/arrow-rs#10439):
148-
a pandas frame's pyarrow-backed capsule handed to the native emit corrupted a
149-
later in-process native write. The guard normalises pandas ->2.2 to a polars
150-
capsule first. Assert the pandas build equals the polars build (proof the
151-
normalisation ran) AND that a native write after it stays intact."""
152+
"""Pins that a pandas build agrees byte-for-byte with the polars build and
153+
that a native write after it still produces a zip — i.e. the guard's happy
154+
path, end to end.
155+
156+
Deliberately NOT the regression test for the crash it was written for. It
157+
was filed as a #122 regression (arrow-rs UAF), but neither assertion can
158+
observe that fault: the texts match with or without the guard, and one native
159+
write cannot detect a corrupted heap. It passed on every build that crashed,
160+
which is why #294 went unexplained for so long. The real fault, and a test
161+
that actually goes red on it, are below."""
152162
import gc
153163

154164
pandas_text = L.build_ags4(
@@ -165,6 +175,60 @@ def test_pandas_build_then_native_write_is_safe() -> None:
165175
assert L.to_excel(L.read(_CLEAN))[:2] == b"PK" # native write uncorrupted
166176

167177

178+
# --- #294: co-resident allocators --------------------------------------------
179+
#
180+
# The fault #122/#294 chased is NOT an arrow-rs use-after-free and has nothing to
181+
# do with the Arrow FFI: pyarrow bundles its own mimalloc as its default memory
182+
# pool, and our cdylib used to bring a second, statically-linked mimalloc v3.
183+
# Two co-resident mimallocs hand out overlapping memory (microsoft/mimalloc#1327;
184+
# apache/arrow GH-50428 is the same fault from Arrow's side), so `import pyarrow`
185+
# followed by `import laterite` was enough to corrupt pyarrow's buffers with no
186+
# laterite call at all. Fixed by pinning mimalloc v2 in
187+
# rust-packages/laterite-py/Cargo.toml.
188+
#
189+
# pyarrow 25.0.1 also fixes it from their side, so this test only goes red on
190+
# pyarrow 24.0.0-25.0.0. It is kept because we do not control which pyarrow a
191+
# user installs — a green run on a fixed pyarrow proves nothing about the pin.
192+
#
193+
# This runs in a SUBPROCESS on purpose. The fault depends on which allocator
194+
# initialises first, so it is a property of process startup — by the time this
195+
# module is imported laterite is already loaded, and an in-process assertion
196+
# cannot express the condition (which is exactly why the #122 regression above
197+
# passed throughout: nothing it asserts can observe the corruption).
198+
#
199+
# The shape of the payload is load-bearing, measured on the faulty build: the
200+
# overlap only shows on a string buffer over 64 bytes with 2+ rows, and only in
201+
# the first pyarrow allocations after our module loads. Shrink the values, or
202+
# put work between the two imports and the table, and this silently stops
203+
# testing anything.
204+
205+
206+
def _run_isolated(body: str) -> subprocess.CompletedProcess[str]:
207+
"""Run `body` in a fresh interpreter with pyarrow imported FIRST."""
208+
return subprocess.run(
209+
[sys.executable, "-c", textwrap.dedent(body)],
210+
capture_output=True,
211+
text=True,
212+
timeout=120,
213+
)
214+
215+
216+
def test_pyarrow_buffers_survive_a_coresident_native_module() -> None:
217+
"""#294: importing laterite must not corrupt pyarrow's heap. Fails with a
218+
UnicodeDecodeError (the string buffer is overwritten) on a mimalloc-v3 build."""
219+
pytest.importorskip("pyarrow")
220+
r = _run_isolated("""
221+
import pyarrow as pa # pyarrow's bundled mimalloc initialises first
222+
import laterite # our cdylib brings the second allocator
223+
224+
vals = ["A" * 32, "B" * 32, "C" * 32, "D" * 32]
225+
table = pa.table({"c": vals})
226+
got = table.column(0).chunk(0).to_pylist()
227+
assert got == vals, f"pyarrow buffer corrupted: {got!r}"
228+
""")
229+
assert r.returncode == 0, f"stdout={r.stdout!r} stderr={r.stderr!r}"
230+
231+
168232
# --- merge: MergeResult repr ------------------------------------------------
169233

170234

‎rust-packages/laterite-cli/Cargo.toml‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,12 @@ tui = ["dep:ratatui", "dep:crossterm"]
5454
# small allocations for a 25 MB file; see perf-campaign T4-followup). Paired
5555
# with the `#[global_allocator]` in the crate root — only a final artifact
5656
# (binary / cdylib) can set one, so it is declared per-surface, not in a leaf.
57-
mimalloc = "0.1"
57+
#
58+
# PINNED to v2 in step with the wheel and the addon (#294). `lat` owns its whole
59+
# process and loads no second allocator, so the co-residency fault that bit the
60+
# wheel cannot reach it — this is version hygiene, not a fix: one workspace,
61+
# one mimalloc major, so a future surface cannot inherit v3 by default.
62+
mimalloc = { version = "0.1", features = ["v2"] }
5863
# The engine. Depended on via its public library API only — the split
5964
# has zero code-consumer breakage (every call site uses the re-exported
6065
# surface: CheckOptions / DictVersion / Findings / ValidatorError /

‎rust-packages/laterite-node/Cargo.toml‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,15 @@ crate-type = ["cdylib"]
3434
# small allocations for a 25 MB file; see perf-campaign T4-followup). Paired
3535
# with the `#[global_allocator]` in the crate root — only a final artifact
3636
# (binary / cdylib) can set one, so it is declared per-surface, not in a leaf.
37-
mimalloc = "0.1"
37+
#
38+
# PINNED to v2 for the same reason as the wheel (#294): libmimalloc-sys defaults
39+
# to mimalloc v3 from 0.1.47, and v3 co-resident with another mimalloc in the
40+
# same process hands out overlapping memory (microsoft/mimalloc#1327, and
41+
# apache/arrow GH-50428 from Arrow's side). Unproven for this addon — no pyarrow
42+
# here — but a Node addon is
43+
# dlopen'd into a process whose other addons we do not control, which is exactly
44+
# the shape that bit the wheel. Kept in step deliberately, not swept.
45+
mimalloc = { version = "0.1", features = ["v2"] }
3846
# `serde-json` lets `parseValue` return a `serde_json::Value` straight across the
3947
# boundary as native JS (number/string/bool/null) — the AGS type system's own
4048
# output shape, no hand-rolled union.

‎rust-packages/laterite-py/Cargo.toml‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,21 @@ crate-type = ["cdylib"]
4747
# small allocations for a 25 MB file; see perf-campaign T4-followup). Paired
4848
# with the `#[global_allocator]` in the crate root — only a final artifact
4949
# (binary / cdylib) can set one, so it is declared per-surface, not in a leaf.
50-
mimalloc = "0.1"
50+
#
51+
# PINNED to mimalloc v2 (#294). libmimalloc-sys defaults to v3 from 0.1.47, and
52+
# v3 co-resident with ANOTHER mimalloc in the same process hands out overlapping
53+
# memory: pyarrow bundles its own mimalloc as its default pool, so
54+
# `import pyarrow; import laterite` was enough to corrupt pyarrow's buffers —
55+
# no laterite call required. microsoft/mimalloc#1327 is the co-residency report;
56+
# apache/arrow GH-50428 is the same fault seen from Arrow's side (a pyarrow
57+
# 24.0.0 regression, fixed in pyarrow 25.0.1) and names switching the
58+
# extension's copy to v2 as the fix. Measured here: pyarrow 24.0.0/25.0.0
59+
# corrupt 44 of 50 shapes under v3, 0 of 50 under v2.
60+
#
61+
# Do NOT unpin just because pyarrow >=25.0.1 is fixed — we do not control the
62+
# pyarrow a user installs, and CPython 3.14 vendors its own mimalloc v3 as a
63+
# second co-residency partner regardless of pyarrow.
64+
mimalloc = { version = "0.1", features = ["v2"] }
5165
laterite-ags4-validator = { path = "../laterite-ags4-validator" }
5266
# The shared parse leaf (#168 Phase 3): the binding parses text/bytes through
5367
# the leaf directly (the validator is no longer on the parse hot-path); its

‎rust-packages/laterite-py/src/lib.rs‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,14 @@ use std::path::Path;
2525
// (measured). Set here because only a final cdylib can choose the allocator; the
2626
// engine leaves the Rust boundary via Arrow release callbacks, so Rust frees what
2727
// Rust allocated (no cross-allocator handoff to polars/pyarrow).
28+
//
29+
// Handoff was never the hazard — CO-RESIDENCY was (#294). We load into processes
30+
// that already contain another mimalloc (pyarrow bundles one as its default
31+
// pool; CPython 3.14 vendors one too), and mimalloc v3 in that situation hands
32+
// out memory the other instance is already using. The version pin that avoids
33+
// it is on the dep in Cargo.toml;
34+
// keep it there, and do not assume this comment's "Rust frees what Rust
35+
// allocated" makes a second allocator in the process safe.
2836
#[global_allocator]
2937
static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;
3038

0 commit comments

Comments
 (0)