Skip to content

Commit 2930a70

Browse files
alxmrsclaude
andauthored
Allocate with mimalloc in einfold's Python extension, as datafusion-python does (#83)
🪭 Claude (einfold) For #79 (peak memory as a goal). With einfold installed, a session that ran xarray-sql's GPT loss again and again kept growing. (The GPT is `benchmarks/ml/02_nanogpt.py`, in xqlsystems/xarray-sql#269.) Two causes: - **The codec kept executed plans.** A plan that has run keeps what its operators hold, such as a hash join's table, and the codec kept the last 1,024 plans. That cost about 2.2 GB per query. It's fixed on #75 ("Hold a plan from its encoding to its decoding"), since the code is there. - **The allocator** is this PR. This library allocated with the system allocator, glibc's, which keeps memory per thread, and einfold's kernels run on blocking threads. A session still grew by about 150 MB per query. With `MALLOC_ARENA_MAX=2` the growth stopped, so this is fragmentation, not retention. datafusion-python links mimalloc, and its sessions don't grow, so this library now allocates with mimalloc too. A buffer one library allocates is freed by its own release callback, through Arrow's C data interface, so the two allocators never free each other's memory. The GPT's loss at batch 8, 64 channels, context 32, run ten times in one session, from xarray-sql's environment: | | after the 1st | after the 5th | after the 10th | |---|---:|---:|---:| | plans kept | 2.9 GB | 9.9 GB (killed at 10 GB) | | | plans freed, glibc | 2.8 GB | 3.4 GB | | | plans freed, mimalloc | 3.3 GB | 3.1 GB | 3.1 GB | DataFusion alone: 2.7 GB after the 1st, 2.3 GB after the 5th. The whole demo at that size, three steps, now peaks at 3.6 GB with einfold against 2.7 GB without; before both fixes it reached 5.9 GB in two. With einfold, a training step takes 1.3 s against 2.5 s. The loss still peaks about 600 MB higher with einfold, which #79 is about. **Tests.** The Python suite passes (12 tests). #75's test of running queries again now checks that the codec holds no tables or plans after each query. ## After CodeRabbit's review - **mimalloc 2, not 3:** mimalloc 3 crashes on macOS arm64 when another copy of it runs in the process, as PyArrow 24's does (apache/datafusion-python#1607). #110's release workflow tests the macOS arm64 wheel. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01M96xSez6PZwTmfaGh2eJgS --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent fdb4e4e commit 2930a70

3 files changed

Lines changed: 34 additions & 0 deletions

File tree

‎python/Cargo.lock‎

Lines changed: 19 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎python/Cargo.toml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,14 @@ datafusion-ffi = "=55.2.0"
2727
datafusion-proto = "=55.2.0"
2828
einfold-datafusion = { path = "../crates/einfold-datafusion" }
2929
futures = "0.3"
30+
# mimalloc, Microsoft's general-purpose memory allocator, which
31+
# datafusion-python (DataFusion's Python bindings) uses too. With the system
32+
# allocator, each of the threads this library computes on keeps memory of its
33+
# own, and a session that runs many queries grew by about 150 MB per query.
34+
# Its version 2, because version 3 crashes on macOS arm64 when a second copy
35+
# of mimalloc runs in the same process, as PyArrow 24 (Apache Arrow's Python
36+
# library) loads one: apache/datafusion-python#1607, microsoft/mimalloc#1287.
37+
mimalloc = { version = "0.1", default-features = false, features = ["v2"] }
3038
pyo3 = { version = "0.29", features = ["extension-module", "abi3", "abi3-py310"] }
3139
# A session hands einfold each logical plan as protobuf, and prost's decoder
3240
# stops at 100 nested messages by default: a chain of 15 joins is deeper.

‎python/src/lib.rs‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ use datafusion_ffi::execution::FFI_TaskContextProvider;
3232
use datafusion_ffi::proto::logical_extension_codec::FFI_LogicalExtensionCodec;
3333
use datafusion_ffi::proto::physical_extension_codec::FFI_PhysicalExtensionCodec;
3434
use datafusion_ffi::query_planner::FFI_QueryPlanner;
35+
use mimalloc::MiMalloc;
3536
use pyo3::exceptions::PyImportError;
3637
use pyo3::prelude::*;
3738
use pyo3::types::{PyCapsule, PyDict};
@@ -41,6 +42,12 @@ mod codec;
4142
mod lean;
4243
mod stack;
4344

45+
/// mimalloc, Microsoft's general-purpose allocator, which datafusion-python
46+
/// uses too, in its version 2, which runs beside PyArrow's own copy (see
47+
/// `Cargo.toml`).
48+
#[global_allocator]
49+
static ALLOCATOR: MiMalloc = MiMalloc;
50+
4451
use bridge::{Bridge, Counts};
4552
use codec::ByReference;
4653

0 commit comments

Comments
 (0)