Repository navigation
Commit 2930a70
Allocate with mimalloc in einfold's Python extension, as datafusion-python does (#83)
🪭 Claude (einfold)
For #79 (peak memory as a goal). With einfold installed, a session that
ran xarray-sql's GPT loss again and again kept growing. (The GPT is
`benchmarks/ml/02_nanogpt.py`, in xqlsystems/xarray-sql#269.) Two
causes:
- **The codec kept executed plans.** A plan that has run keeps what its
operators hold, such as a hash join's table, and the codec kept the last
1,024 plans. That cost about 2.2 GB per query. It's fixed on #75 ("Hold
a plan from its encoding to its decoding"), since the code is there.
- **The allocator** is this PR. This library allocated with the system
allocator, glibc's, which keeps memory per thread, and einfold's kernels
run on blocking threads. A session still grew by about 150 MB per query.
With `MALLOC_ARENA_MAX=2` the growth stopped, so this is fragmentation,
not retention. datafusion-python links mimalloc, and its sessions don't
grow, so this library now allocates with mimalloc too. A buffer one
library allocates is freed by its own release callback, through Arrow's
C data interface, so the two allocators never free each other's memory.
The GPT's loss at batch 8, 64 channels, context 32, run ten times in one
session, from xarray-sql's environment:
| | after the 1st | after the 5th | after the 10th |
|---|---:|---:|---:|
| plans kept | 2.9 GB | 9.9 GB (killed at 10 GB) | |
| plans freed, glibc | 2.8 GB | 3.4 GB | |
| plans freed, mimalloc | 3.3 GB | 3.1 GB | 3.1 GB |
DataFusion alone: 2.7 GB after the 1st, 2.3 GB after the 5th.
The whole demo at that size, three steps, now peaks at 3.6 GB with
einfold against 2.7 GB without; before both fixes it reached 5.9 GB in
two. With einfold, a training step takes 1.3 s against 2.5 s. The loss
still peaks about 600 MB higher with einfold, which #79 is about.
**Tests.** The Python suite passes (12 tests). #75's test of running
queries again now checks that the codec holds no tables or plans after
each query.
## After CodeRabbit's review
- **mimalloc 2, not 3:** mimalloc 3 crashes on macOS arm64 when another
copy of it runs in the process, as PyArrow 24's does
(apache/datafusion-python#1607). #110's release workflow tests the macOS
arm64 wheel.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01M96xSez6PZwTmfaGh2eJgS
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>1 parent fdb4e4e commit 2930a70
3 files changed
Lines changed: 34 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
30 | 38 | | |
31 | 39 | | |
32 | 40 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
| 35 | + | |
35 | 36 | | |
36 | 37 | | |
37 | 38 | | |
| |||
41 | 42 | | |
42 | 43 | | |
43 | 44 | | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
44 | 51 | | |
45 | 52 | | |
46 | 53 | | |
| |||
0 commit comments