You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 488a584
Browse filesBrowse the repository at this point in the historyBrowse files
Track allocator-level memory vs MemoryPool during SLTs to prevent OOMs (apache#22626)
## Which issue does this PR close?
Closesapache#22627.
## What & why
A test opts in with `SET datafusion.runtime.memory_limit = 'N'`; the
wrapping `AccountingMemoryPool` then panics the query if real heap
allocations exceed `N * 1.10`. Catches the silent-OOM bug class where
DF's voluntary `MemoryPool` undercounts. Off by default.
## Changes
**Upstream** (`execution/src/memory_pool/`, `runtime_env.rs`)
- New `MemoryPool::try_resize(usize) -> Result<()>`, default
`Err(NotImplemented)`. `GreedyMemoryPool` (now `AtomicUsize` pool_size)
and `TrackConsumersPool` override.
- `RuntimeEnvBuilder::with_memory_limit` tries `try_resize` before
wholesale replacement so wrappers survive `SET memory_limit`.
**SLT runner** (`datafusion/sqllogictest/`)
- `AccountingAllocator`: global allocator, per-file account
(`HashMap<ContextId, AtomicIsize>`). Context-id stamped on workers
via per-file Tokio runtime + `on_thread_start`.
- `AccountingMemoryPool` wraps DF's pool. `try_resize` retunes the
account to `N * 1.10`. Renders the operator-set default as `Infinite`
so un-opted tests' `SHOW ALL` is unchanged.
- One CLI flag, `--default-pool-size-mb` (replaces `--total-memory-mb`
+ `--datafusion-memory-fraction`).
- README + contributor-guide updated.
## Testing
- `memory_pool` and SLT-accounting unit tests pass.
- Full 475-file SLT corpus passes under `--default-pool-size-mb 16384`.
## User-facing
API non-breaking. `--total-memory-mb` and `--datafusion-memory-fraction`
replaced by `--default-pool-size-mb`. End-user query behavior unchanged.
---
```
[aggregate.slt] killed by allocator overdraft:
account balance = -110983311 bytes, df-pool reserved = 0 MB;
sql = "SELECT array_agg(c13 ORDER BY c13) FROM ..."
```
`df-pool reserved = 0 MB` next to a deeply-negative balance is the
discrepancy this PR surfaces.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
0 commit comments