slt: WindowAggExec untracked allocation (failing test) - #423

Summary

Adds a failing SLT that exposes a real DataFusion accounting bug via the framework in the parent PR.

WindowAggStream holds references to every input batch for the whole stream without telling MemoryPool. The bytes were allocated upstream — Window doesn't double them — but Window is the reason they aren't dropped. Because the pool never sees that retention, spill triggers and backpressure don't fire, and upstream operators keep allocating into a budget they think is free. The framework's allocator-level bank notices on the very next upstream alloc.

The offending lines

Retention without accounting — datafusion/physical-plan/src/windows/window_agg_exec.rs:429

loop {
    return Poll::Ready(Some(match ready!(self.input.poll_next_unpin(cx)) {
        Some(Ok(batch)) => {
            self.batches.push(batch);   // ← no reservation.try_grow(batch_size)
            continue;
        }
        // ...

permalink

The push itself is just an Arc::clone — the bytes are upstream's. But this is where they stop being eligible for drop. With 500 K rows × ~1 KB payload, that's ~500 MB of Arrow buffers kept alive while MemoryPool::reserved() stays near zero.

End-of-stream rebuild — same file, line 362:

let batch = concat_batches(&self.input.schema(), &self.batches)?;

permalink

concat_batches allocates fresh buffers for primitive arrays; for Utf8View it can reuse the data buffers but still rebuilds the view list and null bitmap. Also unaccounted.

WindowAggStream has no reservation field — grep -n "reservation\|try_grow\|MemoryConsumer\|MemoryReservation" window_agg_exec.rs returns nothing.

Why this matters in practice

A query with a single global window over wide rows under a tight pool runs to ~10× the declared budget on the heap. MemoryPool::reserved() will report a small number throughout because nothing in this operator ever calls try_grow. Operators downstream of Window that do spill will never decide to, because the pool looks empty. Eventually a real allocation pushes us over the kernel's tolerance and the process dies — there's no ResourcesExhausted error to catch.

Reproducer

Configuration is intentionally non-engineered: default 10 % headroom from accounting_pool.rs::HEADROOM_FACTOR, 100 MB pool. max(msg) OVER () references the wide payload so the optimizer can't elide the carry through WindowAggExec.

External error: 1 errors in file datafusion/sqllogictest/test_files/memory_accounting_demo.slt

1. statement failed: Other Error: allocator overdraft: account balance at panic = -317042 bytes
[SQL] SELECT msg, max(msg) OVER () AS max_msg
FROM (
    SELECT 'payload-' || cast(i AS varchar) || repeat('q', 1000) AS msg
    FROM generate_series(1, 500000) AS t(i)
)
LIMIT 5

Stack trace at the firing allocation

thread 'slt-file-1' panicked at datafusion/sqllogictest/src/accounting.rs:235:9:
   1: std::panic::panic_any
   2: datafusion_sqllogictest::accounting::track::{{closure}}        ← bank crossed zero
   4: <AccountingAllocator as GlobalAlloc>::alloc
   6: <RawVecInner>::finish_grow
   9: GenericByteViewBuilder::try_append_value
  10: datafusion_physical_expr::expressions::binary::kernels::concat_elements_utf8view
  11: datafusion_physical_expr::expressions::binary::concat_elements
  12: BinaryExpr::evaluate_with_resolved_args                        ← payload concat in upstream projection
  33: ProjectionStream::batch_project

The crashing alloc is innocent on its own — a string-concat in the projection feeding Window. It's just the alloc that happens after Window has silently consumed the budget by retaining every prior batch.

Test plan

• cargo test --features memory-accounting -p datafusion-sqllogictest --test sqllogictests -- memory_accounting_demo --default-pool-size-mb 100 exits non-zero with the message above.