Skip to content

Commit 9c2f8f1

Browse files
cetra3claude
andcommitted
Keep shared view buffers when spilling instead of deduplicating values
Replace value deduplication with a check that needs no hashing of values. A dictionary-encoded Parquet column is read as views into one shared buffer. When a sort combines its input batches with `concat` or `interleave`, that buffer is listed once per input batch, and `gc()` copies it once per row. The spill path always ran `gc()` above 10 KB, so the 1000-value repro from #23564 spilled 83 MB. Before spilling a view array: 1. Merge repeated entries of the same buffer, keyed by pointer and length. 2. Run `gc()` only if it writes less than the merged buffers hold. Both sizes come from the views and buffer lengths. 3. Otherwise rewrite the views to the merged buffer list and keep it. Data without shared buffers takes the same path as on `main`. The value order does not matter, no sample is taken, and the only allocation that grows with the rows is the rewritten views, when they replace a larger `gc()`. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01PVKQ8xyveW8L4ZUQQJoqQw
1 parent 7effeab commit 9c2f8f1

1 file changed

Lines changed: 249 additions & 177 deletions

File tree

  • datafusion/physical-plan/src/spill

0 commit comments

Comments
 (0)