Repository navigation
Commit 9c2f8f1
Keep shared view buffers when spilling instead of deduplicating values
Replace value deduplication with a check that needs no hashing of values.
A dictionary-encoded Parquet column is read as views into one shared
buffer. When a sort combines its input batches with `concat` or
`interleave`, that buffer is listed once per input batch, and `gc()`
copies it once per row. The spill path always ran `gc()` above 10 KB, so
the 1000-value repro from #23564 spilled 83 MB.
Before spilling a view array:
1. Merge repeated entries of the same buffer, keyed by pointer and length.
2. Run `gc()` only if it writes less than the merged buffers hold. Both
sizes come from the views and buffer lengths.
3. Otherwise rewrite the views to the merged buffer list and keep it.
Data without shared buffers takes the same path as on `main`. The value
order does not matter, no sample is taken, and the only allocation that
grows with the rows is the rewritten views, when they replace a larger
`gc()`.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01PVKQ8xyveW8L4ZUQQJoqQw1 parent 7effeab commit 9c2f8f1
1 file changed
Lines changed: 249 additions & 177 deletions
0 commit comments