Prune unused dictionary values before spilling to Arrow IPC #27014

Is your feature request related to a problem or challenge?

IPC spill files can contain large dictionary value tables even when a batch references only a small subset of their entries.

Slicing or merging dictionary-encoded batches can leave dictionary arrays retaining unused values. The existing spill garbage collection traverses nested arrays and compacts StringView/BinaryView buffers, but does not prune dictionary entries that are no longer referenced by the keys.

This can cause a small batch to repeatedly write a much larger dictionary value table across spill and merge passes, increasing temporary disk usage and I/O.

Describe the solution you'd like

Compact oversized dictionary value tables before writing IPC spill batches, remap their keys, and then apply the existing child-array compaction.

Describe alternatives you've considered

n/a

Additional context

I already have a fix for this ready, it's quite simple since we already have a similar case for eg. StringView and we already have code for compacting dicts in arrow, so it's just a bit of plumbing and I'm just opening this for tracking purposes.

Willingness to contribute

None