Add separate spill_batch_size configuration #17595

Is your feature request related to a problem or challenge?

Currently, when spilling RecordBatches to disk, datafusion serializes them using Arrow IPC in units defined by the global batch_size configuration (number of rows). However, it may be beneficial to decouple the spill batch size from the execution batch size. While large batches are good for vectorized execution, they can cause issues when reading back during multi-level merge (e.g. query failures even with fewer streams).

Describe the solution you'd like

1. Add a spill_batch_size configuration option. (maybe in rows or maybe in bytes unit)
2. Benchmark and validate its effect on I/O throughput and query stability (failures in multi-level merge).

Describe alternatives you've considered

No response

Additional context

No response