Arrays without nulls now take 8 fewer bytes each in the spill files
(likely apache/arrow-rs#11694), so the hard-coded spilled_bytes
expectations in the spill tests change accordingly.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

…s in IPC writer (#11817)

* Backport of #11694
* For inclusion in #11665 

Cherry-pick of
[49219f8](49219f8)

Co-authored-by: Aleksandar Zecevic <[email protected]>

…pache#11694)

# Which issue does this PR close?

N/A

# Rationale for this change

The IPC writer writes a validity bitmap for every array that has a
validity buffer, also when the array has no nulls. If the array has no
null buffer, `write_array_data` creates an all-ones bitmap of `ceil(len
/ 8)` bytes for it. Readers do not use it: the arrow-rs reader, like the
C++ reader, ignores the validity buffer when `null_count` is 0.

The format allows a 0-length buffer instead. [`FieldNode.null_count` in
`Message.fbs`](https://github.com/apache/arrow/blob/9f0341314ca96eb58081d4132c77b460d880b2b0/format/Message.fbs#L39-L41):

> Fields with null_count == 0 may choose not to write their physical
validity bitmap out as a materialized buffer, instead setting the length
of the bitmap buffer to 0.

The
[C++](https://github.com/apache/arrow/blob/9f0341314ca96eb58081d4132c77b460d880b2b0/cpp/src/arrow/ipc/writer.cc#L178-L187),
[Go](https://github.com/apache/arrow-go/blob/a6e44ad7a6aaa4c02a7d277032af8eebd5cc2508/arrow/ipc/writer.go#L634-L638)
and
[JS](https://github.com/apache/arrow-js/blob/43c9547baf3766e79eb44d3fedcd0787f899d956/src/visitor/vectorassembler.ts#L95-L98)
writers already do this. So arrow-rs already reads such buffers, in the
integration tests and in the `arrow-testing` gold files, which C++
wrote.

# What changes are included in this PR?

- `write_array_data` writes a 0-length validity buffer when `null_count`
is 0. This also covers sliced arrays whose null buffer has no nulls in
the slice. Arrays with nulls are written as before.
- `test_decimal128_alignment8_is_unaligned` relied on the all-ones
bitmap to move the decimal values to an 8-byte offset. It now uses a
nullable column with a null, so it gets the expected misalignment error
with both the old and the new writer.
- The exact overage limits in the arrow-flight `verify_encoded_split`
tests go down, for example from 120 to 88.

# Are these changes tested?

Yes. `test_write_empty_validity_buffer_without_nulls` encodes a batch
with three columns: an array without a null buffer, a slice without
nulls that keeps its null buffer, and an array with a null. It checks
the validity buffer lengths in the encoded message (`[0, 0, 1]`) and the
round trip. Without the change, the first two columns get 1-byte bitmaps
and the test fails.

# Are there any significant user-facing changes?

The IPC output for arrays without nulls is smaller. The spec allows
this, and the C++, Java, Go, JS, C# and nanoarrow readers accept it.
Code that compares IPC output byte for byte, or expects exact sizes,
will see a difference. There are no API changes.

(cherry picked from commit 49219f8)

…s in IPC writer (#11820)

* Backport of #11694
* For inclusion in #11664
* 58 backport is #11817 (requested in
#11665 (comment))

Cherry-pick of
[49219f8](49219f8)

The cherry-pick applied cleanly but did not compile: on main
`write_array_data` has a `null_count` local introduced by #10128, which
is not on `59_maintenance`. This PR adds that one-line local (`let
null_count = array_data.null_count();`) so the backported code matches
main.

Co-authored-by: Aleksandar Zecevic <[email protected]>