Native Celeborn shuffle: the remote read path costs up to 4.5x local decode and is unbenchmarked #5535

Discovered during post-merge review of #5531.

Component: native/shuffle/src/ipc.rs (read_single_batch),
native/core/src/execution/jni_api.rs:1265 (decodeShuffleBlockWithValidation)

read_ipc_compressed_validated drops with_skip_validation(true), so every remotely fetched batch
pays full Arrow buffer and offset validation. That is the right call for untrusted bytes, but the
cost is significant and is not recorded anywhere. Since the justification for a native Celeborn
reader is that it beats Celeborn's own row decoder, this number should be measured before 8/n
enables the path end to end.

Measurement

Release build, one ShuffleBlockWriter frame of 20 columns (10 Int32, 10 Utf8) x 8192 rows,
300 iterations after 20 warm-up, Apple silicon. "local" is read_ipc_compressed; "validated" is
read_ipc_compressed_validated + validate_remote_schema, i.e. what the reducer runs.

For reference, the local path itself did not regress in #5531: against the pre-PR implementation it
measured -8.8% (None), +0.8% (Lz4), +1.1% (Snappy), -1.8% (Zstd). The table above is purely the
price of the new remote path.

Two hoistable costs on the same path

• decodeShuffleBlockWithValidation calls ShuffleScan::decode and rebuilds expected_types via
  to_arrow_datatype on every block. Both are constant per reducer.
• env.convert_byte_array(expected_schema) copies the serialized schema into a fresh Vec per
  block.

Caching the decoded Vec<DataType> once per reducer, e.g. behind a native handle, would remove
both.

Ask

Publish a benchmark for the remote read path in the 8/n description, and decide whether the
validation cost is acceptable or should be made configurable.