Native Parquet scan returns NULL for nested BINARY values that are not valid UTF-8 when they are read as STRING #7032

Describe the bug

When a Parquet BINARY column without the UTF8 annotation is read as STRING, which a read schema that declares string or spark.sql.parquet.binaryAsString=true does, the native Parquet scan returns NULL for any nested value that is not valid UTF-8. Spark returns the bytes unchanged. Nothing fails, so the NULLs are silent. Top-level values take a different path that skips UTF-8 validation entirely.

• Nested values (a struct field, list element or map value) are converted by parquet_convert_array_impl, which falls back to Arrow's cast_with_options with PARQUET_OPTIONS (native/core/src/parquet/parquet_support.rs lines 74-79 and 333-336). Those options set safe: true, so a value that is not valid UTF-8 becomes NULL.
• Top-level values are decoded by arrow-rs straight into a Utf8 array. arrow-rs validates UTF-8 only for columns annotated UTF8 (parquet 59.3.0 arrow/array_reader/byte_array.rs line 192), so in a release build the invalid bytes reach a Utf8 array unchecked, the unsound state [EPIC] Consistent handling of invalid UTF-8 in native StringType (ingress policy) #4764 describes. A debug build panics instead, in OffsetBuffer::into_array, which validates only under debug_assertions (arrow/buffer/offset_buffer.rs line 141).

Steps to reproduce

In a CometTestBase suite:

spark.sql("SELECT X'41FF42' AS t, named_struct('b', X'41FF42') AS s, array(X'41FF42') AS l")
  .coalesce(1).write.parquet(path)
spark.read.schema("t string, s struct<b:string>, l array<string>").parquet(path)
  .selectExpr("hex(s.b)", "hex(l[0])", "s.b IS NULL")
  .collect()

On main at 66c63bd, the same on Spark 3.4.3, 3.5.9 and 4.1.3:

Expected behavior

The native scan does not turn a nested value into NULL because its bytes are not valid UTF-8, and the top-level path does not admit invalid UTF-8 into a Utf8 array. #4764 proposes decoding at the ingress boundary with decode_utf8_spark_lossy, the decoder CAST(binary AS string) and native shuffle already share, which would cover both paths.

Additional context

Found in a nested-type audit of the native scans. This is another ingress point for the policy #4764 tracks. #4121 covers UTF8-annotated STRING columns, which the native reader rejects with encountered non UTF-8 data. The BINARY-read-as-STRING path is not covered there.