Is your feature request related to a problem or challenge? Please describe what you are trying to do.
With #8004 and #8012 the parquet's ArrowWriter became more flexible in terms of what array types it can accept and write.

Still this doesn't seem sufficient for a lot of cases. For instance, consider Iceberg, where all fields (including nested ones) should be marked with PARQUET_FIELD_ID_META_KEY metadata.

When a non-trivial plan involving nested types (e.g. lists) is constructed to stream the batches to the writer, it is not straight-forward to align the list's inner field metadata to what is expected. DataFusion does provide alias_with_metadata but this will only align the root level fields, not the nested ones.

Consequently an error gets thrown, e.g.

ArrowError("Incompatible type. Field 'a' has type List(non-null Int32, metadata: {\"PARQUET:field_id\": \"1\"}), array has type List(non-null Int32)")

even though the types are logically equivalent.

Describe the solution you'd like
Relax LevelInfoBuilder::types_compatible further, by ignoring nested field names and metadata when comparing data types (i.e. using DataType::equals_datatype instead of strict equality).

Describe alternatives you've considered
Map each batch in the stream produced by the DataFusion plan, such that all batch arrays with nested data types are aligned with the target fields.

This is unwieldy, and the only thing it does is place PARQUET_FIELD_ID_META_KEY in metadata of nested fields, so that the LevelInfoBuilder::types_compatible check passes.

Additional context
Note that in the case of Iceberg the initial, schema-defining write should contain PARQUET_FIELD_ID_META_KEY in metadata, but subsequent batches need not if the types are logically equivalent, hence this issue.