Marking as draft. Spark diffs may need to be updated as well.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 58.59%. Comparing base (f09f8af) to head (f22c55e).
Report is 207 commits behind head on main.

@@             Coverage Diff              @@
##               main    #1771      +/-   ##
============================================
+ Coverage     56.12%   58.59%   +2.46%     
- Complexity      976     1131     +155     
============================================
  Files           119      130      +11     
  Lines         11743    12667     +924     
  Branches       2251     2363     +112     
============================================
+ Hits           6591     7422     +831     
- Misses         4012     4061      +49     
- Partials       1140     1184      +44     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

This is ready for review. Fixes 16 test failures in core2 (both native_datafusion and native_iceberg_compat) without any regressions

in followup PR I'll add a rust test case for map

Thanks @parthchandra

however I feel we need to add some more later. the test passed because

          checkSparkAnswer(spark.sql("SELECT map_keys(map1).id2 FROM tbl"))

and will prob fail for

          checkSparkAnswerAndOperator(spark.sql("SELECT map_keys(map1).id2 FROM tbl"))

Good point @comphead .

The test cannot use checkSparkAnswerAndOperator atm but should eventually.
The plan is not fully native for two reasons -
native_datafusion and native_iceberg_compat do not support DSV2 so the Scan falls back to Spark
with DSV2 the plan is

plan: *(1) Project [map_keys(map1#12).id2 AS map_keys(map1).id2#54]
+- *(1) ColumnarToRow
   +- BatchScan parquet file:/private/var/folders/bz/gg_fqnmj4c17j2c7mdn8ps1m0000gn/T/spark-e0fd616e-7fdc-4d0a-be8a-e6453797e243[map1#12] ParquetScan DataFilters: [], Format: parquet, Location: InMemoryFileIndex(1 paths)[file:/private/var/folders/bz/gg_fqnmj4c17j2c7mdn8ps1m0000gn/T/spark-e0..., PartitionFilters: [], PushedAggregation: [], PushedFilters: [], PushedGroupBy: [], ReadSchema: struct<map1:map<struct<id2:bigint>,struct<id:bigint,id2:bigint,id3:bigint>>> RuntimeFilters: []

with DSV1 sources, the scan is Native but the Project is not.

plan: *(1) Project [map_keys(map1#82).id2 AS map_keys(map1).id2#118]
+- *(1) CometColumnarToRow
   +- CometNativeScan parquet [map1#82] Batched: true, DataFilters: [], Format: CometParquet, Location: InMemoryFileIndex(1 paths)[file:/private/var/folders/bz/gg_fqnmj4c17j2c7mdn8ps1m0000gn/T/spark-d0..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<map1:map<struct<id2:bigint>,struct<id:bigint,id2:bigint,id3:bigint>>>

Filed #1781