feat: Investigate caching opportunities in iceberg_scan.rs #3602

Activity

1. mbutrovich commented on Mar 3, 2026

   

   mbutrovich

   on Mar 3, 2026

   Contributor

   More actions

   Investigated this with instrumentation in adapt_batch_with_expressions. After logging all projection expressions across the full CometIcebergNativeSuite (including schema evolution, partitioned tables, MOR, and complex types), every expression is either a column reference (name@N) or a cast wrapping a column reference (Cast [child: name@N] / COMET_CAST_COLUMN(name@N AS ...)). No literal expressions appeared.

   The two scenarios where we expected literals:

   1. Partition columns — iceberg-rust's RecordBatchTransformer injects identity partition values into the batch as data columns before it reaches us.
   2. Schema evolution (missing columns) — iceberg-rust's RecordBatchTransformer adds missing columns filled with nulls. In the "schema evolution - add column" test, old files written with (id, name) arrive with file_schema=["id", "name", "age"] already populated — the adapter just sees age@2.

   Since iceberg-rust handles both cases internally, our adapter never produces literal expressions, and there's nothing to cache. Closing as not applicable.

   👍React with 👍1Reacted by Oleks V
2. mbutrovich

   closed this as completedon Mar 3, 2026