Extend CastExpr and Converge on Single Cast Expression #20164

Enhance the logical CastExpr to support struct-aware casting, field-level nullability checks, and schema-aware validation. Once complete, retire CastColumnExpr by updating the adapter to emit the enhanced CastExpr and consolidate all casting logic into one expression type.

Scope

This is a follow-up to #20163. It aims to unify casting behavior across the planner and execution layers through a single enhanced expression.

Tasks

Logical Expression Enhancement:

• Extend CastExpr in datafusion/expr/ to support struct field casting
• Add field-level nullability and metadata propagation to CastExpr
• Implement schema-aware validation for target types

Physical Planning & Adapter Integration:

• Update physical planner to convert enhanced CastExpr → optimized physical cast
• Update PhysicalExprAdapter to emit enhanced CastExpr instead of CastColumnExpr
• Ensure adapter-generated casts are compatible with new CastExpr semantics

Testing & Documentation:

• Deprecate CastColumnExpr (or convert to thin wrapper)
• Add SQL logic tests for struct casts in standard SQL queries
• Remove or archive CastColumnExpr once fully transitioned
• Update casting documentation to reflect unified behavior

Files/Modules

• datafusion/expr/src/expr.rs (CastExpr extension)
• datafusion/physical-planner/ (logical → physical conversion)
• datafusion/physical-expr-adapter/src/schema_rewriter.rs (adapter integration)
• datafusion/sqllogictest/test_files/ (new SLT tests for struct casting)

Acceptance Criteria

✅ Enhanced CastExpr supports all CastColumnExpr features
✅ Physical planner correctly converts enhanced CastExpr to physical cast
✅ Adapter emits enhanced CastExpr for schema adaptation
✅ CastColumnExpr is deprecated or removed
✅ SQL logic tests validate struct casts work end-to-end
✅ All existing cast tests continue to pass
✅ Code compiles without warnings

Notes

• Proto deferred: Serialization support for enhanced CastExpr and CastColumnExpr is deferred to keep this PR focused on logical/physical expression unification.
• Design consideration: Ensure enhanced CastExpr doesn't become too broad; keep logical cast semantics clear while allowing physical optimization.

Future: Clean-Up and Removal

Once this issue is complete and CastExpr has been adopted everywhere:

• Archive or completely remove CastColumnExpr
• Update documentation to guide users on struct casting in SQL
• Consider whether other physical expressions could benefit from similar unification patterns