You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Flink: Add unit test to guarantee v2/v1 table without any deletes won't project to exclude the meta-columns. #3431
I think we need to provide a unit test to address the point that if we don't have any eq-deletes or pos-deletes in the task then the best approach is skipping the RowData projection removing the extra columns. In other word :
If it's iceberg v1 table, then we need to assert that the iterator won't do any extra projection to exclude the meta-columns;
if it's iceberg v2 table and there are some eq-deletes or pos-deletes inside it, then we must guarantee that it is projecting the columns to exclude the extra meta columns.
The correct approach to address the above cases in unit tests is : we create a separate unit case with an iceberg table, the iceberg table can be v1 table or v2 table. For the v1 table, we just write few records into it. Finally we use the FlinkInputFormat or DataIterator to open this table and assert the returned RowDatas are not a RowDataProjection type. For the v2 table, the difference is the returned RowData is a RowDataProjection type.
Activity
openinx commentedon Mar 29, 2022
Closed via #3240.