Skip to content

Flink: Add unit test to guarantee v2/v1 table without any deletes won't project to exclude the meta-columns. #3431

Closed

Description

@openinx

Copied from this comment

I think we need to provide a unit test to address the point that if we don't have any eq-deletes or pos-deletes in the task then the best approach is skipping the RowData projection removing the extra columns. In other word :

  • If it's iceberg v1 table, then we need to assert that the iterator won't do any extra projection to exclude the meta-columns;
  • if it's iceberg v2 table and there are some eq-deletes or pos-deletes inside it, then we must guarantee that it is projecting the columns to exclude the extra meta columns.

The correct approach to address the above cases in unit tests is : we create a separate unit case with an iceberg table, the iceberg table can be v1 table or v2 table. For the v1 table, we just write few records into it. Finally we use the FlinkInputFormat or DataIterator to open this table and assert the returned RowDatas are not a RowDataProjection type. For the v2 table, the difference is the returned RowData is a RowDataProjection type.

Activity

  1. openinx commented on Mar 29, 2022

    @openinx
    MemberAuthor

    Closed via #3240.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions