Skip to content

Support pyarrow 12.0.1 for repartitioning  #167

Closed

Description

@raghumdani

Currently repartitioning has two issues:

  1. The stream position set in the delta is current epoch which leads to race condition and correctness issue.
  2. The filter expression in repartitioning uses float("inf") that is unsupported in pyarrow 12.0.1.

Hence, as part of this change, we will fix both the bugs.

Epic: #150

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions