Currently repartitioning has two issues: 1. The stream position set in the delta is current epoch which leads to race condition and correctness issue. 2. The filter expression in repartitioning uses float("inf") that is unsupported in pyarrow 12.0.1. Hence, as part of this change, we will fix both the bugs. Epic: #150
Activity
Fix stream position and support latest pyarrow
Fix stream position and support latest pyarrow (#168)
Workaround for pydantic 2.0 incompatibility, new AWS glue job runner,…
Fix stream position and support latest pyarrow (ray-project#168)