[Python] Legacy dataset can't roundtrip Int64 with nulls if partitioned

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major
    • None
    • Affects Version/s: 4.0.0, 7.0.0
    • Component/s: Python
    • None

      If there is partitioning and the column has nulls, Int64 columns may not round trip successfully using the legacy datasets implementation. 

      Simple reproduction:

       

      import pyarrow as pa
      import pyarrow.parquet as pq
      import pyarrow.dataset as ds
      import tempfile
      
      table = pa.table({
          'x': pa.array([None, 7753285016841556620]),
          'y': pa.array(['a', 'b'])
      })
      
      ds_dir = tempfile.mkdtemp()
      pq.write_to_dataset(table, ds_dir, partition_cols=['y'])
      
      table_after = ds.dataset(ds_dir).to_table()
      print(table['x'])
      print(table_after['x'])
      assert table['x'] == table_after['x']
      
      [
        [
          null,
          7753285016841556620
        ]
      ]
      [
        [
          null
        ],
        [
          7753285016841556992
        ]
      ]
      

            Assignee:
            Unassigned
            Reporter:
            Will Jones
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: