IPC date serialization issue

Describe the bug

When writing a RecordBatch via ipc (and then reading the stream from python) dates are not correct.
I have a date column from 2000-02-01 to 2021-11-02 on rust side but once converted to ipc pyarrow table, I see dates from 1998-01-16 to 2019-11-12. Dates are saved in a Date64Array.
Writing the same recordbatch as csv displays the correct dates.

To Reproduce

I can't really share the code but here is the rust->python (using pyo3) part.

fn df<'a>(py: Python<'a>, batch: &RecordBatch) -> Result<&'a PyAny, Error> {
    // rust ipc write
    let buf: Vec<u8> = Vec::new();
    let mut writer = StreamWriter::try_new(buf, &batch.schema())?;
    writer.write(batch)?;
    writer.finish()?;
    let buf = writer.into_inner()?;

    // python reading
    let bytes = PyBytes::new(py, &buf);
    let locals = PyDict::new(py);
    locals.set_item("buf", bytes)?;
    locals.set_item("pa", PyModule::import(py, "pyarrow")?)?;
    locals.set_item("pd", PyModule::import(py, "pandas")?)?;
    py.run(
        r#"
reader = pa.ipc.open_stream(buf)
batches = [b for b in reader]
table = pa.Table.from_batches(batches) # table.column("date") shows wrong values
df = table.to_pandas(split_blocks=True, self_destruct=True)
df["date"] = pd.to_datetime(df["date"])
df.set_index("date", inplace=True)
    "#,
        Some(&locals),
        None,
    )?;

    Ok(locals.get_item("df").unwrap())
}

Expected behavior

Same dates

Additional context

I am new to arrow so there is a high chance I'm doing something wrong

Note that when using pyarrow feature (which I just discovered reading through the code, is it documented??), this is working fine.

@tafia

@tafia can you possibly provide an example of what is in the batch: &RecordBatch argument? Specifically, what is the exact type (DataType::Timestamp(..))? If so, what is the unit?

Ideally would be some example.

i'll close this since we don't have enough information for a reproduction

for what its worth i tested quickly on current main:

    #[test]
    fn test123() {
        let array = Arc::new(Date64Array::from(vec![949363200000, 1635811200000])) as ArrayRef;
        dbg!(&array);

        let batch = RecordBatch::try_from_iter(vec![("a", array)]).unwrap();

        let stream_data = {
            let mut writer = StreamWriter::try_new(vec![], &batch.schema()).unwrap();
            writer.write(&batch).unwrap();
            writer.into_inner().unwrap()
        };
        std::fs::write("output.bin", &stream_data).unwrap();
    }

which outputs

running 1 test
[arrow-ipc/src/writer.rs:4587:9] &array = PrimitiveArray<Date64>
[
  2000-02-01,
  2021-11-02,
]
test writer::tests::test123 ... ok

and read it in with pyarrow:

>>> import pyarrow as pa
>>> with open("output.bin", "rb") as f:
...     file_bytes = f.read()
...
>>> f
<_io.BufferedReader name='output.bin'>
>>> with pa.ipc.open_stream(file_bytes) as reader:
...     schema = reader.schema
...     batches = [b for b in reader]
...
>>> schema
a: date64[ms] not null
>>> batches
[pyarrow.RecordBatch
a: date64[ms] not null
----
a: [2000-02-01,2021-11-02]]
>>> pa.__version__
'24.0.0'

so its not a glaring bug. feel free to comment again or raise a new issue if this still needs to be resolved

Navigation Menu

Repository navigation

IPC date serialization issue #902

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

IPC date serialization issue #902

Description

Activity

tafia commented on Nov 2, 2021

alamb commented on Nov 4, 2021

Jefffrey commented on Jul 2, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions