Skip to content

Support writing IntervalMonthDayNanoArray to parquet via Arrow Writer  #5849

New issue
New issue
Closed
#5875
Closed
Support writing IntervalMonthDayNanoArray to parquet via Arrow Writer #5849
#5875
Labels
enhancementAny new improvement worthy of a entry in the changelogparquetChanges to the parquet crate
@marvinlanhenke

Description

@marvinlanhenke
marvinlanhenke
opened on Jun 6, 2024

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

While working on the support for converting parquet statistics into ArrayRefs in DataFusion (see apache/datafusion#10453).
I noticed that currently the ColumnWriter does not support writing IntervalUnit::MonthDayNano.

This might be the location:

arrow-rs/parquet/src/arrow/arrow_writer/mod.rs

Lines 854 to 874 in fa8d350

ColumnWriter::FixedLenByteArrayColumnWriter(ref mut typed) => {
let bytes = match column.data_type() {
ArrowDataType::Interval(interval_unit) => match interval_unit {
IntervalUnit::YearMonth => {
let array = column
.as_any()
.downcast_ref::<arrow_array::IntervalYearMonthArray>()
.unwrap();
get_interval_ym_array_slice(array, indices)
}
IntervalUnit::DayTime => {
let array = column
.as_any()
.downcast_ref::<arrow_array::IntervalDayTimeArray>()
.unwrap();
get_interval_dt_array_slice(array, indices)
}
_ => {
return Err(ParquetError::NYI(
format!(
"Attempting to write an Arrow interval type {interval_unit:?} to parquet that is not yet implemented"

Describe the solution you'd like

Support for writing IntervalUnit::MonthDayNano in the ColumnWriter.

Describe alternatives you've considered

Additional context

Related to: #5847

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementAny new improvement worthy of a entry in the changelogparquetChanges to the parquet crate

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions