Skip to content

Panic writing a Dictionary(_, Utf8View) typed column to Parquet #10830

Closed

Description

@adamreeve

Describe the bug

This is encountered when writing a Polars Categorical typed column to a Delta Lake table with delta-rs (see delta-io/delta-rs#3284). Polars uses a Dictionary(UInt32, Utf8View) for categorical columns, and attempting to write these columns to Parquet results in a panic: internal error: entered unreachable code: cannot downcast Utf8View dictionary value to byte array

To Reproduce

use std::sync::Arc;

use arrow_array::{Array, ArrayRef, DictionaryArray, RecordBatch, StringViewArray, UInt32Array};
use arrow_schema::{DataType, Field, Schema};
use parquet::arrow::ArrowWriter;

fn main() {
    let values = StringViewArray::from(vec!["category_value_0", "category_value_1"]);
    let keys = UInt32Array::from(vec![0u32, 1, 0, 1]);
    let dict = DictionaryArray::<arrow_array::types::UInt32Type>::try_new(keys, Arc::new(values))
        .unwrap();

    let schema = Arc::new(Schema::new(vec![Field::new(
        "cat",
        DataType::Dictionary(Box::new(DataType::UInt32), Box::new(DataType::Utf8View)),
        false,
    )]));

    let batch = RecordBatch::try_new(schema.clone(), vec![Arc::new(dict) as ArrayRef]).unwrap();

    let mut buf = Vec::new();
    let mut writer = ArrowWriter::try_new(&mut buf, schema, None).unwrap();
    writer.write(&batch).unwrap(); // panic
    writer.close().unwrap();
}

This panics with something like:

internal error: entered unreachable code: cannot downcast Utf8View dictionary value to byte array
thread 'test_utf8view_dict_write' (373219) panicked at parquet/src/arrow/arrow_writer/byte_array.rs:471:9:

Expected behavior

Successfully write this data to Parquet as a string logical-typed column.

Additional context

No response

Activity

  1. added theissue type on Aug 25, 2026
  2. self-assigned this
    on Aug 25, 2026
  3. added a commit that references this issue on Aug 27, 2026
    b266024
  4. added a commit that references this issue on Sep 2, 2026
    aefca11
  5. added 3 commits that reference this issue on Sep 18, 2026
    2ba166f
    b8686ac
    1fcc70c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions