Skip to content

Support DELTA_BINARY_PACKED and DELTA_BYTE_ARRAY #574

New issue
New issue
Closed
Closed
Support DELTA_BINARY_PACKED and DELTA_BYTE_ARRAY#574
Labels
enhancementNew feature or request

Description

@kazuyukitanimura
kazuyukitanimura
opened on Jun 14, 2024

What is the problem the feature request solves?

There are some tests in Spark 4.0 that uses parquet.writer.version=v2 (ParquetTypeWideningSuite).

The V2 write writes with delta encoding. Comet currently cannot read such files

Describe the potential solution

No response

Additional context

No response

Activity

  1. kazuyukitanimura
    added
    enhancementNew feature or request
    on Jun 14, 2024

  2. parthchandra commented on Jun 15, 2024

    @parthchandra
    parthchandra
    on Jun 15, 2024
    Contributor

    IIRC, the vectorized versions of these encodings in Spark did not improve performance much over the row based implementation in the parquet library

  3. andygrove commented on May 6, 2026

    @andygrove
    andygrove
    on May 6, 2026
    Member

    Closing as superseded: the new native_datafusion and native_iceberg_compat scans (default spark.comet.scan.impl=auto) read DELTA_BINARY_PACKED / DELTA_BYTE_ARRAY / DELTA_LENGTH_BYTE_ARRAY via arrow-rs. Please reopen if there is still a need to support these encodings in the legacy native_comet scan.

  4. andygrove
    closed this as completedon May 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions