feat(parquet): add content defined chunking for arrow writer by kszucs · Pull Request #9450 · apache/arrow-rs

kszucs · 2026-02-20T15:26:52Z

Which issue does this PR close?

Closes #NNN.

Rationale for this change

Rust implementation of apache/arrow#45360

Traditional Parquet writing splits data pages at fixed sizes, so a single inserted or deleted row causes all subsequent pages to shift — resulting in nearly every byte being re-uploaded to content-addressable storage (CAS) systems. CDC determines page boundaries via a rolling gearhash over column values, so unchanged data produces identical pages across different writes enabling storage cost reductions and faster upload times.

See more details in https://huggingface.co/blog/parquet-cdc

The original C++ implementation apache/arrow#45360

Evaluation tool https://github.com/huggingface/dataset-dedupe-estimator where I already integrated this PR to verify that deduplication effectiveness is on par with parquet-cpp (lower is better):

What changes are included in this PR?

Content-defined chunker at parquet/src/column/chunker/
Arrow writer integration integrated in ArrowColumnWriter
Writer properties via CdcOptions struct (min_chunk_size, max_chunk_size, norm_level)
ColumnDescriptor: added repeated_ancestor_def_level field to for nested field values iteration

Are these changes tested?

Yes — unit tests are located in cdc.rs and ported from the C++ implementation.

Are there any user-facing changes?

New experimental API, disabled by default — no behavior change for existing code:

// Simple toggle (256 KiB min, 1 MiB max, norm_level 0)
let props = WriterProperties::builder()
    .set_content_defined_chunking(true)
    .build();

// Excpliti CDC parameters
let props = WriterProperties::builder()
    .set_cdc_options(CdcOptions { min_chunk_size: 128 * 1024, max_chunk_size: 512 * 1024, norm_level: 1 })
    .build();

kszucs · 2026-02-25T11:27:02Z

We don't necessarily need to store the codegen script in the repository. Alternatively we could just reference https://github.com/apache/arrow/blob/main/cpp/src/parquet/chunker_internal_generated.h as a source for cdc_generated.rs. Likely it won't be regenerated at all.

I think it is fine to check this in

kszucs · 2026-03-06T11:53:29Z

@alamb @etseidl could you please take a look? Let me know if you need extra context or if you have limited bandwidth.

etseidl · 2026-03-06T16:12:12Z

Hi @kszucs. 👋 Apologies, I have been unusually bandwidth constrained lately. I will try to give this a good look in the next few days. Thank you for your patience 🙏 (and for adding this to arrow-rs).

kszucs · 2026-03-06T16:16:11Z

Hi @etseidl! No worries, I really appreciate you taking the time to review!

etseidl

Flushing a few early observations/questions. Still need to do the deep dive.

etseidl

Just a few more random comments...I'll get into the meat of the chunking tomorrow. Looking good so far!

…` and use it in content defined chunking

…nments

… chunks

Co-authored-by: Ed Seidl <[email protected]>

alamb · 2026-03-18T20:20:12Z

I think there is something wrong with the tests in this PR.

Specifically, I did an ablation study (what a fancy word!) to verify the test coverage in this PR.

I disconnected all the CDC code from the writer like this

TESTING: Disconnect all cdc logic kszucs/arrow-rs#1

Here is how I ran the tests

cargo test -p parquet --features=arrow -- cdc

Almost all of them still pass (only 3 fail). which suggests they aren't actually testing the CDC code

running 35 tests
test column::chunker::cdc::arrow_tests::test_cdc_find_differences ... ok
test column::chunker::cdc::arrow_tests::test_cdc_empty_table ... ok
test column::chunker::cdc::arrow_tests::test_cdc_f64_column ... ok
test column::chunker::cdc::arrow_tests::test_cdc_array_offsets ... FAILED
test column::chunker::cdc::arrow_tests::test_cdc_prepend ... ok
test column::chunker::cdc::arrow_tests::test_cdc_append ... ok
test column::chunker::cdc::arrow_tests::test_cdc_insert_once ... ok
test column::chunker::cdc::arrow_tests::test_cdc_delete_once ... ok
test column::chunker::cdc::arrow_tests::test_cdc_delete_twice ... FAILED
test column::chunker::cdc::arrow_tests::test_cdc_insert_twice ... FAILED
test column::chunker::cdc::arrow_tests::test_cdc_multiple_row_groups_insert ... ok
test column::chunker::cdc::arrow_tests::test_cdc_roundtrip_dictionary ... ok
test column::chunker::cdc::arrow_tests::test_cdc_multiple_row_groups_append ... ok
test column::chunker::cdc::arrow_tests::test_cdc_roundtrip_large_binary ... ok
test column::chunker::cdc::arrow_tests::test_cdc_deterministic ... ok
test column::chunker::cdc::tests::test_calculate_mask_defaults ... ok
test column::chunker::cdc::tests::test_calculate_mask_invalid ... ok
test column::chunker::cdc::tests::test_calculate_mask_with_norm_level ... ok
test column::chunker::cdc::arrow_tests::test_cdc_roundtrip_string ... ok
test column::chunker::cdc::tests::test_non_nested_non_null_single_chunk ... ok
test column::chunker::cdc::tests::test_nullable_non_nested ... ok
test column::chunker::cdc::arrow_tests::test_cdc_roundtrip_nullable ... ok
test column::chunker::cdc::tests::test_deterministic_chunks ... ok
test column::chunker::cdc::tests::test_max_chunk_size_forces_boundary ... ok
test column::chunker::cdc::arrow_tests::test_cdc_roundtrip_i32 ... ok
test column::chunker::cdc::arrow_tests::test_cdc_roundtrip_list ... ok
test column::chunker::cdc::arrow_tests::test_cdc_update_once ... ok
test column::chunker::cdc::arrow_tests::test_cdc_nullable_column ... ok
test column::chunker::cdc::arrow_tests::test_cdc_string_column ... ok
test column::chunker::cdc::arrow_tests::test_cdc_update_twice ... ok
test column::chunker::cdc::arrow_tests::test_cdc_array_offsets_direct ... ok
test column::chunker::cdc::arrow_tests::test_cdc_roundtrip_multiple_columns ... ok
test column::chunker::cdc::arrow_tests::test_cdc_produces_multiple_pages ... ok
test column::chunker::cdc::arrow_tests::test_cdc_page_reuse_on_append ... ok
test column::chunker::cdc::arrow_tests::test_cdc_state_persists_across_row_groups ... ok

failures:

---- column::chunker::cdc::arrow_tests::test_cdc_array_offsets stdout ----

thread 'column::chunker::cdc::arrow_tests::test_cdc_array_offsets' (9668022) panicked at parquet/src/column/chunker/cdc.rs:1668:9:
assertion `left == right` failed: sliced first page should be 10 values shorter: full=[20480, 12288] sliced=[20480, 12278]
  left: 0
 right: 10
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- column::chunker::cdc::arrow_tests::test_cdc_delete_twice stdout ----

thread 'column::chunker::cdc::arrow_tests::test_cdc_delete_twice' (9668025) panicked at parquet/src/column/chunker/cdc.rs:1574:9:
assertion `left == right` failed: Expected 2 diffs for double delete, got [([16448], [16384])]
  left: 1
 right: 2

---- column::chunker::cdc::arrow_tests::test_cdc_insert_twice stdout ----

thread 'column::chunker::cdc::arrow_tests::test_cdc_insert_twice' (9668031) panicked at parquet/src/column/chunker/cdc.rs:1613:9:
assertion `left == right` failed: Expected 2 diffs for double insert, got [([16384], [16448])]
  left: 1
 right: 2


failures:
    column::chunker::cdc::arrow_tests::test_cdc_array_offsets
    column::chunker::cdc::arrow_tests::test_cdc_delete_twice
    column::chunker::cdc::arrow_tests::test_cdc_insert_twice

test result: FAILED. 32 passed; 3 failed; 0 ignored; 0 measured; 885 filtered out; finished in 0.66s

alamb · 2026-03-18T20:22:52Z

For this PR I think we need an "End to end" test that shows the usecase that the CDC code is intended to solve

For example, perhaps such a test can write two parquet files, with the same data except for some chosen rows in the middle , and verify that most of the pages are the same.

It is not entirely clear to me how a "content addressable filesystem" works (aka how does it know where the parquet pages start/end) so having that documented / mocked out would also be nice

alamb · 2026-03-18T20:23:03Z

Thank you for this work @kszucs and @etseidl

alamb

Thank you

alamb · 2026-03-18T20:27:14Z

+        Ok(())
+    }
+
+    fn write_with_chunkers(


having two code paths (write and write_with_chunkers) is kind of weird to me and seems inconisistent with other optional features like encoding or compression

I wonder if it would encapsulate the code more if we extended write with a Option<&[ContentDefinedChunke]) rather than have two separate external functions

You tell me :)
I merely tried to avoid any breaking changes.

I can tinker as a follow on PR -- I don't think any changes are needed here

alamb · 2026-03-18T20:27:34Z

I think it is fine to check this in

kszucs · 2026-03-18T20:30:07Z

It is not entirely clear to me how a "content addressable filesystem" works (aka how does it know where the parquet pages start/end) so having that documented / mocked out would also be nice

The CDC feature in parquet essentially splits pages according to the columns' content resulting in fairly stable pages even if there are insterted deleted records.

The HF xet filesystem is format agnostic (similarly to for example a deduplicating backup solution like restic) and chunks the byte stream directly. The main issue with parquet is the page level compression which break the deduplication if the page values change - this CDC feature makes the pages more or less stable depending on theit content.

BTW using this feature anyone could implement a "parquet page store" storing only unique parquet pages and some metadata to reassemble the parquet files.

alamb · 2026-03-18T20:39:39Z

BTW using this feature anyone could implement a "parquet page store" storing only unique parquet pages and some metadata to reassemble the parquet files.

Is this easy to show? I realize this is an important usecase for hugging face, but it would be nice to have some example how this could be used by others that are not using the xet filesystem

kszucs · 2026-03-18T20:55:46Z

BTW using this feature anyone could implement a "parquet page store" storing only unique parquet pages and some metadata to reassemble the parquet files.

Is this easy to show? I realize this is an important usecase for hugging face, but it would be nice to have some example how this could be used by others that are not using the xet filesystem

I have been thinking of a page store prototype for a while actually, that would kinda look like:

iterate over the parquet pages using a page reader
use a hash function to assign a unique key to the page based on its content, like xxhash, shar, blake (this is different from the gearhash since chunking is already done by the parquet writer)
write out the page to a hashtable like storage system like kv store, object store, but really depends on the use case
maintain the necessary metadata to reassemble the original parquet file from the stored pages

A format agnostic CAS is different since it does the chunking on the byte stream directly. I have a naive and very simple implementation for that here https://github.com/huggingface/dataset-dedupe-estimator/blob/main/src/store.rs

kszucs · 2026-03-18T20:59:16Z

I'm updating the testing suite to closely match the C++ ones.

kszucs · 2026-03-18T21:19:43Z

@alamb with CDC disabled at the writer level now 85 tests fail from 101 (16 is testing the testing utilities and empty tables).

alamb · 2026-03-20T13:54:27Z

@alamb with CDC disabled at the writer level now 85 tests fail from 101 (16 is testing the testing utilities and empty tables).

Thanks!

alamb · 2026-03-20T13:54:49Z

BTW using this feature anyone could implement a "parquet page store" storing only unique parquet pages and some metadata to reassemble the parquet files.

Is this easy to show? I realize this is an important usecase for hugging face, but it would be nice to have some example how this could be used by others that are not using the xet filesystem

I have been thinking of a page store prototype for a while actually, that would kinda look like:

iterate over the parquet pages using a page reader

use a hash function to assign a unique key to the page based on its content, like xxhash, shar, blake (this is different from the gearhash since chunking is already done by the parquet writer)

write out the page to a hashtable like storage system like kv store, object store, but really depends on the use case

maintain the necessary metadata to reassemble the original parquet file from the stored pages

A format agnostic CAS is different since it does the chunking on the byte stream directly. I have a naive and very simple implementation for that here https://github.com/huggingface/dataset-dedupe-estimator/blob/main/src/store.rs

I filed a ticket to track this idea so it doesn't get lost on a old PR

Create a "end to end" Content Addressable Storage / CDC Chunking Example #9592

alamb · 2026-03-20T13:55:05Z

Thank you very much @kszucs and @etseidl 🙏

kszucs · 2026-03-20T14:05:07Z

Thank you @alamb and @etseidl! Soon adding a config option to datafusion as well and combined with apache/opendal#7185 (and object_store_opendal) datafusion on huggingface will provide a pretty good performance!

alamb · 2026-03-31T20:26:28Z

FWIW I believe I have found a bug in this PR (writing nested lists!):

[Parquet] ArrowWriter with CDC panics on nested ListArrays #9637

(It is not a regression, but just FYI)

kszucs · 2026-03-31T22:10:33Z

Thanks @alamb for catching it, looking at it tomorrow!

## Rationale for this change - closes #21110 Expose the new Content-Defined Chunking feature from parquet-rs apache/arrow-rs#9450 ## What changes are included in this PR? New parquet writer options for enabling CDC. ## Are these changes tested? In-progress. ## Are there any user-facing changes? New config options. Depends on the 58.1 arrow-rs release.

…e#21110) ## Rationale for this change - closes apache#21110 Expose the new Content-Defined Chunking feature from parquet-rs apache/arrow-rs#9450 ## What changes are included in this PR? New parquet writer options for enabling CDC. ## Are these changes tested? In-progress. ## Are there any user-facing changes? New config options. Depends on the 58.1 arrow-rs release.

github-actions Bot added the parquet Changes to the parquet crate label Feb 20, 2026

kszucs marked this pull request as ready for review February 25, 2026 08:12

kszucs requested review from alamb and etseidl February 25, 2026 11:19

kszucs commented Feb 25, 2026

View reviewed changes

Comment thread parquet/src/schema/types.rs Resolved

kszucs commented Feb 25, 2026

View reviewed changes

etseidl reviewed Mar 6, 2026

View reviewed changes

etseidl reviewed Mar 10, 2026

View reviewed changes

Comment thread parquet/src/arrow/arrow_writer/mod.rs Resolved

Comment thread parquet/src/file/properties.rs Resolved

Comment thread parquet/src/file/properties.rs Resolved

Comment thread parquet/src/file/properties.rs Resolved

kszucs and others added 19 commits March 16, 2026 11:48

feat(parquet): add content defined chunking for arrow writer

b66d452

feat(parquet): add repeated_ancestor_def_level to `ColumnDescriptor…

26364c5

…` and use it in content defined chunking

chore: cargo format

cf48df7

chore: fix clippy errors

b05da4d

refactor(parquet): maintain field for better encapsulation

ea0e344

refactor(parquet): simplify the CDC implementation

04711e2

refactor(parquet): hold the cdc chunkers in ArrowWriter

c2b31ff

chore(parquet): remove redundant flush_current_page() method

ad4d2c6

doc(parquet): remove content defined chunking example from dosctrings

2553575

chore(parquet): remove unnecessary mut row_group_writer_factory assig…

5facebb

…nments

fix(parquet): incorporate primitive array offset when calculating cdc…

b25f206

… chunks

chore(parquet): add benchmark for cdc chunking

a699aef

chore(parquet): fix clippy errors

94d2efc

refactor(parquet): do not store the chunker in the row group writer

caef92e

chore(parquet): spell out cdc as content_defined_chunking in properties

947bfdf

chore(parquet): apply suggestions from code review

7622f22

Co-authored-by: Ed Seidl <[email protected]>

chore: address review comments

3f087aa

chore: address review comments

255bec8

chore: add constants for default cdc parameters

7094008

alamb mentioned this pull request Mar 18, 2026

TESTING: Disconnect all cdc logic kszucs/arrow-rs#1

Draft

alamb reviewed Mar 18, 2026

View reviewed changes

test(parquet): closely port the CDC tests from the C++ implementation

8dc0e5b

alamb mentioned this pull request Mar 20, 2026

Create a "end to end" Content Addressable Storage / CDC Chunking Example #9592

Open

alamb merged commit bc74c71 into apache:main Mar 20, 2026
16 checks passed

kszucs mentioned this pull request Mar 23, 2026

feat: add support for parquet content defined chunking options apache/datafusion#21110

Merged

alamb mentioned this pull request Mar 31, 2026

[Parquet] ArrowWriter with CDC panics on nested ListArrays #9637

Closed

kszucs mentioned this pull request Apr 1, 2026

fix(parquet): fix CDC panic on nested ListArrays with null entries kszucs/arrow-rs#2

Closed

alamb mentioned this pull request Apr 6, 2026

Support parquet content-defined chunking options apache/datafusion#21408

Open

kszucs mentioned this pull request Sep 24, 2026

Support content-defined chunking (CDC) in the Parquet writer apache/parquet-java#3817

Open

roeap mentioned this pull request Oct 25, 2026

docs(blogs): draft parquet-is-engine-territory (Kernel Inside #3) open-lakehouse/docs-factory#361

Merged

sunchao mentioned this pull request Oct 28, 2026

GH-3817: Content defined chunking of data pages apache/parquet-java#3818

Open

kszucs mentioned this pull request Nov 5, 2026

feat(parquet/pqarrow): add content-defined chunking apache/arrow-go#1469

Draft

Conversation

kszucs commented Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Uh oh!

kszucs Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

kszucs commented Mar 6, 2026

Uh oh!

etseidl commented Mar 6, 2026

Uh oh!

kszucs commented Mar 6, 2026

Uh oh!

etseidl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

etseidl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alamb commented Mar 18, 2026

Uh oh!

alamb commented Mar 18, 2026

Uh oh!

alamb commented Mar 18, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

kszucs Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kszucs commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb commented Mar 18, 2026

Uh oh!

kszucs commented Mar 18, 2026

Uh oh!

kszucs commented Mar 18, 2026

Uh oh!

kszucs commented Mar 18, 2026

Uh oh!

alamb commented Mar 20, 2026

Uh oh!

alamb commented Mar 20, 2026

Uh oh!

Uh oh!

alamb commented Mar 20, 2026

Uh oh!

kszucs commented Mar 20, 2026

kszucs commented Feb 20, 2026 •

edited

Loading

kszucs Feb 25, 2026 •

edited

Loading

kszucs commented Mar 18, 2026 •

edited

Loading