GH-3817: Content defined chunking of data pages by kszucs · Pull Request #1 · kszucs/parquet-java

kszucs · 2026-09-22T12:27:08Z

Draft on my fork for review before opening upstream.

Rationale for this change

Parquet picks data page boundaries by position, so inserting or deleting a row shifts every later page and a content addressable store (e.g. Hugging Face Xet) has to re-upload the whole file. Arrow C++ and arrow-rs already support content defined chunking; parquet-java has no equivalent.

With 400k rows and 100 inserted near the start, a gear-hash chunker over the file bytes shares 43.5% of the two files with ZSTD and 36.8% with Snappy, against 0.0% and 4.4% without chunking (50% is the ceiling). Measured with a standalone script, not part of this PR.

What changes are included in this PR?

A port of the Arrow C++ chunker: same gear tables, mask derivation, defaults and cut rules, so page boundaries match Arrow C++'s.
CdcOptions, and withContentDefinedChunking(boolean) / withContentDefinedChunking(CdcOptions) on ParquetProperties and ParquetWriter.Builder; parquet.page.content-defined-chunking.* keys in ParquetOutputFormat.
ChunkingColumnWriter wraps the column writer only when chunking is enabled, so the default write path is unchanged.
CdcWriteBenchmarks for the write cost.

Known limitations: row group boundaries stay position based; the default 20 000-row page limit cuts before a default-sized chunk, so it needs raising; a small minimum chunk size can cost a column its dictionary encoding.

Are these changes tested?

Yes: gear tables regenerated from their MD5 spec, page boundaries checked against vectors produced by Arrow C++, and end-to-end writer tests across both page versions, multiple row groups and every primitive type.

Are there any user-facing changes?

New opt-in writer options, off by default, documented in parquet-hadoop/README.md.

Closes apache#3817

…pache#3814) The 2026-09-16 Cancel RC run for 1.19.0 rc9 failed with HTTP 401 because release-cancel-rc.yml still read secrets.PARQUET_NEXUS_USER / PARQUET_NEXUS_PASSWORD, which do not exist on apache/parquet-java. apache#3776 moved prepare-rc and publish to the ASF org-shared NEXUS_STAGE_DEPLOYER_USER / NEXUS_STAGE_DEPLOYER_PW but missed cancel-rc, the last two references in the tree. Nothing was cancelled: verification is step 0, so the run aborted before dropping the staging repo or deleting the SVN artifacts. An absent secret expands to the empty string rather than failing, so the credentials were present-but-blank and the first authenticated curl reported only "error: 401". Add require_env and call it from all three release scripts when DRY_RUN is not 1, so a missing credential is named before any network call. Dry runs stay usable without secrets because every authenticated call is already gated on DRY_RUN. _maven.sh had this check for the deploy path only, which is why prepare-rc's Nexus 401 was equally opaque. Also apply apache#3787's tag-push ordering to publish-release.sh. It pushed the final tag in step 3 and released the staging repo in step 4, so a Nexus failure left a published release tag and the "Final release tag already exists" guard then blocked the retry -- the same trap apache#3787 removed from prepare-rc, where eight orphaned 1.19.0 RC tags accumulated. Keep git tag -a in step 3 for the local reads that follow and move only the push to step 5, immediately before the GitHub release that needs it on the remote. Generated-by: Claude Opus 5 (1M context)

…ache#3750) ### Rationale for this change Protobuf allows empty message definitions, but Parquet forbids empty groups. Converting a message that merely *contains* a field of an empty message type produces a schema with an empty group, which writer construction rejects with `InvalidSchemaException: Cannot write a schema with an empty group`. Such fields appear in real-world schemas (deprecated stubs, marker/placeholder messages), and a single one makes the whole message type unwritable. ### What changes are included in this PR? `ProtoSchemaConverter.addMessageField` terminates a field whose message type has no fields as a `BINARY` column holding the serialized message — the same mechanism PARQUET-1711 uses for recursion beyond `maxRecursion`. Since an empty message serializes to zero bytes, the column is cheap, and field **presence** still round-trips (null = unset vs empty bytes = set): - singular field → `optional binary stub` (or `required`, per the field); - repeated field, parquet-specs mode → LIST-wrapped binary via the existing `addRepeatedPrimitive`, so element cardinality survives; - repeated field, old style → `repeated binary`; - map value type → `optional binary value` inside the `key_value` group (keys stay typed). `ProtoWriteSupport.createMessageWriter`'s existing truncated-field check (primitive BINARY where a message field was declared → `BinaryWriter`) is generalized to look through the LIST/MAP wrapper (`getGroupType` → `getContentType`), so the writer tree lines up with these schemas. A message that is empty at the **root** is still rejected — there is no parent field to hold the bytes, and a Parquet file with zero columns is not representable. **Read side** (second commit, after review): `ProtoMessageConverter` used to cast every message field's Parquet type to a group, so a message field stored as `BINARY` failed with a `ClassCastException` while the converter tree was built — which also means files with PARQUET-1711 recursion truncation have been unreadable by `ProtoParquetReader` since 1.13.0 (apache#995 shipped with an explicit "TODO: ReadSupport"). A new `ProtoBinaryMessageConverter` parses the bytes back through `parentBuilder.newBuilderForField(field)` and hands the message to the existing parent container, so singular fields, LIST elements, old-style repeated fields and map values all round-trip without touching `ListConverter`/`MapConverter`. Parse failures surface as `ParquetDecodingException`. ### Are these changes tested? Yes. New `ProtoEmptyMessageTest` (new test messages `Stub`/`StubBox` in `Trees.proto`) writes through the real write path (`ProtoParquetWriter` → `MessageColumnIO`, both specs-compliant and old style) and reads back with `GroupReadSupport` and with `ProtoParquetReader`: - singular / repeated / map-value empty-message fields round-trip with correct cardinality and zero-byte values; - presence round-trips (set empty message vs unset field); - `ProtoParquetReader` reads the written messages back equal to the originals (specs-compliant and old style), and the same read path round-trips a `BinaryTree` truncated at `maxRecursion`; - an empty root message still fails with `InvalidSchemaException` ("Cannot write a schema with an empty group"). `ProtoSchemaConverterTest.testEmptyMessageFields` pins the converted schema. The full parquet-protobuf suite passes (117 tests). ### Are there any user-facing changes? Message types that previously could not be written to Parquet at all now can; fields of empty message types appear as (possibly LIST/MAP-wrapped) `binary` columns and read back into the original messages with `ProtoParquetReader`. Files with `maxRecursion` truncation, previously unreadable by `ProtoParquetReader`, now read back as well. No change for schemas that were previously writable. Error behavior for an empty root message is unchanged. Closes apache#2142

…pache#3096) ### Rationale for this change If a stream declares in its StreamCapabilities that it supports `ByteBufferPositionedReadable`, then use that API for `readFully(ByteBuffer)` ``` ByteBufferPositionedReadable.readFully(long position, ByteBuffer buf) ``` Adding support for Hadoop `ByteBufferPositionedReadable` streams may improve performance by pushing retry/recovery logic into the filesystem client library. This interface is implemented by the HDFS input stream; we are considering adding it elsewhere. ### What changes are included in this PR? * New SeekableInputStream implementation: `H3ByteBufferInputStream` * Instantiated in `HadoopStreams` if the `FSDataInputStream` is considered suitable. * Tests for the new behavior and that no regressions are caused. #### Class `H3ByteBufferInputStream` The reading is done in a new class, `H3ByteBufferInputStream`, which subclasses `H2ByteBufferInputStream`. This reduces the amount of duplicate code, it just makes it a bit unclean. The purist way to do it would be to create an abstract superclass `HadoopInputStream` to hold all commonality between the the three input streams. I'm happy to do this, just didn't want to doing some larger refactoring without (a) showing the core design worked and (b) getting permission to do this. Should I do this? #### `HadoopStreams` changes Selection of the new input stream is done if and only if the stream declares the capability `in:preadbytebuffer`. There is no equivalent of `isWrappedStreamByteBufferReadable()` which recurses through a chain of wrapped streams looking for the API. If a stream doesn't declare its support for the API, it won't get picked up. This is done knowing that the sole production implemenation which currently exists, the HDFS input stream, does declare this capability. ### Are these changes tested? There is new test suite, for new behavior and ensuring that the integration with HadoopStreams still retains the correct behavior for existing streams. Suite is parameterized on heap and direct buffers. ### Are there any user-facing changes? No Closes apacheGH-3080

…e release (apache#3757) The ASF source release is a `git archive`, so it has no .git directory. buildnumber-maven-plugin leaves buildNumber unset, and tarball builds ship `(build ${buildNumber})` and a blank `git-SHA-1` entry, so they cannot report what they were built from. Set the plugin's existing revisionOnScmFailure so they record `unknown`. Git checkouts still report the real commit. Signed-off-by: 1fanwang <[email protected]>

…pache#3810) Bumps [org.codehaus.mojo:build-helper-maven-plugin](https://github.com/mojohaus/build-helper-maven-plugin) from 3.6.1 to 3.6.2. - [Release notes](https://github.com/mojohaus/build-helper-maven-plugin/releases) - [Commits](mojohaus/build-helper-maven-plugin@3.6.1...3.6.2) --- updated-dependencies: - dependency-name: org.codehaus.mojo:build-helper-maven-plugin dependency-version: 3.6.2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…pache#3809) Bumps [org.codehaus.mojo:buildnumber-maven-plugin](https://github.com/mojohaus/buildnumber-maven-plugin) from 3.2.0 to 3.3.1. - [Release notes](https://github.com/mojohaus/buildnumber-maven-plugin/releases) - [Commits](mojohaus/buildnumber-maven-plugin@3.2.0...buildnumber-maven-plugin-3.3.1) --- updated-dependencies: - dependency-name: org.codehaus.mojo:buildnumber-maven-plugin dependency-version: 3.3.1 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action) from 0.6.3 to 0.6.4. - [Release notes](https://github.com/zizmorcore/zizmor-action/releases) - [Commits](zizmorcore/zizmor-action@70fb788...cc914d7) --- updated-dependencies: - dependency-name: zizmorcore/zizmor-action dependency-version: 0.6.4 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Widen PageReader to extend AutoCloseable with a default no-op close(), so implementations that own resources can release them through the standard try-with-resources / AutoCloseables idiom rather than a bespoke method. Existing implementations remain source- and binary-compatible. Rename ColumnChunkPageReader.releaseBuffers() to close() and let the enclosing ColumnChunkPageReadStore.close() hand its readers to AutoCloseables.uncheckedClose() directly.

…3808) Bumps [org.codehaus.mojo:exec-maven-plugin](https://github.com/mojohaus/exec-maven-plugin) from 3.6.3 to 3.6.4. - [Release notes](https://github.com/mojohaus/exec-maven-plugin/releases) - [Commits](mojohaus/exec-maven-plugin@3.6.3...3.6.4) --- updated-dependencies: - dependency-name: org.codehaus.mojo:exec-maven-plugin dependency-version: 3.6.4 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [actions/setup-java](https://github.com/actions/setup-java) from 6.0.0 to 6.0.1. - [Release notes](https://github.com/actions/setup-java/releases) - [Commits](actions/setup-java@dd06d9c...de7274f) --- updated-dependencies: - dependency-name: actions/setup-java dependency-version: 6.0.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…apache#3753) ### Rationale for this change parquet-avro supports writing both "old" and "new" list encodings via the [parquet.avro.write-old-list-structure](https://github.com/apache/parquet-java/blob/apache-parquet-1.18.0/parquet-avro/src/main/java/org/apache/parquet/avro/AvroWriteSupport.java#L72-L73) config. "old" encodings (aka "2-level"), which wrap the list in a `repeated group array` schema, are the default; "new" encodings (aka "3-level") are opt-in. On the reader side, if you're using `ParquetAvroReader` to read data that was written using `ParquetAvroWriter`, and don't specify a projection, both type sof list encoding get parsed automatically from a combination of the file schema + the `parquet.avro.schema` metadata key. There's no need to set `parquet.avro.write-old-list-structure` key in your Configuration. However, if you're either: - specifying a projection (`AvroReadSupport.setRequestedProjection(...)`), or - reading data _not_ written using ParquetAvroWriter (and thus not containing the `parquet.avro.schema` metadata key), 3-levle list encodings will not be parsed correctly - the reader will inject an extra nested record, named `element`, into the list item type. As a reader this introduces some pain, since you have to look up the underlying file metadata of the upstream Parquet file, and risk reading incorrect data. This PR attempts to automatically detect new list encodings based on the writer file schema. lmk what you think of this change. Automatic inference is always a bit risky, but I tried to be conservative with the approach (only set the list structure property if _all_ list fields in the schema use 3-level encoding; don't override `parquet.avro.write-old-list-structure` if the user is already setting it). any ideas for a better approach here are welcome - this is becoming more of a pain point as 3-level lists become a more popular option among other writer sdks. ### What changes are included in this PR? A new read configuration property `parquet.avro.read.autoDetectListStructure` (defaulting to true) that will instruct AvroReadSupport to automatically set List configuration properties based on parsing the writer file schema. ### Are these changes tested? Yes, unit tests + locally on real data. ### Are there any user-facing changes? Yes, since the new property defaults to `true` - it would impact anyone who's reading 3-level list data without setting the `parquet.avro.write-old-list-structure` key and who's relying on/working around the incorrectly formatted data (e.g. `{"locations": [{"element": {"latitude": 0.0, "longitude": 180.0}}, ...]}` instead of `{"locations": [{"latitude": 0.0, "longitude": 180.0}, ...]}` . additionally, this change also modifies the underlying Configuration object to add the properties. Closes apache#3816

The validation fix is already on master; GitHub issue 3587 stayed open because apache#3588 used Closes #apacheGH-3587. Cover the -1 default at the Util read path and document the public sentinel. Co-authored-by: Cursor Agent <[email protected]> Co-authored-by: Sankalp Thakur <[email protected]>

…he#3793)

Jackson 2.22.2, which parquet-jackson shades, is affected by CVE-2026-89407 and CVE-2026-89425 in jackson-core. Both are fixed in 2.22.3. Generated-by: Claude Code Co-authored-by: Claude Opus 5.5 <[email protected]>

* Use Java 17 instanceof assignments * Use Java 17 instanceof assignments * Use Java 17 switch idiom

apache#3925)

* apacheGH-3558: Properly close buffers * Track the buffers and close all the resources in tests * Disable verification for now * Reduce changes * Spotless * Revert some unrelated changes * Thanks Gang * Revert "Thanks Gang" This reverts commit 6a2fe5e. * Release in try-catch * Fix failing test * Revert unrelated changes * Move registering the buffer up * Revert unrelated change

…ache#3929) The prepare-RC workflow runs dev/ci-before_install.sh, which extracted and built Thrift inside the checkout. prepare-rc.sh then staged the version bump with `git add -A`, so 1.18.2-rc0's "Set version" commit picked up the whole thrift-0.25.0/ tree, compiled .o files included. `mvn deploy` also packaged that tree into the Nexus source-release zip. - Build Thrift in a temporary directory outside the checkout. - Stage only tracked files when committing version bumps. - Refuse to tag and deploy an RC from a working tree that has uncommitted changes or untracked files. Co-authored-by: Claude Opus 5.5 <[email protected]>

)

* Fix typo 'occured' to 'occurred' in parquet-column exception javadocs Signed-off-by: siddiqueirshad <[email protected]> * Fix swapped encoding and decoding descriptions in exception javadocs Signed-off-by: siddiqueirshad <[email protected]> --------- Signed-off-by: siddiqueirshad <[email protected]>

…ache#3933) prepare-rc.sh only ran versions:set, so <scm><tag> in the root POM kept whatever the release branch carried. The 1.18.2 RCs shipped apache-parquet-1.18.0-rc2 and the 1.19.0 RCs shipped HEAD. - prepare-rc.sh writes the RC tag into <scm><tag> in the release commit, including on rc1+ when the version is already set, and refuses to tag and deploy if the committed tag doesn't match. - publish-release.sh checks out the release branch before comparing it with the RC tag (the workflow is dispatched from master), and resets <scm><tag> to HEAD in the next development iteration commit.

* Introduce implementation for FileType * address * address comments and update spec * update to spec * Introduce self-reference reader/writer APIs * Add threshold-driven inline vs self-reference storage for FILE values (#1) * Add threshold-driven inline vs self-reference storage for FILE values Writers hand FILE payloads to FileValueWriter, which decides from a configured threshold whether to keep the bytes inline or store them out of line as a self-reference. Both forms describe the same logical bytes, so content_type and checksum are written identically either way and consumers see no difference beyond which fields are set. The payload is written eagerly, while the record is being written and before the row group's column chunks are flushed. This is what makes the offset knowable in time: offset and size are ordinary column values, and once a value reaches a column writer it is encoded into a buffered page and cannot be revised, so a placeholder could not be patched up later. Writing at that point also leaves each column chunk contiguous on disk, which the read path relies on when coalescing chunks into range reads. Along the way, per parquet-format#603: - Key the self-reference AAD on the file offset rather than a synthetic per-chunk counter. The spec defines AAD suffix field 6 as the offset, and the counter was not recoverable from the file: nothing stores it, so a reader had to walk preceding values to rebuild it, defeating the offset-based design that lets a value be resolved on its own. The ordinal parameter is gone from five signatures as a result. - Decompress into a dynamically sized buffer, supporting every codec. The previous stream-drain loop was broken for all codecs, not just unframed ones: NonBlockedDecompressorStream throws rather than returning -1 once its block is consumed. - Enforce the 2 GiB encrypted-module limit on write, directing oversized values to external references. - Require inline whenever offset is declared. uri is optional per value, so a uri+offset+size schema still permits self-references, and one emitted without an inline column chunk has no reference point to inherit compression and encryption from. Tests cover threshold routing, offset-keyed AAD round-trips in both GCM and CTR modes, tamper detection when an offset is altered, and payloads spanning several buffer doublings across SNAPPY, GZIP, ZSTD and LZ4_RAW. Co-authored-by: Isaac * Fix ZSTD resolution, revert schema tightening, fix test imports Verified the change by building and running the suites locally, which turned up three things: ZSTD self-references could not be resolved at all. ZstandardCodec returns null from createDecompressor because it decompresses only through its stream, so driving the Decompressor directly failed with "Could not obtain a decompressor". Such codecs are framed and report end-of-input properly, so drain the stream for them and keep grow-and-retry for the rest. Four tests were failing on this. Reverted requiring `inline` whenever `offset` is declared, back to requiring it only when `uri` is absent. Two existing tests (testFileLogicalTypeExternalRangedReferenceWithoutInline, testFileLogicalTypeOffsetWithSize) assert that a uri+offset+size schema without `inline` is valid, so the stricter rule contradicted the author's documented intent. The concern is real -- `uri` is optional per value, so such a schema can still emit a self-reference with no reference point -- but it belongs on the write path, and the FILE group declaring `uri` is now documented as needing an always-inline threshold. Fixed two pre-existing test compile errors that blocked the module: TestParquetMetadataConverter used assertEquals/assertTrue with no JUnit import (switched to the AssertJ style used throughout that file), and TestSelfReferenceFileWrite was missing the ParquetReadOptions import. Both fail on the base commit independently of these changes. Also applied spotless formatting. Local results: parquet-column 679/679 pass, parquet-hadoop 761/762. The one failure, testEnumEquivalence on Encoding.ALP, is an artifact of the local workaround for FileType being unreleased -- parquet.thrift was substituted from parquet-format master, which defines an ALP encoding the Java enum does not yet have. It is unrelated to these changes and will not occur once a parquet-format release carries FileType. Co-authored-by: Isaac * remove self reference * Add FILE value writer validation * Added FileValueWriters and FileConverters for writes and reads * save * fix lint * Prevent partial FILE projections * Fix FILE projection formatting * Harden FILE type validation * Validate FILE schema replacements and expand reader coverage Revalidate FILE fields in GroupType.withNewFields and cover duplicate names, invalid annotations, generic projections, converter lifecycle, and persisted reader/writer behavior. Apply the repository's Spotless formatting to the FILE changes. * Address FILE logical type review feedback --------- Co-authored-by: Alkis Evlogimenos <[email protected]>

Bumps [com.github.luben:zstd-jni](https://github.com/luben/zstd-jni) from 1.5.7-24 to 1.5.7-25. - [Commits](https://github.com/luben/zstd-jni/commits) --- updated-dependencies: - dependency-name: com.github.luben:zstd-jni dependency-version: 1.5.7-25 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [com.google.api.grpc:proto-google-common-protos](https://github.com/googleapis/google-cloud-java) from 2.78.0 to 2.79.0. - [Release notes](https://github.com/googleapis/google-cloud-java/releases) - [Changelog](https://github.com/googleapis/google-cloud-java/blob/main/java-document-ai/CHANGELOG.md) - [Commits](googleapis/google-cloud-java@gapic-generator-java/v2.78.0...gax/v2.79.0) --- updated-dependencies: - dependency-name: com.google.api.grpc:proto-google-common-protos dependency-version: 2.79.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Replace jbrotli Brotli codec with brotli4j for CVE-2026-111497 * Fix Brotli codec stream handling and large-page decoding * Bound Brotli decompression and fix stream contracts * Add Brotli codec regression tests for bounds and stream contracts * fix: preserve legacy Brotli codec class alias Accept the legacy Hadoop Brotli codec name while keeping the brotli4j-backed codec as the implementation. Add regression coverage for legacy and current names, other codecs, and unknown or null classes. Based on the patch suggested by Fokko in apache#3880. * fix: close failed Brotli streams and enforce output bounds Release the native encoder even after write or flush failures, retaining the original error and any cleanup errors without try-with-resources self-suppression. Reject negative decode limits and non-empty output under a zero limit. Add six regression tests and strengthen existing exception assertions.

…adoop (apache#3917) * DirectCodecFactory does not need to support past versions of Hadoop * Preserve DirectCodecFactory pool method signatures * Update parquet-hadoop/src/main/java/org/apache/parquet/hadoop/DirectCodecFactory.java Co-authored-by: Fokko Driesprong <[email protected]> * Update parquet-hadoop/src/main/java/org/apache/parquet/hadoop/DirectCodecFactory.java Co-authored-by: Fokko Driesprong <[email protected]> * Fix direct decompressor fallback log message --------- Co-authored-by: Fokko Driesprong <[email protected]>

Implements ALP encoding for FLOAT and DOUBLE types, which converts floating-point values to integers using decimal scaling, then applies Frame of Reference (FOR) encoding and bit-packing for compression. --------- Co-authored-by: Julien Le Dem <[email protected]>

Add opt-in content defined chunking (CDC) to the writer. Data page boundaries are chosen by a rolling gear hash over each column's (definition level, repetition level, value) stream instead of by position, so an insert or delete only changes the pages around it and files deduplicate in content addressable storage. The chunker is a port of Arrow C++ (parquet/chunker_internal) and arrow-rs (column/chunker/cdc.rs): same gear tables, mask derivation, defaults and cut rules. Each column keeps one chunker for the whole file, as arrow-rs does, so chunk boundaries continue across row groups instead of restarting in each. Arrow C++ builds a new chunker for every row group, contrary to its own documentation, so its chunks match ours in the first row group but generally not after it. - CdcOptions: min/max chunk size and normalization level, defaulting to 256 KiB / 1 MiB / 0 as in Arrow. - ParquetProperties and ParquetWriter.Builder: withContentDefinedChunkingEnabled(boolean) and withContentDefinedChunking(CdcOptions). - ParquetOutputFormat: parquet.page.content-defined-chunking.* keys. - ChunkingColumnWriter wraps the column writer only when chunking is enabled. It ends pages at chunk boundaries and, as Arrow C++ does, applies the page size and row count limits counted from the page start, so they cut a chunk in the same places after an edit. - The writer keeps each column's chunker for the whole file, in CdcChunkers, which makes every row group's store. - With chunking on, a column falls back from dictionary encoding only when its dictionary outgrows its size limit, as in Arrow C++, not on its first page, which chunking cuts by content. Pages of nulls alone get an empty dictionary page, as Arrow C++ writes one. - CdcWriteBenchmarks measures the write cost. Off by default, and with it off the writer behaves as before.

- Chunker state lives per file: InternalParquetRecordWriter creates the chunkers of the file up front, one per leaf column, only when chunking is enabled, as Arrow C++ and arrow-rs do, and passes them to every row group's store through a new ParquetProperties.newColumnWriteStore overload. With chunking enabled, a store made without them fails fast rather than silently restarting the chunking. CdcOptions are validated when built. - Content defined chunking is configured through one setter, withContentDefinedChunking(CdcOptions), where null disables it, and named as Arrow names it: CdcOptions, ContentDefinedChunker. - ChunkingColumnWriter wraps each column writer once, when the store is built. It rolls every triplet into the column's chunker and, at record starts, ends the page where needNewChunk() says, as Arrow C++ does, or where a page limit is reached, counted from the page start; cached nulls are flushed on the unchunked schedule's maximum. - Binary values are hashed through one path. - The tests fail when chunking is switched off or broken, cover the cases the Arrow C++ and arrow-rs suites do (V2 pages, dictionary encoding, edits across row groups, a chunker per column, the record writer path), and follow the project's test guidelines. The benchmark writes one mixed dataset.

ColumnWriterBase carries its column's ContentDefinedChunker, passed in by the store as the bloom filter writer is, instead of being wrapped by ChunkingColumnWriter. Without content defined chunking it gets a no-op chunker, as it gets no-op statistics, which the JIT inlines away, so the write path has no added branch. ContentDefinedChunker follows Arrow C++: the column writer rolls each triplet and asks needNewPage() at record starts, built on Roll-like primitives and needNewChunk(). ContentDefinedChunker.forColumns takes the writer properties, for the page limits. Written files are unchanged.

The column writer asks needNewPage(value, repetitionLevel, definitionLevel, writer) before writing each triplet and ends the page when it says so, instead of rolling the value itself and then asking. Rolling is the content defined chunker's own business, so another chunker can answer the same question its own way, and the no-op chunker inlines to a single false.

…oundary Under content defined chunking each column writer applies the page limits at every record start, so the column write store's page size check schedule, which also decides when MessageColumnIO hands its held back nulls to the column writers, must not move a page boundary. Test it through ParquetWriter and document it.

…nstructors The column writers are package private, so their constructors can take the chunker instead of gaining overloads that leave the old ones unused.

…ilter writer The column writer resolves it to the no-op chunker once, as the value collector resolves a missing bloom filter writer to a no-op bloom filter.

The chunker answers only whether a triplet starts a new chunk, as Arrow C++ and arrow-rs decide it: roll the triplet, then ask NeedNewChunk at record starts. The column writer ends its page there unless the page is empty. The column write store applies the page limits on its schedule as without content defined chunking, so the chunker no longer reads the writer's page state.

Check the page limits against a fixed size check schedule, so that a writer applying them itself would fail; pin every page end where a row count limit and the chunks interleave; cover the no-op chunker's null path. Document that page size cuts follow the size check schedule, and claim Arrow's chunking rather than Arrow's pages.

…Options A column writer without content defined chunking holds no chunker instead of a no-op subclass that had to override every method. CdcOptions checks its size range itself, and calculateMask takes the options and rejects a mask width out of range when the writer creates its chunkers, as Arrow C++ and arrow-rs do.

Spell the name out like the rest of the API (withContentDefinedChunking, getContentDefinedChunkingOptions, parquet.page.content-defined-chunking.*), since CDC more often means change data capture. The javadoc still names Arrow's CdcOptions.

As Arrow C++ checks it, align the pages of the original and the edited column by their longest common subsequence and require one changed run per edited place, holding exactly the inserted, deleted or replaced values, with every other page byte for byte the same; the first and last pages stay unless the edit is there, which a chunker that is not content defined fails on a single edit. Row groups of a fixed row count change by exactly the shifted records.