You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 699d0be
Browse filesBrowse the repository at this point in the historyBrowse files
GH-51684: [C++][Parquet] Keep the CDC chunking state across row groups (#51685)
### Rationale for this change
The content-defined chunker was owned by the column writer, which is
recreated for every row group, so the chunking state was reset at each
row group boundary. See #51684.
### What changes are included in this PR?
The file writer creates one chunker per leaf column and passes it to the
column writers of every row group through a private
`ColumnWriter::Make()` overload.
### Are these changes tested?
Yes. `IndependentOfRowGroupBoundaries` writes the same table into one
and into six row groups and expects the same page boundaries apart from
the row group ends. The first commit adds it alone, failing without the
fix.
### Are there any user-facing changes?
- Files with multiple row groups get different page boundaries, since
the chunking continues across row groups.
- `ColumnWriter::Make()` is unchanged, but column writers created
through it no longer chunk, so content-defined chunking requires
`ParquetFileWriter`.
- Multi-threaded writes with content-defined chunking are about 1.7×
slower until #51692 (9.9 ms vs 5.8 ms writing 12 int32 columns of 1M
rows on an M4 Max)
### Deduplication results
A 1 GiB file of 128Mi random int64 values (PLAIN, snappy, default CDC
options) compared with a copy that has 10 rows inserted at 20%. The
ratio is the share of bytes left after deduplicating both files with the
Xet chunker; 50% means the edited copy adds almost nothing.
| Row groups | pyarrow 25.0.1 | This PR | arrow-rs 58.4 |
|---|---|---|---|
| 8 × 16Mi rows | 50.2% | 50.1% | 50.1% |
| 32 × 4Mi rows | 50.6% | 50.4% | 50.5% |
| 128 × 1Mi rows | 52.4% | 51.6% | 51.6% |
| 512 × 256Ki rows | 59.8% | 56.4% | 56.5% |
| 2048 × 64Ki rows | 84.8% | 74.2% | 74.2% |
| Without CDC | 86.3% | 86.3% | 86.3% |
The page boundaries now match arrow-rs, which already keeps the chunker
state across row groups.
### Was AI used for this PR?
In accordance to the [AI generation
guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code),
please disclose below whether and how AI was used in this PR.
Claude Code wrote the code, tests and description under human direction.
**PR code and description written by:**
- [ ] Human
- [x] AI
**Reviewed before submission by:**
- [x] Human
- [x] AI
- [ ] Not reviewed
🤖 Generated with [Claude Code](https://claude.com/claude-code)
* GitHub Issue: #51684
0 commit comments