Skip to content

Commit 699d0be

Browse files
kszucsraulcd
authored andcommitted
GH-51684: [C++][Parquet] Keep the CDC chunking state across row groups (#51685)
### Rationale for this change The content-defined chunker was owned by the column writer, which is recreated for every row group, so the chunking state was reset at each row group boundary. See #51684. ### What changes are included in this PR? The file writer creates one chunker per leaf column and passes it to the column writers of every row group through a private `ColumnWriter::Make()` overload. ### Are these changes tested? Yes. `IndependentOfRowGroupBoundaries` writes the same table into one and into six row groups and expects the same page boundaries apart from the row group ends. The first commit adds it alone, failing without the fix. ### Are there any user-facing changes? - Files with multiple row groups get different page boundaries, since the chunking continues across row groups. - `ColumnWriter::Make()` is unchanged, but column writers created through it no longer chunk, so content-defined chunking requires `ParquetFileWriter`. - Multi-threaded writes with content-defined chunking are about 1.7× slower until #51692 (9.9 ms vs 5.8 ms writing 12 int32 columns of 1M rows on an M4 Max) ### Deduplication results A 1 GiB file of 128Mi random int64 values (PLAIN, snappy, default CDC options) compared with a copy that has 10 rows inserted at 20%. The ratio is the share of bytes left after deduplicating both files with the Xet chunker; 50% means the edited copy adds almost nothing. | Row groups | pyarrow 25.0.1 | This PR | arrow-rs 58.4 | |---|---|---|---| | 8 × 16Mi rows | 50.2% | 50.1% | 50.1% | | 32 × 4Mi rows | 50.6% | 50.4% | 50.5% | | 128 × 1Mi rows | 52.4% | 51.6% | 51.6% | | 512 × 256Ki rows | 59.8% | 56.4% | 56.5% | | 2048 × 64Ki rows | 84.8% | 74.2% | 74.2% | | Without CDC | 86.3% | 86.3% | 86.3% | The page boundaries now match arrow-rs, which already keeps the chunker state across row groups. ### Was AI used for this PR? In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR. Claude Code wrote the code, tests and description under human direction. **PR code and description written by:** - [ ] Human - [x] AI **Reviewed before submission by:** - [x] Human - [x] AI - [ ] Not reviewed 🤖 Generated with [Claude Code](https://claude.com/claude-code) * GitHub Issue: #51684
1 parent 7554a8e commit 699d0be

6 files changed

Lines changed: 139 additions & 43 deletions

File tree

‎cpp/src/parquet/chunker_internal.cc‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -392,8 +392,8 @@ class ContentDefinedChunker::Impl {
392392
}
393393

394394
private:
395-
// Reference to the column's level information
396-
const internal::LevelInfo& level_info_;
395+
// The column's level information
396+
const internal::LevelInfo level_info_;
397397
// Minimum chunk size in bytes, the rolling hash will not be updated until this size is
398398
// reached for each chunk. Note that all data sent through the hash function is counted
399399
// towards the chunk size, including definition and repetition levels.
@@ -422,6 +422,7 @@ ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
422422
int64_t max_chunk_size, int norm_level)
423423
: impl_(new Impl(level_info, min_chunk_size, max_chunk_size, norm_level)) {}
424424

425+
ContentDefinedChunker::ContentDefinedChunker(ContentDefinedChunker&&) noexcept = default;
425426
ContentDefinedChunker::~ContentDefinedChunker() = default;
426427

427428
std::vector<Chunk> ContentDefinedChunker::GetChunks(const int16_t* def_levels,

‎cpp/src/parquet/chunker_internal.h‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -72,10 +72,10 @@ struct Chunk {
7272
/// Implementation details:
7373
///
7474
/// Only the parquet writer must be aware of the content defined chunking, the reader
75-
/// doesn't need to know about it. Each parquet column writer holds a
76-
/// ContentDefinedChunker instance depending on the writer's properties. The chunker's
77-
/// state is maintained across the entire column without being reset between pages and row
78-
/// groups.
75+
/// doesn't need to know about it. The parquet file writer holds one
76+
/// ContentDefinedChunker per leaf column depending on the writer's properties, and passes
77+
/// it to the column writers of every row group. The chunker's state is maintained
78+
/// across the entire column without being reset between pages and row groups.
7979
///
8080
/// The chunker receives the record shredded column data (def_levels, rep_levels, values)
8181
/// and goes over the (def_level, rep_level, value) triplets one by one while adjusting
@@ -118,6 +118,7 @@ class PARQUET_EXPORT ContentDefinedChunker {
118118
/// expense of fragmentation.
119119
ContentDefinedChunker(const LevelInfo& level_info, int64_t min_chunk_size,
120120
int64_t max_chunk_size, int norm_level = 0);
121+
ContentDefinedChunker(ContentDefinedChunker&&) noexcept;
121122
~ContentDefinedChunker();
122123

123124
/// Get the chunk boundaries for the given column data

‎cpp/src/parquet/chunker_internal_test.cc‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -403,6 +403,20 @@ ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data, int column
403403
return result;
404404
}
405405

406+
// The offsets where the data pages of a column end, counted in levels from the start of
407+
// the column across its row groups, which are rows for flat columns
408+
std::vector<int64_t> GetPageEnds(const std::shared_ptr<Buffer>& data, int column_index) {
409+
std::vector<int64_t> page_ends;
410+
int64_t offset = 0;
411+
for (const auto& row_group : GetColumnParquetInfo(data, column_index)) {
412+
for (auto page_length : row_group.page_lengths) {
413+
offset += page_length;
414+
page_ends.push_back(offset);
415+
}
416+
}
417+
return page_ends;
418+
}
419+
406420
// A git-hunk like side-by-side data structure to represent the differences between two
407421
// vectors of uint64_t values.
408422
using ChunkDiff = std::pair<ChunkList, ChunkList>;
@@ -1700,4 +1714,39 @@ TEST_F(TestCDCMultipleRowGroups, Append) {
17001714
}
17011715
}
17021716

1717+
TEST_F(TestCDCMultipleRowGroups, IndependentOfRowGroupBoundaries) {
1718+
// Splitting the data into row groups must not move the content defined page
1719+
// boundaries, only add one at the end of each row group. For example, if the pages of
1720+
// a single row group file end at rows 100, 250 and 400, then with row groups of 200
1721+
// rows the pages must end at rows 100, 200, 250 and 400.
1722+
ASSERT_OK_AND_ASSIGN(auto table, ConcatAndCombine({part1_, part2_, part3_}));
1723+
const int64_t num_rows = table->num_rows();
1724+
const int64_t row_group_length = num_rows / 6;
1725+
ASSERT_OK_AND_ASSIGN(auto single,
1726+
WriteTableToBuffer(table, kMinChunkSize, kMaxChunkSize,
1727+
/*row_group_length=*/num_rows));
1728+
ASSERT_OK_AND_ASSIGN(auto multi, WriteTableToBuffer(table, kMinChunkSize, kMaxChunkSize,
1729+
row_group_length));
1730+
ASSERT_EQ(ReadMetaData(std::make_shared<BufferReader>(single))->num_row_groups(), 1);
1731+
ASSERT_EQ(ReadMetaData(std::make_shared<BufferReader>(multi))->num_row_groups(), 6);
1732+
// compare the page ends column by column
1733+
for (int col = 0; col < table->num_columns(); col++) {
1734+
// the page ends of the single row group file, e.g. 100, 250 and 400
1735+
auto single_page_ends = GetPageEnds(single, col);
1736+
// the page ends of the multiple row group file, e.g. 100, 200, 250 and 400
1737+
auto multi_page_ends = GetPageEnds(multi, col);
1738+
// expect the page ends of the single row group file plus one at each of the 5
1739+
// boundaries between the row groups, e.g. 200, the last row group ends with the data
1740+
// where the single row group file's last page ends too
1741+
auto expected = single_page_ends;
1742+
for (int i = 1; i < 6; i++) {
1743+
expected.push_back(i * row_group_length);
1744+
}
1745+
std::sort(expected.begin(), expected.end());
1746+
EXPECT_EQ(multi_page_ends.size(), single_page_ends.size() + 5) << "column " << col;
1747+
// unlike ASSERT_EQ, ContainerEq prints the page ends that differ
1748+
ASSERT_THAT(multi_page_ends, ::testing::ContainerEq(expected)) << "column " << col;
1749+
}
1750+
}
1751+
17031752
} // namespace parquet::internal

‎cpp/src/parquet/column_writer.cc‎

Lines changed: 41 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -744,7 +744,8 @@ class ColumnWriterImpl {
744744
public:
745745
ColumnWriterImpl(ColumnChunkMetaDataBuilder* metadata,
746746
std::unique_ptr<PageWriter> pager, const bool use_dictionary,
747-
Encoding::type encoding, const WriterProperties* properties)
747+
Encoding::type encoding, const WriterProperties* properties,
748+
internal::ContentDefinedChunker* content_defined_chunker)
748749
: metadata_(metadata),
749750
descr_(metadata->descr()),
750751
level_info_(internal::LevelInfo::ComputeLevelInfo(metadata->descr())),
@@ -763,7 +764,8 @@ class ColumnWriterImpl {
763764
closed_(false),
764765
fallback_(false),
765766
definition_levels_sink_(allocator_),
766-
repetition_levels_sink_(allocator_) {
767+
repetition_levels_sink_(allocator_),
768+
content_defined_chunker_(content_defined_chunker) {
767769
definition_levels_rle_ =
768770
std::static_pointer_cast<ResizableBuffer>(AllocateBuffer(allocator_, 0));
769771
repetition_levels_rle_ =
@@ -775,12 +777,6 @@ class ColumnWriterImpl {
775777
compressor_temp_buffer_ =
776778
std::static_pointer_cast<ResizableBuffer>(AllocateBuffer(allocator_, 0));
777779
}
778-
if (properties_->content_defined_chunking_enabled()) {
779-
auto cdc_options = properties_->content_defined_chunking_options();
780-
content_defined_chunker_.emplace(level_info_, cdc_options.min_chunk_size,
781-
cdc_options.max_chunk_size,
782-
cdc_options.norm_level);
783-
}
784780
}
785781

786782
virtual ~ColumnWriterImpl() = default;
@@ -912,7 +908,9 @@ class ColumnWriterImpl {
912908

913909
std::vector<std::unique_ptr<DataPage>> data_pages_;
914910

915-
std::optional<internal::ContentDefinedChunker> content_defined_chunker_;
911+
// The chunker of the column owned by the file writer, null without content defined
912+
// chunking
913+
internal::ContentDefinedChunker* content_defined_chunker_;
916914

917915
private:
918916
void InitSinks() {
@@ -1286,9 +1284,10 @@ class TypedColumnWriterImpl : public ColumnWriterImpl,
12861284
TypedColumnWriterImpl(ColumnChunkMetaDataBuilder* metadata,
12871285
std::unique_ptr<PageWriter> pager, const bool use_dictionary,
12881286
Encoding::type encoding, const WriterProperties* properties,
1289-
BloomFilter* bloom_filter)
1290-
: ColumnWriterImpl(metadata, std::move(pager), use_dictionary, encoding,
1291-
properties) {
1287+
BloomFilter* bloom_filter,
1288+
internal::ContentDefinedChunker* content_defined_chunker)
1289+
: ColumnWriterImpl(metadata, std::move(pager), use_dictionary, encoding, properties,
1290+
content_defined_chunker) {
12921291
current_encoder_ = MakeEncoder(ParquetType::type_num, encoding, use_dictionary,
12931292
descr_, properties->memory_pool());
12941293
// We have to dynamic_cast as some compilers don't want to static_cast
@@ -1330,7 +1329,7 @@ class TypedColumnWriterImpl : public ColumnWriterImpl,
13301329

13311330
int64_t WriteBatch(int64_t num_values, const int16_t* def_levels,
13321331
const int16_t* rep_levels, const T* values) override {
1333-
if (ARROW_PREDICT_FALSE(properties_->content_defined_chunking_enabled())) {
1332+
if (ARROW_PREDICT_FALSE(content_defined_chunker_ != nullptr)) {
13341333
throw ParquetException(
13351334
"Content-defined chunking is not supported in WriteBatch() or "
13361335
"WriteBatchSpaced(), use WriteArrow() instead.");
@@ -1373,7 +1372,7 @@ class TypedColumnWriterImpl : public ColumnWriterImpl,
13731372
void WriteBatchSpaced(int64_t num_values, const int16_t* def_levels,
13741373
const int16_t* rep_levels, const uint8_t* valid_bits,
13751374
int64_t valid_bits_offset, const T* values) override {
1376-
if (ARROW_PREDICT_FALSE(properties_->content_defined_chunking_enabled())) {
1375+
if (ARROW_PREDICT_FALSE(content_defined_chunker_ != nullptr)) {
13771376
throw ParquetException(
13781377
"Content-defined chunking is not supported in WriteBatch() or "
13791378
"WriteBatchSpaced(), use WriteArrow() instead.");
@@ -1443,8 +1442,7 @@ class TypedColumnWriterImpl : public ColumnWriterImpl,
14431442
bits_buffer_->ZeroPadding();
14441443
}
14451444

1446-
if (ARROW_PREDICT_FALSE(properties_->content_defined_chunking_enabled())) {
1447-
DCHECK(content_defined_chunker_.has_value());
1445+
if (ARROW_PREDICT_FALSE(content_defined_chunker_ != nullptr)) {
14481446
auto chunks = content_defined_chunker_->GetChunks(def_levels, rep_levels,
14491447
num_levels, leaf_array);
14501448
for (size_t i = 0; i < chunks.size(); i++) {
@@ -2700,10 +2698,10 @@ Status TypedColumnWriterImpl<FLBAType>::WriteArrowDense(
27002698
// ----------------------------------------------------------------------
27012699
// Dynamic column writer constructor
27022700

2703-
std::shared_ptr<ColumnWriter> ColumnWriter::Make(ColumnChunkMetaDataBuilder* metadata,
2704-
std::unique_ptr<PageWriter> pager,
2705-
const WriterProperties* properties,
2706-
BloomFilter* bloom_filter) {
2701+
std::shared_ptr<ColumnWriter> ColumnWriter::Make(
2702+
ColumnChunkMetaDataBuilder* metadata, std::unique_ptr<PageWriter> pager,
2703+
const WriterProperties* properties, BloomFilter* bloom_filter,
2704+
internal::ContentDefinedChunker* content_defined_chunker) {
27072705
const ColumnDescriptor* descr = metadata->descr();
27082706
const bool use_dictionary = properties->dictionary_enabled(descr->path()) &&
27092707
descr->physical_type() != Type::BOOLEAN;
@@ -2725,29 +2723,36 @@ std::shared_ptr<ColumnWriter> ColumnWriter::Make(ColumnChunkMetaDataBuilder* met
27252723
}
27262724
return std::make_shared<TypedColumnWriterImpl<BooleanType>>(
27272725
metadata, std::move(pager), use_dictionary, encoding, properties,
2728-
/*bloom_filter=*/nullptr);
2726+
/*bloom_filter=*/nullptr, content_defined_chunker);
27292727
}
27302728
case Type::INT32:
27312729
return std::make_shared<TypedColumnWriterImpl<Int32Type>>(
2732-
metadata, std::move(pager), use_dictionary, encoding, properties, bloom_filter);
2730+
metadata, std::move(pager), use_dictionary, encoding, properties, bloom_filter,
2731+
content_defined_chunker);
27332732
case Type::INT64:
27342733
return std::make_shared<TypedColumnWriterImpl<Int64Type>>(
2735-
metadata, std::move(pager), use_dictionary, encoding, properties, bloom_filter);
2734+
metadata, std::move(pager), use_dictionary, encoding, properties, bloom_filter,
2735+
content_defined_chunker);
27362736
case Type::INT96:
27372737
return std::make_shared<TypedColumnWriterImpl<Int96Type>>(
2738-
metadata, std::move(pager), use_dictionary, encoding, properties, bloom_filter);
2738+
metadata, std::move(pager), use_dictionary, encoding, properties, bloom_filter,
2739+
content_defined_chunker);
27392740
case Type::FLOAT:
27402741
return std::make_shared<TypedColumnWriterImpl<FloatType>>(
2741-
metadata, std::move(pager), use_dictionary, encoding, properties, bloom_filter);
2742+
metadata, std::move(pager), use_dictionary, encoding, properties, bloom_filter,
2743+
content_defined_chunker);
27422744
case Type::DOUBLE:
27432745
return std::make_shared<TypedColumnWriterImpl<DoubleType>>(
2744-
metadata, std::move(pager), use_dictionary, encoding, properties, bloom_filter);
2746+
metadata, std::move(pager), use_dictionary, encoding, properties, bloom_filter,
2747+
content_defined_chunker);
27452748
case Type::BYTE_ARRAY:
27462749
return std::make_shared<TypedColumnWriterImpl<ByteArrayType>>(
2747-
metadata, std::move(pager), use_dictionary, encoding, properties, bloom_filter);
2750+
metadata, std::move(pager), use_dictionary, encoding, properties, bloom_filter,
2751+
content_defined_chunker);
27482752
case Type::FIXED_LEN_BYTE_ARRAY:
27492753
return std::make_shared<TypedColumnWriterImpl<FLBAType>>(
2750-
metadata, std::move(pager), use_dictionary, encoding, properties, bloom_filter);
2754+
metadata, std::move(pager), use_dictionary, encoding, properties, bloom_filter,
2755+
content_defined_chunker);
27512756
default:
27522757
ParquetException::NYI("Column writer not implemented for type: " +
27532758
TypeToString(descr->physical_type()));
@@ -2756,4 +2761,12 @@ std::shared_ptr<ColumnWriter> ColumnWriter::Make(ColumnChunkMetaDataBuilder* met
27562761
return std::shared_ptr<ColumnWriter>(nullptr);
27572762
}
27582763

2764+
std::shared_ptr<ColumnWriter> ColumnWriter::Make(ColumnChunkMetaDataBuilder* metadata,
2765+
std::unique_ptr<PageWriter> pager,
2766+
const WriterProperties* properties,
2767+
BloomFilter* bloom_filter) {
2768+
return Make(metadata, std::move(pager), properties, bloom_filter,
2769+
/*content_defined_chunker=*/nullptr);
2770+
}
2771+
27592772
} // namespace parquet

‎cpp/src/parquet/column_writer.h‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,10 @@ class Encryptor;
5454
class OffsetIndexBuilder;
5555
class WriterProperties;
5656

57+
namespace internal {
58+
class ContentDefinedChunker;
59+
} // namespace internal
60+
5761
class PARQUET_EXPORT LevelEncoder {
5862
public:
5963
LevelEncoder();
@@ -199,6 +203,16 @@ class PARQUET_EXPORT ColumnWriter {
199203
int64_t num_levels, const ::arrow::Array& leaf_array,
200204
ArrowWriteContext* ctx,
201205
bool leaf_field_nullable) = 0;
206+
207+
private:
208+
friend class RowGroupSerializer;
209+
210+
// Private to keep the internal chunker out of the public API, the file writer uses
211+
// it to carry the content defined chunking over between the row groups
212+
static std::shared_ptr<ColumnWriter> Make(
213+
ColumnChunkMetaDataBuilder*, std::unique_ptr<PageWriter>,
214+
const WriterProperties* properties, BloomFilter* bloom_filter,
215+
internal::ContentDefinedChunker* content_defined_chunker);
202216
};
203217

204218
// API to write values to a single column. This is the main client facing API.

0 commit comments

Comments
 (0)