Skip to content

Commit 9a982d3

Browse files
morningmanhubgetereldenmoonclaude
committed
[improvement](be) Import unshredded Parquet Variant values without rebuilding them
### What problem does this PR solve? Issue Number: None Related PR: #66714, #66910, #67016 Problem Summary: Port of branch-4.1 #66714 onto master's #67016 code. When the native Parquet reader materializes an unshredded Variant column (only metadata and value, no typed_value), master rebuilds every row through VariantBatchBuilder (encode_variant_range), decoding and re-encoding the whole value tree although the bytes already are Variant Encoding V1. The direct path seek of #67016 does the same for each found path through append_variant_rows, which re-encodes the rows into a builder with a single metadata dictionary and splits a batch in halves whenever its rows use different dictionaries. Re-encoding also changes what is validated. The builder copies only the metadata keys a value uses, so a corrupt key that the value does not use, for example invalid UTF-8, disappears instead of being reported: extracting a path from such a row returns a value. branch-4.x validates every metadata dictionary when it imports the row, so the read fails. Two #66910 tests cover this; #67016 left them out because master had no importer. This adds ColumnVariantV2::EncodedRowsAppender, as in branch-4.x. It validates each distinct metadata dictionary once and every value, rejects input that borrows bytes from the destination column, and then copies the bytes, interning metadata through a hash index that it keeps across batches. insert_encoded_rows and insert_encoded_batch share the new _append_prevalidated_encoded_data. The Parquet reader uses the appender for complete unshredded roots during materialization and for the direct path seek, and reports VariantUnshreddedDirectImportTime, Rows and Bytes. ### Release note None ### Check List (For Author) - Test: Unit Test - BE UT: ColumnVariantV2Test (appender atomicity on a typed column, rejection of rows borrowed from the destination, metadata interning across chunks, empty appends) and VariantColumnReaderTest (direct import of encoded bytes, a 4097-row import across the batch boundary, profile counters on immediate and late failures, and the two corrupt-metadata cases of #66910) pass, as do the other Variant, JNI and Parquet schema suites (755 tests). With the production change reverted, exactly the six new reader cases fail; the two corrupt-metadata cases fail because the read throws nothing. ColumnVariantV2Test.TypedComparisonBoundariesMatchEncodedWithoutMaterializing fails on a local macOS build with and without this change, and passes on CI. - Behavior changed: Yes. Reading an unshredded Parquet Variant row whose metadata has a corrupt key now fails instead of ignoring the key. - Does this need documentation: No Co-authored-by: daidai <[email protected]> Co-authored-by: lihangyu <[email protected]> Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent cad4abb commit 9a982d3

7 files changed

Lines changed: 927 additions & 64 deletions

File tree

‎be/src/core/column/variant_v2/column_variant_v2.cpp‎

Lines changed: 298 additions & 33 deletions
Large diffs are not rendered by default.

‎be/src/core/column/variant_v2/column_variant_v2.h‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -119,6 +119,34 @@ class ColumnVariantV2 final : public COWHelper<IColumn, ColumnVariantV2> {
119119
std::span<const uint32_t> value_offsets;
120120
};
121121

122+
// Scoped adapter for repeated encoded-row appends. It builds one hash index over destination
123+
// metadata ids and reuses it across calls, while the destination column continues to own all
124+
// copied bytes. Input rows must not borrow metadata or value bytes from the destination,
125+
// including its shredded materialization; use insert_range_from() for that case. Do not mutate
126+
// or destroy the destination while this adapter is alive.
127+
class EncodedRowsAppender {
128+
public:
129+
~EncodedRowsAppender();
130+
131+
EncodedRowsAppender(const EncodedRowsAppender&) = delete;
132+
EncodedRowsAppender& operator=(const EncodedRowsAppender&) = delete;
133+
EncodedRowsAppender(EncodedRowsAppender&&) noexcept;
134+
EncodedRowsAppender& operator=(EncodedRowsAppender&&) = delete;
135+
136+
void append(std::span<const VariantRef> rows);
137+
138+
#ifdef BE_TEST
139+
size_t metadata_comparisons_for_test() const noexcept;
140+
#endif
141+
142+
private:
143+
friend class ColumnVariantV2;
144+
explicit EncodedRowsAppender(ColumnVariantV2& column);
145+
146+
struct Impl;
147+
std::unique_ptr<Impl> _impl;
148+
};
149+
122150
// Borrowed immutable adapter for whole-column E/T readers. The source column owns every
123151
// referenced column, type, and byte; any structural mutation invalidates this view. Encoded
124152
// bytes have already been validated at their insertion or deserialization boundary.
@@ -204,6 +232,10 @@ class ColumnVariantV2 final : public COWHelper<IColumn, ColumnVariantV2> {
204232
// metadata blob. Input buffers must not alias this column; use insert_range_from for that case.
205233
void insert_encoded_rows(const EncodedDataView& data);
206234

235+
// Reuse one appender when encoded rows arrive in bounded batches so destination metadata is
236+
// indexed once for the full import rather than scanned again for every batch.
237+
EncodedRowsAppender create_encoded_rows_appender();
238+
207239
// Direct trusted codec adapter. VariantBatchBuilder already produces canonical metadata,
208240
// validated values, and ColumnString-compatible uint32 offsets, so this path copies its buffers
209241
// without validating the encoded tree a second time.
@@ -282,6 +314,8 @@ class ColumnVariantV2 final : public COWHelper<IColumn, ColumnVariantV2> {
282314
ColumnVariantV2();
283315
ColumnVariantV2(const ColumnVariantV2& other);
284316

317+
void _append_prevalidated_encoded_data(const EncodedDataView& data);
318+
uint32_t _append_metadata(StringRef metadata);
285319
uint32_t _find_or_insert_metadata(StringRef metadata);
286320
void _replace_shredded_state_with(const ColumnVariantV2& replacement);
287321
void _ensure_serialized();

‎be/src/format_v2/parquet/parquet_profile.cpp‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -107,6 +107,12 @@ void ParquetProfile::init(RuntimeProfile* profile) {
107107
TUnit::TIME_NS, parquet_profile);
108108
variant_reconstructed_rows = add_persistent_counter(profile, "VariantReconstructedRows",
109109
TUnit::UNIT, parquet_profile);
110+
variant_unshredded_direct_import_time = add_persistent_counter(
111+
profile, "VariantUnshreddedDirectImportTime", TUnit::TIME_NS, parquet_profile);
112+
variant_unshredded_direct_import_rows = add_persistent_counter(
113+
profile, "VariantUnshreddedDirectImportRows", TUnit::UNIT, parquet_profile);
114+
variant_unshredded_direct_import_bytes = add_persistent_counter(
115+
profile, "VariantUnshreddedDirectImportBytes", TUnit::BYTES, parquet_profile);
110116
variant_unshredded_direct_seek_time = add_persistent_counter(
111117
profile, "VariantUnshreddedDirectSeekTime", TUnit::TIME_NS, parquet_profile);
112118
variant_unshredded_direct_seek_rows = add_persistent_counter(
@@ -355,6 +361,9 @@ ParquetColumnReaderProfile ParquetProfile::column_reader_profile() const {
355361
.materialization_time = materialization_time,
356362
.variant_reconstruction_time = variant_reconstruction_time,
357363
.variant_reconstructed_rows = variant_reconstructed_rows,
364+
.variant_unshredded_direct_import_time = variant_unshredded_direct_import_time,
365+
.variant_unshredded_direct_import_rows = variant_unshredded_direct_import_rows,
366+
.variant_unshredded_direct_import_bytes = variant_unshredded_direct_import_bytes,
358367
.variant_unshredded_direct_seek_time = variant_unshredded_direct_seek_time,
359368
.variant_unshredded_direct_seek_rows = variant_unshredded_direct_seek_rows,
360369
.variant_unshredded_direct_seek_bytes = variant_unshredded_direct_seek_bytes,

‎be/src/format_v2/parquet/parquet_profile.h‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,11 @@ struct ParquetColumnReaderProfile {
4242
RuntimeProfile::Counter* materialization_time = nullptr; // value materialization time (ns)
4343
std::shared_ptr<RuntimeProfile::Counter> variant_reconstruction_time;
4444
std::shared_ptr<RuntimeProfile::Counter> variant_reconstructed_rows;
45+
// Complete unshredded roots imported without rebuilding their Variant value tree. Bytes count
46+
// the per-row metadata and value references presented to the direct importer.
47+
std::shared_ptr<RuntimeProfile::Counter> variant_unshredded_direct_import_time;
48+
std::shared_ptr<RuntimeProfile::Counter> variant_unshredded_direct_import_rows;
49+
std::shared_ptr<RuntimeProfile::Counter> variant_unshredded_direct_import_bytes;
4550
// Pure metadata+value roots can seek a requested path before constructing a root column.
4651
std::shared_ptr<RuntimeProfile::Counter> variant_unshredded_direct_seek_time;
4752
std::shared_ptr<RuntimeProfile::Counter> variant_unshredded_direct_seek_rows;
@@ -184,6 +189,10 @@ struct ParquetProfile {
184189
RuntimeProfile::Counter* materialization_time = nullptr;
185190
std::shared_ptr<RuntimeProfile::Counter> variant_reconstruction_time;
186191
std::shared_ptr<RuntimeProfile::Counter> variant_reconstructed_rows;
192+
// These counters are persistent because direct import can happen after scanner teardown.
193+
std::shared_ptr<RuntimeProfile::Counter> variant_unshredded_direct_import_time;
194+
std::shared_ptr<RuntimeProfile::Counter> variant_unshredded_direct_import_rows;
195+
std::shared_ptr<RuntimeProfile::Counter> variant_unshredded_direct_import_bytes;
187196
std::shared_ptr<RuntimeProfile::Counter> variant_unshredded_direct_seek_time;
188197
std::shared_ptr<RuntimeProfile::Counter> variant_unshredded_direct_seek_rows;
189198
std::shared_ptr<RuntimeProfile::Counter> variant_unshredded_direct_seek_bytes;

‎be/src/format_v2/parquet/reader/variant_column_reader.cpp‎

Lines changed: 101 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -479,9 +479,60 @@ std::optional<std::pair<size_t, size_t>> unshredded_child_indices(
479479
return std::pair {metadata_index, value_index};
480480
}
481481

482-
ColumnVariantV2::MutablePtr encode_variant_column(const ParquetColumnSchema& schema,
483-
const IColumn& physical,
484-
bool require_metadata = true) {
482+
void import_unshredded_variant_range(const ParquetColumnSchema& schema, const ColumnStruct& wrapper,
483+
const ColumnNullable* outer_nullable, size_t metadata_index,
484+
size_t value_index, size_t begin, size_t end,
485+
DorisVector<VariantRef>& encoded_rows,
486+
ColumnVariantV2::EncodedRowsAppender& appender,
487+
int64_t* imported_bytes) {
488+
encoded_rows.clear();
489+
if (encoded_rows.capacity() < end - begin) {
490+
encoded_rows.reserve(end - begin);
491+
}
492+
if (imported_bytes != nullptr) {
493+
*imported_bytes = 0;
494+
}
495+
auto count_imported_bytes = [&](size_t bytes) {
496+
if (imported_bytes == nullptr) {
497+
return;
498+
}
499+
DORIS_CHECK_LE(bytes,
500+
static_cast<size_t>(std::numeric_limits<int64_t>::max() - *imported_bytes));
501+
*imported_bytes += static_cast<int64_t>(bytes);
502+
};
503+
for (size_t row = begin; row < end; ++row) {
504+
if (outer_nullable != nullptr && outer_nullable->get_null_map_data()[row] != 0) {
505+
encoded_rows.push_back(
506+
{.metadata = {.data = VARIANT_EMPTY_METADATA.data(),
507+
.size = VARIANT_EMPTY_METADATA.size()},
508+
.value = {VARIANT_NULL_VALUE.data(), VARIANT_NULL_VALUE.size()}});
509+
count_imported_bytes(VARIANT_EMPTY_METADATA.size());
510+
count_imported_bytes(VARIANT_NULL_VALUE.size());
511+
continue;
512+
}
513+
514+
const Cell metadata = cell_at(wrapper.get_column(metadata_index), row);
515+
if (metadata.is_null) {
516+
throw Exception(ErrorCode::CORRUPTION, "Parquet Variant {} has null metadata at row {}",
517+
schema.name, row);
518+
}
519+
const StringRef metadata_bytes = metadata.column->get_data_at(row);
520+
const Cell value = cell_at(wrapper.get_column(value_index), row);
521+
const StringRef value_bytes =
522+
value.is_null ? StringRef(VARIANT_NULL_VALUE.data(), VARIANT_NULL_VALUE.size())
523+
: value.column->get_data_at(row);
524+
encoded_rows.push_back(
525+
{.metadata = {.data = metadata_bytes.data, .size = metadata_bytes.size},
526+
.value = value_bytes});
527+
count_imported_bytes(metadata_bytes.size);
528+
count_imported_bytes(value_bytes.size);
529+
}
530+
appender.append(std::span<const VariantRef>(encoded_rows));
531+
}
532+
533+
ColumnVariantV2::MutablePtr encode_variant_column(
534+
const ParquetColumnSchema& schema, const IColumn& physical, bool require_metadata = true,
535+
const ParquetColumnReaderProfile* profile = nullptr) {
485536
if (schema.kind != ParquetColumnSchemaKind::VARIANT) {
486537
throw Exception(ErrorCode::INVALID_ARGUMENT, "Parquet column {} is not Variant",
487538
schema.name);
@@ -496,11 +547,50 @@ ColumnVariantV2::MutablePtr encode_variant_column(const ParquetColumnSchema& sch
496547
}
497548

498549
auto variants = ColumnVariantV2::create();
499-
constexpr size_t MAX_RECONSTRUCTION_BATCH_ROWS = 4096;
500-
for (size_t begin = 0; begin < physical.size(); begin += MAX_RECONSTRUCTION_BATCH_ROWS) {
501-
encode_variant_range(schema, wrapper, outer_nullable, begin,
502-
std::min(physical.size(), begin + MAX_RECONSTRUCTION_BATCH_ROWS),
503-
require_metadata, *variants);
550+
// A complete unshredded root already contains exact Variant Encoding V1 bytes. Keep the
551+
// existing lazy materialization boundary, but validate and copy those bytes without rebuilding
552+
// the value tree through VariantBatchBuilder.
553+
const auto unshredded_indices =
554+
require_metadata ? unshredded_child_indices(schema) : std::nullopt;
555+
constexpr size_t MAX_MATERIALIZATION_BATCH_ROWS = 4096;
556+
DorisVector<VariantRef> encoded_rows;
557+
std::optional<ColumnVariantV2::EncodedRowsAppender> encoded_rows_appender;
558+
if (unshredded_indices.has_value()) {
559+
encoded_rows.reserve(std::min(physical.size(), MAX_MATERIALIZATION_BATCH_ROWS));
560+
encoded_rows_appender.emplace(variants->create_encoded_rows_appender());
561+
}
562+
for (size_t begin = 0; begin < physical.size(); begin += MAX_MATERIALIZATION_BATCH_ROWS) {
563+
const size_t end = std::min(physical.size(), begin + MAX_MATERIALIZATION_BATCH_ROWS);
564+
if (unshredded_indices.has_value()) {
565+
DORIS_CHECK(encoded_rows_appender.has_value());
566+
int64_t imported_bytes = 0;
567+
RuntimeProfile::Counter* direct_import_time =
568+
profile == nullptr ? nullptr
569+
: profile->variant_unshredded_direct_import_time.get();
570+
{
571+
// Update the shared timer on every exit, including validation failures. Rows and
572+
// bytes are published only after the complete chunk has been appended.
573+
SCOPED_TIMER(direct_import_time);
574+
import_unshredded_variant_range(
575+
schema, structure, outer_nullable, unshredded_indices->first,
576+
unshredded_indices->second, begin, end, encoded_rows,
577+
*encoded_rows_appender, profile == nullptr ? nullptr : &imported_bytes);
578+
}
579+
if (profile != nullptr) {
580+
const auto imported_rows = static_cast<int64_t>(end - begin);
581+
if (profile->variant_unshredded_direct_import_rows != nullptr) {
582+
COUNTER_UPDATE(profile->variant_unshredded_direct_import_rows.get(),
583+
imported_rows);
584+
}
585+
if (profile->variant_unshredded_direct_import_bytes != nullptr) {
586+
COUNTER_UPDATE(profile->variant_unshredded_direct_import_bytes.get(),
587+
imported_bytes);
588+
}
589+
}
590+
} else {
591+
encode_variant_range(schema, wrapper, outer_nullable, begin, end, require_metadata,
592+
*variants);
593+
}
504594
}
505595
return variants;
506596
}
@@ -1041,27 +1131,6 @@ UnshreddedPathScan scan_unshredded_path(const ParquetColumnSchema& schema, const
10411131
return std::move(typed_values).finish(std::move(path_cache));
10421132
}
10431133

1044-
void append_variant_rows(ColumnVariantV2& output, std::span<const VariantRef> rows) {
1045-
try {
1046-
VariantBatchBuilder builder(VariantBatchBuilder::ReserveHint {.rows = rows.size()});
1047-
for (const VariantRef value : rows) {
1048-
auto row = builder.begin_row();
1049-
row.add_value(value);
1050-
row.finish();
1051-
}
1052-
output.insert_encoded_batch(builder.finish_batch());
1053-
} catch (...) {
1054-
if (rows.size() <= 1) {
1055-
throw;
1056-
}
1057-
// A builder has one metadata dictionary. Split heterogeneous batches while preserving the
1058-
// original validation error for a malformed single row.
1059-
const size_t middle = rows.size() / 2;
1060-
append_variant_rows(output, rows.first(middle));
1061-
append_variant_rows(output, rows.subspan(middle));
1062-
}
1063-
}
1064-
10651134
ColumnPtr normalize_unshredded_path(const ParquetColumnSchema& schema, const IColumn& physical,
10661135
std::pair<size_t, size_t> child_indices,
10671136
std::span<const VariantShreddedPathSegment> path,
@@ -1074,6 +1143,7 @@ ColumnPtr normalize_unshredded_path(const ParquetColumnSchema& schema, const ICo
10741143
DORIS_CHECK_EQ(structure.tuple_size(), schema.children.size());
10751144

10761145
auto values = ColumnVariantV2::create();
1146+
auto appender = values->create_encoded_rows_appender();
10771147
auto nulls = ColumnUInt8::create();
10781148
nulls->reserve(physical.size());
10791149
constexpr size_t MAX_DIRECT_SEEK_BATCH_ROWS = 4096;
@@ -1126,7 +1196,7 @@ ColumnPtr normalize_unshredded_path(const ParquetColumnSchema& schema, const ICo
11261196
count_copied_bytes(null_value);
11271197
}
11281198
}
1129-
append_variant_rows(*values, encoded_rows);
1199+
appender.append(std::span<const VariantRef>(encoded_rows));
11301200
}
11311201
*copied_bytes = total_copied_bytes;
11321202
return ColumnNullable::create(std::move(values), std::move(nulls));
@@ -1503,7 +1573,7 @@ class ParquetVariantShreddedState final : public VariantShreddedState {
15031573
}
15041574
if (!_materialized) {
15051575
SCOPED_TIMER(_profile.variant_reconstruction_time.get());
1506-
_materialized = encode_variant_column(*_schema, *_physical);
1576+
_materialized = encode_variant_column(*_schema, *_physical, true, &_profile);
15071577
update_counter(_profile.variant_reconstructed_rows,
15081578
static_cast<int64_t>(_physical->size()));
15091579
}

0 commit comments

Comments
 (0)