Skip to content

Commit 7df2a08

Browse files
committed
[improvement](variant) Assert Variant sparse byte order in debug builds
### Background Variant V2 persists each row's sparse leaves as a dotted-path map, and every reader merges or binary-searches those keys by raw path bytes. The writer sorts the sparse plan (#70001), and vertical compaction merges the source sparse row with the materialized subcolumns, so byte order is a producer obligation all the way to the compacted segment. Nothing checked it, so a regression would only surface much later as a query failure (`Duplicate Variant object key`). ### What problem does this PR solve? Issue Number: None Problem Summary: Without a check, a writer or compaction merge that breaks the order produces segments whose reads fail, far from the cause. This change asserts the invariant where it is relied on and keeps the assertion out of release builds. ### Details - **Debug-only checks:** `DCHECK(is_variant_sparse_row_sorted(...))` in - `StorageMapRowCursor::bind()` (V2 row cursor used by every hierarchical read), - the exact sparse-path extractor, - `SparseColumnMergeIterator::_merge_to` for both the source row and the merged destination row, because the merge interleaves sparse cells with materialized subcolumn paths. - **Why not repair:** query-time and compaction-time tolerance is deliberately not added on this branch. Segments must be byte-ordered; a violation is a producer bug. - **Performance:** every check is a `DCHECK`, compiled out in release; queries and compaction keep the same scan, sort, and allocation behavior as before. - **Tests:** `VariantAssemblerTest.DebugCheckRejectsUnsortedSparseRows` (death test) feeds a two-cell unsorted row and expects the debug check to fire; `SparseColumnMergeIteratorTest.KeepsByteOrderWhenMergingMaterializedSubcolumn` merges a byte-sorted source row with a materialized `a.d` column and asserts the merged row is `a-b, a.c, a.d`. The import side is already covered by `VariantShredderTest.SparseRowsUseByteOrderWithoutChangingStatisticsSelection`. ### Testing - `./run-be-ut.sh --run --filter='VariantAssemblerTest.*:BinaryColumnExtractIteratorV2Test.*:VariantShredderTest.*'` on dev-x (ASAN): 17 tests passed. - `./run-be-ut.sh --run --filter='VariantColumnWriterReaderTest.*:IndexStorageVariantCompactionReadTest.*:SparseColumnMergeIteratorTest.*'` on dev-x (ASAN): 75 passed, 4 pre-existing skips. - `./run-be-ut.sh --run --filter='SparseColumnMergeIteratorTest.*:VariantAssemblerTest.*'` on dev-x (ASAN): 2 passed (merge interleaving + death test). - Negative check: temporarily removed the writer-side `std::ranges::sort` from `VariantShredder` (#70001). `VariantShredderTest.SparseRowsUseByteOrderWithoutChangingStatisticsSelection` failed its byte-order expectation and the new check aborted in `StorageMapRowCursor::bind` (`Check failed: is_variant_sparse_row_sorted`), so the assertion does catch a producer that stops sorting. The sort was restored afterwards. - `build-support/check-build-hygiene.sh` passed. ### Release note None. ### Check List (For Author) - Behavior changed: No (assertions only, active in debug builds) - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: DeepSeek Harness - Generated: code, tests, commit message - Human review: pending
1 parent 9e598d3 commit 7df2a08

6 files changed

Lines changed: 124 additions & 0 deletions

File tree

‎be/src/core/column/variant_column_utils.cpp‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,15 @@ size_t find_variant_sparse_path_lower_bound(StringRef path, const ColumnString&
3535
return start;
3636
}
3737

38+
bool is_variant_sparse_row_sorted(const ColumnString& sparse_paths, size_t start, size_t end) {
39+
for (size_t i = start + 1; i < end; ++i) {
40+
if (sparse_paths.get_data_at(i) < sparse_paths.get_data_at(i - 1)) {
41+
return false;
42+
}
43+
}
44+
return true;
45+
}
46+
3847
bool is_variant_jsonb_value_semantically_empty(const JsonbValue* value) {
3948
if (value == nullptr || value->isNull()) {
4049
return true;

‎be/src/core/column/variant_column_utils.h‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,11 @@ struct JsonbValue;
3030
size_t find_variant_sparse_path_lower_bound(StringRef path, const ColumnString& sparse_paths,
3131
size_t start, size_t end);
3232

33+
// Readers merge and binary-search sparse keys by raw path bytes, so every segment must persist a
34+
// row in that order: the writer sorts the sparse plan (#70001) and compaction copies cells verbatim.
35+
// Only debug assertions call this, so release reads and compaction pay nothing for the check.
36+
bool is_variant_sparse_row_sorted(const ColumnString& sparse_paths, size_t start, size_t end);
37+
3338
// True when a JSONB value recursively contains no visible scalar payload. Variant V1 and V2
3439
// readers share this rule when deciding whether a materialized nested value contributes a path.
3540
bool is_variant_jsonb_value_semantically_empty(const JsonbValue* value);

‎be/src/storage/segment/variant/binary_column_extract_iterator.cpp‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,8 @@ Status BinaryColumnExtractIterator::_fill_variant_v2_path(MutableColumnPtr& dst,
105105
return Status::Corruption("Invalid Variant sparse offsets [{}, {}) for {} cells", start,
106106
end, paths->size());
107107
}
108+
// Readers binary-search by raw path bytes; a producer that violates the order is a bug.
109+
DCHECK(is_variant_sparse_row_sorted(*paths, start, end));
108110
const size_t pos = find_variant_sparse_path_lower_bound(requested, *paths, start, end);
109111
// Prefix matches such as a.b belong to another physical path.
110112
if (pos < end && paths->get_data_at(pos) == requested) {

‎be/src/storage/segment/variant/sparse_column_merge_iterator.cpp‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@
2626
#include "core/column/column.h"
2727
#include "core/column/column_nullable.h"
2828
#include "core/column/subcolumn_tree.h"
29+
#include "core/column/variant_column_utils.h"
2930
#include "core/data_type/data_type.h"
3031
#include "core/data_type/data_type_nullable.h"
3132
#include "storage/segment/column_reader.h"
@@ -123,6 +124,9 @@ void SparseColumnMergeIterator::_merge_to(MutableColumnPtr& dst) {
123124

124125
size_t offset = src_serialized_sparse_column_offsets[row - 1];
125126
size_t end = src_serialized_sparse_column_offsets[row];
127+
// Compaction only copies cells, and every reader binary-searches by raw path bytes, so the
128+
// source row must already be byte-ordered. Debug-only: release compaction pays nothing.
129+
DCHECK(is_variant_sparse_row_sorted(src_sparse_column_paths, offset, end));
126130
// Iterator over [path, binary value]
127131
for (size_t i = offset; i != end; ++i) {
128132
const StringRef src_sparse_path_string = src_sparse_column_paths.get_data_at(i);
@@ -157,6 +161,12 @@ void SparseColumnMergeIterator::_merge_to(MutableColumnPtr& dst) {
157161
dst_sparse_column_values, src_path, row);
158162
}
159163

164+
// The interleaving above assumes the source sparse cells and the materialized subcolumns
165+
// are byte-ordered. Assert the merged row keeps that invariant for downstream readers;
166+
// debug-only, so release compaction pays nothing.
167+
DCHECK(is_variant_sparse_row_sorted(dst_sparse_column_paths,
168+
dst_sparse_column_offsets[ssize_t(row) - 1],
169+
dst_sparse_column_paths.size()));
160170
// All the sparse columns in this row are null.
161171
dst_sparse_column_offsets.push_back(dst_sparse_column_paths.size());
162172
}

‎be/src/storage/segment/variant/v2/variant_assembler.cpp‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,12 @@ struct StorageMapRowCursor {
9090
DORIS_CHECK(values != nullptr);
9191
DORIS_CHECK_EQ(paths->size(), values->size());
9292
DCHECK(rows == 0 || storage_map->offset_at(static_cast<ssize_t>(rows)) == paths->size());
93+
// Readers binary-search sparse keys by raw path bytes. The writer (#70001) and vertical
94+
// compaction canonicalize every row, so a non-conforming row reaching this cursor is a
95+
// producer bug; keep the check debug-only so release scans pay nothing.
96+
for (size_t row = 0; row < rows; ++row) {
97+
DCHECK(is_variant_sparse_row_sorted(*paths, begin(row), row_end(row)));
98+
}
9399
}
94100

95101
size_t begin(size_t row) const noexcept { return source->offset_at(static_cast<ssize_t>(row)); }

‎be/test/storage/segment/binary_column_extract_iterator_test.cpp‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717

1818
#include "storage/segment/variant/binary_column_extract_iterator.h"
1919

20+
#include <glog/logging.h>
2021
#include <gtest/gtest.h>
2122

2223
#include <algorithm>
@@ -40,6 +41,8 @@
4041
#include "core/value/jsonb_value.h"
4142
#include "exec/common/variant_util.h"
4243
#include "storage/iterators.h"
44+
#include "storage/segment/variant/sparse_column_merge_iterator.h"
45+
#include "storage/segment/variant/v2/variant_assembler.h"
4346

4447
namespace doris::segment_v2 {
4548
namespace {
@@ -789,5 +792,94 @@ TEST(BinaryColumnExtractIteratorV2Test, DoesNotRescanFrozenPathOrdering) {
789792
EXPECT_EQ(variant_v2_json_at(variant, 0), "7");
790793
}
791794

795+
// Debug builds must reject a sparse row that violates the byte-order invariant; the check is a
796+
// DCHECK so release scans and compaction pay nothing.
797+
#if DCHECK_IS_ON()
798+
TEST(VariantAssemblerTest, DebugCheckRejectsUnsortedSparseRows) {
799+
auto sparse = variant_util::create_variant_binary_column();
800+
auto& map = assert_cast<ColumnMap&>(*sparse);
801+
DataTypeInt64 int_type;
802+
auto ints = ColumnInt64::create();
803+
ints->insert_value(1);
804+
ints->insert_value(2);
805+
append_storage_cell(map, "a.c", int_type, *ints, 0);
806+
append_storage_cell(map, "a-b", int_type, *ints, 1);
807+
map.get_offsets().push_back(2);
808+
809+
variant_v2::VariantAssemblerOptions options;
810+
options.storage_map_kind = variant_v2::StorageMapKind::SPARSE;
811+
auto assembler = variant_v2::VariantAssembler::create(std::move(options));
812+
ASSERT_TRUE(assembler.has_value());
813+
EXPECT_DEATH(
814+
{
815+
ColumnNullable::MutablePtr result;
816+
(void)(*assembler)
817+
->assemble({.num_rows = map.size(),
818+
.materialized_columns = {},
819+
.storage_map = &map},
820+
&result);
821+
},
822+
"is_variant_sparse_row_sorted");
823+
}
824+
#endif
825+
826+
// Vertical compaction merges the source sparse row with materialized subcolumns. The interleaving
827+
// assumes both sides are byte-ordered, and the merged row is what later readers see, so it must
828+
// keep that invariant.
829+
TEST(SparseColumnMergeIteratorTest, KeepsByteOrderWhenMergingMaterializedSubcolumn) {
830+
auto sparse = variant_util::create_variant_binary_column();
831+
auto& map = assert_cast<ColumnMap&>(*sparse);
832+
DataTypeInt64 int_type;
833+
auto ints = ColumnInt64::create();
834+
ints->insert_value(1);
835+
ints->insert_value(2);
836+
append_storage_cell(map, "a-b", int_type, *ints, 0);
837+
append_storage_cell(map, "a.c", int_type, *ints, 1);
838+
map.get_offsets().push_back(2);
839+
840+
auto counters = std::make_shared<SparseReadCounters>();
841+
auto cache = std::make_shared<BinaryColumnCache>(
842+
std::make_unique<FixedSparseIterator>(std::move(sparse), counters),
843+
variant_util::create_variant_binary_column());
844+
845+
// Materialized path a.d; the dense iterator yields one nullable bigint row: 3.
846+
auto dense_type = std::make_shared<DataTypeNullable>(std::make_shared<DataTypeInt64>());
847+
auto dense_values = ColumnNullable::create(ColumnInt64::create(), ColumnUInt8::create());
848+
assert_cast<ColumnInt64&>(dense_values->get_nested_column()).insert_value(3);
849+
dense_values->get_null_map_data().push_back(0);
850+
SubstreamReaderTree src_subcolumns;
851+
ASSERT_TRUE(src_subcolumns.add(
852+
PathInData("a.d"),
853+
SubstreamIterator(ColumnNullable::create(ColumnInt64::create(), ColumnUInt8::create()),
854+
std::make_unique<FixedSparseIterator>(
855+
std::move(dense_values),
856+
std::make_shared<SparseReadCounters>()),
857+
dense_type)));
858+
859+
VariantCompactionPaths path_set_info;
860+
OlapReaderStatistics stats;
861+
StorageReadOptions read_options(stats);
862+
SparseColumnMergeIterator iterator(path_set_info, std::move(cache), std::move(src_subcolumns),
863+
&read_options);
864+
ColumnIteratorOptions iterator_options;
865+
iterator_options.stats = &stats;
866+
ASSERT_TRUE(iterator.init(iterator_options).ok());
867+
ASSERT_TRUE(iterator.seek_to_ordinal(0).ok());
868+
auto dst = variant_util::create_variant_binary_column();
869+
size_t rows = 1;
870+
bool has_null = false;
871+
ASSERT_TRUE(iterator.next_batch(&rows, dst, &has_null).ok());
872+
ASSERT_EQ(dst->size(), 1);
873+
const auto& merged = assert_cast<const ColumnMap&>(*dst);
874+
const auto& paths = assert_cast<const ColumnString&>(merged.get_keys());
875+
std::vector<std::string> actual;
876+
const size_t begin = merged.offset_at(0);
877+
const size_t end = begin + merged.size_at(0);
878+
for (size_t cell = begin; cell < end; ++cell) {
879+
actual.push_back(paths.get_data_at(cell).to_string());
880+
}
881+
EXPECT_EQ(actual, (std::vector<std::string> {"a-b", "a.c", "a.d"}));
882+
}
883+
792884
} // namespace
793885
} // namespace doris::segment_v2

0 commit comments

Comments
 (0)