Skip to content

Commit 184f186

Browse files
csun5285claude
andcommitted
[refactor](be) Peel the Nullable in the column reader, as the column writer does
### What problem does this PR solve? Issue Number: None Related PR: apache#68608 Problem Summary: Review follow-up. The read direction of StorageLayout handled a Nullable destination: repack_to_column, which writes the typed column of DATE, DATETIME and DECIMALV2 directly, unwrapped a Nullable itself, while the other layouts left it to ColumnNullable's insert_many_raw_data. The write direction never sees a Nullable: ScalarColumnWriter::append peels it off with peel_nullable, writes the NULL runs itself and hands column_to_storage the nested column. FileColumnIterator now does the same when it reads. next_batch and read_by_rowids peel a Nullable destination once: the iterator appends the NULL runs to its null map and nested column, and the page decoders append the values to the nested column. A dictionary page decoder may replace the column it appends to (a ColumnDictI32 becomes a ColumnString when a page falls back to plain encoding), so the nested column goes back into the Nullable when the read ends, on an early return too. Neither the page decoders nor StorageLayout see a Nullable any more, and repack_to_column loses its Nullable branch. ### Release note None ### Check List (For Author) - Test: Unit Test - PageFormatTest reads every column back by batch and now also by row id, two rows of every three so that the reads start and stop inside NULL runs; the DICT cases are read into a Nullable over ColumnDictI32 as well, which the plain pages at the end of each column make the decoder replace. - The V1 DATE decoder tests read by row id into a ColumnDate; the StorageLayout test of a Nullable destination is removed. - Run on this revision (ASAN build): the 31 suites of the earlier runs, 674 tests, all pass; and 76 more suites that read through a column reader (SegmentIterator, compaction, rowset, variant, inverted index, page decoders): 1015 tests pass, 3 skip themselves, none fail. TestDeltaWriterClusterKey and SegmentCacheTest stop in their first test once its read-back checks have passed: drop_tablet reaches Rowset::clear_cache, which uses a FileCacheFactory that no earlier test of a filtered run installs (enable_file_cache defaults to true). TestDeltaWriter, which drops tablets the same way, was not run. - clang-format 16, build hygiene, clang-tidy on the changed files: no finding on a line the branch changes. - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 5287505 commit 184f186

7 files changed

Lines changed: 198 additions & 132 deletions

File tree

‎be/src/storage/segment/column_reader.cpp‎

Lines changed: 99 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -2924,6 +2924,60 @@ Status FileColumnIterator::next_batch_of_zone_map(size_t* n, MutableColumnPtr& d
29242924
return _reader->next_batch_of_zone_map(n, dst);
29252925
}
29262926

2927+
namespace {
2928+
2929+
// The column the page decoders append values to. As ScalarColumnWriter::append peels the Nullable
2930+
// off before the values reach a page, the iterator peels it here: the decoders append to the
2931+
// nested column and the iterator fills the null map. A dictionary page decoder may replace the
2932+
// column it appends to, so the nested column goes back into dst on destruction.
2933+
class PeeledNullable {
2934+
public:
2935+
explicit PeeledNullable(MutableColumnPtr& dst) : _dst(dst) {
2936+
if (is_column_nullable(*dst)) {
2937+
_nullable = &assert_cast<ColumnNullable&>(*dst);
2938+
_nested = _nullable->get_nested_column_ptr();
2939+
}
2940+
}
2941+
2942+
~PeeledNullable() {
2943+
if (_nullable != nullptr) {
2944+
ColumnPtr nested = std::move(_nested);
2945+
_nullable->change_nested_column(nested);
2946+
}
2947+
}
2948+
2949+
PeeledNullable(const PeeledNullable&) = delete;
2950+
PeeledNullable& operator=(const PeeledNullable&) = delete;
2951+
PeeledNullable(PeeledNullable&&) = delete;
2952+
PeeledNullable& operator=(PeeledNullable&&) = delete;
2953+
2954+
MutableColumnPtr& values() { return _nullable != nullptr ? _nested : _dst; }
2955+
2956+
// Marks not NULL the n rows a decoder has just appended to values().
2957+
void mark_not_null(size_t n) {
2958+
if (_nullable != nullptr) {
2959+
_nullable->push_false_to_nullmap(n);
2960+
}
2961+
}
2962+
2963+
Status append_nulls(size_t n) {
2964+
if (_nullable == nullptr) {
2965+
return Status::InternalError("unexpected column type in column reader");
2966+
}
2967+
_nested->insert_many_defaults(n);
2968+
auto& null_map = _nullable->get_null_map_data();
2969+
null_map.resize_fill(null_map.size() + n, 1);
2970+
return Status::OK();
2971+
}
2972+
2973+
private:
2974+
MutableColumnPtr& _dst;
2975+
ColumnNullable* _nullable = nullptr;
2976+
MutableColumnPtr _nested;
2977+
};
2978+
2979+
} // namespace
2980+
29272981
Status FileColumnIterator::next_batch(size_t* n, MutableColumnPtr& dst, bool* has_null) {
29282982
if (!need_to_read()) {
29292983
DLOG(INFO) << "File column iterator column " << _column_name << " skip reading.";
@@ -2985,49 +3039,50 @@ Status FileColumnIterator::next_batch(size_t* n, MutableColumnPtr& dst, bool* ha
29853039
dst->reserve(*n);
29863040
size_t remaining = *n;
29873041
*has_null = false;
2988-
while (remaining > 0) {
2989-
if (!_page.has_remaining()) {
2990-
bool eos = false;
2991-
RETURN_IF_ERROR(_load_next_page(&eos));
2992-
if (eos) {
2993-
break;
3042+
{
3043+
// Ends before dst is measured below, so that dst holds its nested column again.
3044+
PeeledNullable peeled(dst);
3045+
while (remaining > 0) {
3046+
if (!_page.has_remaining()) {
3047+
bool eos = false;
3048+
RETURN_IF_ERROR(_load_next_page(&eos));
3049+
if (eos) {
3050+
break;
3051+
}
29943052
}
2995-
}
29963053

2997-
// number of rows to be read from this page
2998-
size_t nrows_in_page = std::min(remaining, _page.remaining());
2999-
size_t nrows_to_read = nrows_in_page;
3000-
if (_page.has_null) {
3001-
while (nrows_to_read > 0) {
3002-
bool is_null = false;
3003-
size_t this_run = _page.null_decoder.GetNextRun(&is_null, nrows_to_read);
3004-
// we use num_rows only for CHECK
3005-
size_t num_rows = this_run;
3006-
if (!is_null) {
3007-
RETURN_IF_ERROR(_page.data_decoder->next_batch(&num_rows, dst));
3008-
DCHECK_EQ(this_run, num_rows);
3009-
} else {
3010-
*has_null = true;
3011-
auto* null_col = check_and_get_column<ColumnNullable>(dst.get());
3012-
if (null_col != nullptr) {
3013-
null_col->insert_many_defaults(this_run);
3054+
// number of rows to be read from this page
3055+
size_t nrows_in_page = std::min(remaining, _page.remaining());
3056+
size_t nrows_to_read = nrows_in_page;
3057+
if (_page.has_null) {
3058+
while (nrows_to_read > 0) {
3059+
bool is_null = false;
3060+
size_t this_run = _page.null_decoder.GetNextRun(&is_null, nrows_to_read);
3061+
// we use num_rows only for CHECK
3062+
size_t num_rows = this_run;
3063+
if (!is_null) {
3064+
RETURN_IF_ERROR(_page.data_decoder->next_batch(&num_rows, peeled.values()));
3065+
DCHECK_EQ(this_run, num_rows);
3066+
peeled.mark_not_null(num_rows);
30143067
} else {
3015-
return Status::InternalError("unexpected column type in column reader");
3068+
*has_null = true;
3069+
RETURN_IF_ERROR(peeled.append_nulls(this_run));
30163070
}
3071+
3072+
nrows_to_read -= this_run;
3073+
_page.offset_in_page += this_run;
3074+
_current_ordinal += this_run;
30173075
}
3076+
} else {
3077+
RETURN_IF_ERROR(_page.data_decoder->next_batch(&nrows_to_read, peeled.values()));
3078+
DCHECK_EQ(nrows_to_read, nrows_in_page);
3079+
peeled.mark_not_null(nrows_to_read);
30183080

3019-
nrows_to_read -= this_run;
3020-
_page.offset_in_page += this_run;
3021-
_current_ordinal += this_run;
3081+
_page.offset_in_page += nrows_to_read;
3082+
_current_ordinal += nrows_to_read;
30223083
}
3023-
} else {
3024-
RETURN_IF_ERROR(_page.data_decoder->next_batch(&nrows_to_read, dst));
3025-
DCHECK_EQ(nrows_to_read, nrows_in_page);
3026-
3027-
_page.offset_in_page += nrows_to_read;
3028-
_current_ordinal += nrows_to_read;
3084+
remaining -= nrows_in_page;
30293085
}
3030-
remaining -= nrows_in_page;
30313086
}
30323087
*n -= remaining;
30333088
_opts.stats->bytes_read += (dst->byte_size() - curr_size) + BitmapSize(*n);
@@ -3123,6 +3178,7 @@ Status FileColumnIterator::read_by_rowids(const rowid_t* rowids, const size_t co
31233178
return Status::OK();
31243179
}
31253180

3181+
PeeledNullable peeled(dst);
31263182
size_t remaining = count;
31273183
size_t total_read_count = 0;
31283184
size_t nrows_to_read = 0;
@@ -3155,12 +3211,7 @@ Status FileColumnIterator::read_by_rowids(const rowid_t* rowids, const size_t co
31553211
auto origin_index = _page.data_decoder->current_index();
31563212
if (this_read_count > 0) {
31573213
if (is_null) {
3158-
auto* null_col = check_and_get_column<ColumnNullable>(dst.get());
3159-
if (UNLIKELY(null_col == nullptr)) {
3160-
return Status::InternalError("unexpected column type in column reader");
3161-
}
3162-
3163-
null_col->insert_many_defaults(this_read_count);
3214+
RETURN_IF_ERROR(peeled.append_nulls(this_read_count));
31643215
} else {
31653216
size_t read_count = this_read_count;
31663217

@@ -3169,8 +3220,10 @@ Status FileColumnIterator::read_by_rowids(const rowid_t* rowids, const size_t co
31693220
size_t page_start_off_in_decoder =
31703221
_page.first_ordinal + _page.offset_in_page - origin_index;
31713222
RETURN_IF_ERROR(_page.data_decoder->read_by_rowids(
3172-
&rowids[offset], page_start_off_in_decoder, &read_count, dst));
3223+
&rowids[offset], page_start_off_in_decoder, &read_count,
3224+
peeled.values()));
31733225
DCHECK_EQ(read_count, this_read_count);
3226+
peeled.mark_not_null(read_count);
31743227
}
31753228
}
31763229

@@ -3188,8 +3241,10 @@ Status FileColumnIterator::read_by_rowids(const rowid_t* rowids, const size_t co
31883241
total_read_count += nrows_to_read;
31893242
remaining -= nrows_to_read;
31903243
} else {
3191-
RETURN_IF_ERROR(_page.data_decoder->read_by_rowids(
3192-
&rowids[total_read_count], _page.first_ordinal, &nrows_to_read, dst));
3244+
RETURN_IF_ERROR(_page.data_decoder->read_by_rowids(&rowids[total_read_count],
3245+
_page.first_ordinal, &nrows_to_read,
3246+
peeled.values()));
3247+
peeled.mark_not_null(nrows_to_read);
31933248
total_read_count += nrows_to_read;
31943249
remaining -= nrows_to_read;
31953250
}

‎be/src/storage/storage_layout.cpp‎

Lines changed: 0 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,6 @@
2828
#include "core/column/column_complex.h"
2929
#include "core/column/column_decimal.h"
3030
#include "core/column/column_fixed_length_object.h"
31-
#include "core/column/column_nullable.h"
3231
#include "core/column/column_string.h"
3332
#include "core/column/column_vector.h"
3433
#include "core/value/bitmap_value.h"
@@ -68,13 +67,6 @@ void repack_from_column(const IColumn& column, size_t row_pos, size_t n, uint8_t
6867
// Appends n stored values of a repacked layout, each turned back into its PrimitiveValue.
6968
template <FieldType FT>
7069
void repack_to_column(const uint8_t* values, size_t n, IColumn& dst) {
71-
if (is_column_nullable(dst)) {
72-
// A page holds no NULL row: the n rows it decodes are all not NULL.
73-
auto& nullable = assert_cast<ColumnNullable&>(dst);
74-
nullable.push_false_to_nullmap(n);
75-
repack_to_column<FT>(values, n, nullable.get_nested_column());
76-
return;
77-
}
7870
using StorageValue = typename StorageTypeTraits<FT>::StorageType;
7971
using Column =
8072
typename PrimitiveTypeTraits<storage_field_type_to_primitive_type(FT)>::ColumnType;

‎be/src/storage/storage_layout.h‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -68,9 +68,9 @@ class IColumn;
6868
//
6969
// The fixed-width layouts also read back. to_primitive is the inverse of to_storage for one
7070
// value, and storage_to_column(values, n, dst) appends the n stored values at `values`, which
71-
// need not be aligned, to `dst` as PrimitiveValues: the column of the PrimitiveType, or a
72-
// Nullable over it whose n new rows are marked not NULL, as a page holds no NULL row. The page
73-
// decoders read through it, and the zone map turns its stored bounds back with to_primitive.
71+
// need not be aligned, to `dst`, the column of the PrimitiveType: FileColumnIterator peels a
72+
// Nullable off first, as ScalarColumnWriter::append does. The page decoders read through it,
73+
// and the zone map turns its stored bounds back with to_primitive.
7474
template <FieldType FT>
7575
struct StorageLayout;
7676

‎be/test/storage/segment/bitshuffle_page_test.cpp‎

Lines changed: 4 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,6 @@
2222
#include <memory>
2323

2424
#include "core/assert_cast.h"
25-
#include "core/column/column_nullable.h"
2625
#include "core/column/column_vector.h"
2726
#include "core/data_type/data_type_factory.hpp"
2827
#include "core/uint24.h"
@@ -424,8 +423,8 @@ TEST_F(BitShufflePageTest, TestBitShuffleDateV1DecodesIntoDates) {
424423
}
425424
}
426425

427-
// Rows read by id into a Nullable over ColumnDate come back as their dates, none of them NULL.
428-
TEST_F(BitShufflePageTest, TestBitShuffleDateV1ReadsRowIdsIntoNullableDates) {
426+
// Rows read by id into a ColumnDate come back as their dates.
427+
TEST_F(BitShufflePageTest, TestBitShuffleDateV1ReadsRowIdsIntoDates) {
429428
std::vector<VecDateTimeValue> days;
430429
std::unique_ptr<DataPage> decoded_page;
431430
Slice page;
@@ -436,15 +435,12 @@ TEST_F(BitShufflePageTest, TestBitShuffleDateV1ReadsRowIdsIntoNullableDates) {
436435
ASSERT_TRUE(decoder.init().ok());
437436

438437
const segment_v2::rowid_t rowids[] = {1, 3, 6};
439-
MutableColumnPtr picked = ColumnNullable::create(ColumnDate::create(), ColumnUInt8::create());
438+
MutableColumnPtr picked = ColumnDate::create();
440439
size_t n = 3;
441440
ASSERT_TRUE(decoder.read_by_rowids(rowids, 0, &n, picked).ok());
442441
ASSERT_EQ(n, 3);
443-
const auto& nullable = assert_cast<const ColumnNullable&>(*picked);
444-
const auto& picked_dates =
445-
assert_cast<const ColumnDate&>(nullable.get_nested_column()).get_data();
442+
const auto& picked_dates = assert_cast<const ColumnDate&>(*picked).get_data();
446443
for (size_t i = 0; i < 3; ++i) {
447-
EXPECT_FALSE(nullable.is_null_at(i));
448444
EXPECT_EQ(picked_dates[i], days[rowids[i]]) << "row id " << rowids[i];
449445
}
450446
}

0 commit comments

Comments
 (0)