You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 184f186
Browse filesBrowse the repository at this point in the historyBrowse files
[refactor](be) Peel the Nullable in the column reader, as the column writer does
### What problem does this PR solve?
Issue Number: None
Related PR: apache#68608
Problem Summary: Review follow-up. The read direction of StorageLayout handled a Nullable
destination: repack_to_column, which writes the typed column of DATE, DATETIME and DECIMALV2
directly, unwrapped a Nullable itself, while the other layouts left it to ColumnNullable's
insert_many_raw_data. The write direction never sees a Nullable: ScalarColumnWriter::append
peels it off with peel_nullable, writes the NULL runs itself and hands column_to_storage the
nested column.
FileColumnIterator now does the same when it reads. next_batch and read_by_rowids peel a
Nullable destination once: the iterator appends the NULL runs to its null map and nested
column, and the page decoders append the values to the nested column. A dictionary page
decoder may replace the column it appends to (a ColumnDictI32 becomes a ColumnString when a
page falls back to plain encoding), so the nested column goes back into the Nullable when the
read ends, on an early return too. Neither the page decoders nor StorageLayout see a Nullable
any more, and repack_to_column loses its Nullable branch.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- PageFormatTest reads every column back by batch and now also by row id, two rows of
every three so that the reads start and stop inside NULL runs; the DICT cases are read
into a Nullable over ColumnDictI32 as well, which the plain pages at the end of each
column make the decoder replace.
- The V1 DATE decoder tests read by row id into a ColumnDate; the StorageLayout test of a
Nullable destination is removed.
- Run on this revision (ASAN build): the 31 suites of the earlier runs, 674
tests, all pass; and 76 more suites that read through a column reader
(SegmentIterator, compaction, rowset, variant, inverted index, page
decoders): 1015 tests pass, 3 skip themselves, none fail.
TestDeltaWriterClusterKey and SegmentCacheTest stop in their first test once
its read-back checks have passed: drop_tablet reaches Rowset::clear_cache,
which uses a FileCacheFactory that no earlier test of a filtered run installs
(enable_file_cache defaults to true). TestDeltaWriter, which drops tablets the
same way, was not run.
- clang-format 16, build hygiene, clang-tidy on the changed files: no finding on a line
the branch changes.
- Behavior changed: No
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <[email protected]>
0 commit comments