Repository navigation
Commit 1dcbc03
[test](be) Pin the on-disk pages of every page encoding with golden files
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Nothing pins the bytes of the pages a column writer writes.
The bitshuffle, plain, RLE, frame-of-reference, binary plain and binary prefix
page tests are excluded from the UT build, and the page tests that do run
round-trip values without comparing bytes. SegmentFlusherFormatTest pins whole
Segments, but its columns hold a handful of rows, so no page ever fills up,
and only the encodings the writers pick are written (never PLAIN_V2, for
example).
Add PageFormatTest, in the style of SegmentFlusherFormatTest. For every
registered (storage type, page encoding) pair - DICT once for V1/V2 Segments
(PLAIN dictionary page) and once for V3 (PLAIN_V3 dictionary page), AGG_STATE
once per serialized shape - deterministic rows are converted by
OlapBlockDataConvertor and appended block by block to a ScalarColumnWriter,
the way a Segment writer appends them, with small page budgets so that every
case spans several pages. A file writer keeps each page PageIO writes. The
goldens under be/test/storage/test_data/page_format are one file per page
encoding (bit_shuffle.dat, dict.dat, plain.dat, plain_v2.dat, plain_v3.dat,
prefix.dat, rle.dat), in which the pages of each column follow
one another as they are on disk: the encoded values, the null map, the page
footer and the checksum. layout.txt lists every page with its column, rows and
size, and a failure names the page that changed. Pages are left uncompressed,
so they hold what the page builder encoded and do not depend on a compression
library. Columns hold NULLs, scattered and in a run, unless they are never
nullable (array and map offsets, and the AGG_STATE, HLL, BITMAP and
QUANTILE_STATE columns FE only creates NOT NULL), so pages both with and
without a null map are pinned.
Each column is also read back through ColumnReader and the iterator a Segment
reads it with, and must hold the rows written. layout.txt covers all 86
cases, so a new or removed one changes it, and a second test fails when a
storage type gains an encoding without a spec. Two registered types have no spec: no writer is
given UNSIGNED_INT, as array and map offsets are UNSIGNED_BIGINT, and the root
of a VARIANT is JSONB, which the JSONB pages cover.
FOR is registered for the integer and date types but left out. Its only
writer was the dictionary of a bitmap index, and nothing has written FOR pages
since bitmap indexes were removed; FrameOfReferencePageDecoder cannot read the
values back; and a frame that keeps its original values is written with bytes
ForEncoder never initializes, so its pages are not even the same from run to
run.
The rows cover narrow runs, full-width noise and extremes of every integer
width, NaN with a sign or a payload (written as the canonical NaN), signed
zeros, infinities and denormals, dates at the edges of their range, decimals
at their precision bounds, empty, UTF-8, trailing-space and oversize strings,
a dictionary that overflows into plain pages, empty, single and sparse HLLs,
empty, single and Roaring bitmaps, and quantile states.
Regenerate the goldens with DORIS_PAGE_FORMAT_GOLDEN_OUTPUT_DIR set to a
directory outside the checked-in one, and only for an intended format change.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- PageFormatTest passes against the goldens generated from this tree, also
when run after BitmapValueTest (which leaves bitmap configs changed)
- Changing a page byte, a layout.txt line or the length of a golden file,
or adding a stray golden file, each fails the test and names the page or
file
- Behavior changed: No
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>1 parent bd79755 commit 1dcbc03
9 files changed
Lines changed: 1614 additions & 0 deletions
File tree
- be/test/storage
- segment
- test_data/page_format
Large diffs are not rendered by default.
Binary file not shown.
Binary file not shown.
Large diffs are not rendered by default.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
0 commit comments