Skip to content

Commit 1dcbc03

Browse files
csun5285claude
andcommitted
[test](be) Pin the on-disk pages of every page encoding with golden files
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Nothing pins the bytes of the pages a column writer writes. The bitshuffle, plain, RLE, frame-of-reference, binary plain and binary prefix page tests are excluded from the UT build, and the page tests that do run round-trip values without comparing bytes. SegmentFlusherFormatTest pins whole Segments, but its columns hold a handful of rows, so no page ever fills up, and only the encodings the writers pick are written (never PLAIN_V2, for example). Add PageFormatTest, in the style of SegmentFlusherFormatTest. For every registered (storage type, page encoding) pair - DICT once for V1/V2 Segments (PLAIN dictionary page) and once for V3 (PLAIN_V3 dictionary page), AGG_STATE once per serialized shape - deterministic rows are converted by OlapBlockDataConvertor and appended block by block to a ScalarColumnWriter, the way a Segment writer appends them, with small page budgets so that every case spans several pages. A file writer keeps each page PageIO writes. The goldens under be/test/storage/test_data/page_format are one file per page encoding (bit_shuffle.dat, dict.dat, plain.dat, plain_v2.dat, plain_v3.dat, prefix.dat, rle.dat), in which the pages of each column follow one another as they are on disk: the encoded values, the null map, the page footer and the checksum. layout.txt lists every page with its column, rows and size, and a failure names the page that changed. Pages are left uncompressed, so they hold what the page builder encoded and do not depend on a compression library. Columns hold NULLs, scattered and in a run, unless they are never nullable (array and map offsets, and the AGG_STATE, HLL, BITMAP and QUANTILE_STATE columns FE only creates NOT NULL), so pages both with and without a null map are pinned. Each column is also read back through ColumnReader and the iterator a Segment reads it with, and must hold the rows written. layout.txt covers all 86 cases, so a new or removed one changes it, and a second test fails when a storage type gains an encoding without a spec. Two registered types have no spec: no writer is given UNSIGNED_INT, as array and map offsets are UNSIGNED_BIGINT, and the root of a VARIANT is JSONB, which the JSONB pages cover. FOR is registered for the integer and date types but left out. Its only writer was the dictionary of a bitmap index, and nothing has written FOR pages since bitmap indexes were removed; FrameOfReferencePageDecoder cannot read the values back; and a frame that keeps its original values is written with bytes ForEncoder never initializes, so its pages are not even the same from run to run. The rows cover narrow runs, full-width noise and extremes of every integer width, NaN with a sign or a payload (written as the canonical NaN), signed zeros, infinities and denormals, dates at the edges of their range, decimals at their precision bounds, empty, UTF-8, trailing-space and oversize strings, a dictionary that overflows into plain pages, empty, single and sparse HLLs, empty, single and Roaring bitmaps, and quantile states. Regenerate the goldens with DORIS_PAGE_FORMAT_GOLDEN_OUTPUT_DIR set to a directory outside the checked-in one, and only for an intended format change. ### Release note None ### Check List (For Author) - Test: Unit Test - PageFormatTest passes against the goldens generated from this tree, also when run after BitmapValueTest (which leaves bitmap configs changed) - Changing a page byte, a layout.txt line or the length of a golden file, or adding a stray golden file, each fails the test and names the page or file - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent bd79755 commit 1dcbc03

9 files changed

Lines changed: 1614 additions & 0 deletions

File tree

‎be/test/storage/segment/page_format_test.cpp‎

Lines changed: 1112 additions & 0 deletions
Large diffs are not rendered by default.
31.2 KB
Binary file not shown.
28.7 KB
Binary file not shown.

‎be/test/storage/test_data/page_format/layout.txt‎

Lines changed: 502 additions & 0 deletions
Large diffs are not rendered by default.
85.9 KB
Binary file not shown.
31.6 KB
Binary file not shown.
31.9 KB
Binary file not shown.
19.1 KB
Binary file not shown.
380 Bytes
Binary file not shown.

0 commit comments

Comments
 (0)