|
| 1 | +# Parquet microbenchmark guide for agents |
| 2 | + |
| 3 | +This file applies to `be/benchmark/parquet/`. Read it before changing, running, or interpreting |
| 4 | +these benchmarks. The current suite is a local benchmark foundation, not the complete Parquet |
| 5 | +benchmark system described in the design document. |
| 6 | + |
| 7 | +## What exists today |
| 8 | + |
| 9 | +The benchmark binary registers two groups: |
| 10 | + |
| 11 | +- `ParquetDecoder`: native page decoder benchmarks using in-memory encoded pages. |
| 12 | +- `ParquetReader`: local-file benchmarks that call the format V2 Parquet reader directly. |
| 13 | + |
| 14 | +The relevant files are: |
| 15 | + |
| 16 | +- `benchmark_parquet_decoder.hpp`: deterministic page construction and decoder registration. |
| 17 | +- `benchmark_parquet_reader.hpp`: deterministic local Parquet fixtures and reader registration. |
| 18 | +- `parquet_benchmark_scenarios.h`: scenario definitions and the selected matrix. |
| 19 | +- `README.md`: short human-oriented build and invocation examples. |
| 20 | +- `be/test/format_v2/parquet/parquet_benchmark_scenarios_test.cpp`: matrix invariants. |
| 21 | + |
| 22 | +Do not describe this suite as end-to-end SQL, `FileScannerV2`, remote I/O, V1/V2 comparison, or a |
| 23 | +cross-engine benchmark. The reader benchmark starts at `format::parquet::ParquetReader` and does |
| 24 | +not include FE planning, scanner scheduling, `TableReader`, client latency, or Runtime Profile |
| 25 | +collection. |
| 26 | + |
| 27 | +## Build and list cases |
| 28 | + |
| 29 | +Performance results must come from a Release build: |
| 30 | + |
| 31 | +```shell |
| 32 | +./build.sh --benchmark -j128 |
| 33 | +``` |
| 34 | + |
| 35 | +List all Parquet cases and verify the expected registration counts: |
| 36 | + |
| 37 | +```shell |
| 38 | +be/output/lib/benchmark_test --benchmark_list_tests \ |
| 39 | + | grep -E '^Parquet(Decoder|Reader)/' |
| 40 | + |
| 41 | +be/output/lib/benchmark_test --benchmark_list_tests \ |
| 42 | + | grep -c '^ParquetDecoder/' # currently 152 |
| 43 | + |
| 44 | +be/output/lib/benchmark_test --benchmark_list_tests \ |
| 45 | + | grep -c '^ParquetReader/' # currently 137 |
| 46 | +``` |
| 47 | + |
| 48 | +When running the binary directly from `be/build_RELEASE/bin`, make sure the JVM and third-party |
| 49 | +libraries are discoverable. Prefer the installed `be/output/lib/benchmark_test` produced by |
| 50 | +`build.sh` because the normal Doris environment already configures its dependencies. |
| 51 | + |
| 52 | +## Run smoke verification |
| 53 | + |
| 54 | +A smoke run proves that every registered case initializes and completes. It does not produce a |
| 55 | +stable performance baseline: |
| 56 | + |
| 57 | +```shell |
| 58 | +be/output/lib/benchmark_test \ |
| 59 | + --benchmark_filter='^ParquetDecoder/' \ |
| 60 | + --benchmark_min_time=0.001s \ |
| 61 | + --benchmark_out=parquet-decoder-smoke.json \ |
| 62 | + --benchmark_out_format=json |
| 63 | + |
| 64 | +be/output/lib/benchmark_test \ |
| 65 | + --benchmark_filter='^ParquetReader/' \ |
| 66 | + --benchmark_min_time=0.001s \ |
| 67 | + --benchmark_out=parquet-reader-smoke.json \ |
| 68 | + --benchmark_out_format=json |
| 69 | +``` |
| 70 | + |
| 71 | +Reject a smoke run if the process is non-zero, the expected number of JSON results is absent, or |
| 72 | +any result contains `error_occurred`. Also run the scenario matrix unit test after changing the |
| 73 | +matrix. Never use the 1 ms smoke measurements to claim a speedup or regression. |
| 74 | + |
| 75 | +## Run a performance comparison |
| 76 | + |
| 77 | +Compare the same named cases on the same host, compiler, build flags, CPU set, NUMA node, and power |
| 78 | +policy. Use the same fixture directory and cache state for both revisions. A suitable local command |
| 79 | +is: |
| 80 | + |
| 81 | +```shell |
| 82 | +taskset -c 8 be/output/lib/benchmark_test \ |
| 83 | + --benchmark_filter='^ParquetReader/predicate_scan/plain/null_50/alternating/sel_10/' \ |
| 84 | + --benchmark_min_time=1s \ |
| 85 | + --benchmark_repetitions=10 \ |
| 86 | + --benchmark_report_aggregates_only=true \ |
| 87 | + --benchmark_out=parquet-reader.json \ |
| 88 | + --benchmark_out_format=json |
| 89 | +``` |
| 90 | + |
| 91 | +Use at least three untimed warmups before collecting local warm-cache results. Run revisions in an |
| 92 | +ABBA order when comparing two commits. Do not compare results from different CPUs, cache |
| 93 | +topologies, compiler versions, or build types. |
| 94 | + |
| 95 | +The suite does not currently control the OS page cache. Repeated reader cases are normally |
| 96 | +warm-cache measurements. Do not label them cold-NVMe results. Do not clear a shared machine's page |
| 97 | +cache to manufacture a cold run. |
| 98 | + |
| 99 | +## Current scenario matrix |
| 100 | + |
| 101 | +`ParquetDecoder` contains 19 encoding/type pairs. Each pair is run at 1%, 10%, 50%, and 100% |
| 102 | +selection with clustered and alternating selection ranges, for 152 registered cases. |
| 103 | + |
| 104 | +| Encoding | Physical types | |
| 105 | +|---|---| |
| 106 | +| PLAIN | INT32, INT64, FLOAT, DOUBLE, BYTE_ARRAY, FIXED_LEN_BYTE_ARRAY | |
| 107 | +| Dictionary | INT32, INT64, FLOAT, DOUBLE, BYTE_ARRAY, FIXED_LEN_BYTE_ARRAY | |
| 108 | +| BYTE_STREAM_SPLIT | FLOAT, DOUBLE, FIXED_LEN_BYTE_ARRAY | |
| 109 | +| DELTA_BINARY_PACKED | INT32, INT64 | |
| 110 | +| DELTA_LENGTH_BYTE_ARRAY | BYTE_ARRAY | |
| 111 | +| DELTA_BYTE_ARRAY | BYTE_ARRAY | |
| 112 | + |
| 113 | +`ParquetReader` deliberately uses a single-variable matrix rather than a Cartesian product. After |
| 114 | +deduplication it contains 137 cases covering: |
| 115 | + |
| 116 | +- operations: open-to-first-block, full scan, predicate scan, limit 1, and limit 1000; |
| 117 | +- file encodings: PLAIN, dictionary, BYTE_STREAM_SPLIT, and DELTA_BINARY_PACKED; |
| 118 | +- null ratios: 0%, 1%, 10%, 50%, and 90%; |
| 119 | +- null shapes: clustered and alternating; |
| 120 | +- predicate selectivities: 0%, 1%, 10%, 50%, 90%, and 100%; |
| 121 | +- projection shapes: predicate-only and predicate plus one lazy payload column; |
| 122 | +- schema widths: 4, 32, 128, and 512 columns; |
| 123 | +- predicate position: first or last column. |
| 124 | + |
| 125 | +Except for the axis being varied, reader cases inherit the baseline: nullable INT32, PLAIN, |
| 126 | +alternating 10% nulls, 10% selectivity, 32 columns, predicate at column zero, and predicate plus |
| 127 | +payload projection. |
| 128 | + |
| 129 | +## How decoder data is generated |
| 130 | + |
| 131 | +Decoder pages are constructed in memory before the timed loop. There is no Parquet file, Python |
| 132 | +generator, random seed, or manifest involved. |
| 133 | + |
| 134 | +- Every page contains 65,536 logical values. |
| 135 | +- Integer values use `(row * 17) % 1000003`. |
| 136 | +- Floating-point values use `(row % 1009) * 0.25 - 100.0`. |
| 137 | +- Binary values are fixed at 16 bytes. Their first bytes contain `row % 1009` and the remaining |
| 138 | + bytes are deterministic filler. |
| 139 | +- PLAIN fixed-width values are copied in physical byte order. PLAIN BYTE_ARRAY values use a |
| 140 | + four-byte length followed by payload bytes. |
| 141 | +- BYTE_STREAM_SPLIT pages transpose the byte lanes of the fixed-width PLAIN representation. |
| 142 | +- DELTA pages are produced with the Arrow Parquet DELTA encoders. |
| 143 | +- Dictionary pages contain 256 deterministic entries. IDs repeat from 0 through 255 and are |
| 144 | + encoded with an eight-bit RLE/bit-packed stream. |
| 145 | +- Clustered selection is one continuous range. Alternating selection spreads selected rows evenly |
| 146 | + and intentionally produces many one-row physical ranges. |
| 147 | + |
| 148 | +Page generation, selection construction, decoder creation, dictionary setup, and `set_data` are |
| 149 | +outside the timed decode call. The sinks consume decoder callbacks and prevent compiler removal, |
| 150 | +but they do not build a Doris `Column`. Consequently these cases isolate decoder traversal and |
| 151 | +selection cost; they do not measure definition-level decoding, nullable reconstruction, type |
| 152 | +conversion, or full column materialization. |
| 153 | + |
| 154 | +## How reader Parquet files are generated |
| 155 | + |
| 156 | +Reader fixtures are generated lazily by C++ code using Arrow builders and the Arrow Parquet writer. |
| 157 | +They are stored under: |
| 158 | + |
| 159 | +```text |
| 160 | +${TMPDIR:-/tmp}/doris_parquet_reader_benchmark/ |
| 161 | +``` |
| 162 | + |
| 163 | +Fixture contents and writer settings are: |
| 164 | + |
| 165 | +- 16,384 rows and 4 row groups of 4,096 rows each; |
| 166 | +- all columns are nullable INT32 and reuse the same deterministic Arrow array; |
| 167 | +- every non-null value is `row % 100`; |
| 168 | +- the predicate is `value < selectivity_percent`, so the threshold maps directly to the intended |
| 169 | + non-null selectivity; |
| 170 | +- alternating nulls use a 101-row period and `(row * 37) % 101`, avoiding direct correlation with |
| 171 | + the 100-value predicate period; |
| 172 | +- clustered nulls use contiguous null prefixes inside each 1,024-row cluster; |
| 173 | +- Parquet format version 2.6, DataPage V2, no compression, and statistics disabled; |
| 174 | +- dictionary is explicitly enabled only for dictionary fixtures; other fixtures disable dictionary |
| 175 | + and request their named encoding; |
| 176 | +- after writing, every column chunk in every row group is checked through footer metadata to ensure |
| 177 | + that the writer did not silently choose another encoding. |
| 178 | + |
| 179 | +The fixture name depends only on axes that change file contents. Therefore predicate thresholds, |
| 180 | +projection mode, and operation can reuse the same file. Existing fixtures are footer-validated |
| 181 | +before reuse. Delete only the dedicated `doris_parquet_reader_benchmark` temporary directory when |
| 182 | +fixture rules change; never remove a broad temporary or workspace directory. |
| 183 | + |
| 184 | +Fixture creation and footer verification happen before benchmark iterations. Reader initialization |
| 185 | +and `close()` are excluded from steady-state full/predicate/LIMIT timing. The |
| 186 | +`open_to_first_block` case intentionally includes reader construction, metadata loading, `open()`, |
| 187 | +and the first `get_block()` call. |
| 188 | + |
| 189 | +## Interpret the result correctly |
| 190 | + |
| 191 | +Google Benchmark JSON contains `real_time`, `cpu_time`, repetitions, and custom counters. Use: |
| 192 | + |
| 193 | +- `cpu_time` for decoder CPU efficiency; |
| 194 | +- both `real_time` and `cpu_time` for reader cases, because latency and CPU cost answer different |
| 195 | + questions; |
| 196 | +- median across repetitions as the primary result; |
| 197 | +- p95 and a confidence interval for automated gates; never select the fastest repetition. |
| 198 | + |
| 199 | +Custom counters mean: |
| 200 | + |
| 201 | +- `raw_rows`: logical source rows considered by the case; |
| 202 | +- `selected_rows`: rows returned after the predicate or selection; |
| 203 | +- `ns/raw_row`: normalized source-row cost and the primary comparison across selectivities; |
| 204 | +- `ns/selected_row`: cost per surviving row; it naturally rises as selectivity falls; |
| 205 | +- `selection_ranges`: decoder physical ranges; compare alternating with clustered at the same |
| 206 | + selectivity to expose per-range overhead; |
| 207 | +- `fixture_bytes` or `encoded_bytes`: fixture/page size, not peak memory; |
| 208 | +- `items_per_second`: selected rows per second, so it is not directly comparable across different |
| 209 | + selectivities; |
| 210 | +- `bytes_per_second`: a logical benchmark counter. In reader cases it is estimated from projected |
| 211 | + INT32 values, not measured storage-device bytes or compressed bytes. |
| 212 | + |
| 213 | +Useful comparisons are: |
| 214 | + |
| 215 | +- clustered versus alternating at equal selectivity: sparse-range/cursor overhead; |
| 216 | +- null 0% versus 1/10/50/90% at equal selection: definition-level and nullable reconstruction |
| 217 | + overhead in the reader; |
| 218 | +- predicate-only versus predicate-projected: lazy materialization benefit; |
| 219 | +- width 4/32/128/512 with predicate first/last: schema and metadata traversal overhead; |
| 220 | +- PLAIN versus dictionary/BYTE_STREAM_SPLIT/DELTA at the same reader baseline: encoding path cost; |
| 221 | +- open-to-first-block and LIMIT cases separately from steady-state full scans. |
| 222 | + |
| 223 | +Before attributing a regression, confirm that output row counts are identical, the benchmark name |
| 224 | +and fixture encoding match, and repetition variability is acceptable. A coefficient of variation |
| 225 | +above 3% is inconclusive and should be rerun. Correlate CPU regressions with `perf stat` counters |
| 226 | +such as cycles, instructions, branch misses, and cache misses when possible. |
| 227 | + |
| 228 | +## Coverage audit and required follow-up |
| 229 | + |
| 230 | +The current matrix is not comprehensive. Preserve this distinction in PR descriptions and reports. |
| 231 | + |
| 232 | +### P0: complete the local benchmark phase |
| 233 | + |
| 234 | +1. Add the design's `matrix.yaml`, deterministic corpus generator, `manifest.json`, checksum, and |
| 235 | + standalone corpus verifier. The current runtime-generated files cannot be shared unchanged with |
| 236 | + V1, StarRocks, or DuckDB. |
| 237 | +2. Add correctness oracles. Benchmarks must validate consumed counts and representative checksums |
| 238 | + outside the timed region before their performance samples are trusted. |
| 239 | +3. Add reader-level INT64, FLOAT, DOUBLE, BYTE_ARRAY/string, FIXED_LEN_BYTE_ARRAY, DATE, |
| 240 | + TIMESTAMP, and DECIMAL cases. Today only nullable INT32 reaches the complete reader path. |
| 241 | +4. Extend decoder coverage with 0% and 90% selection, definition levels/null reconstruction, |
| 242 | + dictionary conversion, and real Doris `Column` materialization. The current decoder sink does |
| 243 | + not cover those costs or report decoded bytes per second. |
| 244 | +5. Add the representative nullable sparse corpus requested by the design: 32 INT64 columns, 128 |
| 245 | + row groups, and enough rows to exercise many pages. The current 16K-row/four-row-group fixture |
| 246 | + is a smoke-sized workload. |
| 247 | +6. Add dictionary footprint sweeps based on actual bytes relative to L1/L2 cache, value-width |
| 248 | + sweeps, cardinality sweeps, and dictionary-to-PLAIN fallback/mixed-page files. The current |
| 249 | + dictionary is small and fixed. |
| 250 | +7. Add a Doris V1/V2 SQL runner over the same verified corpus, correctness comparison, Runtime |
| 251 | + Profile collection, normalized result JSON, comparison report, and the first stable local |
| 252 | + baseline. The current 1 ms run only proves executability. |
| 253 | + |
| 254 | +### P1: broaden Parquet format and execution behavior |
| 255 | + |
| 256 | +1. Vary compression codecs: uncompressed, Snappy, ZSTD, LZ4, Brotli, and Gzip where supported. |
| 257 | +2. Cover DataPage V1 and V2, page size, row-group size/count, page boundaries, small files, and |
| 258 | + large files that exceed cache capacity. |
| 259 | +3. Add required/non-nullable columns, correlated and anti-correlated null/predicate distributions, |
| 260 | + random deterministic runs, long null runs, and selections that cross page boundaries. |
| 261 | +4. Add Boolean, INT96 compatibility, logical annotations, timestamp units/time zones, decimal |
| 262 | + physical representations, short/long strings, skew, and empty values. |
| 263 | +5. Add nested ARRAY/MAP/STRUCT and repeated definition/repetition-level cases. |
| 264 | +6. Add row-group statistics, page index, Bloom filter, dictionary filtering, all-filtered batches, |
| 265 | + missing-column/default-value handling, schema mapping, and schema-evolution cases. Keep pruning |
| 266 | + benchmarks separate from decode benchmarks so skipped work is not misattributed to a faster |
| 267 | + decoder. |
| 268 | +7. Add predicate LIMIT 1/1000 cases. Current LIMIT cases only vary reader batch size without a |
| 269 | + predicate. |
| 270 | +8. Add peak tracked memory, allocation counts, and an explicit warm/cold-cache methodology. |
| 271 | + |
| 272 | +### Later phases |
| 273 | + |
| 274 | +Remote S3/FileCache/RTT workloads, prefetch metrics, cross-engine adapters, nightly/weekly jobs, and |
| 275 | +release regression gates belong to later phases. They require isolated infrastructure and must not |
| 276 | +be simulated by silently changing the local reader benchmark. |
| 277 | + |
| 278 | +## Current validation record |
| 279 | + |
| 280 | +At commit `16e05dd5c71`, a Release build completed and the matrix unit test passed 4/4. A 1 ms smoke |
| 281 | +run executed 152 decoder and 137 reader cases with zero benchmark errors. This is an execution |
| 282 | +record only. It is not a reviewed performance baseline because repetitions, host isolation, |
| 283 | +warmups, cache control, `perf` data, variance, and before/after comparison were not collected. |
| 284 | + |
| 285 | +## Rules for extending the suite |
| 286 | + |
| 287 | +- Keep deterministic data construction and validation outside timed regions. |
| 288 | +- Verify actual footer encodings; never trust writer configuration alone. |
| 289 | +- Change one primary axis at a time and add only meaningful second-order interactions. |
| 290 | +- Add or update matrix invariant tests before registering new dimensions. |
| 291 | +- Keep correctness/error-injection tests separate from throughput benchmarks. |
| 292 | +- Do not commit large generated corpora. Commit generation rules and manifests; keep only small |
| 293 | + correctness fixtures in the repository when necessary. |
| 294 | +- Record the exact commit, compiler, build type, host topology, command, repetitions, cache state, |
| 295 | + and raw JSON for any claimed performance result. |
| 296 | +- Never claim a performance improvement from a smoke run or from incomparable machines. |
0 commit comments