Skip to content

Commit 9ec7f76

Browse files
authored
[bench](parquet) Add native reader microbenchmark matrix (apache#65921)
## What problem does this PR solve? Adds a reproducible local microbenchmark foundation for the native Parquet reader described in the [benchmark design]. It separates page decoder cost from end-to-end local reader cost and keeps fixture construction and encoding validation outside measured regions. ## What is changed? - Add 152 native decoder cases covering PLAIN, dictionary, byte-stream-split, and DELTA encodings across fixed-width and binary physical types, with clustered and alternating sparse selections. - Add 137 local reader cases covering open-to-first-block, full scan, predicate scan, LIMIT shapes, null density/pattern, predicate selectivity, projection mode, schema width, predicate position, and four physical encodings. - Generate deterministic fixtures lazily and validate every row group and column encoding from Parquet footer metadata before measuring. - Report rows/s, bytes/s, raw/selected rows, fixture size, ns/raw row, and ns/selected row. - Add matrix constraint tests and local build/run documentation. ## Check List - [x] Release benchmark target builds with `ninja -C be/build_RELEASE -j128 benchmark_test` - [x] Scenario matrix tests pass (4/4) - [x] All 152 decoder benchmarks pass - [x] All 137 reader benchmarks pass - [x] clang-format 16 and `git diff --check` pass
1 parent cbf851d commit 9ec7f76

7 files changed

Lines changed: 1704 additions & 0 deletions

File tree

‎be/benchmark/benchmark_main.cpp‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,11 @@
1717

1818
#include <benchmark/benchmark.h>
1919

20+
#include <cstdlib>
21+
#include <filesystem>
22+
#include <iostream>
23+
#include <vector>
24+
2025
#include "benchmark_arrow_validation.hpp"
2126
#include "benchmark_bit_pack.hpp"
2227
#include "benchmark_bits.hpp"
@@ -39,6 +44,8 @@
3944
#include "core/column/column_string.h"
4045
#include "core/data_type/data_type.h"
4146
#include "core/data_type/data_type_string.h"
47+
#include "parquet/benchmark_parquet_decoder.hpp"
48+
#include "parquet/benchmark_parquet_reader.hpp"
4249
#include "runtime/exec_env.h"
4350
#include "runtime/memory/mem_tracker_limiter.h"
4451
#include "runtime/memory/thread_mem_tracker_mgr.h"
@@ -52,6 +59,35 @@
5259

5360
namespace doris { // change if need
5461

62+
static bool init_benchmark_config(const char* executable) {
63+
std::vector<std::filesystem::path> candidates;
64+
if (const char* doris_home = std::getenv("DORIS_HOME"); doris_home != nullptr) {
65+
candidates.emplace_back(std::filesystem::path(doris_home) / "conf" / "be.conf");
66+
}
67+
candidates.emplace_back(std::filesystem::current_path() / "conf" / "be.conf");
68+
const auto executable_dir = std::filesystem::absolute(executable).parent_path();
69+
candidates.emplace_back(executable_dir / ".." / ".." / "conf" / "be.conf");
70+
candidates.emplace_back(executable_dir / ".." / ".." / ".." / "conf" / "be.conf");
71+
for (const auto& candidate : candidates) {
72+
if (!std::filesystem::exists(candidate)) {
73+
continue;
74+
}
75+
if (std::getenv("DORIS_HOME") == nullptr) {
76+
const auto inferred_home = candidate.parent_path().parent_path().string();
77+
// be.conf contains paths relative to DORIS_HOME. Keep standalone benchmark launches
78+
// equivalent to build/test scripts after locating the same repository config.
79+
if (::setenv("DORIS_HOME", inferred_home.c_str(), 0) != 0) {
80+
return false;
81+
}
82+
}
83+
// Mutable config storage is populated by config::init. Reader benchmarks must use the
84+
// production defaults because zero-initialized safety limits reject every valid footer.
85+
return config::init(candidate.c_str(), false);
86+
}
87+
std::cerr << "Unable to find conf/be.conf for benchmark initialization\n";
88+
return false;
89+
}
90+
5591
static void Example1(benchmark::State& state) {
5692
// init. dont time it.
5793
state.PauseTiming();
@@ -78,6 +114,9 @@ BENCHMARK(Example1);
78114
// ThreadContext + mem tracker, otherwise the allocator throws E-7412. Mirrors
79115
// the minimal subset of be/test/testutil/run_all_tests.cpp::main.
80116
int main(int argc, char** argv) {
117+
if (!doris::init_benchmark_config(argv[0])) {
118+
return 1;
119+
}
81120
doris::config::enable_bmi2_optimizations = true;
82121

83122
SCOPED_INIT_THREAD_CONTEXT();

‎be/benchmark/parquet/AGENTS.md‎

Lines changed: 296 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,296 @@
1+
# Parquet microbenchmark guide for agents
2+
3+
This file applies to `be/benchmark/parquet/`. Read it before changing, running, or interpreting
4+
these benchmarks. The current suite is a local benchmark foundation, not the complete Parquet
5+
benchmark system described in the design document.
6+
7+
## What exists today
8+
9+
The benchmark binary registers two groups:
10+
11+
- `ParquetDecoder`: native page decoder benchmarks using in-memory encoded pages.
12+
- `ParquetReader`: local-file benchmarks that call the format V2 Parquet reader directly.
13+
14+
The relevant files are:
15+
16+
- `benchmark_parquet_decoder.hpp`: deterministic page construction and decoder registration.
17+
- `benchmark_parquet_reader.hpp`: deterministic local Parquet fixtures and reader registration.
18+
- `parquet_benchmark_scenarios.h`: scenario definitions and the selected matrix.
19+
- `README.md`: short human-oriented build and invocation examples.
20+
- `be/test/format_v2/parquet/parquet_benchmark_scenarios_test.cpp`: matrix invariants.
21+
22+
Do not describe this suite as end-to-end SQL, `FileScannerV2`, remote I/O, V1/V2 comparison, or a
23+
cross-engine benchmark. The reader benchmark starts at `format::parquet::ParquetReader` and does
24+
not include FE planning, scanner scheduling, `TableReader`, client latency, or Runtime Profile
25+
collection.
26+
27+
## Build and list cases
28+
29+
Performance results must come from a Release build:
30+
31+
```shell
32+
./build.sh --benchmark -j128
33+
```
34+
35+
List all Parquet cases and verify the expected registration counts:
36+
37+
```shell
38+
be/output/lib/benchmark_test --benchmark_list_tests \
39+
| grep -E '^Parquet(Decoder|Reader)/'
40+
41+
be/output/lib/benchmark_test --benchmark_list_tests \
42+
| grep -c '^ParquetDecoder/' # currently 152
43+
44+
be/output/lib/benchmark_test --benchmark_list_tests \
45+
| grep -c '^ParquetReader/' # currently 137
46+
```
47+
48+
When running the binary directly from `be/build_RELEASE/bin`, make sure the JVM and third-party
49+
libraries are discoverable. Prefer the installed `be/output/lib/benchmark_test` produced by
50+
`build.sh` because the normal Doris environment already configures its dependencies.
51+
52+
## Run smoke verification
53+
54+
A smoke run proves that every registered case initializes and completes. It does not produce a
55+
stable performance baseline:
56+
57+
```shell
58+
be/output/lib/benchmark_test \
59+
--benchmark_filter='^ParquetDecoder/' \
60+
--benchmark_min_time=0.001s \
61+
--benchmark_out=parquet-decoder-smoke.json \
62+
--benchmark_out_format=json
63+
64+
be/output/lib/benchmark_test \
65+
--benchmark_filter='^ParquetReader/' \
66+
--benchmark_min_time=0.001s \
67+
--benchmark_out=parquet-reader-smoke.json \
68+
--benchmark_out_format=json
69+
```
70+
71+
Reject a smoke run if the process is non-zero, the expected number of JSON results is absent, or
72+
any result contains `error_occurred`. Also run the scenario matrix unit test after changing the
73+
matrix. Never use the 1 ms smoke measurements to claim a speedup or regression.
74+
75+
## Run a performance comparison
76+
77+
Compare the same named cases on the same host, compiler, build flags, CPU set, NUMA node, and power
78+
policy. Use the same fixture directory and cache state for both revisions. A suitable local command
79+
is:
80+
81+
```shell
82+
taskset -c 8 be/output/lib/benchmark_test \
83+
--benchmark_filter='^ParquetReader/predicate_scan/plain/null_50/alternating/sel_10/' \
84+
--benchmark_min_time=1s \
85+
--benchmark_repetitions=10 \
86+
--benchmark_report_aggregates_only=true \
87+
--benchmark_out=parquet-reader.json \
88+
--benchmark_out_format=json
89+
```
90+
91+
Use at least three untimed warmups before collecting local warm-cache results. Run revisions in an
92+
ABBA order when comparing two commits. Do not compare results from different CPUs, cache
93+
topologies, compiler versions, or build types.
94+
95+
The suite does not currently control the OS page cache. Repeated reader cases are normally
96+
warm-cache measurements. Do not label them cold-NVMe results. Do not clear a shared machine's page
97+
cache to manufacture a cold run.
98+
99+
## Current scenario matrix
100+
101+
`ParquetDecoder` contains 19 encoding/type pairs. Each pair is run at 1%, 10%, 50%, and 100%
102+
selection with clustered and alternating selection ranges, for 152 registered cases.
103+
104+
| Encoding | Physical types |
105+
|---|---|
106+
| PLAIN | INT32, INT64, FLOAT, DOUBLE, BYTE_ARRAY, FIXED_LEN_BYTE_ARRAY |
107+
| Dictionary | INT32, INT64, FLOAT, DOUBLE, BYTE_ARRAY, FIXED_LEN_BYTE_ARRAY |
108+
| BYTE_STREAM_SPLIT | FLOAT, DOUBLE, FIXED_LEN_BYTE_ARRAY |
109+
| DELTA_BINARY_PACKED | INT32, INT64 |
110+
| DELTA_LENGTH_BYTE_ARRAY | BYTE_ARRAY |
111+
| DELTA_BYTE_ARRAY | BYTE_ARRAY |
112+
113+
`ParquetReader` deliberately uses a single-variable matrix rather than a Cartesian product. After
114+
deduplication it contains 137 cases covering:
115+
116+
- operations: open-to-first-block, full scan, predicate scan, limit 1, and limit 1000;
117+
- file encodings: PLAIN, dictionary, BYTE_STREAM_SPLIT, and DELTA_BINARY_PACKED;
118+
- null ratios: 0%, 1%, 10%, 50%, and 90%;
119+
- null shapes: clustered and alternating;
120+
- predicate selectivities: 0%, 1%, 10%, 50%, 90%, and 100%;
121+
- projection shapes: predicate-only and predicate plus one lazy payload column;
122+
- schema widths: 4, 32, 128, and 512 columns;
123+
- predicate position: first or last column.
124+
125+
Except for the axis being varied, reader cases inherit the baseline: nullable INT32, PLAIN,
126+
alternating 10% nulls, 10% selectivity, 32 columns, predicate at column zero, and predicate plus
127+
payload projection.
128+
129+
## How decoder data is generated
130+
131+
Decoder pages are constructed in memory before the timed loop. There is no Parquet file, Python
132+
generator, random seed, or manifest involved.
133+
134+
- Every page contains 65,536 logical values.
135+
- Integer values use `(row * 17) % 1000003`.
136+
- Floating-point values use `(row % 1009) * 0.25 - 100.0`.
137+
- Binary values are fixed at 16 bytes. Their first bytes contain `row % 1009` and the remaining
138+
bytes are deterministic filler.
139+
- PLAIN fixed-width values are copied in physical byte order. PLAIN BYTE_ARRAY values use a
140+
four-byte length followed by payload bytes.
141+
- BYTE_STREAM_SPLIT pages transpose the byte lanes of the fixed-width PLAIN representation.
142+
- DELTA pages are produced with the Arrow Parquet DELTA encoders.
143+
- Dictionary pages contain 256 deterministic entries. IDs repeat from 0 through 255 and are
144+
encoded with an eight-bit RLE/bit-packed stream.
145+
- Clustered selection is one continuous range. Alternating selection spreads selected rows evenly
146+
and intentionally produces many one-row physical ranges.
147+
148+
Page generation, selection construction, decoder creation, dictionary setup, and `set_data` are
149+
outside the timed decode call. The sinks consume decoder callbacks and prevent compiler removal,
150+
but they do not build a Doris `Column`. Consequently these cases isolate decoder traversal and
151+
selection cost; they do not measure definition-level decoding, nullable reconstruction, type
152+
conversion, or full column materialization.
153+
154+
## How reader Parquet files are generated
155+
156+
Reader fixtures are generated lazily by C++ code using Arrow builders and the Arrow Parquet writer.
157+
They are stored under:
158+
159+
```text
160+
${TMPDIR:-/tmp}/doris_parquet_reader_benchmark/
161+
```
162+
163+
Fixture contents and writer settings are:
164+
165+
- 16,384 rows and 4 row groups of 4,096 rows each;
166+
- all columns are nullable INT32 and reuse the same deterministic Arrow array;
167+
- every non-null value is `row % 100`;
168+
- the predicate is `value < selectivity_percent`, so the threshold maps directly to the intended
169+
non-null selectivity;
170+
- alternating nulls use a 101-row period and `(row * 37) % 101`, avoiding direct correlation with
171+
the 100-value predicate period;
172+
- clustered nulls use contiguous null prefixes inside each 1,024-row cluster;
173+
- Parquet format version 2.6, DataPage V2, no compression, and statistics disabled;
174+
- dictionary is explicitly enabled only for dictionary fixtures; other fixtures disable dictionary
175+
and request their named encoding;
176+
- after writing, every column chunk in every row group is checked through footer metadata to ensure
177+
that the writer did not silently choose another encoding.
178+
179+
The fixture name depends only on axes that change file contents. Therefore predicate thresholds,
180+
projection mode, and operation can reuse the same file. Existing fixtures are footer-validated
181+
before reuse. Delete only the dedicated `doris_parquet_reader_benchmark` temporary directory when
182+
fixture rules change; never remove a broad temporary or workspace directory.
183+
184+
Fixture creation and footer verification happen before benchmark iterations. Reader initialization
185+
and `close()` are excluded from steady-state full/predicate/LIMIT timing. The
186+
`open_to_first_block` case intentionally includes reader construction, metadata loading, `open()`,
187+
and the first `get_block()` call.
188+
189+
## Interpret the result correctly
190+
191+
Google Benchmark JSON contains `real_time`, `cpu_time`, repetitions, and custom counters. Use:
192+
193+
- `cpu_time` for decoder CPU efficiency;
194+
- both `real_time` and `cpu_time` for reader cases, because latency and CPU cost answer different
195+
questions;
196+
- median across repetitions as the primary result;
197+
- p95 and a confidence interval for automated gates; never select the fastest repetition.
198+
199+
Custom counters mean:
200+
201+
- `raw_rows`: logical source rows considered by the case;
202+
- `selected_rows`: rows returned after the predicate or selection;
203+
- `ns/raw_row`: normalized source-row cost and the primary comparison across selectivities;
204+
- `ns/selected_row`: cost per surviving row; it naturally rises as selectivity falls;
205+
- `selection_ranges`: decoder physical ranges; compare alternating with clustered at the same
206+
selectivity to expose per-range overhead;
207+
- `fixture_bytes` or `encoded_bytes`: fixture/page size, not peak memory;
208+
- `items_per_second`: selected rows per second, so it is not directly comparable across different
209+
selectivities;
210+
- `bytes_per_second`: a logical benchmark counter. In reader cases it is estimated from projected
211+
INT32 values, not measured storage-device bytes or compressed bytes.
212+
213+
Useful comparisons are:
214+
215+
- clustered versus alternating at equal selectivity: sparse-range/cursor overhead;
216+
- null 0% versus 1/10/50/90% at equal selection: definition-level and nullable reconstruction
217+
overhead in the reader;
218+
- predicate-only versus predicate-projected: lazy materialization benefit;
219+
- width 4/32/128/512 with predicate first/last: schema and metadata traversal overhead;
220+
- PLAIN versus dictionary/BYTE_STREAM_SPLIT/DELTA at the same reader baseline: encoding path cost;
221+
- open-to-first-block and LIMIT cases separately from steady-state full scans.
222+
223+
Before attributing a regression, confirm that output row counts are identical, the benchmark name
224+
and fixture encoding match, and repetition variability is acceptable. A coefficient of variation
225+
above 3% is inconclusive and should be rerun. Correlate CPU regressions with `perf stat` counters
226+
such as cycles, instructions, branch misses, and cache misses when possible.
227+
228+
## Coverage audit and required follow-up
229+
230+
The current matrix is not comprehensive. Preserve this distinction in PR descriptions and reports.
231+
232+
### P0: complete the local benchmark phase
233+
234+
1. Add the design's `matrix.yaml`, deterministic corpus generator, `manifest.json`, checksum, and
235+
standalone corpus verifier. The current runtime-generated files cannot be shared unchanged with
236+
V1, StarRocks, or DuckDB.
237+
2. Add correctness oracles. Benchmarks must validate consumed counts and representative checksums
238+
outside the timed region before their performance samples are trusted.
239+
3. Add reader-level INT64, FLOAT, DOUBLE, BYTE_ARRAY/string, FIXED_LEN_BYTE_ARRAY, DATE,
240+
TIMESTAMP, and DECIMAL cases. Today only nullable INT32 reaches the complete reader path.
241+
4. Extend decoder coverage with 0% and 90% selection, definition levels/null reconstruction,
242+
dictionary conversion, and real Doris `Column` materialization. The current decoder sink does
243+
not cover those costs or report decoded bytes per second.
244+
5. Add the representative nullable sparse corpus requested by the design: 32 INT64 columns, 128
245+
row groups, and enough rows to exercise many pages. The current 16K-row/four-row-group fixture
246+
is a smoke-sized workload.
247+
6. Add dictionary footprint sweeps based on actual bytes relative to L1/L2 cache, value-width
248+
sweeps, cardinality sweeps, and dictionary-to-PLAIN fallback/mixed-page files. The current
249+
dictionary is small and fixed.
250+
7. Add a Doris V1/V2 SQL runner over the same verified corpus, correctness comparison, Runtime
251+
Profile collection, normalized result JSON, comparison report, and the first stable local
252+
baseline. The current 1 ms run only proves executability.
253+
254+
### P1: broaden Parquet format and execution behavior
255+
256+
1. Vary compression codecs: uncompressed, Snappy, ZSTD, LZ4, Brotli, and Gzip where supported.
257+
2. Cover DataPage V1 and V2, page size, row-group size/count, page boundaries, small files, and
258+
large files that exceed cache capacity.
259+
3. Add required/non-nullable columns, correlated and anti-correlated null/predicate distributions,
260+
random deterministic runs, long null runs, and selections that cross page boundaries.
261+
4. Add Boolean, INT96 compatibility, logical annotations, timestamp units/time zones, decimal
262+
physical representations, short/long strings, skew, and empty values.
263+
5. Add nested ARRAY/MAP/STRUCT and repeated definition/repetition-level cases.
264+
6. Add row-group statistics, page index, Bloom filter, dictionary filtering, all-filtered batches,
265+
missing-column/default-value handling, schema mapping, and schema-evolution cases. Keep pruning
266+
benchmarks separate from decode benchmarks so skipped work is not misattributed to a faster
267+
decoder.
268+
7. Add predicate LIMIT 1/1000 cases. Current LIMIT cases only vary reader batch size without a
269+
predicate.
270+
8. Add peak tracked memory, allocation counts, and an explicit warm/cold-cache methodology.
271+
272+
### Later phases
273+
274+
Remote S3/FileCache/RTT workloads, prefetch metrics, cross-engine adapters, nightly/weekly jobs, and
275+
release regression gates belong to later phases. They require isolated infrastructure and must not
276+
be simulated by silently changing the local reader benchmark.
277+
278+
## Current validation record
279+
280+
At commit `16e05dd5c71`, a Release build completed and the matrix unit test passed 4/4. A 1 ms smoke
281+
run executed 152 decoder and 137 reader cases with zero benchmark errors. This is an execution
282+
record only. It is not a reviewed performance baseline because repetitions, host isolation,
283+
warmups, cache control, `perf` data, variance, and before/after comparison were not collected.
284+
285+
## Rules for extending the suite
286+
287+
- Keep deterministic data construction and validation outside timed regions.
288+
- Verify actual footer encodings; never trust writer configuration alone.
289+
- Change one primary axis at a time and add only meaningful second-order interactions.
290+
- Add or update matrix invariant tests before registering new dimensions.
291+
- Keep correctness/error-injection tests separate from throughput benchmarks.
292+
- Do not commit large generated corpora. Commit generation rules and manifests; keep only small
293+
correctness fixtures in the repository when necessary.
294+
- Record the exact commit, compiler, build type, host topology, command, repetitions, cache state,
295+
and raw JSON for any claimed performance result.
296+
- Never claim a performance improvement from a smoke run or from incomparable machines.

0 commit comments

Comments
 (0)