Skip to content

Commit 28593d5

Browse files
author
Steve Loughran
authored
GH-3451: Add JMH benchmark for variants (#3452)
* GH-3451. Add a JMH benchmark for variants Initial impl. * WiP: add a deeper version. * revert plans for a deeper version * fixes to this benchmark (copilot review) * deser to recurse down * include uuid and bigdecimal * reset counter on benchmark setup * Measure parquet write/read costs. iterations of class code and #of rows are the same for easy compare of overheads. * WiP, on a file read/write benchmark Using the same structure as the iceberg tests do * building a projection schema * projection works * Test to validate impact of lean schema against unshredded files. painful. * GH-561 ongoing work * GH-561 profile driven review of VariantConverters. * GH-561 Variant benchmarks * Move under variant package to access private members * add "variant" group to run.sh * new benchmark VariantConverterBenchmark * spotless * Benchmark enhancements * UUIDs in projection benchmark. * col4 var column now a long string and named as such. * builder and converter benchmarks use @OperationsPerInvocation rather than internal iterators. * revert production code enhancement. including pulling a benchmark from VariantConverters. Pull the warmup/measurement times to a shared constant. If someone thinks the numbers should be bigger: one place to change.
1 parent 492b686 commit 28593d5

6 files changed

Lines changed: 1858 additions & 0 deletions

File tree

‎parquet-benchmarks/pom.xml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,11 @@
5252
<artifactId>parquet-common</artifactId>
5353
<version>${project.version}</version>
5454
</dependency>
55+
<dependency>
56+
<groupId>org.apache.parquet</groupId>
57+
<artifactId>parquet-variant</artifactId>
58+
<version>${project.version}</version>
59+
</dependency>
5560
<dependency>
5661
<groupId>org.apache.hadoop</groupId>
5762
<artifactId>hadoop-client</artifactId>

‎parquet-benchmarks/run.sh‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,7 @@ read | Reading files with different compression, page and block sizes.
4343
write | Writing files.
4444
checksum | Reading and writing with and without CRC checksums.
4545
filter | Filtering column indexes
46+
variant | Variant performance
4647
4748
Examples:
4849
@@ -100,6 +101,9 @@ else
100101
"filter")
101102
BENCHMARK_REGEX="org.apache.parquet.benchmarks.FilteringBenchmarks"
102103
;;
104+
"variant")
105+
BENCHMARK_REGEX="org.apache.parquet.variant.Variant*"
106+
;;
103107
esac
104108

105109
echo JMH command: java -jar ${SCRIPT_PATH}/target/parquet-benchmarks.jar $BENCHMARK_REGEX $JMH_OPTIONS
Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing, software
13+
* distributed under the License is distributed on an "AS IS" BASIS,
14+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
* See the License for the specific language governing permissions and
16+
* limitations under the License.
17+
*/
18+
19+
package org.apache.parquet.variant;
20+
21+
/**
22+
* Common measurement settings.
23+
*/
24+
class VariantBenchmarkMeasurementSettings {
25+
26+
/**
27+
* How long to warm up. Goal: JIT compiler has done its work.
28+
* If flamegraphs show compilation: increase.
29+
*/
30+
static final int SMALL_BENCHMARK_WARMUP = 500;
31+
32+
/**
33+
* Meaurement for the small benchmarks.
34+
*/
35+
static final int SMALL_BENCHMARK_MEASUREMENTS = 500;
36+
}

0 commit comments

Comments
 (0)