The straightforward way of running a Java benchmark is through the following command:

java  -jar ./performance/target/benchmarks.jar 

However, it is like to fail with the following message:

ERROR: Unable to find the resource: /META-INF/BenchmarkList

The fundamental reason for this error is missing some jars in the class path, and it is considered a bug of maven compilation plugin (https://issues.apache.org/jira/browse/MCOMPILER-369)

Since there are many jars required (directly or indirected referenced), we recommend running the benchmarks through maven.

To run all benchmarks, please go to the arrow-performance directory, and invoke the command:

mvn clean install -Dskip.perf.benchmarks=false

We also support filtering specific benchmarks, for example:

mvn clean install -Dskip.perf.benchmarks=false -Dbenchmark.filter=.*IntBenchmarks.setIntDirectly

In addtion, we also support configuring other benchmark parameters, for example:

mvn clean install -Dskip.perf.benchmarks=false -Dbenchmark.filter=.*IntBenchmarks.setIntDirectly -Dbenchmark.forks=1 -Dbenchmark.warmups=5 -Dbenchmark.runs=5

https://issues.apache.org/jira/browse/ARROW-10069

Thank you. Looks good.

In my environment, while benchmark.forks works, benchmark.warmups and benchmark.runs do not work. In other words, the following command executes the benchmark twice with 5 warmups and 5 runs.

mvn clean install  -Dskip.perf.benchmarks=false -Dbenchmark.filter=.*IntBenchmarks.setIntDirectly  -Dbenchmark.forks=2 -Dbenchmark.warmups=6 -Dbenchmark.runs=7

@lidavidm Thanks for your feedback. Please see my reply in line.

LGTM. Two questions:

• Should we include this in the developer docs in Sphinx, or in the README in the Java subproject?

Thanks for the good suggestion. It seems we do not have a development guide for Java. Maybe we should add one in a separate issue.

• How do I run the benchmarks without cleaning and re-building the project? (This seems to do it: mvn install --also-make -Dskip.perf.benchmarks=false -Dbenchmark.filter=.\*IntBenchmarks.setIntDirectly -pl :arrow-performance)

Good question. We can run

mvn exec:exec -Dskip.perf.benchmarks=false ...

However, it is fast to rebuild the code, as there are only a few classes in the performance module. Rebuilding is helpful for making the generated benchmark code up to date.

Thank you. Looks good.

In my environment, while benchmark.forks works, benchmark.warmups and benchmark.runs do not work. In other words, the following command executes the benchmark twice with 5 warmups and 5 runs.

mvn clean install  -Dskip.perf.benchmarks=false -Dbenchmark.filter=.*IntBenchmarks.setIntDirectly  -Dbenchmark.forks=2 -Dbenchmark.warmups=6 -Dbenchmark.runs=7

@kiszk Thanks for your feedback.
The problem was caused by incorrect parameter names. I have fixed it now, please check.

Thank you. Now, these three parameters work correctly as we expect.