Our default is to run jobs like this every 6 hours. What are the constraints that limit this to weekly?

(Also you need to run the seed job to get your Jenkins changes to load. I believe Run Seed Job worked in the past, but you might need to invoke it via the Jenkins UI.)

Run Seed Job

Run Java Jmh benchmarks

@apilloud Thanks a lot! The "problem" with JMH benchmarks is that they take a massive amount of time. They are running a large number of iterations including various forks and warmups to produce a more stable and accurate number.

With the current (default) JMH settings these would run for ~ 14 hours:

~100 benchmarks * 5 forks * (5 warmup iterations + 5 measurement iterations) * 10 seconds

If using just 3 forks it's running for ~ 8.5 hours, which seems more reasonable.
Later, once we have some numbers, we might be able to reduce the number of iterations slightly and see how that impacts (stability of) results.
cc @lukecwik

Codecov Report

Merging #23388 (d02611e) into master (7dd1c6b) will increase coverage by 0.02%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master   #23388      +/-   ##
==========================================
+ Coverage   73.40%   73.43%   +0.02%     
==========================================
  Files         718      718              
  Lines       95652    95680      +28     
==========================================
+ Hits        70215    70263      +48     
+ Misses      24126    24106      -20     
  Partials     1311     1311              

Flags with carried forward coverage won't be shown. Click here to find out more.

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Run Seed Job

It would be great to run this daily or every 2 days. @mosche or @apilloud, do you know how busy or perf machine is already?

Run length of over ~3 hours is going to be disruptive to existing tests regardless of run length. I think the behavior of Jenkins is to just run tests less frequently? Daily would be a good start. Our current tests run every 6 hours and it looks like we are busy ~25% of the time: https://ci-beam.apache.org/label/beam-perf/load-statistics?type=hour

We could easily split the job into two JMH runs since there are two JMH projects currently. We can also split it very finely if we utilize -Pbenchmark=... and run one suite at a time.

Sorry @lukecwik, I was off for a couple of days. Thanks for merging 👍

We could easily split the job into two JMH runs since there are two JMH projects currently.

The benchmarks of core take ~ 7h, compared to ~ 1,5h for the fn harness. That won't be enough yet.

We can also split it very finely if we utilize -Pbenchmark=... and run one suite at a time.

For parametrized benchmarks runtimes vary a lot, but it would certainly be possible to split the benchmarks into equal sized chunks this way. Though, I'm a bit worried that this isn't maintainable long term. Of course it doesn't have to be perfect, but every new benchmark has to be added manually and scheduled appropriately...

I'll think a bit further about it.

I agree that -Pbenchmark= isn't a great solution because of how much maintenance it would take.

@lukecwik looking at the benchmark runtimes, we could split the benchmarks into 3 groups and run them 2/3 times weekly

• ByteStringOutputStreamBenchmark: Tue, Sat
• GetterBasedSchemaProviderBenchmark: Thu, Sun
• Everything else: Mon, Wed, Fri

I was briefly looking at implementing this, but without a plugin to conditionally run steps (conditionalStep) this would have to be implemented as 3 jobs (or use a shell step to trigger gradle tasks, but loosing all the default switches). Thoughts?

Here's the respective dashboard
http://metrics.beam.apache.org/d/kllfR2vVk/java-jmh-benchmarks?orgId=1&from=now-34d&to=now

Do you think we can get it listed on the getting started page http://metrics.beam.apache.org/d/1/getting-started?orgId=1 ?

Do you think we can get it listed on the getting started page http://metrics.beam.apache.org/d/1/getting-started?orgId=1 ?

Absolutely, wasn't aware of the the necessary tag 👍 See #24335