Add Kokoro performance job for release branch#14282
Conversation
|
|
matt-kwong
force-pushed
the
release-jobs
branch
from
b0e524c to
cba95fc
Compare
|
|
|
1 similar comment
|
| source tools/internal_ci/helper_scripts/prepare_build_linux_perf_multilang_rc | ||
|
|
||
| # run 8core client vs 8core server | ||
| tools/run_tests/run_performance_tests.py \ |
There was a problem hiding this comment.
Actually, this is important:
looks like in the kokoro job configuration, the "driver" machine is acquired from a regular pool of workers.
Here's the problem:
- if you have 2 jobs running at the same time (will happen for sure once you have 2 jobs - full_performance_master and full_performance_releases), they will fight over the client and server worker VMs.
- so far that hasn't happened because we were "lucky" and didn't run the "full performance" jobs concurrently with anything.
The remedy for this is to setup a special kokoro pool, that only has 1 VM in it. Then all the jobs that want to access the same set of workers (server, client and client2) will use that pool - which will make the job runs exclusive and they will be sharing the resources correctly.
There was a problem hiding this comment.
Yup, a separate pool should be made for the drivers with a limit of only 1 active workers. I was considering adding a 2nd set of server/clients, but I don't think we not data being generated that frequently. We'll need a 2nd set if we decide to add the full sweep benchmark (Craig has said that it hasn't been used in months and will likely stay unused for the foreseeable future).
|
Added an important comment, please hold merging until the issue is resolved. |
3 participants
Also update current master job to upload to the same BQ table as Jenkins