Skip to content

RBE: rely on labels to select the right worker pool (instead of hardcoding GCE machine types)#24370

Merged
jtattermusch merged 2 commits intogrpc:masterfrom
jtattermusch:e2_rbe_pools
Oct 12, 2020
Merged

RBE: rely on labels to select the right worker pool (instead of hardcoding GCE machine types)#24370
jtattermusch merged 2 commits intogrpc:masterfrom
jtattermusch:e2_rbe_pools

Conversation

@jtattermusch
Copy link
Copy Markdown
Contributor

@jtattermusch jtattermusch commented Oct 8, 2020

Motivation: we want to switch our RBE worker pools from n1 instances to e2 instances. The problem is that currently our RBE configuration relies on hardcoded machine types (e.g. n1-standard-8) to determine on which worker pool should a given test run (there are 2 linux pools, one with 8core "large" machines and one with 2core "small" machines; there is also a windows pool. Tests need to run on the right pools to avoid unexpected failures.

Challenges:

  • currently our bazel toolchain we're using is too old to understand the concept of RBE worker pool labels (and we cannot upgrade our toolchain right now for unrelated reason and even if we could, we could not easily backport that to older release branches).
  • we need a solution that is non-intrusive enough so we can backport it to older release branches if needed (remember that the older branches have the machine types "n1-standard-8" etc hardcoded in them so if we migrate our existing pools to e2, they will suddenly break).

This PR represents an approach that seems to work and that would allow us to switch our existing worker pools to e2 instances seamlessly without breaking the world (and it can be backported to older branches as needed).

@jtattermusch jtattermusch added lang/c++ release notes: no Indicates if PR should not be in release notes labels Oct 8, 2020
@jtattermusch jtattermusch requested a review from gnossen October 8, 2020 15:16
@jtattermusch
Copy link
Copy Markdown
Contributor Author

CC @veblush @nicolasnoble

@jtattermusch
Copy link
Copy Markdown
Contributor Author

@rubensf feel free to review the workaround I used for setting the RBE worker pool labels. I know it's a bit dirty, but it seems to work just fine and considering the challenges I mentioned in the PR description I think it's actually a pretty clever solution to solve the problem we're having (and to expedite the E2 migration).

Copy link
Copy Markdown

@rubensf rubensf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - this should work fine! Thanks for looking into this 👍

Copy link
Copy Markdown
Contributor

@gnossen gnossen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very clever workaround! 👍

@jtattermusch
Copy link
Copy Markdown
Contributor Author

Known failures: #24375

@jtattermusch jtattermusch merged commit 8765c6c into grpc:master Oct 12, 2020
jtattermusch added a commit that referenced this pull request Oct 14, 2020
nicolasnoble added a commit that referenced this pull request Oct 14, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lang/c++ release notes: no Indicates if PR should not be in release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants