Skip to content

Upgrade bazel toolchain to 3.7.1 (and upgrade bazel to 3.7.1) #25001

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jtattermusch wants to merge 6 commits into from
Closed

Upgrade bazel toolchain to 3.7.1 (and upgrade bazel to 3.7.1) #25001

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
bf21b2b
update bazel version in other places too
jtattermusch Dec 15, 2020
619042a
add bazel 3.7.1 to update_mirror.sh
jtattermusch Dec 15, 2020
caddd2f
regenerate dockerfiles
jtattermusch Dec 15, 2020
cb81b2d
upgrade bazel toolchain to 3.7.1
jtattermusch Dec 15, 2020
21c35e6
use create_rbe_exec_properties_dict instead of the deprecated create_…
jtattermusch Dec 15, 2020
064bf34
RBE ubsan doesnt require the crosstool_top hack anymore
jtattermusch Sep 22, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 12 additions & 29 deletions WORKSPACE
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,23 +18,15 @@ register_toolchains(
"//third_party/toolchains/bazel_0.26.0_rbe_windows:cc-toolchain-x64_windows",
)

load("@bazel_toolchains//rules/exec_properties:exec_properties.bzl", "create_exec_properties_dict", "custom_exec_properties", "merge_dicts")
load("@bazel_toolchains//rules/exec_properties:exec_properties.bzl", "create_rbe_exec_properties_dict", "custom_exec_properties")

custom_exec_properties(
name = "grpc_custom_exec_properties",
constants = {
"LARGE_MACHINE": merge_dicts(
create_exec_properties_dict(),
# TODO(jtattermusch): specifying 'labels = {"abc": "xyz"}' in create_exec_properties_dict
# is not possible without https://github.com/bazelbuild/bazel-toolchains/pull/748
# and currently the toolchain we're using is too old for that. To be able to select worker
# pools through labels, we use a workaround and populate the corresponding label values
# manually (see create_exec_properties_dict logic for how labels get transformed)
# Remove this workaround once we transition to a new-enough bazel toolchain.
# The next line corresponds to 'labels = {"os": "ubuntu", "machine_size": "large"}'
{
"label:os": "ubuntu",
"label:machine_size": "large",
"LARGE_MACHINE": create_rbe_exec_properties_dict(
Copy link
Copy Markdown
Contributor Author

@jtattermusch jtattermusch Dec 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gnossen depending on what the bazel distribtests say, we might want to reintroduce parts of the workaround that was used previously (conditionally based on the bazel version used?).
It would be good to get some data on what bazel versions we can continue to support.

labels = {
"os": "ubuntu",
"machine_size": "large",
},
),
},
Expand All @@ -45,23 +37,14 @@ load("@bazel_toolchains//rules:rbe_repo.bzl", "rbe_autoconfig")
# Create toolchain configuration for remote execution.
rbe_autoconfig(
name = "rbe_default",
exec_properties = merge_dicts(
create_exec_properties_dict(
docker_add_capabilities = "SYS_PTRACE",
docker_privileged = True,
os_family = "Linux",
),
# TODO(jtattermusch): specifying 'labels = {"abc": "xyz"}' in create_exec_properties_dict
# is not possible without https://github.com/bazelbuild/bazel-toolchains/pull/748
# and currently the toolchain we're using is too old for that. To be able to select worker
# pools through labels, we use a workaround and populate the corresponding label values
# manually (see create_exec_properties_dict logic for how labels get transformed)
# Remove this workaround once we transition to a new-enough bazel toolchain.
# The next line corresponds to 'labels = {"os": "ubuntu", "machine_size": "small"}'
{
"label:os": "ubuntu",
"label:machine_size": "small",
exec_properties = create_rbe_exec_properties_dict(
docker_add_capabilities = "SYS_PTRACE",
docker_privileged = True,
labels = {
"os": "ubuntu",
"machine_size": "small",
},
os_family = "Linux",
),
# use exec_properties instead of deprecated remote_execution_properties
use_legacy_platform_definition = False,
Expand Down
8 changes: 4 additions & 4 deletions bazel/grpc_deps.bzl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -245,11 +245,11 @@ def grpc_deps():
# list of releases is at https://releases.bazel.build/bazel-toolchains.html
http_archive(
name = "bazel_toolchains",
sha256 = "0b36eef8a66f39c8dbae88e522d5bbbef49d5e66e834a982402c79962281be10",
strip_prefix = "bazel-toolchains-1.0.1",
sha256 = "8c9728dc1bb3e8356b344088dfd10038984be74e1c8d6e92dbb05f21cabbb8e4",
strip_prefix = "bazel-toolchains-3.7.1",
urls = [
"https://mirror.bazel.build/github.com/bazelbuild/bazel-toolchains/archive/1.0.1.tar.gz",
"https://github.com/bazelbuild/bazel-toolchains/releases/download/1.0.1/bazel-toolchains-1.0.1.tar.gz",
"https://github.com/bazelbuild/bazel-toolchains/releases/download/3.7.1/bazel-toolchains-3.7.1.tar.gz",
"https://mirror.bazel.build/github.com/bazelbuild/bazel-toolchains/releases/download/3.7.1/bazel-toolchains-3.7.1.tar.gz",
],
)

Expand Down
4 changes: 4 additions & 0 deletions bazel/update_mirror.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,10 @@ upload github.com/bazelbuild/bazel/releases/download/2.2.0/bazel-2.2.0-linux-x86
upload github.com/bazelbuild/bazel/releases/download/2.2.0/bazel-2.2.0-darwin-x86_64
upload github.com/bazelbuild/bazel/releases/download/2.2.0/bazel-2.2.0-windows-x86_64.exe

upload github.com/bazelbuild/bazel/releases/download/3.7.1/bazel-3.7.1-linux-x86_64
upload github.com/bazelbuild/bazel/releases/download/3.7.1/bazel-3.7.1-darwin-x86_64
upload github.com/bazelbuild/bazel/releases/download/3.7.1/bazel-3.7.1-windows-x86_64.exe

# Collect the github archives to mirror from grpc_deps.bzl
grep -o '"https://github.com/[^"]*"' bazel/grpc_deps.bzl | sed 's/^"https:\/\///' | sed 's/"$//' | while read -r line ; do
echo "Updating mirror for ${line}"
Expand Down
2 changes: 1 addition & 1 deletion templates/tools/dockerfile/bazel.include
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# Bazel installation

# Must be in sync with tools/bazel
ENV BAZEL_VERSION 2.2.0
ENV BAZEL_VERSION 3.7.1

# The correct bazel version is already preinstalled, no need to use //tools/bazel wrapper.
ENV DISABLE_BAZEL_WRAPPER 1
Expand Down
21 changes: 6 additions & 15 deletions third_party/toolchains/BUILD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ licenses(["notice"]) # Apache v2

package(default_visibility = ["//visibility:public"])

load("@bazel_toolchains//rules/exec_properties:exec_properties.bzl", "create_exec_properties_dict", "merge_dicts")
load("@bazel_toolchains//rules/exec_properties:exec_properties.bzl", "create_rbe_exec_properties_dict")

alias(
name = "rbe_windows",
Expand All @@ -30,19 +30,10 @@ platform(
"@bazel_tools//platforms:x86_64",
"@bazel_tools//platforms:windows",
],
exec_properties = merge_dicts(
create_exec_properties_dict(
# See rbe_win2019/Dockerfile for image details
container_image = "docker://gcr.io/grpc-testing/rbe_windows2019_withdbg_toolchain@sha256:7b04ee7e29f942adbf4f70edd2ec4ba20a3e7237e1b54f5cae4b239c6ca41105",
os_family = "Windows",
),
# TODO(jtattermusch): specifying 'labels = {"abc": "xyz"}' in create_exec_properties_dict
# is not possible without https://github.com/bazelbuild/bazel-toolchains/pull/748
# and currently the toolchain we're using is too old for that. To be able to select worker
# pools through labels, we use a workaround and populate the corresponding label values
# manually (see create_exec_properties_dict logic for how labels get transformed)
# Remove this workaround once we transition to a new-enough bazel toolchain.
# The next line corresponds to 'labels = {"os": "windows_2019", "machine_size": "small"}'
{"label:os": "windows_2019", "label:machine_size": "small"}
exec_properties = create_rbe_exec_properties_dict(
# See rbe_win2019/Dockerfile for image details
container_image = "docker://gcr.io/grpc-testing/rbe_windows2019_withdbg_toolchain@sha256:7b04ee7e29f942adbf4f70edd2ec4ba20a3e7237e1b54f5cae4b239c6ca41105",
os_family = "Windows",
labels = {"os": "windows_2019", "machine_size": "small"},
),
)
2 changes: 2 additions & 0 deletions tools/bazel
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,8 @@ then
fi
fi

# IMPORTANT: if you update the version here, other parts of infrastructure might needs updating as well
# (e.g. win RBE builds, sanity checks, bazel toolchains etc.)
VERSION=3.7.1
echo "INFO: Running bazel wrapper (see //tools/bazel for details), bazel version $VERSION will be used instead of system-wide bazel installation." >&2

Expand Down
2 changes: 1 addition & 1 deletion tools/dockerfile/test/bazel/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ RUN python3.6 -m ensurepip && \
# Bazel installation

# Must be in sync with tools/bazel
ENV BAZEL_VERSION 2.2.0
ENV BAZEL_VERSION 3.7.1

# The correct bazel version is already preinstalled, no need to use //tools/bazel wrapper.
ENV DISABLE_BAZEL_WRAPPER 1
Expand Down
2 changes: 1 addition & 1 deletion tools/dockerfile/test/sanity/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ ENV CLANG_TIDY=clang-tidy-8
# Bazel installation

# Must be in sync with tools/bazel
ENV BAZEL_VERSION 2.2.0
ENV BAZEL_VERSION 3.7.1

# The correct bazel version is already preinstalled, no need to use //tools/bazel wrapper.
ENV DISABLE_BAZEL_WRAPPER 1
Expand Down
2 changes: 1 addition & 1 deletion tools/internal_ci/windows/bazel_rbe.bat
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@

@rem TODO(jtattermusch): make this generate less output
@rem TODO(jtattermusch): use tools/bazel script to keep the versions in sync
choco install bazel -y --version 2.2.0 --limit-output
choco install bazel -y --version 3.7.1 --limit-output

cd github/grpc
set PATH=C:\tools\msys64\usr\bin;C:\Python27;%PATH%
Expand Down
5 changes: 0 additions & 5 deletions tools/remote_build/rbe_common.bazelrc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,8 +87,3 @@ build:ubsan --copt=-gmlt
# use double the default value for "moderate" and "long" timeout as sanitizer
# tests tend to be slower
build:ubsan --test_timeout=60,600,1800,3600
# override the config-agnostic crosstool_top
# how to update the bazel toolchain for ubsan:
# - check for the latest released version in https://github.com/bazelbuild/bazel-toolchains/tree/master/configs/experimental/ubuntu16_04_clang
# - you might need to update the bazel_toolchains dependency in grpc_deps.bzl
build:ubsan --crosstool_top=@bazel_toolchains//configs/experimental/ubuntu16_04_clang/1.3/bazel_0.29.1/ubsan:toolchain