-
Notifications
You must be signed in to change notification settings - Fork 4.4k
Fix and verify Java version compatibility of Java tools #26126
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Prebuilt Java tools meant to run on the user-specified `--java_{tool_,}runtime_version` must only use Java 8 language features and APIs. Previously, these tools were compiled with `--java_language_version=8`, but had access to a JDK 21 as the bootclasspath. In fact, `junitrunner` used `Stream#dropWhile`, which isn't available in a JDK 8.
Similarly, Java tools meant to run on the Java runtime provided by a Java (compilation) toolchain must only use Java 11 language features and APIs. This was the case, but they were in fact build with `--java_language_version=8` and a JDK 21, just like the other tools, thus lacking validation for this property.
This PR splits tools into two groups and enforces consistent Java language and bootclasspath runtime for them via `--@rules_java//toolchains:incompatible_language_version_bootclasspath`. Constants in `//src:release_archive.bzl` can be used to set the two minimum versions globally.
| licenses(["notice"]) # Apache 2.0 | ||
|
|
||
| java_binary( | ||
| minimum_java_runtime_java_binary( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If it's sometimes necessary to build this for a higher bytecode version, we could keep it a regular java_binary and wrap it in a transitioning filegroup.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cc @c-mita
|
CI is green after I fixed the bootstrap tests. |
Precompiled Java tools meant to run with the user-specified `--java_{tool_,}runtime_version` must only use Java 8 language features and APIs. Previously, these tools were compiled with `--java_language_version=8`, but had access to a JDK 21 as the bootclasspath. In fact, `junitrunner` used `Stream#dropWhile`, which isn't available in a JDK 8. Similarly, Java tools meant to run on the Java runtime provided by a Java (compilation) toolchain must only use Java 11 language features and APIs. This was the case, but they were in fact build with `--java_language_version=8` and a JDK 21, just like the other tools.
This PR splits tools into two groups and enforces consistent Java language and bootclasspath runtime versions for them via `--@rules_java//toolchains:incompatible_language_version_bootclasspath`. Constants in `//src:release_archive.bzl` can be used to set the two minimum versions globally.
Related to bazelbuild#26100
Closes bazelbuild#26126.
PiperOrigin-RevId: 763699406
Change-Id: Id992546ca6302da11c7280a29417cba893411b51
---
This cherry-pick only contains a few of the original changes. The next
commit will introduce bespoke transitions instead of using
`with_cfg.bzl`.
Upgrades Zlib to 1.3.1 and removes the `-no_adhoc_codesign` and `-no_uuid` linker flags for building the Clang wrapper. Fixes bazelbuild#27026. Incorporates changes from the following commits and pull requests: - Update third_party/zlib/BUILD{,.tools} for 1.3.1 bazelbuild/bazel@b29649f bazelbuild/bazel@2067a8b - Fix upb build with Clang 16 bazelbuild/bazel@4a32d7b bazelbuild#23700 - Remove -Wl{-no_adhoc_codesign,-no_uuid} Required to build under macOS 26.0 to avoid the \`missing LC_UUID\` error from \`wrapped_clang\` and \`xcode-locator\`. bazelbuild#27014 Upgrades `third_party/zlib` using the methodology from bazelbuild#27026: ```txt cd third_party curl -L https://github.com/madler/zlib/releases/download/v1.3.1/zlib-1.3.1.tar.gz | gzip -dc | tar xf - rsync -r zlib-1.3.1/* zlib/ rm -rf zlib-1.3.1 cd .. ``` Several `third_party/zlib` files show up as complete replacements because the previous files used CRLF line endings, and the 1.3.1 files only use LF. To list the substantial changes outside of the Zlib 1.3.1 distribution upgrade: ```txt $ git diff --stat HEAD^ -- ':!:third_party/zlib/[^B]*' distdir_deps.bzl | 4 ++++ third_party/upb/01_remove_werror.patch | 19 +++++++++++++++++++ third_party/upb/BUILD | 4 +++- third_party/zlib/BUILD | 45 ++++++++++++++++++++++++++++++++++++++------- third_party/zlib/BUILD.tools | 15 +++++++++------ tools/cpp/osx_cc_configure.bzl | 2 -- tools/osx/BUILD | 3 +-- 7 files changed, 74 insertions(+), 18 deletions(-) ``` --- The CI infrastructure changed significantly since the 6.5.0 release such that many tests failed in ways unrelated to the original change described above. This commit contains many other changes to get the builds passing again, comprised of cherry-picks and new fixes: - Bumps .bazelversion to 6.3.2 (from the previous 6.2.1) - Sets --test_output=errors in CI to expose more failure information. - Fixes Python 3 related failures: - Updates collections to collections.abc. - Replaces `assertRegexpMatches` with `assertRegex` and `assertEquals` with `assertEqual`. - Updates `pywrapper_template.txt` to use Bash instead of Bourne Shell and replaces `echo | grep` with `[[ ]]` and `=~`. - Disables //src/test/shell/bazel:python_version_test since several platforms no longer ship with a `python2` binary. - Migrates `imp` to `importlib` in `path_test.py` (commit 5b54c74, bazelbuild#24732). - Uses `python3` in `remote_helpers.sh` (commit bb7d83c). - Fixes bazel `py_test` `testSmoke` for Python 3.11+ (commit 9b027c8). - Fixes `bazel_overrides_test.py` on Windows (commit be2186f). - Updates `python` to `python3` in several tests. - Disables //src/test/shell/bazel/android/... and other Android tests due to incompatible CI infrastructure changes, especially after bazelbuild/continuous-integration#2408. - Stops removing `add_opens` and `add_exports` configurations per bazelbuild#19850 (commit 6a7e9ba). - Disables the GoogleTestSecurityManager by default and explicitly enable it for tests that rely on it. (commit 366c4ba). - Adds a shutdown hook for tests that detects `System.exit` (commit 5fc2b01). - Fixes serialization tests for JDK 21 (commit 052407b). - Fixes ProxyHelperTest on JDK@HEAD (commit 4c85356). - Fixes the circuitbreaker test by increasing a sleep in `testRecordFailure_circuitTrips` by 5ms as currently done on `master`. - Removes `python` binary requirement for Bazel tests (commit e9316f0). - Runs idle server GC immediately following a `--nokeep_state_after_build_command` instead of waiting 10 seconds. (commit f6344ff). - Skips `test_cc_toolchain_tsan_feature` on Linux due to new Linux CI images that surface google/sanitizers#1716. - Removes redundant onCompleted call (commit 0f6dd59, bazelbuild#17792). - Disables the `SecurityManager` in `TestRunner`. - Applies `-proc:full` in `SourceCompilationUnit.compile` when supported by the JDK, and applies `-Xlint:-options` otherwise. - Makes `bazel_sandboxing_test` case case insensitive to fix `test_sandbox_block_network_access` on Rocky Linux 8. - Adds or extends timeout values to a number of tests, especially for macOS builds. - Enables `--incompatible_sandbox_hermetic_tmp` by default (commit e2c0276, bazelbuild#19943, bazelbuild#20136). - Sets `--{,tool_}java_language_version=11` and `--{,tool_}java_runtime_version=remotejdk_11` in `.bazelrc` and several tests. - Sets `sandbox_add_mount_pair=/tmp` to fix `bazel_sandboxing_test`. - Replaces the `ubuntu2004` CI build with `ubuntu2004_java11`. - Sets `sandbox_default_allow_network=true` on macos CI to match `master`. See commit 355b000, bazelbuild#23726. - Explicitly sets up the `local_jdk` `javabase` in the tests where it's actually used (commit 77c2791). - Loads the Bash runfiles library in `integration_test_setup.sh` (commit b20085a). - Removed `remote_helper.sh` from tests that don't need it to fix unbound variable errors after cherry-picking commit 77c2791 and commit b20085a. - Adds `@local_jdk//:jdk` to test targets depending upon `remote_helpers.sh` (which invokes `setup_localjdk_javabase`) after cherry-picking commit b20085a. - Ensures `local_jdk/bin/javac` compiles to Java 11 in `external_integration_test.sh`. - Fixes `bazel_proto_library_test` after cherry-picking commit b20085a: - Fixes an incorrect `rlocation` path. - Adds a `new_local_repository` for the builtin `zlib`. - Replaces the numeric `-eq` Bash comparison operator with the `-z` string comparison operator. - Uses Googletest matchers in `option_processor_test` to help macOS CI failures reveal more useful information. - Fixes `//src/test/cpp:option_processor_test` by ignoring `.bazelrc` files (commit 89c69c2; see also: bazelbuild/continuous-integration#2201). - Sets `--{,tool_}java_runtime_version=local_jdk` in `bazel_bootstrap_distfile_test.sh`. This fixed previous macOS and Windows errors when set to Java 11, which no longer ships in recent CI build images. - Fixes and verifies Java version compatibility of Java tools (parts of commit b841e14, bazelbuild#26126). - Ensures Java tools use specific Java versions based on transitions based on `tools/build_defs.bzl` from commit b532a46. - Sets `--noannounce_rc` in `bazel query` expressions from `workspace_test` cases to fix Linux CI builds. - Moves `//src:release_archive.bzl` constants to `//tools:build_defs.bzl` to fix tests that use `copy_tools_directory` from `testenv.sh.tmpl`. - Fixes `repository/downloader:DownloaderTestSuite` for Windows + JDK21 (commit 59cf396, bazelbuild#21593). - Fixes `HttpConnectorTest` missing symbol error after cherry-picking commit 59cf396 by changing a Truth expression to an equivalent assertion. - Supports using newer JDK versions when compiling Bazel from source without an existing Bazel binary (commit af50ad3, bazelbuild#18961, bazelbuild#19125). - Removes `testBuildArmCppBinaryWithMsvcCL` on Windows (commit 2158dac, bazelbuild#26795, bazelbuild#26797). - Prefers IPv6 network in integration tests on macOS (commit 2c51a0c). - Fixes `bazel_command_log_test.sh`: - Updates a `sed` expression to avoid using the `+` operator (commit 130b444). - Makes sure the batch log also clears log messages from the newer JDKs (commit e8966bb). - Ignores warning about JAVA_TOOL_OPTIONS in test (commit 2ceb132, bazelbuild#20274). - Disables bzlmod (thus avoiding any requests to BCR) and filters out irrelevant log entries commit a737169). - Ignores the `Picked up JAVA_TOOL_OPTIONS` message that now appears on macOS runs after cherry-picking commit 2c51a0c. --- The most noteworthy fix in this change sets the right JDK version for `*_for_testing` repos. This fixes several tests broken by an incompatible JVM flag passed to the `javac` compiler from `remotejdk_11`: - Removes the `build_file = "@local_jdk//:BUILD.bazel` parameter from `remotejdk*_for_testing` repositories in `WORKSPACE`. - Updates `dist_http_archive` to parse the correct JDK version from the repo name and then generate a `BUILD` file from `JDK_BUILD_TEMPLATE` instead. The flag came from this expression in `BazelJavaSemantics.getJvmFlags`: ```java if (JavaRuntimeInfo.from(ruleContext).version() >= 17) { jvmFlags.add("-Djava.security.manager=allow"); } ``` Setting `.bazelci/presubmit.yml` Linux jobs to use Java 11 images enabled tests to pass on those platforms without the fix. This is because the `java_runtime` target emitted into `@local_jdk//:BUILD.bazel` (`@local_jdk//:jdk`) sets `version = "11"` on these platforms. That meant the `JavaRuntimeInfo` expression above would return `11` on Linux, but a higher value on macOS and Windows. This led to emitting the `-Djava.security.manager` flag when using the `remotejdk_11` compiler on those platforms, breaking the build. Specifying different macOS and Windows build images with Java 11 isn't possible, necessitating the `dist_http_archive` change to fix tests broken on those platforms. History and further details: The following series of commits introduced the `remotejdk*_for_testing` repos and a mechanism to have the "inner" Bazel reuse the "outer" Bazel's copies: - Add external repos required by tests and a "fetch all" filegroup (commit 17506af) - Allow overriding all external repositories used by our integration tests (commit 99c0e6d) - Move JDK repo definitions to distdir_deps.bzl (commit b741116) In `.bazelci/presubmit.yml`, every job sets `--test_env=TEST_REPOSITORY_HOME=$OUTPUT_BASE/external`. This causes `testenv.sh.tmpl` to override certain repositories with their `_for_testing` variants: ```sh if [[ -n ${TEST_REPOSITORY_HOME:-} ]]; then echo "testenv.sh: Using shared repositories from $TEST_REPOSITORY_HOME." repos=( # ... "remotejdk17_macos_for_testing" # ... ) for repo in "${repos[@]}"; do reponame="${repo%"_for_testing"}" echo "common --override_repository=$reponame=$TEST_REPOSITORY_HOME/$repo" >> $TEST_TMPDIR/bazelrc done fi ``` The `WORKSPACE` file contained entries like the following, created before `@local_jdk//:BUILD.bazel` contained an explicit JDK version for its `//:jdk` target: ```py dist_http_archive( name = "remotejdk11_macos_for_testing", build_file = "@local_jdk//:BUILD.bazel", patch_cmds = EXPORT_WORKSPACE_IN_BUILD_BAZEL_FILE, patch_cmds_win = EXPORT_WORKSPACE_IN_BUILD_BAZEL_FILE_WIN, ) ``` Later, these commits added the JDK version to `JdkRuntimeInfo`: - Add version to JavaRuntimeInfo (commit 7556e11) - [6.2.0] Add version to JavaRuntimeInfo (bazelbuild#17913) (commit 2d04c91) However, they didn't update the `dist_http_archive` entries to adjust their `build_file` parameters accordingly. As a result, the `remotejdk` repos used by the "inner" Bazel during tests all specified the same JDK version as the system JDK. Bazel resolved the correct toolchain for `--java_runtime_version`, which would contain the correct Java binaries, but contained the system JDK `version` parameter. This caused `BazelJavaSemantics` to emit JVM flags for the _wrong_ JDK version. This, in turn, broke test binaries when they ran under the _correct_ `java` binary from the JDK that differed from the system JDK version.
Upgrades Zlib to 1.3.1 and removes the `-no_adhoc_codesign` and `-no_uuid` linker flags for building the Clang wrapper. Fixes bazelbuild#27026. Incorporates changes from the following commits and pull requests: - Update third_party/zlib/BUILD{,.tools} for 1.3.1 bazelbuild/bazel@b29649f bazelbuild/bazel@2067a8b - Fix upb build with Clang 16 bazelbuild/bazel@4a32d7b bazelbuild#23700 - Remove -Wl{-no_adhoc_codesign,-no_uuid} Required to build under macOS 26.0 to avoid the \`missing LC_UUID\` error from \`wrapped_clang\` and \`xcode-locator\`. bazelbuild#27014 Upgrades `third_party/zlib` using the methodology from bazelbuild#27026: ```txt cd third_party curl -L https://github.com/madler/zlib/releases/download/v1.3.1/zlib-1.3.1.tar.gz | gzip -dc | tar xf - rsync -r zlib-1.3.1/* zlib/ rm -rf zlib-1.3.1 cd .. ``` Several `third_party/zlib` files show up as complete replacements because the previous files used CRLF line endings, and the 1.3.1 files only use LF. To list the substantial changes outside of the Zlib 1.3.1 distribution upgrade: ```txt $ git diff --stat HEAD^ -- distdir_deps.bzl third_party/{upb,zlib/BUILD*} tools/{cpp,osx} distdir_deps.bzl | 4 ++++ third_party/upb/01_remove_werror.patch | 19 +++++++++++++++++++ third_party/upb/BUILD | 4 +++- third_party/zlib/BUILD | 45 ++++++++++++++++++++++++++++++++++++++------- third_party/zlib/BUILD.tools | 15 +++++++++------ tools/cpp/osx_cc_configure.bzl | 2 -- tools/osx/BUILD | 3 +-- 7 files changed, 74 insertions(+), 18 deletions(-) ``` --- The CI infrastructure changed significantly since the 6.5.0 release such that many tests failed in ways unrelated to the original change described above. This commit contains many other changes to get the builds passing again, comprised of cherry-picks and new fixes: - Bumps .bazelversion to 6.3.2 (from the previous 6.2.1) - Sets --test_output=errors in CI to expose more failure information. - Fixes Python 3 related failures: - Updates collections to collections.abc. - Replaces `assertRegexpMatches` with `assertRegex` and `assertEquals` with `assertEqual`. - Updates `pywrapper_template.txt` to use Bash instead of Bourne Shell and replaces `echo | grep` with `[[ ]]` and `=~`. - Disables //src/test/shell/bazel:python_version_test since several platforms no longer ship with a `python2` binary. - Migrates `imp` to `importlib` in `path_test.py` (commit 5b54c74, bazelbuild#24732). - Uses `python3` in `remote_helpers.sh` (commit bb7d83c). - Fixes bazel `py_test` `testSmoke` for Python 3.11+ (commit 9b027c8). - Fixes `bazel_overrides_test.py` on Windows (commit be2186f). - Updates `python` to `python3` in several tests. - Disables //src/test/shell/bazel/android/... and other Android tests due to incompatible CI infrastructure changes, especially after bazelbuild/continuous-integration#2408. - Stops removing `add_opens` and `add_exports` configurations per bazelbuild#19850 (commit 6a7e9ba). - Disables the GoogleTestSecurityManager by default and explicitly enable it for tests that rely on it. (commit 366c4ba). - Adds a shutdown hook for tests that detects `System.exit` (commit 5fc2b01). - Fixes serialization tests for JDK 21 (commit 052407b). - Fixes ProxyHelperTest on JDK@HEAD (commit 4c85356). - Fixes the circuitbreaker test by increasing a sleep in `testRecordFailure_circuitTrips` by 5ms as currently done on `master`. - Removes `python` binary requirement for Bazel tests (commit e9316f0). - Runs idle server GC immediately following a `--nokeep_state_after_build_command` instead of waiting 10 seconds. (commit f6344ff). - Skips `test_cc_toolchain_tsan_feature` on Linux due to new Linux CI images that surface google/sanitizers#1716. - Removes redundant onCompleted call (commit 0f6dd59, bazelbuild#17792). - Disables the `SecurityManager` in `TestRunner`. - Applies `-proc:full` in `SourceCompilationUnit.compile` when supported by the JDK, and applies `-Xlint:-options` otherwise. - Makes `bazel_sandboxing_test` case case insensitive to fix `test_sandbox_block_network_access` on Rocky Linux 8. - Adds or extends timeout values to a number of tests, especially for macOS builds. - Enables `--incompatible_sandbox_hermetic_tmp` by default (commit e2c0276, bazelbuild#19943, bazelbuild#20136). - Sets `--{,tool_}java_language_version=11` and `--{,tool_}java_runtime_version=remotejdk_11` in `.bazelrc` and several tests. - Sets `sandbox_add_mount_pair=/tmp` to fix `bazel_sandboxing_test`. - Replaces the `ubuntu2004` CI build with `ubuntu2004_java11`. - Sets `sandbox_default_allow_network=true` on macos CI to match `master`. See commit 355b000, bazelbuild#23726. - Explicitly sets up the `local_jdk` `javabase` in the tests where it's actually used (commit 77c2791). - Loads the Bash runfiles library in `integration_test_setup.sh` (commit b20085a). - Removed `remote_helper.sh` from tests that don't need it to fix unbound variable errors after cherry-picking commit 77c2791 and commit b20085a. - Adds `@local_jdk//:jdk` to test targets depending upon `remote_helpers.sh` (which invokes `setup_localjdk_javabase`) after cherry-picking commit b20085a. - Ensures `local_jdk/bin/javac` compiles to Java 11 in `external_integration_test.sh`. - Fixes `bazel_proto_library_test` after cherry-picking commit b20085a: - Fixes an incorrect `rlocation` path. - Adds a `new_local_repository` for the builtin `zlib`. - Replaces the numeric `-eq` Bash comparison operator with the `-z` string comparison operator. - Uses Googletest matchers in `option_processor_test` to help macOS CI failures reveal more useful information. - Fixes `//src/test/cpp:option_processor_test` by ignoring `.bazelrc` files (commit 89c69c2; see also: bazelbuild/continuous-integration#2201). - Sets `--{,tool_}java_runtime_version=local_jdk` in `bazel_bootstrap_distfile_test.sh`. This fixed previous macOS and Windows errors when set to Java 11, which no longer ships in recent CI build images. - Fixes and verifies Java version compatibility of Java tools (parts of commit b841e14, bazelbuild#26126). - Ensures Java tools use specific Java versions based on transitions based on `tools/build_defs.bzl` from commit b532a46. - Sets `--noannounce_rc` in `bazel query` expressions from `workspace_test` cases to fix Linux CI builds. - Moves `//src:release_archive.bzl` constants to `//tools:build_defs.bzl` to fix tests that use `copy_tools_directory` from `testenv.sh.tmpl`. - Fixes `repository/downloader:DownloaderTestSuite` for Windows + JDK21 (commit 59cf396, bazelbuild#21593). - Fixes `HttpConnectorTest` missing symbol error after cherry-picking commit 59cf396 by changing a Truth expression to an equivalent assertion. - Supports using newer JDK versions when compiling Bazel from source without an existing Bazel binary (commit af50ad3, bazelbuild#18961, bazelbuild#19125). - Removes `testBuildArmCppBinaryWithMsvcCL` on Windows (commit 2158dac, bazelbuild#26795, bazelbuild#26797). - Prefers IPv6 network in integration tests on macOS (commit 2c51a0c). - Fixes `bazel_command_log_test.sh`: - Updates a `sed` expression to avoid using the `+` operator (commit 130b444). - Makes sure the batch log also clears log messages from the newer JDKs (commit e8966bb). - Ignores warning about JAVA_TOOL_OPTIONS in test (commit 2ceb132, bazelbuild#20274). - Disables bzlmod (thus avoiding any requests to BCR) and filters out irrelevant log entries commit a737169). - Ignores the `Picked up JAVA_TOOL_OPTIONS` message that now appears on macOS runs after cherry-picking commit 2c51a0c. --- The most noteworthy fix in this change sets the right JDK version for `*_for_testing` repos. This fixes several tests broken by an incompatible JVM flag passed to the `javac` compiler from `remotejdk_11`: - Removes the `build_file = "@local_jdk//:BUILD.bazel"` parameter from `remotejdk*_for_testing` repositories in `WORKSPACE`. - Updates `dist_http_archive` to parse the correct JDK version from the repo name and then generate a `BUILD` file from `JDK_BUILD_TEMPLATE` instead. The flag came from this expression in `BazelJavaSemantics.getJvmFlags`: ```java if (JavaRuntimeInfo.from(ruleContext).version() >= 17) { jvmFlags.add("-Djava.security.manager=allow"); } ``` Setting `.bazelci/presubmit.yml` Linux jobs to use Java 11 images enabled tests to pass on those platforms without the fix. This is because the `java_runtime` target emitted into `@local_jdk//:BUILD.bazel` (`@local_jdk//:jdk`) sets `version = "11"` on these platforms. That meant the `JavaRuntimeInfo` expression above would return `11` on Linux, but a higher value on macOS and Windows. This led to emitting the `-Djava.security.manager` flag when using the `remotejdk_11` compiler on those platforms, breaking the build. Specifying different macOS and Windows build images with Java 11 isn't possible, necessitating the `dist_http_archive` change to fix tests broken on those platforms. History and further details: The following series of commits introduced the `remotejdk*_for_testing` repos and a mechanism to have the "inner" Bazel reuse the "outer" Bazel's copies: - Add external repos required by tests and a "fetch all" filegroup (commit 17506af) - Allow overriding all external repositories used by our integration tests (commit 99c0e6d) - Move JDK repo definitions to distdir_deps.bzl (commit b741116) In `.bazelci/presubmit.yml`, every job sets `--test_env=TEST_REPOSITORY_HOME=$OUTPUT_BASE/external`. This causes `testenv.sh.tmpl` to override certain repositories with their `_for_testing` variants: ```sh if [[ -n ${TEST_REPOSITORY_HOME:-} ]]; then echo "testenv.sh: Using shared repositories from $TEST_REPOSITORY_HOME." repos=( # ... "remotejdk17_macos_for_testing" # ... ) for repo in "${repos[@]}"; do reponame="${repo%"_for_testing"}" echo "common --override_repository=$reponame=$TEST_REPOSITORY_HOME/$repo" >> $TEST_TMPDIR/bazelrc done fi ``` The `WORKSPACE` file contained entries like the following, created before `@local_jdk//:BUILD.bazel` contained an explicit JDK version for its `//:jdk` target: ```py dist_http_archive( name = "remotejdk11_macos_for_testing", build_file = "@local_jdk//:BUILD.bazel", patch_cmds = EXPORT_WORKSPACE_IN_BUILD_BAZEL_FILE, patch_cmds_win = EXPORT_WORKSPACE_IN_BUILD_BAZEL_FILE_WIN, ) ``` Later, these commits added the JDK version to `JdkRuntimeInfo`: - Add version to JavaRuntimeInfo (commit 7556e11) - [6.2.0] Add version to JavaRuntimeInfo (bazelbuild#17913) (commit 2d04c91) However, they didn't update the `dist_http_archive` entries to adjust their `build_file` parameters accordingly. As a result, the `remotejdk` repos used by the "inner" Bazel during tests all specified the same JDK version as the system JDK. Bazel resolved the correct toolchain for `--java_runtime_version`, which would contain the correct Java binaries, but contained the system JDK `version` parameter. This caused `BazelJavaSemantics` to emit JVM flags for the _wrong_ JDK version. This, in turn, broke test binaries when they ran under the _correct_ `java` binary from the JDK that differed from the system JDK version.
Upgrades Zlib to 1.3.1 and removes the `-no_adhoc_codesign` and `-no_uuid` linker flags for building the Clang wrapper. Fixes #27026. Incorporates changes from the following commits and pull requests: - Update third_party/zlib/BUILD{,.tools} for 1.3.1 b29649f 2067a8b - Fix upb build with Clang 16 4a32d7b #23700 - Remove -Wl{-no_adhoc_codesign,-no_uuid} Required to build under macOS 26.0 to avoid the \`missing LC_UUID\` error from \`wrapped_clang\` and \`xcode-locator\`. #27014 Upgrades `third_party/zlib` using the methodology from #27026: ```txt cd third_party curl -L https://github.com/madler/zlib/releases/download/v1.3.1/zlib-1.3.1.tar.gz | gzip -dc | tar xf - rsync -r zlib-1.3.1/* zlib/ rm -rf zlib-1.3.1 cd .. ``` Several `third_party/zlib` files show up as complete replacements because the previous files used CRLF line endings, and the 1.3.1 files only use LF. To list the substantial changes outside of the Zlib 1.3.1 distribution upgrade: ```txt $ git diff --stat HEAD^ -- distdir_deps.bzl third_party/{upb,zlib/BUILD*} tools/{cpp,osx} distdir_deps.bzl | 4 ++++ third_party/upb/01_remove_werror.patch | 19 +++++++++++++++++++ third_party/upb/BUILD | 4 +++- third_party/zlib/BUILD | 45 ++++++++++++++++++++++++++++++++++++++------- third_party/zlib/BUILD.tools | 15 +++++++++------ tools/cpp/osx_cc_configure.bzl | 2 -- tools/osx/BUILD | 3 +-- 7 files changed, 74 insertions(+), 18 deletions(-) ``` --- The CI infrastructure changed significantly since the 6.5.0 release such that many tests failed in ways unrelated to the original change described above. This commit contains many other changes to get the builds passing again, comprised of cherry-picks and new fixes: - Bumps .bazelversion to 6.3.2 (from the previous 6.2.1) - Sets --test_output=errors in CI to expose more failure information. - Fixes Python 3 related failures: - Updates collections to collections.abc. - Replaces `assertRegexpMatches` with `assertRegex` and `assertEquals` with `assertEqual`. - Updates `pywrapper_template.txt` to use Bash instead of Bourne Shell and replaces `echo | grep` with `[[ ]]` and `=~`. - Disables //src/test/shell/bazel:python_version_test since several platforms no longer ship with a `python2` binary. - Migrates `imp` to `importlib` in `path_test.py` (commit 5b54c74, #24732). - Uses `python3` in `remote_helpers.sh` (commit bb7d83c). - Fixes bazel `py_test` `testSmoke` for Python 3.11+ (commit 9b027c8). - Fixes `bazel_overrides_test.py` on Windows (commit be2186f). - Updates `python` to `python3` in several tests. - Disables //src/test/shell/bazel/android/... and other Android tests due to incompatible CI infrastructure changes, especially after bazelbuild/continuous-integration#2408. - Stops removing `add_opens` and `add_exports` configurations per #19850 (commit 6a7e9ba). - Disables the GoogleTestSecurityManager by default and explicitly enable it for tests that rely on it. (commit 366c4ba). - Adds a shutdown hook for tests that detects `System.exit` (commit 5fc2b01). - Fixes serialization tests for JDK 21 (commit 052407b). - Fixes ProxyHelperTest on JDK@HEAD (commit 4c85356). - Fixes the circuitbreaker test by increasing a sleep in `testRecordFailure_circuitTrips` by 5ms as currently done on `master`. - Removes `python` binary requirement for Bazel tests (commit e9316f0). - Runs idle server GC immediately following a `--nokeep_state_after_build_command` instead of waiting 10 seconds. (commit f6344ff). - Skips `test_cc_toolchain_tsan_feature` on Linux due to new Linux CI images that surface google/sanitizers#1716. - Removes redundant onCompleted call (commit 0f6dd59, #17792). - Disables the `SecurityManager` in `TestRunner`. - Applies `-proc:full` in `SourceCompilationUnit.compile` when supported by the JDK, and applies `-Xlint:-options` otherwise. - Makes `bazel_sandboxing_test` case case insensitive to fix `test_sandbox_block_network_access` on Rocky Linux 8. - Adds or extends timeout values to a number of tests, especially for macOS builds. - Enables `--incompatible_sandbox_hermetic_tmp` by default (commit e2c0276, #19943, #20136). - Sets `--{,tool_}java_language_version=11` and `--{,tool_}java_runtime_version=remotejdk_11` in `.bazelrc` and several tests. - Sets `sandbox_add_mount_pair=/tmp` to fix `bazel_sandboxing_test`. - Replaces the `ubuntu2004` CI build with `ubuntu2004_java11`. - Sets `sandbox_default_allow_network=true` on macos CI to match `master`. See commit 355b000, #23726. - Explicitly sets up the `local_jdk` `javabase` in the tests where it's actually used (commit 77c2791). - Loads the Bash runfiles library in `integration_test_setup.sh` (commit b20085a). - Removed `remote_helper.sh` from tests that don't need it to fix unbound variable errors after cherry-picking commit 77c2791 and commit b20085a. - Adds `@local_jdk//:jdk` to test targets depending upon `remote_helpers.sh` (which invokes `setup_localjdk_javabase`) after cherry-picking commit b20085a. - Ensures `local_jdk/bin/javac` compiles to Java 11 in `external_integration_test.sh`. - Fixes `bazel_proto_library_test` after cherry-picking commit b20085a: - Fixes an incorrect `rlocation` path. - Adds a `new_local_repository` for the builtin `zlib`. - Replaces the numeric `-eq` Bash comparison operator with the `-z` string comparison operator. - Uses Googletest matchers in `option_processor_test` to help macOS CI failures reveal more useful information. - Fixes `//src/test/cpp:option_processor_test` by ignoring `.bazelrc` files (commit 89c69c2; see also: bazelbuild/continuous-integration#2201). - Sets `--{,tool_}java_runtime_version=local_jdk` in `bazel_bootstrap_distfile_test.sh`. This fixed previous macOS and Windows errors when set to Java 11, which no longer ships in recent CI build images. - Fixes and verifies Java version compatibility of Java tools (parts of commit b841e14, #26126). - Ensures Java tools use specific Java versions based on transitions based on `tools/build_defs.bzl` from commit b532a46. - Sets `--noannounce_rc` in `bazel query` expressions from `workspace_test` cases to fix Linux CI builds. - Moves `//src:release_archive.bzl` constants to `//tools:build_defs.bzl` to fix tests that use `copy_tools_directory` from `testenv.sh.tmpl`. - Fixes `repository/downloader:DownloaderTestSuite` for Windows + JDK21 (commit 59cf396, #21593). - Fixes `HttpConnectorTest` missing symbol error after cherry-picking commit 59cf396 by changing a Truth expression to an equivalent assertion. - Supports using newer JDK versions when compiling Bazel from source without an existing Bazel binary (commit af50ad3, #18961, #19125). - Removes `testBuildArmCppBinaryWithMsvcCL` on Windows (commit 2158dac, #26795, #26797). - Prefers IPv6 network in integration tests on macOS (commit 2c51a0c). - Fixes `bazel_command_log_test.sh`: - Updates a `sed` expression to avoid using the `+` operator (commit 130b444). - Makes sure the batch log also clears log messages from the newer JDKs (commit e8966bb). - Ignores warning about JAVA_TOOL_OPTIONS in test (commit 2ceb132, #20274). - Disables bzlmod (thus avoiding any requests to BCR) and filters out irrelevant log entries commit a737169). - Ignores the `Picked up JAVA_TOOL_OPTIONS` message that now appears on macOS runs after cherry-picking commit 2c51a0c. --- The most noteworthy fix in this change sets the right JDK version for `*_for_testing` repos. This fixes several tests broken by an incompatible JVM flag passed to the `javac` compiler from `remotejdk_11`: - Removes the `build_file = "@local_jdk//:BUILD.bazel"` parameter from `remotejdk*_for_testing` repositories in `WORKSPACE`. - Updates `dist_http_archive` to parse the correct JDK version from the repo name and then generate a `BUILD` file from `JDK_BUILD_TEMPLATE` instead. The flag came from this expression in `BazelJavaSemantics.getJvmFlags`: ```java if (JavaRuntimeInfo.from(ruleContext).version() >= 17) { jvmFlags.add("-Djava.security.manager=allow"); } ``` Setting `.bazelci/presubmit.yml` Linux jobs to use Java 11 images enabled tests to pass on those platforms without the fix. This is because the `java_runtime` target emitted into `@local_jdk//:BUILD.bazel` (`@local_jdk//:jdk`) sets `version = "11"` on these platforms. That meant the `JavaRuntimeInfo` expression above would return `11` on Linux, but a higher value on macOS and Windows. This led to emitting the `-Djava.security.manager` flag when using the `remotejdk_11` compiler on those platforms, breaking the build. Specifying different macOS and Windows build images with Java 11 isn't possible, necessitating the `dist_http_archive` change to fix tests broken on those platforms. History and further details: The following series of commits introduced the `remotejdk*_for_testing` repos and a mechanism to have the "inner" Bazel reuse the "outer" Bazel's copies: - Add external repos required by tests and a "fetch all" filegroup (commit 17506af) - Allow overriding all external repositories used by our integration tests (commit 99c0e6d) - Move JDK repo definitions to distdir_deps.bzl (commit b741116) In `.bazelci/presubmit.yml`, every job sets `--test_env=TEST_REPOSITORY_HOME=$OUTPUT_BASE/external`. This causes `testenv.sh.tmpl` to override certain repositories with their `_for_testing` variants: ```sh if [[ -n ${TEST_REPOSITORY_HOME:-} ]]; then echo "testenv.sh: Using shared repositories from $TEST_REPOSITORY_HOME." repos=( # ... "remotejdk17_macos_for_testing" # ... ) for repo in "${repos[@]}"; do reponame="${repo%"_for_testing"}" echo "common --override_repository=$reponame=$TEST_REPOSITORY_HOME/$repo" >> $TEST_TMPDIR/bazelrc done fi ``` The `WORKSPACE` file contained entries like the following, created before `@local_jdk//:BUILD.bazel` contained an explicit JDK version for its `//:jdk` target: ```py dist_http_archive( name = "remotejdk11_macos_for_testing", build_file = "@local_jdk//:BUILD.bazel", patch_cmds = EXPORT_WORKSPACE_IN_BUILD_BAZEL_FILE, patch_cmds_win = EXPORT_WORKSPACE_IN_BUILD_BAZEL_FILE_WIN, ) ``` Later, these commits added the JDK version to `JdkRuntimeInfo`: - Add version to JavaRuntimeInfo (commit 7556e11) - [6.2.0] Add version to JavaRuntimeInfo (#17913) (commit 2d04c91) However, they didn't update the `dist_http_archive` entries to adjust their `build_file` parameters accordingly. As a result, the `remotejdk` repos used by the "inner" Bazel during tests all specified the same JDK version as the system JDK. Bazel resolved the correct toolchain for `--java_runtime_version`, which would contain the correct Java binaries, but contained the system JDK `version` parameter. This caused `BazelJavaSemantics` to emit JVM flags for the _wrong_ JDK version. This, in turn, broke test binaries when they ran under the _correct_ `java` binary from the JDK that differed from the system JDK version.
Precompiled Java tools meant to run with the user-specified
--java_{tool_,}runtime_versionmust only use Java 8 language features and APIs. Previously, these tools were compiled with--java_language_version=8, but had access to a JDK 21 as the bootclasspath. In fact,junitrunnerusedStream#dropWhile, which isn't available in a JDK 8. Similarly, Java tools meant to run on the Java runtime provided by a Java (compilation) toolchain must only use Java 11 language features and APIs. This was the case, but they were in fact build with--java_language_version=8and a JDK 21, just like the other tools.This PR splits tools into two groups and enforces consistent Java language and bootclasspath runtime versions for them via
--@rules_java//toolchains:incompatible_language_version_bootclasspath. Constants in//src:release_archive.bzlcan be used to set the two minimum versions globally.Related to #26100