Skip to content

Conversation

@mbland
Copy link

@mbland mbland commented Oct 29, 2025

Upgrades Zlib to 1.3.1 and removes the -no_adhoc_codesign and -no_uuid linker flags for building the Clang wrapper. Fixes #27026.

Incorporates changes from the following commits and pull requests:

Upgrades third_party/zlib using the methodology from #27026:

cd third_party
curl -L https://github.com/madler/zlib/releases/download/v1.3.1/zlib-1.3.1.tar.gz |
  gzip -dc | tar xf -
rsync -r zlib-1.3.1/* zlib/
rm -rf zlib-1.3.1
cd ..

Several third_party/zlib files show up as complete replacements because the previous files used CRLF line endings, and the 1.3.1 files only use LF.

To list the substantial changes outside of the Zlib 1.3.1 distribution upgrade:

$ git diff --stat HEAD^ -- distdir_deps.bzl third_party/{upb,zlib/BUILD*} tools/{cpp,osx}
 distdir_deps.bzl                       |  4 ++++
 third_party/upb/01_remove_werror.patch | 19 +++++++++++++++++++
 third_party/upb/BUILD                  |  4 +++-
 third_party/zlib/BUILD                 | 45 ++++++++++++++++++++++++++++++++++++++-------
 third_party/zlib/BUILD.tools           | 15 +++++++++------
 tools/cpp/osx_cc_configure.bzl         |  2 --
 tools/osx/BUILD                        |  3 +--
 7 files changed, 74 insertions(+), 18 deletions(-)

The CI infrastructure changed significantly since the 6.5.0 release such that many tests failed in ways unrelated to the original change described above. This commit contains many other changes to get the builds passing again, comprised of cherry-picks and new fixes:


The most noteworthy fix in this change sets the right JDK version for *_for_testing repos. This fixes several tests broken by an incompatible JVM flag passed to the javac compiler from remotejdk_11:

  • Removes the build_file = "@local_jdk//:BUILD.bazel" parameter from remotejdk*_for_testing repositories in WORKSPACE.
  • Updates dist_http_archive to parse the correct JDK version from the repo name and then generate a BUILD file from JDK_BUILD_TEMPLATE instead.

The flag came from this expression in BazelJavaSemantics.getJvmFlags:

if (JavaRuntimeInfo.from(ruleContext).version() >= 17) {
  jvmFlags.add("-Djava.security.manager=allow");
}

Setting .bazelci/presubmit.yml Linux jobs to use Java 11 images enabled tests to pass on those platforms without the fix. This is because the java_runtime target emitted into @local_jdk//:BUILD.bazel (@local_jdk//:jdk) sets version = "11" on these platforms.

That meant the JavaRuntimeInfo expression above would return 11 on Linux, but a higher value on macOS and Windows. This led to emitting the -Djava.security.manager flag when using the remotejdk_11 compiler on those platforms, breaking the build. Specifying different macOS and Windows build images with Java 11 isn't possible, necessitating the dist_http_archive change to fix tests broken on those platforms.

History and further details:

The following series of commits introduced the remotejdk*_for_testing repos and a mechanism to have the "inner" Bazel reuse the "outer" Bazel's copies:

  • Add external repos required by tests and a "fetch all" filegroup (commit 17506af)

  • Allow overriding all external repositories used by our integration tests (commit 99c0e6d)

  • Move JDK repo definitions to distdir_deps.bzl (commit b741116)

In .bazelci/presubmit.yml, every job sets --test_env=TEST_REPOSITORY_HOME=$OUTPUT_BASE/external. This causes testenv.sh.tmpl to override certain repositories with their _for_testing variants:

if [[ -n ${TEST_REPOSITORY_HOME:-} ]]; then
  echo "testenv.sh: Using shared repositories from $TEST_REPOSITORY_HOME."

  repos=(
    # ...
    "remotejdk17_macos_for_testing"
    # ...
  )
  for repo in "${repos[@]}"; do
    reponame="${repo%"_for_testing"}"
    echo "common --override_repository=$reponame=$TEST_REPOSITORY_HOME/$repo" >> $TEST_TMPDIR/bazelrc
  done
fi

The WORKSPACE file contained entries like the following, created before @local_jdk//:BUILD.bazel contained an explicit JDK version for its //:jdk target:

dist_http_archive(
    name = "remotejdk11_macos_for_testing",
    build_file = "@local_jdk//:BUILD.bazel",
    patch_cmds = EXPORT_WORKSPACE_IN_BUILD_BAZEL_FILE,
    patch_cmds_win = EXPORT_WORKSPACE_IN_BUILD_BAZEL_FILE_WIN,
)

Later, these commits added the JDK version to JdkRuntimeInfo:

However, they didn't update the dist_http_archive entries to adjust their build_file parameters accordingly. As a result, the remotejdk repos used by the "inner" Bazel during tests all specified the same JDK version as the system JDK.

Bazel resolved the correct toolchain for --java_runtime_version, which would contain the correct Java binaries, but contained the system JDK version parameter. This caused BazelJavaSemantics to emit JVM flags for the wrong JDK version. This, in turn, broke test binaries when they ran under the correct java binary from the JDK that differed from the system JDK version.

@mbland
Copy link
Author

mbland commented Oct 29, 2025

@meteorcloudy Note that the bulk of the diff in that pull request is the zlib-1.3.1 upgrade. The rest of the changes are exactly from the patch from #27026 (which I copy and pasted, and applied).

The first CI run failed a lot of tests and several builds were DOA. The Windows and RBE builds, in particular, broke as soon as they started.

The second CI run after pushing a commit to update .bazelversion to 6.3.2 got a lot farther. The Windows builds ran to completion, and the RBE build actually passed. The Clang build passed as well.

I suppose we can figure out the rest of the failures between now and some time after BazelCon.

@meteorcloudy
Copy link
Member

Yeah, our release policy does say we promise to backport fixes for OS-compatibility issues into this LTS release in Maintenance stage. Bazel 6 is still in maintenance until Dec 2025 ;)

I'm fine with disabling many of those failing tests if they look irrelevant, for example, android tests. Many of them are failing due to infrastructure changes.

@iancha1992 iancha1992 added team-Rules-CPP Issues for C++ rules team-OSS Issues for the Bazel OSS team: installation, release processBazel packaging, website labels Oct 30, 2025
mbland added a commit to EngFlow-Academy/bzlmod-bootcamp that referenced this pull request Nov 5, 2025
`rules_proto` 6.0.2 would work here, but we don't actually use
`@rules_proto` anywhere, so we just drop it. We'll eventually upgrade
`protobuf` to v28.0, at which point `rules_proto` isn't strictly
necessary anymore, since `protobuf` will contain `proto_library`, et.
al. See:

- https://github.com/bazelbuild/rules_proto/blob/7.1.0/README.md
- https://github.com/bazelbuild/rules_proto/blob/7.1.0/proto/defs.bzl

`protobuf` v21.7 works with Bazel 7 on Linux and Windows, but fails with
the following two errors on macOS. We'll fix the second error eventually
by upgrading to `protobuf` v28.0.

---

Building with Bazel 6 on macOS 26 Tahoe with Xcode 26 produces:

```txt
ERROR: external/zlib/BUILD.bazel:37:11:
  Compiling trees.c [for tool] failed: (Aborted): wrapped_clang failed:
  error executing command (from target @zlib//:zlib)
  external/local_config_cc/wrapped_clang '-D_FORTIFY_SOURCE=1'
    -fstack-protector -fcolor-diagnostics -Wall -Wthread-safety
    -Wself-assign -fno-omit-frame-pointer -g0 -O2 -DNDEBUG
    '-DNS_BLOCK_ASSERTIONS=1' ... (remaining 34 arguments skipped)

dyld[78138]: missing LC_UUID load command in
  external/local_config_cc/wrapped_clang
dyld[78138]: missing LC_UUID load command
```

This is due to the `wrapped_clang` still included in the Bazel 6 version
of `@bazel_tools` containing an explicit `-no_uuid` linker flag. The
only way to fix this at the moment is to build a custom Bazel 6 binary
from source. (There may be a Bazel 6.6.0 release forthcoming with a fix
for this pending the outcome of bazelbuild/bazel#27463.)

For more details, and for instructions on building a new Bazel 6.5.0
binary for macOS 26, see bazelbuild/bazel#27026. We'll return to this in
the next commit message as well.

---

`protobuf` v21.7 uses Zlib 1.2.11, and versions v22.0 up until v28.0 use
Zlib 1.2.13. v28.0 upgrades to Zlib 1.3.1.

- https://github.com/protocolbuffers/protobuf/blob/v21.7/protobuf_deps.bzl#L53
- https://github.com/protocolbuffers/protobuf/blob/v22.0/protobuf_deps.bzl#L54-L55
- https://github.com/protocolbuffers/protobuf/blob/v27.5/protobuf_deps.bzl#L58-L59
- https://github.com/protocolbuffers/protobuf/blob/v28.0/protobuf_deps.bzl#L60-L61

Zlib versions before 1.3.1 cause the error below when building on macOS
with Xcode 16.4 and Clang 17. See:

- madler/zlib#895: Fix platform check for macOS
- madler/zlib@4bd9a71:
  Remove fdopen #defines in zutil.h.
- llvm/llvm-project#74676:
  [clang][PP] Add extension to predefine target OS macros

```txt
$ bazel build //...

Starting local Bazel server and connecting to it...
INFO: Analyzed 2 targets (113 packages loaded, 2278 targets configured).

INFO: From Compiling adler32.c [for tool]:
In file included from external/zlib/adler32.c:8:
external/zlib/zutil.h:170:11: warning: 'OS_CODE' macro redefined [-Wmacro-redefined]
  170 | #  define OS_CODE 19
      |           ^
external/zlib/zutil.h:141:11: note: previous definition is here
  141 | #  define OS_CODE  7
      |           ^
1 warning generated.

[ ...snip OS_CODE warning from crc32.c:30... ]

ERROR: external/zlib/BUILD.bazel:37:11:
  Compiling zutil.c [for tool] failed: (Exit 1): cc_wrapper.sh failed:
  error executing CppCompile command (from target @@zlib//:zlib)
  external/local_config_cc/cc_wrapper.sh -U_FORTIFY_SOURCE
    -fstack-protector -Wall -Wthread-safety -Wself-assign
    -Wunused-but-set-parameter -Wno-free-nonheap-object
    -fcolor-diagnostics ... (remaining 33 arguments skipped)

ERROR: external/zlib/BUILD.bazel:37:11:
Compiling zutil.c [for tool] failed: (Exit 1): wrapped_clang failed:
error executing command (from target @zlib//:zlib)
external/local_config_cc/wrapped_clang '-D_FORTIFY_SOURCE=1'
-fstack-protector -fcolor-diagnostics -Wall -Wthread-safety
-Wself-assign -fno-omit-frame-pointer -g0 -O2 -DNDEBUG
'-DNS_BLOCK_ASSERTIONS=1' ... (remaining 35 arguments skipped)

[ ...snip OS_CODE warning from zutil.c:8... ]

In file included from external/zlib/zutil.c:10:
In file included from external/zlib/gzguts.h:21:
In file included from
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/stdio.h:61:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/_stdio.h:318:7:
error: expected identifier or '('
  318 | FILE    *fdopen(int, const char *) __DARWIN_ALIAS_STARTING(__MAC_10_6, __IPHONE_2_0, __DARWIN_ALIAS(fdopen));
      |          ^
external/zlib/zutil.h:147:33: note: expanded from macro 'fdopen'
  147 | #        define fdopen(fd,mode) NULL /* No fdopen() */
      |                                 ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/17/include/__stddef_null.h:26:16:
note: expanded from macro 'NULL'
   26 | #define NULL ((void*)0)
      |                ^

[ ...snip two more error messages pointing into __stddef_null.h:26... ]

1 warning and 3 errors generated.

ERROR: external/com_frobozz_rules_magic/src/com/frobozz/magic/BUILD:3:12
  Building external/com_frobozz_rules_magic/src/com/frobozz/magic/EmitDigestsWorker.jar
  (1 source file) [for tool] failed: (Exit 1): cc_wrapper.sh failed:
  [ ...snip repeated CppCompile command error... ]
```
mbland added a commit to EngFlow-Academy/bzlmod-bootcamp that referenced this pull request Nov 7, 2025
`rules_proto` 6.0.2 would work here, but we don't actually use
`@rules_proto` anywhere, so we just drop it. We'll eventually upgrade
`protobuf` to v28.0, at which point `rules_proto` isn't strictly
necessary anymore, since `protobuf` will contain `proto_library`, et.
al. See:

- https://github.com/bazelbuild/rules_proto/blob/7.1.0/README.md
- https://github.com/bazelbuild/rules_proto/blob/7.1.0/proto/defs.bzl

`protobuf` v21.7 works with Bazel 7 on Linux and Windows, but fails with
the following two errors on macOS. We'll fix the second error eventually
by upgrading to `protobuf` v28.0.

---

Building with Bazel 6 on macOS 26 Tahoe with Xcode 26 produces:

```txt
ERROR: external/zlib/BUILD.bazel:37:11:
  Compiling trees.c [for tool] failed: (Aborted): wrapped_clang failed:
  error executing command (from target @zlib//:zlib)
  external/local_config_cc/wrapped_clang '-D_FORTIFY_SOURCE=1'
    -fstack-protector -fcolor-diagnostics -Wall -Wthread-safety
    -Wself-assign -fno-omit-frame-pointer -g0 -O2 -DNDEBUG
    '-DNS_BLOCK_ASSERTIONS=1' ... (remaining 34 arguments skipped)

dyld[78138]: missing LC_UUID load command in
  external/local_config_cc/wrapped_clang
dyld[78138]: missing LC_UUID load command
```

This is due to the `wrapped_clang` still included in the Bazel 6 version
of `@bazel_tools` containing an explicit `-no_uuid` linker flag. The
only way to fix this at the moment is to build a custom Bazel 6 binary
from source. (There may be a Bazel 6.6.0 release forthcoming with a fix
for this pending the outcome of bazelbuild/bazel#27463.)

For more details, and for instructions on building a new Bazel 6.5.0
binary for macOS 26, see bazelbuild/bazel#27026. We'll return to this in
the next commit message as well.

---

`protobuf` v21.7 uses Zlib 1.2.11, and versions v22.0 up until v28.0 use
Zlib 1.2.13. v28.0 upgrades to Zlib 1.3.1.

- https://github.com/protocolbuffers/protobuf/blob/v21.7/protobuf_deps.bzl#L53
- https://github.com/protocolbuffers/protobuf/blob/v22.0/protobuf_deps.bzl#L54-L55
- https://github.com/protocolbuffers/protobuf/blob/v27.5/protobuf_deps.bzl#L58-L59
- https://github.com/protocolbuffers/protobuf/blob/v28.0/protobuf_deps.bzl#L60-L61

Zlib versions before 1.3.1 cause the error below when building on macOS
with Xcode 16.4 and Clang 17. See:

- madler/zlib#895: Fix platform check for macOS
- madler/zlib@4bd9a71:
  Remove fdopen #defines in zutil.h.
- llvm/llvm-project#74676:
  [clang][PP] Add extension to predefine target OS macros

```txt
$ bazel build //...

Starting local Bazel server and connecting to it...
INFO: Analyzed 2 targets (113 packages loaded, 2278 targets configured).

INFO: From Compiling adler32.c [for tool]:
In file included from external/zlib/adler32.c:8:
external/zlib/zutil.h:170:11: warning: 'OS_CODE' macro redefined [-Wmacro-redefined]
  170 | #  define OS_CODE 19
      |           ^
external/zlib/zutil.h:141:11: note: previous definition is here
  141 | #  define OS_CODE  7
      |           ^
1 warning generated.

[ ...snip OS_CODE warning from crc32.c:30... ]

ERROR: external/zlib/BUILD.bazel:37:11:
  Compiling zutil.c [for tool] failed: (Exit 1): cc_wrapper.sh failed:
  error executing CppCompile command (from target @@zlib//:zlib)
  external/local_config_cc/cc_wrapper.sh -U_FORTIFY_SOURCE
    -fstack-protector -Wall -Wthread-safety -Wself-assign
    -Wunused-but-set-parameter -Wno-free-nonheap-object
    -fcolor-diagnostics ... (remaining 33 arguments skipped)

ERROR: external/zlib/BUILD.bazel:37:11:
Compiling zutil.c [for tool] failed: (Exit 1): wrapped_clang failed:
error executing command (from target @zlib//:zlib)
external/local_config_cc/wrapped_clang '-D_FORTIFY_SOURCE=1'
-fstack-protector -fcolor-diagnostics -Wall -Wthread-safety
-Wself-assign -fno-omit-frame-pointer -g0 -O2 -DNDEBUG
'-DNS_BLOCK_ASSERTIONS=1' ... (remaining 35 arguments skipped)

[ ...snip OS_CODE warning from zutil.c:8... ]

In file included from external/zlib/zutil.c:10:
In file included from external/zlib/gzguts.h:21:
In file included from
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/stdio.h:61:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/_stdio.h:318:7:
error: expected identifier or '('
  318 | FILE    *fdopen(int, const char *) __DARWIN_ALIAS_STARTING(__MAC_10_6, __IPHONE_2_0, __DARWIN_ALIAS(fdopen));
      |          ^
external/zlib/zutil.h:147:33: note: expanded from macro 'fdopen'
  147 | #        define fdopen(fd,mode) NULL /* No fdopen() */
      |                                 ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/17/include/__stddef_null.h:26:16:
note: expanded from macro 'NULL'
   26 | #define NULL ((void*)0)
      |                ^

[ ...snip two more error messages pointing into __stddef_null.h:26... ]

1 warning and 3 errors generated.

ERROR: external/com_frobozz_rules_magic/src/com/frobozz/magic/BUILD:3:12
  Building external/com_frobozz_rules_magic/src/com/frobozz/magic/EmitDigestsWorker.jar
  (1 source file) [for tool] failed: (Exit 1): cc_wrapper.sh failed:
  [ ...snip repeated CppCompile command error... ]
```
mbland added a commit to EngFlow-Academy/bzlmod-bootcamp that referenced this pull request Nov 7, 2025
`rules_proto` 6.0.2 would work here, but we don't actually use
`@rules_proto` anywhere, so we just drop it. We'll eventually upgrade
`protobuf` to v28.0, at which point `rules_proto` isn't strictly
necessary anymore, since `protobuf` will contain `proto_library`, et.
al. See:

- https://github.com/bazelbuild/rules_proto/blob/7.1.0/README.md
- https://github.com/bazelbuild/rules_proto/blob/7.1.0/proto/defs.bzl

`protobuf` v21.7 works with Bazel 7 on Linux and Windows, but fails with
the following two errors on macOS. We'll fix the second error eventually
by upgrading to `protobuf` v28.0.

---

Building with Bazel 6 on macOS 26 Tahoe with Xcode 26 produces:

```txt
ERROR: external/zlib/BUILD.bazel:37:11:
  Compiling trees.c [for tool] failed: (Aborted): wrapped_clang failed:
  error executing command (from target @zlib//:zlib)
  external/local_config_cc/wrapped_clang '-D_FORTIFY_SOURCE=1'
    -fstack-protector -fcolor-diagnostics -Wall -Wthread-safety
    -Wself-assign -fno-omit-frame-pointer -g0 -O2 -DNDEBUG
    '-DNS_BLOCK_ASSERTIONS=1' ... (remaining 34 arguments skipped)

dyld[78138]: missing LC_UUID load command in
  external/local_config_cc/wrapped_clang
dyld[78138]: missing LC_UUID load command
```

This is due to the `wrapped_clang` still included in the Bazel 6 version
of `@bazel_tools` containing an explicit `-no_uuid` linker flag. The
only way to fix this at the moment is to build a custom Bazel 6 binary
from source. (There may be a Bazel 6.6.0 release forthcoming with a fix
for this pending the outcome of bazelbuild/bazel#27463.)

For more details, and for instructions on building a new Bazel 6.5.0
binary for macOS 26, see bazelbuild/bazel#27026. We'll return to this in
the next commit message as well.

---

`protobuf` v21.7 uses Zlib 1.2.11, and versions v22.0 up until v28.0 use
Zlib 1.2.13. v28.0 upgrades to Zlib 1.3.1.

- https://github.com/protocolbuffers/protobuf/blob/v21.7/protobuf_deps.bzl#L53
- https://github.com/protocolbuffers/protobuf/blob/v22.0/protobuf_deps.bzl#L54-L55
- https://github.com/protocolbuffers/protobuf/blob/v27.5/protobuf_deps.bzl#L58-L59
- https://github.com/protocolbuffers/protobuf/blob/v28.0/protobuf_deps.bzl#L60-L61

Zlib versions before 1.3.1 cause the error below when building on macOS
with Xcode 16.4 and Clang 17. See:

- madler/zlib#895: Fix platform check for macOS
- madler/zlib@4bd9a71:
  Remove fdopen #defines in zutil.h.
- llvm/llvm-project#74676:
  [clang][PP] Add extension to predefine target OS macros

```txt
$ bazel build //...

Starting local Bazel server and connecting to it...
INFO: Analyzed 2 targets (113 packages loaded, 2278 targets configured).

INFO: From Compiling adler32.c [for tool]:
In file included from external/zlib/adler32.c:8:
external/zlib/zutil.h:170:11: warning: 'OS_CODE' macro redefined [-Wmacro-redefined]
  170 | #  define OS_CODE 19
      |           ^
external/zlib/zutil.h:141:11: note: previous definition is here
  141 | #  define OS_CODE  7
      |           ^
1 warning generated.

[ ...snip OS_CODE warning from crc32.c:30... ]

ERROR: external/zlib/BUILD.bazel:37:11:
  Compiling zutil.c [for tool] failed: (Exit 1): cc_wrapper.sh failed:
  error executing CppCompile command (from target @@zlib//:zlib)
  external/local_config_cc/cc_wrapper.sh -U_FORTIFY_SOURCE
    -fstack-protector -Wall -Wthread-safety -Wself-assign
    -Wunused-but-set-parameter -Wno-free-nonheap-object
    -fcolor-diagnostics ... (remaining 33 arguments skipped)

ERROR: external/zlib/BUILD.bazel:37:11:
Compiling zutil.c [for tool] failed: (Exit 1): wrapped_clang failed:
error executing command (from target @zlib//:zlib)
external/local_config_cc/wrapped_clang '-D_FORTIFY_SOURCE=1'
-fstack-protector -fcolor-diagnostics -Wall -Wthread-safety
-Wself-assign -fno-omit-frame-pointer -g0 -O2 -DNDEBUG
'-DNS_BLOCK_ASSERTIONS=1' ... (remaining 35 arguments skipped)

[ ...snip OS_CODE warning from zutil.c:8... ]

In file included from external/zlib/zutil.c:10:
In file included from external/zlib/gzguts.h:21:
In file included from
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/stdio.h:61:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/_stdio.h:318:7:
error: expected identifier or '('
  318 | FILE    *fdopen(int, const char *) __DARWIN_ALIAS_STARTING(__MAC_10_6, __IPHONE_2_0, __DARWIN_ALIAS(fdopen));
      |          ^
external/zlib/zutil.h:147:33: note: expanded from macro 'fdopen'
  147 | #        define fdopen(fd,mode) NULL /* No fdopen() */
      |                                 ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/17/include/__stddef_null.h:26:16:
note: expanded from macro 'NULL'
   26 | #define NULL ((void*)0)
      |                ^

[ ...snip two more error messages pointing into __stddef_null.h:26... ]

1 warning and 3 errors generated.

ERROR: external/com_frobozz_rules_magic/src/com/frobozz/magic/BUILD:3:12
  Building external/com_frobozz_rules_magic/src/com/frobozz/magic/EmitDigestsWorker.jar
  (1 source file) [for tool] failed: (Exit 1): cc_wrapper.sh failed:
  [ ...snip repeated CppCompile command error... ]
```
@mbland
Copy link
Author

mbland commented Nov 19, 2025

@meteorcloudy I've been poking at this the past week and a half. Between cherrypicking fixes from master and applying my own test fixes, all the Linux builds are now passing. (The pull request has become a bit of a beast, however.)

The macOS and Windows builds continue to fail, and I'm pretty sure the lion's share of those are because those images contain Java 21, not Java 11. (i.e., Several of the failures resemble failures I fixed on Linux by ensuring Java 11.)

Tomorrow I can compile a detailed breakdown of what's still failing. Unless, of course, you want to take a look and can beat me to it. (And if you have a tip as to how we can get Java 11 on the macOS and Windows images and make it the default, that would be great!)

@mbland
Copy link
Author

mbland commented Nov 21, 2025

Progress! I'm down to only these tests failing on macOS and Windows:

macOS shard 2

//src/test/shell/bazel:bazel_example_test
//src/test/shell/bazel:bazel_proto_library_test  # passed, but timed out
//src/test/cpp:option_processor_test
//src/test/shell/bazel:bazel_coverage_java_jdk11_toolchain_head_test
//src/test/shell/bazel/remote:remote_execution_test  

macOS shard 3

//src/test/shell/bazel:bazel_java_tools_test  # passed, but timed out
//src/test/py/bazel:launcher_test
//src/test/shell/bazel:bazel_bootstrap_distfile_tar_test
//src/test/shell/bazel:bazel_coverage_java_jdk11_toolchain_released_test

macOS shard 4

//src/test/shell/bazel:bazel_bootstrap_distfile_test
//src/test/shell/bazel:bazel_java_test_jdk11_toolchain_head
//src/test/shell/bazel:bazel_sandboxing_test
//src/test/shell/bazel:external_starlark_execute_test
//src/test/shell/integration:bazel_command_log_test
//src/test/shell/bazel:bazel_repository_cache_test

macOS shard 5

//src/test/py/bazel:bazel_remote_cache_decompression_test
//src/test/shell/bazel:bazel_workspaces_test

Windows shard 2

//src/test/shell/bazel:bazel_bootstrap_distfile_test
//src/test/shell/bazel:bazel_java_test_jdk11_toolchain_head

Windows shard 3

//src/test/py/bazel:bazel_windows_cpp_test

Windows shard 4

//src/test/java/com/google/devtools/build/lib/bazel/repository/downloader:DownloaderTestSuite

@meteorcloudy
Copy link
Member

you have a tip as to how we can get Java 11 on the macOS and Windows images and make it the default

Unfortunately this is no longer possible, we've moved forward with CI image. I'm fine with disabling those tests if they don't actually catch any regression for in new changes.

@loeffel-io
Copy link
Contributor

loeffel-io commented Dec 7, 2025

any update here? we really could need something like 6.5.1 🙏

@mbland
Copy link
Author

mbland commented Dec 8, 2025

@meteorcloudy Modulo the last macOS test timeouts I'm squashing, I've got all the CI builds passing. The pull request has become a beast, so I can split it up or rebase it to squash/drop commits if you like.

(Oddly enough, the CLA check is failing because I cherry-picked a commit by @katre, even though his google.com address is in the commit.)

The core changes from the very first commit remain unchanged. I've cherry-picked a number of commits from master to backport fixes that didn't land on the 6.5.0 branch (applying manual tweaks where necessary). There are a bunch of debugging commits I can rebase out of existence if need be.

Some of the more significant recent commits (without going back to the very beginning) include:

The real pièce de résistance that restored my sanity was 8a5bd6f, to set the right JDK version for *_for_testing repos. Bazel was emitting JVM flags for Java >= 17 into test stubs using a Java 11 runtime. It took weeks to track down what was happening, which amounted to an omission from 7556e11 and 2d04c91.

Please let me know how you'd like to proceed from here.

@loeffel-io
Copy link
Contributor

@meteorcloudy please, we do have a lot of bazel 6.x projects which are broken and we can't upgrade to bazel 7 because rules_docker is not compatible. We need a solution for this 🙏

@iancha1992
Copy link
Member

@meteorcloudy please, we do have a lot of bazel 6.x projects which are broken and we can't upgrade to bazel 7 because rules_docker is not compatible. We need a solution for this 🙏

@loeffel-io @mbland Hi there! Thanks for the ping. We’re currently buried in Bazel 8 and 9 work, so we decided last week to circle back to this in January. We’ll be in touch then—thanks for your patience!

@meteorcloudy
Copy link
Member

@mbland Thank you so much for making this work! Yes, can you please squash everything into one commit and document major changes in the PR description? We'll start a new Bazel 6 release soonish!

@loeffel-io
Copy link
Contributor

@mbland Thank you so much for making this work! Yes, can you please squash everything into one commit and document major changes in the PR description? We'll start a new Bazel 6 release soonish!

🙏

@mbland mbland force-pushed the macos-26-tahoe-fix branch from cd98fc5 to 7b89442 Compare January 5, 2026 22:54
@mbland
Copy link
Author

mbland commented Jan 5, 2026

@meteorcloudy Done. Looks like the CLA check is passing now, too.

I just happen to be aware of the issue from #26366, whereby building Linux binaries on Rocky 8 (instead of CentOS 7) bumps the Glibc version requirement from 2.17 to 2.25. Might be worth a release note akin to that from 8.3.0 or 7.7.0.

@iancha1992
Copy link
Member

@meteorcloudy Done. Looks like the CLA check is passing now, too.

I just happen to be aware of the issue from #26366, whereby building Linux binaries on Rocky 8 (instead of CentOS 7) bumps the Glibc version requirement from 2.17 to 2.25. Might be worth a release note akin to that from 8.3.0 or 7.7.0.

Hi @mbland! Bazel team is currently taking a look. Thank you so much!

@iancha1992 iancha1992 changed the base branch from release-6.5.0 to release-6.6.0 January 7, 2026 14:10
@iancha1992 iancha1992 changed the base branch from release-6.6.0 to release-6.5.0 January 7, 2026 14:12
@iancha1992
Copy link
Member

@mbland Could you please change the base branch to release-6.6.0? If I do it myself, then this commit will belong to me, instead of you. I want to make sure we give credit to you.

@mbland mbland changed the base branch from release-6.5.0 to release-6.6.0 January 7, 2026 14:22
@iancha1992 iancha1992 changed the title [6.5.0] Fixes for using recent Xcodes and macOS 26 [6.6.0] Fixes for using recent Xcodes and macOS 26 Jan 7, 2026
Upgrades Zlib to 1.3.1 and removes the `-no_adhoc_codesign` and
`-no_uuid` linker flags for building the Clang wrapper. Fixes bazelbuild#27026.

Incorporates changes from the following commits and pull requests:

- Update third_party/zlib/BUILD{,.tools} for 1.3.1
  bazelbuild/bazel@b29649f
  bazelbuild/bazel@2067a8b

- Fix upb build with Clang 16
  bazelbuild/bazel@4a32d7b
  bazelbuild#23700

- Remove -Wl{-no_adhoc_codesign,-no_uuid}
  Required to build under macOS 26.0 to avoid the \`missing LC_UUID\`
  error from \`wrapped_clang\` and \`xcode-locator\`.
  bazelbuild#27014

Upgrades `third_party/zlib` using the methodology from bazelbuild#27026:

```txt
cd third_party
curl -L https://github.com/madler/zlib/releases/download/v1.3.1/zlib-1.3.1.tar.gz |
  gzip -dc | tar xf -
rsync -r zlib-1.3.1/* zlib/
rm -rf zlib-1.3.1
cd ..
```

Several `third_party/zlib` files show up as complete replacements because the
previous files used CRLF line endings, and the 1.3.1 files only use LF.

To list the substantial changes outside of the Zlib 1.3.1 distribution upgrade:

```txt
$ git diff --stat HEAD^ -- distdir_deps.bzl third_party/{upb,zlib/BUILD*} tools/{cpp,osx}
 distdir_deps.bzl                       |  4 ++++
 third_party/upb/01_remove_werror.patch | 19 +++++++++++++++++++
 third_party/upb/BUILD                  |  4 +++-
 third_party/zlib/BUILD                 | 45 ++++++++++++++++++++++++++++++++++++++-------
 third_party/zlib/BUILD.tools           | 15 +++++++++------
 tools/cpp/osx_cc_configure.bzl         |  2 --
 tools/osx/BUILD                        |  3 +--
 7 files changed, 74 insertions(+), 18 deletions(-)
```

---

The CI infrastructure changed significantly since the 6.5.0 release such
that many tests failed in ways unrelated to the original change
described above. This commit contains many other changes to get the
builds passing again, comprised of cherry-picks and new fixes:

- Bumps .bazelversion to 6.3.2 (from the previous 6.2.1)
- Sets --test_output=errors in CI to expose more failure information.
- Fixes Python 3 related failures:
  - Updates collections to collections.abc.
  - Replaces `assertRegexpMatches` with `assertRegex` and `assertEquals`
    with `assertEqual`.
  - Updates `pywrapper_template.txt` to use Bash instead of Bourne
    Shell and replaces `echo | grep` with `[[ ]]` and `=~`.
  - Disables //src/test/shell/bazel:python_version_test since several
    platforms no longer ship with a `python2` binary.
  - Migrates `imp` to `importlib` in `path_test.py`
    (commit 5b54c74, bazelbuild#24732).
  - Uses `python3` in `remote_helpers.sh`
    (commit bb7d83c).
  - Fixes bazel `py_test` `testSmoke` for Python 3.11+
    (commit 9b027c8).
  - Fixes `bazel_overrides_test.py` on Windows
    (commit be2186f).
  - Updates `python` to `python3` in several tests.
- Disables //src/test/shell/bazel/android/... and other Android tests
  due to incompatible CI infrastructure changes, especially after
  bazelbuild/continuous-integration#2408.
- Stops removing `add_opens` and `add_exports` configurations per
  bazelbuild#19850 (commit 6a7e9ba).
- Disables the GoogleTestSecurityManager by default and explicitly
  enable it for tests that rely on it.
  (commit 366c4ba).
- Adds a shutdown hook for tests that detects `System.exit`
  (commit 5fc2b01).
- Fixes serialization tests for JDK 21
  (commit 052407b).
- Fixes ProxyHelperTest on JDK@HEAD
  (commit 4c85356).
- Fixes the circuitbreaker test by increasing a sleep in
  `testRecordFailure_circuitTrips` by 5ms as currently done on `master`.
- Removes `python` binary requirement for Bazel tests
  (commit e9316f0).
- Runs idle server GC immediately following a
  `--nokeep_state_after_build_command` instead of waiting 10 seconds.
  (commit f6344ff).
- Skips `test_cc_toolchain_tsan_feature` on Linux due to new Linux CI
  images that surface google/sanitizers#1716.
- Removes redundant onCompleted call
  (commit 0f6dd59, bazelbuild#17792).
- Disables the `SecurityManager` in `TestRunner`.
- Applies `-proc:full` in `SourceCompilationUnit.compile` when
  supported by the JDK, and applies `-Xlint:-options` otherwise.
- Makes `bazel_sandboxing_test` case case insensitive to fix
  `test_sandbox_block_network_access` on Rocky Linux 8.
- Adds or extends timeout values to a number of tests, especially for
  macOS builds.
- Enables `--incompatible_sandbox_hermetic_tmp` by default
  (commit e2c0276, bazelbuild#19943, bazelbuild#20136).
- Sets `--{,tool_}java_language_version=11` and
  `--{,tool_}java_runtime_version=remotejdk_11` in `.bazelrc` and
  several tests.
- Sets `sandbox_add_mount_pair=/tmp` to fix `bazel_sandboxing_test`.
- Replaces the `ubuntu2004` CI build with `ubuntu2004_java11`.
- Sets `sandbox_default_allow_network=true` on macos CI to match
  `master`. See commit 355b000, bazelbuild#23726.
- Explicitly sets up the `local_jdk` `javabase` in the tests where it's
  actually used (commit 77c2791).
- Loads the Bash runfiles library in `integration_test_setup.sh`
  (commit b20085a).
- Removed `remote_helper.sh` from tests that don't need it to fix
  unbound variable errors after cherry-picking
  commit 77c2791 and
  commit b20085a.
- Adds `@local_jdk//:jdk` to test targets depending upon
  `remote_helpers.sh` (which invokes `setup_localjdk_javabase`) after
  cherry-picking commit b20085a.
- Ensures `local_jdk/bin/javac` compiles to Java 11 in
  `external_integration_test.sh`.
- Fixes `bazel_proto_library_test` after cherry-picking
  commit b20085a:
  - Fixes an incorrect `rlocation` path.
  - Adds a `new_local_repository` for the builtin `zlib`.
  - Replaces the numeric `-eq` Bash comparison operator with the `-z`
    string comparison operator.
- Uses Googletest matchers in `option_processor_test` to help macOS CI
  failures reveal more useful information.
- Fixes `//src/test/cpp:option_processor_test` by ignoring `.bazelrc`
  files (commit 89c69c2;
  see also: bazelbuild/continuous-integration#2201).
- Sets `--{,tool_}java_runtime_version=local_jdk` in
  `bazel_bootstrap_distfile_test.sh`. This fixed previous macOS and
  Windows errors when set to Java 11, which no longer ships in recent CI
  build images.
- Fixes and verifies Java version compatibility of Java tools
  (parts of commit b841e14, bazelbuild#26126).
- Ensures Java tools use specific Java versions based on transitions
  based on `tools/build_defs.bzl` from
  commit b532a46.
- Sets `--noannounce_rc` in `bazel query` expressions from
  `workspace_test` cases to fix Linux CI builds.
- Moves `//src:release_archive.bzl` constants to
  `//tools:build_defs.bzl` to fix tests that use `copy_tools_directory`
  from `testenv.sh.tmpl`.
- Fixes `repository/downloader:DownloaderTestSuite` for Windows + JDK21
  (commit 59cf396, bazelbuild#21593).
- Fixes `HttpConnectorTest` missing symbol error after
  cherry-picking commit 59cf396 by
  changing a Truth expression to an equivalent assertion.
- Supports using newer JDK versions when compiling Bazel from source
  without an existing Bazel binary
  (commit af50ad3, bazelbuild#18961, bazelbuild#19125).
- Removes `testBuildArmCppBinaryWithMsvcCL` on Windows
  (commit 2158dac, bazelbuild#26795, bazelbuild#26797).
- Prefers IPv6 network in integration tests on macOS
  (commit 2c51a0c).
- Fixes `bazel_command_log_test.sh`:
  - Updates a `sed` expression to avoid using the `+` operator
    (commit 130b444).
  - Makes sure the batch log also clears log messages from the newer
    JDKs (commit e8966bb).
  - Ignores warning about JAVA_TOOL_OPTIONS in test
    (commit 2ceb132, bazelbuild#20274).
  - Disables bzlmod (thus avoiding any requests to BCR) and filters out
    irrelevant log entries
    commit a737169).
  - Ignores the `Picked up JAVA_TOOL_OPTIONS` message that now appears
    on macOS runs after cherry-picking
    commit 2c51a0c.

---

The most noteworthy fix in this change sets the right JDK version for
`*_for_testing` repos. This fixes several tests broken by an
incompatible JVM flag passed to the `javac` compiler from
`remotejdk_11`:

- Removes the `build_file = "@local_jdk//:BUILD.bazel"` parameter from
  `remotejdk*_for_testing` repositories in `WORKSPACE`.
- Updates `dist_http_archive` to parse the correct JDK version from
  the repo name and then generate a `BUILD` file from
  `JDK_BUILD_TEMPLATE` instead.

The flag came from this expression in `BazelJavaSemantics.getJvmFlags`:

```java
if (JavaRuntimeInfo.from(ruleContext).version() >= 17) {
  jvmFlags.add("-Djava.security.manager=allow");
}
```

Setting `.bazelci/presubmit.yml` Linux jobs to use Java 11 images
enabled tests to pass on those platforms without the fix. This is
because the `java_runtime` target emitted into
`@local_jdk//:BUILD.bazel` (`@local_jdk//:jdk`) sets `version = "11"` on
these platforms.

That meant the `JavaRuntimeInfo` expression above would return `11` on
Linux, but a higher value on macOS and Windows. This led to emitting the
`-Djava.security.manager` flag when using the `remotejdk_11` compiler on
those platforms, breaking the build. Specifying different macOS and
Windows build images with Java 11 isn't possible, necessitating the
`dist_http_archive` change to fix tests broken on those platforms.

History and further details:

The following series of commits introduced the `remotejdk*_for_testing`
repos and a mechanism to have the "inner" Bazel reuse the "outer"
Bazel's copies:

- Add external repos required by tests and a "fetch all" filegroup
  (commit 17506af)

- Allow overriding all external repositories used by our integration
  tests (commit 99c0e6d)

- Move JDK repo definitions to distdir_deps.bzl
  (commit b741116)

In `.bazelci/presubmit.yml`, every job sets
`--test_env=TEST_REPOSITORY_HOME=$OUTPUT_BASE/external`. This causes
`testenv.sh.tmpl` to override certain repositories with their
`_for_testing` variants:

```sh
if [[ -n ${TEST_REPOSITORY_HOME:-} ]]; then
  echo "testenv.sh: Using shared repositories from $TEST_REPOSITORY_HOME."

  repos=(
    # ...
    "remotejdk17_macos_for_testing"
    # ...
  )
  for repo in "${repos[@]}"; do
    reponame="${repo%"_for_testing"}"
    echo "common --override_repository=$reponame=$TEST_REPOSITORY_HOME/$repo" >> $TEST_TMPDIR/bazelrc
  done
fi
```

The `WORKSPACE` file contained entries like the following, created
before `@local_jdk//:BUILD.bazel` contained an explicit JDK version for
its `//:jdk` target:

```py
dist_http_archive(
    name = "remotejdk11_macos_for_testing",
    build_file = "@local_jdk//:BUILD.bazel",
    patch_cmds = EXPORT_WORKSPACE_IN_BUILD_BAZEL_FILE,
    patch_cmds_win = EXPORT_WORKSPACE_IN_BUILD_BAZEL_FILE_WIN,
)
```

Later, these commits added the JDK version to `JdkRuntimeInfo`:

- Add version to JavaRuntimeInfo
  (commit 7556e11)

- [6.2.0] Add version to JavaRuntimeInfo (bazelbuild#17913)
  (commit 2d04c91)

However, they didn't update the `dist_http_archive` entries to adjust
their `build_file` parameters accordingly. As a result, the
`remotejdk` repos used by the "inner" Bazel during tests all specified
the same JDK version as the system JDK.

Bazel resolved the correct toolchain for `--java_runtime_version`, which
would contain the correct Java binaries, but contained the system JDK
`version` parameter. This caused `BazelJavaSemantics` to emit JVM flags
for the _wrong_ JDK version. This, in turn, broke test binaries when
they ran under the _correct_ `java` binary from the JDK that differed
from the system JDK version.
@iancha1992 iancha1992 added this to the 6.6.0 release blockers milestone Jan 7, 2026
@mbland mbland force-pushed the macos-26-tahoe-fix branch from 7b89442 to dbdbe58 Compare January 7, 2026 14:31
@mbland
Copy link
Author

mbland commented Jan 7, 2026

@iancha1992 Done.

I also updated the commit message and pull request description slightly to update the git diff --stat command, fix a typo (build_file = "@local_jdk//:BUILD.bazel), and convert tabs to spaces in the sh block starting with if [[ -n ${TEST_REPOSITORY_HOME:-} ]].

@mbland
Copy link
Author

mbland commented Jan 7, 2026

Copy link
Member

@meteorcloudy meteorcloudy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@meteorcloudy meteorcloudy merged commit 1159b37 into bazelbuild:release-6.6.0 Jan 7, 2026
19 checks passed
@mbland mbland deleted the macos-26-tahoe-fix branch January 7, 2026 16:43
@mbland
Copy link
Author

mbland commented Jan 8, 2026

@meteorcloudy @iancha1992 I noticed postsubmit builds on release-6.6.0 and release-6.6.0rc1 are failing. The failing builds seem to be choking on obsolete Android SDK dependencies.

I'd wondered if I also needed to update one or more of .bazelci/{build_bazel_binaries,postsubmit{,_bzlmod}}.yml to disable the Android tests as I'd done for .bazelci/presubmit.yml. Would you like me to open a pull request to do that, or would one of you like to handle it?

@mbland mbland mentioned this pull request Jan 8, 2026
6 tasks
@iancha1992
Copy link
Member

@meteorcloudy @iancha1992 I noticed postsubmit builds on release-6.6.0 and release-6.6.0rc1 are failing. The failing builds seem to be choking on obsolete Android SDK dependencies.

I'd wondered if I also needed to update one or more of .bazelci/{build_bazel_binaries,postsubmit{,_bzlmod}}.yml to disable the Android tests as I'd done for .bazelci/presubmit.yml. Would you like me to open a pull request to do that, or would one of you like to handle it?

@mbland Yes please :)
We'll start rc1 for 6.6.0 once it's fixed. Thank you!

mbland added a commit to mbland/bazel that referenced this pull request Jan 8, 2026
Fixes postsubmit job failures after merging bazelbuild#27463 by aligning the
`.bazelci` postsubmit jobs with `.bazelci/presubmit.yml`. Specifically:

- Disabled Android setup and test targets due to infrastructure changes
  (including but not limited to bazelbuild/continuous-integration#2408).

- Updated the `ubuntu2004` jobs to `ubuntu2004_java11` (not that it
  should matter after the `*_for_testing` repo fix from bazelbuild#27463).

- Updated the `xcode_version` of `macos` jobs.

Contains other touch-ups to synchronize build images, build flags, test
flags, and test targets, as well as to make minor formatting changes.

Produced most changes via: `vimdiff .bazelci/p{re,ost}submit.yml`.
The remaining differences are:

- No sharding in postsubmit jobs.

- Setting `--noremote_accept_cached` in postsubmit jobs

- Including `include_json_profile` blocks in postsubmit jobs.

- The `kythe_ubuntu2004` job sets `index_upload_gcs` to `False` in
  presubmit and `True` in postsubmit.

- The `docs_ubuntu1804` job only exists in presubmit.

Part of bazelbuild#28117.
@mbland
Copy link
Author

mbland commented Jan 8, 2026

Filed #28193.

meteorcloudy pushed a commit that referenced this pull request Jan 9, 2026
Fixes postsubmit job failures after merging #27463 by aligning the
`.bazelci` postsubmit jobs with `.bazelci/presubmit.yml`. Specifically:

- Disabled Android setup and test targets due to infrastructure changes
(including but not limited to bazelbuild/continuous-integration#2408).

- Updated the `ubuntu2004` jobs to `ubuntu2004_java11` (not that it
should matter after the `*_for_testing` repo fix from #27463).

- Updated the `xcode_version` of `macos` jobs.

Contains other touch-ups to synchronize build images, build flags, test
flags, and test targets, as well as to make minor formatting changes.

Produced most changes via: `vimdiff .bazelci/p{re,ost}submit.yml`. The
remaining differences are:

- No sharding in postsubmit jobs.

- Setting `--noremote_accept_cached` in postsubmit jobs

- Including `include_json_profile` blocks in postsubmit jobs.

- The `kythe_ubuntu2004` job sets `index_upload_gcs` to `False` in
presubmit and `True` in postsubmit.

- The `docs_ubuntu1804` job only exists in presubmit.

Part of #28117.
@iancha1992
Copy link
Member

Hi @mbland! Thanks so much for the postsubmits fixes!
However, we tried to push the rc1 for 6.6.0 just now, and we're facing one last issue.
https://buildkite.com/bazel-trusted/bazel-release/builds/2187#019ba476-d731-4594-be6b-a4db3a6e38e6

Do you want to take a look at this one as well? Our team is very much buried in Bazel 9.0.0 and 8.6.0 at the moment. We'd really appreciate your help. Thanks!

@mbland
Copy link
Author

mbland commented Jan 9, 2026

@iancha1992 I would, but I don't seem to have access to that build report. It gives me a 404. If you can give me access, or send me the report somehow, I'm happy to take a look.

@mbland
Copy link
Author

mbland commented Jan 9, 2026

@iancha1992 FWIW, from the report I can see, the latest release-6.6.0rc1 build passes.

@iancha1992
Copy link
Member

@iancha1992 FWIW, from the report I can see, the latest release-6.6.0rc1 build passes.

Yes, that is the postsubmit passing. But we have an error from our release pipeline.

The error is currently occuring on Rocky Linux 8 only.

INFO: Elapsed time: 169.303s, Critical Path: 100.21s
--
INFO: 2981 processes: 62 internal, 2118 linux-sandbox, 1 local, 800 worker.
INFO: Build completed successfully, 2981 total actions
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
DEBUG: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/ec321eb2cc2d0f8f91b676b6d4c66c29/external/build_bazel_rules_nodejs/index.bzl:122:10: WARNING: check_rules_nodejs_version has been removed. This is a no-op, please remove the call.
ERROR: Skipping ':maven-srcs': no such target '//:maven-srcs': target 'maven-srcs' not declared in package '' defined by /workdir/BUILD (did you mean 'bazel-srcs'? Tip: use `query "//:*"` to see all the targets in that package)
WARNING: Target pattern parsing failed.
ERROR: no such target '//:maven-srcs': target 'maven-srcs' not declared in package '' defined by /workdir/BUILD (did you mean 'bazel-srcs'? Tip: use `query "//:*"` to see all the targets in that package)
INFO: Elapsed time: 6.518s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (2 packages loaded)
🚨 Error: The command exited with status 1

We'll also take a look at this one soon. Thanks!

@mbland
Copy link
Author

mbland commented Jan 9, 2026

Got it: The Rocky Linux job from bazelbuild/continuous-integration/pipelines/bazel-release.yml builds the //:maven-srcs target, added in bazelbuild/continuous-integration@14a0378. (This was a CentOS 7 job until bazelbuild/continuous-integration#2281.)

This target is part of the rules_jvm_external introduction in Bazel 7.0.0. All the other targets in the bazel build invocation exist on the Bazel 6.6.0 branch.

It seems that the change to build :maven-srcs in the release job is intended to fix MODULE.bazel.lock generation, which shouldn't have any bearing on the Bazel 6.6.0 release, as far as I can tell. I'll whip up a pull request to introduce a dummy //:maven-srcs target to see if that does the trick.

mbland added a commit to mbland/bazel that referenced this pull request Jan 9, 2026
Fixes the following Rocky Linux 8 Bazel release job failure described in
bazelbuild#27463 (comment):

```txt
ERROR: Skipping ':maven-srcs': no such target '//:maven-srcs':
  target 'maven-srcs' not declared in package ''
  defined by /workdir/BUILD
WARNING: Target pattern parsing failed.
ERROR: no such target '//:maven-srcs':
  target 'maven-srcs' not declared in package ''
  defined by /workdir/BUILD
```

Reproduced and confirmed the fix by executing the failing release job
command locally:

```txt
./bazel-bin/src/bazel build --nobuild \
  :bazel-srcs :bootstrap-jars :maven-srcs //src:derived_java_srcs
```

Part of bazelbuild#28177.

---

Background from:

- bazelbuild#27463 (comment)

`bazelbuild/continuous-integration/pipelines/bazel-release.yml` defines
the failing Rocky Linux 8 job. (This was a CentOS 7 job prior to
bazelbuild/continuous-integration#2281.)

- https://github.com/bazelbuild/continuous-integration/blob/agent-0.2.7/pipelines/bazel-release.yml#L85

bazelbuild/bazel@3d7937e introduced
the `//:maven-srcs` target in Bazel 7.0.0. All the other targets in the
failing `bazel build` invocation exist on the Bazel 6.6.0 branch.

bazelbuild/continuous-integration@14a0378
added the `//:maven-srcs` target to this job as an attempt to fix
`MODULE.bazel.lock` generation.

Neither the lock file nor the `//:maven-srcs` target itself should have
any bearing on the Bazel 6.6.0 release. This dummy target fixes the
build failure and allows the release to proceed.
@mbland
Copy link
Author

mbland commented Jan 9, 2026

@iancha1992 @meteorcloudy Filed #28208.

@iancha1992
Copy link
Member

iancha1992 commented Jan 12, 2026

The changes in this PR have been included in Bazel 6.6.0 RC1. Please test out the release candidate and report any issues as soon as possible.
If you're using Bazelisk, you can point to the latest RC by setting USE_BAZEL_VERSION=6.6.0rc1. Thanks!

@mbland
Copy link
Author

mbland commented Jan 12, 2026

Can confirm Bazel 6.6.0 RC1 works on macOS 26 Tahoe.

I've built bazel-contrib/rules_scala 7.1.6 in legacy WORKSPACE mode by running ./test_all.sh with the following modifications to accommodate Bazel 6:

  • Set .bazelversion to 6.6.0rc1
  • Set --noenable_bzlmod in .bazelrc and commented out flags unknown to Bazel 6
  • protobuf 31.1 (last version without protocolbuffers/protobuf@da0077e)
  • rules_java 9.0.3 (last version without bazelbuild/rules_java@b5b4100)
  • rules_python 1.6.3 (last version without bazel-contrib/rules_python@726ffa2)
  • com.thesamet.scalapb:compilerplugin_{2.12,2.13,3}:1.0.0-alpha.1 (last version compatible with protobuf < 32.0)
  • Disabled override --test_env test cases in test/EnvAttributeManualTest.scala (since Bazel 6 doesn't contain 01a90d6 or 87b0a1f)

I've also successfully built EngFlow-Academy/bzlmod-bootcamp@7ab6a14 via:

$ bootcamp/run_tests.sh -b 6.6.0rc1

Bazel version: release 6.6.0rc1

File: test/smoke_tests.sh
running test_build
  test_build successful (4s)
running test_magic_tests
  test_magic_tests successful (1s)

File: test/emit_values_test.sh
running test_uses_default_config_values
  test_uses_default_config_values successful (0s)
running test_reads_spells_json
  test_reads_spells_json successful (0s)
running test_uses_config_values_from_repo_env_command_line_option
  test_uses_config_values_from_repo_env_command_line_option successful (2s)
running test_uses_different_toolchain_based_on_game
  test_uses_different_toolchain_based_on_game successful (1s)
running test_uses_toolchain_from_extra_toolchains_command_line_option
  test_uses_toolchain_from_extra_toolchains_command_line_option successful (1s)
running test_fail_if_default_version_not_in_versions_list
  test_fail_if_default_version_not_in_versions_list successful (0s)
running test_fail_if_default_game_not_in_games_list
  test_fail_if_default_game_not_in_games_list successful (0s)

File: test/minimum_dependency_versions_test.sh
running test_minimum_supported_versions
  test_minimum_supported_versions successful (5s)
running test_bazel_7_with_rules_java_8
  test_bazel_7_with_rules_java_8 successful (1s)
running test_bazel_8
  test_bazel_8 successful (5s)

12 test(s) passed in 21s

@loeffel-io
Copy link
Contributor

Bazel 6.6.0 RC1 also work for us!
Thank you so much 🙏

@iancha1992
Copy link
Member

iancha1992 commented Jan 13, 2026

Can confirm Bazel 6.6.0 RC1 works on macOS 26 Tahoe.

I've built bazel-contrib/rules_scala 7.1.6 in legacy WORKSPACE mode by running ./test_all.sh with the following modifications to accommodate Bazel 6:

  • Set .bazelversion to 6.6.0rc1
  • Set --noenable_bzlmod in .bazelrc and commented out flags unknown to Bazel 6
  • protobuf 31.1 (last version without protocolbuffers/protobuf@da0077e)
  • rules_java 9.0.3 (last version without bazelbuild/rules_java@b5b4100)
  • rules_python 1.6.3 (last version without bazel-contrib/rules_python@726ffa2)
  • com.thesamet.scalapb:compilerplugin_{2.12,2.13,3}:1.0.0-alpha.1 (last version compatible with protobuf < 32.0)
  • Disabled override --test_env test cases in test/EnvAttributeManualTest.scala (since Bazel 6 doesn't contain 01a90d6 or 87b0a1f)

I've also successfully built EngFlow-Academy/bzlmod-bootcamp@7ab6a14 via:

$ bootcamp/run_tests.sh -b 6.6.0rc1

Bazel version: release 6.6.0rc1

File: test/smoke_tests.sh
running test_build
  test_build successful (4s)
running test_magic_tests
  test_magic_tests successful (1s)

File: test/emit_values_test.sh
running test_uses_default_config_values
  test_uses_default_config_values successful (0s)
running test_reads_spells_json
  test_reads_spells_json successful (0s)
running test_uses_config_values_from_repo_env_command_line_option
  test_uses_config_values_from_repo_env_command_line_option successful (2s)
running test_uses_different_toolchain_based_on_game
  test_uses_different_toolchain_based_on_game successful (1s)
running test_uses_toolchain_from_extra_toolchains_command_line_option
  test_uses_toolchain_from_extra_toolchains_command_line_option successful (1s)
running test_fail_if_default_version_not_in_versions_list
  test_fail_if_default_version_not_in_versions_list successful (0s)
running test_fail_if_default_game_not_in_games_list
  test_fail_if_default_game_not_in_games_list successful (0s)

File: test/minimum_dependency_versions_test.sh
running test_minimum_supported_versions
  test_minimum_supported_versions successful (5s)
running test_bazel_7_with_rules_java_8
  test_bazel_7_with_rules_java_8 successful (1s)
running test_bazel_8
  test_bazel_8 successful (5s)

12 test(s) passed in 21s

Hi @mbland. For 6.6.0rc1, do you think you can draft the release notes on https://github.com/bazelbuild/bazel/releases for us?

@mbland
Copy link
Author

mbland commented Jan 13, 2026

@iancha1992 The notes from the Bazel 6.6.0 draft release announcement look fine to me. What sort of information would you like for me to add, or would you like for me to do the Markdown conversion, or what?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

team-OSS Issues for the Bazel OSS team: installation, release processBazel packaging, website team-Rules-CPP Issues for C++ rules

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants