Use rules_jvm_external to fetch jars dependencies #17112

meteorcloudy · 2023-01-03T05:35:43Z

Historically, Bazel checks in precompiled jar dependencies under the third_party directory, which has a few drawbacks:

Due to those precompiled artifacts, we cannot check in the third_party directory into google3. This means we often have to split a PR into third_party and non-third_party changes and merge them separately, which makes it very hard to maintain the dependencies under third_party directory.
Checked in jar files are large and will forever increase the size of the Bazel git repository.
Checked in jar files are supposed to be downloaded from Maven Central Repository, but there is no verification on that.

This PR tries to use rules_jvm_external to replace the checked-in guava jar files and all jar dependencies in guava's transitive clousure. To achieve this and not breaking Bazel's offline bootstrap build, we had to

Patch rules_jvm_external to make it easier to vendor the @maven repository and export downloaded jar files.
Vendor BUILD, WORKSPACE and jar files from the @maven repository in the bootstrap distfile.
Point filegroup targets of jar files required by the bootstrap java toolchain to use jar files from the @maven repository.

This PR doesn't yet replace and remove all checked-in jar files, but it provides a framework to do so and can help avoid #17014

This is also working towards resolving #17036

meteorcloudy · 2023-01-03T05:36:30Z

FYI @rjobredeaux3

aiuto

Nice start. Just some questions and a nit or two.

BUILD

WORKSPACE

third_party/BUILD

haxorz · 2023-01-03T18:26:24Z

Hey @meteorcloudy , what do sort of review do you need from me? Or is @aiuto 's review sufficient?

meteorcloudy · 2023-01-04T05:56:51Z

@haxorz I think you are somehow suggested by GitHub, @aiuto and @Wyverald should be enough, thanks!

meteorcloudy · 2023-01-04T05:58:06Z

/cc @ted-xie @comius feel free to take a look, but mostly I'm adding you as FYI

aiuto

This is great.
Will the next phase be to replace the targets java_library targets like //third_party:jsr305 with targets outside of //third_party, so that we eventually reduce //third_party/BUILD to nothing?

Wyverald

nice!!!

meteorcloudy · 2023-01-05T05:31:07Z

so that we eventually reduce //third_party/BUILD to nothing?

Something like that, at least we want to remove all content under third_party that prevents us from checking in this directory into google3.

meteorcloudy · 2023-01-05T05:31:45Z

Thanks for the review, I'll split this change into separate PRs to merge it.

- Add rules_jvm_external.patch Merging bazelbuild#17112

Non-third_party changes in bazelbuild#17112 Merging bazelbuild#17112

- Delete replaced jar files and update BUILD files. Merging bazelbuild#17112

jin · 2023-01-05T07:17:12Z

cc @shs96c @cheister

jin · 2023-01-05T07:18:41Z

WORKSPACE

+
+load("@rules_jvm_external//:defs.bzl", "maven_install")
+
+maven_install(


Please also use pinning to use bazel's downloader and repo caching instead of coursier.

Oh, I see, thanks! Does the pinning also work with Bzlmod?

It should! @shs96c could correct me if it doesn't :)

Nice, I'll try it!

Done, please take another look ;)

Looks like the vendoring of @maven no longer works. glob(["v1/**/*.jar"]) doesn't work anymore since jar files are copied by a genrule from their own repo. Have to figure out a workaround..

https://github.com/bazelbuild/rules_jvm_external#accessing-transitive-dependencies-list may be useful to generate BUILD rules on the Bazel side if all you need is a way to get the complete list of artifacts.

OK, I still have to hack rules_jvm_external to make it work. Basically, I need to mock the maven repository for the bootstrap build, which includes vendoring all the jar files and generate a usable BUILD file.

Pinning works with bzlmod, and if there are problems, that's a bug that we should fix :)

… clouser

Non-third_party changes in bazelbuild#17112 Merging bazelbuild#17112

- Delete replaced jar files and update BUILD files. Merging bazelbuild#17112

- Non-third_party changes in #17112 Merging #17112 Stacking on #17142 Closes #17143. PiperOrigin-RevId: 500980364 Change-Id: Ie03d4bc8e167d8a27ed8b33cd90f2e3af2c472f6

meteorcloudy · 2023-01-10T15:19:38Z

Merged by #17142, #17143, #17144

- Non-third_party changes in #17112 Merging #17112 Stacking on #17142 Closes #17143. PiperOrigin-RevId: 500980364 Change-Id: Ie03d4bc8e167d8a27ed8b33cd90f2e3af2c472f6

@maven

This change is required to support Bazel's offline bootstrap build. More context in bazelbuild/bazel#17112 Instead of checking in jar files in Bazel's source tree, Bazel wants to use rules_jvm_external to fetch jars dependencies. However, to support Bazel's bootstrap build, we need to patch rules_jvm_external for vendoring the @maven repository. - Generate a BUILD.vendor file to be used in the vendored `@maven` repository. Added a jvm_import and a filegroup rule for each downloaded jar artifact. The filegroup rule is required by the bootstrap Java toolchain used in Bazel's bootstrap build. The bootstrap Java toolchain cannot depend on a jvm_import target. Because the jvm_import rule depends on a java_binary tool "AddJarManifestEntry", which requires a Java toolchain. Depending on the jar via a filegroup rule helps avoid this cyclic dependency. - Added a filegroup rule to collect all sources needed for vendoring `@maven`, including BUILD.vendor, WORKSPACE and jar files.

- Non-third_party changes in #17112 Merging #17112 Stacking on #17142 Closes #17143. PiperOrigin-RevId: 500980364 Change-Id: Ie03d4bc8e167d8a27ed8b33cd90f2e3af2c472f6 # Conflicts: # WORKSPACE # maven_install.json

Instead of checking in a precompiled jar, following the example of bazelbuild#17112

Instead of checking in a precompiled jar, following the example of #17112 Closes #19472. PiperOrigin-RevId: 565016180 Change-Id: Ie7f3e0d2c999e5bb97b278156c586f52bc951bab

Instead of checking in a precompiled jar, following the example of #17112 Partial commit for third_party/*, see #19472. Signed-off-by: Pavan Singh <[email protected]>

@maven

This change is required to support Bazel's offline bootstrap build. More context in bazelbuild/bazel#17112 Instead of checking in jar files in Bazel's source tree, Bazel wants to use rules_jvm_external to fetch jars dependencies. However, to support Bazel's bootstrap build, we need to patch rules_jvm_external for vendoring the @maven repository. - Generate a BUILD.vendor file to be used in the vendored `@maven` repository. Added a jvm_import and a filegroup rule for each downloaded jar artifact. The filegroup rule is required by the bootstrap Java toolchain used in Bazel's bootstrap build. The bootstrap Java toolchain cannot depend on a jvm_import target. Because the jvm_import rule depends on a java_binary tool "AddJarManifestEntry", which requires a Java toolchain. Depending on the jar via a filegroup rule helps avoid this cyclic dependency. - Added a filegroup rule to collect all sources needed for vendoring `@maven`, including BUILD.vendor, WORKSPACE and jar files. - Fix `testonly` attribute handling in Bzlmod. - Set `fetch_sources` to false by default in Bzlmod. - Fix `excluded_artifacts` in Bzlmod. Signed-off-by: kon72 <[email protected]>

meteorcloudy requested review from Wyverald and aiuto January 3, 2023 05:36

meteorcloudy requested review from comius, haxorz and ted-xie January 3, 2023 05:37

sgowroji added team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. awaiting-review PR is awaiting review from an assigned reviewer labels Jan 3, 2023

aiuto reviewed Jan 3, 2023

View reviewed changes

BUILD Show resolved Hide resolved

WORKSPACE Show resolved Hide resolved

third_party/BUILD Outdated Show resolved Hide resolved

rjobredeaux3 mentioned this pull request Jan 3, 2023

Update aosp_gradle_core in preparation for updating manifest merger to version 30.1.3 #17014

Closed

meteorcloudy removed the request for review from haxorz January 4, 2023 05:57

meteorcloudy requested review from fweikert and removed request for comius and ted-xie January 4, 2023 05:58

aiuto approved these changes Jan 4, 2023

View reviewed changes

Wyverald approved these changes Jan 4, 2023

View reviewed changes

meteorcloudy added a commit to meteorcloudy/bazel that referenced this pull request Jan 5, 2023

Use rules_jvm_external to fetch jars dependencies (part 1)

a12ae6a

- Add rules_jvm_external.patch Merging bazelbuild#17112

meteorcloudy added a commit to meteorcloudy/bazel that referenced this pull request Jan 5, 2023

Use rules_jvm_external to fetch jars dependencies (part 2)

db9eacf

Non-third_party changes in bazelbuild#17112 Merging bazelbuild#17112

meteorcloudy added a commit to meteorcloudy/bazel that referenced this pull request Jan 5, 2023

Use rules_jvm_external to fetch jars dependencies (part 3)

1d5d6c4

- Delete replaced jar files and update BUILD files. Merging bazelbuild#17112

This was referenced Jan 5, 2023

Use rules_jvm_external to fetch jars dependencies (part 1) #17142

Closed

Use rules_jvm_external to fetch jars dependencies (part 2) #17143

Closed

Use rules_jvm_external to fetch jars dependencies (part 3) #17144

Closed

jin reviewed Jan 5, 2023

View reviewed changes

Use rules_jvm_external to fetch jars dependencies

c182e3b

meteorcloudy added 8 commits January 5, 2023 15:40

Fix tests

d624d8a

Address reviewer comments and replace more jars in guava's transitive…

da808f9

… clouser

Replace guava-testlib as well

e2a8c8d

More fixes

6281f47

Remove guava related setup in compile.sh

0398b94

Fix verify_workspace.sh in a better way

d7bd1f4

Pin Maven dependencies

a5d28d4

Use rules_jvm_external also in the Bzlmod build

dbc9a8f

meteorcloudy force-pushed the rje_support branch from 7c9c053 to dbc9a8f Compare January 5, 2023 08:26

meteorcloudy added a commit to meteorcloudy/bazel that referenced this pull request Jan 5, 2023

Use rules_jvm_external to fetch jars dependencies (part 2)

d552b54

Non-third_party changes in bazelbuild#17112 Merging bazelbuild#17112

meteorcloudy added a commit to meteorcloudy/bazel that referenced this pull request Jan 5, 2023

Use rules_jvm_external to fetch jars dependencies (part 3)

ef28626

- Delete replaced jar files and update BUILD files. Merging bazelbuild#17112

meteorcloudy added 4 commits January 5, 2023 19:40

Make the vendored @maven repo work with pined jar dependencies

337ea09

Fix patch file

3a76b0f

more patch file fixes

ff63add

Fix bazel_determinism_test

2579300

jin approved these changes Jan 5, 2023

View reviewed changes

meteorcloudy mentioned this pull request Jan 10, 2023

Security Policy violation Binary Artifacts #17036

Closed

meteorcloudy closed this Jan 10, 2023

meteorcloudy mentioned this pull request Mar 23, 2023

Add targets to make it easier to vendor the @maven repository bazel-contrib/rules_jvm_external#877

Closed

cushon added a commit to cushon/bazel that referenced this pull request Sep 9, 2023

Use rules_jvm_external to fetch turbine

9c886d3

Instead of checking in a precompiled jar, following the example of bazelbuild#17112

cushon mentioned this pull request Sep 9, 2023

Use rules_jvm_external to fetch turbine #19472

Closed

cushon added a commit to cushon/bazel that referenced this pull request Sep 11, 2023

Use rules_jvm_external to fetch turbine

aaeeabc

Instead of checking in a precompiled jar, following the example of bazelbuild#17112


		load("@rules_jvm_external//:defs.bzl", "maven_install")

		maven_install(

Use rules_jvm_external to fetch jars dependencies #17112

Use rules_jvm_external to fetch jars dependencies #17112

Uh oh!

Conversation

meteorcloudy commented Jan 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

meteorcloudy commented Jan 3, 2023

Uh oh!

aiuto left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

haxorz commented Jan 3, 2023

Uh oh!

meteorcloudy commented Jan 4, 2023

Uh oh!

meteorcloudy commented Jan 4, 2023

Uh oh!

aiuto left a comment

Choose a reason for hiding this comment

Uh oh!

Wyverald left a comment

Choose a reason for hiding this comment

Uh oh!

meteorcloudy commented Jan 5, 2023

Uh oh!

meteorcloudy commented Jan 5, 2023

Uh oh!

jin commented Jan 5, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

meteorcloudy Jan 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

meteorcloudy commented Jan 10, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

meteorcloudy commented Jan 3, 2023 •

edited

Loading

meteorcloudy Jan 5, 2023 •

edited

Loading