Add standalone protoc plugin binaries · Pull Request #13308 · protocolbuffers/protobuf

ghost · 2023-07-13T17:57:13Z

This PR adds targets to build protoc plugins for all the built-in plugins. This will result in the following binaries being available:

protoc-gen-cpp
protoc-gen-csharp
protoc-gen-java
protoc-gen-kotlin
protoc-gen-objectivec
protoc-gen-php
protoc-gen-python
protoc-gen-pyi
protoc-gen-ruby
protoc-gen-rust

The goal of this PR is for these to be added as assets to each GitHub release, although we are unable to find the code that builds the assets for releases - if it is in this repository, let us know and we're happy to do any work required!

There's multiple motivations for this PR:

Optimally, people want to version their plugins separately from their compiler. That is, the version of protoc should be decoupled from the version of each plugin, as is the case with the rest of the Protobuf ecosystem outside of the built-in plugins. You should be able to, for example, take advantage of fixes in protoc v24.0 whilst continuing to use the Java generated code from v23.0, in case there are breaking changes you're unable to adopt yet.
This partially addresses protoc should not populate json_name in descriptor if no json_name was specified #5587 - the core motivation behind this issue is that when piping a FileDescriptorSet into protoc with --descriptor_set_in, and then invoking one of the built-in plugins, different code can result, as the value for json_name will be subtly different when taking an external FileDescriptorSet vs protoc producing one internally and then invoking one of the built-in plugins.
Selfishly, it makes life a lot easier for alternative Protobuf compilers. For example, for us with buf, to use the built-in plugins, we have a convoluted mechanism to proxy to protoc that cannot deal with issue protoc should not populate json_name in descriptor if no json_name was specified #5587, and requires people to install both protoc and buf when they may just want one of the built-in plugins. Having separate, independently-versionable plugins would make our lives a lot easier.

The majority of this PR is something we have already been maintaining for our remote plugins feature within Buf, we're just hoping we can get this upstreamed, as we think it would benefit the Protobuf community.

Follow-ups that would be nice:

Adding to GitHub releases, as mentioned.
Get these plugins installed as part of the Homebrew protobuf formula. There's precedent for this with i.e. the grpc formula, which installs each language-specific plugin.
Change protoc to default to using an on-disk plugin vs a built-in plugin, that is if protoc-gen-cpp exists, use protoc-gen-cpp over the builtin cpp plugin. This allows people to independently version plugins as mentioned, and mirrors the behavior of the Well-Known Types (if a WKT is in the include path, it is used instead of the WKTs included as part of the protoc distribution). We're happy to put up a PR for this as well.

google-cla · 2023-07-13T17:57:18Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

ghost · 2023-07-13T18:14:04Z

CLA has been signed.

thomasvl

For objc changes

fowles · 2023-07-18T14:32:56Z

+#include "google/protobuf/compiler/plugin.h"
+
+int main(int argc, char *argv[]) {
+  google::protobuf::compiler::java::KotlinGenerator generator;


doesn't this also need the JavaGenerator to be useful?

Sorry, could you please elaborate if possible? Right now I have a separate entrypoint for Java in src/google/protobuf/compiler/java/main.cc, and this entrypoint here using only KotlinGenerator matches what we already do to make an unofficial protoc-gen-kotlin.

I have two distinct directions of concern/thought here.

I am imagining a hypothetical user that has installed protoc-gen-kotlin. They are able to generate kotlin code, but that code depends on a java library that they are unable to generate. Seems like not a great situation.

A user may have different versions of protoc-gen-kotlin and protoc-gen-java. I literally don't know if their generated kotlin stubs will work. I am not certain how tight the coupling is between these two and don't relish finding out the hard way.

A user may have different versions of protoc-gen-kotlin and protoc-gen-java. I literally don't know if their generated kotlin stubs will work. I am not certain how tight the coupling is between these two and don't relish finding out the hard way.

I don't think this is any different than the rest of the Protobuf ecosystem though - you could have different versions of dependent plugins, say a Protobuf and grpc plugin, and it may or may not cause an issue. I can see the integration here may be tighter, but I'd think that it's an orthogonal issue. I'd love to have a way to declare library and plugin dependencies for plugins, but it's not part of the plugin API at the moment

(1) is genuinely different, we don't have a way to end up in this state right now.

(2) has existing issues, but they require having two distinct full copies of protoc installed, which is not nearly as easy as having two different binaries installed.

Perhaps we're talking about two different things here - my argument is that neither (1) or (2) is any different from the rest of the plugin ecosystem, and I don't know if there's a reason to special case kotlin and Java. As an example, if you generate code with protoc-gen-go-grpc, you need generated code from protoc-gen-go, but if you don't have protoc-gen-go installed, you can't generate it.

If there's a kotlin/Java-specific solution to this though, that's great. But I think my argument is: massively in favor of dependencies between plugins (we actually represent the dependencies in BSR), but not sure builtins are any different than the rest of the ecosystem, and not sure if this is the place to have special-case guardrails. But I may be missing something massive here (wouldn't be the first time), apologies if so.

fowles · 2023-07-18T16:56:05Z

Optimally, people want to version their plugins separately from their compiler. That is, the version of protoc should be decoupled from the version of each plugin, as is the case with the rest of the Protobuf ecosystem outside of the built-in plugins. You should be able to, for example, take advantage of fixes in protoc v24.0 whilst continuing to use the Java generated code from v23.0, in case there are breaking changes you're unable to adopt yet.

Can you clarify if you are referring to the same language in v23 and v24 here, or do you mean something like "you want v24 java, but v23 C++"?

bufdev · 2023-07-18T17:18:07Z

Sorry, should have clarified here:

I don't think this PR is intending to make it so that v23 java and v24 java can be generated into the same codebase, that's a separate issue entirely heh.
It would be nice to easily be able to do v23 java and v24 c++ in the same monorepo as an example, something that is more difficult now if you have a single point of entry for protoc.
It would also be nice if the core compiler functionality of protoc would be able to work with older plugins, so for example there's a bug fix in the compiler of protoc in v24, but I continue to use protoc-gen-java v23.

Does that make sense?

fowles · 2023-09-08T15:40:37Z

Sorry this has lingered for so long. There are some internal things that are moving around that make this direction look really promising, so odds are strong we will take this eventually, but moving the behemoth takes time :)

github-actions · 2023-12-08T10:02:09Z

We triage inactive PRs and issues in order to make it easier to find active work. If this PR should remain active, please add a comment.

This PR is labeled inactive because the last activity was over 90 days ago. This PR will be closed and archived after 14 additional days without activity.

github-actions · 2023-12-23T10:02:11Z

We triage inactive PRs and issues in order to make it easier to find active work. If this PR should remain active or becomes active again, please reopen it.

This PR was closed and archived because there has been no new activity in the 14 days since the inactive label was added.

ghost · 2024-01-02T17:48:52Z

Oops, this PR probably shouldn't be closed. I could of course just open a new PR later, but the relevant discussion is probably worth keeping.

github-actions · 2025-01-10T10:02:51Z

We triage inactive PRs and issues in order to make it easier to find active work. If this PR should remain active, please add a comment.

This PR is labeled inactive because the last activity was over 90 days ago. This PR will be closed and archived after 14 additional days without activity.

ghost · 2025-01-15T20:57:05Z

Still relevant, needs to be rebased though.

JasonLunn · 2025-02-03T22:49:31Z

@jchadwick-buf - can you rebase this PR?

ghost · 2025-03-26T23:22:19Z

Sorry for the delay. I have been a bit busy.

I've rebased. I see the Kotlin generator moved but otherwise not much needed to be changed, seems to be building. Please let me know if there's anything else that needs to be done.

JasonLunn · 2025-03-27T14:14:37Z

Looks close! The last test run looks like it only failed due to comparison with goldens:

Installed files do not match goldens!
If this is expected, please update the golden file: cmake/installed_bin_golden.txt
The following diffs were found:
--- cmake/installed_bin_golden.txt	2025-03-26 23:42:40.086630065 +0000
+++ static/bin.txt	2025-03-26 23:42:41.302627899 +0000
@@ -1,3 +1,13 @@
 protoc
+protoc-gen-cpp
+protoc-gen-csharp
+protoc-gen-java
+protoc-gen-kotlin
+protoc-gen-objectivec
+protoc-gen-php
+protoc-gen-pyi
+protoc-gen-python
+protoc-gen-ruby
+protoc-gen-rust
 protoc-gen-upb
 protoc-gen-upbdefs

ghost · 2025-04-02T17:13:44Z

Thanks, I've applied those changes.

ghost · 2025-04-04T16:38:49Z

I see that the CL failed internally. Let me know if there's anything I should do to help make this pass in google3; otherwise, I'll cross my fingers that it isn't anything too ridiculous and leave it to y'all.

This PR adds targets to build `protoc` plugins for all the built-in plugins. This will result in the following binaries being available: - `protoc-gen-cpp` - `protoc-gen-csharp` - `protoc-gen-java` - `protoc-gen-kotlin` - `protoc-gen-objectivec` - `protoc-gen-php` - `protoc-gen-python` - `protoc-gen-pyi` - `protoc-gen-ruby` - `protoc-gen-rust` The goal of this PR is for these to be added as assets to each GitHub release, although we are unable to find the code that builds the assets for releases - if it is in this repository, let us know and we're happy to do any work required! There's multiple motivations for this PR: - Optimally, people want to version their plugins separately from their compiler. That is, the version of `protoc` should be decoupled from the version of each plugin, as is the case with the rest of the Protobuf ecosystem outside of the built-in plugins. You should be able to, for example, take advantage of fixes in `protoc` v24.0 whilst continuing to use the Java generated code from v23.0, in case there are breaking changes you're unable to adopt yet. - This partially addresses #5587 - the core motivation behind this issue is that when piping a `FileDescriptorSet` into `protoc` with `--descriptor_set_in`, and then invoking one of the built-in plugins, different code can result, as the value for `json_name` will be subtly different when taking an external `FileDescriptorSet` vs `protoc` producing one internally and then invoking one of the built-in plugins. - Selfishly, it makes life a lot easier for alternative Protobuf compilers. For example, for us with `buf`, to use the built-in plugins, we have a convoluted [mechanism to proxy to protoc](https://github.com/bufbuild/buf/blob/main/private/bufpkg/bufpluginexec/protoc_proxy_handler.go) that cannot deal with issue #5587, and requires people to install both `protoc` and `buf` when they may just want one of the built-in plugins. Having separate, independently-versionable plugins would make our lives a lot easier. The majority of this PR is something [we have already been maintaining](https://github.com/bufbuild/plugins/tree/main/plugins/protocolbuffers/cpp/v23.4) for our remote plugins feature within Buf, we're just hoping we can get this upstreamed, as we think it would benefit the Protobuf community. Follow-ups that would be nice: - Adding to GitHub releases, as mentioned. - Get these plugins installed as part of the Homebrew `protobuf` formula. There's precedent for this with i.e. the `grpc` formula, which installs each language-specific plugin. - Change `protoc` to default to using an on-disk plugin vs a built-in plugin, that is if `protoc-gen-cpp` exists, use `protoc-gen-cpp` over the builtin `cpp` plugin. This allows people to independently version plugins as mentioned, and mirrors the behavior of the Well-Known Types (if a WKT is in the include path, it is used instead of the WKTs included as part of the `protoc` distribution). We're happy to put up a PR for this as well. Closes #13308 COPYBARA_INTEGRATE_REVIEW=#13308 from jchadwick-buf:standalone-plugins 3f78f87 FUTURE_COPYBARA_INTEGRATE_REVIEW=#13308 from jchadwick-buf:standalone-plugins 3f78f87 PiperOrigin-RevId: 743624811

Follow-up to #23736 and #13308 but with only Bazel targets added per feedback. This PR adds targets to build protoc plugins for all the built-in plugins. This will result in the following binaries being available: - protoc-gen-cpp - protoc-gen-csharp - protoc-gen-java - protoc-gen-kotlin - protoc-gen-objectivec - protoc-gen-php - protoc-gen-python - protoc-gen-pyi - protoc-gen-ruby - protoc-gen-rust Closes #23919 COPYBARA_INTEGRATE_REVIEW=#23919 from bufdev:standalone-plugins 45d44f6 PiperOrigin-RevId: 917373434

ghost marked this pull request as ready for review July 13, 2023 18:06

ghost requested review from a team as code owners July 13, 2023 18:06

ghost requested review from deannagarcia and removed request for a team July 13, 2023 18:06

perezd requested a review from fowles July 13, 2023 18:08

mkruskal-google added the 🅰️ safe for tests Mark a commit as safe to run presubmits over label Jul 13, 2023

github-actions Bot removed the 🅰️ safe for tests Mark a commit as safe to run presubmits over label Jul 13, 2023

thomasvl approved these changes Jul 13, 2023

View reviewed changes

fowles reviewed Jul 18, 2023

View reviewed changes

fowles requested a review from zhangskz July 18, 2023 14:34

github-actions Bot added the inactive Denotes the issue/PR has not seen activity in the last 90 days. label Dec 8, 2023

github-actions Bot closed this Dec 23, 2023

fowles reopened this Jan 2, 2024

fowles removed the request for review from a team January 2, 2024 18:06

github-actions Bot removed the inactive Denotes the issue/PR has not seen activity in the last 90 days. label Oct 2, 2024

JasonLunn added protoc wait for user action labels Oct 11, 2024

github-actions Bot added the inactive Denotes the issue/PR has not seen activity in the last 90 days. label Jan 10, 2025

github-actions Bot removed the inactive Denotes the issue/PR has not seen activity in the last 90 days. label Jan 16, 2025

Add standalone protoc plugin binaries

a70e667

JasonLunn added 🅰️ safe for tests Mark a commit as safe to run presubmits over and removed wait for user action labels Mar 26, 2025

github-actions Bot removed the 🅰️ safe for tests Mark a commit as safe to run presubmits over label Mar 26, 2025

JasonLunn added the wait for user action label Mar 27, 2025

Add standalone plugins to installed_bin_golden.txt

3f78f87

JasonLunn added 🅰️ safe for tests Mark a commit as safe to run presubmits over and removed wait for user action labels Apr 3, 2025

github-actions Bot removed the 🅰️ safe for tests Mark a commit as safe to run presubmits over label Apr 3, 2025

googleberg approved these changes Apr 3, 2025

View reviewed changes

copybara-service Bot mentioned this pull request Apr 4, 2025

Add standalone protoc plugin binaries (#13308) #21101

Closed

ghost closed this by deleting the head repository Sep 3, 2025

srikrsna-buf mentioned this pull request Oct 1, 2025

Add standalone protoc plugin binaries #23736

Closed

bufdev mentioned this pull request Oct 10, 2025

Add bazel targets for standalone plugin binaries #23919

Closed

loci-dev mentioned this pull request Feb 11, 2026

UPSTREAM PR #23919: Add bazel targets for standalone plugin binaries auroralabs-loci/protobuf#10

Open

This pull request was closed.

Uh oh!

Conversation

ghost commented Jul 13, 2023

Uh oh!

google-cla Bot commented Jul 13, 2023

Uh oh!

ghost commented Jul 13, 2023

Uh oh!

thomasvl left a comment

Choose a reason for hiding this comment

Uh oh!

fowles Jul 18, 2023

Choose a reason for hiding this comment

Uh oh!

ghost Jul 18, 2023

Choose a reason for hiding this comment

Uh oh!

fowles Jul 18, 2023

Choose a reason for hiding this comment

Uh oh!

bufdev Jul 18, 2023

Choose a reason for hiding this comment

Uh oh!

fowles Jul 18, 2023

Choose a reason for hiding this comment

Uh oh!

bufdev Jul 18, 2023

Choose a reason for hiding this comment

Uh oh!

fowles commented Jul 18, 2023

Uh oh!

bufdev commented Jul 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fowles commented Sep 8, 2023

Uh oh!

github-actions Bot commented Dec 8, 2023

Uh oh!

github-actions Bot commented Dec 23, 2023

Uh oh!

ghost commented Jan 2, 2024

Uh oh!

github-actions Bot commented Jan 10, 2025

Uh oh!

ghost commented Jan 15, 2025

Uh oh!

JasonLunn commented Feb 3, 2025

Uh oh!

ghost commented Mar 26, 2025

Uh oh!

JasonLunn commented Mar 27, 2025

Uh oh!

ghost commented Apr 2, 2025

Uh oh!

ghost commented Apr 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

bufdev commented Jul 18, 2023 •

edited

Loading