You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As outlined in #447, TinyGo should support common serialization formats. While most people may think of JSON, I personally want TinyGo to support protocol buffers. This issue is meant to track my progress on adding Protocol Buffer support to TinyGo.
Design
The default Protocol Buffer package (https://pkg.go.dev/google.golang.org/protobuf) makes heavy use of reflection. TinyGo only supports a small amount of the reflect package. Heavy use of reflection leads to slow code, so the folks at PlanetScale wrote vtprotobuf, which generates explicit serialization / deserialization code. This code works today with TinyGo. However, it requires the existing, slow code to compile.
So the plan is simple: Support compiling the code generated by protoc-gen-go and running and of the code contained in init() functions. Use the code generated by vtprotobuf to serialize and deserialize the data.
Reacted by Yuki Ito, Marten Mooij, gitslav, 澄潭, Jared Weinfurtner, Alexandre Bourget, Jacob LeGrone, Philipp Keidel, Christian Stewart, Ukjae Jeong and 8 more
Reacted by Alexandre Bourget, Thomas Auinger and dyegoaurelio
@kyleconroy This is a great idea, I'm using vtprotobuf a lot currently and it works great. Having some kind of auto-transpiler that automatically converts the reflection-based protobuf-go to use vtprotobuf would be great.
Quick question: as of March 2023, it seems like most of the reflection features are supported - is there anything missing for the existing reflection version to work currently?
How are you able to use vtprotobuf with tinygo? Isn't the generated code from vtprotobuf depends on the result of protobuf-go? I am still getting the same error with my _vtproto.pb.go file.
How are you able to use vtprotobuf with tinygo? Isn't the generated code from vtprotobuf depends on the result of protobuf-go?
The reflection methods are stubbed out, so it still compiles and the vtprotobuf version doesn't invoke the reflection methods, so it works fine for me.
I am still getting the same error with my _vtproto.pb.go file.
Post the error and make sure you're on the latest tinygo version
I am on the latest tinygo v0.31.1, and I am also using it with wasmtime. The error I got is exactly the same with the one you posted before without using vtprotobuf. Did you change any of the code generated by protoc --go_out ?
vtprotobuf generates static code for Marshal and Unmarshal. We just need a version of protoc-gen-go that does not import reflect nor any of the "heavy" protobuf packages that add a lot of complexity to the resulting binary.
I have forked to protoc-gen-go-lite here and have already dropped everything but the protoc plugin, and am working on adjusting the compiler to generate bare structs without referencing reflect: aperturerobotics/protobuf-go-lite#1
Please join the effort there as this is intended to make it easier to build lightweight tinygo protobuf projects.
The vtprotobuf approach described in this issue now works. I tested vtprotobuf v0.6.0 with TinyGo 0.44.0-dev-eae3f875 on macOS/ARM64. A generated message containing integer, string, and byte fields successfully round-tripped through MarshalVT and UnmarshalVT, with all decoded values checked.
=== RUN TestVTRoundTrip
--- PASS: TestVTRoundTrip (0.00s)
PASS
For comparison, calling standard proto.Marshal still fails:
Activity
paralin commentedon Mar 4, 2023
@kyleconroy This is a great idea, I'm using vtprotobuf a lot currently and it works great. Having some kind of auto-transpiler that automatically converts the reflection-based protobuf-go to use vtprotobuf would be great.
Quick question: as of March 2023, it seems like most of the reflection features are supported - is there anything missing for the existing reflection version to work currently?
dgryski commentedon Mar 23, 2023
And even more reflect is working, including the last bug on you roadmap.
gitslav commentedon Jun 15, 2023
@kyleconroy @dgryski looking at the roadmap is it time for the confetti emoji?
rockwotj commentedon Nov 15, 2023
So it seems that the default protobuf compiler is not supported still even with the improvements to reflection. I have yet to test
vtprotobufrockwotj commentedon Nov 21, 2023
FWIW I have successfully used https://github.com/planetscale/vtprotobuf to marshal/unmarshal protobuf in tinygo -target=wasi
(this can be closed then?)
wrnlb666 commentedon Apr 8, 2024
How are you able to use vtprotobuf with tinygo? Isn't the generated code from vtprotobuf depends on the result of protobuf-go? I am still getting the same error with my
_vtproto.pb.gofile.rockwotj commentedon Apr 8, 2024
The reflection methods are stubbed out, so it still compiles and the vtprotobuf version doesn't invoke the reflection methods, so it works fine for me.
Post the error and make sure you're on the latest tinygo version
wrnlb666 commentedon Apr 9, 2024
I am on the latest tinygo v0.31.1, and I am also using it with wasmtime. The error I got is exactly the same with the one you posted before without using vtprotobuf. Did you change any of the code generated by
protoc --go_out?rockwotj commentedon Apr 9, 2024
no modifications needed. Can you post a reproduction?
wrnlb666 commentedon Apr 9, 2024
I am so sorry to bother u, it was me making a dumb mistake causing the error, I fixed the error and it works totally fine. Thank you so much.
paralin commentedon Apr 10, 2024
vtprotobuf generates static code for Marshal and Unmarshal. We just need a version of protoc-gen-go that does not import reflect nor any of the "heavy" protobuf packages that add a lot of complexity to the resulting binary.
I have forked to protoc-gen-go-lite here and have already dropped everything but the protoc plugin, and am working on adjusting the compiler to generate bare structs without referencing reflect: aperturerobotics/protobuf-go-lite#1
Please join the effort there as this is intended to make it easier to build lightweight tinygo protobuf projects.
protobuf-go-lite: keep protoc-gen-go, drop unused, fix linter errors
protobuf-go-lite: keep protoc-gen-go, drop unused, fix linter errors
protobuf-go-lite: keep protoc-gen-go, drop unused, fix linter errors
protobuf-go-lite: keep protoc-gen-go, drop unused, fix linter errors
protobuf-go-lite: keep protoc-gen-go, drop unused, fix linter errors
protobuf-go-lite: keep protoc-gen-go, drop unused, fix linter errors
paralin commentedon Apr 11, 2024
I have completed protobuf-go-lite which is a reflection-free fork of protobuf-go merged with vtprotobuf and protoc-gen-go-json.
Library: https://github.com/aperturerobotics/protobuf-go-lite
Example: https://github.com/aperturerobotics/protobuf-project/tree/protobuf-lite
RPCs are available as well with starpc: https://github.com/aperturerobotics/starpc
protobuf-go-lite now supports reflection-free protobuf and protojson encoding!
Merge protobuf-go and vtprotobuf and remove reflection dependency
refactor: switch to protobuf-go-lite
feat: implement features/json merging in protoc-gen-go-json
davecheney commentedon Oct 28, 2026
The vtprotobuf approach described in this issue now works. I tested vtprotobuf v0.6.0 with TinyGo
0.44.0-dev-eae3f875on macOS/ARM64. A generated message containing integer, string, and byte fields successfully round-tripped throughMarshalVTandUnmarshalVT, with all decoded values checked.For comparison, calling standard
proto.Marshalstill fails:Can we close this as completed for the original roadmap of importing generated protobuf code and using vtprotobuf for serialization?