Skip to content

Serialize and deserialize protocol buffers #2667

Open

Description

@kyleconroy

As outlined in #447, TinyGo should support common serialization formats. While most people may think of JSON, I personally want TinyGo to support protocol buffers. This issue is meant to track my progress on adding Protocol Buffer support to TinyGo.

Design

The default Protocol Buffer package (https://pkg.go.dev/google.golang.org/protobuf) makes heavy use of reflection. TinyGo only supports a small amount of the reflect package. Heavy use of reflection leads to slow code, so the folks at PlanetScale wrote vtprotobuf, which generates explicit serialization / deserialization code. This code works today with TinyGo. However, it requires the existing, slow code to compile.

So the plan is simple: Support compiling the code generated by protoc-gen-go and running and of the code contained in init() functions. Use the code generated by vtprotobuf to serialize and deserialize the data.

Roadmap

Activity

  1. paralin commented on Mar 4, 2023

    @paralin
    Contributor

    @kyleconroy This is a great idea, I'm using vtprotobuf a lot currently and it works great. Having some kind of auto-transpiler that automatically converts the reflection-based protobuf-go to use vtprotobuf would be great.

    Quick question: as of March 2023, it seems like most of the reflection features are supported - is there anything missing for the existing reflection version to work currently?

  2. dgryski commented on Mar 23, 2023

    @dgryski
    Member

    And even more reflect is working, including the last bug on you roadmap.

  3. gitslav commented on Jun 15, 2023

    @gitslav
    Contributor

    @kyleconroy @dgryski looking at the roadmap is it time for the confetti emoji?

  4. rockwotj commented on Nov 15, 2023

    @rockwotj
    Contributor

    So it seems that the default protobuf compiler is not supported still even with the improvements to reflection. I have yet to test vtprotobuf

    % ./wasmtime-v14.0.4-x86_64-linux/wasmtime run protobuf-example.wasm
    panic: unimplemented: (reflect.Type).MethodByName()
    Error: failed to run main module `protobuf-example.wasm`
    
    Caused by:
        0: failed to invoke command default
        1: error while executing at wasm backtrace:
               0: 0x5a15 - <unknown>!runtime._panic
               1: 0x5939 - <unknown>!(reflect.rawType).MethodByName
               2: 0x6484 - <unknown>!(*reflect.rawType).MethodByName
               3: 0x103253 - <unknown>!(*google.golang.org/protobuf/internal/impl.MessageInfo).makeStructInfo
               4: 0xb95e6 - <unknown>!(*google.golang.org/protobuf/internal/impl.MessageInfo).init
               5: 0xb8e02 - <unknown>!google.golang.org/protobuf/proto.protoMethods
               6: 0x34c67 - <unknown>!(google.golang.org/protobuf/proto.MarshalOptions).marshal
               7: 0x2d45a - <unknown>!runtime.run$1
               8: 0x2cf76 - <unknown>!runtime.run$1$gowrapper
               9:  0xdc1 - <unknown>!tinygo_launch
              10: 0x2ce48 - <unknown>!_start
           note: using the `WASMTIME_BACKTRACE_DETAILS=1` environment variable may show more debugging information
        2: wasm trap: wasm `unreachable` instruction executed
    
  5. rockwotj commented on Nov 21, 2023

    @rockwotj
    Contributor

    FWIW I have successfully used https://github.com/planetscale/vtprotobuf to marshal/unmarshal protobuf in tinygo -target=wasi

    (this can be closed then?)

  6. wrnlb666 commented on Apr 8, 2024

    @wrnlb666

    How are you able to use vtprotobuf with tinygo? Isn't the generated code from vtprotobuf depends on the result of protobuf-go? I am still getting the same error with my _vtproto.pb.go file.

  7. rockwotj commented on Apr 8, 2024

    @rockwotj
    Contributor

    How are you able to use vtprotobuf with tinygo? Isn't the generated code from vtprotobuf depends on the result of protobuf-go?

    The reflection methods are stubbed out, so it still compiles and the vtprotobuf version doesn't invoke the reflection methods, so it works fine for me.

    I am still getting the same error with my _vtproto.pb.go file.

    Post the error and make sure you're on the latest tinygo version

  8. wrnlb666 commented on Apr 9, 2024

    @wrnlb666

    I am on the latest tinygo v0.31.1, and I am also using it with wasmtime. The error I got is exactly the same with the one you posted before without using vtprotobuf. Did you change any of the code generated by protoc --go_out ?

  9. rockwotj commented on Apr 9, 2024

    @rockwotj
    Contributor

    no modifications needed. Can you post a reproduction?

  10. wrnlb666 commented on Apr 9, 2024

    @wrnlb666

    I am so sorry to bother u, it was me making a dumb mistake causing the error, I fixed the error and it works totally fine. Thank you so much.

  11. paralin commented on Apr 10, 2024

    @paralin
    Contributor

    vtprotobuf generates static code for Marshal and Unmarshal. We just need a version of protoc-gen-go that does not import reflect nor any of the "heavy" protobuf packages that add a lot of complexity to the resulting binary.

    I have forked to protoc-gen-go-lite here and have already dropped everything but the protoc plugin, and am working on adjusting the compiler to generate bare structs without referencing reflect: aperturerobotics/protobuf-go-lite#1

    Please join the effort there as this is intended to make it easier to build lightweight tinygo protobuf projects.

  12. paralin commented on Apr 11, 2024

    @paralin
    Contributor

    I have completed protobuf-go-lite which is a reflection-free fork of protobuf-go merged with vtprotobuf and protoc-gen-go-json.

    Library: https://github.com/aperturerobotics/protobuf-go-lite
    Example: https://github.com/aperturerobotics/protobuf-project/tree/protobuf-lite

    RPCs are available as well with starpc: https://github.com/aperturerobotics/starpc

    protobuf-go-lite now supports reflection-free protobuf and protojson encoding!

  13. davecheney commented on Oct 28, 2026

    @davecheney
    Contributor

    The vtprotobuf approach described in this issue now works. I tested vtprotobuf v0.6.0 with TinyGo 0.44.0-dev-eae3f875 on macOS/ARM64. A generated message containing integer, string, and byte fields successfully round-tripped through MarshalVT and UnmarshalVT, with all decoded values checked.

    === RUN   TestVTRoundTrip
    --- PASS: TestVTRoundTrip (0.00s)
    PASS
    

    For comparison, calling standard proto.Marshal still fails:

    panic: unimplemented: (reflect.Type).MethodByName()
    

    Can we close this as completed for the original roadmap of importing generated protobuf code and using vtprotobuf for serialization?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions