BQ Write: prototype derive macro for proto client #7088

Objective

Offer customers higher write throughput to BQ without having to deal with the ugliness that is protobuf.

Background

We have the google-cloud-bigquery-derive crate (under src/bigquery-derive) which defines FromSql and FromRow macros.

Applications can annotate their structs, and the library knows how to convert between our representation of Rows (SqlValue) and the application's struct (including conversions for datatypes like wkt::Timestamp which are represented as Strings).

It would be awesome if we had a similar derive macro for the Write client to convert an application's struct to protobuf.

Details

Ultimately what we need is to take the definition of a struct and represent it as a crate::model::ProtoDescriptor. And to take a concrete struct and convert it to a byte::Bytes. (I think. I think crate::model::ProtoRows is for when you have >1 rows). This conversion needs to handle the nuances of the datatypes (like encoding wkt::Timestamp as a string).

So to sketch, we probably want a trait like:

pub trait ToSql {
  fn schema() -> ProtoDescriptor;
  fn to_row(self) -> Result<bytes::Bytes>;
}

and the derive macro generates the implementation of this trait for given structs.

Things to look out for...

In order to implement to_row() we probably need to use prost. We ideally do not want to expose this anywhere in the public API. We should also be concerned.

We should ideally not use dynamic proto messages. (Like we should not dynamically create a proto message on every call to to_row(). I think that would be expensive.

Applications can annotate their structs with #[prost::Message]. This is close, but I am not sure it handles the conversions for custom structs. If I had to guess, the solution for the impl looks like:

#[google_cloud_bigquery::ToSql]
struct Foo {
  a: i64,
  b: wkt::Timestamp,
}

// Generates code like:

#[prost::Message]
struct __google_cloud_bigquery_Foo {
  a: i64,
  b: String,
}

impl __google_cloud_bigquery_Foo {
  fn try_from(v: Foo) -> Result<Self> {
    Self {
      a: v.a,
      // TODO : define the `to_sql()` function for all primitive types?
      // Basically turning a `wkt::Timestamp` into a `String`.
      b: v.b.to_sql()?,
    }
  }
}

impl ToSql for Foo {
  fn schema() -> todo!(),
  fn to_sql(self) -> Result<bytes::Bytes> {
    // convert to our __google_cloud_bigquery_Foo
    // encode that type, because it is a prost::Message
  }
}

How reliant is this implementation on prost? Can prost be swapped out for google's protobuf in the future without breaking applications? What does that look like?

Testing

We will want an integration test similar to this that encompasses all the BQ data types.

For an implementation plan, I would start with the basics.

• Can we make this thing work for a struct with a String field. Then add in an int field. Then add in a wkt::Timestamp.
• The proto surface is currently hidden. For local development, make build_proto() in src/bigquery/src/write/writer_builder.rs pub, so we can use it in the integration tests crate.