Skip to content
This repository was archived by the owner on Oct 31, 2025. It is now read-only.

feat!: proto3 JSON serializer and deserializer#2

Merged
alexander-fenster merged 5 commits intomainfrom
code
Aug 3, 2021
Merged

feat!: proto3 JSON serializer and deserializer#2
alexander-fenster merged 5 commits intomainfrom
code

Conversation

@alexander-fenster
Copy link
Copy Markdown
Contributor

@alexander-fenster alexander-fenster commented Aug 2, 2021

And this is the real code to be reviewed.

The implementation is in typescript/src, tests are in typescript/test. The implementation of both the serializer and the deserializer is basically recursive, it's the regular traversing of the tree (the JSON tree for the deserializer, and the protobuf object for the serializer), and then I just have a separate pair of functions to serialize/deserialize any proto3 type that has a special JSON representation according to the spec linked below.

README.md describes the idea and has all the samples :) I'll just copy the main part of it here.

proto3 JSON serializer for TypeScript / JavaScript

This library implements proto3 JSON serialization and deserialization for protobuf.js protobuf objects according to the spec.

Note that the spec requires special representation of some google.protobuf.* types (Value, Struct, Timestamp, Duration, etc.), so you cannot just use .toObject() since the result won't be understood by protobuf in other languages. Hence this module.

JavaScript:

const serializer = require('proto3-json-serializer');

TypeScript:

import * as serializer from 'proto3-json-serializer';

Serialization: protobuf.js object to proto3 JSON

const root = protobuf.loadSync('test.proto');
const Type = root.lookupType('test.Message');
const message = Type.fromObject({...});

const serialized = serializer.toProto3JSON(message);

Serialization works with any object created by calling .create(), .decode(), or .fromObject() for a loaded protobuf type. It relies on the $type field so it will not work with a static object.

Deserialization: proto3 JSON to protobuf.js object

To deserialize an object from proto3 JSON, we must know its type (as returned by root.lookupType('...')). Pass this type as the first parameter to .fromProto3JSON:

const root = protobuf.loadSync('test.proto');
const Type = root.lookupType('test.Message');
const json = {...};

const deserialized = serializer.fromProto3JSON(Type, json);

Complete example

const assert = require('assert');
const path = require('path');
const protobuf = require('protobufjs');
const serializer = require('proto3-json-serializer');

// We'll take sample protos from google-proto-files but the code will work with any protos
const protos = require('google-proto-files');

// Load some proto file
const rpcProtos = protos.getProtoPath('rpc');
const root = protobuf.loadSync([
    path.join(rpcProtos, 'status.proto'),
    path.join(rpcProtos, 'error_details.proto'),
]);
const Status = root.lookupType('google.rpc.Status');

// If you have a protobuf object that follows proto3 JSON syntax
// https://developers.google.com/protocol-buffers/docs/proto3#json
// (this is an example of google.rpc.Status message in JSON)
const json = {
    code: 3,
    message: 'Test error message',
    details: [
        {
            '@type': 'google.rpc.BadRequest',
            fieldViolations: [
                {
                    field: 'field',
                    description: 'must not be null',
                },
            ],
        },
    ],
};

// You can deserialize it into a protobuf.js object:
const deserialized = serializer.fromProto3JSON(Status, json);
console.log(deserialized);

// And serialize it back
const serialized = serializer.toProto3JSON(deserialized);
assert.deepStrictEqual(serialized, json);

@alexander-fenster alexander-fenster requested review from a team, bcoe, sofisl and summer-ji-eng August 2, 2021 22:54
@generated-files-bot
Copy link
Copy Markdown

Warning: This pull request is touching the following templated files:

@google-cla google-cla Bot added the cla: yes This human has signed the Contributor License Agreement. label Aug 2, 2021
@snippet-bot
Copy link
Copy Markdown

snippet-bot Bot commented Aug 2, 2021

Here is the summary of possible violations 😱

Details

There is a possible violation for not having product prefix.

The end of the violation section. All the stuff below is FYI purposes only.


Here is the summary of changes.

You are about to add 1 region tag.

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

  • Refresh this comment

Copy link
Copy Markdown

@bcoe bcoe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I'm understanding correctly, this will allow us to reduce the amount of nesting that users of GAPIC libraries need to use when interacting with complex APIs, like ai-platform.

Presumably this will eventually require a breaking change for libraries?

Comment thread package.json Outdated
Comment thread package.json
"gts": "^3.1.0",
"jsdoc": "^3.6.7",
"jsdoc-fresh": "^1.1.0",
"jsdoc-region-tag": "^1.3.0",
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need jsdoc-region-tag for this library? It handles edge-cases in JSDoc parsing, e.g., snippet region tags.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure, I just copied it :)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(but I do have one region tag in quickstart.js)


function testGoogleProtobufStruct(root: protobuf.Root) {
const MessageWithStruct = root.lookupType('test.MessageWithStruct');
const message = MessageWithStruct.fromObject({
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, if I'm understanding, we're going to be able to represent these deeply nested structures much more elegantly for users, e.g., the parser turns:

    structField: {
      fields: {
        stringField: {
          stringValue: 'test',
        },
        numberField: {
          numberValue: 42,
        },
      },
    },
  });

Into:

 const json = {
    structField: {
      stringField: 'test',
      numberField: 42,
    },
  };

Copy link
Copy Markdown
Contributor Author

@alexander-fenster alexander-fenster Aug 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's right. We are not yet enabling this for libraries, for now we just need it for the transport layer in REGAPIC since the HTTP endpoint expects this short form rather than the long one, so gax should be able to read and write this short form JSON.

Enabling this for libraries is (1) breaking, (2) will require .d.ts updates to follow the short structure, so it's a separate story. But even now, aiplatform could start using this library instead of using their own conversion code.

@alexander-fenster alexander-fenster added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 3, 2021
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 3, 2021
@alexander-fenster alexander-fenster merged commit 96255a7 into main Aug 3, 2021
@alexander-fenster alexander-fenster deleted the code branch August 3, 2021 21:25
@release-please release-please Bot mentioned this pull request Aug 3, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

cla: yes This human has signed the Contributor License Agreement.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants