feat: adapt package to convert TableSchema to ProtoDescriptor by alvarowolfx · Pull Request #326 · googleapis/nodejs-bigquery-storage

alvarowolfx · 2023-02-16T22:15:10Z

This sub package goal is to allow users to convert BigQuery Schema/Table Schema to protobuf Descriptors and then convert plain JSON objects to protobuf, making it easier to use with the Storage Write API and not depending on protofiles for sending data.

Example:

import {adapt} from '@google-cloud/bigquery-storage')';
const {Type} = require('protobufjs');
require('protobufjs/ext/descriptor');

const protoDescriptor =
adapt.convertStorageSchemaToProto2Descriptor(
  {
    fields: [
      {
        name: 'foo',
        type: TableFieldSchema.Type.STRING,
        mode: TableFieldSchema.Mode.NULLABLE,
      },
      {
        name: 'bar',
        type: TableFieldSchema.Type.DOUBLE,
        mode: TableFieldSchema.Mode.REQUIRED,
      },
      {
        name: 'baz',
        type: TableFieldSchema.Type.STRING,
        mode: TableFieldSchema.Mode.REPEATED,
      },
      {
        name: 'bat',
        type: TableFieldSchema.Type.BOOL,
        mode: TableFieldSchema.Mode.REPEATED,
      },
    ],
  },
  'Test'
);
const TestProto = Type.fromDescriptor(protoDescriptor);
const raw = {
  foo: 'name',
  bar: 42,
  baz: ['A', 'B'],
  bat: [true, false],
};
const serialized = TestProto.encode(raw).finish();
const decoded = TestProto.decode(serialized).toJSON();
});

snippet-bot · 2023-02-22T21:41:54Z

Here is the summary of changes.

You are about to add 1 region tag.

samples/append_rows_table_to_proto2.js:22, tag bigquerystorage_jsonstreamwriter_pending

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

Refresh this comment

feywind

Cool, sorry for leaving so many comments :D but hopefully there's something useful in there. I don't see anything worth holding it back, but I'd definitely check out the behaviour on the circular package dependencies that Seth mentioned.

feywind · 2023-03-02T18:03:18Z

+    const root = Root.fromJSON(namespace);
+    const SampleData = root.lookupType('SampleData');
+
+    /**


Should these go closer to the snippet open tag so it's more visible to users copying and pasting?

feywind · 2023-03-02T18:08:40Z

+  }
+  // [END bigquerystorage_write_pending_complexschema]
+  appendRowsProto2();
+}


As a user I kind of love the comprehensive nature of this sample, but I also worry that it's really long. I wonder if we want to have more than one for smaller aspects of the process. (Not really a comment on your stuff so much as thinking out loud in regards to snippet tags...)

feywind · 2023-03-02T22:34:47Z

+    if (field.typeName) {
+      type = field.typeName;
+    } else {
+      type = fieldTypeLabelMap[field.type].replace('TYPE_', '').toLowerCase();


I was reminded the other day, in case it matters, that replace(string, string) will only replace the first occurrence. You'd need to use /TYPE_/g to get all of them. (not sure if that applies here)

feywind · 2023-03-02T22:37:18Z

+  return useProto3
+    ? bqModeToFieldLabelMapProto3[mode]
+    : bqModeToFieldLabelMapProto2[mode];
+}


Overall comments, it might be good to go back and put in some jsdoc/tsdoc comments describing what the functions do. Or at least some basic non-doc comments. I'd also consider thinking about breaking this up into multiple files if you can, because it's sort of long.

feywind · 2023-03-02T22:40:57Z

+
+const {Root} = protobuf;
+
+describe('Adapt Protos', () => {


If you're interested, I recently discovered a package snap-shot-it that lets you just record the output of a test, and then it will compare against the saved one. Not super essential though.

…files

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

generated-files-bot · 2023-03-09T15:48:14Z

Warning: This pull request is touching the following templated files:

README.md - README.md is managed by synthtool. However, a partials file can be used to update the README, e.g.: https://github.com/googleapis/nodejs-storage/blob/main/.readme-partials.yaml
samples/README.md - samples/README.md is managed by synthtool. However, a partials file can be used to update the README, e.g.: https://github.com/googleapis/nodejs-storage/blob/main/.readme-partials.yaml

feat: initial iteration on converting TableSchema to ProtoDescriptor

8aa66b6

product-auto-label Bot added size: l Pull request size is large. api: bigquerystorage Issues related to the googleapis/nodejs-bigquery-storage API. labels Feb 16, 2023

fix: lint issues

2914e3b

alvarowolfx changed the title ~~feat: initial iteration on converting TableSchema to ProtoDescriptor~~ feat: adapt package to convert TableSchema to ProtoDescriptor Feb 17, 2023

feat: add schema conversion functions

b413a35

product-auto-label Bot added size: xl Pull request size is extra large. and removed size: l Pull request size is large. labels Feb 17, 2023

alvarowolfx added 4 commits February 17, 2023 18:41

feat: add normalize proto descriptor function

6ede7ac

fix: lint issues

94570f5

fix: lint issues on proto conversion

037e64e

test: improve type conversion coverage for adapt package

57a7a4b

alvarowolfx mentioned this pull request Feb 22, 2023

feat: add storage sample using adapt package #327

Closed

alvarowolfx marked this pull request as ready for review February 22, 2023 18:19

alvarowolfx requested review from a team, chalmerlowe, feywind and shollyman and removed request for chalmerlowe February 22, 2023 18:19

feat: add storage sample using adapt package

9979f4c

revert: proto files update

4b84daf

shollyman reviewed Feb 23, 2023

View reviewed changes

Comment thread package.json Outdated

shollyman reviewed Feb 23, 2023

View reviewed changes

Comment thread src/adapt/proto.ts Outdated

Comment thread src/adapt/proto.ts Outdated

alvarowolfx mentioned this pull request Feb 27, 2023

feat: storage write api veneer #328

Merged

feat: improve docs and import patterns

579ae01

feywind approved these changes Mar 2, 2023

View reviewed changes

alvarowolfx added 2 commits March 7, 2023 15:27

feat: migrate protoc to protobufjs and split adapt package into more …

b42d7cf

…files

fix: lint issues and protobufjs gen instructions

62cd6ec

alvarowolfx added the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 8, 2023

gcf-owl-bot Bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 8, 2023

🦉 Updates from OwlBot post-processor

5a606fb

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

shollyman reviewed Mar 8, 2023

View reviewed changes

Comment thread samples/customer_record.json

Comment thread samples/customer_record.json

Comment thread samples/append_rows_table_to_proto2.js Outdated

Comment thread src/adapt/proto_mappings.ts

fix: naming issues

6e78627

alvarowolfx added 2 commits March 17, 2023 17:40

feat: remove namespace conversion need by using Type.fromDescriptor

bf6b545

fix: address comment points and remove bigquery core dep

3f76517

alvarowolfx requested review from feywind and shollyman March 20, 2023 20:40

shollyman approved these changes Mar 21, 2023

View reviewed changes

feywind approved these changes Mar 21, 2023

View reviewed changes

Comment thread samples/append_rows_proto2.js Outdated

alvarowolfx added the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 21, 2023

gcf-owl-bot Bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 21, 2023

alvarowolfx and others added 2 commits March 21, 2023 14:19

Merge branch 'main' into proto-adapt-module

bf85877

fix: remove extra files

1f48245

alvarowolfx added the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 22, 2023

gcf-owl-bot Bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 22, 2023

alvarowolfx merged commit 2d189e9 into googleapis:main Mar 22, 2023

release-please Bot mentioned this pull request Mar 22, 2023

chore(main): release 3.4.0 #324

Merged

release-please Bot mentioned this pull request Aug 13, 2025

chore(main): release 5.2.0 #583

Closed

Conversation

alvarowolfx commented Feb 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

snippet-bot Bot commented Feb 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

feywind left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

feywind Mar 2, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

feywind Mar 2, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

feywind Mar 2, 2023

Choose a reason for hiding this comment

Uh oh!

feywind Mar 2, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

feywind Mar 2, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

generated-files-bot Bot commented Mar 9, 2023

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alvarowolfx commented Feb 16, 2023 •

edited

Loading

snippet-bot Bot commented Feb 22, 2023 •

edited

Loading