adapt.convertStorageSchemaToProto2Descriptor converts the names of RECORD type fields to lower case, which causes data not to be inserted if the data schema keeps the casing of the table
Environment details
- OS: any
- Node.js version: v18.6.0
- npm version: 9.6.6
@google-cloud/bigquery-storage version: 3.4.0
Steps to reproduce
- Create a table with a RECORD type column with any upper case letters.
Example:
[
{
"name": "stringField",
"type": "STRING",
"mode": "NULLABLE"
},
{
"name": "recordField",
"type": "RECORD",
"mode": "NULLABLE",
"fields": [
{
"name": "recoredSubField",
"type": "STRING",
"mode": "NULLABLE"
}
]
},
{
"name": "repeatedField",
"type": "RECORD",
"mode": "REPEATED",
"fields": [
{
"name": "repeatedSubField",
"type": "STRING",
"mode": "NULLABLE"
}
]
}
]
- Adapt its schema to proto descriptor using
adapt.convertStorageSchemaToProto2Descriptor and check the output:
const { adapt } = require('@google-cloud/bigquery-storage');
const { BigQuery } = require('@google-cloud/bigquery');
const bigquery = new BigQuery({ projectId});
const dataset = bigquery.dataset(datasetId);
const table = await dataset.table(tableId);
const [metadata] = await table.getMetadata();
const storageSchema = adapt.convertBigQuerySchemaToStorageTableSchema(
metadata.schema
);
const protoDescriptor = adapt.convertStorageSchemaToProto2Descriptor(
storageSchema,
'InputData'
);
console.log(protoDescriptor)
Ouput:
DescriptorProto {
field: [
FieldDescriptorProto {
name: 'stringField',
number: 1,
type: 9,
label: 1,
options: [Object]
},
FieldDescriptorProto {
name: 'recordfield',
number: 2,
type: 11,
typeName: 'InputData_recordField',
label: 1
},
FieldDescriptorProto {
name: 'repeatedfield',
number: 3,
type: 11,
typeName: 'InputData_repeatedField',
label: 3
}
],
...
}
Note that the STRING field stringField keeps its original casing in FieldDescriptorProto name, while recordField and repeteadField are converted to recordfield and repeatedfield.
Thanks!
adapt.convertStorageSchemaToProto2Descriptorconverts the names of RECORD type fields to lower case, which causes data not to be inserted if the data schema keeps the casing of the tableEnvironment details
@google-cloud/bigquery-storageversion: 3.4.0Steps to reproduce
Example:
[ { "name": "stringField", "type": "STRING", "mode": "NULLABLE" }, { "name": "recordField", "type": "RECORD", "mode": "NULLABLE", "fields": [ { "name": "recoredSubField", "type": "STRING", "mode": "NULLABLE" } ] }, { "name": "repeatedField", "type": "RECORD", "mode": "REPEATED", "fields": [ { "name": "repeatedSubField", "type": "STRING", "mode": "NULLABLE" } ] } ]adapt.convertStorageSchemaToProto2Descriptorand check the output:Ouput:
Note that the STRING field
stringFieldkeeps its original casing in FieldDescriptorProto name, whilerecordFieldandrepeteadFieldare converted torecordfieldandrepeatedfield.Thanks!