feat: add table.createInsertStream for native streaming inserts#997
feat: add table.createInsertStream for native streaming inserts#997steffnay merged 21 commits intogoogleapis:mainfrom
Conversation
product-auto-label
Bot
added
the
api: bigquery
google-cla
Bot
added
the
cla: yes
steffnay
added
the
owlbot:run
gcf-owl-bot
Bot
removed
the
owlbot:run
| maxOutstandingBytes: 1 * 1024 * 1024, | ||
|
|
||
| // The maximum time we'll wait to send batched rows, in milliseconds. | ||
| maxDelayMillis: 10000, |
There was a problem hiding this comment.
Just based off similar batching of messages in nodejs-pubsub, do you have an idea of a timeframe that is better aligned for BigQuery inserts?
There was a problem hiding this comment.
Not really. I think it has more to do with the customer's requirements than BigQuery limitations. Since it's configurable, aligning the default with pub/sub makes sense to me.
feywind
left a comment
feywind
left a comment
There was a problem hiding this comment.
Sorry for the flood of comments ^^; I'm happy to talk over any of them and/or help fix some of the any comments if you like.
steffnay
added
the
owlbot:run
gcf-owl-bot
Bot
removed
the
owlbot:run
feywind
left a comment
feywind
left a comment
There was a problem hiding this comment.
I can't remember what all I commented on now, but overall it looks good :)
| const opts = typeof options === 'object' ? options : {}; | ||
|
|
||
| if (opts.insertRowsOptions) { | ||
| this.insertRowsOptions = opts.insertRowsOptions; |
There was a problem hiding this comment.
Ah geez, I never actually gave this link to you... Here's one method:
| }; | ||
| }), | ||
| // eslint-disable-next-line @typescript-eslint/no-explicit-any | ||
| row: rows[(insertError as any).index], |
| maxOutstandingBytes: 1 * 1024 * 1024, | ||
|
|
||
| // The maximum time we'll wait to send batched rows, in milliseconds. | ||
| maxDelayMillis: 10000, |
There was a problem hiding this comment.
Not really. I think it has more to do with the customer's requirements than BigQuery limitations. Since it's configurable, aligning the default with pub/sub makes sense to me.
steffnay
added
the
owlbot:run
gcf-owl-bot
Bot
removed
the
owlbot:run
3 participants
This feature adds the ability to use a native Duplex stream for inserting rows into BigQuery via the
/insertAllendpoint and reading the API response. Implements batching of rows viaRowBatchandRowQueueclasses.Adds:
Table.createInsertStream()
RowQueue
RowBatch
Ensure the tests and linter pass
Code coverage does not decrease (if any source code was changed)
Appropriate docs were updated (if necessary)
Fixes #506 🦕