Skip to content
This repository was archived by the owner on Feb 7, 2026. It is now read-only.
This repository was archived by the owner on Feb 7, 2026. It is now read-only.

Implement writable streams for natively streaming inserts #506

@metasloth

Description

@metasloth

Is your feature request related to a problem? Please describe.
It would be really useful to be able to natively stream records directly into a table (via /insertAll). This would make it much more accessible to transfer significant amounts of data on low-resource platforms like cloud functions.

Describe the solution you'd like
A Table method similar to createWriteStream() in the @google-cloud/storage package. Being able to just call mySourceStream.pipe(myTable.createWriteStream()) would be super helpful.

Describe alternatives you've considered
I've managed to get roughly the functionality I'm looking for by calling insert() in arbitrarily sized batches, i.e.

class TableWriteableStream extends Writable {
  constructor (destinationTable) {
    super({ objectMode: true })
    this.destinationTable = destinationTable
    this.records = []
  }

  _write (chunk, encoding, callback) {
    this.records.push(chunk)
    // Every 500 records, call insert()
    if (this.records.length >= 500) {
      const recordsToInsert = this.records
      this.records = []
      this.destinationTable
        .insert(recordsToInsert)
        .then(() => { callback() })
    } else {
      callback()
    }
  }

  _final (callback) {
      // Insert any remaining records ...

Additional context
This is somewhat related to #75, though that seems focused on compression (which would be a great option for this feature, just like the 'gzip' option for the @google-cloud/storage function).

Metadata

Metadata

Assignees

Labels

api: bigqueryIssues related to the googleapis/nodejs-bigquery API.type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions