Google BigQuery Storage: Node.js Samples

Node.js idiomatic client for BigQuery Storage.

The BigQuery Storage product is divided into two major APIs: Write and Read API. BigQuery Storage API does not provide functionality related to managing BigQuery resources such as datasets, jobs, or tables.

The BigQuery Storage Write API is a unified data-ingestion API for BigQuery. It combines streaming ingestion and batch loading into a single high-performance API. You can use the Storage Write API to stream records into BigQuery in real time or to batch process an arbitrarily large number of records and commit them in a single atomic operation.

Read more in our introduction guide.

Using a system provided default stream, this code sample demonstrates using the schema of a destination stream/table to construct a writer, and send several batches of row data to the table.

const {adapt, managedwriter} = require('@google-cloud/bigquery-storage');
const {WriterClient, JSONWriter} = managedwriter;

async function appendJSONRowsDefaultStream() {      
  const projectId = 'my_project';
  const datasetId = 'my_dataset';
  const tableId = 'my_table';

  const destinationTable = `projects/${projectId}/datasets/${datasetId}/tables/${tableId}`;
  const writeClient = new WriterClient({projectId});

  try {
    const writeStream = await writeClient.getWriteStream({
      streamId: `${destinationTable}/streams/_default`,
      view: 'FULL'
    });
    const protoDescriptor = adapt.convertStorageSchemaToProto2Descriptor(
      writeStream.tableSchema,
      'root'
    );

    const connection = await writeClient.createStreamConnection({
      streamId: managedwriter.DefaultStream,
      destinationTable,
    });
    const streamId = connection.getStreamId();

    const writer = new JSONWriter({
      streamId,
      connection,
      protoDescriptor,
    });

    let rows = [];
    const pendingWrites = [];

    // Row 1
    let row = {
      row_num: 1,
      customer_name: 'Octavia',
    };
    rows.push(row);

    // Row 2
    row = {
      row_num: 2,
      customer_name: 'Turing',
    };
    rows.push(row);

    // Send batch.
    let pw = writer.appendRows(rows);
    pendingWrites.push(pw);

    rows = [];

    // Row 3
    row = {
      row_num: 3,
      customer_name: 'Bell',
    };
    rows.push(row);

    // Send batch.
    pw = writer.appendRows(rows);
    pendingWrites.push(pw);

    const results = await Promise.all(
      pendingWrites.map(pw => pw.getResult())
    );
    console.log('Write results:', results);
  } catch (err) {
    console.log(err);
  } finally {
    writeClient.close();
  }
}

The BigQuery Storage Read API provides fast access to BigQuery-managed storage by using an gRPC based protocol. When you use the Storage Read API, structured data is sent over the wire in a binary serialization format. This allows for additional parallelism among multiple consumers for a set of results.

Read more how to use the BigQuery Storage Read API.

See sample code on the Quickstart section.

Table of Contents

• Before you begin
• Samples
  • Append_rows_buffered
  • Append_rows_json_writer_committed
  • Append_rows_json_writer_default
  • Append_rows_pending
  • Append_rows_proto2
  • Append_rows_table_to_proto2
  • Customer_record_pb
  • BigQuery Storage Quickstart
  • Sample_data_pb

Before you begin

Before running the samples, make sure you've followed the steps outlined in Using the client library.

cd samples

npm install

cd ..

Samples

Append_rows_buffered

View the source code.

Usage:

node samples/append_rows_buffered.js

---

Append_rows_json_writer_committed

View the source code.

Usage:

node samples/append_rows_json_writer_committed.js

---

Append_rows_json_writer_default

View the source code.

Usage:

node samples/append_rows_json_writer_default.js

---

Append_rows_pending

View the source code.

Usage:

node samples/append_rows_pending.js

---

Append_rows_proto2

View the source code.

Usage:

node samples/append_rows_proto2.js

---

Append_rows_table_to_proto2

View the source code.

Usage:

node samples/append_rows_table_to_proto2.js

---

Customer_record_pb

View the source code.

Usage:

node samples/customer_record_pb.js

---

BigQuery Storage Quickstart

Read data from a table via read stream.

View the source code.

Usage:

node quickstart.js

---

Sample_data_pb

View the source code.

Usage:

node samples/sample_data_pb.js