I wrote this code in an App Engine standard (Java7) application which writes a set of rows in a BigQuery table
My code started from here: https://cloud.google.com/bigquery/loading-data-post-request#bigquery-import-file-java
private static final Charset UTF8 = Charset.forName("UTF-8");
// Instantiates a client
BigQuery bigquery = BigQueryOptions.getDefaultInstance().getService();
// Create the reference for the Table
TableId tableId = TableId.of(project, dataset, table);
WriteChannelConfiguration writeChannelConfiguration = WriteChannelConfiguration
.newBuilder(tableId)
.setFormatOptions(FormatOptions.json())
.build();
// Open a new channel for write
TableDataWriteChannel writer = bigquery.writer(writeChannelConfiguration);
// Retrieves the object from DB
List<T> objects = ....;
// Write data to writer
try (OutputStream stream = Channels.newOutputStream(writer)) {
// Access to iterator of the objects
Iterator<T> iterator = objects.iterator();
while (iterator.hasNext()) {
// Retrieve the next element
T object = iterator.next();
// Serialize the object in json format
JsonObject json = serialize(object);
// Write the json to the stream
stream.write(json.toString().getBytes(UTF8));
// Increment the counter
index++;
// Check if additional records are available
if (iterator.hasNext()) {
stream.write("\n".getBytes(UTF8));
}
}
}
// Get job status
Job job = writer.getJob();
// Wait for job to end
try {
job = job.waitFor();
} catch (InterruptedException | TimeoutException e) {
throw new IOException("WaitFor error", e);
}
// Retrieve job statistics
LoadStatistics stats = job.getStatistics();
LOG.log(Level.INFO, "Executed Job: {0}", new Object[] {stats});
In localhost works fine, w/o problems. All the rows are written and no errors raised.
But after I ran this code online I got this error
{"message":"400\nFailed to parse Content-Range header.","errors":[{"message":"400\nFailed to parse Content-Range header."},{"message":"400\nFailed to parse Content-Range header."}]}
Failed to parse Content-Range header.
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:86)
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.write(HttpBigQueryRpc.java:466)
at com.google.cloud.bigquery.TableDataWriteChannel$1.call(TableDataWriteChannel.java:57)
at com.google.cloud.bigquery.TableDataWriteChannel$1.call(TableDataWriteChannel.java:54)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:94)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:54)
at com.google.cloud.bigquery.TableDataWriteChannel.flushBuffer(TableDataWriteChannel.java:53)
at com.google.cloud.BaseWriteChannel.close(BaseWriteChannel.java:160)
at java.nio.channels.Channels$1.close(Channels.java:178)
Caused by: com.google.api.client.http.HttpResponseException: 400
Failed to parse Content-Range header.
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1070)
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.write(HttpBigQueryRpc.java:449)
... 69 more
Can this library be used on App Engine standard environment? I'm asking this because for the PubSub counterpart there is a warning
However, we recommend using the older Google APIs Client Libraries if running on Google App Engine standard environment
that I cannot see on BigQuery page.
My goal here is loading a bunch of rows directly inside the table w/o using a temporary file, loading the data directly from the application memory.
I enabled the full Log of HTTP Transport, here is a part of it
com.google.api.client.http.HttpRequest execute: -------------- REQUEST -------------- (HttpRequest.java:962)
PUT https://www.googleapis.com/upload/bigquery/v2/projects/noovle-erp-orchestrator-dev/jobs?uploadType=resumable&upload_id=AEnB2Ur7N3UM_EkF6fpM9NQFmKhsEtS1y2IsEXM0DFjnPIAn6udvTx6gA5dWDzbfwscWkbLUXy9llTBhssjG8n-llBgSlYs-2g
Accept-Encoding: gzip
Authorization: <Not Logged>
Content-Range: bytes 0--1/0
User-Agent: Google-HTTP-Java-Client/1.22.0 (gzip)
x-goog-api-client: gl-java/1.7.0-google-v6 gccl/0.20.1-beta
Content-Length: 0
Content-Length is 0 and Content-Range is 0 as well. Maybe this type of in-memory load does not work because I do not have in advance the size of the full data?
If so, why in localhost works fine?
I wrote this code in an App Engine standard (Java7) application which writes a set of rows in a BigQuery table
My code started from here: https://cloud.google.com/bigquery/loading-data-post-request#bigquery-import-file-java
In localhost works fine, w/o problems. All the rows are written and no errors raised.
But after I ran this code online I got this error
Can this library be used on App Engine standard environment? I'm asking this because for the PubSub counterpart there is a warning
that I cannot see on BigQuery page.
My goal here is loading a bunch of rows directly inside the table w/o using a temporary file, loading the data directly from the application memory.
I enabled the full Log of HTTP Transport, here is a part of it
Content-Length is 0 and Content-Range is 0 as well. Maybe this type of in-memory load does not work because I do not have in advance the size of the full data?
If so, why in localhost works fine?