Skip to content

BigQuery: obscure exception when writing 0 bytes #2249

@NicolaSpreafico

Description

@NicolaSpreafico

I wrote this code in an App Engine standard (Java7) application which writes a set of rows in a BigQuery table

My code started from here: https://cloud.google.com/bigquery/loading-data-post-request#bigquery-import-file-java

private static final Charset UTF8 = Charset.forName("UTF-8");

// Instantiates a client
BigQuery bigquery = BigQueryOptions.getDefaultInstance().getService();

// Create the reference for the Table
TableId tableId = TableId.of(project, dataset, table);

WriteChannelConfiguration writeChannelConfiguration = WriteChannelConfiguration
    .newBuilder(tableId)
    .setFormatOptions(FormatOptions.json())
    .build();

// Open a new channel for write
TableDataWriteChannel writer = bigquery.writer(writeChannelConfiguration);

// Retrieves the object from DB
List<T> objects = ....;

// Write data to writer
try (OutputStream stream = Channels.newOutputStream(writer)) {
    // Access to iterator of the objects
    Iterator<T> iterator = objects.iterator();
    
    while (iterator.hasNext()) {
        // Retrieve the next element
        T object = iterator.next();
    
        // Serialize the object in json format
        JsonObject json = serialize(object);
    
        // Write the json to the stream
        stream.write(json.toString().getBytes(UTF8));
    
        // Increment the counter
        index++;
    
        // Check if additional records are available
        if (iterator.hasNext()) {
            stream.write("\n".getBytes(UTF8));
        }
    }
}

// Get job status
Job job = writer.getJob();

// Wait for job to end
try {
    job = job.waitFor();
} catch (InterruptedException | TimeoutException e) {
    throw new IOException("WaitFor error", e);
}

// Retrieve job statistics
LoadStatistics stats = job.getStatistics();
LOG.log(Level.INFO, "Executed Job: {0}", new Object[] {stats});

In localhost works fine, w/o problems. All the rows are written and no errors raised.

But after I ran this code online I got this error

{"message":"400\nFailed to parse Content-Range header.","errors":[{"message":"400\nFailed to parse Content-Range header."},{"message":"400\nFailed to parse Content-Range header."}]}

Failed to parse Content-Range header.
	at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:86)
	at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.write(HttpBigQueryRpc.java:466)
	at com.google.cloud.bigquery.TableDataWriteChannel$1.call(TableDataWriteChannel.java:57)
	at com.google.cloud.bigquery.TableDataWriteChannel$1.call(TableDataWriteChannel.java:54)
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:94)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:54)
	at com.google.cloud.bigquery.TableDataWriteChannel.flushBuffer(TableDataWriteChannel.java:53)
	at com.google.cloud.BaseWriteChannel.close(BaseWriteChannel.java:160)
	at java.nio.channels.Channels$1.close(Channels.java:178)
Caused by: com.google.api.client.http.HttpResponseException: 400
Failed to parse Content-Range header.
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1070)
	at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.write(HttpBigQueryRpc.java:449)
	... 69 more

Can this library be used on App Engine standard environment? I'm asking this because for the PubSub counterpart there is a warning

However, we recommend using the older Google APIs Client Libraries if running on Google App Engine standard environment

that I cannot see on BigQuery page.

My goal here is loading a bunch of rows directly inside the table w/o using a temporary file, loading the data directly from the application memory.

I enabled the full Log of HTTP Transport, here is a part of it

com.google.api.client.http.HttpRequest execute: -------------- REQUEST  -------------- (HttpRequest.java:962)
PUT https://www.googleapis.com/upload/bigquery/v2/projects/noovle-erp-orchestrator-dev/jobs?uploadType=resumable&upload_id=AEnB2Ur7N3UM_EkF6fpM9NQFmKhsEtS1y2IsEXM0DFjnPIAn6udvTx6gA5dWDzbfwscWkbLUXy9llTBhssjG8n-llBgSlYs-2g
Accept-Encoding: gzip
Authorization: <Not Logged>
Content-Range: bytes 0--1/0
User-Agent: Google-HTTP-Java-Client/1.22.0 (gzip)
x-goog-api-client: gl-java/1.7.0-google-v6 gccl/0.20.1-beta
Content-Length: 0

Content-Length is 0 and Content-Range is 0 as well. Maybe this type of in-memory load does not work because I do not have in advance the size of the full data?
If so, why in localhost works fine?

Metadata

Metadata

Labels

api: bigqueryIssues related to the BigQuery API.priority: p2Moderately-important priority. Fix may not be included in next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions