-
Notifications
You must be signed in to change notification settings - Fork 1.1k
[BigQuery] Streaming insert drops records? #3344
Copy link
Copy link
Closed
Labels
🚨This issue needs some love.This issue needs some love.api: bigqueryIssues related to the BigQuery API.Issues related to the BigQuery API.priority: p2Moderately-important priority. Fix may not be included in next release.Moderately-important priority. Fix may not be included in next release.status: blockedResolving the issue is dependent on other work.Resolving the issue is dependent on other work.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Metadata
Metadata
Assignees
Labels
🚨This issue needs some love.This issue needs some love.api: bigqueryIssues related to the BigQuery API.Issues related to the BigQuery API.priority: p2Moderately-important priority. Fix may not be included in next release.Moderately-important priority. Fix may not be included in next release.status: blockedResolving the issue is dependent on other work.Resolving the issue is dependent on other work.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
I'm facing an issue with BigQuery streaming inserts (
Table.insert(...), specificallyinsert(Iterable<InsertAllRequest.RowToInsert> rows, boolean skipInvalidRows, boolean ignoreUnknownValues)withskipInvalidRows = falseandignoreUnknownValues = false) where (sometimes) records don't seem to be available after one or more insert requests. TheInsertAllRequests complete successfully, i.e. no exceptions are thrown and no errors are reported (InsertAllResponse.hasErrorsreturnsfalse). I checked availability of streamed data in the BigQuery Web UI and using theTable.list(...)API. According to https://cloud.google.com/bigquery/streaming-data-into-bigquery I would expect streamed data to be available for query a few seconds after insertion. In cases where some records were missing after the initial check, I tried again after 10s, 30s, 60s, 1h, ... but to no avail. So it looks like the records have been dropped for some reason.