Skip to content

Commit 9a7e94a

Browse files
pongadpongad
authored
---
yaml --- r: 8555 b: refs/heads/bq-perf c: 6125c56 h: refs/heads/master i: 8553: 0f92b58 8551: a2b0b78
1 parent 62f86a8 commit 9a7e94a

404 files changed

Lines changed: 5778 additions & 32138 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎[refs]‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -74,4 +74,4 @@ refs/tags/v0.23.0: e5405e1f6d144441b889acd3b6405fdcc3cdfd72
7474
refs/tags/v0.23.1: 30bcf8076ef9d71cc5a858d026cb907bb0954bec
7575
refs/tags/v0.24.0: b3cf61898d9c63d028fe088c14486721318d5fd5
7676
refs/heads/snehashah-bugfix: 05db5cd15de0a559442b4b73137776c73289e4b0
77-
refs/heads/bq-perf: 40b0da87d50d9d6e996d50dd158f686ba656f6c8
77+
refs/heads/bq-perf: 6125c56eda925e89cd3c8432332af2514d3f4009

‎branches/bq-perf/.github/CODEOWNERS‎

Lines changed: 28 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -2,35 +2,33 @@
22
# This file controls who is tagged for review for any given pull request.
33

44
# For anything not explicitly taken by someone else.
5-
* @garrettjonesgoogle
5+
* @garrettjonesgoogle
66
# Owners per module.
7-
google-cloud-bigquery/src/* @vam-google @tswast
8-
google-cloud-bigtable/src/* @igorbernstein2
9-
google-cloud-compute/src/* @andreamlin
10-
google-cloud-contrib/src/* @garrettjonesgoogle @jean-philippe-martin
11-
google-cloud-core-grpc/src/* @garrettjonesgoogle
12-
google-cloud-core-http/src/* @garrettjonesgoogle
13-
google-cloud-core/src/* @garrettjonesgoogle
14-
google-cloud-datastore/src/* @neozwu @vam-google
15-
google-cloud-dep-verification/src/* @garrettjonesgoogle
16-
google-cloud-dlp/src/* @neozwu
17-
google-cloud-dns/src/* @garrettjonesgoogle
18-
google-cloud-errorreporting/src/* @garrettjonesgoogle
19-
google-cloud-examples/src/* @garrettjonesgoogle
20-
google-cloud-firestore/src/* @schmidt-sebastian
21-
google-cloud-language/src/* @garrettjonesgoogle
22-
google-cloud-logging/src/* @saicheems @andreamlin @jabubake
23-
google-cloud-monitoring/src/* @garrettjonesgoogle
24-
google-cloud-pubsub/src/* @pongad @garrettjonesgoogle
25-
google-cloud-resourcemanager/src/* @garrettjonesgoogle
26-
google-cloud-spanner/src/* @vam-google @vkedia
27-
google-cloud-speech/src/* @vam-google
28-
google-cloud-storage/src/* @andreamlin @neozwu
29-
google-cloud-testing/src/* @vam-google
30-
google-cloud-trace/src/* @garrettjonesgoogle
31-
google-cloud-translate/src/* @andreamlin @saicheems
32-
google-cloud-util/src/* @garrettjonesgoogle
33-
google-cloud-video-intelligence/src/* @vam-google
34-
google-cloud-vision/src/* @vam-google
35-
google-cloud/src/* @garrettjonesgoogle
7+
google-cloud-bigquery/* @vam-google @tswast
8+
google-cloud-compute/* @garrettjonesgoogle
9+
google-cloud-contrib/* @garrettjonesgoogle @jean-philippe-martin
10+
google-cloud-core-grpc/* @garrettjonesgoogle
11+
google-cloud-core-http/* @garrettjonesgoogle
12+
google-cloud-core/* @garrettjonesgoogle
13+
google-cloud-datastore/* @garrettjonesgoogle
14+
google-cloud-dep-verification/* @garrettjonesgoogle
15+
google-cloud-dlp/* @neozwu
16+
google-cloud-dns/* @garrettjonesgoogle
17+
google-cloud-errorreporting/* @garrettjonesgoogle
18+
google-cloud-examples/* @garrettjonesgoogle
19+
google-cloud-language/* @garrettjonesgoogle
20+
google-cloud-logging/* @garrettjonesgoogle @jabubake
21+
google-cloud-monitoring/* @garrettjonesgoogle
22+
google-cloud-pubsub/* @pongad
23+
google-cloud-resourcemanager/* @garrettjonesgoogle
24+
google-cloud-spanner/* @vam-google @vkedia
25+
google-cloud-speech/* @vam-google
26+
google-cloud-storage/* @garrettjonesgoogle
27+
google-cloud-testing/* @vam-google
28+
google-cloud-trace/* @garrettjonesgoogle
29+
google-cloud-translate/* @garrettjonesgoogle
30+
google-cloud-util/* @garrettjonesgoogle
31+
google-cloud-video-intelligence/* @vam-google
32+
google-cloud-vision/* @vam-google
33+
google-cloud/* @garrettjonesgoogle
3634

‎branches/bq-perf/README.md‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -15,29 +15,29 @@ Java idiomatic client for [Google Cloud Platform][cloud-platform] services.
1515
This client supports the following Google Cloud Platform services at a [GA](#versioning) quality level:
1616
- [Stackdriver Logging](google-cloud-logging) (GA)
1717
- [Cloud Datastore](google-cloud-datastore) (GA)
18-
- [Cloud Natural Language](google-cloud-language) (GA)
1918
- [Cloud Storage](google-cloud-storage) (GA)
2019
- [Cloud Translation](google-cloud-translate) (GA)
21-
- [Cloud Vision](google-cloud-vision) (GA)
2220

2321
This client supports the following Google Cloud Platform services at a [Beta](#versioning) quality level:
2422

2523
- [BigQuery](google-cloud-bigquery) (Beta)
26-
- [Cloud Data Loss Prevention](google-cloud-dlp) (Beta)
27-
- [Stackdriver Error Reporting](google-cloud-errorreporting) (Beta)
2824
- [Cloud Firestore](google-cloud-firestore) (Beta)
29-
- [Stackdriver Monitoring](google-cloud-monitoring) (Beta)
3025
- [Cloud Pub/Sub](google-cloud-pubsub) (Beta)
3126
- [Cloud Spanner](google-cloud-spanner) (Beta)
27+
- [Cloud Natural Language](google-cloud-language) (Beta)
3228
- [Cloud Video Intelligence](google-cloud-video-intelligence) (Beta)
33-
- [Stackdriver Trace](google-cloud-trace) (Beta)
29+
- [Cloud Vision](google-cloud-vision) (Beta)
3430

3531
This client supports the following Google Cloud Platform services at an [Alpha](#versioning) quality level:
3632

3733
- [Cloud Compute](google-cloud-compute) (Alpha)
34+
- [Cloud Data Loss Prevention](google-cloud-dlp) (Alpha)
3835
- [Cloud DNS](google-cloud-dns) (Alpha)
36+
- [Stackdriver Error Reporting](google-cloud-errorreporting) (Alpha)
37+
- [Stackdriver Monitoring](google-cloud-monitoring) (Alpha)
3938
- [Cloud Resource Manager](google-cloud-resourcemanager) (Alpha)
4039
- [Cloud Speech](google-cloud-speech) (Alpha)
40+
- [Cloud Trace](google-cloud-trace) (Alpha)
4141

4242
> Note: google-cloud-java is a work-in-progress, and may occasionally
4343
> make backwards-incompatible changes.
@@ -50,16 +50,16 @@ If you are using Maven, add this to your pom.xml file
5050
<dependency>
5151
<groupId>com.google.cloud</groupId>
5252
<artifactId>google-cloud</artifactId>
53-
<version>0.30.0-alpha</version>
53+
<version>0.26.0-alpha</version>
5454
</dependency>
5555
```
5656
If you are using Gradle, add this to your dependencies
5757
```Groovy
58-
compile 'com.google.cloud:google-cloud:0.30.0-alpha'
58+
compile 'com.google.cloud:google-cloud:0.26.0-alpha'
5959
```
6060
If you are using SBT, add this to your dependencies
6161
```Scala
62-
libraryDependencies += "com.google.cloud" % "google-cloud" % "0.30.0-alpha"
62+
libraryDependencies += "com.google.cloud" % "google-cloud" % "0.26.0-alpha"
6363
```
6464

6565
For running on Google App Engine, see [more instructions here](./APPENGINE.md).

‎branches/bq-perf/google-cloud-bigquery/README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,16 +22,16 @@ If you are using Maven, add this to your pom.xml file
2222
<dependency>
2323
<groupId>com.google.cloud</groupId>
2424
<artifactId>google-cloud-bigquery</artifactId>
25-
<version>0.30.0-beta</version>
25+
<version>0.26.0-beta</version>
2626
</dependency>
2727
```
2828
If you are using Gradle, add this to your dependencies
2929
```Groovy
30-
compile 'com.google.cloud:google-cloud-bigquery:0.30.0-beta'
30+
compile 'com.google.cloud:google-cloud-bigquery:0.26.0-beta'
3131
```
3232
If you are using SBT, add this to your dependencies
3333
```Scala
34-
libraryDependencies += "com.google.cloud" % "google-cloud-bigquery" % "0.30.0-beta"
34+
libraryDependencies += "com.google.cloud" % "google-cloud-bigquery" % "0.26.0-beta"
3535
```
3636

3737
Example Application

‎branches/bq-perf/google-cloud-bigquery/pom.xml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
33
<modelVersion>4.0.0</modelVersion>
44
<artifactId>google-cloud-bigquery</artifactId>
5-
<version>0.30.1-beta-SNAPSHOT</version>
5+
<version>0.26.1-beta-SNAPSHOT</version>
66
<packaging>jar</packaging>
77
<name>Google Cloud BigQuery</name>
88
<url>https://github.com/GoogleCloudPlatform/google-cloud-java/tree/master/google-cloud-bigquery</url>
@@ -12,7 +12,7 @@
1212
<parent>
1313
<groupId>com.google.cloud</groupId>
1414
<artifactId>google-cloud-pom</artifactId>
15-
<version>0.30.1-alpha-SNAPSHOT</version>
15+
<version>0.26.1-alpha-SNAPSHOT</version>
1616
</parent>
1717
<properties>
1818
<site.installationModule>google-cloud-bigquery</site.installationModule>

‎branches/bq-perf/google-cloud-bigquery/src/benchmark/java/com/google/cloud/bigquery/benchmark/Benchmark.java‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,8 @@
2020
import com.google.cloud.bigquery.BigQuery;
2121
import com.google.cloud.bigquery.BigQueryOptions;
2222
import com.google.cloud.bigquery.FieldValue;
23+
import com.google.cloud.bigquery.Job;
2324
import com.google.cloud.bigquery.QueryJobConfiguration;
24-
import com.google.cloud.bigquery.QueryResponse;
2525
import java.io.FileInputStream;
2626
import java.util.List;
2727
import org.threeten.bp.Clock;
@@ -53,13 +53,13 @@ public static void main(String[] args) throws Exception {
5353
}
5454

5555
Instant start = clock.instant();
56-
QueryResponse queryResponse =
57-
bq.query(QueryJobConfiguration.newBuilder(request).setUseLegacySql(false).build());
56+
Job job = bq.query(QueryJobConfiguration.newBuilder(request).setUseLegacySql(false).build());
57+
job = job.waitFor();
5858

5959
int rows = 0;
6060
int cols = 0;
6161
Duration firstByte = null;
62-
for (List<FieldValue> row : queryResponse.getResult().iterateAll()) {
62+
for (List<FieldValue> row : job.getQueryResults().getResult().iterateAll()) {
6363
rows++;
6464
if (cols == 0) {
6565
cols = row.size();

‎branches/bq-perf/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQuery.java‎

Lines changed: 18 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,6 @@
2727
import com.google.common.base.Function;
2828
import com.google.common.collect.ImmutableList;
2929
import com.google.common.collect.Lists;
30-
3130
import java.io.Serializable;
3231
import java.util.ArrayList;
3332
import java.util.List;
@@ -989,11 +988,14 @@ Page<FieldValueList> listTableData(String datasetId, String tableId,
989988
*/
990989
boolean cancel(JobId jobId);
991990

991+
// TODO(pongad): rewrite query() samples.
992+
992993
/**
993-
* Runs the query associated with the request, using an internally-generated random JobId.
994+
* Runs the query associated with the request, using an internally-generated random JobId. The returned job is always completed.
994995
*
995996
* <p>Example of running a query.
996-
* <pre> {@code
997+
*
998+
* <pre>{@code
997999
* String query = "SELECT distinct(corpus) FROM `bigquery-public-data.samples.shakespeare`";
9981000
* QueryJobConfiguration queryConfig = QueryJobConfiguration.of(query);
9991001
*
@@ -1013,7 +1015,8 @@ Page<FieldValueList> listTableData(String datasetId, String tableId,
10131015
* }</pre>
10141016
*
10151017
* <p>Example of running a query with query parameters.
1016-
* <pre> {@code
1018+
*
1019+
* <pre>{@code
10171020
* String query =
10181021
* "SELECT distinct(corpus) FROM `bigquery-public-data.samples.shakespeare` where word_count > ?";
10191022
* QueryJobConfiguration queryConfig = QueryJobConfiguration.newBuilder(query)
@@ -1033,32 +1036,31 @@ Page<FieldValueList> listTableData(String datasetId, String tableId,
10331036
* @throws InterruptedException if the current thread gets interrupted while waiting for the query
10341037
* to complete
10351038
*/
1036-
QueryResponse query(QueryJobConfiguration configuration, QueryOption... options)
1037-
throws InterruptedException;
1039+
Job query(QueryJobConfiguration configuration, JobOption... options) throws InterruptedException;
10381040

10391041
/**
1040-
* Runs the query associated with the request, using the given job id.
1042+
* Runs the query associated with the request, using the given JobId. The returned job is always completed.
10411043
*
1042-
* <p>See {@link #query(QueryJobConfiguration, QueryOption...)} for examples on populating a
1043-
* {@link QueryJobConfiguration}.
1044+
* <p>See {@link #query(QueryJobConfiguration, JobOption...)} for examples on populating a {@link
1045+
* QueryJobConfiguration}.
10441046
*
1045-
* <p>
1046-
* The recommended way to create a randomly generated JobId is the following:
1047+
* <p>The recommended way to create a randomly generated JobId is the following:
10471048
*
1048-
* <pre> {@code
1049-
* JobId jobId = JobId.of();
1049+
* <pre>{@code
1050+
* JobId jobId = JobId.of();
10501051
* }</pre>
10511052
*
10521053
* For a user specified job id with an optional prefix use the following:
1053-
* <pre> {@code
1054-
* JobId jobId = JobId.of("my_prefix-my_unique_job_id");
1054+
*
1055+
* <pre>{@code
1056+
* JobId jobId = JobId.of("my_prefix-my_unique_job_id");
10551057
* }</pre>
10561058
*
10571059
* @throws BigQueryException upon failure
10581060
* @throws InterruptedException if the current thread gets interrupted while waiting for the query
10591061
* to complete
10601062
*/
1061-
QueryResponse query(QueryJobConfiguration configuration, JobId jobId, QueryOption... options)
1063+
Job query(QueryJobConfiguration configuration, JobId jobId, JobOption... options)
10621064
throws InterruptedException;
10631065

10641066
/**

‎branches/bq-perf/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQueryImpl.java‎

Lines changed: 6 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,6 @@
2020
import static com.google.common.base.Preconditions.checkArgument;
2121

2222
import com.google.api.gax.paging.Page;
23-
import com.google.api.gax.retrying.RetrySettings;
2423
import com.google.api.services.bigquery.model.GetQueryResultsResponse;
2524
import com.google.api.services.bigquery.model.TableDataInsertAllRequest;
2625
import com.google.api.services.bigquery.model.TableDataInsertAllRequest.Rows;
@@ -32,10 +31,10 @@
3231
import com.google.cloud.PageImpl.NextPageFetcher;
3332
import com.google.cloud.RetryHelper;
3433
import com.google.cloud.RetryHelper.RetryHelperException;
35-
import com.google.cloud.RetryOption;
3634
import com.google.cloud.Tuple;
3735
import com.google.cloud.bigquery.InsertAllRequest.RowToInsert;
3836
import com.google.cloud.bigquery.spi.v2.BigQueryRpc;
37+
import com.google.common.annotations.VisibleForTesting;
3938
import com.google.common.base.Function;
4039
import com.google.common.collect.ImmutableList;
4140
import com.google.common.collect.ImmutableMap;
@@ -584,24 +583,15 @@ public Boolean call() {
584583
}
585584

586585
@Override
587-
public QueryResponse query(QueryJobConfiguration configuration, QueryOption... options)
586+
public Job query(QueryJobConfiguration configuration, JobOption... options)
588587
throws InterruptedException {
589588
return query(configuration, JobId.of(), options);
590589
}
591590

592591
@Override
593-
public QueryResponse query(QueryJobConfiguration configuration, JobId jobId, QueryOption... options)
592+
public Job query(QueryJobConfiguration configuration, JobId jobId, JobOption... options)
594593
throws InterruptedException {
595-
try {
596-
JobInfo jobInfo = JobInfo.newBuilder(configuration).setJobId(jobId).build();
597-
Job job = create(jobInfo);
598-
RetrySettings waitSettings =
599-
RetryOption.mergeToSettings(
600-
Job.DEFAULT_QUERY_JOB_WAIT_SETTINGS, QueryOption.filterRetryOptions(options));
601-
return job.waitForQueryResults(waitSettings, QueryOption.filterQueryResultsOptions(options));
602-
} catch (RetryHelper.RetryHelperException e) {
603-
throw BigQueryException.translateAndThrow(e);
604-
}
594+
return create(JobInfo.of(jobId, configuration), options).waitFor();
605595
}
606596

607597
@Override
@@ -677,7 +667,8 @@ public TableDataWriteChannel writer(WriteChannelConfiguration writeChannelConfig
677667
writeChannelConfiguration.setProjectId(getOptions().getProjectId()));
678668
}
679669

680-
private Map<BigQueryRpc.Option, ?> optionMap(Option... options) {
670+
@VisibleForTesting
671+
static Map<BigQueryRpc.Option, ?> optionMap(Option... options) {
681672
Map<BigQueryRpc.Option, Object> optionMap = Maps.newEnumMap(BigQueryRpc.Option.class);
682673
for (Option option : options) {
683674
Object prev = optionMap.put(option.getRpcOption(), option.getValue());

‎branches/bq-perf/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/QueryResponse.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@
2727
/**
2828
* Google Cloud BigQuery Query Response. This class contains the results of a Query Job
2929
* ({@link BigQuery#getQueryResults(JobId, BigQuery.QueryResultsOption...)}) or of a
30-
* Query Request ({@link BigQuery#query(QueryJobConfiguration, QueryOption...)}).
30+
* Query Request ({@link BigQuery#query(QueryJobConfiguration, JobOption...)}).
3131
*
3232
* <p>Example usage of a query response:
3333
* <pre> {@code

0 commit comments

Comments
 (0)