Skip to content

Commit 823f0de

Browse files
ajkannanajkannan
committed
Merge pull request #512 from mziccard/fix-bigquery-example
Fix bigquery example and related issues
2 parents 45f5661 + 6af6db4 commit 823f0de

8 files changed

Lines changed: 212 additions & 141 deletions

File tree

‎README.md‎

Lines changed: 78 additions & 78 deletions
Original file line numberDiff line numberDiff line change
@@ -12,10 +12,10 @@ Java idiomatic client for [Google Cloud Platform][cloud-platform] services.
1212

1313
This client supports the following Google Cloud Platform services:
1414

15+
- [Google Cloud BigQuery] (#google-cloud-bigquery)
1516
- [Google Cloud Datastore] (#google-cloud-datastore)
16-
- [Google Cloud Storage] (#google-cloud-storage)
1717
- [Google Cloud Resource Manager] (#google-cloud-resource-manager)
18-
- [Google Cloud BigQuery] (#google-cloud-bigquery)
18+
- [Google Cloud Storage] (#google-cloud-storage)
1919

2020
> Note: This client is a work-in-progress, and may occasionally
2121
> make backwards-incompatible changes.
@@ -42,14 +42,14 @@ libraryDependencies += "com.google.gcloud" % "gcloud-java" % "0.1.1"
4242
Example Applications
4343
--------------------
4444

45+
- [`BigQueryExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/BigQueryExample.java) - A simple command line interface providing some of Cloud BigQuery's functionality
46+
- Read more about using this application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/BigQueryExample.html).
4547
- [`DatastoreExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/DatastoreExample.java) - A simple command line interface for the Cloud Datastore
4648
- Read more about using this application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/DatastoreExample.html).
47-
- [`StorageExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/StorageExample.java) - A simple command line interface providing some of Cloud Storage's functionality
48-
- Read more about using this application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/StorageExample.html).
4949
- [`ResourceManagerExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/ResourceManagerExample.java) - A simple command line interface providing some of Cloud Resource Manager's functionality
5050
- Read more about using this application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/ResourceManagerExample.html).
51-
- [`BigQueryExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/BigQueryExample.java) - A simple command line interface providing some of Cloud BigQuery's functionality
52-
- Read more about using this application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/BigQueryExample.html).
51+
- [`StorageExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/StorageExample.java) - A simple command line interface providing some of Cloud Storage's functionality
52+
- Read more about using this application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/StorageExample.html).
5353

5454
Specifying a Project ID
5555
-----------------------
@@ -109,6 +109,51 @@ Next, choose a method for authenticating API requests from within your project:
109109
4. Google Cloud SDK credentials
110110
5. Compute Engine credentials
111111
112+
Google Cloud BigQuery
113+
----------------------
114+
115+
- [API Documentation][bigquery-api]
116+
- [Official Documentation][cloud-bigquery-docs]
117+
118+
#### Preview
119+
120+
Here is a code snippet showing a simple usage example from within Compute/App Engine. Note that you
121+
must [supply credentials](#authentication) and a project ID if running this snippet elsewhere.
122+
123+
```java
124+
import com.google.gcloud.bigquery.BaseTableInfo;
125+
import com.google.gcloud.bigquery.BigQuery;
126+
import com.google.gcloud.bigquery.BigQueryOptions;
127+
import com.google.gcloud.bigquery.Field;
128+
import com.google.gcloud.bigquery.JobStatus;
129+
import com.google.gcloud.bigquery.LoadJobInfo;
130+
import com.google.gcloud.bigquery.Schema;
131+
import com.google.gcloud.bigquery.TableId;
132+
import com.google.gcloud.bigquery.TableInfo;
133+
134+
BigQuery bigquery = BigQueryOptions.defaultInstance().service();
135+
TableId tableId = TableId.of("dataset", "table");
136+
BaseTableInfo info = bigquery.getTable(tableId);
137+
if (info == null) {
138+
System.out.println("Creating table " + tableId);
139+
Field integerField = Field.of("fieldName", Field.Type.integer());
140+
bigquery.create(TableInfo.of(tableId, Schema.of(integerField)));
141+
} else {
142+
System.out.println("Loading data into table " + tableId);
143+
LoadJobInfo loadJob = LoadJobInfo.of(tableId, "gs://bucket/path");
144+
loadJob = bigquery.create(loadJob);
145+
while (loadJob.status().state() != JobStatus.State.DONE) {
146+
Thread.sleep(1000L);
147+
loadJob = bigquery.getJob(loadJob.jobId());
148+
}
149+
if (loadJob.status().error() != null) {
150+
System.out.println("Job completed with errors");
151+
} else {
152+
System.out.println("Job succeeded");
153+
}
154+
}
155+
```
156+
112157
Google Cloud Datastore
113158
----------------------
114159
@@ -149,45 +194,6 @@ if (entity == null) {
149194
}
150195
```
151196
152-
Google Cloud Storage
153-
----------------------
154-
155-
- [API Documentation][storage-api]
156-
- [Official Documentation][cloud-storage-docs]
157-
158-
*Follow the [activation instructions][cloud-storage-activation] to use the Google Cloud Storage API with your project.*
159-
160-
#### Preview
161-
162-
Here is a code snippet showing a simple usage example from within Compute/App Engine. Note that you must [supply credentials](#authentication) and a project ID if running this snippet elsewhere.
163-
164-
```java
165-
import static java.nio.charset.StandardCharsets.UTF_8;
166-
167-
import com.google.gcloud.storage.Blob;
168-
import com.google.gcloud.storage.BlobId;
169-
import com.google.gcloud.storage.Storage;
170-
import com.google.gcloud.storage.StorageOptions;
171-
172-
import java.nio.ByteBuffer;
173-
import java.nio.channels.WritableByteChannel;
174-
175-
Storage storage = StorageOptions.defaultInstance().service();
176-
BlobId blobId = BlobId.of("bucket", "blob_name");
177-
Blob blob = Blob.load(storage, blobId);
178-
if (blob == null) {
179-
BlobInfo blobInfo = BlobInfo.builder(blobId).contentType("text/plain").build();
180-
storage.create(blobInfo, "Hello, Cloud Storage!".getBytes(UTF_8));
181-
} else {
182-
System.out.println("Updating content for " + blobId.name());
183-
byte[] prevContent = blob.content();
184-
System.out.println(new String(prevContent, UTF_8));
185-
WritableByteChannel channel = blob.writer();
186-
channel.write(ByteBuffer.wrap("Updated content".getBytes(UTF_8)));
187-
channel.close();
188-
}
189-
```
190-
191197
Google Cloud Resource Manager
192198
----------------------
193199
@@ -219,48 +225,42 @@ while (projectIterator.hasNext()) {
219225
}
220226
```
221227
222-
Google Cloud BigQuery
228+
Google Cloud Storage
223229
----------------------
224230
225-
- [API Documentation][bigquery-api]
226-
- [Official Documentation][cloud-bigquery-docs]
231+
- [API Documentation][storage-api]
232+
- [Official Documentation][cloud-storage-docs]
233+
234+
*Follow the [activation instructions][cloud-storage-activation] to use the Google Cloud Storage API with your project.*
227235
228236
#### Preview
229237
230-
Here is a code snippet showing a simple usage example from within Compute/App Engine. Note that you
231-
must [supply credentials](#authentication) and a project ID if running this snippet elsewhere.
238+
Here is a code snippet showing a simple usage example from within Compute/App Engine. Note that you must [supply credentials](#authentication) and a project ID if running this snippet elsewhere.
232239
233240
```java
234-
import com.google.gcloud.bigquery.BaseTableInfo;
235-
import com.google.gcloud.bigquery.BigQuery;
236-
import com.google.gcloud.bigquery.BigQueryOptions;
237-
import com.google.gcloud.bigquery.Field;
238-
import com.google.gcloud.bigquery.JobStatus;
239-
import com.google.gcloud.bigquery.LoadJobInfo;
240-
import com.google.gcloud.bigquery.Schema;
241-
import com.google.gcloud.bigquery.TableId;
242-
import com.google.gcloud.bigquery.TableInfo;
241+
import static java.nio.charset.StandardCharsets.UTF_8;
243242
244-
BigQuery bigquery = BigQueryOptions.defaultInstance().service();
245-
TableId tableId = TableId.of("dataset", "table");
246-
BaseTableInfo info = bigquery.getTable(tableId);
247-
if (info == null) {
248-
System.out.println("Creating table " + tableId);
249-
Field integerField = Field.of("fieldName", Field.Type.integer());
250-
bigquery.create(TableInfo.of(tableId, Schema.of(integerField)));
243+
import com.google.gcloud.storage.Blob;
244+
import com.google.gcloud.storage.BlobId;
245+
import com.google.gcloud.storage.Storage;
246+
import com.google.gcloud.storage.StorageOptions;
247+
248+
import java.nio.ByteBuffer;
249+
import java.nio.channels.WritableByteChannel;
250+
251+
Storage storage = StorageOptions.defaultInstance().service();
252+
BlobId blobId = BlobId.of("bucket", "blob_name");
253+
Blob blob = Blob.load(storage, blobId);
254+
if (blob == null) {
255+
BlobInfo blobInfo = BlobInfo.builder(blobId).contentType("text/plain").build();
256+
storage.create(blobInfo, "Hello, Cloud Storage!".getBytes(UTF_8));
251257
} else {
252-
System.out.println("Loading data into table " + tableId);
253-
LoadJobInfo loadJob = LoadJobInfo.of(tableId, "gs://bucket/path");
254-
loadJob = bigquery.create(loadJob);
255-
while (loadJob.status().state() != JobStatus.State.DONE) {
256-
Thread.sleep(1000L);
257-
loadJob = bigquery.getJob(loadJob.jobId());
258-
}
259-
if (loadJob.status().error() != null) {
260-
System.out.println("Job completed with errors");
261-
} else {
262-
System.out.println("Job succeeded");
263-
}
258+
System.out.println("Updating content for " + blobId.name());
259+
byte[] prevContent = blob.content();
260+
System.out.println(new String(prevContent, UTF_8));
261+
WritableByteChannel channel = blob.writer();
262+
channel.write(ByteBuffer.wrap("Updated content".getBytes(UTF_8)));
263+
channel.close();
264264
}
265265
```
266266

‎gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/FormatOptions.java‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,8 @@
1616

1717
package com.google.gcloud.bigquery;
1818

19+
import static com.google.common.base.Preconditions.checkNotNull;
20+
1921
import com.google.common.base.MoreObjects;
2022

2123
import java.io.Serializable;
@@ -85,7 +87,7 @@ public static FormatOptions datastoreBackup() {
8587
* Default options for the provided format.
8688
*/
8789
public static FormatOptions of(String format) {
88-
if (format.equals(CSV)) {
90+
if (checkNotNull(format).equals(CSV)) {
8991
return csv();
9092
}
9193
return new FormatOptions(format);

‎gcloud-java-examples/README.md‎

Lines changed: 27 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,27 @@ To run examples from your command line:
3939

4040
4. Run an example using Maven from command line.
4141

42+
Here's an example run of `BigQueryExample`.
43+
44+
Before running the example, go to the [Google Developers Console][developers-console] to ensure
45+
that BigQuery API is enabled. You can upload a CSV file `my_csv_file` to the `my_bucket` bucket
46+
(replace `my_csv_file` and `my_bucket` with actual file and bucket names) using the GCS
47+
[web browser](https://console.developers.google.com/storage/browser). The CSV file will be used to
48+
load data into a BigQuery table and should look something like:
49+
```csv
50+
value1
51+
value2
52+
value3
53+
```
54+
Then you are ready to run the following example:
55+
```
56+
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="create dataset new_dataset_id"
57+
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="create table new_dataset_id new_table_id field_name:string"
58+
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="list tables new_dataset_id"
59+
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="load new_dataset_id new_table_id CSV gs://my_bucket/my_csv_file"
60+
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="query 'select * from new_dataset_id.new_table_id'"
61+
```
62+
4263
Here's an example run of `DatastoreExample`.
4364

4465
Note that you have to enable the Google Cloud Datastore API on the [Google Developers Console][developers-console] before running the following commands.
@@ -48,16 +69,6 @@ To run examples from your command line:
4869
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.DatastoreExample" -Dexec.args="my_name delete"
4970
```
5071

51-
Here's an example run of `StorageExample`.
52-
53-
Before running the example, go to the [Google Developers Console][developers-console] to ensure that Google Cloud Storage API is enabled and that you have a bucket. Also ensure that you have a test file (`test.txt` is chosen here) to upload to Cloud Storage stored locally on your machine.
54-
```
55-
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="upload /path/to/test.txt <bucket_name>"
56-
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="list <bucket_name>"
57-
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="download <bucket_name> test.txt"
58-
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="delete <bucket_name> test.txt"
59-
```
60-
6172
Here's an example run of `ResourceManagerExample`.
6273

6374
Be sure to change the placeholder project ID "my-project-id" with your own globally unique project ID.
@@ -67,14 +78,14 @@ To run examples from your command line:
6778
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.ResourceManagerExample" -Dexec.args="get my-project-id"
6879
```
6980

70-
Here's an example run of `BigQueryExample`.
81+
Here's an example run of `StorageExample`.
7182

72-
Before running the example, go to the [Google Developers Console][developers-console] to ensure that Google Cloud BigQuery API is enabled.
83+
Before running the example, go to the [Google Developers Console][developers-console] to ensure that Google Cloud Storage API is enabled and that you have a bucket. Also ensure that you have a test file (`test.txt` is chosen here) to upload to Cloud Storage stored locally on your machine.
7384
```
74-
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="create dataset new_dataset_id"
75-
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="create table new_dataset_id new_table_id field_name:string"
76-
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="list tables new_dataset_id"
77-
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="query 'select * from new_dataset_id.new_table_id'"
85+
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="upload /path/to/test.txt <bucket_name>"
86+
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="list <bucket_name>"
87+
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="download <bucket_name> test.txt"
88+
$mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="delete <bucket_name> test.txt"
7889
```
7990

8091
Troubleshooting

0 commit comments

Comments
 (0)