Skip to content

Commit eea926a

Browse files
mziccardmziccard
committed
---
yaml --- r: 2079 b: refs/heads/pubsub-alpha c: d299e06 h: refs/heads/master i: 2077: ace63d0 2075: 36e78ff 2071: ed54e9c 2063: 0b69450 2047: 9b1d5b2
1 parent 69a2fb1 commit eea926a

75 files changed

Lines changed: 15524 additions & 1 deletion

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎[refs]‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ refs/heads/master: 689bbb466df4b2d5d2483d6edb8ac5c7c7f7c6fa
33
refs/heads/travis: e21ee7b88a5edc3f3d8c71f90c3fc32abf7e8dd6
44
refs/heads/gh-pages: 4e0561bb4504bf647db669a14417b2b2c87ba45d
55
refs/heads/bigquery: 762fa5830e6c398c0396177e3e7fd243bd62cfc3
6-
refs/heads/pubsub-alpha: 9bcebd3da2bd48224b81f5353c9d7a44eb69effb
6+
refs/heads/pubsub-alpha: d299e0635aa984d5e7bc4cc0ed9b759b46bb50f6
77
refs/heads/resource-manager: ebf4adc5ee835cd2086c4ac5b4e78d01a5a005a7
88
refs/heads/update-datastore: 482954f2c5055231e5b3122ea91d2ba00ce8187c
99
refs/tags/0.0.9: 22f1839238f66c39e67ed4dfdcd273b1ae2e8444

‎branches/pubsub-alpha/gcloud-java-bigquery/README.md‎

Lines changed: 322 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,322 @@
1+
Google Cloud Java Client for BigQuery
2+
====================================
3+
4+
Java idiomatic client for [Google Cloud BigQuery] (https://cloud.google.com/bigquery).
5+
6+
[![Build Status](https://travis-ci.org/GoogleCloudPlatform/gcloud-java.svg?branch=master)](https://travis-ci.org/GoogleCloudPlatform/gcloud-java)
7+
[![Coverage Status](https://coveralls.io/repos/GoogleCloudPlatform/gcloud-java/badge.svg?branch=master)](https://coveralls.io/r/GoogleCloudPlatform/gcloud-java?branch=master)
8+
<!-- TODO(mziccard): add in the maven shield once the artifact is pushed to maven -->
9+
10+
- [Homepage] (https://googlecloudplatform.github.io/gcloud-java/)
11+
- [API Documentation] (http://googlecloudplatform.github.io/gcloud-java/apidocs/index.html?com/google/gcloud/bigquery/package-summary.html)
12+
13+
> Note: This client is a work-in-progress, and may occasionally
14+
> make backwards-incompatible changes.
15+
16+
Quickstart
17+
----------
18+
If you are using Maven, add this to your pom.xml file
19+
<!-- TODO(mziccard): add mvn dependency code -->
20+
21+
If you are using Gradle, add this to your dependencies
22+
<!-- TODO(mziccard): add gradle dependency code -->
23+
24+
If you are using SBT, add this to your dependencies
25+
<!-- TODO(mziccard): add sbt dependency code -->
26+
27+
Example Application
28+
-------------------
29+
30+
<!-- TODO(mziccard): add example application -->
31+
32+
Authentication
33+
--------------
34+
35+
See the [Authentication](https://github.com/GoogleCloudPlatform/gcloud-java#authentication) section in the base directory's README.
36+
37+
About Google Cloud BigQuery
38+
--------------------------
39+
40+
[Google Cloud BigQuery][cloud-bigquery] is a fully managed, NoOps, low cost data analytics service.
41+
Data can be streamed into BigQuery at millions of rows per second to enable real-time analysis.
42+
With BigQuery you can easily deploy Petabyte-scale Databases.
43+
44+
Be sure to activate the Google Cloud BigQuery API on the Developer's Console to use BigQuery from
45+
your project.
46+
47+
See the ``gcloud-java`` API [bigquery documentation][bigquery-api] to learn how to interact
48+
with Google Cloud BigQuery using this Client Library.
49+
50+
Getting Started
51+
---------------
52+
#### Prerequisites
53+
For this tutorial, you will need a
54+
[Google Developers Console](https://console.developers.google.com/) project with the BigQuery API
55+
enabled. You will need to [enable billing](https://support.google.com/cloud/answer/6158867?hl=en) to
56+
use Google Cloud BigQuery.
57+
[Follow these instructions](https://cloud.google.com/docs/authentication#preparation) to get your
58+
project set up. You will also need to set up the local development environment by [installing the
59+
Google Cloud SDK](https://cloud.google.com/sdk/) and running the following commands in command line:
60+
`gcloud auth login` and `gcloud config set project [YOUR PROJECT ID]`.
61+
62+
#### Installation and setup
63+
You'll need to obtain the `gcloud-java-bigquery` library. See the [Quickstart](#quickstart) section
64+
to add `gcloud-java-bigquery` as a dependency in your code.
65+
66+
#### Creating an authorized service object
67+
To make authenticated requests to Google Cloud BigQuery, you must create a service object with
68+
credentials. You can then make API calls by calling methods on the BigQuery service object. The
69+
simplest way to authenticate is to use
70+
[Application Default Credentials](https://developers.google.com/identity/protocols/application-default-credentials).
71+
These credentials are automatically inferred from your environment, so you only need the following
72+
code to create your service object:
73+
74+
```java
75+
import com.google.gcloud.bigquery.BigQuery;
76+
import com.google.gcloud.bigquery.BigQueryOptions;
77+
78+
BigQuery bigquery = BigQueryOptions.defaultInstance().service();
79+
```
80+
81+
For other authentication options, see the
82+
[Authentication](https://github.com/GoogleCloudPlatform/gcloud-java#authentication) page.
83+
84+
#### Creating a dataset
85+
With BigQuery you can create datasets. A dataset is a grouping mechanism that holds zero or more
86+
tables. Add the following import at the top of your file:
87+
88+
```java
89+
import com.google.gcloud.bigquery.DatasetInfo;
90+
```
91+
Then, to create the dataset, use the following code:
92+
93+
```java
94+
// Create a dataset
95+
String datasetId = "my_dataset_id";
96+
bigquery.create(DatasetInfo.builder(datasetId).build());
97+
```
98+
99+
#### Creating a table
100+
With BigQuery you can create different types of tables: normal tables with an associated schema,
101+
external tables backed by data stored on [Google Cloud Storage][cloud-storage] and view tables that
102+
are created from a BigQuery SQL query. In this code snippet we show how to create a normal table
103+
with only one string field. Add the following imports at the top of your file:
104+
105+
```java
106+
import com.google.gcloud.bigquery.BaseTableInfo;
107+
import com.google.gcloud.bigquery.Field;
108+
import com.google.gcloud.bigquery.Schema;
109+
import com.google.gcloud.bigquery.TableId;
110+
import com.google.gcloud.bigquery.TableInfo;
111+
```
112+
Then add the following code to create the table:
113+
114+
```java
115+
TableId tableId = TableId.of(datasetId, "my_table_id");
116+
// Table field definition
117+
Field stringField = Field.of("StringField", Field.Type.string());
118+
// Table schema definition
119+
Schema schema = Schema.of(stringField);
120+
// Create a table
121+
TableInfo createdTableInfo = bigquery.create(TableInfo.of(tableId, schema));
122+
```
123+
124+
#### Loading data into a table
125+
BigQuery provides several ways to load data into a table: streaming rows or loading data from a
126+
Google Cloud Storage file. In this code snippet we show how to stream rows into a table.
127+
Add the following imports at the top of your file:
128+
129+
```java
130+
import com.google.gcloud.bigquery.InsertAllRequest;
131+
import com.google.gcloud.bigquery.InsertAllResponse;
132+
133+
import java.util.HashMap;
134+
import java.util.Map;
135+
```
136+
Then add the following code to insert data:
137+
138+
```java
139+
Map<String, Object> firstRow = new HashMap<>();
140+
Map<String, Object> secondRow = new HashMap<>();
141+
firstRow.put("StringField", "value1");
142+
secondRow.put("StringField", "value2");
143+
// Create an insert request
144+
InsertAllRequest insertRequest = InsertAllRequest.builder(tableId)
145+
.addRow(firstRow)
146+
.addRow(secondRow)
147+
.build();
148+
// Insert rows
149+
InsertAllResponse insertResponse = bigquery.insertAll(insertRequest);
150+
// Check if errors occurred
151+
if (insertResponse.hasErrors()) {
152+
System.out.println("Errors occurred while inserting rows");
153+
}
154+
```
155+
156+
#### Querying data
157+
BigQuery enables querying data by running queries and waiting for the result. Queries can be run
158+
directly or through a Query Job. In this code snippet we show how to run a query directly and wait
159+
for the result. Add the following imports at the top of your file:
160+
161+
```java
162+
import com.google.gcloud.bigquery.FieldValue;
163+
import com.google.gcloud.bigquery.QueryRequest;
164+
import com.google.gcloud.bigquery.QueryResponse;
165+
166+
import java.util.Iterator;
167+
import java.util.List;
168+
```
169+
Then add the following code to run the query and wait for the result:
170+
171+
```java
172+
// Create a query request
173+
QueryRequest queryRequest =
174+
QueryRequest.builder("SELECT * FROM my_dataset_id.my_table_id")
175+
.maxWaitTime(60000L)
176+
.maxResults(1000L)
177+
.build();
178+
// Request query to be executed and wait for results
179+
QueryResponse queryResponse = bigquery.query(queryRequest);
180+
while (!queryResponse.jobComplete()) {
181+
Thread.sleep(1000L);
182+
queryResponse = bigquery.getQueryResults(queryResponse.jobId());
183+
}
184+
// Read rows
185+
Iterator<List<FieldValue>> rowIterator = queryResponse.result().iterateAll();
186+
System.out.println("Table rows:");
187+
while (rowIterator.hasNext()) {
188+
System.out.println(rowIterator.next());
189+
}
190+
```
191+
#### Complete source code
192+
193+
Here we put together all the code shown above into one program. This program assumes that you are
194+
running on Compute Engine or from your own desktop. To run this example on App Engine, simply move
195+
the code from the main method to your application's servlet class and change the print statements to
196+
display on your webpage.
197+
198+
```java
199+
import com.google.gcloud.bigquery.BaseTableInfo;
200+
import com.google.gcloud.bigquery.BigQuery;
201+
import com.google.gcloud.bigquery.BigQueryOptions;
202+
import com.google.gcloud.bigquery.DatasetInfo;
203+
import com.google.gcloud.bigquery.Field;
204+
import com.google.gcloud.bigquery.FieldValue;
205+
import com.google.gcloud.bigquery.InsertAllRequest;
206+
import com.google.gcloud.bigquery.InsertAllResponse;
207+
import com.google.gcloud.bigquery.QueryRequest;
208+
import com.google.gcloud.bigquery.QueryResponse;
209+
import com.google.gcloud.bigquery.Schema;
210+
import com.google.gcloud.bigquery.TableId;
211+
import com.google.gcloud.bigquery.TableInfo;
212+
213+
import java.util.HashMap;
214+
import java.util.Iterator;
215+
import java.util.List;
216+
import java.util.Map;
217+
218+
public class GcloudBigQueryExample {
219+
220+
public static void main(String[] args) throws InterruptedException {
221+
222+
// Create a service instance
223+
BigQuery bigquery = BigQueryOptions.defaultInstance().service();
224+
225+
// Create a dataset
226+
String datasetId = "my_dataset_id";
227+
bigquery.create(DatasetInfo.builder(datasetId).build());
228+
229+
TableId tableId = TableId.of(datasetId, "my_table_id");
230+
// Table field definition
231+
Field stringField = Field.of("StringField", Field.Type.string());
232+
// Table schema definition
233+
Schema schema = Schema.of(stringField);
234+
// Create a table
235+
TableInfo createdTableInfo = bigquery.create(TableInfo.of(tableId, schema));
236+
237+
// Define rows to insert
238+
Map<String, Object> firstRow = new HashMap<>();
239+
Map<String, Object> secondRow = new HashMap<>();
240+
firstRow.put("StringField", "value1");
241+
secondRow.put("StringField", "value2");
242+
// Create an insert request
243+
InsertAllRequest insertRequest = InsertAllRequest.builder(tableId)
244+
.addRow(firstRow)
245+
.addRow(secondRow)
246+
.build();
247+
// Insert rows
248+
InsertAllResponse insertResponse = bigquery.insertAll(insertRequest);
249+
// Check if errors occurred
250+
if (insertResponse.hasErrors()) {
251+
System.out.println("Errors occurred while inserting rows");
252+
}
253+
254+
// Create a query request
255+
QueryRequest queryRequest =
256+
QueryRequest.builder("SELECT * FROM my_dataset_id.my_table_id")
257+
.maxWaitTime(60000L)
258+
.maxResults(1000L)
259+
.build();
260+
// Request query to be executed and wait for results
261+
QueryResponse queryResponse = bigquery.query(queryRequest);
262+
while (!queryResponse.jobComplete()) {
263+
Thread.sleep(1000L);
264+
queryResponse = bigquery.getQueryResults(queryResponse.jobId());
265+
}
266+
// Read rows
267+
Iterator<List<FieldValue>> rowIterator = queryResponse.result().iterateAll();
268+
System.out.println("Table rows:");
269+
while (rowIterator.hasNext()) {
270+
System.out.println(rowIterator.next());
271+
}
272+
}
273+
}
274+
```
275+
276+
Troubleshooting
277+
---------------
278+
279+
To get help, follow the `gcloud-java` links in the `gcloud-*`[shared Troubleshooting document](https://github.com/GoogleCloudPlatform/gcloud-common/blob/master/troubleshooting/readme.md#troubleshooting).
280+
281+
Java Versions
282+
-------------
283+
284+
Java 7 or above is required for using this client.
285+
286+
Testing
287+
-------
288+
289+
This library has tools to help make tests for code using Cloud BigQuery.
290+
291+
See [TESTING] to read more about testing.
292+
293+
Versioning
294+
----------
295+
296+
This library follows [Semantic Versioning] (http://semver.org/).
297+
298+
It is currently in major version zero (``0.y.z``), which means that anything
299+
may change at any time and the public API should not be considered
300+
stable.
301+
302+
Contributing
303+
------------
304+
305+
Contributions to this library are always welcome and highly encouraged.
306+
307+
See [CONTRIBUTING] for more information on how to get started.
308+
309+
License
310+
-------
311+
312+
Apache 2.0 - See [LICENSE] for more information.
313+
314+
315+
[CONTRIBUTING]:https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/CONTRIBUTING.md
316+
[LICENSE]: https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/LICENSE
317+
[TESTING]: https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/TESTING.md#testing-code-that-uses-bigquery
318+
[cloud-platform]: https://cloud.google.com/
319+
320+
[cloud-bigquery]: https://cloud.google.com/bigquery/
321+
[cloud-storage]: https://cloud.google.com/storage/
322+
[bigquery-api]: http://googlecloudplatform.github.io/gcloud-java/apidocs/index.html?com/google/gcloud/bigquery/package-summary.html

‎branches/pubsub-alpha/gcloud-java-bigquery/pom.xml‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
<?xml version="1.0"?>
2+
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
3+
<modelVersion>4.0.0</modelVersion>
4+
<groupId>com.google.gcloud</groupId>
5+
<artifactId>gcloud-java-bigquery</artifactId>
6+
<packaging>jar</packaging>
7+
<name>GCloud Java bigquery</name>
8+
<description>
9+
Java idiomatic client for Google Cloud BigQuery.
10+
</description>
11+
<parent>
12+
<groupId>com.google.gcloud</groupId>
13+
<artifactId>gcloud-java-pom</artifactId>
14+
<version>0.1.1-SNAPSHOT</version>
15+
</parent>
16+
<properties>
17+
<site.installationModule>gcloud-java-bigquery</site.installationModule>
18+
</properties>
19+
<dependencies>
20+
<dependency>
21+
<groupId>${project.groupId}</groupId>
22+
<artifactId>gcloud-java-core</artifactId>
23+
<version>${project.version}</version>
24+
</dependency>
25+
<dependency>
26+
<groupId>${project.groupId}</groupId>
27+
<artifactId>gcloud-java-storage</artifactId>
28+
<version>${project.version}</version>
29+
<scope>test</scope>
30+
</dependency>
31+
<dependency>
32+
<groupId>com.google.apis</groupId>
33+
<artifactId>google-api-services-bigquery</artifactId>
34+
<version>v2-rev254-1.21.0</version>
35+
<scope>compile</scope>
36+
<exclusions>
37+
<exclusion>
38+
<groupId>com.google.guava</groupId>
39+
<artifactId>guava-jdk5</artifactId>
40+
</exclusion>
41+
</exclusions>
42+
</dependency>
43+
<dependency>
44+
<groupId>junit</groupId>
45+
<artifactId>junit</artifactId>
46+
<version>4.12</version>
47+
<scope>test</scope>
48+
</dependency>
49+
<dependency>
50+
<groupId>org.easymock</groupId>
51+
<artifactId>easymock</artifactId>
52+
<version>3.3</version>
53+
<scope>test</scope>
54+
</dependency>
55+
</dependencies>
56+
</project>

0 commit comments

Comments
 (0)