Skip to content

Commit e05e401

Browse files
mziccardmziccard
committed
---
yaml --- r: 2425 b: refs/heads/update-datastore c: 7a72399 h: refs/heads/master i: 2423: 8def21f
1 parent 8d5ffbf commit e05e401

8 files changed

Lines changed: 96 additions & 73 deletions

File tree

‎[refs]‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ refs/heads/gh-pages: 4e0561bb4504bf647db669a14417b2b2c87ba45d
55
refs/heads/bigquery: 762fa5830e6c398c0396177e3e7fd243bd62cfc3
66
refs/heads/pubsub-alpha: 1a0e970f265af871e02274085b9662b3fe29058b
77
refs/heads/resource-manager: ebf4adc5ee835cd2086c4ac5b4e78d01a5a005a7
8-
refs/heads/update-datastore: 2de9135149e57cec18b070028452db1efef03125
8+
refs/heads/update-datastore: 7a723996f7c352fd93ceae91f41e474de256873d
99
refs/tags/0.0.9: 22f1839238f66c39e67ed4dfdcd273b1ae2e8444
1010
refs/tags/v0.0.10: 207ebd2a3472fddee69fe1298eb90429e3306efd
1111
refs/tags/v0.0.11: ffbfba48a6426ff63c08ff2117e58681f251fbf2

‎branches/update-datastore/README.md‎

Lines changed: 2 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -45,61 +45,15 @@ Example Applications
4545
- [`StorageExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/StorageExample.java) - A simple command line interface providing some of Cloud Storage's functionality
4646
- Read more about using this application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/StorageExample.html).
4747

48-
Specifying a Project ID
49-
-----------------------
50-
51-
Most `gcloud-java` libraries require a project ID. There are multiple ways to specify this project ID.
52-
53-
1. When using `gcloud-java` libraries from within Compute/App Engine, there's no need to specify a project ID. It is automatically inferred from the production environment.
54-
2. When using `gcloud-java` elsewhere, you can do one of the following:
55-
* Supply the project ID when building the service options. For example, to use Datastore from a project with ID "PROJECT_ID", you can write:
56-
57-
```java
58-
Datastore datastore = DatastoreOptions.builder().projectId("PROJECT_ID").build().service();
59-
```
60-
* Specify the environment variable `GCLOUD_PROJECT` to be your desired project ID.
61-
* Set the project ID using the [Google Cloud SDK](https://cloud.google.com/sdk/?hl=en). To use the SDK, [download the SDK](https://cloud.google.com/sdk/?hl=en) if you haven't already, and set the project ID from the command line. For example:
62-
63-
```
64-
gcloud config set project PROJECT_ID
65-
```
66-
67-
`gcloud-java` determines the project ID from the following sources in the listed order, stopping once it finds a value:
68-
69-
1. Project ID supplied when building the service options
70-
2. Project ID specified by the environment variable `GCLOUD_PROJECT`
71-
3. App Engine project ID
72-
4. Compute Engine project ID
73-
5. Google Cloud SDK project ID
74-
7548
Authentication
7649
--------------
7750

7851
There are multiple ways to authenticate to use Google Cloud services.
7952

8053
1. When using `gcloud-java` libraries from within Compute/App Engine, no additional authentication steps are necessary.
8154
2. When using `gcloud-java` libraries elsewhere, there are two options:
82-
* [Generate a JSON service account key](https://cloud.google.com/storage/docs/authentication?hl=en#service_accounts). After downloading that key, you must do one of the following:
83-
* Define the environment variable GOOGLE_APPLICATION_CREDENTIALS to be the location of the key. For example:
84-
```bash
85-
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/my/key.json
86-
```
87-
* Supply the JSON credentials file when building the service options. For example, this Storage object has the necessary permissions to interact with your Google Cloud Storage data:
88-
```java
89-
Storage storage = StorageOptions.builder()
90-
.authCredentials(AuthCredentials.createForJson(new FileInputStream("/path/to/my/key.json"))
91-
.build()
92-
.service();
93-
```
94-
* If running locally for development/testing, you can use use Google Cloud SDK. Download the SDK if you haven't already, then login using the SDK (`gcloud auth login` in command line). Be sure to set your project ID as described above.
95-
96-
`gcloud-java` looks for credentials in the following order, stopping once it finds credentials:
97-
98-
1. Credentials supplied when building the service options
99-
2. App Engine credentials
100-
3. Key file pointed to by the GOOGLE_APPLICATION_CREDENTIALS environment variable
101-
4. Google Cloud SDK credentials
102-
5. Compute Engine credentials
55+
* [Generate a JSON service account key](https://cloud.google.com/storage/docs/authentication?hl=en#service_accounts). Supply a path to the downloaded JSON credentials file when building the options supplied to datastore/storage constructor.
56+
* If running locally for development/testing, you can use use [Google Cloud SDK](https://cloud.google.com/sdk/?hl=en). To use the SDK authentication, [download the SDK](https://cloud.google.com/sdk/?hl=en) if you haven't already. Then login using the SDK (`gcloud auth login` in command line), and set your current project using `gcloud config set project PROJECT_ID`.
10357

10458
Google Cloud Datastore
10559
----------------------

‎branches/update-datastore/gcloud-java-core/pom.xml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,12 +20,12 @@
2020
<dependency>
2121
<groupId>com.google.auth</groupId>
2222
<artifactId>google-auth-library-credentials</artifactId>
23-
<version>0.3.0</version>
23+
<version>0.1.0</version>
2424
</dependency>
2525
<dependency>
2626
<groupId>com.google.auth</groupId>
2727
<artifactId>google-auth-library-oauth2-http</artifactId>
28-
<version>0.3.0</version>
28+
<version>0.1.0</version>
2929
</dependency>
3030
<dependency>
3131
<groupId>com.google.http-client</groupId>

‎branches/update-datastore/gcloud-java-core/src/main/java/com/google/gcloud/Page.java‎

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -16,20 +16,30 @@
1616

1717
package com.google.gcloud;
1818

19+
import java.util.Iterator;
20+
1921
/**
2022
* Interface for Google Cloud paginated results.
2123
*
2224
* <p>
23-
* A typical {@code Page} usage:
25+
* Use {@code Page} to iterate through all values (also in next pages):
26+
* <pre> {@code
27+
* Page<T> page = ...; // get a Page<T> instance
28+
* Iterator<T> iterator = page.iterateAll();
29+
* while (iterator.hasNext()) {
30+
* T value = iterator.next();
31+
* // do something with value
32+
* }}</pre>
33+
* <p>
34+
* Or handle pagination explicitly:
2435
* <pre> {@code
2536
* Page<T> page = ...; // get a Page<T> instance
2637
* while (page != null) {
2738
* for (T value : page.values()) {
2839
* // do something with value
2940
* }
3041
* page = page.nextPage();
31-
* }
32-
* }</pre>
42+
* }}</pre>
3343
*/
3444
public interface Page<T> {
3545

@@ -38,6 +48,12 @@ public interface Page<T> {
3848
*/
3949
Iterable<T> values();
4050

51+
/**
52+
* Returns an iterator for all values, possibly also in the next pages. Once current page's values
53+
* are traversed the iterator fetches next page, if any.
54+
*/
55+
Iterator<T> iterateAll();
56+
4157
/**
4258
* Returns the cursor for the nextPage or {@code null} if no more results.
4359
*/

‎branches/update-datastore/gcloud-java-core/src/main/java/com/google/gcloud/PageImpl.java‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,11 @@
1616

1717
package com.google.gcloud;
1818

19+
import com.google.common.collect.AbstractIterator;
20+
1921
import java.io.Serializable;
2022
import java.util.Collections;
23+
import java.util.Iterator;
2124
import java.util.Objects;
2225

2326
/**
@@ -35,6 +38,33 @@ public interface NextPageFetcher<T> extends Serializable {
3538
Page<T> nextPage();
3639
}
3740

41+
static class PageIterator<T> extends AbstractIterator<T> {
42+
43+
private Iterator<T> currentPageIterator;
44+
private Page<T> currentPage;
45+
46+
PageIterator(Page<T> currentPage) {
47+
this.currentPageIterator = currentPage.values().iterator();
48+
this.currentPage = currentPage;
49+
}
50+
51+
@Override
52+
protected T computeNext() {
53+
if (currentPageIterator.hasNext()) {
54+
return currentPageIterator.next();
55+
}
56+
Page<T> nextPage = currentPage.nextPage();
57+
if (nextPage != null) {
58+
currentPage = nextPage;
59+
currentPageIterator = currentPage.values().iterator();
60+
if (currentPageIterator.hasNext()) {
61+
return currentPageIterator.next();
62+
}
63+
}
64+
return endOfData();
65+
}
66+
}
67+
3868
/**
3969
* Creates a {@code PageImpl} object. In order for the object to be serializable the {@code
4070
* results} parameter must be serializable.
@@ -50,6 +80,11 @@ public Iterable<T> values() {
5080
return results == null ? Collections.EMPTY_LIST : results;
5181
}
5282

83+
@Override
84+
public Iterator<T> iterateAll() {
85+
return new PageIterator<T>(this);
86+
}
87+
5388
@Override
5489
public String nextPageCursor() {
5590
return cursor;

‎branches/update-datastore/gcloud-java-core/src/test/java/com/google/gcloud/PageImplTest.java‎

Lines changed: 24 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -26,21 +26,38 @@
2626

2727
public class PageImplTest {
2828

29+
private static final ImmutableList<String> VALUES = ImmutableList.of("1", "2");
30+
private static final ImmutableList<String> NEXT_VALUES = ImmutableList.of("3", "4");
31+
private static final ImmutableList<String> ALL_VALUES = ImmutableList.<String>builder()
32+
.addAll(VALUES)
33+
.addAll(NEXT_VALUES)
34+
.build();
35+
2936
@Test
30-
public void testPage() throws Exception {
31-
ImmutableList<String> values = ImmutableList.of("1", "2");
32-
final PageImpl<String> nextResult =
33-
new PageImpl<>(null, "c", Collections.<String>emptyList());
37+
public void testPage() {
38+
final PageImpl<String> nextResult = new PageImpl<>(null, "c", NEXT_VALUES);
3439
PageImpl.NextPageFetcher<String> fetcher = new PageImpl.NextPageFetcher<String>() {
35-
3640
@Override
3741
public PageImpl<String> nextPage() {
3842
return nextResult;
3943
}
4044
};
41-
PageImpl<String> result = new PageImpl<>(fetcher, "c", values);
45+
PageImpl<String> result = new PageImpl<>(fetcher, "c", VALUES);
4246
assertEquals(nextResult, result.nextPage());
4347
assertEquals("c", result.nextPageCursor());
44-
assertEquals(values, ImmutableList.copyOf(result.values().iterator()));
48+
assertEquals(VALUES, result.values());
49+
}
50+
51+
@Test
52+
public void testIterateAll() {
53+
final PageImpl<String> nextResult = new PageImpl<>(null, "c", NEXT_VALUES);
54+
PageImpl.NextPageFetcher<String> fetcher = new PageImpl.NextPageFetcher<String>() {
55+
@Override
56+
public PageImpl<String> nextPage() {
57+
return nextResult;
58+
}
59+
};
60+
PageImpl<String> result = new PageImpl<>(fetcher, "c", VALUES);
61+
assertEquals(ALL_VALUES, ImmutableList.copyOf(result.iterateAll()));
4562
}
4663
}

‎branches/update-datastore/gcloud-java-examples/src/main/java/com/google/gcloud/examples/StorageExample.java‎

Lines changed: 7 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,7 @@
5353
import java.security.cert.CertificateException;
5454
import java.util.Arrays;
5555
import java.util.HashMap;
56+
import java.util.Iterator;
5657
import java.util.List;
5758
import java.util.Map;
5859
import java.util.concurrent.TimeUnit;
@@ -214,12 +215,9 @@ String parse(String... args) {
214215
public void run(Storage storage, String bucketName) {
215216
if (bucketName == null) {
216217
// list buckets
217-
Page<BucketInfo> bucketPage = storage.list();
218-
while (bucketPage != null) {
219-
for (BucketInfo b : bucketPage.values()) {
220-
System.out.println(b);
221-
}
222-
bucketPage = bucketPage.nextPage();
218+
Iterator<BucketInfo> bucketInfoIterator = storage.list().iterateAll();
219+
while (bucketInfoIterator.hasNext()) {
220+
System.out.println(bucketInfoIterator.next());
223221
}
224222
} else {
225223
// list a bucket's blobs
@@ -228,12 +226,9 @@ public void run(Storage storage, String bucketName) {
228226
System.out.println("No such bucket");
229227
return;
230228
}
231-
Page<Blob> blobPage = bucket.list();
232-
while (blobPage != null) {
233-
for (Blob b : blobPage.values()) {
234-
System.out.println(b.info());
235-
}
236-
blobPage = blobPage.nextPage();
229+
Iterator<Blob> blobIterator = bucket.list().iterateAll();
230+
while (blobIterator.hasNext()) {
231+
System.out.println(blobIterator.next().info());
237232
}
238233
}
239234
}

‎branches/update-datastore/gcloud-java-storage/src/test/java/com/google/gcloud/storage/RemoteGcsHelperTest.java‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@
3333
import java.io.InputStream;
3434
import java.nio.file.Files;
3535
import java.nio.file.Paths;
36+
import java.util.Iterator;
3637
import java.util.List;
3738
import java.util.UUID;
3839
import java.util.concurrent.ExecutionException;
@@ -90,6 +91,11 @@ public Page<BlobInfo> nextPage() {
9091
public Iterable<BlobInfo> values() {
9192
return BLOB_LIST;
9293
}
94+
95+
@Override
96+
public Iterator<BlobInfo> iterateAll() {
97+
return BLOB_LIST.iterator();
98+
}
9399
};
94100
private static String keyPath = "/does/not/exist/key." + UUID.randomUUID().toString() + ".json";
95101

0 commit comments

Comments
 (0)