Skip to content

Commit a4a9530

Browse files
author
Ajay Kannan
committed
---
yaml --- r: 1965 b: refs/heads/pubsub-alpha c: a07fbda h: refs/heads/master i: 1963: f2b8f6b
1 parent 053aa3c commit a4a9530

12 files changed

Lines changed: 156 additions & 150 deletions

File tree

[refs]

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ refs/heads/master: 689bbb466df4b2d5d2483d6edb8ac5c7c7f7c6fa
33
refs/heads/travis: e21ee7b88a5edc3f3d8c71f90c3fc32abf7e8dd6
44
refs/heads/gh-pages: 4e0561bb4504bf647db669a14417b2b2c87ba45d
55
refs/heads/bigquery: 762fa5830e6c398c0396177e3e7fd243bd62cfc3
6-
refs/heads/pubsub-alpha: bbf3cb62911c50e287087664816e92b2950158d8
6+
refs/heads/pubsub-alpha: a07fbda2724b3faef46b43a251aea0ac9b42625a
77
refs/heads/resource-manager: ebf4adc5ee835cd2086c4ac5b4e78d01a5a005a7
88
refs/heads/update-datastore: 482954f2c5055231e5b3122ea91d2ba00ce8187c
99
refs/tags/0.0.9: 22f1839238f66c39e67ed4dfdcd273b1ae2e8444

branches/pubsub-alpha/gcloud-java-datastore/README.md

Lines changed: 122 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -56,33 +56,136 @@ Cloud Datastore for your project.
5656
See the ``gcloud-java`` API [datastore documentation][datastore-api] to learn how to interact
5757
with the Cloud Datastore using this Client Library.
5858

59-
Here is a code snippet showing a simple usage example from within Compute/App Engine. Note that you must [supply credentials](https://github.com/GoogleCloudPlatform/gcloud-java#authentication) and a project ID if running this snippet elsewhere.
59+
Getting Started
60+
---------------
61+
#### Prerequisites
62+
For this tutorial, you will need a [Google Developers Console](https://console.developers.google.com/) project with the Datastore API enabled. [Follow these instructions](https://cloud.google.com/docs/authentication#preparation) to get your project set up. You will also need to set up the local development environment by [installing the Google Cloud SDK](https://cloud.google.com/sdk/) and running the following commands in command line: `gcloud auth login` and `gcloud config set project [YOUR PROJECT ID]`.
63+
64+
#### Installation and setup
65+
You'll need to obtain the `gcloud-java-datastore` library. See the [Quickstart](#quickstart) section to add `gcloud-java-datastore` as a dependency in your code.
66+
67+
#### Creating an authorized service object
68+
To make authenticated requests to Google Cloud Datastore, you must create a service object with credentials. You can then make API calls by calling methods on the Datastore service object. The simplest way to authenticate is to use [Application Default Credentials](https://developers.google.com/identity/protocols/application-default-credentials). These credentials are automatically inferred from your environment, so you only need the following code to create your service object:
6069

6170
```java
6271
import com.google.gcloud.datastore.Datastore;
6372
import com.google.gcloud.datastore.DatastoreOptions;
64-
import com.google.gcloud.datastore.DateTime;
73+
74+
Datastore datastore = DatastoreOptions.defaultInstance().service();
75+
```
76+
77+
For other authentication options, see the [Authentication](https://github.com/GoogleCloudPlatform/gcloud-java#authentication) page.
78+
79+
#### Storing data
80+
Objects in Datastore are known as entities. Entities are grouped by "kind" and have keys for easy access. In this code snippet, we will create a new entity representing a person and store that data by the person's name. First, add the following imports at the top of your file:
81+
82+
```java
6583
import com.google.gcloud.datastore.Entity;
6684
import com.google.gcloud.datastore.Key;
6785
import com.google.gcloud.datastore.KeyFactory;
86+
```
6887

69-
Datastore datastore = DatastoreOptions.defaultInstance().service();
70-
KeyFactory keyFactory = datastore.newKeyFactory().kind(KIND);
71-
Key key = keyFactory.newKey(keyName);
72-
Entity entity = datastore.get(key);
73-
if (entity == null) {
74-
entity = Entity.builder(key)
75-
.set("name", "John Do")
76-
.set("age", 30)
77-
.set("access_time", DateTime.now())
78-
.build();
79-
datastore.put(entity);
80-
} else {
81-
System.out.println("Updating access_time for " + entity.getString("name"));
82-
entity = Entity.builder(entity)
83-
.set("access_time", DateTime.now())
84-
.build();
85-
datastore.update(entity);
88+
Then add the following code to put an entity in Datastore.
89+
90+
```java
91+
KeyFactory keyFactory = datastore.newKeyFactory().kind("Person");
92+
Key key = keyFactory.newKey("John Doe");
93+
Entity entity = Entity.builder(key)
94+
.set("age", 30)
95+
.set("favorite_food", "pizza")
96+
.build();
97+
datastore.put(entity);
98+
```
99+
100+
Later, if you want to get this entity back, add the following to your code:
101+
102+
```java
103+
Entity johnEntity = datastore.get("John Doe");
104+
```
105+
106+
#### Running a query
107+
In addition to retrieving entities by their keys, you can perform queries to retrieve entities by the values of their properties. A typical query includes an entity kind, filters to select entities with matching values, and sort orders to sequence the results. `gcloud-java-datastore` supports two types of queries: `StructuredQuery` (that allows you to construct query elements) and `GqlQuery` (which operates using [GQL syntax](https://cloud.google.com/datastore/docs/apis/gql/gql_reference)) in string format. In this tutorial, we will use a simple `StructuredQuery`.
108+
109+
Suppose that you've added more people to Datastore, and now you want to find all people whose favorite food is pizza. Import the following:
110+
111+
```java
112+
import com.google.gcloud.datastore.Query;
113+
import com.google.gcloud.datastore.QueryResults;
114+
import com.google.gcloud.datastore.StructuredQuery;
115+
import com.google.gcloud.datastore.StructuredQuery.PropertyFilter;
116+
```
117+
118+
Then add the following code to your program:
119+
120+
```java
121+
Query<Entity> query = Query.entityQueryBuilder()
122+
.kind("Person")
123+
.filter(PropertyFilter.eq("favorite_food", "pizza"))
124+
.build();
125+
QueryResults<Entity> results = datastore.run(query);
126+
while (results.hasNext()) {
127+
Entity currentEntity = results.next();
128+
}
129+
```
130+
131+
#### Complete source code
132+
133+
Here we put together all the code shown above into one program. This program assumes that you are running on Compute Engine or from your own desktop. To run this example on App Engine, simply move the code from the main method to your application's servlet class.
134+
135+
```java
136+
import com.google.gcloud.datastore.Datastore;
137+
import com.google.gcloud.datastore.DatastoreOptions;
138+
import com.google.gcloud.datastore.Entity;
139+
import com.google.gcloud.datastore.Key;
140+
import com.google.gcloud.datastore.KeyFactory;
141+
import com.google.gcloud.datastore.Query;
142+
import com.google.gcloud.datastore.QueryResults;
143+
import com.google.gcloud.datastore.StructuredQuery;
144+
import com.google.gcloud.datastore.StructuredQuery.PropertyFilter;
145+
146+
public class GcloudJavaDatastoreExample {
147+
148+
public static void main(String[] args) {
149+
// Create datastore service object.
150+
// By default, credentials are inferred from the runtime environment.
151+
Datastore datastore = DatastoreOptions.defaultInstance().service();
152+
153+
// Add an entity to Datastore
154+
KeyFactory keyFactory = datastore.newKeyFactory().kind("Person");
155+
Key key = keyFactory.newKey("John Doe");
156+
Entity entity = Entity.builder(key)
157+
.set("age", 51)
158+
.set("favorite_food", "pizza")
159+
.build();
160+
datastore.put(entity);
161+
162+
// Get an entity from Datastore
163+
Entity johnEntity = datastore.get(key);
164+
165+
// Add a couple more entities to make the query results more interesting
166+
Key janeKey = keyFactory.newKey("Jane Doe");
167+
Entity janeEntity = Entity.builder(janeKey)
168+
.set("age", 44)
169+
.set("favorite_food", "pizza")
170+
.build();
171+
Key joeKey = keyFactory.newKey("Joe Shmoe");
172+
Entity joeEntity = Entity.builder(joeKey)
173+
.set("age", 27)
174+
.set("favorite_food", "sushi")
175+
.build();
176+
datastore.put(janeEntity, joeEntity);
177+
178+
// Run a query
179+
Query<Entity> query = Query.entityQueryBuilder()
180+
.kind("Person")
181+
.filter(PropertyFilter.eq("favorite_food", "pizza"))
182+
.build();
183+
QueryResults<Entity> results = datastore.run(query);
184+
while (results.hasNext()) {
185+
Entity currentEntity = results.next();
186+
// Do something using the entity. (e.g. send an invite a pizza party)
187+
}
188+
}
86189
}
87190
```
88191

branches/pubsub-alpha/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/FullEntity.java

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,7 @@
1919
import com.google.api.services.datastore.DatastoreV1;
2020

2121
/**
22-
* A full entity is a {@link BaseEntity} that holds all the properties associated with a
23-
* Datastore entity (as opposed to {@link ProjectionEntity}).
22+
* A full entity is a {@link BaseEntity} that with a complete set of properties.
2423
*/
2524
public class FullEntity<K extends IncompleteKey> extends BaseEntity<K> {
2625

branches/pubsub-alpha/gcloud-java-storage/src/main/java/com/google/gcloud/spi/DefaultStorageRpc.java

Lines changed: 7 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -429,8 +429,8 @@ public void onFailure(GoogleJsonError e, HttpHeaders responseHeaders) {
429429
}
430430

431431
@Override
432-
public Tuple<String, byte[]> read(StorageObject from, Map<Option, ?> options, long position,
433-
int bytes) throws StorageException {
432+
public byte[] read(StorageObject from, Map<Option, ?> options, long position, int bytes)
433+
throws StorageException {
434434
try {
435435
Get req = storage.objects()
436436
.get(from.getBucket(), from.getName())
@@ -439,13 +439,12 @@ public Tuple<String, byte[]> read(StorageObject from, Map<Option, ?> options, lo
439439
.setIfMetagenerationNotMatch(IF_METAGENERATION_NOT_MATCH.getLong(options))
440440
.setIfGenerationMatch(IF_GENERATION_MATCH.getLong(options))
441441
.setIfGenerationNotMatch(IF_GENERATION_NOT_MATCH.getLong(options));
442-
StringBuilder range = new StringBuilder();
443-
range.append("bytes=").append(position).append("-").append(position + bytes - 1);
444-
req.getRequestHeaders().setRange(range.toString());
442+
MediaHttpDownloader downloader = req.getMediaHttpDownloader();
443+
downloader.setContentRange(position, Ints.checkedCast(position + bytes - 1));
444+
downloader.setDirectDownloadEnabled(true);
445445
ByteArrayOutputStream output = new ByteArrayOutputStream();
446-
req.executeMedia().download(output);
447-
String etag = req.getLastResponseHeaders().getETag();
448-
return Tuple.of(etag, output.toByteArray());
446+
req.executeMediaAndDownloadTo(output);
447+
return output.toByteArray();
449448
} catch (IOException ex) {
450449
throw translate(ex);
451450
}

branches/pubsub-alpha/gcloud-java-storage/src/main/java/com/google/gcloud/spi/StorageRpc.java

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -259,7 +259,7 @@ StorageObject compose(Iterable<StorageObject> sources, StorageObject target,
259259
byte[] load(StorageObject storageObject, Map<Option, ?> options)
260260
throws StorageException;
261261

262-
Tuple<String, byte[]> read(StorageObject from, Map<Option, ?> options, long position, int bytes)
262+
byte[] read(StorageObject from, Map<Option, ?> options, long position, int bytes)
263263
throws StorageException;
264264

265265
String open(StorageObject object, Map<Option, ?> options) throws StorageException;

branches/pubsub-alpha/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BlobReadChannel.java

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,8 +26,9 @@
2626
/**
2727
* A channel for reading data from a Google Cloud Storage object.
2828
*
29-
* Implementations of this class may buffer data internally to reduce remote calls. This interface
30-
* implements {@link Restorable} to allow saving the reader's state to continue reading afterwards.
29+
* Implementations of this class may buffer data internally to reduce remote calls.
30+
*
31+
* This class is @{link Serializable}, which allows incremental reads.
3132
*/
3233
public interface BlobReadChannel extends ReadableByteChannel, Closeable,
3334
Restorable<BlobReadChannel> {

branches/pubsub-alpha/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BlobReadChannelImpl.java

Lines changed: 4 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,6 @@
2323
import com.google.gcloud.RestorableState;
2424
import com.google.gcloud.RetryHelper;
2525
import com.google.gcloud.spi.StorageRpc;
26-
import com.google.gcloud.spi.StorageRpc.Tuple;
2726

2827
import java.io.IOException;
2928
import java.io.Serializable;
@@ -42,7 +41,6 @@ class BlobReadChannelImpl implements BlobReadChannel {
4241
private final StorageOptions serviceOptions;
4342
private final BlobId blob;
4443
private final Map<StorageRpc.Option, ?> requestOptions;
45-
private String lastEtag;
4644
private int position;
4745
private boolean isOpen;
4846
private boolean endOfStream;
@@ -119,19 +117,12 @@ public int read(ByteBuffer byteBuffer) throws IOException {
119117
}
120118
final int toRead = Math.max(byteBuffer.remaining(), chunkSize);
121119
try {
122-
Tuple<String, byte[]> result = runWithRetries(new Callable<Tuple<String, byte[]>>() {
120+
buffer = runWithRetries(new Callable<byte[]>() {
123121
@Override
124-
public Tuple<String, byte[]> call() {
122+
public byte[] call() {
125123
return storageRpc.read(storageObject, requestOptions, position, toRead);
126124
}
127125
}, serviceOptions.retryParams(), StorageImpl.EXCEPTION_HANDLER);
128-
if (lastEtag != null && !Objects.equals(result.x(), lastEtag)) {
129-
StringBuilder messageBuilder = new StringBuilder();
130-
messageBuilder.append("Blob ").append(blob).append(" was updated while reading");
131-
throw new StorageException(0, messageBuilder.toString(), false);
132-
}
133-
lastEtag = result.x();
134-
buffer = result.y();
135126
} catch (RetryHelper.RetryHelperException e) {
136127
throw StorageException.translateAndThrow(e);
137128
}
@@ -161,7 +152,6 @@ static class StateImpl implements RestorableState<BlobReadChannel>, Serializable
161152
private final StorageOptions serviceOptions;
162153
private final BlobId blob;
163154
private final Map<StorageRpc.Option, ?> requestOptions;
164-
private final String lastEtag;
165155
private final int position;
166156
private final boolean isOpen;
167157
private final boolean endOfStream;
@@ -171,7 +161,6 @@ static class StateImpl implements RestorableState<BlobReadChannel>, Serializable
171161
this.serviceOptions = builder.serviceOptions;
172162
this.blob = builder.blob;
173163
this.requestOptions = builder.requestOptions;
174-
this.lastEtag = builder.lastEtag;
175164
this.position = builder.position;
176165
this.isOpen = builder.isOpen;
177166
this.endOfStream = builder.endOfStream;
@@ -182,7 +171,6 @@ static class Builder {
182171
private final StorageOptions serviceOptions;
183172
private final BlobId blob;
184173
private final Map<StorageRpc.Option, ?> requestOptions;
185-
private String lastEtag;
186174
private int position;
187175
private boolean isOpen;
188176
private boolean endOfStream;
@@ -194,11 +182,6 @@ private Builder(StorageOptions options, BlobId blob, Map<StorageRpc.Option, ?> r
194182
this.requestOptions = reqOptions;
195183
}
196184

197-
Builder lastEtag(String lastEtag) {
198-
this.lastEtag = lastEtag;
199-
return this;
200-
}
201-
202185
Builder position(int position) {
203186
this.position = position;
204187
return this;
@@ -232,7 +215,6 @@ static Builder builder(
232215
@Override
233216
public BlobReadChannel restore() {
234217
BlobReadChannelImpl channel = new BlobReadChannelImpl(serviceOptions, blob, requestOptions);
235-
channel.lastEtag = lastEtag;
236218
channel.position = position;
237219
channel.isOpen = isOpen;
238220
channel.endOfStream = endOfStream;
@@ -242,8 +224,8 @@ public BlobReadChannel restore() {
242224

243225
@Override
244226
public int hashCode() {
245-
return Objects.hash(serviceOptions, blob, requestOptions, lastEtag, position, isOpen,
246-
endOfStream, chunkSize);
227+
return Objects.hash(serviceOptions, blob, requestOptions, position, isOpen, endOfStream,
228+
chunkSize);
247229
}
248230

249231
@Override
@@ -258,7 +240,6 @@ public boolean equals(Object obj) {
258240
return Objects.equals(this.serviceOptions, other.serviceOptions)
259241
&& Objects.equals(this.blob, other.blob)
260242
&& Objects.equals(this.requestOptions, other.requestOptions)
261-
&& Objects.equals(this.lastEtag, other.lastEtag)
262243
&& this.position == other.position
263244
&& this.isOpen == other.isOpen
264245
&& this.endOfStream == other.endOfStream

branches/pubsub-alpha/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BlobWriteChannel.java

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,8 +26,8 @@
2626
* A channel for writing data to a Google Cloud Storage object.
2727
*
2828
* Implementations of this class may further buffer data internally to reduce remote calls. Written
29-
* data will only be visible after calling {@link #close()}. This interface implements
30-
* {@link Restorable} to allow saving the writer's state to continue writing afterwards.
29+
* data will only be visible after calling {@link #close()}. This class is serializable, to allow
30+
* incremental writes.
3131
*/
3232
public interface BlobWriteChannel extends WritableByteChannel, Closeable,
3333
Restorable<BlobWriteChannel> {

branches/pubsub-alpha/gcloud-java-storage/src/main/java/com/google/gcloud/storage/Storage.java

Lines changed: 2 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,6 @@
3333
import java.io.InputStream;
3434
import java.io.Serializable;
3535
import java.net.URL;
36-
import java.nio.ByteBuffer;
3736
import java.util.Arrays;
3837
import java.util.Collections;
3938
import java.util.HashSet;
@@ -1406,29 +1405,14 @@ private static void checkContentType(BlobInfo blobInfo) throws IllegalArgumentEx
14061405
BatchResponse apply(BatchRequest batchRequest);
14071406

14081407
/**
1409-
* Return a channel for reading the blob's content. The blob's latest generation is read. If the
1410-
* blob changes while reading (i.e. {@link BlobInfo#etag()} changes), subsequent calls to
1411-
* {@code blobReadChannel.read(ByteBuffer)} may throw {@link StorageException}.
1412-
*
1413-
* <p>The {@link BlobSourceOption#generationMatch(long)} option can be provided to ensure that
1414-
* {@code blobReadChannel.read(ByteBuffer)} calls will throw {@link StorageException} if blob`s
1415-
* generation differs from the expected one.
1408+
* Return a channel for reading the blob's content.
14161409
*
14171410
* @throws StorageException upon failure
14181411
*/
14191412
BlobReadChannel reader(String bucket, String blob, BlobSourceOption... options);
14201413

14211414
/**
1422-
* Return a channel for reading the blob's content. If {@code blob.generation()} is set
1423-
* data corresponding to that generation is read. If {@code blob.generation()} is {@code null}
1424-
* the blob's latest generation is read. If the blob changes while reading (i.e.
1425-
* {@link BlobInfo#etag()} changes), subsequent calls to {@code blobReadChannel.read(ByteBuffer)}
1426-
* may throw {@link StorageException}.
1427-
*
1428-
* <p>The {@link BlobSourceOption#generationMatch()} and
1429-
* {@link BlobSourceOption#generationMatch(long)} options can be used to ensure that
1430-
* {@code blobReadChannel.read(ByteBuffer)} calls will throw {@link StorageException} if the
1431-
* blob`s generation differs from the expected one.
1415+
* Return a channel for reading the blob's content.
14321416
*
14331417
* @throws StorageException upon failure
14341418
*/

0 commit comments

Comments
 (0)