Skip to content

Commit 9613ce0

Browse files
committed
---
yaml --- r: 2501 b: refs/heads/update-datastore c: 78f8f28 h: refs/heads/master i: 2499: 9331ba1
1 parent 9a2f602 commit 9613ce0

4 files changed

Lines changed: 92 additions & 125 deletions

File tree

[refs]

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ refs/heads/gh-pages: 4e0561bb4504bf647db669a14417b2b2c87ba45d
55
refs/heads/bigquery: 762fa5830e6c398c0396177e3e7fd243bd62cfc3
66
refs/heads/pubsub-alpha: 1a0e970f265af871e02274085b9662b3fe29058b
77
refs/heads/resource-manager: ebf4adc5ee835cd2086c4ac5b4e78d01a5a005a7
8-
refs/heads/update-datastore: b48d1631ad7f7c6f9d7c40d93303cc79d45540a8
8+
refs/heads/update-datastore: 78f8f288f428618d4e5160407681adefd9c077e6
99
refs/tags/0.0.9: 22f1839238f66c39e67ed4dfdcd273b1ae2e8444
1010
refs/tags/v0.0.10: 207ebd2a3472fddee69fe1298eb90429e3306efd
1111
refs/tags/v0.0.11: ffbfba48a6426ff63c08ff2117e58681f251fbf2

branches/update-datastore/gcloud-java-datastore/README.md

Lines changed: 19 additions & 122 deletions
Original file line numberDiff line numberDiff line change
@@ -56,136 +56,33 @@ Cloud Datastore for your project.
5656
See the ``gcloud-java`` API [datastore documentation][datastore-api] to learn how to interact
5757
with the Cloud Datastore using this Client Library.
5858

59-
Getting Started
60-
---------------
61-
#### Prerequisites
62-
For this tutorial, you will need a [Google Developers Console](https://console.developers.google.com/) project with the Datastore API enabled. [Follow these instructions](https://cloud.google.com/docs/authentication#preparation) to get your project set up. You will also need to set up the local development environment by [installing the Google Cloud SDK](https://cloud.google.com/sdk/) and running the following commands in command line: `gcloud auth login` and `gcloud config set project [YOUR PROJECT ID]`.
63-
64-
#### Installation and setup
65-
You'll need to obtain the `gcloud-java-datastore` library. See the [Quickstart](#quickstart) section to add `gcloud-java-datastore` as a dependency in your code.
66-
67-
#### Creating an authorized service object
68-
To make authenticated requests to Google Cloud Datastore, you must create a service object with credentials. You can then make API calls by calling methods on the Datastore service object. The simplest way to authenticate is to use [Application Default Credentials](https://developers.google.com/identity/protocols/application-default-credentials). These credentials are automatically inferred from your environment, so you only need the following code to create your service object:
59+
Here is a code snippet showing a simple usage example from within Compute/App Engine. Note that you must [supply credentials](https://github.com/GoogleCloudPlatform/gcloud-java#authentication) and a project ID if running this snippet elsewhere.
6960

7061
```java
7162
import com.google.gcloud.datastore.Datastore;
7263
import com.google.gcloud.datastore.DatastoreOptions;
73-
74-
Datastore datastore = DatastoreOptions.defaultInstance().service();
75-
```
76-
77-
For other authentication options, see the [Authentication](https://github.com/GoogleCloudPlatform/gcloud-java#authentication) page.
78-
79-
#### Storing data
80-
Objects in Datastore are known as entities. Entities are grouped by "kind" and have keys for easy access. In this code snippet, we will create a new entity representing a person and store that data by the person's name. First, add the following imports at the top of your file:
81-
82-
```java
64+
import com.google.gcloud.datastore.DateTime;
8365
import com.google.gcloud.datastore.Entity;
8466
import com.google.gcloud.datastore.Key;
8567
import com.google.gcloud.datastore.KeyFactory;
86-
```
87-
88-
Then add the following code to put an entity in Datastore.
89-
90-
```java
91-
KeyFactory keyFactory = datastore.newKeyFactory().kind("Person");
92-
Key key = keyFactory.newKey("John Doe");
93-
Entity entity = Entity.builder(key)
94-
.set("age", 51)
95-
.set("favorite_food", "pizza")
96-
.build();
97-
datastore.put(entity);
98-
```
99-
100-
Later, if you want to get this entity back, add the following to your code:
101-
102-
```java
103-
Entity johnEntity = datastore.get(key);
104-
```
105-
106-
#### Running a query
107-
In addition to retrieving entities by their keys, you can perform queries to retrieve entities by the values of their properties. A typical query includes an entity kind, filters to select entities with matching values, and sort orders to sequence the results. `gcloud-java-datastore` supports two types of queries: `StructuredQuery` (that allows you to construct query elements) and `GqlQuery` (which operates using [GQL syntax](https://cloud.google.com/datastore/docs/apis/gql/gql_reference)) in string format. In this tutorial, we will use a simple `StructuredQuery`.
10868

109-
Suppose that you've added more people to Datastore, and now you want to find all people whose favorite food is pizza. Import the following:
110-
111-
```java
112-
import com.google.gcloud.datastore.Query;
113-
import com.google.gcloud.datastore.QueryResults;
114-
import com.google.gcloud.datastore.StructuredQuery;
115-
import com.google.gcloud.datastore.StructuredQuery.PropertyFilter;
116-
```
117-
118-
Then add the following code to your program:
119-
120-
```java
121-
Query<Entity> query = Query.entityQueryBuilder()
122-
.kind("Person")
123-
.filter(PropertyFilter.eq("favorite_food", "pizza"))
124-
.build();
125-
QueryResults<Entity> results = datastore.run(query);
126-
while (results.hasNext()) {
127-
Entity currentEntity = results.next();
128-
}
129-
```
130-
131-
#### Complete source code
132-
133-
Here we put together all the code shown above into one program. This program assumes that you are running on Compute Engine or from your own desktop. To run this example on App Engine, simply move the code from the main method to your application's servlet class.
134-
135-
```java
136-
import com.google.gcloud.datastore.Datastore;
137-
import com.google.gcloud.datastore.DatastoreOptions;
138-
import com.google.gcloud.datastore.Entity;
139-
import com.google.gcloud.datastore.Key;
140-
import com.google.gcloud.datastore.KeyFactory;
141-
import com.google.gcloud.datastore.Query;
142-
import com.google.gcloud.datastore.QueryResults;
143-
import com.google.gcloud.datastore.StructuredQuery;
144-
import com.google.gcloud.datastore.StructuredQuery.PropertyFilter;
145-
146-
public class GcloudJavaDatastoreExample {
147-
148-
public static void main(String[] args) {
149-
// Create datastore service object.
150-
// By default, credentials are inferred from the runtime environment.
151-
Datastore datastore = DatastoreOptions.defaultInstance().service();
152-
153-
// Add an entity to Datastore
154-
KeyFactory keyFactory = datastore.newKeyFactory().kind("Person");
155-
Key key = keyFactory.newKey("John Doe");
156-
Entity entity = Entity.builder(key)
157-
.set("age", 51)
158-
.set("favorite_food", "pizza")
159-
.build();
160-
datastore.put(entity);
161-
162-
// Get an entity from Datastore
163-
Entity johnEntity = datastore.get(key);
164-
165-
// Add a couple more entities to make the query results more interesting
166-
Key janeKey = keyFactory.newKey("Jane Doe");
167-
Entity janeEntity = Entity.builder(janeKey)
168-
.set("age", 44)
169-
.set("favorite_food", "pizza")
170-
.build();
171-
Key joeKey = keyFactory.newKey("Joe Shmoe");
172-
Entity joeEntity = Entity.builder(joeKey)
173-
.set("age", 27)
174-
.set("favorite_food", "sushi")
175-
.build();
176-
datastore.put(janeEntity, joeEntity);
177-
178-
// Run a query
179-
Query<Entity> query = Query.entityQueryBuilder()
180-
.kind("Person")
181-
.filter(PropertyFilter.eq("favorite_food", "pizza"))
182-
.build();
183-
QueryResults<Entity> results = datastore.run(query);
184-
while (results.hasNext()) {
185-
Entity currentEntity = results.next();
186-
// Do something using the entity. (e.g. send an invite a pizza party)
187-
}
188-
}
69+
Datastore datastore = DatastoreOptions.defaultInstance().service();
70+
KeyFactory keyFactory = datastore.newKeyFactory().kind(KIND);
71+
Key key = keyFactory.newKey(keyName);
72+
Entity entity = datastore.get(key);
73+
if (entity == null) {
74+
entity = Entity.builder(key)
75+
.set("name", "John Do")
76+
.set("age", 30)
77+
.set("access_time", DateTime.now())
78+
.build();
79+
datastore.put(entity);
80+
} else {
81+
System.out.println("Updating access_time for " + entity.getString("name"));
82+
entity = Entity.builder(entity)
83+
.set("access_time", DateTime.now())
84+
.build();
85+
datastore.update(entity);
18986
}
19087
```
19188

branches/update-datastore/gcloud-java-storage/src/main/java/com/google/gcloud/spi/DefaultStorageRpc.java

Lines changed: 22 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,6 @@
3535
import com.google.api.client.googleapis.batch.json.JsonBatchCallback;
3636
import com.google.api.client.googleapis.json.GoogleJsonError;
3737
import com.google.api.client.googleapis.json.GoogleJsonResponseException;
38-
import com.google.api.client.googleapis.media.MediaHttpDownloader;
3938
import com.google.api.client.http.ByteArrayContent;
4039
import com.google.api.client.http.GenericUrl;
4140
import com.google.api.client.http.HttpHeaders;
@@ -59,16 +58,18 @@
5958
import com.google.api.services.storage.model.Objects;
6059
import com.google.api.services.storage.model.StorageObject;
6160
import com.google.common.base.MoreObjects;
61+
import com.google.common.collect.ImmutableList;
6262
import com.google.common.collect.ImmutableSet;
63+
import com.google.common.collect.Lists;
6364
import com.google.common.collect.Maps;
64-
import com.google.common.primitives.Ints;
6565
import com.google.gcloud.storage.StorageException;
6666
import com.google.gcloud.storage.StorageOptions;
6767

6868
import java.io.ByteArrayOutputStream;
6969
import java.io.IOException;
7070
import java.io.InputStream;
7171
import java.util.ArrayList;
72+
import java.util.Iterator;
7273
import java.util.List;
7374
import java.util.Map;
7475
import java.util.Set;
@@ -82,6 +83,7 @@ public class DefaultStorageRpc implements StorageRpc {
8283
// see: https://cloud.google.com/storage/docs/concepts-techniques#practices
8384
private static final Set<Integer> RETRYABLE_CODES = ImmutableSet.of(504, 503, 502, 500, 429, 408);
8485
private static final long MEGABYTE = 1024L * 1024L;
86+
private static final int MAX_BATCH_DELETES = 100;
8587

8688
public DefaultStorageRpc(StorageOptions options) {
8789
HttpTransport transport = options.httpTransportFactory().create();
@@ -361,6 +363,24 @@ public byte[] load(StorageObject from, Map<Option, ?> options)
361363

362364
@Override
363365
public BatchResponse batch(BatchRequest request) throws StorageException {
366+
List<List<Tuple<StorageObject, Map<Option, ?>>>> partitionedToDelete =
367+
Lists.partition(request.toDelete, MAX_BATCH_DELETES);
368+
Iterator<List<Tuple<StorageObject, Map<Option, ?>>>> iterator = partitionedToDelete.iterator();
369+
BatchRequest chunkRequest = new BatchRequest(iterator.hasNext() ? iterator.next() :
370+
ImmutableList.<Tuple<StorageObject, Map<Option, ?>>>of(), request.toUpdate, request.toGet);
371+
BatchResponse response = batchChunk(chunkRequest);
372+
Map<StorageObject, Tuple<Boolean, StorageException>> deletes =
373+
Maps.newHashMapWithExpectedSize(request.toDelete.size());
374+
deletes.putAll(response.deletes);
375+
while (iterator.hasNext()) {
376+
chunkRequest = new BatchRequest(iterator.next(), null, null);
377+
BatchResponse deleteBatchResponse = batchChunk(chunkRequest);
378+
deletes.putAll(deleteBatchResponse.deletes);
379+
}
380+
return new BatchResponse(deletes, response.updates, response.gets);
381+
}
382+
383+
private BatchResponse batchChunk(BatchRequest request) {
364384
com.google.api.client.googleapis.batch.BatchRequest batch = storage.batch();
365385
final Map<StorageObject, Tuple<Boolean, StorageException>> deletes =
366386
Maps.newConcurrentMap();

branches/update-datastore/gcloud-java-storage/src/test/java/com/google/gcloud/storage/ITStorageTest.java

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@
2525
import static org.junit.Assert.assertTrue;
2626
import static org.junit.Assert.fail;
2727

28+
import com.google.api.client.util.Lists;
2829
import com.google.common.collect.ImmutableList;
2930
import com.google.common.collect.ImmutableMap;
3031
import com.google.gcloud.Page;
@@ -65,6 +66,7 @@ public class ITStorageTest {
6566
private static final String CONTENT_TYPE = "text/plain";
6667
private static final byte[] BLOB_BYTE_CONTENT = {0xD, 0xE, 0xA, 0xD};
6768
private static final String BLOB_STRING_CONTENT = "Hello Google Cloud Storage!";
69+
private static final int MAX_BATCH_DELETES = 100;
6870

6971
@BeforeClass
7072
public static void beforeClass() {
@@ -623,6 +625,54 @@ public void testBatchRequest() {
623625
assertTrue(deleteResponse.deletes().get(1).get());
624626
}
625627

628+
@Test
629+
public void testBatchRequestManyDeletes() {
630+
List<BlobId> blobsToDelete = Lists.newArrayListWithCapacity(2 * MAX_BATCH_DELETES);
631+
for (int i = 0; i < 2 * MAX_BATCH_DELETES; i++) {
632+
blobsToDelete.add(BlobId.of(BUCKET, "test-batch-request-many-deletes-blob-" + i));
633+
}
634+
BatchRequest.Builder builder = BatchRequest.builder();
635+
for (BlobId blob : blobsToDelete) {
636+
builder.delete(blob);
637+
}
638+
String sourceBlobName1 = "test-batch-request-many-deletes-source-blob-1";
639+
String sourceBlobName2 = "test-batch-request-many-deletes-source-blob-2";
640+
BlobInfo sourceBlob1 = BlobInfo.builder(BUCKET, sourceBlobName1).build();
641+
BlobInfo sourceBlob2 = BlobInfo.builder(BUCKET, sourceBlobName2).build();
642+
assertNotNull(storage.create(sourceBlob1));
643+
assertNotNull(storage.create(sourceBlob2));
644+
BlobInfo updatedBlob2 = sourceBlob2.toBuilder().contentType(CONTENT_TYPE).build();
645+
646+
BatchRequest updateRequest = builder
647+
.get(BUCKET, sourceBlobName1)
648+
.update(updatedBlob2)
649+
.build();
650+
BatchResponse response = storage.apply(updateRequest);
651+
assertEquals(2 * MAX_BATCH_DELETES, response.deletes().size());
652+
assertEquals(1, response.updates().size());
653+
assertEquals(1, response.gets().size());
654+
655+
// Check deletes
656+
for (BatchResponse.Result<Boolean> deleteResult : response.deletes()) {
657+
assertFalse(deleteResult.failed());
658+
assertFalse(deleteResult.get());
659+
}
660+
661+
// Check updates
662+
BlobInfo remoteUpdatedBlob2 = response.updates().get(0).get();
663+
assertEquals(sourceBlob2.bucket(), remoteUpdatedBlob2.bucket());
664+
assertEquals(sourceBlob2.name(), remoteUpdatedBlob2.name());
665+
assertEquals(updatedBlob2.contentType(), remoteUpdatedBlob2.contentType());
666+
667+
// Check gets
668+
BlobInfo remoteBlob1 = response.gets().get(0).get();
669+
assertEquals(sourceBlob1.bucket(), remoteBlob1.bucket());
670+
assertEquals(sourceBlob1.name(), remoteBlob1.name());
671+
672+
assertTrue(storage.delete(BUCKET, sourceBlobName1));
673+
assertTrue(storage.delete(BUCKET, sourceBlobName2));
674+
}
675+
626676
@Test
627677
public void testBatchRequestFail() {
628678
String blobName = "test-batch-request-blob-fail";

0 commit comments

Comments
 (0)