Skip to content

Commit ee90ede

Browse files
jean-philippe-martinaozarov
jean-philippe-martin
authored and
aozarov
committed
---
yaml --- r: 4037 b: refs/heads/gcs-nio c: edd0f21 h: refs/heads/master i: 4035: 36b0eee
1 parent 01a894c commit ee90ede

11 files changed

Lines changed: 302 additions & 78 deletions

File tree

‎[refs]‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ refs/tags/v0.0.12: 2fd8066e891fb3dfea69b65f6bf6461db79342b9
1111
refs/heads/compute-alpha: 969cba2627f1d53d352cc4a5ffe0879dacf65e6c
1212
refs/heads/dns-alpha: 2f90e7e338349287ace33375896907af0f032ca1
1313
refs/heads/dns-alpha-batch: 17442b07867021b85d0452f5f3eda29a3413288f
14-
refs/heads/gcs-nio: b584349290cb7ebcad782ba24d2a700d97ac9cb6
14+
refs/heads/gcs-nio: edd0f2119a6a80d078d79498d7e6e100f85bf4a3
1515
refs/heads/logging-alpha: db5312bffa7fccac194f6a7feb8cc3066de16aff
1616
refs/tags/v0.1.0: a615317f7424ed58621b1f65d5c4d8cbbe8a6ed8
1717
refs/tags/v0.1.1: 7a7f6985fe465e9dd6a075af55493f42b4933be0

‎branches/gcs-nio/gcloud-java-contrib/README.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,12 @@ Contents
77
--------
88

99
* [gcloud-java-nio](./gcloud-java-nio/): NIO Filesystem Provider for Google Cloud Storage.
10+
* [gcloud-java-nio-examples](./gcloud-java-nio-examples/): How to add GCS NIO after the fact.
11+
12+
See also
13+
--------
14+
15+
* [gcloud-java-examples](../gcloud-java-examples) for an example of how to use NIO normally.
1016

1117
Contributing
1218
------------

‎branches/gcs-nio/gcloud-java-contrib/gcloud-java-nio-examples/README.md‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
Example of adding the Google Cloud Storage NIO Provider to a legacy jar
2+
=======================================================================
3+
4+
This project shows how to add GCS capabilities to a jar file for a Java 7 application
5+
that uses Java NIO without the need to recompile.
6+
7+
Note that whenever possible, you instead want to recompile the app and use the normal
8+
dependency mechanism to add a dependency to gcloud-java-nio. You can see examples of
9+
this in the [gcloud-java-examples](../../gcloud-java-examples) project.
10+
11+
To run this example:
12+
13+
1. Before running the example, go to the [Google Developers Console][developers-console] to ensure that Google Cloud Storage API is enabled.
14+
15+
2. Log in using gcloud SDK (`gcloud auth login` in command line)
16+
17+
3. Compile the JAR with:
18+
```
19+
mvn package -DskipTests -Dmaven.javadoc.skip=true -Dmaven.source.skip=true
20+
```
21+
22+
4. Run the sample with:
23+
24+
```
25+
java -cp gcloud-java-contrib/gcloud-java-nio/target/gcloud-java-nio-0.1.6-SNAPSHOT-shaded.jar:gcloud-java-contrib/gcloud-java-nio-examples/target/gcloud-java-nio-examples-0.1.6-SNAPSHOT.jar com.google.gcloud.nio.examples.ListFilesystems
26+
```
27+
28+
Notice that it lists gcs, which it wouldn't if you ran it without the nio jar:
29+
```
30+
java -cp gcloud-java-contrib/gcloud-java-nio-examples/target/gcloud-java-nio-examples-0.1.6-SNAPSHOT.jar com.google.gcloud.nio.examples.ListFilesystems
31+
```
32+
33+
The sample doesn't have anything about GCS in it. It gets that ability from the nio jar that
34+
we're adding to the classpath. You can use the nio "fat shaded" jar for this purpose as it also
35+
includes the dependencies for gcloud-java-nio.
36+
The underlying mechanism is Java's standard [ServiceLoader](https://docs.oracle.com/javase/7/docs/api/java/util/ServiceLoader.html)
37+
facility, the [standard way](http://docs.oracle.com/javase/7/docs/technotes/guides/io/fsp/filesystemprovider.html) to plug in NIO providers like this one.
38+
39+
If you have access to a project's source code you can also simply add gcloud-java-nio as
40+
a dependency and let Maven pull in the required dependencies (this is what the nio unit tests do).
41+
This approach is preferable as the fat jar approach may waste memory on multiple copies of dependencies.

‎branches/gcs-nio/gcloud-java-contrib/gcloud-java-nio-examples/pom.xml‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
<?xml version="1.0"?>
2+
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
3+
<modelVersion>4.0.0</modelVersion>
4+
<artifactId>gcloud-java-nio-examples</artifactId>
5+
<packaging>jar</packaging>
6+
<name>GCloud Java NIO example</name>
7+
<description>
8+
Demonstrates how to use the gcloud-java-nio jar to add GCS functionality to legacy code.
9+
</description>
10+
<parent>
11+
<groupId>com.google.gcloud</groupId>
12+
<artifactId>gcloud-java-contrib</artifactId>
13+
<version>0.1.6-SNAPSHOT</version>
14+
</parent>
15+
<properties>
16+
<site.installationModule>nio</site.installationModule>
17+
</properties>
18+
<dependencies>
19+
<dependency>
20+
<groupId>${project.groupId}</groupId>
21+
<artifactId>gcloud-java-storage</artifactId>
22+
<version>${project.version}</version>
23+
</dependency>
24+
<dependency>
25+
<groupId>com.google.guava</groupId>
26+
<artifactId>guava</artifactId>
27+
<version>19.0</version>
28+
</dependency>
29+
<dependency>
30+
<groupId>com.google.code.findbugs</groupId>
31+
<artifactId>jsr305</artifactId>
32+
<version>2.0.1</version>
33+
</dependency>
34+
<dependency>
35+
<groupId>javax.inject</groupId>
36+
<artifactId>javax.inject</artifactId>
37+
<version>1</version>
38+
</dependency>
39+
<dependency>
40+
<groupId>com.google.auto.service</groupId>
41+
<artifactId>auto-service</artifactId>
42+
<version>1.0-rc2</version>
43+
<scope>provided</scope> <!-- to leave out of the all-deps jar -->
44+
</dependency>
45+
<dependency>
46+
<groupId>com.google.auto.value</groupId>
47+
<artifactId>auto-value</artifactId>
48+
<version>1.1</version>
49+
<scope>provided</scope> <!-- to leave out of the all-deps jar -->
50+
</dependency>
51+
<dependency>
52+
<groupId>junit</groupId>
53+
<artifactId>junit</artifactId>
54+
<version>4.12</version>
55+
<scope>test</scope>
56+
</dependency>
57+
<dependency>
58+
<groupId>com.google.guava</groupId>
59+
<artifactId>guava-testlib</artifactId>
60+
<version>19.0</version>
61+
<scope>test</scope>
62+
</dependency>
63+
<dependency>
64+
<groupId>com.google.truth</groupId>
65+
<artifactId>truth</artifactId>
66+
<version>0.27</version>
67+
<scope>test</scope>
68+
</dependency>
69+
<dependency>
70+
<groupId>org.mockito</groupId>
71+
<artifactId>mockito-core</artifactId>
72+
<version>1.9.5</version>
73+
</dependency>
74+
<dependency>
75+
<groupId>org.apache.maven.plugins</groupId>
76+
<artifactId>maven-assembly-plugin</artifactId>
77+
<version>2.5.4</version>
78+
</dependency>
79+
</dependencies>
80+
<build>
81+
<plugins>
82+
<plugin>
83+
<groupId>org.codehaus.mojo</groupId>
84+
<artifactId>exec-maven-plugin</artifactId>
85+
<configuration>
86+
<skip>false</skip>
87+
</configuration>
88+
</plugin>
89+
</plugins>
90+
</build>
91+
</project>

‎branches/gcs-nio/gcloud-java-contrib/gcloud-java-nio-examples/src/main/java/com/google/gcloud/nio/examples/ListFilesystems.java‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
/*
2+
* Copyright 2015 Google Inc. All Rights Reserved.
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
package com.google.gcloud.nio.examples;
18+
19+
import java.io.IOException;
20+
import java.net.URI;
21+
import java.nio.file.FileSystem;
22+
import java.nio.file.FileSystems;
23+
import java.nio.file.Files;
24+
import java.nio.file.Path;
25+
import java.nio.file.Paths;
26+
import java.nio.file.spi.FileSystemProvider;
27+
28+
/**
29+
* ListFilesystems is a super-simple program that lists the available NIO filesystems.
30+
*/
31+
public class ListFilesystems {
32+
33+
/**
34+
* See the class documentation.
35+
*/
36+
public static void main(String[] args) throws IOException {
37+
listFilesystems();
38+
}
39+
40+
private static void listFilesystems() {
41+
System.out.println("Installed filesystem providers:");
42+
for (FileSystemProvider p : FileSystemProvider.installedProviders()) {
43+
System.out.println(" " + p.getScheme());
44+
}
45+
}
46+
47+
}

‎branches/gcs-nio/gcloud-java-contrib/gcloud-java-nio/pom.xml‎

Lines changed: 34 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
11
<?xml version="1.0"?>
22
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
33
<modelVersion>4.0.0</modelVersion>
4-
<groupId>com.google.gcloud</groupId>
54
<artifactId>gcloud-java-nio</artifactId>
65
<packaging>jar</packaging>
76
<name>GCloud Java NIO</name>
@@ -92,40 +91,40 @@
9291
<compilerArgument>-Xlint:unchecked</compilerArgument>
9392
</configuration>
9493
</plugin>
95-
<!-- shade the jar so we can demo adding the NIO jar to add functionality. -->
96-
<plugin>
97-
<groupId>org.apache.maven.plugins</groupId>
98-
<artifactId>maven-shade-plugin</artifactId>
99-
<version>2.4.3</version>
100-
<configuration>
101-
<shadedArtifactAttached>true</shadedArtifactAttached>
102-
<relocations>
103-
<relocation>
104-
<pattern>com</pattern>
105-
<shadedPattern>shaded.gcloud-nio.com</shadedPattern>
106-
<excludes>
107-
<exclude>com.google.gcloud.**</exclude>
108-
</excludes>
109-
</relocation>
110-
<relocation>
111-
<pattern>org</pattern>
112-
<shadedPattern>shaded.gcloud-nio.org</shadedPattern>
113-
</relocation>
114-
<relocation>
115-
<pattern>google</pattern>
116-
<shadedPattern>shaded.gcloud-nio.google</shadedPattern>
117-
</relocation>
118-
</relocations>
119-
</configuration>
120-
<executions>
121-
<execution>
122-
<phase>package</phase>
123-
<goals>
124-
<goal>shade</goal>
125-
</goals>
126-
</execution>
127-
</executions>
128-
</plugin>
94+
<!-- shade the jar so we can demo adding the NIO jar to add functionality. -->
95+
<plugin>
96+
<groupId>org.apache.maven.plugins</groupId>
97+
<artifactId>maven-shade-plugin</artifactId>
98+
<version>2.4.3</version>
99+
<configuration>
100+
<shadedArtifactAttached>true</shadedArtifactAttached>
101+
<relocations>
102+
<relocation>
103+
<pattern>com</pattern>
104+
<shadedPattern>shaded.gcloud-nio.com</shadedPattern>
105+
<excludes>
106+
<exclude>com.google.gcloud.**</exclude>
107+
</excludes>
108+
</relocation>
109+
<relocation>
110+
<pattern>org</pattern>
111+
<shadedPattern>shaded.gcloud-nio.org</shadedPattern>
112+
</relocation>
113+
<relocation>
114+
<pattern>google</pattern>
115+
<shadedPattern>shaded.gcloud-nio.google</shadedPattern>
116+
</relocation>
117+
</relocations>
118+
</configuration>
119+
<executions>
120+
<execution>
121+
<phase>package</phase>
122+
<goals>
123+
<goal>shade</goal>
124+
</goals>
125+
</execution>
126+
</executions>
127+
</plugin>
129128
</plugins>
130129
</build>
131130
</project>

‎branches/gcs-nio/gcloud-java-contrib/pom.xml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717
</properties>
1818
<modules>
1919
<module>gcloud-java-nio</module>
20+
<module>gcloud-java-nio-examples</module>
2021
</modules>
2122
<build>
2223
<plugins>

‎branches/gcs-nio/gcloud-java-examples/README.md‎

Lines changed: 27 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -37,9 +37,9 @@ To run examples from your command line:
3737

3838
2. Set your current project using `gcloud config set project PROJECT_ID`. This step is not necessary for `ResourceManagerExample`.
3939

40-
3. Compile using Maven (`mvn compile` in command line from your base project directory)
40+
3. Compile using Maven: `cd gcloud-java-examples` in command line from your base project directory and then `mvn package appassembler:assemble -DskipTests -Dmaven.javadoc.skip=true -Dmaven.source.skip=true`.
4141

42-
4. Run an example using Maven from command line.
42+
4. Run an example from the command line using the Maven-generated scripts.
4343

4444
* Here's an example run of `BigQueryExample`.
4545

@@ -55,62 +55,57 @@ To run examples from your command line:
5555
```
5656
Then you are ready to run the following example:
5757
```
58-
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.bigquery.BigQueryExample" -Dexec.args="create dataset new_dataset_id"
59-
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.bigquery.BigQueryExample" -Dexec.args="create table new_dataset_id new_table_id field_name:string"
60-
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.bigquery.BigQueryExample" -Dexec.args="list tables new_dataset_id"
61-
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.bigquery.BigQueryExample" -Dexec.args="load new_dataset_id new_table_id CSV gs://my_bucket/my_csv_file"
62-
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.bigquery.BigQueryExample" -Dexec.args="query 'select * from new_dataset_id.new_table_id'"
58+
target/appassembler/bin/BigQueryExample create dataset new_dataset_id
59+
target/appassembler/bin/BigQueryExample create table new_dataset_id new_table_id field_name:string
60+
target/appassembler/bin/BigQueryExample list tables new_dataset_id
61+
target/appassembler/bin/BigQueryExample load new_dataset_id new_table_id CSV gs://my_bucket/my_csv_file
62+
target/appassembler/bin/BigQueryExample query 'select * from new_dataset_id.new_table_id'
6363
```
6464
6565
* Here's an example run of `DatastoreExample`.
6666
6767
Be sure to change the placeholder project ID "your-project-id" with your own project ID. Also note that you have to enable the Google Cloud Datastore API on the [Google Developers Console][developers-console] before running the following commands.
6868
```
69-
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.datastore.DatastoreExample" -Dexec.args="your-project-id my_name add my\ comment"
70-
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.datastore.DatastoreExample" -Dexec.args="your-project-id my_name display"
71-
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.datastore.DatastoreExample" -Dexec.args="your-project-id my_name delete"
69+
target/appassembler/bin/DatastoreExample your-project-id my_name add my\ comment
70+
target/appassembler/bin/DatastoreExample your-project-id my_name display
71+
target/appassembler/bin/DatastoreExample your-project-id my_name delete
7272
```
7373
7474
* Here's an example run of `ResourceManagerExample`.
7575
7676
Be sure to change the placeholder project ID "your-project-id" with your own globally unique project ID.
7777
```
78-
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.resourcemanager.ResourceManagerExample" -Dexec.args="create your-project-id"
79-
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.resourcemanager.ResourceManagerExample" -Dexec.args="list"
80-
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.resourcemanager.ResourceManagerExample" -Dexec.args="get your-project-id"
78+
target/appassembler/bin/ResourceManagerExample create your-project-id
79+
target/appassembler/bin/ResourceManagerExample list
80+
target/appassembler/bin/ResourceManagerExample get your-project-id
8181
```
8282
8383
* Here's an example run of `StorageExample`.
8484
8585
Before running the example, go to the [Google Developers Console][developers-console] to ensure that Google Cloud Storage API is enabled and that you have a bucket. Also ensure that you have a test file (`test.txt` is chosen here) to upload to Cloud Storage stored locally on your machine.
8686
```
87-
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.storage.StorageExample" -Dexec.args="upload /path/to/test.txt <bucket_name>"
88-
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.storage.StorageExample" -Dexec.args="list <bucket_name>"
89-
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.storage.StorageExample" -Dexec.args="download <bucket_name> test.txt"
90-
mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.storage.StorageExample" -Dexec.args="delete <bucket_name> test.txt"
87+
target/appassembler/bin/StorageExample upload /path/to/test.txt <bucket_name>
88+
target/appassembler/bin/StorageExample list <bucket_name>
89+
target/appassembler/bin/StorageExample download <bucket_name> test.txt
90+
target/appassembler/bin/StorageExample delete <bucket_name> test.txt
9191
```
92-
* Here's an example run of `Stat`, illustrating the use of the gcloud-java-nio jar.
9392
94-
Before running the example, go to the [Google Developers Console][developers-console] to ensure that Google Cloud Storage API is enabled and that you have a bucket with a file in it.
95-
Compile the JAR with:
96-
```
97-
mvn package -DskipTests -Dmaven.javadoc.skip=true -Dmaven.source.skip=true
98-
```
99-
Then run the sample with:
93+
* Here's an example run of `Stat`, illustrating the use of gcloud-java-nio.
94+
95+
Before running the example, go to the [Google Developers Console][developers-console] to ensure that Google Cloud Storage API is enabled and that you have a bucket with a file in it.
96+
97+
Run the sample with (from the gcloud-java-examples folder):
10098
```
101-
java -cp gcloud-java-contrib/gcloud-java-nio/target/gcloud-java-nio-0.1.6-SNAPSHOT-shaded.jar:gcloud-java-examples/target/gcloud-java-examples-0.1.6-SNAPSHOT.jar com.google.gcloud.examples.nio.Stat --check
99+
target/appassembler/bin/Stat --check
100+
102101
```
103102
Or, if you have a file in `gs://mybucket/myfile.txt`, you can run:
104103
```
105-
java -cp gcloud-java-contrib/gcloud-java-nio/target/gcloud-java-nio-0.1.6-SNAPSHOT-shaded.jar:gcloud-java-examples/target/gcloud-java-examples-0.1.6-SNAPSHOT.jar com.google.gcloud.examples.nio.Stat gs://mybucket/myfile.txt
104+
target/appassembler/bin/Stat gs://mybucket/myfile.txt
106105
```
107-
The sample doesn't have anything about GCS in it. It gets that ability from the nio jar that
108-
we're adding to the classpath. You can use the nio "fat shaded" jar for this purpose as it also
109-
includes the dependencies for gcloud-java-nio.
110106
111-
If you have access to a project's source code you can also simply add gcloud-java-nio as
112-
a dependency and let Maven pull in the required dependencies (this is what the nio unit tests do).
113-
This approach is preferable as the fat jar approach may waste memory on multiple copies of dependencies.
107+
The sample doesn't have anything special about GCS in it, it just opens files via the NIO API.
108+
It lists gcloud-java-nio as a dependency, and that enables it to interpret `gs://` paths.
114109
115110
Troubleshooting
116111
---------------

0 commit comments

Comments
 (0)