Skip to content

Commit 6a5a3c8

Browse files
yoonjs2yoonjs2
committed
2 parents 6d04fa7 + a6c8b5f commit 6a5a3c8

File tree

21 files changed

+362
-37
lines changed

21 files changed

+362
-37
lines changed

‎bin/common.sh‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,7 @@ if [[ -z "$ZEPPELIN_MEM" ]]; then
105105
fi
106106

107107
JAVA_OPTS+=" ${ZEPPELIN_JAVA_OPTS} -Dfile.encoding=${ZEPPELIN_ENCODING} ${ZEPPELIN_MEM}"
108+
JAVA_OPTS+=" -Dlog4j.configuration=file://${ZEPPELIN_CONF_DIR}/log4j.properties"
108109
export JAVA_OPTS
109110

110111
# jvm options for interpreter process
@@ -117,6 +118,7 @@ if [[ -z "${ZEPPELIN_INTP_MEM}" ]]; then
117118
fi
118119

119120
JAVA_INTP_OPTS="${ZEPPELIN_INTP_JAVA_OPTS} -Dfile.encoding=${ZEPPELIN_ENCODING}"
121+
JAVA_INTP_OPTS+=" -Dlog4j.configuration=file://${ZEPPELIN_CONF_DIR}/log4j.properties"
120122
export JAVA_INTP_OPTS
121123

122124

‎docs/install/virtual_machine.md‎

Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -26,11 +26,13 @@ Apache Zeppelin distribution includes a scripts directory
2626

2727
`scripts/vagrant/zeppelin-dev`
2828

29-
This script creates a virtual machine that launches a repeatable, known set of core dependencies required for developing Zeppelin. It can also be used to run an existing Zeppelin build if you don't plan to build from source. For pyspark users, this script also includes several helpful [Python Libraries](#pythonextras).
29+
This script creates a virtual machine that launches a repeatable, known set of core dependencies required for developing Zeppelin. It can also be used to run an existing Zeppelin build if you don't plan to build from source.
30+
For PySpark users, this script includes several helpful [Python Libraries](#python-extras).
31+
For SparkR users, this script includes several helpful [R Libraries](#r-extras).
3032

3133
####Installing the required components to launch a virtual machine.
3234

33-
This script requires three applications, [Ansible](http://docs.ansible.com/ansible/intro_installation.html#latest-releases-via-pip "Ansible"), [Vagrant](http://www.vagrantup.com/downloads "Vagrant") and [Virtual Box](https://www.virtualbox.org/ "Virtual Box"). All of these applications are freely available as Open Source projects and extremely easy to set up on most operating systems.
35+
This script requires three applications, [Ansible](http://docs.ansible.com/ansible/intro_installation.html#latest-releases-via-pip "Ansible"), [Vagrant](http://www.vagrantup.com "Vagrant") and [Virtual Box](https://www.virtualbox.org/ "Virtual Box"). All of these applications are freely available as Open Source projects and extremely easy to set up on most operating systems.
3436

3537
### Create a Zeppelin Ready VM in 4 Steps (5 on Windows)
3638

@@ -92,23 +94,24 @@ The virtual machine consists of:
9294
- libfontconfig to avoid phatomJs missing dependency issues
9395
- openjdk-7-jdk
9496
- Python addons: pip, matplotlib, scipy, numpy, pandas
97+
- [R](https://www.r-project.org/) and R Packages required to run the R Interpreter and the related R tutorial notebook, including: Knitr, devtools, repr, rCharts, ggplot2, googleVis, mplot, htmltools, base64enc, data.table
9598
9699
### How to build & run Zeppelin
97100
98-
This assumes you've already cloned the project either on the host machine in the zeppelin-dev directory (to be shared with the guest machine) or cloned directly into a directory while running inside the guest machine.
101+
This assumes you've already cloned the project either on the host machine in the zeppelin-dev directory (to be shared with the guest machine) or cloned directly into a directory while running inside the guest machine. The following build steps will also include Python and R support via PySpark and SparkR:
99102
100103
```
101104
cd /incubator-zeppelin
102-
mvn clean package -Pspark-1.5 -Ppyspark -Dhadoop.version=2.2.0 -Phadoop-2.2 -DskipTests
105+
mvn clean package -Pspark-1.6 -Ppyspark -Phadoop-2.4 -Psparkr -DskipTests
103106
./bin/zeppelin-daemon.sh start
104107
```
105108
106109
On your host machine browse to `http://localhost:8080/`
107110
108-
If you [turned off port forwarding](#tweakvm) in the `Vagrantfile` browse to `http://192.168.51.52:8080`
111+
If you [turned off port forwarding](#tweaking-the-virtual-machine) in the `Vagrantfile` browse to `http://192.168.51.52:8080`
109112
110113
111-
### [Tweaking the Virtual Machine](id:tweakvm)
114+
### Tweaking the Virtual Machine
112115
113116
If you plan to run this virtual machine along side other Vagrant images, you may wish to bind the virtual machine to a specific IP address, and not use port fowarding from your local host.
114117
@@ -123,7 +126,7 @@ config.vm.network "private_network", ip: "192.168.51.52"
123126
This approach usually is typically required if running other virtual machines that discover each other directly by IP address, such as Spark Masters and Slaves as well as Cassandra Nodes, Elasticsearch Nodes, and other Spark data sources. You may wish to launch nodes in virtual machines with IP addresses in a subnet that works for your local network, such as: 192.168.51.53, 192.168.51.54, 192.168.51.53, etc..
124127
125128
126-
### [Python Extras](id:pythonextras)
129+
### Python Extras
127130
128131
With Zeppelin running, **Numpy**, **SciPy**, **Pandas** and **Matplotlib** will be available. Create a pyspark notebook, and try the below code.
129132
@@ -175,10 +178,7 @@ plt.title('How fast do you want to go today?')
175178
show(plt)
176179
```
177180

181+
### R Extras
178182

179-
180-
181-
182-
183-
184-
183+
With zeppelin running, an R Tutorial notebook will be available. The R packages required to run the examples and graphs in this tutorial notebook were installed by this virtual machine.
184+
The installed R Packages include: Knitr, devtools, repr, rCharts, ggplot2, googleVis, mplot, htmltools, base64enc, data.table

‎docs/manual/interpreters.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,8 @@ Zeppelin interpreter setting is the configuration of a given interpreter on Zepp
3636

3737
<img src="/assets/themes/zeppelin/img/screenshots/interpreter_setting.png">
3838

39+
Properties are exported as environment variable when property name is consisted of upper characters, numbers and underscore ([A-Z_0-9]). Otherwise set properties as JVM property.
40+
3941
Each notebook can be binded to multiple Interpreter Settings using setting icon on upper right corner of the notebook.
4042

4143
<img src="/assets/themes/zeppelin/img/screenshots/interpreter_binding.png" width="800px">

‎lens/pom.xml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@
3434
<url>http://www.apache.org</url>
3535

3636
<properties>
37-
<lens.version>2.2.0-beta-incubating</lens.version>
37+
<lens.version>2.5.0-beta</lens.version>
3838
<spring-shell.version>1.1.0.RELEASE</spring-shell.version>
3939
<hadoop-common.version>2.4.0</hadoop-common.version>
4040
</properties>

‎scripts/vagrant/zeppelin-dev/README.md‎

Lines changed: 13 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -13,11 +13,13 @@ limitations under the License.
1313
-->
1414
## Vagrant Virtual Machine for Apache Zeppelin
1515

16-
This script creates a virtual machine that launches a repeatable, known set of core dependencies required for developing Zeppelin. It can also be used to run an existing Zeppelin build if you don't plan to build from source. For pyspark users, this script also includes several helpful [Python Libraries and one obscure configuration to help with matplotlib plotting inside Zeppelin](#pythonextras)
16+
This script creates a virtual machine that launches a repeatable, known set of core dependencies required for developing Zeppelin. It can also be used to run an existing Zeppelin build if you don't plan to build from source.
17+
For PySpark users, this script includes several helpful [Python Libraries](#python-extras).
18+
For SparkR users, this script includes several helpful [R Libraries](#r-extras).
1719

1820
####Installing the required components to launch a virtual machine.
1921

20-
This script requires three applications, [Ansible](http://docs.ansible.com/ansible/intro_installation.html#latest-releases-via-pip "Ansible"), [Vagrant](http://www.vagrantup.com/downloads "Vagrant") and [Virtual Box](https://www.virtualbox.org/ "Virtual Box"). All of these applications are freely available as Open Source projects and extremely easy to set up on most operating systems.
22+
This script requires three applications, [Ansible](http://docs.ansible.com/ansible/intro_installation.html#latest-releases-via-pip "Ansible"), [Vagrant](http://www.vagrantup.com "Vagrant") and [Virtual Box](https://www.virtualbox.org/ "Virtual Box"). All of these applications are freely available as Open Source projects and extremely easy to set up on most operating systems.
2123

2224
### Create a Zeppelin Ready VM in 4 Steps (5 on Windows)
2325

@@ -77,23 +79,24 @@ The virtual machine consists of:
7779
- libfontconfig to avoid phatomJs missing dependency issues
7880
- openjdk-7-jdk
7981
- Python addons: pip, matplotlib, scipy, numpy, pandas
82+
- [R](https://www.r-project.org/) and R Packages required to run the R Interpreter and the related R tutorial notebook, including: Knitr, devtools, repr, rCharts, ggplot2, googleVis, mplot, htmltools, base64enc, data.table
8083

8184
### How to build & run Zeppelin
8285

83-
This assumes you've already cloned the project either on the host machine in the zeppelin-dev directory (to be shared with the guest machine) or cloned directly into a directory while running inside the guest machine.
86+
This assumes you've already cloned the project either on the host machine in the zeppelin-dev directory (to be shared with the guest machine) or cloned directly into a directory while running inside the guest machine. The following build steps will also include Python and R support via PySpark and SparkR:
8487

8588
```
8689
cd /incubator-zeppelin
87-
mvn clean package -Pspark-1.5 -Ppyspark -Dhadoop.version=2.2.0 -Phadoop-2.2 -DskipTests
90+
mvn clean package -Pspark-1.6 -Ppyspark -Phadoop-2.4 -Psparkr -DskipTests
8891
./bin/zeppelin-daemon.sh start
8992
```
9093

9194
On your host machine browse to `http://localhost:8080/`
9295

93-
If you [turned off port forwarding](#tweakvm) in the `Vagrantfile` browse to `http://192.168.51.52:8080`
96+
If you [turned off port forwarding](#tweaking-the-virtual-machine) in the `Vagrantfile` browse to `http://192.168.51.52:8080`
9497

9598

96-
### [Tweaking the Virtual Machine](id:tweakvm)
99+
### Tweaking the Virtual Machine
97100

98101
If you plan to run this virtual machine along side other Vagrant images, you may wish to bind the virtual machine to a specific IP address, and not use port fowarding from your local host.
99102

@@ -108,7 +111,7 @@ config.vm.network "private_network", ip: "192.168.51.52"
108111
This approach usually is typically required if running other virtual machines that discover each other directly by IP address, such as Spark Masters and Slaves as well as Cassandra Nodes, Elasticsearch Nodes, and other Spark data sources. You may wish to launch nodes in virtual machines with IP Addresses in a subnet that works for your local network, such as: 192.168.51.53, 192.168.51.54, 192.168.51.53, etc..
109112

110113

111-
### [Python Extras](id:pythonextras)
114+
### Python Extras
112115

113116
With zeppelin running, Numpy, SciPy, Pandas and Matplotlib will be available. Create a pyspark notebook, and try
114117

@@ -160,8 +163,7 @@ plt.title('How fast do you want to go today?')
160163
show(plt)
161164
```
162165

166+
### R Extras
163167

164-
165-
166-
167-
168+
With zeppelin running, an R Tutorial notebook will be available. The R packages required to run the examples and graphs in this tutorial notebook were installed by this virtual machine.
169+
The installed R Packages include: Knitr, devtools, repr, rCharts, ggplot2, googleVis, mplot, htmltools, base64enc, data.table

‎spark/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -91,7 +91,12 @@ public class PySparkInterpreter extends Interpreter implements ExecuteResultHand
9191
public PySparkInterpreter(Properties property) {
9292
super(property);
9393

94-
scriptPath = System.getProperty("java.io.tmpdir") + "/zeppelin_pyspark.py";
94+
try {
95+
File scriptFile = File.createTempFile("zeppelin_pyspark-", ".py");
96+
scriptPath = scriptFile.getAbsolutePath();
97+
} catch (IOException e) {
98+
throw new InterpreterException(e);
99+
}
95100
}
96101

97102
private void createPythonScript() {
@@ -235,6 +240,7 @@ private int findRandomOpenPortOnAllLocalInterfaces() {
235240
@Override
236241
public void close() {
237242
executor.getWatchdog().destroyProcess();
243+
new File(scriptPath).delete();
238244
gatewayServer.shutdown();
239245
}
240246

‎spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -254,7 +254,7 @@ private DepInterpreter getDepInterpreter() {
254254
}
255255

256256
public SparkContext createSparkContext() {
257-
System.err.println("------ Create new SparkContext " + getProperty("master") + " -------");
257+
logger.info("------ Create new SparkContext {} -------", getProperty("master"));
258258

259259
String execUri = System.getenv("SPARK_EXECUTOR_URI");
260260
String[] jars = SparkILoop.getAddedJars();

‎spark/src/main/java/org/apache/zeppelin/spark/ZeppelinR.java‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -111,8 +111,12 @@ public ZeppelinR(String rCmdPath, String libPath, int sparkRBackendPort) {
111111
this.rCmdPath = rCmdPath;
112112
this.libPath = libPath;
113113
this.port = sparkRBackendPort;
114-
scriptPath = System.getProperty("java.io.tmpdir") + "/zeppelin_sparkr.R";
115-
114+
try {
115+
File scriptFile = File.createTempFile("zeppelin_sparkr-", ".R");
116+
scriptPath = scriptFile.getAbsolutePath();
117+
} catch (IOException e) {
118+
throw new InterpreterException(e);
119+
}
116120
}
117121

118122
/**

‎zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreter.java‎

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -71,7 +71,7 @@ public RemoteInterpreter(Properties property,
7171
this.interpreterRunner = interpreterRunner;
7272
this.interpreterPath = interpreterPath;
7373
this.localRepoPath = localRepoPath;
74-
env = new HashMap<String, String>();
74+
env = getEnvFromInterpreterProperty(property);
7575
this.connectTimeout = connectTimeout;
7676
this.maxPoolSize = maxPoolSize;
7777
this.remoteInterpreterProcessListener = remoteInterpreterProcessListener;
@@ -92,12 +92,31 @@ public RemoteInterpreter(Properties property,
9292
this.interpreterRunner = interpreterRunner;
9393
this.interpreterPath = interpreterPath;
9494
this.localRepoPath = localRepoPath;
95+
env.putAll(getEnvFromInterpreterProperty(property));
9596
this.env = env;
9697
this.connectTimeout = connectTimeout;
9798
this.maxPoolSize = 10;
9899
this.remoteInterpreterProcessListener = remoteInterpreterProcessListener;
99100
}
100101

102+
private Map<String, String> getEnvFromInterpreterProperty(Properties property) {
103+
Map<String, String> env = new HashMap<String, String>();
104+
for (Object key : property.keySet()) {
105+
if (isEnvString((String) key)) {
106+
env.put((String) key, property.getProperty((String) key));
107+
}
108+
}
109+
return env;
110+
}
111+
112+
static boolean isEnvString(String key) {
113+
if (key == null || key.length() == 0) {
114+
return false;
115+
}
116+
117+
return key.matches("^[A-Z_0-9]*");
118+
}
119+
101120
@Override
102121
public String getClassName() {
103122
return className;

‎zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterServer.java‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -151,6 +151,7 @@ public void createInterpreter(String interpreterGroupId, String noteId, String
151151
Class<Interpreter> replClass = (Class<Interpreter>) Object.class.forName(className);
152152
Properties p = new Properties();
153153
p.putAll(properties);
154+
setSystemProperty(p);
154155

155156
Constructor<Interpreter> constructor =
156157
replClass.getConstructor(new Class[] {Properties.class});
@@ -179,6 +180,19 @@ public void createInterpreter(String interpreterGroupId, String noteId, String
179180
}
180181
}
181182

183+
private void setSystemProperty(Properties properties) {
184+
for (Object key : properties.keySet()) {
185+
if (!RemoteInterpreter.isEnvString((String) key)) {
186+
String value = properties.getProperty((String) key);
187+
if (value == null || value.isEmpty()) {
188+
System.clearProperty((String) key);
189+
} else {
190+
System.setProperty((String) key, properties.getProperty((String) key));
191+
}
192+
}
193+
}
194+
}
195+
182196
private Interpreter getInterpreter(String noteId, String className) throws TException {
183197
if (interpreterGroup == null) {
184198
throw new TException(

0 commit comments

Comments
 (0)