Skip to content

Commit 4efb39f

Browse files
Leemoonsoominahlee
Leemoonsoo
authored and
minahlee
committed
[ZEPPELIN-1046] bin/install-interpreter.sh for netinst package
### What is this PR for? Implementation of bin/install-interpreter.sh for netinst package which suggested in the [discussion](http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Ask-opinion-regarding-0-6-0-release-package-tp3298p3314.html). Some usages will be ``` # download all interpreters provided by Apache Zeppelin project bin/install-interpreter.sh --all # download an interpreter with name (for example markdown interpreter) bin/install-interpreter.sh --name md # download an (3rd party) interpreter with specific maven artifact name bin/install-interpreter.sh --name md -t org.apache.zeppelin:zeppelin-markdown:0.6.0-SNAPSHOT ``` If it looks fine, i'll continue the work (refactor code, and add test) ### What type of PR is it? Feature ### Todos * [x] - working implementation * [x] - refactor * [x] - add test ### What is the Jira issue? * Open an issue on Jira https://issues.apache.org/jira/browse/ZEPPELIN/ * Put link here, and add [ZEPPELIN-*Jira number*] in PR title, eg. [ZEPPELIN-533] ### How should this be tested? Outline the steps to test the PR here. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? * Is there breaking changes for older versions? * Does this needs documentation? Author: Lee moon soo <[email protected]> Author: AhyoungRyu <[email protected]> Closes #1042 from Leemoonsoo/netinst and squashes the following commits: f81d16e [Lee moon soo] address mina's comment 049bc89 [Lee moon soo] Update docs 7307c67 [Lee moon soo] Merge remote-tracking branch 'AhyoungRyu/netinst-docs' into netinst 7e749ad [Lee moon soo] Address mina's comment 0eedd2a [AhyoungRyu] Address @minahlee feedback 13f2d04 [Lee moon soo] generate netinst package 03c664e [AhyoungRyu] Add a new line 5d0a971 [AhyoungRyu] Revert install.md to latest version 13899fb [AhyoungRyu] Reorganize interpreter installation docs 4c1f029 [Lee moon soo] Proxy support 9079580 [Lee moon soo] fix artifact name 1077296 [Lee moon soo] update test aebca17 [Lee moon soo] Add docs d547551 [Lee moon soo] Remove test entries 6ee06b8 [Lee moon soo] Make DependencyResolver in zeppelin-interpreter module not aware of ZEPPELIN_HOME 7b1b36a [Lee moon soo] update usage 49f0568 [Lee moon soo] Add conf/interpreter-list 1b558fd [Lee moon soo] update some text ec7d152 [Lee moon soo] add tip 2c81a3f [Lee moon soo] update 78a7c52 [Lee moon soo] Refactor and add test 47f5706 [Lee moon soo] Install multiple interpreters at once 38e2556 [Lee moon soo] Initial implementation of install-interpreter.sh
1 parent df7dd5c commit 4efb39f

File tree

13 files changed

+680
-41
lines changed

13 files changed

+680
-41
lines changed

‎bin/install-interpreter.sh‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
#!/bin/bash
2+
#
3+
# Licensed to the Apache Software Foundation (ASF) under one
4+
# or more contributor license agreements. See the NOTICE file
5+
# distributed with this work for additional information
6+
# regarding copyright ownership. The ASF licenses this file
7+
# to you under the Apache License, Version 2.0 (the
8+
# "License"); you may not use this file except in compliance
9+
# with the License. You may obtain a copy of the License at
10+
#
11+
# http://www.apache.org/licenses/LICENSE-2.0
12+
#
13+
# Unless required by applicable law or agreed to in writing, software
14+
# distributed under the License is distributed on an "AS IS" BASIS,
15+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
16+
# See the License for the specific language governing permissions and
17+
# limitations under the License.
18+
#
19+
# Run Zeppelin
20+
#
21+
22+
bin=$(dirname "${BASH_SOURCE-$0}")
23+
bin=$(cd "${bin}">/dev/null; pwd)
24+
25+
. "${bin}/common.sh"
26+
27+
28+
ZEPPELIN_INSTALL_INTERPRETER_MAIN=org.apache.zeppelin.interpreter.install.InstallInterpreter
29+
ZEPPELIN_LOGFILE="${ZEPPELIN_LOG_DIR}/install-interpreter.log"
30+
JAVA_OPTS+=" -Dzeppelin.log.file=${ZEPPELIN_LOGFILE}"
31+
32+
if [[ -d "${ZEPPELIN_HOME}/zeppelin-zengine/target/classes" ]]; then
33+
ZEPPELIN_CLASSPATH+=":${ZEPPELIN_HOME}/zeppelin-zengine/target/classes"
34+
fi
35+
addJarInDir "${ZEPPELIN_HOME}/zeppelin-server/target/lib"
36+
37+
if [[ -d "${ZEPPELIN_HOME}/zeppelin-interpreter/target/classes" ]]; then
38+
ZEPPELIN_CLASSPATH+=":${ZEPPELIN_HOME}/zeppelin-interpreter/target/classes"
39+
fi
40+
addJarInDir "${ZEPPELIN_HOME}/zeppelin-interpreter/target/lib"
41+
42+
addJarInDir "${ZEPPELIN_HOME}/lib"
43+
44+
CLASSPATH+=":${ZEPPELIN_CLASSPATH}"
45+
$ZEPPELIN_RUNNER $JAVA_OPTS -cp $CLASSPATH $ZEPPELIN_INSTALL_INTERPRETER_MAIN ${@}

‎conf/interpreter-list‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
#
18+
# [name] [maven artifact] [description]
19+
20+
alluxio org.apache.zeppelin:zeppelin-alluxio:0.6.0 Alluxio interpreter
21+
angular org.apache.zeppelin:zeppelin-angular:0.6.0 HTML and AngularJS view rendering
22+
cassandra org.apache.zeppelin:zeppelin-cassandra:0.6.0 Cassandra interpreter
23+
elasticsearch org.apache.zeppelin:zeppelin-elasticsearch:0.6.0 Elasticsearch interpreter
24+
file org.apache.zeppelin:zeppelin-file:0.6.0 HDFS file interpreter
25+
flink org.apache.zeppelin:zeppelin-flink:0.6.0 Flink interpreter
26+
hbase org.apache.zeppelin:zeppelin-hbase:0.6.0 Hbase interpreter
27+
ignite org.apache.zeppelin:zeppelin-ignite:0.6.0 Ignite interpreter
28+
jdbc org.apache.zeppelin:zeppelin-jdbc:0.6.0 Jdbc interpreter
29+
kylin org.apache.zeppelin:zeppelin-kylin:0.6.0 Kylin interpreter
30+
lens org.apache.zeppelin:zeppelin-lens:0.6.0 Lens interpreter
31+
livy org.apache.zeppelin:zeppelin-livy:0.6.0 Livy interpreter
32+
md org.apache.zeppelin:zeppelin-markdown:0.6.0 Markdown support
33+
postgresql org.apache.zeppelin:zeppelin-postgresql:0.6.0 Postgresql interpreter
34+
python org.apache.zeppelin:zeppelin-python:0.6.0 Python interpreter
35+
shell org.apache.zeppelin:zeppelin-shell:0.6.0 Shell command

‎dev/create_release.sh‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -103,6 +103,7 @@ function make_binary_release() {
103103
git_clone
104104
make_source_package
105105
make_binary_release all "-Pspark-1.6 -Phadoop-2.4 -Pyarn -Ppyspark -Psparkr -Pr"
106+
make_binary_release netinst "-Pspark-1.6 -Phadoop-2.4 -Pyarn -Ppyspark -pl !alluxio,!angular,!cassandra,!elasticsearch,!file,!flink,!hbase,!ignite,!jdbc,!kylin,!lens,!livy,!markdown,!postgresql,!python,!shell"
106107

107108
# remove non release files and dirs
108109
rm -rf "${WORKING_DIR}/zeppelin"

‎docs/_includes/themes/zeppelin/_navigation.html‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,8 +21,8 @@
2121
<li><a href="{{BASE_PATH}}/index.html">What is Apache Zeppelin ?</a></li>
2222
<li role="separator" class="divider"></li>
2323
<li class="title"><span><b>Getting Started</b><span></li>
24-
<li><a href="{{BASE_PATH}}/install/install.html">Quick Start</a></li>
25-
<li><a href="{{BASE_PATH}}/install/install.html#zeppelin-configuration">Configuration</a></li>
24+
<li><a href="{{BASE_PATH}}/install/install.html">Install</a></li>
25+
<li><a href="{{BASE_PATH}}/install/install.html#apache-zeppelin-configuration">Configuration</a></li>
2626
<li><a href="{{BASE_PATH}}/quickstart/explorezeppelinui.html">Explore Zeppelin UI</a></li>
2727
<li><a href="{{BASE_PATH}}/quickstart/tutorial.html">Tutorial</a></li>
2828
<li role="separator" class="divider"></li>
@@ -42,6 +42,7 @@
4242
<li><a href="{{BASE_PATH}}/manual/interpreters.html">Overview</a></li>
4343
<li role="separator" class="divider"></li>
4444
<li class="title"><span><b>Usage</b><span></li>
45+
<li><a href="{{BASE_PATH}}/manual/interpreterinstallation.html">Interpreter Installation</a></li>
4546
<li><a href="{{BASE_PATH}}/manual/dynamicinterpreterload.html">Dynamic Interpreter Loading</a></li>
4647
<li><a href="{{BASE_PATH}}/manual/dependencymanagement.html">Interpreter Dependency Management</a></li>
4748
<li role="separator" class="divider"></li>
@@ -110,3 +111,4 @@
110111
</nav><!--/.navbar-collapse -->
111112
</div>
112113
</div>
114+

‎docs/install/install.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,9 @@ Although it can be unstable somehow since it is on development status, you can e
6060
### Downloading Binary Package
6161

6262
If you want to install Apache Zeppelin with a stable binary package, please visit [Apache Zeppelin download Page](http://zeppelin.apache.org/download.html).
63+
64+
If you have downloaded `netinst` binary, [install additional interpreters](../manual/interpreterinstallation.html) before you start Zeppelin. Or simply run `./bin/install-interpreter.sh --all`.
65+
6366
After unpacking, jump to [Starting Apache Zeppelin with Command Line](#starting-apache-zeppelin-with-command-line) section.
6467

6568
### Building from Source
@@ -388,4 +391,4 @@ You can configure Apache Zeppelin with both **environment variables** in `conf/z
388391
<td>1024000</td>
389392
<td>Size in characters of the maximum text message to be received by websocket.</td>
390393
</tr>
391-
</table>
394+
</table>

‎docs/manual/interpreterinstallation.md‎

Lines changed: 164 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
---
2+
layout: page
3+
title: "Interpreter Installation"
4+
description: ""
5+
group: manual
6+
---
7+
<!--
8+
Licensed under the Apache License, Version 2.0 (the "License");
9+
you may not use this file except in compliance with the License.
10+
You may obtain a copy of the License at
11+
12+
http://www.apache.org/licenses/LICENSE-2.0
13+
14+
Unless required by applicable law or agreed to in writing, software
15+
distributed under the License is distributed on an "AS IS" BASIS,
16+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
17+
See the License for the specific language governing permissions and
18+
limitations under the License.
19+
-->
20+
{% include JB/setup %}
21+
22+
# Interpreter Installation
23+
24+
Apache Zeppelin provides **Interpreter Installation** mechanism for whom downloaded Zeppelin `netinst` binary package, or just want to install another 3rd party interpreters.
25+
26+
## Community managed interpreters
27+
Apache Zeppelin provides several interpreters as [community managed interpreters](#available-community-managed-interpreters).
28+
If you downloaded `netinst` binary package, you need to install by using below commands.
29+
30+
#### Install all community managed interpreters
31+
32+
```
33+
./bin/install-interpreter.sh --all
34+
```
35+
36+
#### Install specific interpreters
37+
38+
```
39+
./bin/install-interpreter.sh --name md,shell,jdbc,python
40+
```
41+
42+
You can get full list of community managed interpreters by running
43+
44+
```
45+
./bin/install-interpreter.sh --list
46+
```
47+
48+
Once you have installed interpreters, you need to restart Zeppelin. And then [create interpreter setting](../manual/interpreters.html#what-is-zeppelin-interpreter) and [bind it with your notebook](../manual/interpreters.html#what-is-zeppelin-interpreter-setting).
49+
50+
51+
## 3rd party interpreters
52+
53+
You can also install 3rd party interpreters located in the maven repository by using below commands.
54+
55+
#### Install 3rd party interpreters
56+
57+
```
58+
./bin/install-interpreter.sh --name interpreter1 --artifact groupId1:artifact1:version1
59+
```
60+
61+
The above command will download maven artifact `groupId1:artifact1:version1` and all of it's transitive dependencies into `interpreter/interpreter1` directory.
62+
63+
Once you have installed interpreters, you'll need to add interpreter class name into `zeppelin.interpreters` property in [configuration](../install/install.html#apache-zeppelin-configuration).
64+
And then restart Zeppelin, [create interpreter setting](../manual/interpreters.html#what-is-zeppelin-interpreter) and [bind it with your notebook](../manual/interpreters.html#what-is-zeppelin-interpreter-setting).
65+
66+
67+
#### Install multiple 3rd party interpreters at once
68+
69+
```
70+
./bin/install-interpreter.sh --name interpreter1,interpreter2 --artifact groupId1:artifact1:version1,groupId2:artifact2:version2
71+
```
72+
73+
`--name` and `--artifact` arguments will recieve comma separated list.
74+
75+
## Available community managed interpreters
76+
77+
You can also find the below community managed interpreter list in `conf/interpreter-list` file.
78+
<table class="table-configuration">
79+
<tr>
80+
<th>Name</th>
81+
<th>Maven Artifact</th>
82+
<th>Description</th>
83+
</tr>
84+
<tr>
85+
<td>alluxio</td>
86+
<td>org.apache.zeppelin:zeppelin-alluxio:0.6.0</td>
87+
<td>Alluxio interpreter</td>
88+
</tr>
89+
<tr>
90+
<td>angular</td>
91+
<td>org.apache.zeppelin:zeppelin-angular:0.6.0</td>
92+
<td>HTML and AngularJS view rendering</td>
93+
</tr>
94+
<tr>
95+
<td>cassandra</td>
96+
<td>org.apache.zeppelin:zeppelin-cassandra:0.6.0</td>
97+
<td>Cassandra interpreter</td>
98+
</tr>
99+
<tr>
100+
<td>elasticsearch</td>
101+
<td>org.apache.zeppelin:zeppelin-elasticsearch:0.6.0</td>
102+
<td>Elasticsearch interpreter</td>
103+
</tr>
104+
<tr>
105+
<td>file</td>
106+
<td>org.apache.zeppelin:zeppelin-file:0.6.0</td>
107+
<td>HDFS file interpreter</td>
108+
</tr>
109+
<tr>
110+
<td>flink</td>
111+
<td>org.apache.zeppelin:zeppelin-flink:0.6.0</td>
112+
<td>Flink interpreter</td>
113+
</tr>
114+
<tr>
115+
<td>hbase</td>
116+
<td>org.apache.zeppelin:zeppelin-hbase:0.6.0</td>
117+
<td>Hbase interpreter</td>
118+
</tr>
119+
<tr>
120+
<td>ignite</td>
121+
<td>org.apache.zeppelin:zeppelin-ignite:0.6.0</td>
122+
<td>Ignite interpreter</td>
123+
</tr>
124+
<tr>
125+
<td>jdbc</td>
126+
<td>org.apache.zeppelin:zeppelin-jdbc:0.6.0</td>
127+
<td>Jdbc interpreter</td>
128+
</tr>
129+
<tr>
130+
<td>kylin</td>
131+
<td>org.apache.zeppelin:zeppelin-kylin:0.6.0</td>
132+
<td>Kylin interpreter</td>
133+
</tr>
134+
<tr>
135+
<td>lens</td>
136+
<td>org.apache.zeppelin:zeppelin-lens:0.6.0</td>
137+
<td>Lens interpreter</td>
138+
</tr>
139+
<tr>
140+
<td>livy</td>
141+
<td>org.apache.zeppelin:zeppelin-livy:0.6.0</td>
142+
<td>Livy interpreter</td>
143+
</tr>
144+
<tr>
145+
<td>md</td>
146+
<td>org.apache.zeppelin:zeppelin-markdown:0.6.0</td>
147+
<td>Markdown support</td>
148+
</tr>
149+
<tr>
150+
<td>postgresql</td>
151+
<td>org.apache.zeppelin:zeppelin-postgresql:0.6.0</td>
152+
<td>Postgresql interpreter</td>
153+
</tr>
154+
<tr>
155+
<td>python</td>
156+
<td>org.apache.zeppelin:zeppelin-python:0.6.0</td>
157+
<td>Python interpreter</td>
158+
</tr>
159+
<tr>
160+
<td>shell</td>
161+
<td>org.apache.zeppelin:zeppelin-shell:0.6.0</td>
162+
<td>Shell command</td>
163+
</tr>
164+
</table>

‎zeppelin-interpreter/src/main/java/org/apache/zeppelin/dep/AbstractDependencyResolver.java‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717

1818
package org.apache.zeppelin.dep;
1919

20+
import java.net.URL;
2021
import java.util.Collection;
2122
import java.util.Iterator;
2223
import java.util.LinkedList;
@@ -25,6 +26,7 @@
2526
import org.sonatype.aether.RepositorySystem;
2627
import org.sonatype.aether.RepositorySystemSession;
2728
import org.sonatype.aether.repository.Authentication;
29+
import org.sonatype.aether.repository.Proxy;
2830
import org.sonatype.aether.repository.RemoteRepository;
2931
import org.sonatype.aether.repository.RepositoryPolicy;
3032
import org.sonatype.aether.resolution.ArtifactResult;
@@ -44,6 +46,16 @@ public AbstractDependencyResolver(String localRepoPath) {
4446
repos.add(Booter.newLocalRepository());
4547
}
4648

49+
public void setProxy(URL proxyUrl, String proxyUser, String proxyPassword) {
50+
Authentication auth = new Authentication(proxyUser, proxyPassword);
51+
Proxy proxy = new Proxy(proxyUrl.getProtocol(), proxyUrl.getHost(), proxyUrl.getPort(), auth);
52+
synchronized (repos) {
53+
for (RemoteRepository repo : repos) {
54+
repo.setProxy(proxy);
55+
}
56+
}
57+
}
58+
4759
public List<RemoteRepository> getRepos() {
4860
return this.repos;
4961
}

‎zeppelin-interpreter/src/main/java/org/apache/zeppelin/dep/DependencyResolver.java‎

Lines changed: 8 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@
1919

2020
import java.io.File;
2121
import java.io.IOException;
22+
import java.net.URL;
2223
import java.util.Arrays;
2324
import java.util.Collection;
2425
import java.util.Iterator;
@@ -63,11 +64,6 @@ public List<File> load(String artifact)
6364
return load(artifact, new LinkedList<String>());
6465
}
6566

66-
public List<File> load(String artifact, String destPath)
67-
throws RepositoryException, IOException {
68-
return load(artifact, new LinkedList<String>(), destPath);
69-
}
70-
7167
public synchronized List<File> load(String artifact, Collection<String> excludes)
7268
throws RepositoryException, IOException {
7369
if (StringUtils.isBlank(artifact)) {
@@ -85,25 +81,20 @@ public synchronized List<File> load(String artifact, Collection<String> excludes
8581
return libs;
8682
}
8783
}
88-
89-
public List<File> load(String artifact, Collection<String> excludes, String destPath)
84+
85+
public List<File> load(String artifact, File destPath) throws IOException, RepositoryException {
86+
return load(artifact, new LinkedList<String>(), destPath);
87+
}
88+
89+
public List<File> load(String artifact, Collection<String> excludes, File destPath)
9090
throws RepositoryException, IOException {
9191
List<File> libs = new LinkedList<File>();
9292

9393
if (StringUtils.isNotBlank(artifact)) {
9494
libs = load(artifact, excludes);
9595

96-
// find home dir
97-
String home = System.getenv("ZEPPELIN_HOME");
98-
if (home == null) {
99-
home = System.getProperty("zeppelin.home");
100-
}
101-
if (home == null) {
102-
home = "..";
103-
}
104-
10596
for (File srcFile : libs) {
106-
File destFile = new File(home + "/" + destPath, srcFile.getName());
97+
File destFile = new File(destPath, srcFile.getName());
10798
if (!destFile.exists() || !FileUtils.contentEquals(srcFile, destFile)) {
10899
FileUtils.copyFile(srcFile, destFile);
109100
logger.info("copy {} to {}", srcFile.getAbsolutePath(), destPath);

0 commit comments

Comments
 (0)