Skip to content

Commit 86ca513

Browse files
author
HyungSung
authored
Merge pull request #9 from AhyoungRyu/ZEPPELIN-1280-ahyoung
Minor update for spark_cluster_mode.md
2 parents 633c930 + cde5f8d commit 86ca513

File tree

2 files changed

+14
-15
lines changed

2 files changed

+14
-15
lines changed

‎docs/assets/themes/zeppelin/img/docs-img/zeppelin_yarn_conf.png‎

138 KB
Loading

‎docs/install/spark_cluster_mode.md‎

Lines changed: 14 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
---
22
layout: page
33
title: "Apache Zeppelin on Spark cluster mode"
4-
description: ""
4+
description: "This document will guide you how you can build and configure the environment on 3 types of Spark cluster manager with Apache Zeppelin using docker scripts."
55
group: install
66
---
77
<!--
@@ -56,12 +56,12 @@ spark_standalone bash;
5656
```
5757

5858
### 3. Configure Spark interpreter in Zeppelin
59-
Set Spark master as `spark://localhost:7077` in Zeppelin **Interpreters** setting page.
59+
Set Spark master as `spark://<hostname>:7077` in Zeppelin **Interpreters** setting page.
6060

6161
<img src="../assets/themes/zeppelin/img/docs-img/standalone_conf.png" />
6262

6363
### 4. Run Zeppelin with Spark interpreter
64-
After running single paragraph with Spark interpreter in Zeppelin, browse `https://localhost:8080` and check whether Spark cluster is running well or not.
64+
After running single paragraph with Spark interpreter in Zeppelin, browse `https://<hostname>:8080` and check whether Spark cluster is running well or not.
6565

6666
<img src="../assets/themes/zeppelin/img/docs-img/spark_ui.png" />
6767

@@ -73,7 +73,7 @@ ps -ef | grep spark
7373

7474

7575
## Spark on Yarn mode
76-
You can simply set up [Spark on Yarn](http://spark.apache.org/docs/latest/running-on-yarn.html) docker environment with below steps.
76+
You can simply set up [Spark on Yarn](http://spark.apache.org/docs/latest/running-on-yarn.html) environment with below steps using Docker scripts.
7777

7878
> **Note :** Since Apache Zeppelin and Spark use same `8080` port for their web UI, you might need to change `zeppelin.server.port` in `conf/zeppelin-site.xml`.
7979
@@ -113,31 +113,30 @@ docker run -it \
113113

114114
### 3. Verify running Spark on Yarn.
115115

116-
You can simply verify the processes of Spark and Yarn is running well in Docker with below command.
117-
116+
You can simply verify the processes of Spark and Yarn are running well in Docker with below command.
118117

119118
```
120-
ps -ef
119+
ps -ef | grep spark
121120
```
122121

123-
and also You can see the hdfs web ui on `http://localhost:50070/` and yarn ui on `http://localhost:8088/cluster` and spark ui on `http://localhost:8080/`.
124-
122+
You can also check each application web UI for HDFS on `http://<hostname>:50070/`, YARN on `http://<hostname>:8088/cluster` and Spark on `http://<hostname>:8080/`.
125123

126124
### 4. Configure Spark interpreter in Zeppelin
127-
Set following configurations to the `zeppelin-env.sh`.
125+
Set following configurations to `conf/zeppelin-env.sh`.
128126

129127
```
130128
export MASTER=yarn-client
131-
export HADOOP_CONF_DIR=[PATH OF HADOOP CONF]
132-
export SPARK_HOME=[PATH OF SPARK HOME]
129+
export HADOOP_CONF_DIR=[your_hadoop_conf_path]
130+
export SPARK_HOME=[your_spark_home_path]
133131
```
134-
Hadoop configuration path(HADOOP_CONF_DIR) is the `scripts/docker/spark-cluster-managers/spark_yarn_cluster/hdfs_conf`.
135132

136-
Please make sure the spark master as `yarn-client` in Zeppelin **Interpreters** setting page.
133+
`HADOOP_CONF_DIR`(Hadoop configuration path) is defined in `/scripts/docker/spark-cluster-managers/spark_yarn_cluster/hdfs_conf`.
134+
135+
Don't forget to set Spark `master` as `yarn-client` in Zeppelin **Interpreters** setting page like below.
137136

138137
<img src="../assets/themes/zeppelin/img/docs-img/zeppelin_yarn_conf.png" />
139138

140139
### 5. Run Zeppelin with Spark interpreter
141-
After running single paragraph with Spark interpreter in Zeppelin, browse `http://localhost:8088/cluster/apps` and check zeppelin application running well or not.
140+
After running a single paragraph with Spark interpreter in Zeppelin, browse `http://<hostname>:8088/cluster/apps` and check Zeppelin application is running well or not.
142141

143142
<img src="../assets/themes/zeppelin/img/docs-img/yarn_applications.png" />

0 commit comments

Comments
 (0)