Skip to content

Commit 4c8d72d

Browse files
author
astroshim
committed
merge with Ayoung's
2 parents 8c62cf1 + 86ca513 commit 4c8d72d

File tree

2 files changed

+12
-13
lines changed

2 files changed

+12
-13
lines changed

‎docs/assets/themes/zeppelin/img/docs-img/zeppelin_yarn_conf.png‎

138 KB
Loading

‎docs/install/spark_cluster_mode.md‎

Lines changed: 12 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
---
22
layout: page
33
title: "Apache Zeppelin on Spark cluster mode"
4-
description: ""
4+
description: "This document will guide you how you can build and configure the environment on 3 types of Spark cluster manager with Apache Zeppelin using docker scripts."
55
group: install
66
---
77
<!--
@@ -56,12 +56,12 @@ spark_standalone bash;
5656
```
5757

5858
### 3. Configure Spark interpreter in Zeppelin
59-
Set Spark master as `spark://localhost:7077` in Zeppelin **Interpreters** setting page.
59+
Set Spark master as `spark://<hostname>:7077` in Zeppelin **Interpreters** setting page.
6060

6161
<img src="../assets/themes/zeppelin/img/docs-img/standalone_conf.png" />
6262

6363
### 4. Run Zeppelin with Spark interpreter
64-
After running single paragraph with Spark interpreter in Zeppelin, browse `https://localhost:8080` and check whether Spark cluster is running well or not.
64+
After running single paragraph with Spark interpreter in Zeppelin, browse `https://<hostname>:8080` and check whether Spark cluster is running well or not.
6565

6666
<img src="../assets/themes/zeppelin/img/docs-img/spark_ui.png" />
6767

@@ -115,29 +115,28 @@ docker run -it \
115115

116116
You can simply verify the processes of Spark and YARN is running well in Docker with below command.
117117

118-
119118
```
120-
ps -ef
119+
ps -ef | grep spark
121120
```
122121

123-
and also You can see the hdfs web ui on `http://localhost:50070/` and yarn ui on `http://localhost:8088/cluster` and spark ui on `http://localhost:8080/`.
124-
122+
You can also check each application web UI for HDFS on `http://<hostname>:50070/`, YARN on `http://<hostname>:8088/cluster` and Spark on `http://<hostname>:8080/`.
125123

126124
### 4. Configure Spark interpreter in Zeppelin
127-
Set following configurations to the `zeppelin-env.sh`.
125+
Set following configurations to `conf/zeppelin-env.sh`.
128126

129127
```
130128
export MASTER=yarn-client
131-
export HADOOP_CONF_DIR=[PATH OF HADOOP CONF]
132-
export SPARK_HOME=[PATH OF SPARK HOME]
129+
export HADOOP_CONF_DIR=[your_hadoop_conf_path]
130+
export SPARK_HOME=[your_spark_home_path]
133131
```
134-
Hadoop configuration path(HADOOP_CONF_DIR) is the `scripts/docker/spark-cluster-managers/spark_yarn_cluster/hdfs_conf`.
135132

136-
Please make sure the spark master as `yarn-client` in Zeppelin **Interpreters** setting page.
133+
`HADOOP_CONF_DIR`(Hadoop configuration path) is defined in `/scripts/docker/spark-cluster-managers/spark_yarn_cluster/hdfs_conf`.
134+
135+
Don't forget to set Spark `master` as `yarn-client` in Zeppelin **Interpreters** setting page like below.
137136

138137
<img src="../assets/themes/zeppelin/img/docs-img/zeppelin_yarn_conf.png" />
139138

140139
### 5. Run Zeppelin with Spark interpreter
141-
After running single paragraph with Spark interpreter in Zeppelin, browse `http://localhost:8088/cluster/apps` and check zeppelin application running well or not.
140+
After running a single paragraph with Spark interpreter in Zeppelin, browse `http://<hostname>:8088/cluster/apps` and check Zeppelin application is running well or not.
142141

143142
<img src="../assets/themes/zeppelin/img/docs-img/yarn_applications.png" />

0 commit comments

Comments
 (0)