You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/install/spark_cluster_mode.md
+64Lines changed: 64 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -72,3 +72,67 @@ ps -ef | grep spark
72
72
```
73
73
74
74
75
+
## Spark on Mesos mode
76
+
You can simply set up [Spark on Mesos](http://spark.apache.org/docs/latest/running-on-mesos.html) docker environment with below steps.
77
+
78
+
> **Note :** Since Apache Zeppelin and Spark use same `8080` port for their web UI, you might need to change `zeppelin.server.port` in `conf/zeppelin-site.xml`.
79
+
80
+
### 1. Build Docker file
81
+
You can find docker script files under `scripts/docker/spark-cluster-managers`.
82
+
83
+
84
+
```
85
+
cd $ZEPPELIN_HOME/scripts/docker/spark-cluster-managers/spark_mesos
86
+
docker build -t "spark_mesos" .
87
+
```
88
+
89
+
90
+
### 2. Run docker
91
+
92
+
```
93
+
docker run --net=host -it \
94
+
-p 8080:8080 \
95
+
-p 7077:7077 \
96
+
-p 8888:8888 \
97
+
-p 8081:8081 \
98
+
-p 8082:8082 \
99
+
-p 5050:5050 \
100
+
-p 5051:5051 \
101
+
-p 4040:4040 \
102
+
-h sparkmaster \
103
+
--name spark_mesos \
104
+
spark_mesos bash;
105
+
```
106
+
107
+
### 3. Verify running Spark on Mesos.
108
+
109
+
You can simply verify the processes of Spark and Mesos is running well in Docker with below command.
110
+
111
+
112
+
```
113
+
ps -ef
114
+
```
115
+
116
+
You can also check each application web UI for Mesos on `http://<hostname>:5050/cluster` and Spark on `http://<hostname>:8080/`.
117
+
118
+
119
+
### 4. Configure Spark interpreter in Zeppelin
120
+
Set following configurations to `conf/zeppelin-env.sh`.
121
+
122
+
```
123
+
export MASTER=mesos://127.0.1.1:5050
124
+
export MESOS_NATIVE_JAVA_LIBRARY=[PATH OF libmesos.so]
125
+
export SPARK_HOME=[PATH OF SPARK HOME]
126
+
```
127
+
128
+
129
+
Don't forget to set Spark `master` as `mesos://127.0.1.1:5050` in Zeppelin **Interpreters** setting page like below.
After running a single paragraph with Spark interpreter in Zeppelin, browse `http://<hostname>:5050/#/frameworks` and check Zeppelin application is running well or not.
0 commit comments