You can leverage [Zeppelin Dynamic Form]({{BASE_PATH}}/manual/dynamicform.html) inside your queries. You can use both the `text input` and `select form` parameterization features.

You can leverage [Zeppelin Dynamic Form]({{BASE_PATH}}/manual/dynamicform.html) inside your queries. You can use both the `text input` and `select form` parameterization features.

You can leverage [Zeppelin Dynamic Form]({{BASE_PATH}}/manual/dynamicform.html). You can use both the `text input` and `select form` parameterization features.

<img src="{{BASE_PATH}}/assets/themes/zeppelin/img/docs-img/markdown-interpreter-setting.png" width="60%" />

<img src="{{BASE_PATH}}/assets/themes/zeppelin/img/docs-img/markdown-example.png" width="70%" />

Apache Zeppelin [Table Display System]({{BASE_PATH}}/displaysystem/basicdisplaysystem.html#table) provides built-in data visualization capabilities. Python interpreter leverages it to visualize Pandas DataFrames though similar `z.show()` API, same as with [Matplotlib integration](#matplotlib-integration).

There is a convenience `%python.sql` interpreter that matches Apache Spark experience in Zeppelin and enables usage of SQL language to query [Pandas DataFrames](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html) and visualization of results though built-in [Table Display System]({{BASE_PATH}}/displaysystem/basicdisplaysystem.html#table).

<img src="{{BASE_PATH}}/assets/themes/zeppelin/img/docs-img/shell-example.png" />

description: "Apache Spark is a fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs."

It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs

Apache Spark is supported in Zeppelin with

Spark Interpreter group, which consists of five interpreters.

<td>Creates a SparkContext and provides a scala environment</td>

<td>%pyspark</td>

<td>Provides a python environment</td>

<td>%r</td>

<td>%sql</td>

<td>%dep</td>

In **conf/zeppelin-env.sh**, export `SPARK_HOME` environment variable with your Spark installation path.

for example

You can optionally export HADOOP\_CONF\_DIR and SPARK\_SUBMIT\_OPTIONS

For Windows, ensure you have `winutils.exe` in `%HADOOP_HOME%\bin`. For more details please see [Problems running Hadoop on Windows](https://wiki.apache.org/hadoop/WindowsProblems)

for example,

That's it. Zeppelin will work with any version of Spark and any deployment type without rebuilding Zeppelin in this way. (Zeppelin 0.5.6-incubating release works up to Spark 1.6.1 )

SparkContext, SQLContext, ZeppelinContext are automatically created and exposed as variable names 'sc', 'sqlContext' and 'z', respectively, both in scala and python environments.

There are two ways to load external library in spark interpreter. First is using Interpreter setting menu and second is loading Spark properties.

Once `SPARK_HOME` is set in `conf/zeppelin-env.sh`, Zeppelin uses `spark-submit` as spark interpreter runner. `spark-submit` supports two ways to load configurations. The first is command line options such as --master and Zeppelin can pass these options to `spark-submit` by exporting `SPARK_SUBMIT_OPTIONS` in conf/zeppelin-env.sh. Second is reading configuration options from `SPARK_HOME/conf/spark-defaults.conf`. Spark properites that user can set to distribute libraries are:

<th>Applicable Interpreter</th>

<td>%spark</td>

<td>%spark</td>

<td>Comma-separated list of maven coordinates of jars to include on the driver and executor classpaths. Will search the local maven repo, then maven central and any additional remote repositories given by --repositories. The format for the coordinates should be groupId:artifactId:version.</td>

<td>%pyspark</td>

* SPARK\_SUBMIT\_OPTIONS in conf/zeppelin-env.sh

* SPARK_HOME/conf/spark-defaults.conf

`%dep` interpreter load libraries to `%spark` and `%pyspark` but not to `%spark.sql` interpreter so we recommend you to use first option instead.

When your code requires external library, instead of doing download/copy/restart Zeppelin, you can easily do following jobs using `%dep` interpreter.

* Load libraries recursively from Maven repository

Dep interpreter leverages scala environment. So you can write any Scala code here.

Note that `%dep` interpreter should be used before `%spark`, `%pyspark`, `%sql`.

%dep

Zeppelin automatically injects ZeppelinContext as variable 'z' in your scala/python environment. ZeppelinContext provides some additional functions and utility.

ZeppelinContext extends map and it's shared between scala, python environment.

So you can put some object from scala and read it from python, vise versa.

%pyspark

ZeppelinContext provides functions for creating forms.

In scala and python environments, you can create forms programmatically.

%pyspark

Interpreter setting can choose one of 'shared', 'scoped', 'isolated' option. Spark interpreter creates separate scala compiler per each notebook but share a single SparkContext in 'scoped' mode (experimental). It creates separate SparkContext per each notebook in 'isolated' mode.

2. Set SPARK\_HOME in `[ZEPPELIN\_HOME]/conf/zeppelin-env.sh` to use spark-submit

(Additionally, you might have to set `export HADOOP\_CONF\_DIR=/etc/hadoop/conf`)

3. Add the two properties below to spark configuration (`[SPARK_HOME]/conf/spark-defaults.conf`):