<title>{{ page.title }} - Spark {{site.SPARK_VERSION_SHORT}} Documentation</title>

<meta name="description" content="">

<link rel="stylesheet" href="css/bootstrap.min.css">

<style>

body {

<a href="#" class="dropdown-toggle" data-toggle="dropdown">Programming Guides<b class="caret"></b></a>

<ul class="dropdown-menu">

<li><a href="quick-start.html">Quick Start</a></li>

<li class="divider"></li>

<li><a href="streaming-programming-guide.html">Spark Streaming</a></li>

<li><a href="sql-programming-guide.html">Spark SQL</a></li>

<li><a href="mllib-guide.html">MLlib (Machine Learning)</a></li>

<li><a href="graphx-programming-guide.html">GraphX (Graph Processing)</a></li>

</ul>

</li>

For example, we can use Bagel to implement PageRank. Here, vertices represent pages, edges represent links between pages, and messages represent shares of PageRank sent to the pages that a particular page links to.

In Spark, RDDs are not persisted in memory by default. To avoid recomputation, they must be explicitly cached when using them multiple times (see the [Spark Programming Guide][RDD Persistence]). Graphs in GraphX behave the same way. **When using a graph multiple times, make sure to call [`Graph.cache()`][Graph.cache] on it first.**

[Graph.cache]: api/scala/index.html#org.apache.spark.graphx.Graph@cache():Graph[VD,ED]

In iterative computations, *uncaching* may also be necessary for best performance. By default, cached RDDs and graphs will remain in memory until memory pressure forces them to be evicted in LRU order. For iterative computation, intermediate results from previous iterations will fill up the cache. Though they will eventually be evicted, the unnecessary data stored in memory will slow down garbage collection. It would be more efficient to uncache intermediate results as soon as they are no longer necessary. This involves materializing (caching and forcing) a graph or RDD every iteration, uncaching all other datasets, and only using the materialized dataset in future iterations. However, because graphs are composed of multiple RDDs, it can be difficult to unpersist them correctly. **For iterative computation we recommend using the Pregel API, which correctly unpersists intermediate results.**

Apache Spark is a fast and general-purpose cluster computing system.

It also supports a rich set of higher-level tools including [Shark](http://shark.cs.berkeley.edu) (Hive on Spark), [MLlib](mllib-guide.html) for machine learning, [GraphX](graphx-programming-guide.html) for graph processing, and [Spark Streaming](streaming-programming-guide.html).

contains Spark packages for many popular HDFS versions. If you'd like to build Spark from

scratch, visit the [building with Maven](building-with-maven.html) page.

For its Scala API, Spark {{site.SPARK_VERSION}} depends on Scala {{site.SCALA_BINARY_VERSION}}.

If you write applications in Scala, you will need to use a compatible Scala version

./bin/spark-shell --master local[2]

The `--master` option specifies the

locally with one thread, or `local[N]` to run locally with N threads. You should start by using

`local` for testing. For a full list of options, run Spark shell with the `--help` option.

**Programming guides:**

* [Quick Start](quick-start.html): a quick introduction to the Spark API; start here!

* [Spark Streaming](streaming-programming-guide.html): Spark's API for processing data streams

* [Spark SQL](sql-programming-guide.html): Support for running relational queries on Spark

* [MLlib (Machine Learning)](mllib-guide.html): Spark's built-in machine learning library

The Spark Java API exposes all the Spark features available in the Scala version to Java.

To learn the basics of Spark, we recommend reading through the

easy to follow even if you don't know Scala.

This guide will show how to use the Spark features described there in Java.

declared in the [org.apache.spark.api.java.StorageLevels](api/java/index.html?org/apache/spark/api/java/StorageLevels.html) class. To

define your own storage level, you can use StorageLevels.create(...).

The Java API supports other Spark features, including

$\cdot n$`.

In each iteration, the sampling over the distributed dataset

computation of the sum of the partial results from each worker machine is performed by the

standard spark routines.

The Spark Python API (PySpark) exposes the Spark programming model to Python.

To learn the basics of Spark, we recommend reading through the

easy to follow even if you don't know Scala.

This guide will show how to use the Spark features described there in Python.

This tutorial provides a quick introduction to using Spark. We will first introduce the API through Spark's

interactive shell (in Python or Scala),

then show how to write standalone applications in Java, Scala, and Python.

To follow along with this guide, first download a packaged release of Spark from the

[Spark website](http://spark.apache.org/downloads.html). Since we won't be using HDFS,

textFile: spark.RDD[String] = spark.MappedRDD@2ee9b6e3

{% endhighlight %}

{% highlight scala %}

scala> textFile.count() // Number of items in this RDD

res1: String = # Apache Spark

{% endhighlight %}

{% highlight scala %}

scala> val linesWithSpark = textFile.filter(line => line.contains("Spark"))

{% endhighlight %}

{% highlight python %}

u'# Apache Spark'

{% endhighlight %}

{% highlight python %}

wordCounts: spark.RDD[(String, Int)] = spark.ShuffledAggregatedRDD@71f027b8

{% endhighlight %}

{% highlight scala %}

scala> wordCounts.collect()

{% endhighlight %}

{% highlight python %}

It may seem silly to use Spark to explore and cache a 100-line text file. The interesting part is

that these same functions can be used on very large data sets, even when they are striped across

tens or hundreds of nodes. You can also do this interactively by connecting `bin/spark-shell` to

</div>

<div data-lang="python" markdown="1">

It may seem silly to use Spark to explore and cache a 100-line text file. The interesting part is

that these same functions can be used on very large data sets, even when they are striped across

tens or hundreds of nodes. You can also do this interactively by connecting `bin/pyspark` to

</div>

</div>

The Master URLs for Mesos are in the form `mesos://host:5050` for a single-master Mesos

The driver also needs some configuration in `spark-env.sh` to interact properly with Mesos: