RDDs

RDDs means “resilient distributed datasets”, which are Spark’s main programming abstraction. RDDs represent a collection of items distributed across many compute nodes that can be manipulated in parallel. Spark Core provides many APIs for building and manipulating these collections.

RDDs can be splitted into partitions (Each partition can be compared with blocks in HDFS). Partitions are distributed into multiple worker nodes. RDDs are stored in RAM by default.

On each RDD, we can perform 2 types of operations.
(i) Transformations
(ii) Actions

  • RDDs are lazy
  • Execution style is called ‘Lazy Evaluation’ (RDD will be computed whenever action is performed from beginning of the flow to Actioned RDD).
  • When computation happens on RDDs, they stored in RAM by default. Optionally, we can save RDDs into Disk.
  • If RDD has multiple partitions, computations happens parallel.
  • If RDD size is bigger than RAM availability, RDD will wait for resources to release. Otherwise, it will failed.

Each RDD is available, till next RDD is constructed.

Example:
Suppose we have RDD1->RDD2->RDD3, and RDD3.Count() is ACTION.
Step 1: RDD1 is created
Step 2: RDD2 is created, and RDD1 is deleted.
Step 3: RDD3 is created and RDD2 is deleted. Once COUNT action is completed, RDD3 will be deleted.

When RDD will be created

  1. When you parallelize an object 
    ex: val l = List(10,20,30,40,50,60)
    val rdd = sc.parallelize(l,2)
  2. When spark context has read data from a file (local/hdfs/s3) 
    ex: val rddx = sc.textFile("/usr/file")
  3. When perform an operation (transformation/filter) on rdd, another rdd will be created 
    ex: val rddy = rddx.map(x => x+10)

Key advantages of SPARK RDD models:

  1. More Parallel computing than MapReduce
  2. In Memory computation
  3. Reusability of transformations

Persist()

If we would like to reuse an RDD in multiple actions, we can ask spark to persist it using RDD.persist(). We can ask Spark to persist our data in a number of different places. After computing it the first time, Spark will store the RDD contents in memory (partitioned across the machines in your cluster), and reuse them in future actions. Persisting RDDs on disk instead of memory is also possible. The behavior of not persisting by default may again seem unusual, but it makes a lot of sense for big datasets. If you will not reuse the RDD, there’s no reason to waste storage space when Spark could instead stream through the data once and just compute the result.

How fault tolerance is applied?

If any worker node which has persist RDD is down, DAG will observe it and re-execute the flow till persist end.

What if persist RDD has partitions?

If any machine which has partition as a part of persistent RDD is down, all remaining partition will be deallocated from RAM and flow will be re-executed till persistent RDD creates.