{"id":53748,"date":"2016-03-09T19:00:43","date_gmt":"2016-03-09T17:00:43","guid":{"rendered":"https:\/\/www.javacodegeeks.com\/?p=53748"},"modified":"2016-03-09T14:50:55","modified_gmt":"2016-03-09T12:50:55","slug":"get-started-using-apache-spark-graphx-scala","status":"publish","type":"post","link":"https:\/\/www.javacodegeeks.com\/2016\/03\/get-started-using-apache-spark-graphx-scala.html","title":{"rendered":"How to Get Started Using Apache Spark GraphX with Scala"},"content":{"rendered":"

Editor’s Note:<\/strong> Don’t miss our new free on-demand training course about how to create data pipeline applications using Apache Spark \u2013 learn more here.<\/a><\/p>\n

This post will help you get started using Apache Spark GraphX with Scala on the MapR Sandbox. GraphX is the Apache Spark component for graph-parallel computations, built upon a branch of mathematics called graph theory. It is a distributed graph processing framework that sits on top of the Spark core.<\/p>\n

Overview of some graph concepts<\/h2>\n

A graph is a mathematical structure used to model relations between objects. A graph is made up of vertices and edges that connect them. The vertices are the objects and the edges are the relationships between them.<\/p>\n

\"image00_edge-vertex-relationship\"<\/a><\/p>\n

A directed graph<\/strong> is a graph where the edges have a direction associated with them. An example of a directed graph is a Twitter follower. User Bob can follow user Carol without implying that user Carol follows user Bob.<\/p>\n

\"image02_bob-follows-carol\"<\/a><\/p>\n

A regular graph<\/strong> is a graph where each vertex has the same number of edges. An example of a regular graph is Facebook friends. If Bob is a friend of Carol, then Carol is also a friend of Bob.<\/p>\n

GraphX Property Graph<\/h4>\n

GraphX extends the Spark RDD with a Resilient Distributed Property Graph.<\/p>\n

The property graph<\/a> is a directed multigraph which can have multiple edges in parallel. Every edge and vertex has user defined properties associated with it. The parallel edges allow multiple relationships between the same vertices.<\/p>\n

\"image01_flight-relationship\"<\/a><\/p>\n

In this activity, you will use GraphX to analyze flight data.<\/p>\n

Scenario<\/h2>\n

As a starting simple example, we will analyze three flights. For each flight, we have the following information:<\/p>\n\n\n\n\n\n\n\n
Originating Airport<\/strong><\/td>\nDestination Airport<\/strong><\/td>\nDistance<\/strong><\/td>\n<\/tr>\n<\/thead>\n
SFO<\/td>\nORD<\/td>\n1800 miles<\/td>\n<\/tr>\n
ORD<\/td>\nDFW><\/td>\n800 miles<\/td>\n<\/tr>\n
DFW<\/td>\nSFO><\/td>\n1400 miles<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

 <\/p>\n

In this scenario, we are going to represent the airports as vertices and routes as edges. For our graph we will have three vertices, each representing an airport. The distance between the airports is a route property, as shown below:\"image04_3-vertex-relationship\"<\/a><\/p>\n

Vertex Table for Airports<\/strong><\/h4>\n\n\n\n<\/thead>\n\n\n\n\n
ID<\/strong><\/td>\nProperty<\/strong><\/td>\n<\/tr>\n<\/thead>\n
1<\/td>\nSFO<\/td>\n<\/tr>\n
2<\/td>\nORD<\/td>\n<\/tr>\n
3<\/td>\nDFW<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

 <\/p>\n

Edges Table for Routes<\/strong><\/h4>\n\n\n\n<\/thead>\n\n\n\n\n
SrcId<\/strong><\/td>\nDestId<\/strong><\/td>\nProperty<\/strong><\/td>\n<\/tr>\n<\/thead>\n
1<\/td>\n2<\/td>\n1800<\/td>\n<\/tr>\n
2<\/td>\n3<\/td>\n800<\/td>\n<\/tr>\n
3<\/td>\n1<\/td>\n1400<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

 <\/p>\n

Software<\/h4>\n

This tutorial will run on the MapR Sandbox, which includes Spark.<\/p>\n