Real time applications
using MapReduce
BY
ATHITHYA R
20PIT002
.
Analysis of logs, data analysis, recommendation
mechanisms, fraud detection, user behavior analysis,
genetic algorithms, scheduling problems, resource
planning among others, is applications that use
MapReduce.
• Let’s describe some practical applications of this
technology.
.
1 — Social Networks
• We are social networking users like Facebook, Twitter,
and LinkedIn to connect with our friends and community.
• Many of the features, such as who visited your LinkedIn
profile, who read your post on Facebook or Twitter, can be
evaluated using the MapReduce, programming model.
• 2 — Entertainment
• Netflix uses Hadoop and MapReduce to solve problems
such as discovering the most popular movies, based on what
you watched, what do you like? Providing suggestions to
registered users taken into account their interests.
• MapReduce can determine how users are watching
movies, analyzing their logs and clicks.
3 — Electronic Commerce
Many e-commerce providers, such as the Amazon, Walmart, and eBay,
use the MapReduce programming model to identify favorite products based on
users’ interests or buying behavior.
It includes creating product recommendation mechanisms for e-commerce
catalogs, analyzing site records, purchase history, user interaction logs, and so
on.
It’s used to establish a user’s sentimental profile for a particular product by
reviewing comments or reviews or analyzing search logs by identifying which
items are most popular based on the search and which products are missing
Many Internet service providers use MapReduce to analyze site records and
understand site visits, engagement, locations, mobile devices, and browsers.
4 — Fraud Detection
Hadoop and MapReduce are used in the financial industries, including
companies such as banks, insurance providers, payment locations for fraud
detection, trend identification or business metrics through transaction analysis.
Banks analyze the data of the credit card and the related expenses, for
categorization of these expenses and make recommendations for different
offers, analyzing anonymous purchasing behavior.
5 — Search and Advertisement Mechanisms
We can utilize it to analyze and understand search behavior, trends, and
missing results for specific keywords.
Google and Yahoo use MapReduce to understand users’ behavior, such as
popular searches over a period of an event such as presidential elections.
Google AdWords uses MapReduce to understand the impressions of ads
served, click-through rates, and engagement behavior of users.
6 — Data Warehouse
We can utilize MapReduce to analyze large data volumes in data
warehouses while implementing specific business logic for data insights.
7 - Geographical Data
Large data sets including road, intersection, and feature data ,
Problems that Google Maps has used MapReduce to solve
Locating roads connected to a given intersection
Rendering of map tiles
Finding nearest feature to a given address or location
Example
Input: List of roads and intersections
Map: Creates pairs of connected points (road,
intersection) or (road, road)
Sort: Sort by key
Reduce: Get list of pairs with same key
Output: List of all points that connect to a particular road
8 - PageRank
Program implemented by Google to rank any type of
recursive “documents” using MapReduce.
Initially developed at Stanford University by Google
founders, Larry Page and Sergey Brin, in 1995.
Led to a functional prototype named Google in 1998.
Still provides the basis for all of Google's web search tools.
Simulates a “random-surfer”
Begins with pair (URL, list-of-URLs)
Maps to (URL, (PR, list-of-URLs))
Maps again taking above data, and for each u in list-of-
URLs returns (u, PR/|list-of-URLs|), as well as (u, new-list-of-
URLs)
Reduce receives (URL, list-of-URLs), and many (URL,
value) pairs and calculates (URL, (new-PR, list-of-URLs))
PageRank: Problems
Has some bugs – Google Jacking
Favors Older websites
Easy to manipulate
