Introduction to Big Data
Instructor: Li Yang
What’s big data?
• Term for data sets so large and complex that traditional data processing
and storage techniques fail
• Large Volum: (M,G,TB,PB)
• Large Variety: (online data: web, photo, video, social; offline data: sensor
data,…)
• Varied Velocity: (periodic, realtime)
• Veracity: (quality of data: usually poor for big data)
Why big data?
• High tech results in the need for restoring and processing huge amounts of data
• Web and Super (cloud) computing
• Traditional data processing techniques (RDBMSs) fail:
• Fit for numeric, well-structured, clean (no missing) data
• Scaling requires for high costs (expensive hardware)
• Fault tolerance (ability to rescue the hardware failure) is also expensive
• Traditional data processing techniques can’t scale to fit for big data without massive code
development
Evolution of big data techniques
• Hadoop
• HDFS
• Map Reduce
• Spark: designed to run on top of Hadoop (upgrade)
• User-friendly
• Efficient: 100 times faster in memory and 10 times faster running in disk than MapReduce
• Combines SQL, Streaming, and other complicated analytics
• Runs ‘everywhere’ (not only on Hadoop but also Mesos, …)
Big data analytics: comparison of Hadoop MapReduce and Apache Spark, 2016
Big Data Examples
• Walmart (more details later)
• Data mining: discover consumer’s purchase pattern
• Hadoop and NoSQL technique
• Uber
• Machine learning: predict the demand everywhere and set the local price
• Netflix (more details later)
• Machine learning: cater each consumer’s preference (recommendation engine)
• Hadoop, SQL, Cassandra: online on-demand video streaming data
• eBay
• requirement: rapidly data analysis for streaming data and quick action on it
• Apache Spark, Storm, Kafka
• Procter&Gamble
• marketing, product development, supply chain
• Hadoop
Example of Big Data: Walmart
How Big Data Analysis helped increase Walmarts Sales turnover?
• Walmart is an American multinational retail corporation that operates a chain of hypermarkets,
discount department stores, and grocery stores from the united states, headquartered in
Bentonville, Arkansas. (by Wikipedia)
• Walmart ranks ? in Fortune 500 in 2021.
Walmart had a banner 2020, with
U.S. e-commerce sales up 79%
as pandemic-weary customers
consolidated shopping trips to
fewer retailers and took
advantage of the big-box giant’s
strong curbside pickup offering.
Its Sam’s Club and international
businesses also boomed for
similar reasons.
Example of Big Data: Walmart
How Big Data Analysis helped increase Walmarts Sales turnover?
• Walmart is an American multinational retail corporation that operates a chain of hypermarkets,
discount department stores, and grocery stores from the united states, headquartered in
Bentonville, Arkansas. (by Wikipedia)
• Walmart ranks ? # 1 in Fortune 500 in 2021.
Walmart had a banner 2020, with
U.S. e-commerce sales up 79%
as pandemic-weary customers
consolidated shopping trips to
fewer retailers and took
advantage of the big-box giant’s
strong curbside pickup offering.
Its Sam’s Club and international
businesses also boomed for
similar reasons.
Walmart Data Source 1: consumers
Walmart tracks and targets
every consumer individually
• Walmart gathers information on what
customer’s buy, where they live and what are
the products they like through in-store Wi-Fi
• Walmart collects every clickable action on
Walmart.com-what consumers buy in-store
and online
• Walmart also pay attention to the local news,
trending on social network, even local
weather.
.
Walmart Data Source 2: employees and itself
Walmart tracks every
employe
• Walmart collects the online retailers’
informatio
• Walmart gathers the employees’ information
to optimize its own organization and improve
ef ciency
fi
n
Example of Big Data: Walmart
• Summary: American multinational retail giant Walmart collects 2.5
petabytes of unstructured data from 1 million customers every hour
Usage of Big Data by Walmart
• Launching new products
• Design the most popular product to catch the trend (Christmas products)
• Better Predictive Analysis
• Demand
• Pricing
• Logistics
• Customized Recommendations
• Designed coupon
• Designed advertising
Big Data Analytic Solutions
• Social Media Big Data Solutions
• Social Media Data is unstructured, informal and generally ungrammatical
• Big part of Walmart’s data driven decision are based on social media data: (Facebook comments, Pinterest pins,
Twitter Tweets, LinkedIn shares …)
1. Social Genome: developed by WalmartLabs; social network data; better analyze the context of their users
2. Shopycat-gift recommendation engine at Walmart: app developed by Walmart; help consumers to buy ideal
gift for their friends during the holiday rush; also give detail reference information for the recommendations
3. Inventory management at Walmart: help managers to optimize the storage for the products; how many cashiers
and self-checkout should be open?
• Mobile Big Data Solutions
• More than half of the Walmart’s customers use Smartphones
• Walmart’s mobile application: a shopping list that can tell customers the position of their wants and helps them by
providing discounts; geofencing feature of Walmart’s mobile app senses whenever a user enters the Walmart store in US.
• https://www.forbes.com/sites/bernardmarr/2017/08/29/how-walmart-is-using-machine-learning-ai-iot-and-
big-data-to-boost-retail-performance/?sh=68bd71496cb1
Example of Big Data: Netflix
Net ix Recommender System — A Big Data Case Study, Kasula, 2020
• Netflix an American over-the-top content platform and production company headquartered in Los
Gatos, CA. The company's primary business is a subscription-based streaming service offering
online streaming from a library of films and television series, including those produced in-house.
(by Wikipedia)
• Their main source of income comes from users’ subscription fees. They allow users to stream data
from a wide range of their movies and TV shows at any time on a variety of internet-connected
services
• The primary asset of Netflix is their technology. Especially their recommendation system. The
study of the recommendation system is a branch of information filtering systems (Recommender
system, 2020).
• Most of the recommender systems study users by using their history. Recommender systems
have two primary approaches. They are collaborative filtering and content-filtering.
fl
Big Data Source: Netflix
• Internal source of data:
• Billion ratings from its members
• Stream related data such as the duration, time of playing, type of the device, day of the week and other
context-related information.
• The pattern and the titles that their subscribers add to their queues
• All the metadata related to a title in their catalog such as director, actor, genre, rating and reviews from different
platforms.
• The search-related text information by Netflix subscribers
• External source of data:
• box office information, performance and critic reviews
• demographics, culture, language, and other temporal data
Big Data Example: Netflix
• What does Netflix want from Big Data?
• Recommend the `next content’ to its user
• What is the `next content’ for each consumer?
• What are the big-data challenges for Netflix?
• volume: approximately 105TB of data with respect to videos alone; 10,000 GB of rating data alone
• velocity: collect data about the time of the data, the types of devices you watch content on, the duration of your watch
• Veracity: bias, noise, and abnormalities in data; Not all movies were rated equally by an individual
• Variety: most of the data in a structured format such as time of the day, duration of watch, popularity, social data, search-related
information, stream related data, etc. However, Netflix could also be using unstructured data. For example, thumbnail pictures that
it uses for personalization.
Data Ecosystem: Netflix
Big Data Example: Netflix
• What are advanced techniques used for Big Data?
• Data Storage and preprocessing
• Hadoop, Cassandra, S3
• Machine learning
• Supervised learning: classification, regression
• Unsupervised learning: clustering, compression, dimension reduction
• Other techniques
• Matrix factorization
• Singular valuation decomposition
• Probabilistic graphic model
• Ensemble method
Big Data Example: Netflix
• What are the results obtain from Big Data for Netflix?
• The overall engagement rate by the user with Netflix has increased with the help of the
recommender system. This led to lower cancellation rates and increased streaming hours volume
• Member satisfaction increased with the development and changes to the recommendation system.
• Personalization and recommendations save Netflix more than $1Billion per year.
• Examples:
• the winning algorithm was able to increase the predicting ratings and improved ‘Cinematch’
by 10.06% (Netflix Prize, 2020).
• According to (Netflix Technology Blog, 2017b), Singular Value Decomposition was able to
reduce the RMSE to 89.14% whereas Restricted Boltzmann Machines helped in reducing
RMSE to 89.90%
