Big Data
Sunnie S Chung
A Brief History
Relational database
management systems
Time
19751985
19851995
19952005
20052010
2020
Let us first see what a
relational database
system is
User/Application
Data Management
Query
Query
Query
Data
DataBase Management System (DBMS)
Example: At a Company
Query 1: Is there an employee named Nemo?
Query 2: What is Nemos salary?
Query 3: How many departments are there in the company?
Query 4: What is the name of Nemos department?
Query 5: How many employees are there in the
Accounts department?
Employee
Department
ID
Name
DeptID
Salary
ID
Name
10
Nemo
12
120K
12
IT
20
Dory
156
79K
34
Accounts
40
Gill
89
76K
89
HR
52
Ray
34
85K
156
Marketing
DataBase Management System (DBMS)
High-level
Query Q
Answer
DBMS
Data
Translates Q into
best execution plan
for current conditions,
runs plan
Example: Store that Sells Cars
Make
Model OwnerID
ID Name
Owners of
12
12 Nemo
Honda Accords Honda Accord
who are <=
Honda Accord
156
156 Dory
23 years old
Join ([Link] = [Link])
Filter (Make = Honda and
Model = Accord)
Cars
Age
22
21
Filter (Age <= 23)
Owners
Make
Model
OwnerID
ID
Name
Age
Honda
Accord
12
12
Nemo
22
Toyota
Camry
34
34
Ray
42
Mini
Cooper
89
89
Gill
36
Honda
Accord
156
156
Dory
21
DataBase Management System (DBMS)
High-level
Query Q
Answer
DBMS
Keeps data safe
and correct
despite failures,
concurrent
updates, online
processing, etc.
Data
Translates Q into
best execution plan
for current conditions,
runs plan
A Brief History
Relational database
management systems
Time
19751985
19851995
19952005
20052010
2020
Assumptions and
requirements changed
over time
Semi-structured and
unstructured data (Web)
Hardware developments
Developments in
system software
Changes in
data sizes
Big Data: How much data?
Google processes 20 PB a day (2008)
Wayback Machine has 3 PB + 100 TB/month (3/2009)
eBay has 6.5 PB of user data + 50 TB/day (5/2009)
Facebook has 36 PB of user data + 80-90 TB/day (6/2010)
CERNs LHC: 15 PB a year (any day now)
LSST: 6-10 PB a year (~2015)
640K ought to be
enough for
anybody.
From [Link]
From: [Link]
NEW REALITIES
The quest for knowledge used to
TBwith
disks
< $100
begin
grand
theories.
Everything is data
Now it begins with massive amounts
Rise of data-driven culture
of data.
Very publicly espoused
Welcome
to theWired,
Petabyte
by Google,
etc. Age.
Sloan Digital Sky Survey,
Terraserver, etc.
From: [Link]
FOX AUDIENCE
NETWORK
Greenplum parallel DB
42 Sun X4500s (Thumper) each
with:
48 500GB drives
16GB RAM
2 dual-core Opterons
Big and growing
200 TB data (mirrored)
Fact table of 1.5 trillion rows
Growing 5TB per day
4-7 Billion rows per day
From: [Link]
Also extensive use of R
and Hadoop
Yahoo! runs a 4000
node Hadoop cluster
(probably the largest).
Overall, there are
38,000 nodes running
Hadoop at Yahoo!
As reported by FAN, Feb, 2009
A SCENARIO FROM FAN
How many female WWF
fans under the age of 30
visited the Toyota
community over the last 4
days and saw a Class A ad?
How are these people
similar to those that
visited Nissan?
Open-ended question about
statistical densities
(distributions)
From: [Link]
MULTILINGUAL
DEVELOPMENT
SQL or MapReduce
Sequential code in a
variety of languages
Perl
Python
Java
R
Mix and Match!
From: [Link]
SE HABLA MAPREDUCE
SQL SPOKEN HERE
QUI SI PARLA PYTHON
HIER JAVA GESPROCKEN
R PARL ICI
From: [Link]
What is important to learn
Principles of query processing (35%)
Indexes
Query execution plans and operators
Query optimization
Data storage (15%)
Databases Vs. Filesystems (Google/Hadoop Distributed
FileSystem)
Data layouts (row-stores, column-stores, partitioning,
compression)
Scalable data processing (40%)
Parallel query plans and operators
Systems based on MapReduce
Scalable key-value stores
Processing rapid, high-speed data streams
Concurrency control and recovery (10%)
Consistency models for data (ACID, BASE, Serializability)
Write-ahead logging
