Data Warehousingand Data Mining
Data Warehousingand Data Mining
Magister Komputer
Data Warehousing
and
OLAP Technology
Oleh :
Nama : Sunaryo Tandi
N I M : (0801050005)
November 5, 2008 1
Data Mining:
Concepts and Techniques
— Chapter 3 —
Jiawei Han
Department of Computer Science
University of Illinois at Urbana-Champaign
www.cs.uiuc.edu/~hanj
©2006 Jiawei Han and Micheline Kamber, All rights reserved
November 5, 2008 2
Chapter 3: Data Warehousing and OLAP Technology: An Overview
November 5, 2008 3
A producer wants to know….
Which are our
lowest/highest margin
customers ?
Who are my customers
What is the most and what products
effective distribution are they buying?
channel?
November 5, 2008 6
Data Warehouse—Integrated
November 5, 2008 7
Data Warehouse—Time Variant
November 5, 2008 8
Data Warehouse—Nonvolatile
November 5, 2008 9
Data Warehouse vs. Heterogeneous DBMS
November 5, 2008 12
OLTP vs. OLAP
OLTP OLAP
users clerk, IT professional knowledge worker
function day to day operations decision support
DB design application-oriented subject-oriented
data current, up-to-date historical,
detailed, flat relational summarized, multidimensional
isolated integrated, consolidated
usage repetitive ad-hoc
access read/write lots of scans
index/hash on prim. key
unit of work short, simple transaction complex query
# records accessed tens millions
#users thousands hundreds
DB size 100MB-GB 100GB-TB
metric transaction throughput query throughput, response
November 5, 2008 13
Application-Orientation vs. Subject-Orientation
Application-Orientation Subject-Orientation
Operational Data
Database Warehouse
Credit
Loans Customer
Card
Vendor
Product
Trust
Savings Activity
November 5, 2008 14
Why Separate Data Warehouse?
High performance for both systems
DBMS— tuned for OLTP: access methods, indexing, concurrency
control, recovery
Warehouse—tuned for OLAP: complex OLAP queries,
multidimensional view, consolidation
Different functions and different data:
missing data: Decision support requires historical data which
operational DBs do not typically maintain
data consolidation: DS requires consolidation (aggregation,
summarization) of data from heterogeneous sources
data quality: different sources typically use inconsistent data
representations, codes and formats which have to be reconciled
Note: There are more and more systems which perform OLAP
analysis directly on relational databases
November 5, 2008 15
To summarize ...
November 5, 2008 16
Chapter 3: Data Warehousing and OLAP
Technology: An Overview
November 5, 2008 17
From Tables and Spreadsheets to Data Cubes
time,location,supplier
3-D cuboids
time,item,location
time,item,supplier item,location,supplier
4-D(base) cuboid
time, item, location, supplier
November 5, 2008 19
Conceptual Modeling of Data Warehouses
November 5, 2008 20
Example of Star Schema
time
time_key item
day item_key
day_of_the_week Sales Fact Table item_name
month brand
quarter time_key type
year supplier_type
item_key
branch_key
branch location
location_key
branch_key location_key
branch_name units_sold street
branch_type city
dollars_sold state_or_province
country
avg_sales
Measures
November 5, 2008 21
Example of Snowflake Schema
time
time_key item
day item_key supplier
day_of_the_week Sales Fact Table item_name supplier_key
month brand supplier_type
quarter time_key type
year item_key supplier_key
branch_key
location
branch location_key
location_key
branch_key
units_sold street
branch_name
city_key
branch_type
dollars_sold city
city_key
avg_sales city
state_or_province
Measures country
November 5, 2008 22
Example of Fact Constellation
time
time_key item Shipping Fact Table
day item_key
day_of_the_week Sales Fact Table item_name time_key
month brand
quarter time_key type item_key
year supplier_type shipper_key
item_key
branch_key from_location
November 5, 2008 24
Defining Star Schema in DMQL
November 5, 2008 26
Defining Fact Constellation in DMQL
all all
November 5, 2008 29
View of Warehouses and Hierarchies
Specification of hierarchies
Schema hierarchy
day < {month < quarter;
week} < year
Set_grouping hierarchy
{1..10} < inexpensive
November 5, 2008 30
Multidimensional Data
Office Day
Month
November 5, 2008 31
A Sample Data Cube
Total annual sales
Date of TV in U.S.A.
1Qtr 2Qtr 3Qtr 4Qtr sum
t
uc
TV
od
PC U.S.A
Pr
VCR
Country
sum
Canada
Mexico
sum
November 5, 2008 32
Cuboids Corresponding to the Cube
all
0-D(apex) cuboid
product date country
1-D cuboids
3-D(base) cuboid
product, date, country
November 5, 2008 33
Browsing a Data Cube
Visualization
OLAP capabilities
Interactive manipulation
November 5, 2008 34
Typical OLAP Operations
Roll up (drill-up): summarize data
by climbing up hierarchy or by dimension reduction
Drill down (roll down): reverse of roll-up
from higher level summary to lower level summary or detailed
data, or introducing new dimensions
Slice and dice: project and select
Pivot (rotate):
reorient the cube, visualization, 3D to series of 2D planes
Other operations
drill across: involving (across) more than one fact table
drill through: through the bottom level of the cube to its back-end
relational tables (using SQL)
November 5, 2008 35
Fig. 3.10 Typical OLAP
Operations
November 5, 2008 36
A Star-Net Query Model
Customer Orders
Shipping Method
Customer
CONTRACTS
AIR-EXPRESS
ORDER
TRUCK
PRODUCT LINE
Time Product
ANNUALY QTRLY DAILY PRODUCT ITEM PRODUCT GROUP
CITY
SALES PERSON
COUNTRY
DISTRICT
REGION
DIVISION
Location Each circle is
called a footprint Promotion Organization
November 5, 2008 37
Chapter 3: Data Warehousing and OLAP
Technology: An Overview
November 5, 2008 38
Design of Data Warehouse: A Business
Analysis Framework
Monitor
& OLAP Server
Other Metadata
sources Integrator
Analysis
Operational Extract Query
DBs Transform Data Serve Reports
Load
Refresh
Warehouse Data mining
Data Marts
November 5, 2008 42
Data Warehouse Development: A
Recommended Approach
Multi-Tier Data
Warehouse
Distributed
Data Marts
Enterprise
Data Data
Data
Mart Mart
Warehouse
Data Sources
Data Marts
Data Warehouse
November 5, 2008 44
Problems with Data Mart Centric Solution
November 5, 2008 45
True Warehouse
Data Sources
Data Warehouse
Data Marts
November 5, 2008 46
Data Warehouse Back-End Tools and Utilities
Data extraction
get data from multiple, heterogeneous, and external sources
Data cleaning
detect errors in the data and rectify them when possible
Data transformation
convert data from legacy or host format to warehouse format
Load
sort, summarize, consolidate, compute views, check integrity,
and build indicies and partitions
Refresh
propagate the updates from the data sources to the
warehouse
November 5, 2008 47
Metadata Repository
Meta data is the data defining warehouse objects. It stores:
Description of the structure of the data warehouse
schema, view, dimensions, hierarchies, derived data defn, data
mart locations and contents
Operational meta-data
data lineage (history of migrated data and transformation path),
currency of data (active, archived, or purged), monitoring
information (warehouse usage statistics, error reports, audit trails)
The algorithms used for summarization
The mapping from operational environment to the data warehouse
Data related to system performance
warehouse schema, view and derived data definitions
Business data
business terms and definitions,48 ownership of data, charging
November 5, 2008
OLAP Server Architectures
Relational OLAP (ROLAP)
Use relational or extended-relational DBMS to store and manage
warehouse data and OLAP middle ware
Include optimization of DBMS backend, implementation of
aggregation navigation logic, and additional tools and services
Greater scalability
Multidimensional OLAP (MOLAP)
Sparse array-based multidimensional storage engine
Fast indexing to pre-computed summarized data
Hybrid OLAP (HOLAP) (e.g., Microsoft SQLServer)
Flexibility, e.g., low level: relational, high-level: array
Specialized SQL servers (e.g., Redbricks)
Specialized support for SQL queries over star/snowflake schemas
November 5, 2008 49
Chapter 3: Data Warehousing and OLAP
Technology: An Overview
November 5, 2008 50
Efficient Data Cube Computation
November 5, 2008 51
Cube Operation
Cube definition and computation in DMQL
define cube sales[item, city, year]: sum(sales_in_dollars)
compute cube sales
Transform it into a SQL-like language (with a new operator
cube by, introduced by Gray et al.’96) ()
SELECT item, city, year, SUM (amount)
FROM SALES (city) (item) (year)
November 5, 2008 53
Indexing OLAP Data: Bitmap Index
Index on a particular column
Each value in the column has a bit vector: bit-op is fast
The length of the bit vector: # of records in the base table
The i-th bit is set if the i-th row of the base table has the value for
the indexed column
not suitable for high cardinality domains
November 5, 2008 57
Data Warehouse Usage
Layer2
MDDB
MDDB
Meta
Data
Filtering&Integration Database API Filtering
Layer1
Data cleaning Data
Databases Data
Data integration Warehouse Repository
November 5, 2008 60
Chapter 3: Data Warehousing and OLAP
Technology: An Overview
Summary
November 5, 2008 61
Summary: Data Warehouse and OLAP Technology
Thank You
November 5, 2008 65