Marko Grobelnik marko.grobelnik@ijs.

si Jozef Stefan Institute Ljubljana, Slovenia

Stavanger, May 8th 2012

Introduction

Techniques Tools Applications Literature

What is Big data? Why Big-Data? When Big-Data is really a problem?

Big-data is similar to Small-data, but bigger but having data bigger consequently requires different approaches: to solve:
techniques, tools & architectures New problems and old problems in a better way.

From Understanding Big Data by IBM

Big-Data

Key enablers for the growth of Big Data are:

Increase of storage capacities Increase of processing power Availability of data

NoSQL

MapReduce Storage Servers

DatabasesMongoDB, CouchDB, Cassandra, Redis, BigTable, Hbase, Hypertable, Voldemort, Riak, ZooKeeper Hadoop, Hive, Pig, Cascading, Cascalog, mrjob, Caffeine, S4, MapR, Acunu, Flume, Kafka, Azkaban, Oozie, Greenplum S3, Hadoop Distributed File System EC2, Google App Engine, Elastic, Beanstalk, Heroku R, Yahoo! Pipes, Mechanical Turk, Solr/Lucene, ElasticSearch, Datameer, BigSheets, Tinkerpop

Processing

when the operations on data are complex:

e.g. simple counting is not a complex problem Modeling and reasoning with data of different kinds can get extremely complex

Good news about big-data:

Often, because of vast amount of data, modeling techniques can get simpler (e.g. smart counting can replace complex model based analytics) as long as we deal with the scale

Research areas (such as IR, KDD, ML, NLP, SemWeb, ) are subcubes within the data cube

Usage Quality Context Dynamicity Scalability

Good recommendations can make a big difference when keeping a user on a web site

the key is how rich context model a system is using to select information for a user Bad recommendations <1% users, good ones >5% users click

Contextual personalized recommendations generated in ~20ms

Domain Sub-domain Page URL URL sub-directories Page Meta Tags Page Title Page Content Named Entities Has Query Referrer Query

Referring Domain Referring URL Outgoing URL GeoIP Country GeoIP State GeoIP City Absolute Date Day of the Week Day period Hour of the day User Agent

Zip Code State Income Age Gender Country Job Title Job Industry

Trend Detection System

Log Files (~100M page clicks per day) Stream of clicks

User profiles

Stream of profiles

Trends and updated segments NYT articles

Segment Stock Market Health Green Energy Hybrid cars Travel Keywords Stock Market, mortgage, banking, investors, Wall Street, turmoil, New York Stock Exchange diabetes, heart disease, disease, heart, illness Hybrid cars, energy, power, model, carbonated, fuel, bulbs, Hybrid cars, vehicles, model, engines, diesel travel, wine, opening, tickets, hotel, sites, cars, search, restaurant

Sales
Segments
Campaign to sell segments

Advertisers

50Gb of uncompressed log files 10Gb of compressed log files 0.5Gb of processed log files 50-100M clicks 4-6M unique users 7000 unique pages with more then 100 hits Index size 2Gb Pre-processing & indexing time
~10min on workstation (4 cores & 32Gb) ~1hour on EC2 (2 cores & 16Gb)

Alarms Server

Telecom Network (~25 000 devices)

~10-100/sec

Alarms

Live feed of data

Alarms Explorer Server

Alarms Explorer Server implements three real-time scenarios on the alarms stream:

system is used in British Telecom

1. Root-Cause-Analysis finding which device is responsible for occasional flood of alarms 2. Short-Term Fault Prediction predict which device will fail in next 15mins 3. Long-Term Anomaly Detection detect unusual trends in the network

Operator

Big board display

Presented in Planetary-Scale Views on a Large Instant-Messaging Network by Jure Leskovec and Eric Horvitz WWW2008

Observe social and communication phenomena at a planetary scale Largest social network analyzed to date

Research questions: How does communication change with user demographics (age, sex, language, country)? How does geography affect communication? What is the structure of the communication network?
33

We collected the data for June 2006 Log size: 150Gb/day (compressed) Total: 1 month of communication data: 4.5Tb of compressed data Activity over June 2006 (30 days)
245 million users logged in 180 million users engaged in conversations 17,5 million new accounts activated More than 30 billion conversations More than 255 billion exchanged messages

34

35

36

Count the number of users logging in from particular location on the earth
37

Logins from Europe

38

Hops
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 20 6 degrees of separation [Milgram 60s] 21 Average distance between two random users is 6.622 23 90% of nodes can be reached in < 8 hops 24 25 19

Nodes
10 78 396 8648 3299252 28395849 79059497 52995778 10321008 1955007 518410 149945 44616 13740 4476 1542 536 167 71 29 16 10 3 2 3