Web Data Analysis
Department of Communication PhD Student Workshop
Web Mining for Communication Research
April 22-25, 2014
http://weblab.com.cityu.edu.hk/blog/project/workshops
Cheng-Jun Wang
Outline
I. Key features of web data
II. Major approaches to web data
analysis
i.
ii.
iii.
iv.
Network analysis
Temporal analysis
Spatial analysis
Sentiment analysis
III. Reflections on web data analysis
FEATURES OF WEB DATA
Traditional vs. Web Data
Analysis of traditional
(cross-sectional, fat)
data
ID
V1
V2
V3
...
V..
1
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
1,00
0
...
...
...
...
...
Analysis of web
(longitudinal, tall)
data
Time series analysis
ID
Network analysis1
Spatial analysis
2
Text mining
...
etc.
Time
V1
...
...
...
...
...
...
Multiple regression
1,000
...
...
Log-linear model
...
...
...
10,000
...
...
...
...
...
...
...
...
Multilevel analysis
Structural equation modeling
etc.
APPROACHES TO WEB
DATA ANALYSIS
5
What Can We Do with Web
Data?
Features
Temporal features
Spatial features
Structural/behavioral features (e.g., RT,
@)
Content features (term/topic/sentiment)
Approaches
Time series analysis
Spatial analysis
Network analysis
Text mining
6
Frequently Used Tools
Operation
Pull-down
menus
Programmingbased
Open Source Commercial
OpenOffice
Google Docs
Spreadsheet
SPSS
Excel
Stata
SAS
NETWORK ANALYSIS
(ELEMENTARY-LEVEL)
8
What Is a Network?
A network consists of
Nodes (actors, agents,
etc.)
Edges (relations, ties, etc.)
The same set of nodes
and edges can also be
called:
a graph
a matrix
a web
a map
etc.
A pair of adjacent nodes are
neighbors. (Are A and C neighbors?)
9
Key Concepts
Network
Node
Edge
Ego-network
Component
Triadic closure
Individual-level
analysis:
Centrality metrics
Group-level
analysis
Transitivity
Global-level
analysis
Density
Modularity
10
Examples of Nodes and
Edges
Nodes:
Persons (e.g., Facebook
users)
Organizations
(McDonald restaurants)
Nations (EU members)
Machines (web servers)
Locations (airports)
Ideas (words in articles)
etc.
Edges:
Kinship links (family
ties)
Friendship ties (factual
or perceived)
Business transactions
Travel routes (highways,
subways, air flights)
Similarities (word cooccurrences in articles)
etc.
11
Examples of Innovative
Network Analysis
Food Flavor Network
Music Notes Network
http://www.nature.com/srep/2011/111215/srep0
0196/full/srep00196.html
http://www.eie.polyu.edu.hk/~xfliu/publications/
LiuXF.2010.physa.Music.pdf
12
More on Edges
Directed (one-way) vs. undirected (two-way)
Observed (directly measured, e.g., hyperlinks)
vs. hidden (inferred , e.g., co-occurrences)
Formal (institutionally arranged, top-down) vs.
informal (self-organized, bottom-up)
Static (unchanged over time) vs. dynamic
(evolving)
Positive (e.g., friending) vs. negative (e.g., defriending)
The key challenge to innovative network analysis is
to identify hidden, informal, and evolving edges
13
Classification of Online Social
Networks
Manifestation
of Ties
Direction of Ties
Undirected
Directed
Directly
Observed
Friendship networks
(e.g., Facebook,
Google+)
Microblog networks
(e.g., Twitter, Sina
Weibo)
Indirectly
Inferred
Semantic networks
(e.g., recommendation
systems, social tagging
systems)
Newsgroups, blogs,
WWW hyperlink
networks
Source: Ackland and Zhu (forthcoming). Social network analysis, Sage.14
Components
A component is a
subset of a network:
i.
ii.
every node in the
subset has a path to
every other
the subset is not
part of some larger
set
Most online social
networks have one
(or a few) giant
components
15
Components in a High School
Network
Source: Bearman, Moody & Stovel (2004).
http://www.jstor.org/discover/10.1086/386272?uid=3738176&uid=2&uid=4&sid=
21103
878405327
16
Components in World Wide
Web
Daisy Model (Donato et al., 2005)
Bowtie Model (Broder et al., 2000)
SCC: strongly connected component
IN: unilaterally connected to SCC
OUT: unilaterally connected by SCC
Teapot Model (Zhu et al., 2008)
17
Ego-Centric Network
Ego-network: a subset of a network
including a particularly designated node
(ego) and its neighbors (alters)
For example, followers of a VIP account on
Twitter or Sina Weibo form an ego-network
All snowballing samples of online social
networks are ego-networks.
An important property of ego-networks is
the depth (see next slide).
Family tree is a special case of egonetworks (see the second next slide).
18
The Depth of Ego Networks
1.0 Ego Network
1.5 Ego Network
2.0 Ego Network
2.5 Ego Network
19
Family Tree: Special EgoNetworks
Is it an
undirected or
directed graph?
Are there
multiple paths
from a parent
node to a child
node?
What are the
similarities or
differences
between family
trees and other
types of ego20
networks?
Triadic Closure (Transitivity)
B
A
t0
t1
Why are friends (B and C) of a common friend (A) more likely to become
friends themselves: 1. chances to meet each other; 2. similarity between them.
21
Triads of Undirected
Networks
Closed Triad
Connected Pair
Open Triad
Unconnected
22
Triads of Directed Networks
The 1st number: N of bidirectional edges;
The 2nd number: N of
uni-directional edges;
The 3rd number: N of
nonexistent edges;
The letter code:
directed variations of
the same triad, with U
for up, D for down,
C for circle, and T for
transitive (i.e.,
having 2 paths that
lead to the same
endpoint).
23
Measure of Triadic Closure
Nodes in a graph
usually have multiple
triads each.
Therefore, there is a
need to measure
quantitatively the
overall degree of triadic
closure for each node.
Clustering coefficient
(CC) is the most
frequently used
measure for the
purpose.
B
A
C
D
24
Triadic Closure: Driven by
Social Selection or Social
Influence?
(b) Focal closure
(a) Triadic closure
Person
Person
Person
Person
Focus
(e.g.,
recommended
(c) Membership closure
books on
Amazon)
Person
Person
Focus
(e.g.,
recommended
groups on
Person
Facebook)
25
Goals of Social Network
Analysis
Perer & Shneiderman (2008):
1.Overall network metrics (e.g., number of nodes,
number of edges, density, diameter), global
2.Node rankings (degree, betweenness, closeness
centrality), individual
3.Edge rankings (weight, betweenness centrality),
local
4.Node rankings in pairs (degree vs. betweenness,
plotted on a scatter gram), local
5.Edge rankings in pairs, local
6.Cohesive subgroups (finding communities), local
7.Multiplexity (analyzing comparisons between
different edge types, such as friends vs. enemies),
cross-levels
26
Levels of Network Analysis
Individual-level: nodes, focusing on
who are the most
popular/important/influential nodes in
the network?
Local-level: groups (or clusters,
communities, components, etc.),
focusing on how are the nodes
clustered/grouped together?
Global-level: network, focusing on how
densely/closely is the network
connected as a whole?
27
Individual-level Analysis
Find popular/important/influential nodes
usually based on centrality metrics
Degree centrality: How many nodes are you
connected to?
Closeness centrality: How close are you to
other nodes?
Betweenness centrality: How many paths
are through you?
Eigenvalue: How many important nodes are
round you?
28
Interpretation of Centrality
Scores
High centrality scores:
Individuals with high
centrality scores are
often more likely to be:
Low centrality scores:
Individuals with low
scores are in
peripheral positions:
leaders
key conduits of
information
early adopters of
anything that spreads
in a network
who may be
protected from
negative contagion
and influence
who may be
associated with less
work overload in an
organization
29
Example: Krackhardts Kite
Graph
F
A
E
G
H
D
A network of 10
nodes and 18 edges:
Who has the highest
degree centrality?
Who has the highest
betweenness
centrality?
Who has the highest
closeness centrality?
30
Degree Centrality
Number of
neighbors a node is
directly connected
Indicates how well
the node is
connected within
the graph
Degree of G = 6
31
Betweenness Centrality
The number of
shortest paths
between pairs of
other nodes through a
node (as compared
with total number of
shortest paths in the
graph)
Indicates how critical
the node is to the flow
of information or
resource in the graph
Betweenness of H = 14
32
Closeness Centrality
Number of steps
along the shortest
path from the focal
node to all other
node
Indicates how
quickly information
travels between the
node and anyone
else in the graph
Closeness of D and E =
14, respectively
14 == 1*5 + 2*3 + 3*1
33
Eigenvalue Centrality
The extent to which a node
is a big fish connected with
other big fish in a big pond.
Calculated by assessing
how well connected a node
is to the parts of the
network with the greatest
connectivity.
Nodes with high
eigenvector scores have
many connections who
have many connections,
etc., similar to the logic of
Google PageRank.
Highly connected individuals
within highly interconnected
clusters, or big fish in big
ponds, have high eigenvector
centrality.
34
Group-level Analysis
Central Question: How are nodes
clustered (grouped) together?
based on clustering analysis, a method
to merge an n number of nodes into a g
number of groups such that:
the nodes within the same group are
maximally similar or homogeneous
the nodes between the groups are
maximally different or heterogeneous
35
Process of Clustering
Analysis
1
At step 1, there are 10 clusters,
each with a node that is uniquely
different from all others.
At step 2, nodes 1 and 2 are
considered to be similar enough
to form a cluster; same goes
between nodes 9 and 10. There
are now 8 clusters.
At step 3, node 3 joins the
cluster of 1 and 2, and node 8
joins the cluster of 9 and 10. The
process keeps on until every
node is included in the one giant
cluster at step 6.
An optimal solution is to keep a
small number of clusters with
maximal similarity within and
maximal difference between.
2
3
4
5
6
7
8
9
10
1
36
Island Method for Group
Detection
By raising the
threshold of edge
strength (e.g., mean,
median, or k
standard deviation
above the mean), an
increasing number of
groups
(communities) will
emerge successively
from a giant
connected
component.
37
Group-level Metrics in
NodeXL
Vertex counts
Edge counts
Geodesic distances
Group density
Number of edges between each pair
of groups
38
Global-level Analysis
Key question: How
densely or closely
connected is the
network as a whole?
Fig a (top): connected
based on 67%
agreement
Fig b (bottom):
connected based on
75% agreement
39
Global-level Metrics in NodeXL
(1)
Graph Type
Vertices
Unique Edges
Edges With Duplicates
Total Edges
Directed or undirected.
The number of vertices in the graph.
The number of edges that do not have duplicates.
The number of edges that have duplicates.
The number of edges in the graph. This is the sum of Unique
Edges and Edges With Duplicates.
Self-Loops
The number of edges that connect a vertex to itself.
Reciprocated Vertex Pair In a directed graph, this is the N of vertex pairs that have edges
Ratio
in both directions divided by the N of vertex pairs that are
connected by any edge. Duplicate edges and self-loops are
ignored. In an undirected graph, this is undefined.
Reciprocated Edge Ratio In a directed graph, this is the number of edges that are
reciprocated divided by the total number of edges. Duplicate
edges and self-loops are ignored. In an undirected graph, this is
undefined and is not calculated.
Connected Components The number of connected components in the graph. A
connected component is a set of vertices that are connected to
each other but not to the rest of the graph.
40
Global-level Metrics in NodeXL
(2)
Single-Vertex Connected
Components
Maximum Vertices in a
Connected Component
Maximum Edges in a
Connected Component
Maximum Geodesic
Distance (Diameter)
The N of connected components that have only one vertex.
Average Geodesic
Distance
The average geodesic distance among all vertex pairs, where
geodesic distance is the distance between two vertices along the
shortest path between them.
Graph Density
A ratio that compares the N of edges with the maximum N of
edges the graph would have if all the vertices were connected to
each other. Duplicate edges and self-loops are ignored.
Modularity
When the graph has groups, this is a measure of the "quality" of
the grouping. Graphs with high modularity have dense
connections among the vertices within the group but sparse
connections among vertices in different groups. When the graph
does not have groups, this is undefined.
The N of vertices in the connected component that has the most
vertices.
The N of edges in the connected component that has the most
edges.
The maximum geodesic distance among all vertex pairs, where
geodesic distance is the shortest path between two vertices.
41
HANDS-ON TUTORIALS
42
use R! aRe you
suRe?
NodeXL
and
Super R logo
Source: www.redbubble.com/
43
R for Web Data Analysis
Networ
k
Analysi
s
R packages
igraph, Statnet,
Rsiena
Spatial
Analysis
http://cran.rproject.org/web/views/Spat
ial.html
Sp, Spatial,
OpenStreetMap,
RgoogleMaps
Temporal
Analysis
http://cran.rtseries, forecast, urca,
project.org/web/views/Time wavelets, SpatioTemporal
Series.html
http://cran.rproject.org/web/views/Spat
ioTemporal.html
Text Mining
http://cran.rproject.org/web/views/Natu
ralLanguageProcessing.ht
ml
tm, Rweka, openNLP,
wordcloud, topicmodels,
RTextTools, sentiment,
ReadMe
Machine
http://cran.r-
Nnet, rpart, trees, party,
44
came, I saw,
and I
walked away?
Picture: Gareth Jenkins/Solent
http://www.telegraph.co.uk/news/picturegalleries/picturesoftheday/8561204/Pictures-of-the-day-7-June-2011.html?image=6
Figure from the movie Daddy Day Care (2003)
http://img0.joyreactor.com/pics/post/gif-eddie-murphy-reaction-gifs-party-394848.gif
Plunge
into the water!
45
HANDS-ON!
46
Demo 1. Software
Installation
Download and install R, Rstudio, and
NodeXL
http://cran.r-project.org/
https://www.rstudio.com/ide/
http://nodexl.codeplex.com/
Learn the basics of R
http://tryr.codeschool.com/
More information
https://www.rstudio.com/training/online.html
47
NETWORK ANALYSIS
(ADVANCED-LEVEL)
48
Network Topology
49
Regular or random?
Regular network
Nodes are connected in
a regular neighborhood
with a fixed number k
of edges per each node
They do not exhibit the
small world
characteristics
They may exhibit
clustering
Random network
Random networks
have randomly
connected edges
each node has an
average edges
They exhibit the small
world characteristics
They do not exhibit
clustering
50
Small-World Networks
Between order and chaos
Network generation
Watts and Strogatz
(1999) propose a
model for networks
between order and
chaos
The model is built by simply
Re-wiring at random a small
percentage of the regular edges
Which dramatically shortens the
average path length without
destroying clustering
Such that
The network exhibits the
small world feature as
random networks
And exhibits clustering, as
regular lattices
Watts and Strogatz (1999)
51
Scale-free network
Power law
Long-tail distribution
P(k) ~ k-a, 0<a<2
log(P) ~-a*log(k)
Zipf distribution
Pareto distribution
Properties
Scale-invariance
P(c*k) ~ (c*k) a
Thus, P(c*k) ~ c a k-a
P(c*k) k-a
No average
Universality
Barabsi, Albert, and Jeong, Scale-free characteristics of random networks: The topology of the world wide web, Physical A.,
281, 2000, pp.69-77.
52
Demo2. Generate the
Network
R scripthttp://chengjun.github.io/web_data_analysis/demo2_simulate_networks/
install.packages("igraph")
library(igraph)
size = 50
g = graph.tree(size, children = 2); plot(g)
g = graph.star(size); plot(g)
g = graph.full(size); plot(g)
g = graph.ring(size); plot(g)
g = connect.neighborhood(graph.ring(size), 2); plot(g)
g = erdos.renyi.game(size, 0.1)
# small-world network
g = rewire.edges(erdos.renyi.game(size, 0.1), prob = 0.8 ); plot(g)
# scale-free network
g = barabasi.game(size) ; plot(g)
53
The Political Blogosphere VS.
Congressmens Retweet
Network
Peng, Zhu, Liu, Wu, Liu (2014)
L. A. Adamic and N. Glance, 'The
Political Blogosphere and the 2004
U.S. Election: Divided They Blog',
LinkKDD 2005
Friendship, Interaction
networks and Vote agreement
of congressmen in the United
States. 7th APNC, Montreal,
Canada
54
How to Represent a
Network?
A
e1
B
e3
e
2
C
e
4
e6 e5
E
A,
A,
A,
C,
C,
C,
B
D
C
D
E
F
55
Demo 3. Describe the
Network
Compute graph metrics using NodeXL
Step 1 paste the
edgelist here
56
Demo 3. Describe the
Network
NodeXL: Set node attributes
Step 2 paste the
node attribute here,
and name it as
party
Remember to click here to shift to
the Vertices window
57
Network
NodeXL: Calculating graph
metrics
Step 3 Click graph
metrics here
Remember to set the graph as
directed here
58
Network
NodeXL: Set vertex color and
vertex size
Step 4 Set vertex color
and vertex size by
click here
Remember to set the party as a
categorical variable
59
Demo 3. Describe the
Network
R script
http://chengjun.github.io/web_data_analysis/demo3_describe_the_network/
Graph Statistics
Centrality Measures
Algorithms of graphs
Shortest path
Connected component algorithms
60
The exponential random
graph model (p*)
An ERGM (p*) model is a statistical model for the
ties in a network
Independent (pairs of) ties (p1, Holland and
Leinhardt, 1981; Fienberg and Wasserman, 1979,
1981)
Markov graphs (Frank and Strauss, 1986)
Extensions (Pattison & Wasserman, 1999; Robins,
Pattison & Wasserman, 1999; Wasserman &
Pattison, 1996)
New specifications (Snijders et al., 2006; Hunter
& Handcock, 2006)
61
Why do we use stochastic
network models?
To capture complex social phenomena that
caused by regularities and randomness.
To infer whether certain network signatures will
appear more often than by chance
To distinguish between different social processes
(e.g. homophily vs. structural balance)
To better understand the way local social
processes interact and combine to shape global
network patterns
Deterministic approaches are not always good
enough
62
Procedures of ERGM
Assume we have an observed network of size n. What are the
mechanisms driving the formation of our network (e.g. reciprocity,
transitivity)?
Given those mechanisms, are some network configurations (e.g.
mutual dyads, transitive triplets) more common than you would
expect by chance?
Include a parameter for each configuration in the model. Parameter
values will help us identify a probability distribution for all graphs of
size n. (e.g. if we have a high value for the reciprocity parameter,
graphs that have a lot of mutual dyads will be more probable than
ones that do not)
Estimate the parameters: find the parameter values that best match
the observed network. We do that using MCMC-MLE: Markov Chain
Monte Carlo Maximum Likelihood Estimation techniques.
Once we have our probability distribution, we can draw random
graphs from it and compare any of their characteristics to those of our
observed
network.
http://www.kateto.net/wordpress/wp-content/uploads/2012/12/COMM%20645%20-%20ERGM.pdf
63
Network Configurations:
Undirected Networks
4-star
Edge
K-star
2-star
:
:
Triangle
3-star
64
Network Configurations:
Directed Networks
Arc
Reciprocity
isolate
2-mixed star
2-in star
2-out star
K-in star
Transitive triad
:
:
K-out star
:
:
Cyclic triad
65
Exponential Random Graph Models
ERGM
Y: all the possible ties
y: the observed ties
X: node attributes
g(y,X): network configurations(a vector) .
: a vector of model parameters
k(): normalizing constant
66
One example
denotes the vector of change statistics
Johan Koskinen (2012) An introduction to ERGM. 8th UKSNA Conference, Bristol
67
Tie-Network configuration
matrix
edges
2-star
K-star
Triangle
Y1,2
Y1,3
Y2, 3
Yn, n-1
68
Online Collective Identity
Ackland (2011) Online collective identity. SN
69
Demo 4. ERGM with R
R script
Load data
Build up network objects
Set the node attributes
Plot the network
Fitting a basic ERG model
http://chengjun.github.io/web_data_analysis/demo4_ergm_analysis/
70
TEMPORAL ANALYSIS
71
Time Series Analysis
Time series data can
be analyzed within
either time domain
or frequency domain.
Time domain:
ARIMA/VAR analysis
Survival analysis
Multilevel analysis
Frequency domain:
Where time domain analysis is
routinely conducted, frequency domain
analysis rarely adopted.
Fourier
transformation
Spectrum analysis
(comparing ak and bk
of different time
series).
72
Time Series Analysis
Forecasting and Univariate Modeling
Frequency analysis
Decomposition and Filtering
Seasonality
Stationarity, Unit Roots, and Cointegration
Nonlinear Time Series Analysis
Dynamic Regression Models
Multivariate Time Series Models
73
Survival Analysis of
Blogging Behavior
Source: Zhu et al., ICA 2010
74
SPATIAL ANALYSIS
75
Spatial Analysis
Spatial Data:
Location names
IP addresses
Map visits
GPS usage
etc.
Well-developed for
offline data but underdeveloped/utilized for
web data beyond visual
inspections.
Spatial Analysis:
Spatial clusters/patterns
(by visual inspections)
Spatial autocorrelation
Spatial Regression
Spatial Dependence
(correlation between
nearby locations)
Spatial interaction
(correlation between
geo-coded variables)
76
Geospatial Distribution of the
Communication on Twitter
Conover MD, Davis C, Ferrara E, McKelvey K, Menczer F, et al. (2013) The
Geospatial Characteristics of a Social Movement Communication Network.
PLoS ONE 8(3): e55957. doi:10.1371/journal.pone.0055957
77
Spatial Distribution of Tweets in
Milan
kernel smoother of point density
Are the tweets randomly distributed?
78
Temporal Distribution of Tweets
in Milan
79
SENTIMENT ANALYSIS
80
Sentiment Analysis
Decompose sentiment
Emotion
Joy
surprise
Anger
Sadness
Fear
disgust
Polarity
Positivity
Negativity
Neutral
Lexicon method
Carlo Strapparava and
Alessandro Valituttis
emotions lexicon
Janyce Wiebes
subjectivity lexicon
Liu Bings polarity
lexicon
Supervised machine
learning
Combine lexicon and
machine learning
81
Sentiment in the Tweet Stream
Miller (2011) Social scientists wade into the tweet
stream. Science
82
Twitter Mood Predicts the Stock
Market?
Decompose
sentiment
Emotion
Calm
Alert
Sure
Vital
Kind
Happy
Bollen (2011)
Twitter mood predicts
the stock market. JOCS
83
Calm Sentiment Predicts the
Stock Market
84
Demo 5. Sentiment Analysis
with Supervised Machine
Learning
R script
http://chengjun.github.io/web_data_analysis/demo5_sentiment_analysis/
Figure source: http://courtneylambert.co/official-twitter-stats-from-chirp
85
REFLECTION ON WEB
DATA ANALYSIS
86
Google Correlate & Google Flu
Prediction
http://www.google.com/trends/correlate/comic?p=2
87
Nature reported that Google flu trends (GFT) was predicting more than
double the proportion of doctor visits for influenza-like illness (ILI) than the
Centers for Disease Control and Prevention (CDC), which bases its
estimates on surveillance reports from laboratories across the United States
(1, 2).
Lazer et al. (2014) The parable of Google Flu Traps in big data analysis. Science
88
Tweet Sentiment and U. S.
Election 2012
Figures source: election.twitter.com
89
Facebook Insight and Twitter
Mention
Result
Facebook Insight
http://www.cnn.com/election/2012/facebook-insights/
http://www.zerogeography.net/2012/11/obama-wins-election-ontwitter.html
http://www.huffingtonpost.com/simon-jackman/pollster-predictions_b_2081013.html
90
Predict Political Orientation
with Machine Learning
Colleoni et al (2014) Echo chamber or public sphere Predicting political
orientation and measuring political homophily in Twitter. JOC
91
To Move on
R Style Guide
R bloggers
stackoverflow
github
http://adv-r.had.co.nz/Style.html
92
Yet, It Is Not Finished
Creating an R package (intro) http://gastonsanchez.com/teaching/
93
