Theory of Integration in Statistics
Mohammed Nasser
Department of Statistics
Courtesy:
An unknown writer
Relation between ML and Intgration
Statistical
Concepts/Techniques
Concepts in Integration Theory
Probability measures
An Integral of some non-negative
function w.r.t. a particular
measure
Probability of an event
Evaluation of the integral over that
event
Mean, Moments, Variance, - --
Integration of a rv/ positive power
of a rv,a deviated rv w.r.t.
appropriate probability measure
Lp space
Relation between ML and Integration
Statistical
Concepts/Techniques
Concepts in Integration Theory
Covariance, Correlation
Integration of product of two
deviated rvs/ two standardized rvs
w.r.t. appropriate probability
measure
Distributional Convergence
In Stochastic Process --
Integral seen as a linear functional on
the set of Probability measures
Pn P
f (w)dPn f (w)dP f C (),
<,d is a metric space
MEASURE THEORY HELPS STATISTICS IN TWO WAYS
Real Analysis And
Complex Analysis
Probability Theory
(Convergence, Limit, Cont., Integration and
Diff.)
Theory of Statistics (Sampling
Distribution and Inference)
.
Analysis and Statistics
What is Integration?
f,a function
S
if ix
The product as well as the series should be defined
The series should be convergent
What is Integration?
f,a function
R2
R/R2
Integration is nothng but infinite sum of infinitesimal quantities
What is Integration?
f,a function
Rn
Integration is nothng but infinite sum of infinitesimal quantities
Whenever we want to measure overall characteristic of
something complicated we need some type of integration.
What is Integration?
f,a function
, any set
Integration =Integrate+ion
How long will the patient survive?
What percent of people earn more than 50,000 Tk
permonth?
What is Integration?
f,a function
B,
B-space
Bochner Integral
Lebesgue versus Riemann
Folland (1984) summarized the difference between the
Riemann and Lebesgue approaches thus:
"to compute the Riemann integral of f, one partitions the domain
[a, b] into subintervals", while in the Lebesgue integral, "one is
in effect partitioning the range of f".
Lebesgue versus Riemann
A little perspective:
Riemann did this...
...
Lebesgue versus Riemann
A little perspective:
Lebesgue did this...
Lebesgue versus Riemann
A little perspective:
Lebesgue did this...
Why is this better????
Lebesgue versus Riemann
Consider the function:
1 at irrationals
f
0 at rationals
Height
If we follow Lebesgues reasoning,
then the integral of this function over
the set [0,1] should be:
Width of Irrationals
(Height) x (width of irrationals)
= 1 x (measure of irrationals)
If we can measure the size of superlevel sets, we can integrate a lot of function!
Lebesgue versus Riemann
Measure Theory begins from this simple motivation.
If you remember a couple of simple principles, measure and
integration theory becomes quite intuitive.
The first principle is this...
Integration is about functions. Measure theory is about sets.
The connection between functions and sets is super/sub-level
sets.
Lebesgue versus Riemann
Let f be a bounded function on [a,b]:
f is Riemann integrable on [a,b] if and only the set of
discontinuities of f has Lebesgue measure 0
If f is Riemann integrable on [a,b] , f is Lebesgue
integrable on [a,b] . Two are equal.
Lebesgue versus Riemann
In case of improper integrals
Their does not exist any clear-cut
relationship.
f may be not Lebesgue integrable despite its Riemann
integrability
The same is true if types of integration change their places
Caution:The type is very important.
For examples, see Lebesgue Measure and Intgration by Jain
and Gupta (1987), pp 165-167.
-Algebras??
Measure theory is about measuring the size of things.
In math we measure the size of sets.
How big is the set A?
Set A
-Algebras??
Size should have the following property:
The size of the union of disjoint sets should equal the
sum of the sizes of the individual sets.
Makes sense.... but there is a problem with this...
Consider the interval [0,1]. It is the disjoint union of all
real numbers between 0 and 1. Therefore, according to
above, the size of [0,1] should be the sum of the sizes of a
single real number.
If the size of a singleton is 0 then the size of [0,1] is 0.
If the size of a singleton is non-zero, then the size of [0,1] is
infinity!
This shows we have to be a bit more careful.
Sigma Algebras (-algebra)
Consider a set
Let F be a collection of subsets of
F is a -algebra if:
1) F
C
A
A
F
2)
3) A1 , A2 F A1 A2 F
4) A1 , A2 , A3 ,... F
A F
i
i 1
An algebra of sets is
closed under finite set
operations. A -algebra
is closed under
countable set operations.
In mathematics, often
refers to countable.
General Measurable Functions I
f: 1
1 and 2 are endowed with algebras F1 and F2 respectively.
Let us take any B in F2 and take inverse of B, f-1{B}. Since B,
subset of 2 , f-1{B} is a subset 1..
If f-1{B} belong to F1 for all B in F2 , f is called measurable w.r. t.
F1 and F2
General Measurable Functions II
f: 1
In applications 2 is a metric space and F2 ,, Borel -algebra , the algebra generated by open sets.
How can we test measurability of a f?
The theorem that helps: f-1((S))=(f-1(S))
If f 1(S) F1
That implies f is F1 -F2 measurable.
Algebras are what we need for super/sub-level sets of a vector
space of indicator functions.
1A1
1A2
A1
A2
A1 A2
11A1 21A2
A1 A2
Superlevel Sets
A1
A2
A1 A2
11A1 21A2
A1 A2
Superlevel Sets
A1
A2
Simple Functions
Simple functions are finite linear combinations of of
indicator functions. n
1
i 1
i Ai
algebras let us take limits of these. That is more interesting!!
Measurable Functions
Given a Measurable Space (,F)
We say that a function:
respect to F if:
f :
y f ( )
is measurable with
f ({ y }) F
1
This simply says that a function is measurable with respect
to the -algebra if all its superlevel sets are in the -algebra.
Hence, a measurable function is one that when we slice it
like Lebesgue, we can measure the width of the part of the
function above the slice. Simple...right?
Another equivalent way to think of measurable functions is as
the pointwise limit of simple functions.
In fact, if a measurable function is non-negative, we can say it
is the increasing pointwise limit of simple functions.
This is simply going back to Lebesgues picture...
Intuition behind measurable functions .
They are constant on the sets in the sigma algebra.
[0,1)
F { , [0,1), [0, 12 ), [ 12 ,1)}
What do measurable functions look like?
Yes
Yes
1
2
1
2
No
1
2
Intuitively speaking, Measurable functions are constant on the sets in the -alg.
More accurately, they are limits of functions that are constant on the sets in the
-algebra. The in -algebra gives us this.
Information and s-Algebras.
1
2
Since measurable functions are constant on the -algebra, if
I am trying to determine information from a measurable
function, the -algebra determines the information that I can
obtain.
I measure y f ( ) What is the most information that I
determine about ?
The best possible I can do is to say that A with A F
-algebras determine the amount of information possible in a
function.
Measures and Integration
A measurable space (F) defines the sets can be measured.
Now we actually have to measure them...
What are the properties that size should satisfy.
If you think about it long enough, there are really only two...
Definition of a measure:
Given a Measurable Space (,F),
A measure is a function
satisfying two properties:
:F
1) ( ) 0
2) If A1 , A2 , A3 ,... are disjoint
i 1
i 1
(Ai ) ( Ai )
(2) is known as countable additivity. However, it can
be interpreted as linearity and left continuity for sets.
Different Measures
Infinite different measures are available on a sample space
Counting measure: Let (A)= n if A contains n number
of elements, otherwise.
Discrete measure: Let ={x1, x2, -- - ,xn,-,- } .
p
i 1
xi
1, px 0, i
(A)
xi A
xi
Different Measures
Lebesgue measure ) : There is a unique measure m on
( R ,B ) that satisfies
m ( [a , b] ) = b - a
for every finite interval [a , b] , . This is called the Lebesgue
measure . If we restrict m to measurable space ( [0 , 1] ,
B) , then m is a probability measure
Lebesgue Stieltjes measure ) :Let F be a nondecreasing and right-continuous function from from R to
R.There is a unique measure m on ( R ,B ) that satisfies
m ( [a , b] ) = F(b) F(a)
for every finite interval [a , b] . This is called the LebesgueStieltjes measure . If we take bounded F, then m is a
Measure-Probability Measure-Cumulative
Distribution Function
Bounded measure
Probability
measure
Cumulative
Distribution
Function
Point function Vs Set Function
F(x)=Pr{X<=x}
Point
Function
Set
Function
What is the relation between them??
Cumulative Distribution Function
Vs Probability Function
F(x)||f(x), p(x)
Directly
Related to
measure
No easy
connection
with
measure
What is the relation between them??
Continuity of Functions
For all sn
f(s )
f(s)
it is true in a general metric space but not
in general topological space.
f: <S1,d1>
<S2,d2>
is continuous at s in S1.
Fundamental properties of measures (or size):
Left Continuity: This is a trivial consequence of the definition!
Let
Ai A then lim ( Ai ) ( A)
Right Continuity: Depends on boundedness!
Let
Ai A
( Ai )
then lim ( Ai ) ( A)
and for some i
(Here is why we need boundedness. Consider Ai [i, ) then A )
lim ( Ai ) ( A)
then
Ai A
Let
(A)= (A 1)+ (A 2)- (A 1)+------+
(A n 0- (A n-1)-------
A=A 1U(A 2-A 1)U------(A n-A n-1)-------
Continuity above-property
of Measure
Induced Measures by Measurable
Functions
f: 1
1 and 2 are endowed with algebras F1 and F2 respectively.
Let us take any B in F2 and take inverse of B, f-1{B}. Since B,
subset of 2 , f-1{B} is a subset 1..
If f-1{B} belong to F1 for all B in F2 , f is called measurable w.r. t.
F1 and F2
Any Measurable Function
Induces a Measure on Its Measurable
Codomain
But How?
The domain should be a measure space with a measure.
Let be the measure on the domain
<, A,>
f(A)
<S,B>
f-1(B)
m(B)=(f-1(B)) for all B in B . The definition is possible since f-1{B}
belong to A.. M is written as f-1.
Now we can define the integral.
Simple Functions:
1
i 1
n
i Ai
d ( A )
i 1
Positive Functions:
Since positive measurable functions can be written as the
increasing pointwise limit of simple functions, we define the
integral as
d
fd sup
f
For a general measurable function write
fd f d f d
How should I think about
fd
A
This picture says everything!!!!
f
This is a set! and the
integral is measuring
the size of this set!
Integrals are like
measures! They
measure the size of a
set. We just describe
that set by a
function.
Therefore, integrals
should satisfy the
properties of
measures.
This leads us to another important principle...
Measures and integrals are different descriptions of of
the same concept. Namely, size.
Therefore, they should satisfy the same properties!!
Lebesgue defined the integral so that this would be
true!
Measures are:
Left Continuous
Ai A lim ( Ai ) ( A)
Integrals are:
Left Continuous
f i f lim f i d fd
(Monotone Convergence Thm.)
Bdd Right Cont.
Ai A and ( Ai )
lim ( Ai ) ( A)
Bdd Right Cont.
f i f and f i d
lim f i d fd
(Bounded Convergence Thm.)
(Fatous Lemma, etc...)
etc...
etc...
The Lp Spaces
p
|
f
|
d
A function is in L (,F,) if
p
The Lp spaces are Banach Spaces with norm:
| f | d
p
1/ p
L2 is a Hilbert Space with inner product:
f , g fgd
1 p
Examples of Normed spaces
Lp spaces
Lp={f:<X, , >C| (|f|pd )1/p < , 1<=p<} with
||f||= (|f|pd )1/pis a Banach space if we consider f=g a.e.
are equal. For counting measure the condition is not
needed,
C(X) is dense in it if X=Rk, =(Rk) and =Lebesgue
Stieltjes measure.
0<p<1 it is a Frchet soace
For p=, if define ||f||=ess sup|f| < and ess sup
g=inf{c : {:g()> c}=0}, under this norm the space is
Banach
Examples of Normed spaces
Lp spaces (Cont.)
The space is very very large.
The space is not only important for Kernel methods
but also for the development of Fucntional Analysis as
well as Foundation of Theory of Statistics
When X is finite, , the counting measure and p=2,
we have our well-known, Rk
Checking of Existence of Mean of Cauchy Distribution:
Let X be a standard Cauchy variate with
probability density
1 function,
f ( x)
(1 x )
2
; x
Since X be a continuous random variable,
therefore by definition of mean,
E ( X ) xf ( x)dx;
Provided that the integration
exists.
Here we use two way of integral to determine the
above integration, they are;
(i)Lebesgue Integral
(ii)Riemann Integral
Checking of Existence of Mean of Cauchy Distribution:
Lebesgue Integral Way
xf (x)dx
Now,
x
0 1 x2 dx
x
dx
2
1 x
1
1
x
dx x
dx
2
2
1 x
1 x
1
1
_
x
dx
x
dx
0 1 x2
1 x2
x
1 1 x2 dx
x
dx
2
1 x
[1, )
1 x2 z
2 xdx dz
n,n 1
1
log z
1x
2 n1
1
1 (n 1)2
log
2 n1
1 n2
put
1(n 1)2
2
n+1
1+(n+1)2
1+n2
Checking of Existence of Mean of Cauchy Distribution:
Since
And
1 (n 1) 2
1 (n 1) 2
log
log
2
1 n
1 2n 2
1 (n 1) 2 1
lim log
2
n
1 n
2
dx
2
does
1 x
0
E( X )
not
exist
does not exist
Therefore mean of the Cauchy distribution does
not exist
Checking of Existence of Mean of Cauchy Distribution:
Riemann Integral Way
Method (i):
We know
that,
x
lim
dx 0;
2
a
a 1 x
a
x
But lim
dx
2
a
2a 1 x
1a
1
lim log z
a
1 4a
2
1
1 a2
lim log
a
2
1 4a2
1
2
E (X )
x
Since 2
1 x
function
does not
exist.
is an odd
Checking of Existence of Mean of Cauchy Distribution:
Riemann Integral Way
Method (ii):
x
2
2
2
x
x
1 x
lim
1 0
2
2
x
1
1 x
1 x
x
x
dx
2
1
x
0
E (X )
does not exist by
quotient test
does not
exist
Therefore, mean of the Cauchy distribution does
not exist.
Measure-Probability Measure-Cumulative
Distribution Function
Bounded measure
Probability
measure
Cumulative
Distribution
Function
Discrete P(A)=1,
#(A)=finite or
Probability
Measures
Continuous
P{x}=0 for
all x
Absolutely
Continous w.r.t.
Lebesgue measure
Non A.C.
On Euclidean sample
space
Broad Categories of
Probability Measures
Expectation
A random variable is a measurable function.
X ( )
The expectation of a random variable is its integral:
E ( X ) XdP
A density function is the Radon-Nikodym derivative wrt
Lebesgue measure:
dP
fX
E ( X ) XdP xf X ( x)dx
dx
Counting measure p=dP/d
=xipi
Expectation
Change of variables. Let f be measurable from (, F , v)
to (,) and g be Borel on (,). Then
(1)
i.e. , if either integral exists, then so does the other, and the
two are the same.
Note that integration domains are indicated on both
sides of 1. This result extends the change of variable
formula for Reimann integrals, i.e. ,
Expectation
In statistics we will talk about expectations with respect
to different measures.
E P ( X ) XdP
E Q ( X ) XdQ
And write expectations in terms of the different measures:
dP
P
E ( X ) XdP X
dQ XdQ E Q ( X )
dQ
dP
or dP dQ
where
dQ
