Lecture 5: The multivariate normal
distribution
The bivariate normal distribution
Suppose µx , µy , σx ≥ 0, σy ≥ 0 and −1 ≤ ρ ≤ 1 are constants.
Define the 2 × 2 matrix Σ by
2
σx ρσx σy
Σ= .
ρσx σy σy2
Then define a joint probability density function by
1 1
fX ,Y (x, y ) = √ exp − Q(x, y )
2π det Σ 2
where
Q(x, y ) = (x − µ)T Σ−1 (x − µ)
and
x µx
x= , µ= .
y µy
If random variables (X , Y ) have joint probability density given by
fX ,Y above, then we say that (X , Y ) have a bivariate normal
distribution and write
(X , Y )T ∼ N2 (µ, Σ).
It can be proved that the function fX ,Y (x, y ) integrates to 1 and
therefore defines a valid joint pdf.
The notes contain expansions of Q(x, y ) and det Σ.
Remarks
1 The vector µ = (µx , µy )T is called the mean vector and the
matrix Σ is called the covariance matrix (or sometimes
variance-covariance matrix).
2 Functions of the form F (x) = x T Σ−1 x are called quadratic
forms. Quadratic forms are functions Rn → R which satisfy
certain properties. They crop up in several areas of
mathematics and statistics.
3 The matrix Σ and its inverse Σ−1 are positive definite. A
matrix A is positive definite if
x T Ax ≥ 0
for all non-zero vectors x.
4 It follows that when µx = µy = 0, Q(x, y ) is a positive
definite quadratic form.
Pictures
σx = σy, ρ = 0
3
90%
2
y
−1
80%
−2
95%
−3 99%
−3 −2 −1 0 1 2 3
x
σx = 2σy, ρ = 0
3
0
y
Pictures −1
80%
−2
95%
−3 99%
−3 −2 −1 0 1 2 3
x
σx = 2σy, ρ = 0
3
2
99%
1 80%
y
−1 90%
95%
−2
−3
−3 −2 −1 0 1 2 3
x
2σx = σy, ρ = 0
3 99%
−1 90%
95%
Pictures −2
−3
−3 −2 −1 0 1 2 3
x
2σx = σy, ρ = 0
3 99%
95%
2
80%
y
−1
−2 90%
−3
−3 −2 −1 0 1 2 3
x
Pictures
σx = σy, ρ = 0.75
3
90%
2
y
−1
80%
−2
95%
−3 99%
−3 −2 −1 0 1 2 3
x
σx = σy, ρ = − 0.75
3 99%
0
y
Pictures −1
80%
−2
95%
−3 99%
−3 −2 −1 0 1 2 3
x
σx = σy, ρ = − 0.75
3 99%
90%
2
y
−1
80%
−2
95%
−3
−3 −2 −1 0 1 2 3
x
2σx = σy, ρ = 0.75
3
−1
Pictures −2
80%
95%
−3
−3 −2 −1 0 1 2 3
x
2σx = σy, ρ = 0.75
3
95%
2
80%
1
y
−1
90%
−2
99%
−3
−3 −2 −1 0 1 2 3
x
Comments
1 Q(x, y ) ≥ 0 with equality only when x = µ. It follows that
the density function has its mode at x = µ.
2 Changing the values of µx , µy does not change the shape of
the plots, but corresponds to a translation of the xy -plane
i.e. changing µx , µy just shifts the contours / surface to a
new mode position.
3 The contours of equal density are circular when σx = σy and
ρ = 0 and elliptical when σx 6= σy or ρ 6= 0.
4 σx and σy control the extent to which the distribution is
dispersed.
5 The parameter ρ is the correlation of X , Y
i.e. Cor (X , Y ) = ρ. Thus for non-zero ρ, the contours are at
an angle to the axes.
Marginals and conditionals
Suppose (X , Y )T ∼ N2 (µ, Σ). Then:-
1 The marginal distributions are normal:
X ∼ N(µx , σx2 ) and
Y ∼ N(µy , σy2 ).
2 The conditional distributions are normal:
σx
X |Y = y ∼ N(µx + ρ (y − µy ), σx2 (1 − ρ2 )) and
σy
σy
Y |X = x ∼ N(µy + ρ (x − µx ), σy2 (1 − ρ2 )).
σx
3 When ρ = 0, X and Y are independent.
4 Linear combinations of X and Y are also normally distributed:
aX + bY ∼ N(aµx + bµy , a2 σx2 + b 2 σy2 + 2abρσx σy )
where a, b are constants.
Example 5.1
Suppose (X , Y )T ∼ N2 (µ, Σ) where µx = 2, µy = 3, σx = 1,
σy = 1 and ρ = 0.5.
Simulate a sample of size 500 from this distribution and draw a
scatter plot.
Use simulation to find Pr X 2 + Y 2 < 9 .
Solution
The marginal distribution of X is X ∼ N(2, 12 ).
Using the formula for the conditional
σy
Y |X = x ∼ N(µy + ρ (x − µx ), σy2 (1 − ρ2 ))
σx
∼ N(3 + 0.5(x − 2), 0.75).
Example 5.1
Suppose (X , Y )T ∼ N2 (µ, Σ) where µx = 2, µy = 3, σx = 1,
σy = 1 and ρ = 0.5.
Simulate a sample of size 500 from this distribution and draw a
scatter plot.
Use simulation to find Pr X 2 + Y 2 < 9 .
Solution
The marginal distribution of X is X ∼ N(2, 12 ).
Using the formula for the conditional
σy
Y |X = x ∼ N(µy + ρ (x − µx ), σy2 (1 − ρ2 ))
σx
∼ N(3 + 0.5(x − 2), 0.75).
Simulation results
1 n p t s = 500
2 x = rnorm ( n p t s , mean=2, s d = 1 )
3 y = rnorm ( n p t s , mean=3+0.5∗ ( x −2) , s d=s q r t ( 0 . 7 5 ) )
●
6
● ●
● ●
● ● ●●
5
● ● ● ●
● ●●●
● ● ● ●● ● ●
●
● ● ●● ● ● ● ●
●
●●●● ● ● ● ● ●
●● ● ● ●●
● ● ●●●● ● ● ●
4
●●●●● ● ●
●●●●● ●●●
● ●●●● ●
●● ●●●● ●
● ●● ●● ● ●
● ●● ● ●●●● ● ● ●●●● ●●●●●● ● ● ●
● ● ●● ● ●
● ● ● ●
● ●● ●● ●●
● ●●●●●●●● ●●●●●●●●●●●
●●●● ●
●● ●●● ●●●● ●
● ●●●●
●
● ●
●
●● ●●● ● ● ●●
● ●
●● ●
●● ●●●●●●● ●●●
● ● ●● ●●● ● ● ●
● ●● ● ●● ● ● ●● ● ●● ● ●● ●
y
● ●●●
3
● ● ●
● ● ● ● ●
● ●
●●● ● ●●●●●● ●● ● ●● ●●● ●
● ●
● ●●● ●● ●●● ●● ●●
● ●● ● ●● ● ● ●● ● ●●
● ●●●●● ●●
●●●●● ●●
●●
● ● ●●●●●●● ●● ● ● ●
●●
● ●● ● ●●● ● ● ●●●● ●●●
●●● ● ● ●●●● ●
●●●● ● ● ●●● ●●
●
● ● ●●● ● ●●
● ● ● ●● ● ●● ● ●●●
● ● ● ● ●● ●
2
●●
● ●● ●● ●● ●●●● ●●●● ● ● ●
● ●● ● ● ●
●
● ● ● ●● ● ●● ●●● ● ●
● ●●●●
● ●
● ● ● ●
● ● ●● ●
●● ● ● ●
● ● ● ● ● ● ●●
1
●
●●
● ● ●
●●●
●
●
● ●
0
0 1 2 3 4 5
x
Probability calculation
To find Pr X 2 + Y 2 < 9 approximately, count the number of
points in the region:
1 n p t s = 10000
2 x = rnorm ( n p t s , mean=2, s d = 1 )
3 y = rnorm ( n p t s , mean=3+0.5∗ ( x −2) , s d=s q r t ( 0 . 7 5 ) )
4 f = xˆ2+y ˆ2
5 sum ( f <9)/ n p t s
Answer ' 0.2776
Extra example
Suppose
X 4 8 2
∼ N2 , .
Y 1 2 5
The random variable Z is defined by Z = X + 3Y . What is the
distribution of Z ?
Extra example
We have Z = X + 3Y . Using result 4 on page 31, we have
E [Z ] = 1 × µx + 3 × µy = 1 × 4 + 3 × 1 = 7.
Now from the variance-covariance matrix, we have ρσx σy = 2.
Thus
Var (Z ) = 12 × σx2 + 32 × σy2 + 2 × 1 × 3 × (ρσx σy )
= 1×8+9×5+2×1×3×2
= 65.
Therefore Z ∼ N(7, 65).
Extra example
We have Z = X + 3Y . Using result 4 on page 31, we have
E [Z ] = 1 × µx + 3 × µy = 1 × 4 + 3 × 1 = 7.
Now from the variance-covariance matrix, we have ρσx σy = 2.
Thus
Var (Z ) = 12 × σx2 + 32 × σy2 + 2 × 1 × 3 × (ρσx σy )
= 1×8+9×5+2×1×3×2
= 65.
Therefore Z ∼ N(7, 65).
Extra example
We have Z = X + 3Y . Using result 4 on page 31, we have
E [Z ] = 1 × µx + 3 × µy = 1 × 4 + 3 × 1 = 7.
Now from the variance-covariance matrix, we have ρσx σy = 2.
Thus
Var (Z ) = 12 × σx2 + 32 × σy2 + 2 × 1 × 3 × (ρσx σy )
= 1×8+9×5+2×1×3×2
= 65.
Therefore Z ∼ N(7, 65).
Extra example
We have Z = X + 3Y . Using result 4 on page 31, we have
E [Z ] = 1 × µx + 3 × µy = 1 × 4 + 3 × 1 = 7.
Now from the variance-covariance matrix, we have ρσx σy = 2.
Thus
Var (Z ) = 12 × σx2 + 32 × σy2 + 2 × 1 × 3 × (ρσx σy )
= 1×8+9×5+2×1×3×2
= 65.
Therefore Z ∼ N(7, 65).
Extra example
−20 0 20 40
The multivariate normal distribution
The multivariate normal distribution is defined on vectors in Rn .
Suppose that X is a random vector with n entries, i.e.
X = (X1 , . . . , Xn )T .
Then
X ∼ Nn (µ, Σ)
if X1 , . . . , Xn have joint PDF given by
1 1
fX (x) = √ exp − Q(x)
2π det Σ 2
where
Q(x) = (x − µ)T Σ−1 (x − µ).
This definition makes sense for any column vector µ ∈ Rn and any
positive definite n × n matrix Σ.
Remarks
1 The vector µ is the mean of the distribution and Σ is called
the covariance matrix.
2 All the marginal distributions of X are normal. (We do not
specify their parameters here, however).
3 Similarly, all the conditional distributions of X are normal.
(Again, we do not specify the parameters of these
distributions here).
