Basics of Probability
Theory
Background
Games of chance and gambling in the 17 th
century sparked interests in probability theory
A short history of probability:
http://homepages.wmich.edu/~mackey/Teaching/145/probHist.html
Probability Theory
“Probability theory is nothing but
common sense reduced to calculation”.
Pierre-Simon Laplace in 1812
Probability Theory
Classical/Frequentist View
Bayesian View
Classical/Frequentist View
Probability is estimated in terms of the
frequencies of random, repeatable events
E.g., coin toss, dice roll etc.
Definition of Probability
Experiment: toss a coin twice
Sample space: possible outcomes of an experiment
S = {HH, HT, TH, TT}
Event: a subset of possible outcomes
A={HH}, B={HT, TH}
Probability of an event : a number assigned to an event, i.e.,
P(A)
Axiom 1: 0 ≤ P(A) ≤ 1
Axiom 2: P(S) = 1
Definition of Probability
Experiment: toss a coin twice
Sample space: possible outcomes of an experiment
S = {HH, HT, TH, TT}
Event: a subset of possible outcomes
A={HH}, B={HT, TH}
Probability of an event : a number assigned to an event, i.e.,
P(A)
Axiom 3:
If an experiment can result in any one of ‘N’ different equally
likely outcomes, and if exactly ‘n’ of these outcomes correspond to
event A, then the probability of event A is:
P(A) = n / N
Example
A coin is tossed twice. What is the probability that at least
1 head occurs?
S = {HH, HT, TH, TT}
A = {HH, HT, TH}
P (A) = n / N = 3/4 (Axiom 3)
Additive Rule
If A and B are two events, then the probability that A or B
will occur is the sum of the probability of each event minus
the overlap
P(A U B) = P(A) + P(B) – P(A ∩ B)
Additive Rule
If A and B are two events, then the probability that A or B
will occur is the sum of the probability of each event minus
the overlap
P(A U B) = P(A) + P(B) – P(A ∩ B)
Additive Rule
If A and B are two events, then the probability that A or B
will occur is the sum of the probability of each event minus
the overlap
P(A U B) = P(A) + P(B) – P(A ∩ B)
If A and B are mutually exclusive, then
P(A U B) = P(A) + P(B)
Conditioning
If A and B are events with Pr(A) > 0, the conditional
probability of B given A is
Pr( AB)
Pr( B | A)
Pr( A)
Conditioning
If A and B are events with Pr(A) > 0, the conditional
probability of B given A is
Pr( AB)
Pr( B | A)
Pr( A)
Example: Drug test
A = {Patient is a Woman}
Women Men B = {Drug fails}
Success 200 1800 Pr(B|A) = ?
Failure 1800 200 Pr(A|B) = ?
Conditioning
If A and B are events with Pr(A) > 0, the conditional
probability of B given A is
Pr( AB)
Pr( B | A)
Pr( A)
Example: Drug test
A = {Patient is a Woman}
Women Men B = {Drug fails}
Success 200 1800 Pr(B|A) = 1800/2000 = 9/10
Failure 1800 200 Pr(A|B) = 1800/2000 = 9/10
Independence
Two events A and B are independent if and only if
P(B|A) = P(B) or P(A|B) = P(A)
Example:
A = {You catch Covid-19}
B = {Joe Biden catches Covid-19}
Otherwise, A and B are dependent
The Product/Multiplicative Rule
Enables us to calculate the joint probability
P(AB) that two events A and B will both
occur
P(A∩B) = P(A) P(B|A)
P(B∩A) = P(B) P(A|B)
P(A∩B) = P(B∩A)
The Product/Multiplicative Rule
P(A∩B) = P(A) P(B|A)
Example:
Suppose that we have a fuse box containing 20
fuses, of which 5 are defective. If 2 fuses are
selected at random and removed from the box in
succession without replacing the first, what is the
probability that both fuses are defective?
The Product/Multiplicative Rule
Solution:
A = {First fuse is defective}
B = {Second fuse is defective}
A∩B = {A occurs and then B occurs after A has
occurred}
P(A) = 5/20 = 1/4
P(B|A) = 4/19
P(A∩B) = P(A) P(B|A) = 1/4 * 4/19 = 1/19
Independence
P(A∩B) = P(A) P(B|A)
Two events A and B are independent if and only if
P(A ∩ B) = P(A) P(B)
Therefore, to obtain the probability that two
independent events will both occur, we simply find
the product of their individual probabilities
A set of events {Ai} is independent in case
Pr(i Ai ) i Pr( Ai )
Probability Theory
Classical/Frequentist View
Bayesian View
Bayesian View
More general view based on the Bayes’ Rule
Probabilities provide a quantification of
uncertainty in real-life events or phenomena
Examples
What is the probability that it will rain tomorrow?
How likely is it that this stock will go up by the
end of the month?
What is the likelihood that two companies will be
merged together?
Bayes’ Rule
Given two events A and B and suppose that Pr(A) > 0. Then
Pr( B A)
Pr( B | A)
Pr( A)
Pr( B A) Pr( B ) Pr( A | B)
Pr( B A) Pr( B ) Pr( A | B )
Pr( B | A)
Pr( A) Pr( A)
Bayes’ Rule
Given two events A and B and suppose
that Pr(A) > 0. Then
Thomas Bayes
Pr( B ) Pr( A | B ) (1701 - 1761)
Pr( B | A) England
Pr( A)
Bayes’ Rule
Given two events A and B and suppose that Pr(A) > 0. Then
Pr( B ) Pr( A | B )
Pr( B | A)
Pr( A)
Example:
Pr(R) = 0.8
R: It is a rainy day
Pr(W|R) R R
W: The grass is wet
W 0.7 0.3
Pr(R|W) = ?
W 0.3 0.7
Bayes’ Rule
Given two events A and B and suppose that Pr(A) > 0. Then
Pr( B ) Pr( A | B )
Pr( B | A)
Pr( A)
Example:
Pr(R) = 0.8
Pr(W|R) R R R: It is a rainy day
W 0.7 0.3 W: The grass is wet
Pr(R|W) = 0.56
W 0.3 0.7
Pr( R W ) Pr( R) Pr(W | R) 0.8 * 0.7
Pr( R | W ) 0.56
Pr(W ) Pr(W ) 1.0
Bayes’ Rule
R R
R: It rains
W 0.7 0.3
W: The grass is wet
W 0.3 0.7
Information
Pr(W|R)
R W
Inference
Pr(R|W)
Bayes’ Rule
R R
R: It rains
W 0.7 0.3
W: The grass is wet
W 0.3 0.7
Information: Pr(E|H)
Hypothesis H Evidence E
Posterior Likelihood
Inference: Pr(H|E) Prior
Pr( E | H ) Pr( H )
Pr( H | E )
Pr( E )
Bayes’ Rule
R R
R: It rains
W 0.7 0.3
W: The grass is wet
W 0.3 0.7
Information: Pr(E|H)
Hypothesis H Evidence E
Posterior Likelihood
Inference: Pr(H|E) Prior
Pr( E | H ) Pr( H )
Pr( H | E )
Pr( E )
Reference
Chapter 2: Probability & Statistics for
Engineers & Scientists, 9th edition, Walpole et
al, Prentice Hall (2012).
