Introduction to Probability

The language of uncertainty: what a probability is, how to count your way to one, how new information changes it, and the classic mistakes that fool even doctors and courts.

impossible certain as likely as not 0 rolling a 7 with one die 1/36 double six with two dice 1/6 a six with one die 1/2 heads on a fair coin 3/4 at least one head in two flips 1 rolling 1 to 6 on a die 0 0.25 0.5 0.75 1 0 0.5 1 0 rolling a 7 with one die 1/36 double six with two dice 1/6 a six with one die 1/2 heads on a fair coin 3/4 at least one head in two flips 1 rolling 1 to 6 on a die
Every probability lives somewhere on this line, from 0 (impossible) to 1 (certain). Nothing is ever below 0 or above 1.

Part I was about describing data you already have. Part II is about reasoning when you don't know what will happen next. A spam filter has to decide whether an email is spam before anyone reads it. A weather service says "70% chance of rain" about a day that hasn't happened yet. A model predicts whether a customer will cancel. All of these speak the same language: probability.

What is probability?

A probability is a number between 0 and 1 that says how likely something is. 0 means it can't happen, 1 means it's certain, and 0.5 means it's as likely as not. You'll also see probabilities written as percentages (0.25 is 25%).

There are three common ways to arrive at that number:

Whichever way you get there, the same three basic rules always hold:

  1. Every probability is between 0 and 1.
  2. The probabilities of all possible outcomes add up to exactly 1. Something has to happen: for a coin, P(heads) + P(tails) = 0.5 + 0.5 = 1.
  3. The complement rule: the probability that A does not happen is 1 − P(A). If there's a 70% chance of rain, there's a 30% chance of no rain.

The building blocks

Probability has its own vocabulary. These five terms come up constantly:

The counting formula

P(A) = number of favourable outcomestotal number of possible outcomes
This only works when every outcome is equally likely, like the faces of a fair die. It doesn't work for "will it rain tomorrow?", where "rain" and "no rain" are two outcomes but not equally likely ones.

Example 1: flipping a coin twice

H H H T T H T T 3 of the 4 equally likely outcomes contain at least one head H H H T T H T T 3 of the 4 outcomes have at least one head
The four equally likely outcomes of two coin flips. Three of them include at least one head.

The sample space has 4 equally likely outcomes. Three of them (HH, HT, TH) contain at least one head, so P(at least one head) = 3/4 = 0.75. Only TT has no heads, so P(no heads) = 1/4. The two answers add up to 1, as the complement rule says they must.

Example 2: rolling two dice

second die first die 1 1 2 2 3 3 4 4 5 5 6 6 1,1 1,2 1,3 1,4 1,5 1,6 2,1 2,2 2,3 2,4 2,5 2,6 3,1 3,2 3,3 3,4 3,5 3,6 4,1 4,2 4,3 4,4 4,5 4,6 5,1 5,2 5,3 5,4 5,5 5,6 6,1 6,2 6,3 6,4 6,5 6,6 second die first die 1 1 2 2 3 3 4 4 5 5 6 6 1,1 1,2 1,3 1,4 1,5 1,6 2,1 2,2 2,3 2,4 2,5 2,6 3,1 3,2 3,3 3,4 3,5 3,6 4,1 4,2 4,3 4,4 4,5 4,6 5,1 5,2 5,3 5,4 5,5 5,6 6,1 6,2 6,3 6,4 6,5 6,6
All 36 equally likely outcomes of rolling two dice. The 11 highlighted cells show at least one 5. The darker cell, (5,5), is counted once, not twice.

What's the probability that at least one die shows a 5? Counting the highlighted cells gives 11 out of 36, about 0.306.

A tempting shortcut is "1/6 for the first die plus 1/6 for the second, so 2/6 = 12/36". That's one too many, because it counts the outcome (5,5) twice. The fix is the addition rule:

P(A or B) = P(A) + P(B) − P(A and B)
Here: 6/36 + 6/36 − 1/36 = 11/36. If A and B can't happen together, P(A and B) = 0 and you simply add.

The complement rule gives the same answer another way. The chance that neither die shows a 5 is 5/6 × 5/6 = 25/36, so the chance that at least one does is 1 − 25/36 = 11/36. "At least one" questions are often easiest through their complement.

Checking probabilities by simulation

When you're not sure a calculation is right, you can simply run the experiment many times on a computer and count. With NumPy, a million coin-flip pairs take a fraction of a second:

Pythonimport numpy as np rng = np.random.default_rng(42) for trials in [100, 10_000, 1_000_000]: flips = rng.integers(0, 2, size=(trials, 2)) # 1 = heads, 0 = tails at_least_one_head = (flips.max(axis=1) == 1).mean() print(f"{trials:>9,} trials: P(at least one head) = {at_least_one_head:.4f}") print("Exact answer: 0.7500")
Output 100 trials: P(at least one head) = 0.7800 10,000 trials: P(at least one head) = 0.7424 1,000,000 trials: P(at least one head) = 0.7501 Exact answer: 0.7500

With 100 trials the estimate is off by a few percentage points; with a million it's almost exact. That's the law of large numbers at work: the more times you repeat a random experiment, the closer the observed frequency gets to the true probability. It's also why the "long-run frequency" view of probability works at all.

Real case: a game of solitaire and the Monte Carlo method

In 1946, while recovering from an illness, the mathematician Stanislaw Ulam wondered about the chance of winning a game of solitaire. Working it out exactly meant analysing an enormous number of card arrangements. He realised it would be far simpler to play many games and count the wins. Ulam and John von Neumann developed the idea into a general method for hard calculations in physics, and named it after the famous casino in Monaco: the Monte Carlo method. Today it's used everywhere from finance to climate models to checking the maths in an article like this one.

Counting at scale: combinatorics

Listing outcomes works for two coins. It doesn't work when there are millions of them. Combinatorics is the maths of counting without listing.

The multiplication principle

If one choice can be made in a ways and a second, independent choice in b ways, together there are a × b combinations. Say a product team is testing a landing page with 3 headlines, 4 hero images and 2 button colours. That's 3 × 4 × 2 = 24 different page versions.

Why it matters: each version needs enough visitors to measure it reliably. Add a third button colour and you have 3 × 4 × 3 = 36 versions. That's 50% more versions, so you need 50% more traffic (or a longer test) to give each one the same number of visitors. Combinatorics tells you the cost of an experiment before you run it.

Permutations: when order matters

With repetition allowed, choosing k items from n options, where each position can be any of the n, gives nk arrangements. A 4-digit PIN has 10 choices for each digit, so there are 104 = 10,000 possible PINs, and a random guess has a 1 in 10,000 chance.

Without repetition, once an item is used it's gone. The number of ordered arrangements of k items chosen from n is:

P(n, k) = n!(n − k)!
n! ("n factorial") means n × (n − 1) × ⋯ × 1. For example, 5! = 120.

A delivery driver has 10 stops to choose from and time for 5 of them. The first stop can be any of 10, the second any of the 9 left, then 8, 7 and 6: 10 × 9 × 8 × 7 × 6 = 30,240 possible ordered routes. Arrange all n items and you get n! orderings, which grows absurdly fast. A shuffled deck of 52 cards can be ordered in 52! ways, about 8 × 1067.

Real case: UPS and the 120-stop route

A UPS driver typically makes around 120 deliveries per shift, and the number of possible orders for 120 stops is 120!, a number with 199 digits. Nobody can check them all. Instead, UPS built a route-optimisation system called ORION (On-Road Integrated Optimization and Navigation), which it rolled out across its US delivery routes in the mid-2010s. For a 120-stop route it evaluates more than 200,000 options to find an efficient one. UPS has said it expects ORION to cut about 100 million miles and 10 million gallons of fuel a year, worth $300 million to $400 million annually. Counting is what tells you why clever search, rather than brute force, is the only option.

Combinations: when order doesn't matter

Often you only care which items are chosen, not their order. That's a combination, written C(n, k) and read "n choose k":

C(n, k) = n!k! × (n − k)!
It's the number of permutations divided by k!, because each group of k items can be ordered in k! ways, and we only want to count it once.

Here's where it bites in machine learning. You have 20 features and want to try every possible subset of 5 to find the best model. That's C(20, 5) = 15,504 models. At one second per model, about 4.3 hours. Fine. Now take 50 features and subsets of 10: over 10 billion models, or about 326 years at one second each. That's why data scientists use smarter strategies, such as step-by-step feature selection or LASSO regularisation, instead of trying every subset.

Pythonfrom math import comb, perm, factorial print(f"Ordered routes, 10 stops choose 5: {perm(10, 5):,}") print(f"Feature subsets, 20 choose 5: {comb(20, 5):,}") print(f"Feature subsets, 50 choose 10: {comb(50, 10):,}") print(f"Years at 1 second per model: {comb(50, 10) / (365.25 * 24 * 3600):,.0f}") print(f"Orderings of a 52-card deck: {factorial(52):.2e}")
OutputOrdered routes, 10 stops choose 5: 30,240 Feature subsets, 20 choose 5: 15,504 Feature subsets, 50 choose 10: 10,272,278,170 Years at 1 second per model: 326 Orderings of a 52-card deck: 8.07e+67

Independent and dependent events

The multiplication rule

For independent events, the probability that both happen is the product of their probabilities:

P(A and B) = P(A) × P(B)   (only if A and B are independent)

For any two events, independent or not, there's a general version:

P(A and B) = P(A) × P(B | A)
P(B | A) is read "the probability of B given A". When A and B are independent, P(B | A) is just P(B), and this becomes the rule above.

Example: two aces. Draw two cards from a shuffled deck without putting the first back. P(first is an ace) = 4/52. If it was, 3 aces remain among 51 cards, so P(second is an ace | first was) = 3/51. Together: 4/52 × 3/51 = 1/221, about 0.45%. If you put the first card back and reshuffle, the draws become independent: (4/52)² = 1/169, about 0.59%.

Example: a sales funnel. 3% of visitors click an ad, and 10% of those who click go on to buy. A purchase through the ad requires a click first, so these are dependent: P(click and buy) = P(click) × P(buy | click) = 0.03 × 0.10 = 0.003. Out of 1,000,000 visitors, expect about 3,000 purchases. Raise the click rate from 3% to 3.1% and that becomes 3,100. Small probability changes add up at scale.

Real case: when assuming independence went badly wrong

In 1999, Sally Clark, an English solicitor, was convicted of murdering her two baby sons, who had died suddenly. At her trial, the paediatrician Sir Roy Meadow told the court that the chance of a single sudden infant death in a family like hers was about 1 in 8,543. He then squared it, as you would for independent events, to get about 1 in 73 million for two such deaths.

Multiplying assumes the two deaths were independent. But genetic and environmental factors shared within a family can make a second sudden death far more likely after a first. In 2001 the Royal Statistical Society issued a statement saying the 1 in 73 million figure had "no statistical basis". The figure also invited a second error, which you'll meet below: confusing the chance of the evidence with the chance of guilt. Sally Clark's conviction was overturned on appeal in 2003. Check independence before you multiply.

Conditional probability

Conditional probability answers the question: how does knowing one thing change the probability of another? P(A | B) means "the probability of A, given that B has happened".

P(A | B) = P(A and B)P(B)

The intuition: knowing B happened shrinks your world down to just the outcomes where B is true. Then you ask what fraction of that smaller world also has A.

Example: spotting fraud

Imagine a bank looks at 10,000 card transactions. 100 are fraudulent (1%). 500 were made from a location the customer had never used before (5%). And 80 transactions are both fraudulent and from a new location. If a transaction comes from a new location, how likely is it to be fraud?

All 10,000 transactions 1% fraud keep only the 500 from a new location 80 fraud 420 genuine New location only: 500 transactions 16% fraud All 10,000 transactions 1% fraud keep only the 500 from a new location 80 fraud 420 genuine New location only: 500 transactions 16% fraud
Conditioning means zooming in. Across all transactions, 1% are fraud. Among the 500 from a new location, 80 are fraud: 16%.
Pythontotal = 10_000 fraud = 100 new_location = 500 fraud_and_new_location = 80 p_fraud = fraud / total p_fraud_given_new = fraud_and_new_location / new_location # zoom into the 500 p_new_given_fraud = fraud_and_new_location / fraud # zoom into the 100 print(f"P(fraud) = {p_fraud:.0%}") print(f"P(fraud | new location) = {p_fraud_given_new:.0%}") print(f"P(new location | fraud) = {p_new_given_fraud:.0%}")
OutputP(fraud) = 1% P(fraud | new location) = 16% P(new location | fraud) = 80%

Knowing the location raised the fraud probability from 1% to 16%, sixteen times higher. That's what makes it a useful signal. Real fraud systems combine many signals like this one (amount, device, time of day) to sharpen the estimate further.

Now look at the last line. P(new location | fraud) is 80%, but P(fraud | new location) is only 16%. P(A | B) and P(B | A) are different numbers, and mixing them up is one of the most common and costly mistakes in reasoning with probability. Most fraud comes from new locations, but most new-location transactions are not fraud, simply because genuine transactions vastly outnumber fraudulent ones. The tool that connects the two directions is Bayes' theorem.

Bayes' theorem: updating beliefs with evidence

Bayes' theorem tells you how to update a probability when new evidence arrives. It's named after Thomas Bayes, an 18th-century English minister whose essay on the idea was published after his death, in 1763.

P(A | B) = P(B | A) × P(A)P(B)

Example: a simple spam filter

Let's use round numbers to keep the arithmetic easy. Suppose half of all incoming email is spam, so the prior is P(spam) = 0.5. The word "free" appears in 80% of spam emails but only 10% of genuine ones. A new email contains "free". How likely is it to be spam?

First, the overall chance of seeing "free":

P("free") = 0.80 × 0.5 + 0.10 × 0.5 = 0.40 + 0.05 = 0.45

Then Bayes' theorem:

P(spam | "free") = 0.80 × 0.50.45 ≈ 0.889

Before reading the email, the belief was 50%. After seeing one word, it's 88.9%. If the email also contains "lottery" (in 60% of spam, 2% of genuine email), we can update again:

Pythonp_spam = 0.50 p_free = {"spam": 0.80, "genuine": 0.10} # P("free" | class) p_lottery = {"spam": 0.60, "genuine": 0.02} # P("lottery" | class) # One word top = p_free["spam"] * p_spam bottom = top + p_free["genuine"] * (1 - p_spam) print(f"P(spam | 'free') = {top / bottom:.1%}") # Two words, treating them as independent within each class (the "naive" assumption) top = p_free["spam"] * p_lottery["spam"] * p_spam bottom = top + p_free["genuine"] * p_lottery["genuine"] * (1 - p_spam) print(f"P(spam | 'free', 'lottery') = {top / bottom:.1%}")
OutputP(spam | 'free') = 88.9% P(spam | 'free', 'lottery') = 99.6%
Before reading the email 50.0% After seeing "free" 88.9% After "free" and "lottery" 99.6% Before reading the email 50.0% After seeing "free" 88.9% After "free" and "lottery" 99.6%
Each piece of evidence updates the belief. Start with a prior, update it with the evidence, and the result is the posterior, which becomes the prior for the next piece of evidence.

Chaining evidence like this, word by word, is the idea behind the Naive Bayes classifier, a simple and fast machine learning model that still works well for text. It's "naive" because it assumes the words are independent of each other within each class (among spam emails, and separately among genuine ones). That's rarely exactly true, as "free" and "lottery" probably do appear together more often than chance, but the simplification usually works surprisingly well.

Real case: "A Plan for Spam"

In August 2002 the programmer Paul Graham published an essay called "A Plan for Spam". He proposed giving every word a spam probability, learned from a person's own email, and combining the probabilities of the most telling words in each message using Bayes' rule. On his own email, he reported, the filter missed fewer than 5 spams per 1,000 and wrongly blocked no genuine emails. The essay helped popularise Bayesian spam filtering, which spread into many spam filters and email programs in the following years. Modern filters use far more complex models, but the core idea of combining evidence into an updated probability remains the same.

The base rate trap

Bayes' theorem matters most when the thing you're looking for is rare. Here's the classic example. A disease affects 1 in 1,000 people. A test for it catches 99% of people who have it, and wrongly flags 1% of people who don't. Someone tests positive. What's the chance they have the disease?

Most people answer 99%. The real answer is about 9%. The easiest way to see why is to imagine 100,000 people:

0.1% 99.9% 99% 1% 1% 99% 100,000 people 100 have the disease 99,900 do not 99 test positive 1 tests negative 999 test positive 98,901 test negative Of the 1,098 positive tests, only 99 are real: 99 / 1,098 ≈ 9% 0.1% 99.9% 99% / 1% 1% / 99% 100,000 people 100 have it 99,900 do not 99 positive 1 negative 999 positive 98,901 negative Positive tests: 99 + 999 = 1,098 Only 99 of them are sick: about 9%
The same probabilities as counts of people, sometimes called natural frequencies. The 999 false alarms from the large healthy group swamp the 99 true positives from the small sick group.

Out of 100,000 people, 100 have the disease and 99 of them test positive. Of the 99,900 who don't, 1% (999 people) also test positive. So 1,098 people test positive in total, and only 99 of them are actually sick: 99 / 1,098 ≈ 9%. The disease is so rare (the base rate, or prior, is so low) that the false alarms from the huge healthy group outnumber the true cases. Bayes' theorem gives the same answer:

Pythonprior = 1 / 1000 # P(disease) sensitivity = 0.99 # P(positive | disease) false_alarm = 0.01 # P(positive | no disease) p_positive = sensitivity * prior + false_alarm * (1 - prior) posterior = sensitivity * prior / p_positive print(f"P(disease | positive test) = {posterior:.1%}")
OutputP(disease | positive test) = 9.0%

A positive result is still very informative: it raised the chance from 0.1% to about 9%, a ninety-fold jump. But it's far from a diagnosis, which is why a positive screening test is usually followed by a second, different test.

Real case: even doctors fall for it

The psychologist Gerd Gigerenzer has spent years testing how doctors read screening tests. When a group of 160 gynaecologists was given the facts about a mammography screening test as percentages and asked how likely a woman with a positive result was to have breast cancer, the most common answer was 90%. The correct answer was about 10%. Gigerenzer's fix is the frequency tree above: in one of his training sessions, the share of gynaecologists who got the right answer rose from 21% to 87% once the same information was presented as counts of people instead of percentages.

The same trap is behind the second error in the Sally Clark case. Even if a 1 in 73 million figure had been right, it was the chance of the evidence if she were innocent, not the chance she was innocent given the evidence. Turning one into the other is exactly what Bayes' theorem does, and it needs the prior. Confusing the two is so common in courtrooms that it has a name: the prosecutor's fallacy.

Key takeaways

Quick check

Five questions to test whether this stuck. The answers are one click away.

You flip a fair coin 3 times. What's the sample space, and what's P(at least one tail)?

There are 2³ = 8 outcomes: HHH, HHT, HTH, HTT, THH, THT, TTH, TTT. Only HHH has no tails, so by the complement rule P(at least one tail) = 1 − 1/8 = 7/8 = 0.875.

A friend says the chance of at least one six when rolling two dice is 1/6 + 1/6 = 1/3. What's wrong?

Adding counts the outcome (6,6) twice. By the addition rule it's 1/6 + 1/6 − 1/36 = 11/36, about 0.306. Or by the complement: 1 − (5/6)² = 11/36.

You want to arrange 3 of your 6 books on a shelf, and the order matters. Permutation or combination, and how many arrangements?

A permutation, since order matters: 6! / (6 − 3)! = 6 × 5 × 4 = 120 arrangements. If you only cared which 3 books you picked, it would be the combination C(6, 3) = 20.

"80% of people who like jazz also like classical music." Is that P(classical | jazz) or P(jazz | classical)?

P(classical | jazz): the probability of liking classical, given that someone likes jazz. The group you're restricting to, here jazz fans, goes after the bar. And it tells you nothing directly about P(jazz | classical), which could be very different.

A disease affects 1 in 1,000 people, and a test is 99% accurate both ways. Someone tests positive. Are they probably sick?

No. Their chance of having the disease is only about 9%. Out of 100,000 people, the test finds 99 of the 100 sick people but also flags 999 healthy ones, so most positives are false alarms. The low base rate is what Bayes' theorem accounts for and intuition ignores.

Next up, Day 8 Random Variables and Probability Distributions

Today we worked with single events. Next: how to describe all the possible outcomes of a random process at once, and what their probabilities look like as a distribution.