On Day 7 we computed the probability of single events: rolling a 5, a transaction being fraud, an email being spam. But most questions in data science are about a whole range of possible values. How many support tickets will arrive in the next hour? How long will a user browse before choosing something? How much will the next customer spend? Answering those needs two new ideas: random variables and probability distributions.
What is a random variable?
A random variable is a rule that turns each outcome of a random experiment into a number. That's it. Once outcomes are numbers, you can average them, measure their spread and draw them, with every tool from Part I.
Take Day 7's experiment of flipping a coin twice. The outcomes are HH, HT, TH and TT, which aren't numbers. Define X as "the number of heads". Now each outcome has a number: HH gives 2, HT and TH give 1, TT gives 0.
Because two of the four equally likely outcomes give X = 1, P(X = 1) = 2/4 = 1/2, while P(X = 0) and P(X = 2) are 1/4 each. That list of values and their probabilities is the random variable's distribution, which is the main subject of this article.
Random variables get capital letters, like X or Y. A particular value gets a lowercase letter, like x. So P(X = x) reads "the probability that the random variable X takes the value x", and P(X = 1) = 0.5 is a specific example.
Discrete and continuous random variables
Random variables come in two main kinds, and the kind decides which tools you use.
- Discrete random variables take separate, countable values with gaps between them: usually whole numbers. The number of ads a user clicks in a session (0, 1, 2, but never 1.7). The number of bug reports filed today. The face of a die.
- Continuous random variables can take any value in a range, with no gaps. The time a user spends browsing (23.7 seconds, or 23.71, or 23.712...). A page's load time. A person's height.
A quick test: if you'd answer "how many?", it's usually discrete; if you'd answer "how much?" or "how long?", it's usually continuous. (A few variables are a mix of both. Daily rainfall is exactly 0 on many days but can be any positive amount on the rest. They're less common, and the ideas below still apply in pieces.)
For a discrete variable you can list every value and ask "what's the probability of exactly this one?" For a continuous variable there are infinitely many possible values, and the probability of hitting any single exact value is zero. You can only ask about ranges. Hold on to this: it explains why discrete and continuous variables need different tools.
What is a probability distribution?
A probability distribution is a complete description of a random variable: every value it can take, and how likely each one is. The simplest example is one roll of a fair die. X is the face shown, and each face is equally likely:
| x | 1 | 2 | 3 | 4 | 5 | 6 | Total |
|---|---|---|---|---|---|---|---|
| P(X = x) | 1/6 | 1/6 | 1/6 | 1/6 | 1/6 | 1/6 | 1 |
Every valid distribution follows the two rules from Day 7: each probability is between 0 and 1, and all the probabilities add up to exactly 1, because one of the values must happen.
How you write a distribution down depends on the kind of variable. Discrete variables use a probability mass function (PMF). Continuous variables use a probability density function (PDF). Both can be summarised by a cumulative distribution function (CDF). Let's take them in turn.
The probability mass function (PMF)
For a discrete random variable, the PMF gives the probability of each possible value:
Think of it as a lookup table: give it a value, and it gives you that value's probability.
Example: support tickets per hour
Suppose a support team looks back over a year of peak-time hours and finds this distribution for X, the number of tickets that arrive in an hour:
| x (tickets) | 0 | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|---|
| P(X = x) | 0.05 | 0.15 | 0.30 | 0.25 | 0.15 | 0.10 |
The probabilities add up to 1, so it's a valid PMF. Now questions become lookups and sums. The chance of exactly 2 tickets is P(X = 2) = 0.30. The chance of more than 3 is P(X = 4) + P(X = 5) = 0.25.
A PMF from real data
In practice you often estimate a PMF from data by counting how often each value appears. The tips dataset that ships with seaborn records 244 restaurant bills, collected by one waiter over a few months. One column is the size of the dining party:
That's an empirical PMF: almost two in three parties (63.9%) were a party of two, and parties of five or six were rare. With more data, these proportions settle closer to the true probabilities for that restaurant, the law of large numbers from Day 7 again.
Expected value and variance
On Day 3 we summarised data with a mean, and on Day 4 with a variance and standard deviation. Random variables have exactly the same summaries.
Expected value: the long-run average
The expected value of X, written E[X] or μ, is the average you'd get if you could repeat the experiment forever. For a discrete variable, multiply each value by its probability and add:
Notice that 2.6 isn't a value X can ever take: you can't receive 2.6 tickets in an hour. That's normal. A fair die's expected value is 3.5, which no roll ever shows. The expected value is a long-run average, not a prediction for any single hour. Over 100 peak hours, though, you should expect about 260 tickets, and that's exactly the kind of number you staff a team around.
Variance: the spread around it
Variance works just like Day 4's, with probabilities as the weights:
A typical hour sees 2.6 tickets, give or take about 1.3. You'll see E[X] and Var(X) for every named distribution on Day 9 and Day 10.
An American roulette wheel has 38 pockets: the numbers 1 to 36 plus 0 and 00. Bet $1 on a single number and, if it comes up, you're paid $35 and keep your dollar. Otherwise you lose the dollar. So:
E[winnings] = 35 × (1/38) + (−1) × (37/38) = −2/38 ≈ −$0.0526
On average, every dollar bet loses about 5.3 cents. That's the "house edge". On a European wheel, with a single zero and 37 pockets, it's 1/37, about 2.7 cents. Any single spin can go either way, but over millions of bets the casino's average take settles right on the expected value. This is the law of large numbers working for the house.
The probability density function (PDF)
Now try the PMF idea on a continuous variable, such as how long a user browses before picking something to watch. What's the probability they browse for exactly 62.000000... seconds? Zero. There are infinitely many possible times, so any single one has probability 0. A bar chart would need infinitely many bars, all of height zero.
The fix is to spread probability along a curve, the probability density function, and to measure probability as area under it:
The height of the curve, f(x), is called the density. It tells you where values are concentrated, but density is not a probability. It can even be bigger than 1. If a quantity is spread evenly between 0 and 0.5, its density is 2 across that range, because 2 × 0.5 = 1 is the total area. Only areas are probabilities.
Example: how long do viewers browse?
In a 2015 paper on Netflix's recommender system, Netflix's Carlos Gomez-Uribe and Neil Hunt wrote that their consumer research suggested "a typical Netflix member loses interest after perhaps 60 to 90 seconds of choosing". That makes browsing time a quantity worth understanding.
Let's model it. Suppose browsing time X, in seconds, follows an exponential distribution with an average of 90 seconds: most people choose quickly, and a few browse for a long time. (This is an illustrative model, not Netflix's data. We'll meet the exponential distribution properly on Day 10.) SciPy can compute the areas for us:
Under this model, almost half of users (48.7%) choose within a minute, and 13.5% are still browsing after three minutes. That last group is the one at risk of giving up.
Where do the areas come from? SciPy's cdf method, which we'll meet in a moment, gives the area to the left of a point, so the area between 30 and 120 is the area left of 120 minus the area left of 30. And sf, the "survival function", is simply 1 minus the CDF: the area to the right.
The cumulative distribution function (CDF)
Many practical questions are about thresholds rather than exact values. What fraction of customers cancel within 30 days? What share of requests finish in under 200 milliseconds? What's the chance of 2 tickets or fewer? The CDF answers all of them:
For a discrete variable, F(x) adds up the PMF from the left up to x. For a continuous variable, it's the area under the PDF to the left of x. Either way, every CDF has the same basic behaviour: it starts at 0, never goes down, and ends at 1. For a discrete variable it climbs in steps, one step per possible value; for a continuous variable it climbs smoothly.
Building a CDF from a PMF
| x (tickets) | P(X = x) | F(x) = P(X ≤ x) |
|---|---|---|
| 0 | 0.05 | 0.05 |
| 1 | 0.15 | 0.05 + 0.15 = 0.20 |
| 2 | 0.30 | 0.20 + 0.30 = 0.50 |
| 3 | 0.25 | 0.50 + 0.25 = 0.75 |
| 4 | 0.15 | 0.75 + 0.15 = 0.90 |
| 5 | 0.10 | 0.90 + 0.10 = 1.00 |
The CDF turns questions about ranges into subtraction. Two details are worth knowing:
- "More than" uses the complement: P(X > 3) = 1 − F(3) = 0.25.
- For discrete variables, watch the endpoints. P(1 ≤ X ≤ 3) is F(3) − F(0), not F(3) − F(1), because F(1) already includes X = 1, which we want to keep. For continuous variables this doesn't matter: P(X = 1) is zero, so < and ≤ give the same answer.
Running the CDF backwards: percentiles
The CDF answers "what's the probability of being at or below x?" Turn it around and you get "below which value do p% of outcomes fall?" That's a percentile (or quantile), and the function that computes it is the inverse CDF. The median is the 50th percentile, and the quartiles from Day 4 are the 25th and 75th.
In SciPy the inverse CDF is called ppf, the "percent point function". Half of users choose within 62.4 seconds, and 90% within 207 seconds. Notice that the median (62.4) is below the mean (90): this distribution is right-skewed, exactly the pattern from Day 3.
When a doctor says a baby is "at the 75th percentile for length", they're reading a CDF: 75% of healthy babies of that age and sex are shorter. In the US, the Centers for Disease Control and Prevention recommends the World Health Organization's growth standards for children under two, and uses the 2nd and 98th percentiles as cut-offs that flag possible problems, such as unusually low weight for length. Percentiles turn a raw measurement into "how unusual is this?", which is exactly what a CDF is for.
An empirical CDF from real data
You can also estimate a CDF straight from data: F(x) is just the fraction of observations at or below x. Back to the restaurant bills:
60% of the bills were $20 or less, half were under $17.80, and only one in ten topped $32.24. No formula needed: the data is its own distribution.
PMF, PDF or CDF?
| Tool | For | Answers | Watch out for |
|---|---|---|---|
| PMF, p(x) | Discrete variables | "What's the chance of exactly x?" | The values must add up to 1. |
| PDF, f(x) | Continuous variables | "Where are values concentrated?" Probabilities come from areas. | The height is density, not probability, and can exceed 1. |
| CDF, F(x) | Both | "What's the chance of x or less?" Plus ranges by subtraction, and percentiles by running it backwards. | For discrete variables, < and ≤ give different answers. |
Key takeaways
- A random variable turns random outcomes into numbers. It's discrete (countable values) or continuous (any value in a range).
- The PMF gives exact probabilities for discrete variables. The PDF gives density for continuous ones, and probability is the area under it.
- The CDF is a running total that works for both, turns ranges into subtraction, and gives percentiles when run backwards.
- E[X] is the long-run average and Var(X) the spread around it: Day 3 and Day 4, now for random variables. A casino's profit is an expected value.
Quick check
Six questions to test whether this stuck. The answers are one click away.
Is "the number of typos in a blog post" discrete or continuous? What about "the time between two typos"?
The number of typos is discrete: 3 or 4, never 3.5. The time between them is continuous: it could be 12.3 seconds or 12.34 seconds, with no gaps between possible values.
A PMF has P(X = 1) = 0.2, P(X = 2) = 0.5, and P(X = 3) unknown. These are the only possible values. What is P(X = 3)?
0.3. A PMF must add up to 1, so P(X = 3) = 1 − 0.2 − 0.5 = 0.3.
For that same PMF, what is the expected value E[X]?
E[X] = 1(0.2) + 2(0.5) + 3(0.3) = 0.2 + 1.0 + 0.9 = 2.1.
Why is it meaningless to ask "what's the probability a session lasts exactly 47.000 minutes"?
Session length is continuous, so any single exact value has probability 0. A question with a real answer is "what's the probability the session lasts between 46 and 48 minutes?", which is the area under the PDF over that range.
A PDF reaches a height of 3 at one point. Is something wrong?
No. Density isn't probability, so it can exceed 1, as long as the total area under the curve is 1. A variable spread evenly between 0 and 1/3 has a density of exactly 3.
A CDF has F(10) = 0.4 and F(20) = 0.75. What is P(10 < X ≤ 20)?
0.35. F(20) − F(10) = 0.75 − 0.4. Once you have a CDF, the probability of any range is just a subtraction.
Some PMFs turn up so often that they have names: Bernoulli, binomial, geometric and Poisson. Next time: how to recognise each one, and what its expected value and variance tell you.