On Day 3 we learned how the mean, median and mode describe the centre of a dataset. But the centre on its own is like knowing a city's average yearly temperature: it won't tell you whether to pack a jacket or sunscreen. Today we look at the other half of the story: how spread out is the data?
Why the centre isn't enough
Look at the three restaurants above. On paper they're identical, because each one averages 30 minutes. In real life they're nothing alike. Restaurant C is so consistent you could set a timer by it. Restaurant A might arrive in 11 minutes or in 48. If you're ordering before a meeting, the number you actually care about is not the average. It's how much the delivery time varies.
That variation is called dispersion (or spread), and the numbers that measure it are called measures of dispersion. They answer questions like: are the values huddled close to the centre, or scattered far from it?
Designers of uniforms, seats and cockpits often reached for one convenient target: the "average man". In December 1952, Lieutenant Gilbert S. Daniels of the US Air Force's Aero Medical Laboratory published a short report that tested that idea with a simple question: how many airmen actually are average?
He used body measurements of 4,063 airmen from the Air Force's 1950 anthropometric survey and picked ten measurements that matter for designing clothing (height, chest, sleeve length, waist and so on). He counted a man as "approximately average" on a measurement if he fell within 0.3 standard deviations of the mean. Then he filtered, one measurement at a time.
Not one man out of 4,063 was average on all ten measurements. Daniels' conclusion: the "average man" is "a misleading and illusory concept as a basis for design criteria", especially when more than one measurement matters. Equipment built for the average fits almost nobody. In later retellings, most famously Todd Rose's book The End of Average (2016), his report is credited with pushing the Air Force toward equipment that adjusts to the person, such as adjustable seats.
The lesson for data work: an average describes a dataset, not the individuals in it. When the spread is large, many real values sit far from the average, and you only find that out by measuring the spread.
In this article we'll cover five measures of spread, from the simplest to the most robust:
- Range: the gap between the largest and smallest values
- Variance: the average squared distance from the mean
- Standard deviation: the square root of variance, back in the original units
- Median absolute deviation (MAD): a typical distance from the median, built to resist outliers
- Interquartile range (IQR): the width of the middle 50% of the data
Range
Range is the simplest measure of spread there is: the biggest value minus the smallest.
Example
Here are the salaries (in US dollars) of 10 data scientists at one company:
[78.3k, 72k, 75k, 80k, 73.5k, 70k, 78k, 85k, 71k, 73k]
So these salaries span a $15,000 gap. That's a useful first impression, and it takes two seconds to compute.
The limits of range
Range has two big problems. First, it ignores everything in the middle: only the maximum and the minimum count. Second, it's extremely sensitive to outliers, because one extreme value is the maximum or the minimum. Watch what happens when we swap the $85k salary for $300k (perhaps a typo, perhaps the head of the team):
[78.3k, 72k, 75k, 80k, 73.5k, 70k, 78k, 71k, 73k, 300k]
The range jumps from $15k to $230k because of one value, while the other nine salaries didn't change at all. We'll come back to this exact list several times today, as a test of which measures get fooled.
Adding values to a dataset can widen the range or leave it alone, but it can never shrink it. Measure the heights of 10 people and then of 10,000, and the bigger group will almost always have a larger range, simply because you had more chances to meet someone unusually tall or short. So comparing ranges across groups of very different sizes is unfair from the start.
Variance
We want a measure that uses every value, not just the two extremes. A natural idea: measure how far each point is from the mean, then average those distances. Variance is that idea, with one twist.
Why we can't just average the deviations
A point's deviation is its distance from the mean, with a sign: positive if it's above, negative if it's below. Let's try averaging deviations for this small dataset, whose mean is 50:
[46, 69, 32, 60, 52, 41]
The deviations are −4, +19, −18, +10, +2 and −9. Add them up and you get exactly 0. That's not a coincidence of this dataset. The deviations from the mean always add up to exactly zero, for any dataset, because that's what makes the mean the balance point. So "average deviation" is always 0 and tells us nothing.
The fix is to make every deviation positive before adding. Variance does this by squaring each deviation. (Taking the absolute value would also work, and we'll meet that idea later in MAD. Squaring became the standard because it has much nicer mathematical properties, which later topics like regression rely on.)
Population vs. sample variance
There are two versions of the formula, depending on whether your data is the entire population or a sample from it (the distinction from Day 1).
The only difference is dividing by n − 1 instead of n. This is called Bessel's correction, and we'll see why it exists in a moment.
Worked example
Let's treat [46, 69, 32, 60, 52, 41] as a sample.
Step 1. Find the mean:
Step 2. Find each deviation and square it:
| Value | Deviation (x − 50) | Squared deviation |
|---|---|---|
| 46 | −4 | 16 |
| 69 | +19 | 361 |
| 32 | −18 | 324 |
| 60 | +10 | 100 |
| 52 | +2 | 4 |
| 41 | −9 | 81 |
| Total | 0 | 886 |
Step 3. Divide by n − 1, which is 5 here:
Why divide by n − 1?
Here's the intuition. The deviations in the formula are measured from the sample mean, not the true population mean, which we don't know. The sample mean is calculated from these very points, so it always sits right in the middle of them. In fact, no other number gives a smaller total of squared distances to your sample. That means distances measured from the sample mean come out a little too small, compared with distances from the true mean. Dividing by a slightly smaller number, n − 1, scales the answer back up by just the right amount.
You don't have to take that on trust. Here's a simulation: a population whose true variance we know is exactly 100, and 100,000 small samples of 5 values drawn from it.
Dividing by n is wrong in a consistent direction: it lands around 80 instead of 100. That's just what the theory predicts: for samples of 5 it comes out low by a factor of 4/5 on average. Dividing by n − 1 lands on 100. A formula that is right on average is called unbiased, and that's what Bessel's correction buys you. With bigger samples the gap shrinks, because n and n − 1 become almost the same number. (One small footnote: taking the square root brings back a slight bias, so the sample SD still runs a little low on average. In practice it's rarely worth worrying about.)
The limitation of variance
Because we squared the deviations, the answer is in squared units. If your data is in dollars, the variance is in "dollars squared", which nobody can picture. That's why we almost always take one more step.
Standard deviation
Standard deviation (SD) is the square root of the variance. That one step brings the number back into the original units of your data, so you can read it right next to the mean.
For our dataset:
How to read it: 13.31 is a typical distance between a value and the mean of 50. Be careful with the word "average" here. It's not literally the average distance: the plain average of the distances (4, 19, 18, 10, 2 and 9) is 10.33. Squaring gives big deviations extra weight, so the SD always comes out at least as large as the average distance, and larger still when a few points sit far away. Think of it as "a typical distance from the mean, with far-off points counting extra".
NumPy's np.std() and np.var() divide by n unless you pass ddof=1. pandas' .std() and .var() divide by n − 1 unless you pass ddof=0. So the same column can give two different answers depending on the library:
For 6 values the gap is about 9.5%. The two answers differ by a factor of √(n / (n − 1)), so the gap fades as n grows: for 1,000 values it's about 0.05%. It's the small datasets where it matters. If your data is a sample (and in data science it nearly always is), use the n − 1 version.
Standard deviation in practice
| Use case | How SD helps |
|---|---|
| Describing a distribution | High SD means widely spread values; low SD means values huddle close to the mean. |
| Finance and risk | The SD of an investment's returns, usually called its volatility, is a standard way to measure how bumpy the ride is. |
| Quality control | Factories track the SD of a product's measurements to check that it's being made consistently (see the case below). |
| Machine learning | Standardization rescales each feature using its mean and SD, so features measured in different units become comparable. |
| Spotting unusual values | Values more than 2 or 3 SDs from the mean are often flagged for a closer look. As we'll see below, this rule has a serious blind spot. |
On 16 May 1924, Walter Shewhart, an engineer at Bell Labs, wrote an internal memo that introduced the control chart. The idea: keep measuring a product as it's made, and draw limits at the mean plus and minus three standard deviations. Small wobbles inside those limits are normal, everyday variation that you should leave alone. A point outside them is a signal that something specific has gone wrong, such as a worn machine or a bad batch of material, and is worth investigating. Control charts with 3-SD limits are still a basic tool in manufacturing quality control a century later.
The limitation: outliers fool it too
Variance and SD are both built around the mean, and as we saw all through Day 3, the mean gets dragged around by outliers. Worse, the squaring step makes big deviations count even more. So one extreme value can inflate the SD dramatically.
That creates a nasty trap for the "flag anything more than 3 SDs away" rule. Let's test it on our salary list with the $300k value in it:
The $300k salary is nearly four times everyone else's, yet it's only 2.84 SDs above the mean, so a 3-SD rule would not flag it. The outlier has stretched the very ruler we're using to measure it: it pulled the mean up to $97k and blew the SD up to $71k. It's like measuring how unusual a giant is with a tape measure that stretches whenever the giant holds it. This effect is called masking.
In small datasets it can get even worse. For a sample of n values, no point can ever be more than (n − 1) / √n SDs from the mean. With 5 values that limit is 1.79, so a 3-SD rule can't flag anything at all, no matter how extreme. We'll meet the SD-based "how many SDs away" score properly, as the z-score, on Day 10.
We need measures of spread that an outlier can't stretch. That's what the next two are for.
Median absolute deviation (MAD)
MAD is a robust stand-in for the standard deviation. Instead of measuring distances from the mean and averaging their squares, it measures distances from the median and takes their median. That double swap, median instead of mean at both steps, is what makes it resistant to extreme values. We used it briefly in Day 1's salary example; here's the full recipe.
How to calculate MAD
- Find the median of the data.
- Find each value's absolute deviation from that median (the distance, ignoring the sign).
- Find the median of those absolute deviations. That's the MAD.
Worked example
Six test scores: [94, 85, 78, 92, 100, 96]
Step 1. Sorted, they're [78, 85, 92, 94, 96, 100]. With 6 values, the median is the average of the middle two: (92 + 94) / 2 = 93.
Step 2. Distances from 93, in the same sorted order: [15, 8, 1, 1, 3, 7].
Step 3. Sort those: [1, 1, 3, 7, 8, 15]. Their median is (3 + 7) / 2 = 5.
So the MAD is 5. Because it's a median, you can read it like this: half of the scores are within 5 points of the median score, and half are further away.
Now the real test. Suppose the 100 gets mistyped as 1000:
One typo multiplied the SD by more than 46. The MAD moved from 5 to 5.5. That's what robust means.
Putting MAD on the same scale as SD
On the same data, MAD and SD are naturally different sizes, so you can't compare them directly. For data shaped like the classic bell curve, the SD is about 1.4826 times the MAD. Multiplying the MAD by 1.4826 gives a robust estimate of the SD. SciPy does this for you with scale="normal":
For the clean scores that's 7.41, close to the true SD of 8.01, and with the typo it barely moves, to 8.15. The ordinary SD jumped to 371.97.
Some books and tools use MAD for the mean absolute deviation (the average distance from the mean, the 10.33 we computed earlier), which is not robust. pandas used to have a .mad() method that computed exactly that. It was deprecated in pandas 1.5 and no longer exists in pandas 2. For the robust median version, use scipy.stats.median_abs_deviation, and always say which MAD you mean.
MAD for spotting outliers
The modified z-score, recommended by Iglewicz and Hoaglin and described in the NIST/SEMATECH Engineering Statistics Handbook, swaps the mean and SD for the median and MAD:
Let's give it the salary list that fooled the 3-SD rule:
A score of 43.5 against a threshold of 3.5. The $300k salary can't drag the median or the MAD, so it can't hide.
Interquartile range (IQR)
The IQR is the other robust measure of spread, and the one you'll see most often. It looks only at the middle 50% of the data and ignores the top and bottom 25%, which is exactly where outliers live.
Quartiles first
Quartiles are the three values that cut sorted data into four equal-sized chunks:
| Quartile | What it is | Percentile |
|---|---|---|
| Q1 (lower quartile) | A quarter of the data sits below it | 25th |
| Q2 (the median) | Half the data sits below it | 50th |
| Q3 (upper quartile) | Three quarters of the data sits below it | 75th |
Worked example by hand
17 students' test scores: [98, 94, 94, 62, 73, 87, 77, 68, 72, 85, 99, 100, 86, 60, 52, 100, 80]
Step 1, sort: [52, 60, 62, 68, 72, 73, 77, 80, 85, 86, 87, 94, 94, 98, 99, 100, 100]
Step 2, find the median (Q2): with 17 values, the middle one is the 9th, so Q2 = 85.
Step 3, find Q1 and Q3: one common hand method takes the median of each half, leaving out the overall median. The lower half is [52, 60, 62, 68, 72, 73, 77, 80], so Q1 = (68 + 72) / 2 = 70. The upper half is [86, 87, 94, 94, 98, 99, 100, 100], so Q3 = (94 + 98) / 2 = 96.
Step 4, IQR: 96 − 70 = 26.
So the middle half of the class scored within a 26-point band. The lowest score (52) and the highest (100) have no say in that number at all.
Now the same thing in NumPy
NumPy says the IQR is 22, not 26. Neither answer is wrong. There is no single agreed definition of a quartile, as we mentioned on Day 1. When the 25% mark falls between two data points, different methods settle it differently. By default, NumPy and pandas count positions in the sorted list from 0 and put the 25% mark at position 0.25 × (n − 1), interpolating if it lands between two values. With 17 values that's position 4, which holds the value 72; the 75% mark is position 12, which holds 94. The hand method instead takes the middle of each half. On large datasets the methods give nearly the same answer; on 17 values the gap is visible.
NumPy can use other methods too. For this particular dataset, method="weibull" happens to give 70 and 96, matching the hand method:
That match is a coincidence of this dataset, not a general rule. The practical lesson: before comparing an IQR from one tool with an IQR from another, check that they used the same quartile method. A mismatch usually means two different rulers, not a mistake.
The IQR fence rule for outliers
The IQR's most common job is the "fence" rule, popularised by the statistician John Tukey along with the box plot (he described both in his 1977 book Exploratory Data Analysis):
Upper fence = Q3 + 1.5 × IQR
With the hand-calculated quartiles (Q1 = 70, Q3 = 96, IQR = 26):
Upper fence = 96 + 1.5 × 26 = 96 + 39 = 135
No score is below 31 or above 135, so nothing gets flagged. (With NumPy's quartiles the fences are 39 and 127, and nothing gets flagged either.)
This is the rule behind the whiskers and outlier dots in a standard box plot. Matplotlib uses 1.5 × IQR by default, as do most plotting libraries, and it computes the quartiles the way NumPy does. So a box plot drawn by Matplotlib would show a box from 72 to 94 here, not 70 to 96.
Now our salary list, the one that slipped past the 3-SD rule:
The fence sits at about $87k, so the $300k salary is flagged immediately. Here's the scorecard for the three outlier checks we've run on this list:
| Method | Built on | Score for $300k | Flagged? |
|---|---|---|---|
| More than 3 SDs from the mean | Mean and SD | 2.84 SDs | No, masked by the outlier |
| Modified z-score above 3.5 | Median and MAD | 43.5 | Yes |
| Beyond the IQR fence | Quartiles | $300k vs. a fence at about $87k | Yes |
These rules find values worth a second look, not values to throw away. The $300k might be a typo, or it might be the team's director. Check where it came from before you decide. Day 3's 500 cm height was clearly an error; a large salary might be perfectly real.
Spread and centre belong together
A measure of spread means most when it's reported next to a measure of centre. Here are two companies, both with a mean salary of exactly $96,000:
- Company A: [$20k, $20k, $20k, $20k, $400k]
- Company B: [$90k, $94k, $96k, $98k, $102k]
Report only the mean and these companies look identical. Add the SD and the difference jumps out: about $170k for Company A against about $4.5k for B. Notice something else, though. For Company A, "mean ± SD" would be $96k ± $170k, a range that starts below zero. That's the SD telling you that the mean is the wrong summary for this data in the first place. The data is badly skewed, and the median ($20k), which matches four of the five employees, tells the real story.
That gives a simple rule:
- Roughly symmetric data, no wild outliers: report the mean ± standard deviation.
- Skewed data or outliers: report the median with the IQR (or the MAD).
Choosing a measure of spread
| Measure | Uses every value? | Units | Robust to outliers? | Best for |
|---|---|---|---|---|
| Range | No, only two | Original | No | A quick first look |
| Variance | Yes | Squared | No | Maths and models behind the scenes |
| Standard deviation | Yes | Original | No | Clean, roughly symmetric data |
| MAD | Yes, via two medians | Original | Yes | Messy data, outlier detection |
| IQR | No, only the middle 50% | Original | Yes | Skewed data, box plots, fences |
Key takeaway
Every time you report a centre, report a spread right next to it. A mean or median on its own is an incomplete summary, and it can easily be a misleading one: a cockpit built for the average pilot fits nobody.
- Range is quick but fragile: one value decides it.
- Variance and SD use every value and power a lot of statistics, but an outlier can inflate them and even hide itself.
- MAD and IQR stay put when outliers appear, which makes them the safer choice for messy, real-world data.
- Check your defaults: NumPy's SD divides by n, pandas' by n − 1, and quartile methods vary between tools.
Quick check
Five questions to test whether this stuck. The answers are one click away.
Two datasets have the same range of $50,000. Do they have the same amount of spread?
Not necessarily. Range only looks at the maximum and the minimum. In one dataset, everything in between could be tightly bunched; in the other, spread evenly across the whole gap. You'd need the SD, MAD or IQR to tell them apart.
You compute a standard deviation and get a negative number. What went wrong?
Something in the calculation, because an SD can never be negative. It comes from squared deviations (zero or positive) and a square root (which never returns a negative number). An SD of exactly 0 is possible, and means every value is identical.
A colleague runs np.std(x) and you run x.std() on the same pandas column. Your answers differ slightly. Who's right?
You're computing different things. NumPy divides by n by default (the population formula); pandas divides by n − 1 (the sample formula). For sample data the n − 1 version is the standard choice, so pass ddof=1 to NumPy. The gap is biggest for small datasets.
A $300k salary in a list of $70k to $80k salaries is only 2.84 SDs above the mean. Why isn't it further?
Because the outlier itself inflated both the mean and the SD, the ruler used to measure it. This is called masking. Robust methods built on the median, such as the modified z-score (MAD) or the IQR fences, flag it easily.
Why does MAD use the median twice, instead of the mean at either step?
Because the whole point of MAD is to resist outliers, and the mean is exactly the number outliers drag around. Using the median for the centre and for summarising the distances means one wild value can barely move the answer, as the typo test showed: the SD jumped from 8.01 to 371.97 while the MAD went from 5 to 5.5.
An SD of 10 is huge for exam scores out of 20 and tiny for house prices. Next time: how to compare spread across datasets that aren't even on the same scale.