Measures of Dispersion

Range, variance, standard deviation, MAD and the IQR: five ways to measure how spread out your data is, and how to pick one that a single bad value can't fool.

every restaurant averages 30 minutes Restaurant A range 37 min · SD 13.3 min Restaurant B range 15 min · SD 5.4 min Restaurant C range 6 min · SD 1.9 min 10 20 30 40 50 delivery time in minutes, seven orders each every restaurant averages 30 minutes Restaurant A range 37 min · SD 13.3 min Restaurant B range 15 min · SD 5.4 min Restaurant C range 6 min · SD 1.9 min 10 20 30 40 50 delivery time in minutes, seven orders each
Three restaurants, seven deliveries each, and every one of them averages exactly 30 minutes. Only Restaurant C delivered every order in under 35 minutes. The average can't tell you that; the spread can.

On Day 3 we learned how the mean, median and mode describe the centre of a dataset. But the centre on its own is like knowing a city's average yearly temperature: it won't tell you whether to pack a jacket or sunscreen. Today we look at the other half of the story: how spread out is the data?

Why the centre isn't enough

Look at the three restaurants above. On paper they're identical, because each one averages 30 minutes. In real life they're nothing alike. Restaurant C is so consistent you could set a timer by it. Restaurant A might arrive in 11 minutes or in 48. If you're ordering before a meeting, the number you actually care about is not the average. It's how much the delivery time varies.

That variation is called dispersion (or spread), and the numbers that measure it are called measures of dispersion. They answer questions like: are the values huddled close to the centre, or scattered far from it?

Real case: the US Air Force and the "average pilot"

Designers of uniforms, seats and cockpits often reached for one convenient target: the "average man". In December 1952, Lieutenant Gilbert S. Daniels of the US Air Force's Aero Medical Laboratory published a short report that tested that idea with a simple question: how many airmen actually are average?

He used body measurements of 4,063 airmen from the Air Force's 1950 anthropometric survey and picked ten measurements that matter for designing clothing (height, chest, sleeve length, waist and so on). He counted a man as "approximately average" on a measurement if he fell within 0.3 standard deviations of the mean. Then he filtered, one measurement at a time.

All airmen measured 4,063 Average on height 1,055 + average chest 302 + average sleeve length 143 after 9 measurements 2 after all 10 0, nobody All airmen measured 4,063 Average on height 1,055 + average chest 302 + average sleeve length 143 after 9 measurements 2 after all 10 0, nobody
How many of the 4,063 airmen stayed "approximately average" as Daniels added one body measurement after another. Data from Daniels (1952), The "Average Man"?, Technical Note WCRD 53-7.

Not one man out of 4,063 was average on all ten measurements. Daniels' conclusion: the "average man" is "a misleading and illusory concept as a basis for design criteria", especially when more than one measurement matters. Equipment built for the average fits almost nobody. In later retellings, most famously Todd Rose's book The End of Average (2016), his report is credited with pushing the Air Force toward equipment that adjusts to the person, such as adjustable seats.

The lesson for data work: an average describes a dataset, not the individuals in it. When the spread is large, many real values sit far from the average, and you only find that out by measuring the spread.

In this article we'll cover five measures of spread, from the simplest to the most robust:

Range

Range is the simplest measure of spread there is: the biggest value minus the smallest.

Range = Maximum − Minimum

Example

Here are the salaries (in US dollars) of 10 data scientists at one company:

[78.3k, 72k, 75k, 80k, 73.5k, 70k, 78k, 85k, 71k, 73k]

Range = $85,000 − $70,000 = $15,000

So these salaries span a $15,000 gap. That's a useful first impression, and it takes two seconds to compute.

The limits of range

Range has two big problems. First, it ignores everything in the middle: only the maximum and the minimum count. Second, it's extremely sensitive to outliers, because one extreme value is the maximum or the minimum. Watch what happens when we swap the $85k salary for $300k (perhaps a typo, perhaps the head of the team):

[78.3k, 72k, 75k, 80k, 73.5k, 70k, 78k, 71k, 73k, 300k]

Range = $300,000 − $70,000 = $230,000

The range jumps from $15k to $230k because of one value, while the other nine salaries didn't change at all. We'll come back to this exact list several times today, as a test of which measures get fooled.

One more catch: range grows with more data

Adding values to a dataset can widen the range or leave it alone, but it can never shrink it. Measure the heights of 10 people and then of 10,000, and the bigger group will almost always have a larger range, simply because you had more chances to meet someone unusually tall or short. So comparing ranges across groups of very different sizes is unfair from the start.

Variance

We want a measure that uses every value, not just the two extremes. A natural idea: measure how far each point is from the mean, then average those distances. Variance is that idea, with one twist.

Why we can't just average the deviations

A point's deviation is its distance from the mean, with a sign: positive if it's above, negative if it's below. Let's try averaging deviations for this small dataset, whose mean is 50:

[46, 69, 32, 60, 52, 41]

The deviations are −4, +19, −18, +10, +2 and −9. Add them up and you get exactly 0. That's not a coincidence of this dataset. The deviations from the mean always add up to exactly zero, for any dataset, because that's what makes the mean the balance point. So "average deviation" is always 0 and tells us nothing.

The fix is to make every deviation positive before adding. Variance does this by squaring each deviation. (Taking the absolute value would also work, and we'll meet that idea later in MAD. Squaring became the standard because it has much nicer mathematical properties, which later topics like regression rely on.)

deviation squared mean = 50 32 −18 324 41 −9 81 46 −4 16 52 +2 4 60 +10 100 69 +19 361 add them up 0 886 deviation squared mean = 50 32 −18 324 41 −9 81 46 −4 16 52 +2 4 60 +10 100 69 +19 361 add them up 0 886
Each bar is one value's deviation from the mean. The signed deviations cancel out to 0; the squared deviations can't cancel, and add up to 886.

Population vs. sample variance

There are two versions of the formula, depending on whether your data is the entire population or a sample from it (the distinction from Day 1).

Population variance
σ2 = ∑ (xi − μ)2N
Sample variance
s2 = ∑ (xi − x)2n − 1
μ is the population mean and N the population size; x is the sample mean and n the sample size.

The only difference is dividing by n − 1 instead of n. This is called Bessel's correction, and we'll see why it exists in a moment.

Worked example

Let's treat [46, 69, 32, 60, 52, 41] as a sample.

Step 1. Find the mean:

x = 46 + 69 + 32 + 60 + 52 + 416 = 3006 = 50

Step 2. Find each deviation and square it:

ValueDeviation (x − 50)Squared deviation
46−416
69+19361
32−18324
60+10100
52+24
41−981
Total0886

Step 3. Divide by n − 1, which is 5 here:

s2 = 8865 = 177.2

Why divide by n − 1?

Here's the intuition. The deviations in the formula are measured from the sample mean, not the true population mean, which we don't know. The sample mean is calculated from these very points, so it always sits right in the middle of them. In fact, no other number gives a smaller total of squared distances to your sample. That means distances measured from the sample mean come out a little too small, compared with distances from the true mean. Dividing by a slightly smaller number, n − 1, scales the answer back up by just the right amount.

You don't have to take that on trust. Here's a simulation: a population whose true variance we know is exactly 100, and 100,000 small samples of 5 values drawn from it.

Pythonimport numpy as np rng = np.random.default_rng(42) # 100,000 samples, each with 5 values, from a population with # mean 50 and standard deviation 10 (so the true variance is 100) samples = rng.normal(loc=50, scale=10, size=(100_000, 5)) print("Divide by n, average answer:", round(samples.var(axis=1, ddof=0).mean(), 1)) print("Divide by n - 1, average answer:", round(samples.var(axis=1, ddof=1).mean(), 1))
OutputDivide by n, average answer: 80.1 Divide by n - 1, average answer: 100.1

Dividing by n is wrong in a consistent direction: it lands around 80 instead of 100. That's just what the theory predicts: for samples of 5 it comes out low by a factor of 4/5 on average. Dividing by n − 1 lands on 100. A formula that is right on average is called unbiased, and that's what Bessel's correction buys you. With bigger samples the gap shrinks, because n and n − 1 become almost the same number. (One small footnote: taking the square root brings back a slight bias, so the sample SD still runs a little low on average. In practice it's rarely worth worrying about.)

The limitation of variance

Because we squared the deviations, the answer is in squared units. If your data is in dollars, the variance is in "dollars squared", which nobody can picture. That's why we almost always take one more step.

Standard deviation

Standard deviation (SD) is the square root of the variance. That one step brings the number back into the original units of your data, so you can read it right next to the mean.

σ = √σ2  (population)    s = √s2  (sample)

For our dataset:

s = √177.2 ≈ 13.31

How to read it: 13.31 is a typical distance between a value and the mean of 50. Be careful with the word "average" here. It's not literally the average distance: the plain average of the distances (4, 19, 18, 10, 2 and 9) is 10.33. Squaring gives big deviations extra weight, so the SD always comes out at least as large as the average distance, and larger still when a few points sit far away. Think of it as "a typical distance from the mean, with far-off points counting extra".

Pythonimport numpy as np data = [46, 69, 32, 60, 52, 41] print(np.var(data, ddof=1)) # sample variance print(np.std(data, ddof=1)) # sample standard deviation
Output177.2 13.311649033834989
Watch out: NumPy and pandas disagree by default

NumPy's np.std() and np.var() divide by n unless you pass ddof=1. pandas' .std() and .var() divide by n − 1 unless you pass ddof=0. So the same column can give two different answers depending on the library:

Pythonimport pandas as pd import statistics print("np.std(data) ", np.std(data)) # ddof=0: divides by n print("pd.Series(data).std() ", pd.Series(data).std()) # ddof=1: divides by n - 1 print("statistics.pstdev(data)", statistics.pstdev(data)) # population version print("statistics.stdev(data) ", statistics.stdev(data)) # sample version
Outputnp.std(data) 12.151817422372122 pd.Series(data).std() 13.311649033834989 statistics.pstdev(data) 12.151817422372122 statistics.stdev(data) 13.311649033834989

For 6 values the gap is about 9.5%. The two answers differ by a factor of √(n / (n − 1)), so the gap fades as n grows: for 1,000 values it's about 0.05%. It's the small datasets where it matters. If your data is a sample (and in data science it nearly always is), use the n − 1 version.

Standard deviation in practice

Use caseHow SD helps
Describing a distributionHigh SD means widely spread values; low SD means values huddle close to the mean.
Finance and riskThe SD of an investment's returns, usually called its volatility, is a standard way to measure how bumpy the ride is.
Quality controlFactories track the SD of a product's measurements to check that it's being made consistently (see the case below).
Machine learningStandardization rescales each feature using its mean and SD, so features measured in different units become comparable.
Spotting unusual valuesValues more than 2 or 3 SDs from the mean are often flagged for a closer look. As we'll see below, this rule has a serious blind spot.
Real case: Shewhart's control chart, 1924

On 16 May 1924, Walter Shewhart, an engineer at Bell Labs, wrote an internal memo that introduced the control chart. The idea: keep measuring a product as it's made, and draw limits at the mean plus and minus three standard deviations. Small wobbles inside those limits are normal, everyday variation that you should leave alone. A point outside them is a signal that something specific has gone wrong, such as a worn machine or a bad batch of material, and is worth investigating. Control charts with 3-SD limits are still a basic tool in manufacturing quality control a century later.

The limitation: outliers fool it too

Variance and SD are both built around the mean, and as we saw all through Day 3, the mean gets dragged around by outliers. Worse, the squaring step makes big deviations count even more. So one extreme value can inflate the SD dramatically.

That creates a nasty trap for the "flag anything more than 3 SDs away" rule. Let's test it on our salary list with the $300k value in it:

Pythonsalaries = np.array([78.3, 72, 75, 80, 73.5, 70, 78, 71, 73, 300]) # in $1,000s mean = salaries.mean() sd = salaries.std(ddof=1) z = (300 - mean) / sd # how many SDs above the mean the $300k sits print(f"Mean: {mean:.2f} SD: {sd:.2f}") print(f"$300k is {z:.2f} SDs above the mean")
OutputMean: 97.08 SD: 71.38 $300k is 2.84 SDs above the mean

The $300k salary is nearly four times everyone else's, yet it's only 2.84 SDs above the mean, so a 3-SD rule would not flag it. The outlier has stretched the very ruler we're using to measure it: it pulled the mean up to $97k and blew the SD up to $71k. It's like measuring how unusual a giant is with a tape measure that stretches whenever the giant holds it. This effect is called masking.

In small datasets it can get even worse. For a sample of n values, no point can ever be more than (n − 1) / √n SDs from the mean. With 5 values that limit is 1.79, so a 3-SD rule can't flag anything at all, no matter how extreme. We'll meet the SD-based "how many SDs away" score properly, as the z-score, on Day 10.

We need measures of spread that an outlier can't stretch. That's what the next two are for.

Median absolute deviation (MAD)

MAD is a robust stand-in for the standard deviation. Instead of measuring distances from the mean and averaging their squares, it measures distances from the median and takes their median. That double swap, median instead of mean at both steps, is what makes it resistant to extreme values. We used it briefly in Day 1's salary example; here's the full recipe.

How to calculate MAD

  1. Find the median of the data.
  2. Find each value's absolute deviation from that median (the distance, ignoring the sign).
  3. Find the median of those absolute deviations. That's the MAD.

Worked example

Six test scores: [94, 85, 78, 92, 100, 96]

Step 1. Sorted, they're [78, 85, 92, 94, 96, 100]. With 6 values, the median is the average of the middle two: (92 + 94) / 2 = 93.

Step 2. Distances from 93, in the same sorted order: [15, 8, 1, 1, 3, 7].

Step 3. Sort those: [1, 1, 3, 7, 8, 15]. Their median is (3 + 7) / 2 = 5.

So the MAD is 5. Because it's a median, you can read it like this: half of the scores are within 5 points of the median score, and half are further away.

Now the real test. Suppose the 100 gets mistyped as 1000:

Pythonfrom scipy import stats scores = [94, 85, 78, 92, 100, 96] typo = [94, 85, 78, 92, 1000, 96] for name, x in [("Clean", scores), ("With typo", typo)]: print(f"{name:10} SD = {np.std(x, ddof=1):7.2f} MAD = {stats.median_abs_deviation(x)}")
OutputClean SD = 8.01 MAD = 5.0 With typo SD = 371.97 MAD = 5.5

One typo multiplied the SD by more than 46. The MAD moved from 5 to 5.5. That's what robust means.

Putting MAD on the same scale as SD

On the same data, MAD and SD are naturally different sizes, so you can't compare them directly. For data shaped like the classic bell curve, the SD is about 1.4826 times the MAD. Multiplying the MAD by 1.4826 gives a robust estimate of the SD. SciPy does this for you with scale="normal":

Pythonprint(stats.median_abs_deviation(scores, scale="normal")) print(stats.median_abs_deviation(typo, scale="normal"))
Output7.41301109252801 8.15431220178081

For the clean scores that's 7.41, close to the true SD of 8.01, and with the typo it barely moves, to 8.15. The ordinary SD jumped to 371.97.

Name clash: "MAD" can mean two things

Some books and tools use MAD for the mean absolute deviation (the average distance from the mean, the 10.33 we computed earlier), which is not robust. pandas used to have a .mad() method that computed exactly that. It was deprecated in pandas 1.5 and no longer exists in pandas 2. For the robust median version, use scipy.stats.median_abs_deviation, and always say which MAD you mean.

MAD for spotting outliers

The modified z-score, recommended by Iglewicz and Hoaglin and described in the NIST/SEMATECH Engineering Statistics Handbook, swaps the mean and SD for the median and MAD:

Mi = 0.6745 × (xi − median)MAD
Values with |Mi| greater than 3.5 are labelled potential outliers. The 0.6745 is 1 / 1.4826, the same bell-curve scaling as above.

Let's give it the salary list that fooled the 3-SD rule:

Pythonmed = np.median(salaries) mad = stats.median_abs_deviation(salaries) m = 0.6745 * (300 - med) / mad print(f"Median: {med} MAD: {mad}") print(f"Modified z-score of $300k: {m:.1f}")
OutputMedian: 74.25 MAD: 3.5 Modified z-score of $300k: 43.5

A score of 43.5 against a threshold of 3.5. The $300k salary can't drag the median or the MAD, so it can't hide.

Interquartile range (IQR)

The IQR is the other robust measure of spread, and the one you'll see most often. It looks only at the middle 50% of the data and ignores the top and bottom 25%, which is exactly where outliers live.

Quartiles first

Quartiles are the three values that cut sorted data into four equal-sized chunks:

QuartileWhat it isPercentile
Q1 (lower quartile)A quarter of the data sits below it25th
Q2 (the median)Half the data sits below it50th
Q3 (upper quartile)Three quarters of the data sits below it75th
IQR = Q3 − Q1

Worked example by hand

17 students' test scores: [98, 94, 94, 62, 73, 87, 77, 68, 72, 85, 99, 100, 86, 60, 52, 100, 80]

Step 1, sort: [52, 60, 62, 68, 72, 73, 77, 80, 85, 86, 87, 94, 94, 98, 99, 100, 100]

Step 2, find the median (Q2): with 17 values, the middle one is the 9th, so Q2 = 85.

Step 3, find Q1 and Q3: one common hand method takes the median of each half, leaving out the overall median. The lower half is [52, 60, 62, 68, 72, 73, 77, 80], so Q1 = (68 + 72) / 2 = 70. The upper half is [86, 87, 94, 94, 98, 99, 100, 100], so Q3 = (94 + 98) / 2 = 96.

Step 4, IQR: 96 − 70 = 26.

So the middle half of the class scored within a 26-point band. The lowest score (52) and the highest (100) have no say in that number at all.

Now the same thing in NumPy

Pythonscores17 = [52, 60, 62, 68, 72, 73, 77, 80, 85, 86, 87, 94, 94, 98, 99, 100, 100] q1, q3 = np.percentile(scores17, [25, 75]) print(f"Q1={q1}, Q3={q3}, IQR={q3 - q1}")
OutputQ1=72.0, Q3=94.0, IQR=22.0

NumPy says the IQR is 22, not 26. Neither answer is wrong. There is no single agreed definition of a quartile, as we mentioned on Day 1. When the 25% mark falls between two data points, different methods settle it differently. By default, NumPy and pandas count positions in the sorted list from 0 and put the 25% mark at position 0.25 × (n − 1), interpolating if it lands between two values. With 17 values that's position 4, which holds the value 72; the 75% mark is position 12, which holds 94. The hand method instead takes the middle of each half. On large datasets the methods give nearly the same answer; on 17 values the gap is visible.

NumPy can use other methods too. For this particular dataset, method="weibull" happens to give 70 and 96, matching the hand method:

Pythonprint(np.percentile(scores17, [25, 75], method="weibull"))
Output[70. 96.]

That match is a coincidence of this dataset, not a general rule. The practical lesson: before comparing an IQR from one tool with an IQR from another, check that they used the same quartile method. A mismatch usually means two different rulers, not a mistake.

The IQR fence rule for outliers

The IQR's most common job is the "fence" rule, popularised by the statistician John Tukey along with the box plot (he described both in his 1977 book Exploratory Data Analysis):

Lower fence = Q1 − 1.5 × IQR
Upper fence = Q3 + 1.5 × IQR
Points beyond either fence are flagged as potential outliers.

With the hand-calculated quartiles (Q1 = 70, Q3 = 96, IQR = 26):

Lower fence = 70 − 1.5 × 26 = 70 − 39 = 31
Upper fence = 96 + 1.5 × 26 = 96 + 39 = 135

No score is below 31 or above 135, so nothing gets flagged. (With NumPy's quartiles the fences are 39 and 127, and nothing gets flagged either.)

lower fence 31 upper fence 135 Q1 = 70 Q3 = 96 median = 85 52 100 IQR = 26 30 50 70 90 110 130 lower fence 31 upper fence 135 Q1 = 70 Q3 = 96 median = 85 52 100 IQR = 26 30 50 70 90 110 130
A box plot of the 17 scores, using the hand-calculated quartiles, with each score shown as a dot underneath. The box spans the middle 50%, the line inside is the median, and the whiskers reach the most extreme scores still inside the fences. The dashed fences are drawn here to teach the rule; real box plots usually leave them out.

This is the rule behind the whiskers and outlier dots in a standard box plot. Matplotlib uses 1.5 × IQR by default, as do most plotting libraries, and it computes the quartiles the way NumPy does. So a box plot drawn by Matplotlib would show a box from 72 to 94 here, not 70 to 96.

Now our salary list, the one that slipped past the 3-SD rule:

Pythonq1, q3 = np.percentile(salaries, [25, 75]) iqr = q3 - q1 upper = q3 + 1.5 * iqr print(f"Q1={q1:.2f} Q3={q3:.2f} IQR={iqr:.2f}") print(f"Upper fence: {upper:.2f}") print("Flagged:", salaries[salaries > upper])
OutputQ1=72.25 Q3=78.22 IQR=5.97 Upper fence: 87.19 Flagged: [300.]

The fence sits at about $87k, so the $300k salary is flagged immediately. Here's the scorecard for the three outlier checks we've run on this list:

MethodBuilt onScore for $300kFlagged?
More than 3 SDs from the meanMean and SD2.84 SDsNo, masked by the outlier
Modified z-score above 3.5Median and MAD43.5Yes
Beyond the IQR fenceQuartiles$300k vs. a fence at about $87kYes
"Flagged" doesn't mean "delete"

These rules find values worth a second look, not values to throw away. The $300k might be a typo, or it might be the team's director. Check where it came from before you decide. Day 3's 500 cm height was clearly an error; a large salary might be perfectly real.

Spread and centre belong together

A measure of spread means most when it's reported next to a measure of centre. Here are two companies, both with a mean salary of exactly $96,000:

both average $96k Company A SD ≈ $169,941 Company B SD ≈ $4,472 four people at $20k one at $400k everyone between $90k and $102k $0k $100k $200k $300k $400k both average $96k Company A SD ≈ $169,941 Company B SD ≈ $4,472 four people at $20k one at $400k everyone between $90k and $102k $0k $100k $200k $300k $400k
Same mean, very different reality. Company A's average comes from one $400k salary; Company B's average describes its typical employee. The SDs shown are sample SDs.

Report only the mean and these companies look identical. Add the SD and the difference jumps out: about $170k for Company A against about $4.5k for B. Notice something else, though. For Company A, "mean ± SD" would be $96k ± $170k, a range that starts below zero. That's the SD telling you that the mean is the wrong summary for this data in the first place. The data is badly skewed, and the median ($20k), which matches four of the five employees, tells the real story.

That gives a simple rule:

Choosing a measure of spread

MeasureUses every value?UnitsRobust to outliers?Best for
RangeNo, only twoOriginalNoA quick first look
VarianceYesSquaredNoMaths and models behind the scenes
Standard deviationYesOriginalNoClean, roughly symmetric data
MADYes, via two mediansOriginalYesMessy data, outlier detection
IQRNo, only the middle 50%OriginalYesSkewed data, box plots, fences

Key takeaway

Every time you report a centre, report a spread right next to it. A mean or median on its own is an incomplete summary, and it can easily be a misleading one: a cockpit built for the average pilot fits nobody.

Quick check

Five questions to test whether this stuck. The answers are one click away.

Two datasets have the same range of $50,000. Do they have the same amount of spread?

Not necessarily. Range only looks at the maximum and the minimum. In one dataset, everything in between could be tightly bunched; in the other, spread evenly across the whole gap. You'd need the SD, MAD or IQR to tell them apart.

You compute a standard deviation and get a negative number. What went wrong?

Something in the calculation, because an SD can never be negative. It comes from squared deviations (zero or positive) and a square root (which never returns a negative number). An SD of exactly 0 is possible, and means every value is identical.

A colleague runs np.std(x) and you run x.std() on the same pandas column. Your answers differ slightly. Who's right?

You're computing different things. NumPy divides by n by default (the population formula); pandas divides by n − 1 (the sample formula). For sample data the n − 1 version is the standard choice, so pass ddof=1 to NumPy. The gap is biggest for small datasets.

A $300k salary in a list of $70k to $80k salaries is only 2.84 SDs above the mean. Why isn't it further?

Because the outlier itself inflated both the mean and the SD, the ruler used to measure it. This is called masking. Robust methods built on the median, such as the modified z-score (MAD) or the IQR fences, flag it easily.

Why does MAD use the median twice, instead of the mean at either step?

Because the whole point of MAD is to resist outliers, and the mean is exactly the number outliers drag around. Using the median for the centre and for summarising the distances means one wild value can barely move the answer, as the typo test showed: the SD jumped from 8.01 to 371.97 while the MAD went from 5 to 5.5.

Next up, Day 5 Relative Measures of Dispersion

An SD of 10 is huge for exam scores out of 20 and tiny for house prices. Next time: how to compare spread across datasets that aren't even on the same scale.