Until now we've described one variable at a time: its centre on Day 3, its spread on Days 4 and 5. But most interesting questions in data science are about pairs. Does more advertising go with more sales? Do longer delivery times go with more cancellations? Today is about measuring whether two variables move together, and how much to trust that measurement.
Why a data scientist cares
Imagine you work at a streaming service and your manager says: "Users who watch more than three hours a day seem to cancel faster. Can you look into it?" Your first step is to check whether hours watched and cancellations are related at all, in which direction, and how strongly. That's what correlation measures.
Correlation is one of the first things you check on any new dataset. It helps you:
- See which features move together with the thing you're trying to predict.
- Spot redundant features: two columns that are very highly correlated carry much of the same information.
- Catch multicollinearity (inputs that are strongly correlated with each other) before it makes a regression model unstable.
- Find patterns worth investigating further. Note the word "investigating": as we'll see, a correlation is a lead, not a conclusion.
What is correlation?
A correlation coefficient is a single number that describes how two numeric variables move together. The most common one is written r. It always falls between −1 and +1, and it tells you two things at once:
- Direction, from its sign. Positive means the two tend to rise together; negative means one tends to fall as the other rises.
- Strength, from its size. Close to ±1 means the points hug a straight line; close to 0 means there's no straight-line pattern.
Some everyday examples. Positive: taller people tend to weigh more; hotter days tend to bring more ice cream sales. Negative: the colder the winter, the higher the heating bill; when a product's price goes up, demand usually goes down. Close to zero: a person's height and the last digit of their phone number.
How strong is "strong"?
The most widely quoted rule of thumb comes from the statistician Jacob Cohen, in his 1988 book on statistical power. He suggested these rough labels for the size of r, ignoring its sign:
| |r| around | Cohen's label |
|---|---|
| 0.10 | Small |
| 0.30 | Medium |
| 0.50 and above | Large |
Cohen himself presented these as rough conventions, and what counts as strong depends on the field. In psychology, an r of 0.3 between two human traits can be a meaningful finding. In physics or engineering, two instruments measuring the same thing should agree at 0.99 or better. Always read a correlation in the context it came from.
One more helpful translation: r², the square of r, is the share of one variable's variation that a straight line through the other can account for. An r of 0.5 sounds like "halfway there", but r² is only 0.25: a straight line on one variable explains a quarter of the other's variation. That's a useful check on how impressive a correlation really is.
Covariance: the building block
Correlation is built on top of covariance, so let's start there. Covariance asks: when one variable is above its average, is the other one usually above its average too?
Here's the recipe. For each data point, measure how far X is from its mean and how far Y is from its mean, and multiply the two distances together:
- If both are above their means, the product is positive (positive × positive).
- If both are below their means, the product is also positive (negative × negative).
- If one is above and the other below, the product is negative.
Average those products and you get the covariance. If most points land in the "both above" or "both below" groups, it comes out positive. If most land in the mixed groups, it comes out negative. If they're balanced, it's near zero.
A real example: Palmer penguins
The Palmer penguins dataset holds measurements of 344 penguins from three species, collected by Dr. Kristen Gorman at Palmer Station, Antarctica, between 2007 and 2009. It's free and loads straight into seaborn. After dropping two penguins with missing measurements, we have 342:
The covariance is positive, as the picture suggests: longer flippers go with heavier penguins. But look at what happened when we changed the units. The same penguins, the same relationship, and the covariance dropped from 9,824.4 to 9.82, a thousand times smaller, just because we wrote the mass in kilograms instead of grams.
That's covariance's big weakness, and the same problem Day 5 solved for spread: its size depends on the units, so on its own it can't tell you whether a relationship is strong or weak. The fix is also similar. Divide by something that carries the same units, and they cancel.
Pearson's correlation coefficient (r)
Pearson's r divides the covariance by the standard deviations of both variables:
Think of it as covariance measured in standard deviations instead of in grams or millimetres. Change the units of either variable and r doesn't move:
An r of 0.87 is a very strong positive correlation by anyone's standard: heavier penguins really do tend to have longer flippers. (In pandas, penguins[["flipper_length_mm", "body_mass_g"]].corr() gives the same number.)
What r = 1, 0 and −1 mean
- r = +1: every point lies exactly on a rising straight line.
- r = −1: every point lies exactly on a falling straight line.
- r = 0: no straight-line relationship. That last word matters a lot, as you're about to see.
What Pearson's r assumes
- A straight-line relationship. r only measures how well a straight line fits. A strong curved pattern can still give a small r.
- No extreme outliers. Because it's built from means and standard deviations, a single far-off point can drag it around (Day 4's problem again).
- Numeric data on an interval or ratio scale (Day 2), so that equal differences mean equal amounts.
You'll often read that both variables must be normally distributed to use Pearson's r. As a description of how well a straight line fits, r doesn't need normality at all. Normality matters when you go further and compute a p-value or confidence interval for r, especially in small samples. We'll cover p-values properly on Day 12.
Weakness 1: it's blind to curves
Here's the cleanest possible example. Y is exactly X squared, a perfect relationship with no randomness at all:
Y is completely determined by X, yet r is 0. The left half of the U slopes down, the right half slopes up, and the two cancel out. So r = 0 does not mean "unrelated". It means "no straight-line relationship".
A famous demonstration: Anscombe's quartet
In 1973 the statistician Francis Anscombe built and published four small datasets in The American Statistician to make exactly this point. They're the four panels at the top of this page, and they come with seaborn:
The numbers are almost identical, but the pictures tell four different stories. Dataset I is a genuine, noisy straight line. Dataset II is a smooth curve that a straight line describes badly. Dataset III is a perfect straight line knocked off course by one outlier. In Dataset IV there's no relationship at all among ten of the points, which all share the same x; a single far-off point creates the entire correlation. Half a century later, his lesson still holds: always plot your data before you trust a correlation.
Weakness 2: it's sensitive to outliers
Eight points that rise steadily together, then one extra point far away from the rest:
One point turned a strong positive relationship into a negative one. Before you report an r, look for points like this one and find out whether they're errors or real.
Spearman's rank correlation (ρ)
Spearman's rank correlation, written ρ ("rho"), makes one change to Pearson's: it replaces every value with its rank (1 for the smallest, 2 for the next, and so on), then computes Pearson's r on those ranks. That one change means:
- It measures monotonic relationships, not just straight lines (explained below).
- It's less sensitive to outliers. An extreme value can only ever be ranked first or last, however extreme it is.
- It works for ordinal data such as ratings, rankings and satisfaction levels (Day 2), where only the order of the values means anything.
Monotonic relationships
A relationship is monotonic if, as X goes up, Y always moves the same way: it keeps rising (or at least never falls), or keeps falling (or at least never rises). It doesn't have to rise at a steady rate. Think of driving up a winding mountain road: the slope keeps changing, but you never lose height. That's monotonic.
Notice the right-hand panel. Spearman is better at curves, but only at curves that never turn around. A U-shaped relationship defeats both measures. Only a plot will show it to you.
How to compute Spearman's ρ by hand
- Rank the X values from smallest to largest. If two values tie, give each the average of the ranks they would share.
- Rank the Y values the same way.
- Compute Pearson's r on the two lists of ranks.
When there are no ties, step 3 has a shortcut:
Worked example
Five students, the hours they studied, and their exam scores:
| Student | Hours | Score | Rank of hours | Rank of score | d | d² |
|---|---|---|---|---|---|---|
| A | 2 | 52 | 1 | 1 | 0 | 0 |
| B | 4 | 58 | 2 | 3 | −1 | 1 |
| C | 5 | 55 | 3 | 2 | +1 | 1 |
| D | 7 | 70 | 4 | 4 | 0 | 0 |
| E | 9 | 88 | 5 | 5 | 0 | 0 |
| Total | 2 |
Only students B and C are out of order (B studied more but scored a little less), so ρ is a little below 1. SciPy agrees:
And on the outlier example from earlier:
Spearman's ρ stays positive where Pearson's r flipped sign, but it still falls a long way, from 0.95 to 0.37. "Less sensitive to outliers" is not the same as "immune". The extra point has the largest X and the smallest Y, which scrambles the ranks too. Plot first, always.
In April 2017, Netflix replaced its five-star rating system with a thumbs up or thumbs down. The company said that in testing, thumbs produced about 200% more ratings than stars, and that many members were confused about what the stars did. Its reasons were about engagement and clarity, not statistics, but the change touches a statistical point worth knowing. Star ratings are ordinal: nothing guarantees that the jump from 3 to 4 stars means the same as the jump from 1 to 2. If you want to correlate star ratings with something else, Spearman (or Kendall, next) is the safer choice, because they only use the order of the ratings.
Kendall's tau (τ)
Kendall's tau is another rank-based correlation, with a different way of counting. It looks at every possible pair of observations and asks a simple question: do X and Y put these two in the same order?
- If the observation with the larger X also has the larger Y, the pair is concordant (they agree).
- If the one with the larger X has the smaller Y, the pair is discordant (they disagree).
This gives tau a lovely plain-language reading. With no ties, a tau of 0.6 means that 80% of all pairs agree on the order and 20% disagree (80% − 20% = 60%). Pearson and Spearman have no interpretation that simple.
Worked example
Two judges rank the same five contestants (1 = best):
Five contestants make 10 possible pairs. The judges disagree on only two of them: the order of the first two contestants, and the order of the last two. That's 8 concordant and 2 discordant pairs, so τ = (8 − 2) / 10 = 0.6. The p-value of 0.23 is a reminder that with only five data points, even a decent-looking agreement could easily happen by chance.
Kendall's tau is often preferred when samples are small or contain many tied values, and whenever you're comparing two rankings, because "the share of pairs that agree" is exactly the question you're asking.
Search engines are evaluated by having people judge which results are relevant. That raises a worry: different judges disagree, so would different judges crown a different winner? In a 2000 study, Ellen Voorhees of the US National Institute of Standards and Technology re-ran the evaluation of the systems in the TREC search competitions with different sets of human judges and used Kendall's tau to compare the resulting rankings of the systems. The rankings stayed highly correlated, at around 0.9, so the competition's conclusions held up even though individual judges disagreed. Comparing rankings with Kendall's tau has been widely used in that field ever since.
Choosing the right coefficient
| Your situation | Use |
|---|---|
| Two numeric variables, and the scatter plot looks like a straight-line cloud with no wild outliers | Pearson's r |
| The relationship is curved but always moves one way, or there are outliers | Spearman's ρ |
| Ordinal data such as ratings or rankings | Spearman's ρ or Kendall's τ |
| A small sample, many tied values, or comparing two rankings | Kendall's τ |
| The plot shows a U shape or another pattern that changes direction | None of them. Describe the pattern you see, or model it directly |
Whichever you choose, the first step never changes: make a scatter plot and look at it.
The most important lesson: correlation is not causation
This may be the most important idea in all of statistics. Two variables moving together does not mean one causes the other. There are three common reasons a correlation can appear without one variable causing the other.
1. A third variable drives both
The classic textbook illustration: ice cream sales and shark attacks rise and fall together. Ice cream doesn't attract sharks. Hot weather drives both: more people buy ice cream and more people swim in the sea. A hidden variable like hot weather is called a confounding variable (or a lurking variable).
For years, a large body of observational studies found that women who took hormone therapy after menopause had a 40% to 50% lower risk of coronary heart disease, so hormone therapy was widely expected to protect the heart. Then came a randomised trial, the Women's Health Initiative, which assigned 16,608 women to hormone therapy or a placebo by chance. Published in JAMA in July 2002, after the trial was stopped early, it found no heart protection at all: the hazard ratio for coronary heart disease was 1.29, meaning risk was slightly higher in the hormone group.
The leading explanation is a "healthy user" effect: women who chose hormone therapy in the observational studies tended to be healthier to begin with. Their health caused both the therapy use and the lower heart risk. The correlation was real; the causal story was wrong. Randomisation breaks the link with hidden variables like these, which is why it's the gold standard, and the core idea behind the A/B tests on Day 13.
2. Pure coincidence
Search through enough pairs of variables and some will line up by chance. The writer Tyler Vigen made this famous with his Spurious Correlations project. One example: from 1999 to 2009, the number of films Nicolas Cage appeared in each year tracks the number of people in the US who drowned by falling into a swimming pool.
An r of 0.67, and a p-value below 0.05, which is the usual threshold for calling a result "statistically significant". Yet nothing connects the two. These are yearly totals, so there isn't even a shared season to blame. It's a coincidence, found by searching through a huge number of possible pairs and keeping the ones that happen to match. Eleven data points make coincidences like this easy to find. The more pairs you test, the more of them you'll find.
3. The groups are hiding the real pattern
Sometimes a correlation in the whole dataset points one way while every group inside it points the other. This is called Simpson's paradox, and the penguins show it beautifully:
Looking at all penguins together, longer bills go with shallower bills (r = −0.24). Within each species, longer bills go with deeper bills, and fairly strongly. The overall trend is driven by differences between species: Gentoo penguins have long but shallow bills, Adelie penguins short but deep ones. Species is the confounding variable. If you'd reported the pooled correlation, you'd have described a relationship that doesn't hold for any actual kind of penguin.
So whenever you see a correlation, ask three questions. Could a third variable be driving both? Could it be chance, especially if I tested many pairs? Is there a grouping in my data that I've ignored? And remember that the cause can also run the opposite way from the one you expect.
Correlation matrices and feature selection
In practice you'll rarely look at one pair at a time. A correlation matrix shows the correlation between every pair of columns at once, and a heatmap makes it easy to scan. Let's use another real dataset: the diabetes data that ships with scikit-learn. It has 442 patients, ten baseline measurements (age, sex, body mass index, blood pressure and six blood tests, called s1 to s6), and a measure of how much their disease progressed a year later. It comes from a well-known 2004 statistics paper by Efron, Hastie, Johnstone and Tibshirani.
Body mass index (bmi) and s5 (a blood fat measurement, likely the log of serum triglycerides) have the strongest correlations with disease progression, both around 0.57 to 0.59. Sex barely correlates at all. Now the matrix of correlations between the features themselves:
To draw a heatmap like this yourself, you'd use sns.heatmap(corr, annot=True, cmap="coolwarm", vmin=-1, vmax=1).
The standout is s1 and s2, at 0.90. That makes sense once you know what they are: s1 is total cholesterol and s2 is LDL cholesterol, which is one of the parts that make up total cholesterol. They carry much of the same information. Putting both into a plain linear regression invites multicollinearity: the model can't tell which of the two deserves the credit, so their coefficients become unstable and hard to interpret. Common fixes are to keep just one of them, combine them, or use a regularised model that copes with correlated inputs.
First, a low correlation with the target doesn't make a feature useless. Correlation only measures one-variable, straight-line (or, for Spearman, monotonic) relationships. A feature can matter through a curve, or in combination with other features. Second, a high correlation doesn't make a feature a cause, for every reason in the previous section. Use correlations to guide where you look, not to make the final decision.
Key takeaway
Correlation is one of the most useful tools in data science and one of the most misused. Used well, it points you toward relationships worth understanding. Used carelessly, it produces confident-sounding numbers that describe nothing real.
- Covariance gives the direction of a relationship, but its size depends on the units.
- Pearson's r standardises it to −1 to +1, but only measures straight-line relationships and is easily swayed by outliers.
- Spearman's ρ and Kendall's τ work on ranks, so they handle monotonic curves, ordinal data and outliers better.
- Always plot first. Anscombe's four datasets share one r and tell four stories.
- Correlation is not causation. Check for confounders, chance and hidden groups before you tell anyone that X drives Y.
Quick check
Five questions to test whether this stuck. The answers are one click away.
Two variables have a Pearson r of 0.05. Can you conclude they're unrelated?
No. Pearson only detects straight-line relationships. A perfect U-shaped relationship, like y = x² from −5 to 5, gives an r of exactly 0. Plot the data before ruling anything out.
You want to correlate customer ratings (1 to 5 stars) with delivery time in minutes. Which coefficient fits best?
Spearman's ρ or Kendall's τ. Star ratings are ordinal, so the gaps between them aren't guaranteed to be equal, and both of these only use the order of the values. With only five possible ratings there will be lots of ties, which Kendall's tau-b handles well.
You change a column from grams to kilograms. What happens to its covariance and its correlation with another column?
The covariance shrinks by a factor of 1,000, as the penguin example showed (9,824.4 to 9.82). The correlation doesn't change at all, because Pearson's r divides out the units.
Across all penguins, bill length and bill depth have r = −0.24, but within each species r is positive. Which should you report?
The within-species correlations, or better, both together with the explanation. The pooled number is driven by differences between species, not by how bills grow within any real species. This is Simpson's paradox, and species is the confounding variable.
Two features in your dataset correlate at 0.90 with each other. What should you consider before fitting a linear regression?
Multicollinearity. With two such similar inputs, the model can't separate their effects, so their coefficients become unstable. Consider keeping one, combining them, or using a regularised model, as with s1 and s2 in the diabetes data.
That wraps up descriptive statistics. Part II begins with probability: the language for reasoning about chance, and the foundation for everything from p-values to A/B tests.