The Foundation

How data scientists actually think about numbers, starting with why one salary out of ten can make an "average" lie to you.

0 50K 100K 150K 200K one salary, far from the rest 0 50K 100K 150K 200K one salary, far from the rest
Ten salaries from one startup, in thousands of dollars a year. What does a typical employee here earn? Hold that thought.

Welcome to Day 1. If you want to build real data science and analytics skills, statistics is where that has to start. Not because it's a box to tick, but because almost nothing else in this field works without it.

The series gets more advanced as it goes. Whether you're completely new or brushing up, the goal for every post is the same: by the end, you should be able to use the idea, not just recognise the word. Let's get into it.

Why statistics?

Statistics is the backbone of data science. Before you build a model, make a prediction, or draw a business conclusion, you need to understand your data, and statistics is the toolkit for doing that.

Think of statistics as the language your data speaks. Without it, you're staring at a spreadsheet full of numbers. With it, you can find patterns, measure how uncertain you should be, and make decisions with real confidence behind them. Every major technique in data science, from linear regression to neural networks, sits on a statistical foundation.

A real-world comparison

A doctor doesn't examine every cell in your body to check your health. They look at a handful of key indicators (blood pressure, cholesterol, temperature) and use those to understand the full picture. Data scientists do the same thing with data: a few well-chosen summary numbers tell you far more than staring at every raw row ever will.

The two branches of statistics

Statistics splits into two main branches, and almost every technique in this series falls into one of them.

Descriptive statisticsInferential statistics
GoalSummarise and describe the data you already haveDraw conclusions about a population from a sample
ScopeThe dataset in handA broader, unseen population
OutputCharts, averages, distributionsEstimates with a margin of error, hypothesis test results, confidence intervals
Example"The average exam score in this class is 74.""Students nationally likely score between 70 and 78."
Descriptive What does this data show? mean, median, std dev histograms, box plots Inferential What can we conclude beyond it? hypothesis tests confidence intervals then Descriptive What does this data show? mean, median, std dev histograms, box plots then Inferential What can we conclude beyond it? hypothesis tests confidence intervals

You'll almost always use them in that order: descriptive first, to understand what you're looking at, then inferential, to go beyond it. Skip the first step and the second one is built on sand.

Descriptive statistics

Descriptive statistics is about making raw data understandable. It helps you collect, organise, summarise, and present data so its key features are visible at a glance. Instead of getting lost in thousands of rows, you see the shape of the thing.

The four tools that do most of the work

Central tendency

Mean, median and mode. Where the "middle" of your data sits.

Dispersion

Range, variance, standard deviation, IQR and MAD. How spread out the data is, and how much to trust that middle value.

Frequency distributions

How often each value, or range of values, shows up.

Graphs and tables

Histograms, box plots, bar charts and scatter plots, which turn all of the above into something you can see.

None of these tools mean much on their own. The real skill is knowing which combination answers your actual question, which is exactly what the next example is about.

A concrete example

Here are the salaries from the top of the page: ten employees at a startup, in thousands of US dollars a year.

[40K, 45K, 48K, 50K, 52K, 55K, 60K, 65K, 70K, 200K]

Pythonimport pandas as pd salaries = pd.Series([40, 45, 48, 50, 52, 55, 60, 65, 70, 200]) print("Mean: ", salaries.mean()) print("Median:", salaries.median()) # pandas uses the sample std dev (ddof=1) by default print("Std dev (sample):", round(salaries.std(), 2)) print("Variance (sample):", round(salaries.var(), 2)) print("Range: ", salaries.max() - salaries.min()) q1, q3 = salaries.quantile([0.25, 0.75]) print("IQR: ", round(q3 - q1, 2)) mad = (salaries - salaries.median()).abs().median() print("MAD: ", mad)
OutputMean: 68.5 Median: 53.5 Std dev (sample): 47.09 Variance (sample): 2217.83 Range: 160 IQR: 15.25 MAD: 7.5

The mean says the "typical" employee earns about $68.5K. The median says $53.5K. That's a $15K gap between two numbers that are both supposed to describe the same "average" employee, and it comes entirely from one person on the list.

0 50K 100K 150K 200K Median = 53.5K Mean = 68.5K the $200K outlier 9 of 10 salaries sit in here 0 50K 100K 150K 200K Median = 53.5K Mean = 68.5K the $200K outlier 9 of 10 salaries sit in here
The median stays inside the cluster. The mean gets dragged toward the one extreme value.

The $200K salary is an outlier: a value far removed from the rest of the data. The mean uses every value in its calculation, so that one number drags it upward. The median only cares about the middle position in a sorted list, so it barely notices. This is why the median is usually a better "typical value" for skewed data like income, house prices, or app session times, where a few extreme values sit alongside a lot of ordinary ones.

The standard deviation makes the distortion even clearer. Above, it comes out to about $47K, which would normally suggest salaries are spread wildly, but nine of the ten people are within $30K of each other. Drop the outlier and recompute:

Python# drop the last value, the 200K salary trimmed = salaries[:-1] print("Mean: ", round(trimmed.mean(), 2)) print("Median:", trimmed.median()) print("Std dev:", round(trimmed.std(), 2))
OutputMean: 53.89 Median: 52.0 Std dev: 9.66

One data point took the standard deviation from $47K down to under $10K. That's the whole lesson in one number: a single outlier can dominate the mean and the standard deviation while leaving the median almost untouched. A good data scientist checks the middle and the spread together, and looks at the actual data before trusting either one.

Now look at the IQR ($15.25K) and the MAD ($7.5K). Both stayed small without anyone removing the outlier. The IQR only looks at the middle 50% of the data (the gap between the first quartile, Q1, and the third quartile, Q3), so a value sitting far outside that range barely moves it. A box plot shows exactly this:

0 50K 100K 150K 200K Q1 = 48.5K Q3 = 63.75K Median = 53.5K IQR = 15.25K 200K, drawn on its own 0 50K 100K 150K 200K Q1 = 48.5K Q3 = 63.75K Median = 53.5K IQR = 15.25K 200K, on its own
The box is the IQR: the middle half of the salaries. The $200K value sits far outside it, so the box doesn't move. Day 4 covers the exact rule box plots use to flag an outlier.

The MAD (median absolute deviation) works the same way: it uses the median as the centre, measures how far every value sits from it, and then takes the median of those distances. Both the IQR and the MAD are built to ignore exactly the kind of extreme value that inflates the mean and the standard deviation, which is why they're often better choices for skewed, real-world data.

A quirk worth knowing

Quartiles don't have one universal definition. Pandas and NumPy use linear interpolation by default, which is why the code above gives Q1 = 48.5K and Q3 = 63.75K. Split the same data into a lower and an upper half and take the median of each instead, a method plenty of introductory courses teach, and you get Q1 = 48K and Q3 = 65K, for an IQR of 17K instead of 15.25K. Neither answer is wrong. They're different conventions, so don't be alarmed if your textbook and your code disagree slightly on quartiles.

Why does pandas give a different std dev than you might expect?

There are two versions of standard deviation. The population version divides by n (the count of values). The sample version divides by n−1. Why the smaller number? The spread is measured around the sample's own mean, and a sample's points always sit a little closer to their own mean than to the true population mean. So dividing by n comes out slightly too small on average, and dividing by n−1 corrects for it. This fix is called Bessel's correction, and pandas uses the sample version by default.

On our 10 salaries, the population std dev is $44.68K and the sample std dev is $47.09K: close, but not identical. The difference matters the moment you move from "describing this dataset" to "estimating something about a bigger population," which is exactly where inferential statistics picks up.

Frequency distributions and graphs

The third tool, frequency distributions, is just counting how often each value shows up. Say 20 users rated an app between 1 and 5 stars:

RatingCountShare
1 star00%
2 stars15%
3 stars420%
4 stars630%
5 stars945%
0 1 star 1 2 stars 4 3 stars 6 4 stars 9 5 stars mode: the most frequent rating number of users giving each rating 0 1 star 1 2 stars 4 3 stars 6 4 stars 9 5 stars mode: most frequent users giving each rating

As a table, this is a frequency distribution. As a bar chart, it's instantly readable: most people are happy, a handful are lukewarm, and almost nobody is unhappy. This is also where the mode comes from: the most frequent value in the data, 5 stars here. The mean and median describe a distribution's centre, while the mode describes its peak. For categorical or heavily repeated data like ratings, the mode is often the most useful of the three.

Why descriptive statistics matter in data science

This isn't theory you tick off before the "real" modelling starts. Descriptive statistics is where most of a data scientist's actual time goes.

In practice, this usually starts with one line of code. Running salaries.describe() on the series above gives you the count, mean, std dev, min, max, and quartiles in a single call, which is why .describe() is so often the very first thing a data scientist runs on a new dataset.

Common pitfall

Reporting the mean alone, without checking the spread or plotting the data first, is one of the most common mistakes new data scientists make. If you'd only seen "average salary: $68.5K" without the raw numbers, you'd have walked away with a genuinely wrong idea of what this startup pays most of its people.


Inferential statistics

Descriptive statistics tells you about the data you have. Inferential statistics lets you go further and reason about data you don't have.

The population is the entire group you're actually interested in. The sample is the smaller, manageable subset of that group you actually collect data from. By carefully analysing a sample, you make educated, mathematically grounded guesses about the much larger population. That's essential whenever collecting data from everyone is impossible or just impractical.

Two more words come with this. A number that describes a whole population is a parameter: for example, the true average salary of every data scientist in India. The same kind of number calculated from a sample is a statistic. You almost never get to see a parameter directly. You estimate it from a statistic, and inferential statistics is the set of methods for doing that honestly.

the full group you care about Population draws a sample Sample what you actually measure analyse and infer Conclusions about the population Population everyone you care about draws a sample Sample what you measure analyse and infer Conclusions about the population
A bigger sample isn't automatically a better one

You can't survey every voter before an election, or test every light bulb a factory makes. Inference lets you draw reliable conclusions from a manageable sample instead, but only if that sample actually looks like the population.

In 1936, a magazine called The Literary Digest mailed about 10 million ballots, got roughly 2.4 million back, and predicted that Alf Landon would beat Franklin Roosevelt for the US presidency. Roosevelt won about 61% of the vote and 46 of the 48 states. The magazine's mailing lists leaned toward wealthier households, and the people who bothered to reply leaned even further against Roosevelt. Meanwhile George Gallup, polling only about 50,000 people chosen with more care, called the result correctly. A huge biased sample lost to a small representative one.

Key techniques

A business example

Say an e-commerce company tests a new product recommendation algorithm. They show it to a randomly selected group of users for a month and measure purchase rates. Using inferential statistics, specifically a hypothesis test, they decide whether the improvement in that group is statistically significant: likely to hold across all their users, rather than just random chance.

A clinical trial works the same way. A new medication is tested on 200 patients, and inferential statistics decides whether the observed benefit is strong and consistent enough to conclude the medication works for patients broadly, not just the 200 in the trial.

A first taste of the math

Hypothesis tests and confidence intervals get built up properly later in the series, but it's worth seeing the actual computation once, even in miniature. Say you survey 1,000 people about their monthly spend on a subscription app. The sample gives you a mean of $42 and a standard deviation of $15. How confident can you be about the true average across everyone, not just your 1,000 respondents?

Pythonfrom scipy import stats import numpy as np n, mean, std = 1000, 42, 15 standard_error = std / np.sqrt(n) ci_low, ci_high = stats.norm.interval(0.95, loc=mean, scale=standard_error) print(f"95% CI: (${ci_low:.2f}, ${ci_high:.2f})")
Output95% CI: ($41.07, $42.93)

The only new idea in that code is standard_error. It measures how much the sample mean would wobble if you repeated the survey with a fresh 1,000 people: the standard deviation divided by the square root of the sample size. Bigger samples wobble less, which is why this interval is so narrow. Day 11 builds the idea up properly.

And that's a confidence interval. It says: if you ran this survey over and over, 95% of the intervals you'd compute this way would contain the true population average. It does not mean there's a 95% chance the true value sits in this one specific range. That mix-up is common enough that confidence intervals get a full article of their own.

One more note: with a sample of 1,000, the normal distribution used above and the slightly more exact t-distribution give practically the same critical value, 1.96 versus 1.9623, so using the simpler normal version cost no real accuracy. The gap only starts to matter with small samples, which is where the t-distribution earns its keep.

Why inferential statistics matter in data science

ApplicationTechnique
Testing whether a new marketing strategy leads to higher salesHypothesis testing
Estimating a city's average income from a 1,000-person surveyConfidence intervals
Predicting housing prices from historical dataRegression analysis
Comparing test scores between two teaching methodsTwo-sample t-test

The role of probability

You can't fully understand inferential statistics without probability. It's the mathematical engine behind inference, giving you a principled way to reason about uncertainty.

When someone says a result is "statistically significant at the 5% level," that's a probability statement. When you build a 95% confidence interval, probability defines what that 95% means. When a spam filter scores an email, it's working with probabilities too. If 2 of every 100 emails it has seen were spam, it starts every new email at a 2% chance of being spam before reading a single word. That starting number is called a prior. The email's actual content (its words, links, and sender) then pushes that number up or down. Day 7 shows exactly how that update works, using Bayes' Theorem.

Probability underpins statistical inference (how confident should you be in a conclusion), risk assessment (how likely is a bad outcome), predictive modelling (what's the most likely future value), and anomaly detection (how unusual is this data point). It's a big enough topic that it gets its own part of the series, starting on Day 7.

Descriptive vs. inferential, side by side

ScenarioDescriptiveInferential
Student exam scores"The average score in Class A is 74, with a std dev of 8.""Is Class A's performance significantly better than Class B's?"
Product sales"Last month's sales peaked on weekends, with a median of $320 a day.""Will this weekend pattern hold next quarter?"
Healthcare"40% of patients in this trial reported side effects.""We estimate 35% to 45% of all patients on this drug experience side effects."
Customer survey"In our survey of 500 customers, 65% rated us 4 stars or higher.""We're 95% confident that between about 61% and 69% of all customers would rate us 4 stars or higher."
App engagement"Users in our sample opened the app 5.2 times a day on average.""Does a new notification design actually increase daily opens, or is the change just noise?"

Key takeaway

Statistics isn't about memorising formulas. It's about building intuition for data. Descriptive statistics helps you know your data deeply. Inferential statistics lets you make confident, honest claims about things you haven't directly observed, as long as your sample genuinely represents what you're asking about.

A habit for every project

Start with descriptive statistics, always. Compute summary measures, plot distributions, check for outliers and missing values. Only once you understand what you're working with should you move on to inference and model building. Skipping this step is one of the most common mistakes new data scientists make, and you've now seen exactly why, with one $200K salary as proof.

Quick check

Four questions to test whether this stuck. The answers are one click away.

Why did the mean and median disagree so much in the salary example?

Because the mean uses every value in its calculation, so the single $200K outlier pulls it upward. The median only depends on the middle position in a sorted list, so it's far less sensitive to extreme values.

You're handed a dataset and told "the average is 50." What's the first thing you should ask?

What's the spread, and what does the distribution actually look like? An average alone tells you very little: the same "50" could mean values tightly clustered around 50, or a skewed distribution being pulled toward one side, the same way the $68.5K average salary looked nothing like what most of the ten employees actually earned.

Descriptive or inferential: "Based on a poll of 1,200 voters, we estimate 51% to 57% of the country supports the policy."

Inferential. It uses a sample (1,200 voters) to make a claim about a much larger population (the country), with a range that accounts for uncertainty. A purely descriptive version would just report the poll's own number: "54% of our 1,200 respondents support the policy." Notice the range is centred on that 54%.

A news website's online poll gets 300,000 votes. Should you trust it more than a carefully chosen random sample of 1,000 people?

No. An online poll only hears from people who visit that site and choose to vote, and those people can differ a lot from the population as a whole. That's the same trap The Literary Digest fell into in 1936. A smaller sample that actually represents the population beats a huge one that doesn't.

Next up, Day 2 Types of Data and Scales of Measurement

Qualitative versus quantitative data, discrete versus continuous, and the four scales of measurement (nominal, ordinal, interval and ratio), plus why that label decides which statistical tools you're even allowed to use.