Welcome to Day 1. If you want to build real data science and analytics skills, statistics is where that has to start. Not because it's a box to tick, but because almost nothing else in this field works without it.
The series gets more advanced as it goes. Whether you're completely new or brushing up, the goal for every post is the same: by the end, you should be able to use the idea, not just recognise the word. Let's get into it.
Why statistics?
Statistics is the backbone of data science. Before you build a model, make a prediction, or draw a business conclusion, you need to understand your data, and statistics is the toolkit for doing that.
Think of statistics as the language your data speaks. Without it, you're staring at a spreadsheet full of numbers. With it, you can find patterns, measure how uncertain you should be, and make decisions with real confidence behind them. Every major technique in data science, from linear regression to neural networks, sits on a statistical foundation.
A doctor doesn't examine every cell in your body to check your health. They look at a handful of key indicators (blood pressure, cholesterol, temperature) and use those to understand the full picture. Data scientists do the same thing with data: a few well-chosen summary numbers tell you far more than staring at every raw row ever will.
The two branches of statistics
Statistics splits into two main branches, and almost every technique in this series falls into one of them.
| Descriptive statistics | Inferential statistics | |
|---|---|---|
| Goal | Summarise and describe the data you already have | Draw conclusions about a population from a sample |
| Scope | The dataset in hand | A broader, unseen population |
| Output | Charts, averages, distributions | Estimates with a margin of error, hypothesis test results, confidence intervals |
| Example | "The average exam score in this class is 74." | "Students nationally likely score between 70 and 78." |
You'll almost always use them in that order: descriptive first, to understand what you're looking at, then inferential, to go beyond it. Skip the first step and the second one is built on sand.
Descriptive statistics
Descriptive statistics is about making raw data understandable. It helps you collect, organise, summarise, and present data so its key features are visible at a glance. Instead of getting lost in thousands of rows, you see the shape of the thing.
The four tools that do most of the work
Central tendency
Mean, median and mode. Where the "middle" of your data sits.
Dispersion
Range, variance, standard deviation, IQR and MAD. How spread out the data is, and how much to trust that middle value.
Frequency distributions
How often each value, or range of values, shows up.
Graphs and tables
Histograms, box plots, bar charts and scatter plots, which turn all of the above into something you can see.
None of these tools mean much on their own. The real skill is knowing which combination answers your actual question, which is exactly what the next example is about.
A concrete example
Here are the salaries from the top of the page: ten employees at a startup, in thousands of US dollars a year.
[40K, 45K, 48K, 50K, 52K, 55K, 60K, 65K, 70K, 200K]
The mean says the "typical" employee earns about $68.5K. The median says $53.5K. That's a $15K gap between two numbers that are both supposed to describe the same "average" employee, and it comes entirely from one person on the list.
The $200K salary is an outlier: a value far removed from the rest of the data. The mean uses every value in its calculation, so that one number drags it upward. The median only cares about the middle position in a sorted list, so it barely notices. This is why the median is usually a better "typical value" for skewed data like income, house prices, or app session times, where a few extreme values sit alongside a lot of ordinary ones.
The standard deviation makes the distortion even clearer. Above, it comes out to about $47K, which would normally suggest salaries are spread wildly, but nine of the ten people are within $30K of each other. Drop the outlier and recompute:
One data point took the standard deviation from $47K down to under $10K. That's the whole lesson in one number: a single outlier can dominate the mean and the standard deviation while leaving the median almost untouched. A good data scientist checks the middle and the spread together, and looks at the actual data before trusting either one.
Now look at the IQR ($15.25K) and the MAD ($7.5K). Both stayed small without anyone removing the outlier. The IQR only looks at the middle 50% of the data (the gap between the first quartile, Q1, and the third quartile, Q3), so a value sitting far outside that range barely moves it. A box plot shows exactly this:
The MAD (median absolute deviation) works the same way: it uses the median as the centre, measures how far every value sits from it, and then takes the median of those distances. Both the IQR and the MAD are built to ignore exactly the kind of extreme value that inflates the mean and the standard deviation, which is why they're often better choices for skewed, real-world data.
Quartiles don't have one universal definition. Pandas and NumPy use linear interpolation by default, which is why the code above gives Q1 = 48.5K and Q3 = 63.75K. Split the same data into a lower and an upper half and take the median of each instead, a method plenty of introductory courses teach, and you get Q1 = 48K and Q3 = 65K, for an IQR of 17K instead of 15.25K. Neither answer is wrong. They're different conventions, so don't be alarmed if your textbook and your code disagree slightly on quartiles.
There are two versions of standard deviation. The population version divides by n (the count of values). The sample version divides by n−1. Why the smaller number? The spread is measured around the sample's own mean, and a sample's points always sit a little closer to their own mean than to the true population mean. So dividing by n comes out slightly too small on average, and dividing by n−1 corrects for it. This fix is called Bessel's correction, and pandas uses the sample version by default.
On our 10 salaries, the population std dev is $44.68K and the sample std dev is $47.09K: close, but not identical. The difference matters the moment you move from "describing this dataset" to "estimating something about a bigger population," which is exactly where inferential statistics picks up.
Frequency distributions and graphs
The third tool, frequency distributions, is just counting how often each value shows up. Say 20 users rated an app between 1 and 5 stars:
| Rating | Count | Share |
|---|---|---|
| 1 star | 0 | 0% |
| 2 stars | 1 | 5% |
| 3 stars | 4 | 20% |
| 4 stars | 6 | 30% |
| 5 stars | 9 | 45% |
As a table, this is a frequency distribution. As a bar chart, it's instantly readable: most people are happy, a handful are lukewarm, and almost nobody is unhappy. This is also where the mode comes from: the most frequent value in the data, 5 stars here. The mean and median describe a distribution's centre, while the mode describes its peak. For categorical or heavily repeated data like ratings, the mode is often the most useful of the three.
Why descriptive statistics matter in data science
This isn't theory you tick off before the "real" modelling starts. Descriptive statistics is where most of a data scientist's actual time goes.
- Exploratory data analysis (EDA). The first step in any project: spotting trends, patterns, outliers, and anomalies before you build anything.
- Data cleaning. Finding impossible values (a person aged 500) or patterns in missing data.
- Data visualisation. Turning raw numbers into histograms, box plots, and dashboards that actually communicate something to stakeholders.
- Business intelligence. Understanding what already happened (sales last quarter, churn rate, average order value) before you try to predict what happens next.
In practice, this usually starts with one line of code. Running salaries.describe() on the series above gives you the count, mean, std dev, min, max, and quartiles in a single call, which is why .describe() is so often the very first thing a data scientist runs on a new dataset.
Reporting the mean alone, without checking the spread or plotting the data first, is one of the most common mistakes new data scientists make. If you'd only seen "average salary: $68.5K" without the raw numbers, you'd have walked away with a genuinely wrong idea of what this startup pays most of its people.
Inferential statistics
Descriptive statistics tells you about the data you have. Inferential statistics lets you go further and reason about data you don't have.
The population is the entire group you're actually interested in. The sample is the smaller, manageable subset of that group you actually collect data from. By carefully analysing a sample, you make educated, mathematically grounded guesses about the much larger population. That's essential whenever collecting data from everyone is impossible or just impractical.
Two more words come with this. A number that describes a whole population is a parameter: for example, the true average salary of every data scientist in India. The same kind of number calculated from a sample is a statistic. You almost never get to see a parameter directly. You estimate it from a statistic, and inferential statistics is the set of methods for doing that honestly.
You can't survey every voter before an election, or test every light bulb a factory makes. Inference lets you draw reliable conclusions from a manageable sample instead, but only if that sample actually looks like the population.
In 1936, a magazine called The Literary Digest mailed about 10 million ballots, got roughly 2.4 million back, and predicted that Alf Landon would beat Franklin Roosevelt for the US presidency. Roosevelt won about 61% of the vote and 46 of the 48 states. The magazine's mailing lists leaned toward wealthier households, and the people who bothered to reply leaned even further against Roosevelt. Meanwhile George Gallup, polling only about 50,000 people chosen with more care, called the result correctly. A huge biased sample lost to a small representative one.
Key techniques
- Hypothesis testing is formally testing a claim about a population, like whether a new drug actually reduces blood pressure.
- Confidence intervals give you a range the true population parameter likely falls in, like being 95% confident the true average sits between 50 and 60.
- Regression analysis models the relationship between variables, both to understand it and to predict outcomes.
- Predictive modelling uses historical sample data to forecast future events.
A business example
Say an e-commerce company tests a new product recommendation algorithm. They show it to a randomly selected group of users for a month and measure purchase rates. Using inferential statistics, specifically a hypothesis test, they decide whether the improvement in that group is statistically significant: likely to hold across all their users, rather than just random chance.
A clinical trial works the same way. A new medication is tested on 200 patients, and inferential statistics decides whether the observed benefit is strong and consistent enough to conclude the medication works for patients broadly, not just the 200 in the trial.
A first taste of the math
Hypothesis tests and confidence intervals get built up properly later in the series, but it's worth seeing the actual computation once, even in miniature. Say you survey 1,000 people about their monthly spend on a subscription app. The sample gives you a mean of $42 and a standard deviation of $15. How confident can you be about the true average across everyone, not just your 1,000 respondents?
The only new idea in that code is standard_error. It measures how much the sample mean would wobble if you repeated the survey with a fresh 1,000 people: the standard deviation divided by the square root of the sample size. Bigger samples wobble less, which is why this interval is so narrow. Day 11 builds the idea up properly.
And that's a confidence interval. It says: if you ran this survey over and over, 95% of the intervals you'd compute this way would contain the true population average. It does not mean there's a 95% chance the true value sits in this one specific range. That mix-up is common enough that confidence intervals get a full article of their own.
One more note: with a sample of 1,000, the normal distribution used above and the slightly more exact t-distribution give practically the same critical value, 1.96 versus 1.9623, so using the simpler normal version cost no real accuracy. The gap only starts to matter with small samples, which is where the t-distribution earns its keep.
Why inferential statistics matter in data science
- Generalisation. Drawing reliable conclusions about a whole population from a small, well-chosen sample.
- Decisions under uncertainty. Putting a number on how confident you are in a conclusion, and letting that guide business strategy.
- A/B testing. The backbone of product development at tech companies, used to test features, interface changes, and pricing.
- Machine learning. Logistic regression is a statistical model whose coefficients come with standard errors and significance tests, and Naive Bayes is Bayes' Theorem put to work. Even judging a model's accuracy, with cross-validation and confidence intervals on its metrics, is an inferential exercise.
| Application | Technique |
|---|---|
| Testing whether a new marketing strategy leads to higher sales | Hypothesis testing |
| Estimating a city's average income from a 1,000-person survey | Confidence intervals |
| Predicting housing prices from historical data | Regression analysis |
| Comparing test scores between two teaching methods | Two-sample t-test |
The role of probability
You can't fully understand inferential statistics without probability. It's the mathematical engine behind inference, giving you a principled way to reason about uncertainty.
When someone says a result is "statistically significant at the 5% level," that's a probability statement. When you build a 95% confidence interval, probability defines what that 95% means. When a spam filter scores an email, it's working with probabilities too. If 2 of every 100 emails it has seen were spam, it starts every new email at a 2% chance of being spam before reading a single word. That starting number is called a prior. The email's actual content (its words, links, and sender) then pushes that number up or down. Day 7 shows exactly how that update works, using Bayes' Theorem.
Probability underpins statistical inference (how confident should you be in a conclusion), risk assessment (how likely is a bad outcome), predictive modelling (what's the most likely future value), and anomaly detection (how unusual is this data point). It's a big enough topic that it gets its own part of the series, starting on Day 7.
Descriptive vs. inferential, side by side
| Scenario | Descriptive | Inferential |
|---|---|---|
| Student exam scores | "The average score in Class A is 74, with a std dev of 8." | "Is Class A's performance significantly better than Class B's?" |
| Product sales | "Last month's sales peaked on weekends, with a median of $320 a day." | "Will this weekend pattern hold next quarter?" |
| Healthcare | "40% of patients in this trial reported side effects." | "We estimate 35% to 45% of all patients on this drug experience side effects." |
| Customer survey | "In our survey of 500 customers, 65% rated us 4 stars or higher." | "We're 95% confident that between about 61% and 69% of all customers would rate us 4 stars or higher." |
| App engagement | "Users in our sample opened the app 5.2 times a day on average." | "Does a new notification design actually increase daily opens, or is the change just noise?" |
Key takeaway
Statistics isn't about memorising formulas. It's about building intuition for data. Descriptive statistics helps you know your data deeply. Inferential statistics lets you make confident, honest claims about things you haven't directly observed, as long as your sample genuinely represents what you're asking about.
Start with descriptive statistics, always. Compute summary measures, plot distributions, check for outliers and missing values. Only once you understand what you're working with should you move on to inference and model building. Skipping this step is one of the most common mistakes new data scientists make, and you've now seen exactly why, with one $200K salary as proof.
Quick check
Four questions to test whether this stuck. The answers are one click away.
Why did the mean and median disagree so much in the salary example?
Because the mean uses every value in its calculation, so the single $200K outlier pulls it upward. The median only depends on the middle position in a sorted list, so it's far less sensitive to extreme values.
You're handed a dataset and told "the average is 50." What's the first thing you should ask?
What's the spread, and what does the distribution actually look like? An average alone tells you very little: the same "50" could mean values tightly clustered around 50, or a skewed distribution being pulled toward one side, the same way the $68.5K average salary looked nothing like what most of the ten employees actually earned.
Descriptive or inferential: "Based on a poll of 1,200 voters, we estimate 51% to 57% of the country supports the policy."
Inferential. It uses a sample (1,200 voters) to make a claim about a much larger population (the country), with a range that accounts for uncertainty. A purely descriptive version would just report the poll's own number: "54% of our 1,200 respondents support the policy." Notice the range is centred on that 54%.
A news website's online poll gets 300,000 votes. Should you trust it more than a carefully chosen random sample of 1,000 people?
No. An online poll only hears from people who visit that site and choose to vote, and those people can differ a lot from the population as a whole. That's the same trap The Literary Digest fell into in 1936. A smaller sample that actually represents the population beats a huge one that doesn't.
Qualitative versus quantitative data, discrete versus continuous, and the four scales of measurement (nominal, ordinal, interval and ratio), plus why that label decides which statistical tools you're even allowed to use.