MACHINE LEARNING

MASTERY

THE BEGINNER’S
GUIDE TO
Data
Science
A Journey from Data to
Insight with Statistical
Techniques

Vinod Chugani
This is Just a Sample

Thank-you for your interest in The Beginner’s Guide to Data Science.

This is just a sample of the full text. You can purchase the complete book online from:
https://machinelearningmastery.com/beginners-guide-data-science/

MACHINE LEARNING
MASTERY

THE BEGINNER’S
GUIDE TO
Data
Science
A Journey from Data to
Insight with Statistical
Techniques

Vinod Chugani
This is Just a Sample ii

Disclaimer
The information contained within this eBook is strictly for educational purposes. If you wish to
apply ideas contained in this eBook, you are taking full responsibility for your actions.
The author has made every effort to ensure the accuracy of the information within this book was
correct at time of publication. The author does not assume and hereby disclaims any liability to any
party for any loss, damage, or disruption caused by errors or omissions, whether such errors or
omissions result from accident, negligence, or any other cause.
No part of this eBook may be reproduced or transmitted in any form or by any means, electronic or
mechanical, recording or by any information storage and retrieval system, without written
permission from the author.

Credits
Founder: Jason Brownlee
Lead Editor: Adrian Tam
Author: Vinod Chugani
Technical Reviewers: Yoyo Chan and Kanwal Mehreen

Copyright
The Beginner’s Guide to Data Science
© 2024 MachineLearningMastery.com. All Rights Reserved.

Edition: v1.00
Contents

This is Just a Sample i

Preface iv

Introduction v

1 Revealing the Invisible: Visualizing Missing Values in Ames Housing 1

The Ames Properties Dataset . . . . . . . . . . . . . . . . . . . . . . . 1
Loading & Sizing Up the Dataset. . . . . . . . . . . . . . . . . . . . . . 2
Uncovering & Visualizing Missing Values . . . . . . . . . . . . . . . . . . 4
Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

4 Harmonizing Data: A Symphony of Segmenting, Concatenating, Pivoting, and Merging 11

Segmenting and Concatenating: Choreographing with Pandas . . . . . . . . . 11
Pivoting and Merging: Dancing with Pandas. . . . . . . . . . . . . . . . . 15
Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

11 Garage or Not? Housing Insights Through the Chi-Squared Test 21

Understanding the Chi-Squared Test . . . . . . . . . . . . . . . . . . . . 21
How the Chi-Squared Test Works. . . . . . . . . . . . . . . . . . . . . . 22
Unraveling the Correlation Between External Quality and Garage Presence . . . 22
Important Caveats . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Preface

Data science involves applying scientific techniques to extract insights from data, essentially
telling a story through data analysis. It involves utilizing sophisticated methods like machine
learning models to identify patterns or validate hypotheses. However, cultivating a data
science mindset is the most fundamental and crucial aspect.
This mindset does not rely on complex models; even basic statistical techniques are
sufficient to illustrate a data science workflow. This book operates on this premise: without
delving into advanced machine learning, it teaches how to approach data and substantiate
hypotheses. The essence of data science lies in constructing a compelling narrative supported
by empirical evidence.
In this guide, you will be led through each step, from data manipulation in Python to
employing fundamental statistical functions in NumPy and SciPy to draw insights from the
data. Whether you are new to data science or seeking to enhance your skills, this guide
requires minimal prerequisites yet instills the mindset of an experienced data scientist.
Introduction

Welcome to The Beginner’s Guide to Data Science. This book is a primer to data science but
does not involve any advanced machine learning models. The advanced models are deliberately
avoided because you should focus on what data science is about—–telling stories.
Data science can be difficult because there are unlimited tools you can use as long as they
can help you tell a story. As a beginner, you can get lost because you see a lot of statistical
tests, models, equations, and plots being used, and you may feel like you are jumping around.
However, you will find data science is easier to grasp once you focus on the objective rather
than the tool.

How to Learn Data Science

Watch how a data scientist works. What do you need to get the job done?
B You must know where to find and collect the data.
B You must be very proficient in programming.
B You must know a lot about probability and statistics.
B You must be an expert in machine learning models.
B You must be an artist who can create beautiful charts.
B You need to be a very good presenter.
None of these are wrong. However, you do not need to spend years mastering everything
before you can work on a data science project. Because data science is about telling stories,
you can quickly ramp up your data science journey by asking two questions: What story do
you want to tell, and how?
This book is created to walk you through a data science project but using only the bare
minimum of mathematics and programming. No sophisticated machine learning models are
involved. Building fancy charts and web pages is also not the focus. Rather, you will start with
a data set you can readily download and ask various questions about it. As a data scientist,
you will back your statement with data derived from some statistical model by juggling the
dataset with a few lines of Python code.
 vi

Book Organization
This book is in two parts. The first part is about data wrangling, particularly with the pandas
library. This is useful because you often need to process the dataset to create derived data or
filter for useful data. This part includes the following chapters:
1. Revealing the Invisible: Visualizing Missing Values in Ames Housing
2. Exploring Dictionaries, Classifying Variables, and Imputing Data
3. Beyond SQL: Transforming Real Estate Data into Actionable Insights with Pandas
4. Harmonizing Data: A Symphony of Segmenting, Concatenating, Pivoting, and
Merging
These chapters guide you using Python and the pandas library to manipulate a dataset. This
is a useful and important skill because you should manipulate the entire dataset rather than
processing each number from the dataset. With pandas, you can filter the data or find the
average with just one line of code.
The second part of the book assumes you can process the data readily. Then, you will
be shown how to obtain information from the raw data to make a statement.
This part includes the following chapters:
4. Decoding Data: An Introduction to Descriptive Statistics
5. From Data to Map: Visualizing Ames House Prices with Python
6. Feature Relationships 101: Lessons from the Ames Housing Data
7. Mastering Pair Plots for Visualization and Hypothesis Creation
8. Inferential Insights: How Confidence Intervals Illuminate the Ames Real Estate Market
9. Testing Assumptions in Real Estate: A Dive into Hypothesis Testing
10. Garage or Not? Housing Insights Through the Chi-Squared Test
11. Leveraging ANOVA and Kruskal-Wallis Tests to Analyze the Impact of the Great
Recession on Housing Prices
12. Spotting the Exception: Classical Methods for Outlier Detection in Data Science
13. Skewness Be Gone: Transformative Tricks for Data Scientists
14. Finding Value with Data: The Cohesive Force Behind Luxury Real Estate Decisions
All these chapters are based on the same dataset, but each asks a different question. This is
not a simple question like what is the average price of a house, but one like whether having
a garage makes a house more expensive. To answer such a question, you must find the right
mathematical tool, apply the appropriate Python function, and interpret the result correctly.
To tell a story about the data, you focus on the objective, and there can be multiple ways
to achieve that. Therefore, you should consider the methods and models used in each chapter
as examples rather than the only golden rule for the task. It may not be the best way for a
task either, but surely, it is a simple solution so you can learn the thought process easily.
 vii

Appendix B outlines the tools you might find useful. In Appendix C you will find a
summary of how you should apply the thought process to a data science project.

Requirements for This Book

This book focuses heavily on statistical models rather than more advanced machine learning
ones. Even so, you do not need to be an expert in statistics to enjoy it since most statistical
concepts are well explained when they are introduced. However, if you are interested in
learning more, you may find the other book by Machine Learning Mastery, Statistical Methods
for Machine Learning 1 useful.
The code in this book is in Python. It is the most popular language in this domain, and
you can easily find help online. It does not mean you need to be a Python expert. All you need
to know is to install and setup Python with the NumPy and SciPy libraries. The examples
assume that you have a Python environment available. This may be on your workstation or
laptop, it may be in a VM or a Docker instance that you run, or it may be a server instance
that you can configure in the cloud. Please find Appendix A if you need instructions to setup
your Python environment.
The most important requirement of this book is a good mindset. There is a lot to cover in
this book. Each topic can be drilled down and expanded into a much larger context. However,
remember that you should focus on the high-level goal of the data instead of all the details in
the models and code. In many cases, you will see that a function from SciPy is used, and there
are many arguments you can specify for that function. However, the example only covered a
minimum variation to complete the job. You should finish the story in each chapter first and
go back to change the code to try a different idea afterward. This would be the fastest way
to learn.

Your Outcomes from Reading This Book

This book will lead you from being a programmer with little to no data science knowledge to
a developer with sufficient knowledge to carry out basic workflow in data science.
Specifically, you will know:
B How to develop and evaluate an idea about a dataset.
B The basic but commonly used statistical models in data science.
B How to present data in visualizations.
B How to deliver your argument quantitatively.
This book is not a textbook. You are provided with the theoretical background at a bare
minimum. This book, however, is a cookbook or playbook such that there are step-by-step
instructions you can follow to finish a project.
You do not need to read the chapters in their order. While the chapters are ordered to
help you go from the beginning to the end of a data science journey, you can skip to chapters
1
https://machinelearningmastery.com/statistics_for_machine_learning/
 viii

as your needs or interests motivate you. To get the most from this book, you should also
attempt to improve the results, try a different function or model, apply the method to a
similar but different problem, and so on. You are welcome to share your findings with us at
[email protected].

Next
Let’s dive in. Next up is Part I, where you will take a tour of the pandas library in Python
to wrangle with data, which is an essential skill you will be using throughout this book.
 1
Revealing the Invisible:
Visualizing Missing Values in
Ames Housing

The digital age has ushered in an era where data-driven decision-making is pivotal in various
domains, real estate being a prime example. Comprehensive datasets, like the Ames housing
dataset, offer a treasure trove for data enthusiasts. Through meticulous exploration and
analysis of such datasets, one can uncover patterns, gain insights, and make informed decisions.
Starting from this chapter, you will embark on a captivating journey through the intricate
lanes of Ames properties, focusing primarily on data science techniques.
Let’s get started.

Overview
This chapter is divided into three parts; they are:
B The Ames Properties Dataset
B Loading & Sizing Up the Dataset
B Uncovering & Visualizing Missing Values

1.1 The Ames Properties Dataset

Every dataset has a story to tell, and understanding its background can offer invaluable
context. While the Ames Housing Dataset is widely known in academic circles, the dataset
we’re analyzing today, Ames.csv1 , which is a more comprehensive collection of property details
from Ames.
Dr. Dean De Cock, a dedicated academician, recognized the need for a new, robust dataset
in the domain of real estate. He meticulously compiled the Ames Housing Dataset, which has
since become a cornerstone for budding data scientists and researchers. This dataset stands
out due to its comprehensive details, capturing myriad facets of real estate properties. It has
been a foundation for numerous predictive modeling exercises and offers a rich landscape for
exploratory data analysis.
1
The original dataset was in a tab-separated value format. The CSV version is available at https://raw.github
usercontent.com/Padre-Media/dataset/main/Ames.csv
1.2 Loading & Sizing Up the Dataset 2

The Ames Housing Dataset was envisioned as a modern alternative to the older Boston
Housing Dataset. Covering residential sales in Ames, Iowa between 2006 and 2010, it presents
a diverse array of variables, setting the stage for advanced regression techniques.
This time frame is particularly significant in U.S. history. The period leading up to
2007–2008 saw the dramatic inflation of housing prices, fueled by speculative frenzy and
subprime mortgages. This culminated in the devastating collapse of the housing bubble
in late 2007, an event vividly captured in narratives like “The Big Short.” The aftermath
of this collapse rippled across the nation, leading to the Great Recession. Housing prices
plummeted, foreclosures skyrocketed, and many Americans found themselves underwater on
their mortgages. The Ames dataset provides a glimpse into this turbulent period, capturing
property sales during national economic upheaval.

1.2 Loading & Sizing Up the Dataset

For those who are venturing into the realm of data science, having the right tools in your
arsenal is paramount. If you require some help to setup your Python environment, the guide
is in Appendix A.

Dataset Dimensions. Before diving into intricate analyses, it’s essential to familiarize yourself
with the basic structure and data types of the dataset. This step provides a roadmap for
subsequent exploration and ensures you tailor your analyses based on the nature of data.
With the Python environment in place, let’s load and gauge the dataset in terms of rows
(individual properties) and columns (attributes of these properties).

# Load the Ames dataset

import pandas as pd
Ames = pd.read_csv('Ames.csv')

# Dataset shape
print(Ames.shape)

rows, columns = Ames.shape

print(f"The dataset comprises {rows} properties described across {columns} attributes.")

Listing 1.1: Loading the dataset

This prints:

(2579, 85)
The dataset comprises 2579 properties described across 85 attributes.

Output 1.1: Shape of the loaded DataFrame

Data Types. Recognizing the data type of each attribute helps shape our analysis approach.
Numerical attributes might be summarized using measures like mean or median, while mode
(the most frequently seen value) is apt for categorical attributes.
1.2 Loading & Sizing Up the Dataset 3

import pandas as pd
Ames = pd.read_csv('Ames.csv')

# Determine the data type for each feature

data_types = Ames.dtypes

# Tally the total by data type

type_counts = data_types.value_counts()
print(type_counts)

Listing 1.2: Investigating the data type

This shows:

object 44
int64 27
float64 14
dtype: int64

Output 1.2: Count of attribute data types

The Data Dictionary. Each dataset usually comes with a data dictionary to describe each of
its features. It specifies the meaning, possible values, and even the logic behind its collection.
Whether you’re deciphering the meaning behind an unfamiliar feature or discerning the
significance of particular values, the data dictionary serves as a comprehensive guide. It
bridges the gap between raw data and actionable insights, ensuring that the analyses and
decisions are well-informed.

import pandas as pd
Ames = pd.read_csv('Ames.csv')

# Determine the data type for each feature

data_types = Ames.dtypes

# View a few datatypes from the dataset (first and last 5 features)
print(data_types)

Listing 1.3: Investigating the data type

This shows, for example, Ground Living Area and Sale Price are numerical (int64) data types,
while Sale Condition (object) is a categorical data type:

PID int64
GrLivArea int64
SalePrice int64
MSSubClass int64
MSZoning object
...
SaleCondition object
GeoRefNo float64
Prop_Addr object
Latitude float64
1.3 Uncovering & Visualizing Missing Values 4

Longitude float64
Length: 85, dtype: object

Output 1.3: Data type of each attribute column

and you can match that with the data dictionary, an excerpt as follows:

Sale Condition (Nominal): Condition of sale

Normal Normal Sale

Abnorml Abnormal Sale - trade, foreclosure, short sale
AdjLand Adjoining Land Purchase

C
Alloca Allocation - two linked properties with separate deeds, typically condo C
with a garage unit
Family Sale between family members
Partial Home was not completed when last assessed (associated with New Homes)

SalePrice (Continuous): Sale price $$

Output 1.4: Excerpt from the data dictionary showing the “SaleCondition” has string
values and “SalePrice” is numerical

1.3 Uncovering & Visualizing Missing Values

Real-world datasets seldom arrive perfectly curated, often presenting analysts with the
challenge of missing values. These gaps in data can arise due to various reasons, such as
errors in data collection, system limitations, or the absence of information. Addressing missing
values is not merely a technical necessity but a critical step that significantly impacts the
integrity and reliability of subsequent analyses.
Understanding the patterns of missing values is essential for informed data analysis.
This insight guides the selection of appropriate imputation methods, which fill in missing
data based on available information, thereby influencing the accuracy and interpretability of
results. Additionally, assessing missing value patterns informs decisions on feature selection;
features with extensive missing data may be excluded to enhance model performance and
focus on more reliable information. In essence, grasping the patterns of missing values ensures
robust and trustworthy data analyses, guiding imputation strategies and optimizing feature
inclusion for more accurate insights.

NaN or None? In pandas, the isnull() function is used to detect missing values in a
DataFrame or Series. Specifically, it identifies the following types of missing data:
B np.nan (Not a Number), often used to denote missing numerical data
B None, which is a built-in object in Python to denote the absence of a value or a null
value
Both nan and NaN are just different ways to refer to np.nan in NumPy, and isnull() will
identify them as missing values. Here is a quick example.
1.3 Uncovering & Visualizing Missing Values 5

import pandas as pd
import numpy as np

# Create a DataFrame with various types of missing values

df = pd.DataFrame({
'A': [1, 2, np.nan, 4, 5],
'B': ['a', 'b', None, 'd', 'e'],
'C': [np.nan, np.nan, np.nan, np.nan, np.nan],
'D': [1, 2, 3, 4, 5]
})

# Use isnull() to identify missing values

missing_data = df.isnull().sum()

print(df)
print()
print(missing_data)

Listing 1.4: Showing NaN values

This prints:

A B C D
0 1.0 a NaN 1
1 2.0 b NaN 2
2 NaN None NaN 3
3 4.0 d NaN 4
4 5.0 e NaN 5

A 1
B 1
C 5
D 0
dtype: int64

Output 1.5: A DataFrame with NaN values

Visualizing Missing Values. When it comes to visualizing missing data, tools like missingno,
matplotlib, and seaborn come in handy. By sorting the features based on the percentage of
missing values and placing them into a DataFrame, you can easily rank the features most
affected by missing data.

import pandas as pd
Ames = pd.read_csv('Ames.csv')

# Calculating the percentage of missing values for each column

missing_data = Ames.isnull().sum()
missing_percentage = (missing_data / len(Ames)) * 100

# Combining the counts and percentages into a DataFrame for better visualization
missing_info = pd.DataFrame({'Missing Values': missing_data,
'Percentage': missing_percentage})
1.3 Uncovering & Visualizing Missing Values 6

# Sorting the DataFrame by the percentage of missing values in descending order

missing_info = missing_info.sort_values(by='Percentage', ascending=False)

# Display columns with missing values

print(missing_info[missing_info['Missing Values'] > 0])

Listing 1.5: Calculating exposure of missing values from a DataFrame

This shows:

Missing Values Percentage

PoolQC 2570 99.651028
MiscFeature 2482 96.238852
Alley 2411 93.485847
Fence 2054 79.643273
FireplaceQu 1241 48.119426
LotFrontage 462 17.913920
GarageCond 129 5.001939
GarageQual 129 5.001939
GarageFinish 129 5.001939
GarageYrBlt 129 5.001939
GarageType 127 4.924389
Longitude 97 3.761148
Latitude 97 3.761148
BsmtExposure 71 2.753005
BsmtFinType2 70 2.714230
BsmtFinType1 69 2.675456
BsmtQual 69 2.675456
BsmtCond 69 2.675456
GeoRefNo 20 0.775494
Prop_Addr 20 0.775494
MasVnrArea 14 0.542846
MasVnrType 14 0.542846
BsmtFullBath 2 0.077549
BsmtHalfBath 2 0.077549
GarageArea 1 0.038775
BsmtFinSF1 1 0.038775
Electrical 1 0.038775
TotalBsmtSF 1 0.038775
BsmtUnfSF 1 0.038775
BsmtFinSF2 1 0.038775
GarageCars 1 0.038775

Output 1.6: Count of missing values in each attribute

The missingno package facilitates a swift, graphical representation of missing data. The white
lines or gaps in the visualization denote missing values. However, it will only accommodate
up to 50 labeled variables. Past that range, labels begin to overlap or become unreadable,
and by default large displays omit them.
1.3 Uncovering & Visualizing Missing Values 7

import pandas as pd
import missingno as msno
import matplotlib.pyplot as plt

Ames = pd.read_csv('Ames.csv')
msno.matrix(Ames, sparkline=False, fontsize=20)
plt.show()

Listing 1.6: Illustrating missing values using the missingno package

Figure 1.1: Visual representation of missing values using missingno.matrix().

You can get a bar chart using msno.bar to show the top 15 features with least count of
non-missing values, i.e., those with the most missing values.

import pandas as pd
import missingno as msno
import matplotlib.pyplot as plt
Ames = pd.read_csv('Ames.csv')

# Calculating the percentage of missing values for each column

missing_data = Ames.isnull().sum()
missing_percentage = (missing_data / len(Ames)) * 100

# Combining the counts and percentages into a DataFrame for better visualization
missing_info = pd.DataFrame({'Missing Values': missing_data,
'Percentage': missing_percentage})

# Sort the DataFrame columns by the percentage of missing values

sorted_df = Ames[missing_info.sort_values(by='Percentage', ascending=False).index]

# Select the top 15 columns with the most missing values

top_15_missing = sorted_df.iloc[:, :15]
1.3 Uncovering & Visualizing Missing Values 8

#Visual with missingno

msno.bar(top_15_missing)
plt.show()

Listing 1.7: Showing the number of missing values in a bar chart

Figure 1.2: Using missingno.bar() to visualize features with missing data.

The illustration above denotes that Pool Quality, Miscellaneous Feature, and the type of Alley
access to the property are the three features with the highest number of missing values since
their bar is the shortest.

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

Ames = pd.read_csv('Ames.csv')
missing_data = Ames.isnull().sum()
missing_percentage = (missing_data / len(Ames)) * 100
missing_info = pd.DataFrame({'Missing Values': missing_data,
'Percentage': missing_percentage})

# Filter to show only the top 15 columns with the most missing values
top_15_missing_info = missing_info.nlargest(15, 'Percentage')

# Create the horizontal bar plot using seaborn

plt.figure(figsize=(12, 8))
sns.barplot(x='Percentage', y=top_15_missing_info.index, hue=top_15_missing_info.index,
data=top_15_missing_info, orient='h')
plt.title('Top 15 Features with Missing Percentages', fontsize=20)
plt.xlabel('Percentage of Missing Values', fontsize=16)
plt.ylabel('Features', fontsize=16)
#plt.yticks(fontsize=11)
plt.show()

Listing 1.8: Showing top columns with the highest percentage of missing values
1.4 Further Reading 9

Figure 1.3: Using seaborn horizontal bar plots to visualize missing data.

A horizontal bar plot using seaborn allows you to list features with the highest missing values
in a vertical format, adding both readability and aesthetic value.
Handling missing values is more than just a technical requirement; it’s a significant step
that can influence the quality of your machine learning models. Understanding and visualizing
these missing values are the first steps in this intricate dance.

1.4 Further Reading

This section provides more resources on the topic if you want to go deeper.

Papers
Dean De Cock. “Ames, Iowa: Alternative to the Boston Housing Data as an End of
Semester Regression Project”. Journal of Statistics Education, 19(3), 2011.
https://jse.amstat.org/v19n3/decock.pdf

Resources
Dean De Cock. Ames Housing Dataset. 2011.
https://jse.amstat.org/v19n3/decock/AmesHousing.txt
Dean De Cock. Ames Housing Data Dictionary. 2011.
https://jse.amstat.org/v19n3/decock/DataDocumentation.txt
1.5 Summary 10

1.5 Summary
In this chapter, you embarked on an exploration of the Ames Properties dataset, a
comprehensive collection of housing data tailored for data science applications.
Specifically, you learned:
B About the context of the Ames dataset, including the pioneers and academic
importance behind it.
B How to extract dataset dimensions, data types, and missing values.
B How to use packages like missingno, Matplotlib, and seaborn to quickly visualize
your missing data.
As you learn about missing data, you will see how you can fill in the missing values in the
next chapter.
 4
Harmonizing Data: A Symphony
of Segmenting, Concatenating,
Pivoting, and Merging

In a data science project, the data you collect is often not in the shape that you want it to be.
Often you will need to create derived features, aggregate subsets of data into a summarized
form, or select a portion of the data according to some complex logic. This is not a hypothetical
situation. In a project big or small, the data you obtained at the first step is very likely far
from ideal.
As a data scientist, you must be handy to format the data into the right shape to make
your subsequent steps easier. In the following, you will learn how to slice and dice the dataset
in pandas as well as reassemble them into a very different form to make the useful data more
pronounced, so that analysis can be easier.
Let’s get started.

Overview
This chapter is divided into two parts; they are:
B Segmenting and Concatenating: Choreographing with Pandas
B Pivoting and Merging: Dancing with Pandas

4.1 Segmenting and Concatenating: Choreographing with

Pandas
One intriguing question you might pose is: How does the year a property was built influence
its price? To investigate this, you can segment the dataset by “SalePrice” into four quartiles—
Low, Medium, High, and Premium—and analyze the construction years within these segments.
This methodical division of the dataset not only paves the way for a focused analysis but also
reveals trends that might be concealed within a collective review.

Segmentation Strategy: Quartiles of “SalePrice”

Let’s begin by creating a new column that neatly classifies the “SalePrice” of properties into
your defined price categories:
4.1 Segmenting and Concatenating: Choreographing with Pandas 12

import pandas as pd

# Load the dataset

Ames = pd.read_csv('Ames.csv')

# Define the quartiles

quantiles = Ames['SalePrice'].quantile([0.25, 0.5, 0.75])

# Function to categorize each row

def categorize_by_price(row):
if row['SalePrice'] <= quantiles.iloc[0]:
return 'Low'
elif row['SalePrice'] <= quantiles.iloc[1]:
return 'Medium'
elif row['SalePrice'] <= quantiles.iloc[2]:
return 'High'
else:
return 'Premium'

# Apply the function to create a new column

Ames['Price_Category'] = Ames.apply(categorize_by_price, axis=1)
print(Ames[['SalePrice','Price_Category']])

Listing 4.1: Adding the price category column

By executing the above code, you have enriched your dataset with a new column entitled
“Price_Category.” Here’s a glimpse of the output you’ve obtained:

SalePrice Price_Category
0 126000 Low
1 139500 Medium
2 124900 Low
3 114000 Low
4 227000 Premium
... ... ...
2574 121000 Low
2575 139600 Medium
2576 145000 Medium
2577 217500 Premium
2578 215000 Premium

[2579 rows x 2 columns]

Output 4.1: Price and the corresponding category

Visualizing Trends with the Empirical Cumulative Distribution Function

The empirical cumulative distribution function (ECDF) is an increasing function range from
0 to 1. Below you will see how you can use ECDF to show the year a house was built is a
distinctive feature for different price categories.
You can now split the original dataset into four DataFrames and proceed to visualize
the cumulative distribution of construction years within each price category. This visual will
4.1 Segmenting and Concatenating: Choreographing with Pandas 13

help you understand at a glance the historical trends in property construction as they relate
to pricing.

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

def categorize_by_price(row):
if row['SalePrice'] <= quantiles.iloc[0]:
return 'Low'
elif row['SalePrice'] <= quantiles.iloc[1]:
return 'Medium'
elif row['SalePrice'] <= quantiles.iloc[2]:
return 'High'
else:
return 'Premium'

Ames = pd.read_csv('Ames.csv')
quantiles = Ames['SalePrice'].quantile([0.25, 0.5, 0.75])
Ames['Price_Category'] = Ames.apply(categorize_by_price, axis=1)

# Split original dataset into 4 DataFrames by Price Category

low_priced_homes = Ames.query('Price_Category == "Low"')
medium_priced_homes = Ames.query('Price_Category == "Medium"')
high_priced_homes = Ames.query('Price_Category == "High"')
premium_priced_homes = Ames.query('Price_Category == "Premium"')

# Setting the style for aesthetic looks

sns.set_style("whitegrid")

# Create a figure
plt.figure(figsize=(10, 6))

# Plot each ECDF on the same figure

sns.ecdfplot(data=low_priced_homes, x='YearBuilt', color='skyblue', label='Low')
sns.ecdfplot(data=medium_priced_homes, x='YearBuilt', color='orange', label='Medium')
sns.ecdfplot(data=high_priced_homes, x='YearBuilt', color='green', label='High')
sns.ecdfplot(data=premium_priced_homes, x='YearBuilt', color='red', label='Premium')

# Adding labels and title for clarity

plt.title('ECDF of Year Built by Price Category', fontsize=16)
plt.xlabel('Year Built', fontsize=14)
plt.ylabel('ECDF', fontsize=14)
plt.legend(title='Price Category', title_fontsize=14, fontsize=14)

# Show the plot

plt.show()

Listing 4.2: Creating the ECDF plot

Below is the ECDF plot, which provides a visual summation of the data you’ve categorized.
An ECDF, or Empirical Cumulative Distribution Function, is a statistical tool used to describe
the distribution of data points in a dataset. It represents the proportion or percentage of data
points that fall below or at a certain value. Essentially, it gives you a way to visualize the
4.1 Segmenting and Concatenating: Choreographing with Pandas 14

distribution of data points across different values, providing insights into the shape, spread,
and central tendency of the data. ECDF plots are particularly useful because they allow for
easy comparison between different datasets. Notice how the curves for each price category
give you a narrative of housing trends over the years.

Figure 4.1: The ECDF plot

From the plot, it is evident that lower and medium-priced homes have a higher frequency of
being built in earlier years, while high and premium-priced homes tend to be of more recent
construction.

Stacking Datasets with Pandas.concat()

As data scientists, you often need to stack datasets or their segments to extract deeper insights.
The Pandas.concat() function is your Swiss Army knife for such tasks, enabling you to combine
DataFrames with precision and flexibility. This powerful function is similar to UNION operation
in SQL when it comes to combining rows from different datasets. Yet, Pandas.concat() stands
out by offering greater flexibility—it allows both vertical and horizontal concatenation of
DataFrames. This feature becomes indispensable when datasets have non-matching columns
but you need to align them by common columns. Here’s how you can combine the segmented
DataFrames to compare the broader market categories of “affordable” and “luxury” homes:

...
# Stacking Low and Medium categories into an "affordable_homes" DataFrame
affordable_homes = pd.concat([low_priced_homes, medium_priced_homes])

# Stacking High and Premium categories into a "luxury_homes" DataFrame

luxury_homes = pd.concat([high_priced_homes, premium_priced_homes])

Listing 4.3: Using concat() to create DataFrame

Through this, you can compare and analyze the characteristics that differentiate more accessible
homes from their expensive counterparts. Should the two DataFrames you pass on to concat()
4.2 Pivoting and Merging: Dancing with Pandas 15

have different columns, the resulting DataFrame will have the superset of columns from both,
while some rows filled in NaN for the columns that they have no data from the original
DataFrame.

4.2 Pivoting and Merging: Dancing with Pandas

Having segmented the dataset into “affordable” and “luxury” homes and explored the
distribution of their construction years, you now turn your attention to another dimension
that influences property value: amenities, with a focus on the number of fireplaces. Before
you delve into merging datasets—a task for which Pandas.merge() stands as a robust tool
comparable to JOIN in SQL—you must first examine your data through a finer lens.
Pivot tables are an excellent tool for summarizing and analyzing specific data points
within the segments. They provide you with the ability to aggregate data and reveal patterns
that can inform your subsequent merge operations. By creating pivot tables, you can compile
a clear and organized overview of the average living area and the count of homes, categorized
by the number of fireplaces. This preliminary analysis will not only enrich your understanding
of the two market segments but also set a solid foundation for the intricate merging techniques
you want to show.

Creating Insightful Summaries with Pivot Tables

Let’s commence by constructing pivot tables for the “affordable” and “luxury” home categories.
These tables will summarize the average gross living area (GrLivArea) and provide a count
of homes for each category of fireplaces present. Such analysis is crucial as it illustrates a key
aspect of home desirability and value—the presence and number of fireplaces—and how these
features vary across different segments of the housing market.

...
# Creating pivot tables with both mean living area and home count
aggfunc = {'GrLivArea': 'mean', 'Fireplaces': 'count'}
pivot_affordable = affordable_homes.pivot_table(index='Fireplaces', aggfunc=aggfunc)
pivot_luxury = luxury_homes.pivot_table(index='Fireplaces', aggfunc=aggfunc)

# Renaming columns and index labels separately

rename_rules = {'GrLivArea': 'AvLivArea', 'Fireplaces': 'HmCount'}

pivot_affordable.rename(columns=rename_rules, inplace=True)
pivot_affordable.index.name = 'Fire'

pivot_luxury.rename(columns=rename_rules, inplace=True)
pivot_luxury.index.name = 'Fire'

# View the pivot tables

print(pivot_affordable)
print(pivot_luxury)

Listing 4.4: Using pivot table to summarize the number of homes and the average
area
4.2 Pivoting and Merging: Dancing with Pandas 16

With these pivot tables, you can now easily visualize and compare how features like fireplaces
correlate with the living area and how frequently they occur within each segment. The first
pivot table was crafted from the “affordable” homes DataFrame and demonstrates that most
properties within this grouping do not have any fireplaces.

HmCount AvLivArea
Fire
0 931 1159.050483
1 323 1296.808050
2 38 1379.947368

Output 4.2: DataFrame of affordable homes

The second pivot table which was derived from the “luxury” homes DataFrame illustrates
that properties within this subset have a range of zero to four fireplaces, with one fireplace
being the most common.

HmCount AvLivArea
Fire
0 310 1560.987097
1 808 1805.243812
2 157 1998.248408
3 11 2088.090909
4 1 2646.000000

Output 4.3: DataFrame of luxury homes

With the creation of the pivot tables, you’ve distilled the data into a form that’s ripe for the
next analytical step—melding these insights using Pandas.merge() to see how these features
interplay across the broader market.
The pivot table above is the simplest one. The more advanced version allows you to
specify not only the index but also the columns in the argument. The idea is similar: you pick
two columns, one specified as index and the other as columns argument, in which the values
of these two columns are aggregated and become a matrix. The value in the matrix is then
the result as specified by the aggfunc argument.
You can consider the following example, which produces a similar result as above:

...
pivot = Ames.pivot_table(index="Fireplaces",
columns="Price_Category",
aggfunc={'GrLivArea':'mean', 'Fireplaces':'count'})
print(pivot)

Listing 4.5: More advanced pivot table syntax

This prints:
4.2 Pivoting and Merging: Dancing with Pandas 17

Fireplaces GrLivArea
Price_Category High Low Medium Premium High Low Medium Premium
Fireplaces
0 228.0 520.0 411.0 82.0 1511.912281 1081.496154 1257.172749 1697.439024
1 357.0 116.0 207.0 451.0 1580.644258 1184.112069 1359.961353 1983.031042
2 52.0 9.0 29.0 105.0 1627.384615 1184.888889 1440.482759 2181.914286
3 5.0 NaN NaN 6.0 1834.600000 NaN NaN 2299.333333
4 NaN NaN NaN 1.0 NaN NaN NaN 2646.000000
Output 4.4: The number of homes and average living area by the number of fireplaces
and price category

You can see the result is the same by comparing, for example, the count of low and medium
homes of zero fireplaces to be 520 and 411, respectively, which 931 = 520+411 as you obtained
previously. You see the second-level columns are labeled with Low, Medium, High, and
Premium because you specified “Price_Category” as columns argument in pivot_table(). The
dictionary to the aggfunc argument gives the top-level columns.

Towards Deeper Insights: Leveraging Pandas.merge() for Comparative

Analysis
Having illuminated the relationship between fireplaces, home count, and living areas within
the segmented datasets, you are well-positioned to take your analysis one step further. With
Pandas.merge(), you can overlay these insights, akin to how JOIN operation in SQL combines
records from two or more tables based on a related column. This technique will allow you
to merge the segmented data on a common attribute, enabling a comparative analysis that
transcends categorization.
You first operation uses an outer join to combine the affordable and luxury home datasets,
ensuring no data is lost from either category. This method is particularly illuminating as it
reveals the full spectrum of homes, regardless of whether they share a common number of
fireplaces.

...
pivot_outer_join = pd.merge(pivot_affordable, pivot_luxury, on='Fire', how='outer',
suffixes=('_aff', '_lux')).fillna(0)
print(pivot_outer_join)

Listing 4.6: Outer join using merge() function

HmCount_aff AvLivArea_aff HmCount_lux AvLivArea_lux

Fire
0 931.0 1159.050483 310 1560.987097
1 323.0 1296.808050 808 1805.243812
2 38.0 1379.947368 157 1998.248408
3 0.0 0.000000 11 2088.090909
4 0.0 0.000000 1 2646.000000

Output 4.5: Count of homes and average living area by number of fireplaces

In this case, the outer join functions similarly to a right join, capturing every distinct category
of fireplaces present across both market segments. It is interesting to note that there are no
properties within the affordable price range that have 3 or 4 fireplaces. You need to specify
4.2 Pivoting and Merging: Dancing with Pandas 18

two strings for the suffixes argument because the “HmCount” and “AvLivArea” columns exist
in both DataFrames pivot_affordable and pivot_luxury. You see “HmCount_aff” is zero for 3
and 4 fireplaces because you need them as a placeholder for the outer join to match the rows
in pivot_luxury.
Next, you can use the inner join, focusing on the intersection where affordable and luxury
homes share the same number of fireplaces. This approach highlights the core similarities
between the two segments.

...
pivot_inner_join = pd.merge(pivot_affordable, pivot_luxury, on='Fire', how='inner',
suffixes=('_aff', '_lux'))
print(pivot_inner_join)

Listing 4.7: Inner join using merge() function

HmCount_aff AvLivArea_aff HmCount_lux AvLivArea_lux

Fire
0 931 1159.050483 310 1560.987097
1 323 1296.808050 808 1805.243812
2 38 1379.947368 157 1998.248408

Output 4.6: Result of inner join shows fewer number of rows

Interestingly, in this context, the inner join mirrors the functionality of a left join, showcasing
categories present in both datasets. You do not see the rows corresponding to 3 and 4
fireplaces because it is the result of an inner join, and there are no such rows in the DataFrame
pivot_affordable.

Lastly, a cross join allows you to examine every possible combination of affordable and
luxury home attributes, offering a comprehensive view of how different features interact across
the entire dataset. The result is sometimes called the Cartesian product of rows from the two
DataFrames.

...
# Resetting index to display cross join
pivot_affordable.reset_index(inplace=True)
pivot_luxury.reset_index(inplace=True)

pivot_cross_join = pd.merge(pivot_affordable, pivot_luxury, how='cross',

suffixes=('_aff', '_lux')).round(2)
print(pivot_cross_join)

Listing 4.8: Cross join using merge() function

The result is as follows, which demonstrates the result of cross-join but does not provide any
special insight into the context of this dataset.

Fire_aff HmCount_aff AvLivArea_aff Fire_lux HmCount_lux AvLivArea_lux

0 0 931 1159.05 0 310 1560.99
1 0 931 1159.05 1 808 1805.24
2 0 931 1159.05 2 157 1998.25
3 0 931 1159.05 3 11 2088.09
4.2 Pivoting and Merging: Dancing with Pandas 19

4 0 931 1159.05 4 1 2646.00

5 1 323 1296.81 0 310 1560.99
6 1 323 1296.81 1 808 1805.24
7 1 323 1296.81 2 157 1998.25
8 1 323 1296.81 3 11 2088.09
9 1 323 1296.81 4 1 2646.00
10 2 38 1379.95 0 310 1560.99
11 2 38 1379.95 1 808 1805.24
12 2 38 1379.95 2 157 1998.25
13 2 38 1379.95 3 11 2088.09
14 2 38 1379.95 4 1 2646.00

Output 4.7: Result of cross join is a Cartesian product

Deriving Insights from Merged Data

Each join type demonstrated above sheds light on different aspects of the housing market:
B The outer join reveals the broadest range of properties, emphasizing the diversity in
amenities like fireplaces across all price points.
B The inner join refines your view, focusing on the direct comparisons where affordable
and luxury homes overlap in their fireplace counts, providing a clearer picture of
standard market offerings.
B The cross join offers an exhaustive combination of features, you may need that to
generate an exhaustive list of all possible conditions.
After conducting these merges, you observe amongst affordable homes that:
B Homes with no fireplaces have an average gross living area of approximately 1159
square feet and constitute the largest segment.
B As the number of fireplaces increases to one, the average living area expands to around
1296 square feet, underscoring a noticeable uptick in living space.
B Homes with two fireplaces, though fewer in number, boast an even larger average
living area of approximately 1379 square feet, highlighting a trend where additional
amenities correlate with more generous living spaces.
In contrast, you observe amongst luxury homes that:
B The luxury segment presents a starting point with homes without fireplaces averaging
1560 square feet, significantly larger than their affordable counterparts.
B The leap in the average living area is more pronounced in luxury homes as the number
of fireplaces increases, with one-fireplace homes averaging about 1805 square feet.
B Homes with two fireplaces further amplify this trend, offering an average living area
of nearly 1998 square feet. Homes with three to four fireplaces, although not many,
have even larger living space, peaking at 2646 square feet.
These observations offer a fascinating glimpse into how amenities such as fireplaces not only add
to the desirability of homes but also appear to be a marker of larger living spaces, particularly
as you move from affordable to luxury market segments.
4.3 Further Reading 20

4.3 Further Reading

This section provides more resources on the topic if you want to go deeper.

Online
concat() method. pandas.
https://pandas.pydata.org/docs/reference/api/pandas.concat.html
DataFrame.pivot_table() method. pandas.
https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.pivot_table.html
merge() method. pandas.
https://pandas.pydata.org/docs/reference/api/pandas.merge.html

4.4 Summary
In this comprehensive exploration of data harmonization techniques using Python and Pandas,
you’ve delved into the intricacies of segmenting, concatenating, pivoting, and merging datasets.
From dividing datasets into meaningful segments based on price categories to visualizing trends
in construction years, and from stacking datasets to analyzing broader market categories using
Pandas.concat(), to summarizing and analyzing data points within segments using pivot
tables, you’ve covered a wide array of essential data manipulation and analysis techniques.
Additionally, by leveraging Pandas.merge() to compare segmented datasets and derive insights
from different types of merge operations (outer, inner, cross), you’ve unlocked the power of
data integration and exploration. Armed with these techniques, data scientists and analysts
can navigate the complex landscape of data with confidence, uncovering hidden patterns, and
extracting valuable insights that drive informed decision-making.
Specifically, you learned:
B How to divide datasets into meaningful segments based on price categories and visualize
trends in construction years.
B The use of Pandas.concat() to stack datasets and analyze broader market categories.
B The role of pivot tables in summarizing and analyzing data points within segments.
B How to leverage Pandas.merge() to compare segmented datasets and derive insights
from different types of merge operations (outer, inner, cross).
Starting from the next chapter, you will see examples of making a statement from data using
data science techniques. The first would be the simplest one: Describe what you can see from
the data from the most superficial aspect.
Garage or Not? Housing Insights
Through the Chi-Squared Test
11
The chi-squared test for independence is a statistical procedure employed to assess the
relationship between two categorical variables—determining whether they are correlated or
independent. The exploration of the visual appeal of a property and its impact on its valuation
is intriguing. But how often do you associate the outlook of a house with functional features
like a garage? With the chi-squared test, you can determine whether there exists a statistically
significant correlation between features.
Let’s get started.

Overview
This chapter is divided into four parts; they are:
B Understanding the Chi-Squared Test
B How the Chi-Squared Test Works
B Unraveling the Correlation Between External Quality and Garage Presence
B Important Caveats

11.1 Understanding the Chi-Squared Test

The chi-squared (χ2 ) test is useful because of its ability to test for relationships between
categorical variables. It’s particularly valuable when working with nominal or ordinal data,
where the variables are divided into categories or groups.
B Are two categorical variables independent of each other?
If the variables are independent, changes in one variable are not related to changes
in the other. There is no correlation between them.
B Is there a significant correlation between the two categorical variables?
Conversely, if the variables are correlated, changes in one variable usually change the
other. The chi-squared test helps to quantify whether this correlation is statistically
significant.
11.2 How the Chi-Squared Test Works 22

In this study, you will focus on the visual appeal of a house (categorized as “Great” or
“Average”) and its relation to the presence or absence of a garage. For the results of the
chi-squared test to be valid, the following conditions must be satisfied:
B Independence: The observations must be independent, meaning the occurrence of one
outcome shouldn’t affect another. Your dataset satisfies this as each entry represents
a distinct house.
B Large Sample Size: The dataset should be randomly sampled and sufficiently large
to be representative. Your data, sourced from Ames, Iowa, meets this criterion.
B Minimum Expected Frequency for Each Category: Every cell in the contingency table
should have an expected frequency of at least 5. This is vital for the reliability
of the test, as the chi-squared test relies on a large sample approximation. You will
demonstrate this condition below by creating and visualizing the expected frequencies.

11.2 How the Chi-Squared Test Works

The chi-squared test compares the observed frequencies from data to the expected frequencies
from assumptions. It works by comparing the observed frequencies of the categories in a
contingency table to the expected frequencies that would be expected under the assumption
of independence. The contingency table is a cross-tabulation of the two categorical variables,
showing how many observations fall into each combination of categories.
B Null Hypothesis (H0 ): The null hypothesis in the chi-squared test assumes
independence between the two variables, i.e., the observed frequencies of Great and
Average houses and that of with and without garages should match with each other.
B Alternative Hypothesis (H1 ): The alternative hypothesis suggests that there is a
significant correlation between the two variables, i.e., the observed frequencies (with
or without garage) should differ based on the value of another variable (quality of a
house).
The test statistic in the chi-squared test is calculated by comparing the observed and expected
frequencies in each cell of the contingency table. The larger the difference between observed
and expected frequencies, the larger the chi-squared statistic becomes. The chi-squared test
produces a p-value, which indicates the probability of observing the observed correlation
(or a more extreme one) under the assumption of independence. If the p-value is below a
chosen significance level α (commonly 0.05), the null hypothesis of independence is rejected,
suggesting a significant correlation.

11.3 Unraveling the Correlation Between External Quality

and Garage Presence
Using the Ames housing dataset, you set out to determine whether there’s a correlation
between the external quality of a house and the presence or absence of a garage. Let’s delve
into the specifics of your analysis, supported by the corresponding Python code.
11.3 Unraveling the Correlation Between External Quality and Garage Presence 23

import pandas as pd
from scipy.stats import chi2_contingency

# Load the dataset

Ames = pd.read_csv('Ames.csv')

# Extracting the relevant columns

data = Ames[['ExterQual', 'GarageFinish']].copy()

# Filling missing values in the 'GarageFinish' column with 'No Garage'

data['GarageFinish'] = data['GarageFinish'].fillna('No Garage')

# Grouping 'GarageFinish' into 'With Garage' and 'No Garage'

data['Garage Group'] \
= data['GarageFinish'] \
.apply(lambda x: 'With Garage' if x != 'No Garage' else 'No Garage')

# Grouping 'ExterQual' into 'Great' and 'Average'

data['Quality Group'] \
= data['ExterQual'].apply(lambda x: 'Great' if x in ['Ex', 'Gd'] else 'Average')

# Constructing the simplified contingency table

simplified_contingency_table = pd.crosstab(data['Quality Group'], data['Garage Group'])

#Printing the Observed Frequency

print("Observed Frequencies:")
observed_df = pd.DataFrame(simplified_contingency_table,
index=["Average", "Great"],
columns=["No Garage", "With Garage"])
print(observed_df)
print()

# Performing the chi-squared test

chi2_stat, p_value, _, expected_freq = chi2_contingency(simplified_contingency_table)

# Printing the Expected Frequencies

print("Expected Frequencies:")
print(pd.DataFrame(expected_freq,
index=["Average", "Great"],
columns=["No Garage", "With Garage"]).round(1))
print()

# Printing the results of the test

print(f"Chi-squared Statistic: {chi2_stat:.4f}")
print(f"p-value: {p_value:.4e}")

Listing 11.1: Running chi-squared test using SciPy

The output should be:

Observed Frequencies:
No Garage With Garage
Average 121 1544
Great 8 906
11.4 Important Caveats 24

Expected Frequencies:
No Garage With Garage
Average 83.3 1581.7
Great 45.7 868.3

Chi-squared Statistic: 49.4012

p-value: 2.0862e-12

Output 11.1: Frequencies and the result of chi-squared test

The code above performs three steps:

1. Data Loading & Preparation:

B You began by loading the dataset and extracting the pertinent columns: ExterQual
(Exterior Quality) and GarageFinish (Garage Finish).
B Recognizing the missing values in GarageFinish, you sensibly imputed them with the
label "No Garage", indicating houses devoid of garages.

2. Data Grouping for Simplification:

B You further categorized the GarageFinish data into two groups: “With Garage” (for
houses with any kind of garage) and “No Garage”.
B Similarly, you grouped the ExterQual data into “Great” (houses with excellent or good
exterior quality) and “Average” (houses with average or fair exterior quality).

3. Chi-squared Test:
B With the data aptly prepared, you constructed a contingency table to depict the
observed frequencies between the newly formed categories. They are the two tables
printed in the output.
B You then performed a chi-squared test on this contingency table using SciPy. The p-
value is printed and found much less than α (0.05). The extremely low p-value suggests
rejecting the null hypothesis, meaning there is a statistically significant relationship
between the external quality of a house and the presence of a garage in this dataset.
B A glance at the expected frequencies satisfies the third condition of a chi-squared test,
which requires a minimum of 5 occurrences in each cell.
Through this analysis, you not only refined and simplified the data to make it more interpretable
but also provided statistical evidence of a correlation between two categorical variables of
interest.

11.4 Important Caveats

The chi-squared test, despite its utility, has its limitations:
11.5 Further Reading 25

B No Causation: While the test can determine correlation, it doesn’t infer causation.
So, even though there’s a significant link between the external quality of a house and
its garage presence, you can’t conclude that one causes the other.
B Directionality: The test indicates an correlation but doesn’t specify its direction.
However, your data suggests that houses labeled as “Great” in terms of external
quality are more likely to have garages than those labeled as “Average”.
B Magnitude: The test doesn’t provide insights into the strength of a relationship. Other
metrics, like Cramér’s V, would be more informative in this regard.
B External Validity: Your conclusions are specific to the Ames dataset. Caution is
advised when generalizing these findings to other regions.

11.5 Further Reading

This section provides more resources on the topic if you want to go deeper.

Online
H. B. Berman. Chi-square Test for Independence. Stat Trek.
https://stattrek.com/chi-square-test/independence
Chi-square test. Wikipedia.
https://en.wikipedia.org/wiki/Chi-squared_test
scipy.stats.chi2_contingency API. SciPy.
https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.chi2_contingenc
y.html

11.6 Summary
In this chapter, you delved into the chi-squared test and its application on the Ames housing
dataset. You discovered a significant correlation between the external quality of a house and
the presence of a garage.
Specifically, you learned:
B The fundamentals and practicality of the chi-squared test.
B The chi-squared test revealed a significant correlation between the external quality
of a house and the presence of a garage in the Ames dataset. Houses with a “Great”
external quality rating showed a higher likelihood of having a garage when compared
to those with an “Average” rating, a trend that was statistically significant.
B The vital caveats and limitations of the chi-squared test.
In the next chapter, you will learn about ANOVA, which can be considered as an extension
of the t-test.
This is Just a Sample

Thank-you for your interest in The Beginner’s Guide to Data Science.

This is just a sample of the full text. You can purchase the complete book online from:
https://machinelearningmastery.com/beginners-guide-data-science/

MACHINE LEARNING
MASTERY

THE BEGINNER’S
GUIDE TO
Data
Science
A Journey from Data to
Insight with Statistical
Techniques

Vinod Chugani