MECHANICAL SYLLABUS 21 Regulations Anna University

OCS353 DATA SCIENCE FUNDAMENTALS L T P C 2 0 2 3

COURSE OBJECTIVES:
● Familiarize students with the data science process.
● Understand the data manipulation functions in Numpy and Pandas.
● Explore different types of machine learning approaches.
● Understand and practice visualization techniques using tools.
● Learn to handle large volumes of data with case studies

UNIT I INTRODUCTION 6
Data Science: Benefits and uses – facets of data - Data Science Process: Overview – Defining research
goals – Retrieving data – data preparation - Exploratory Data analysis – build the model – presenting
findings and building applications - Data Mining - Data Warehousing – Basic statistical descriptions of Data

UNIT II DATA MANIPULATION 9

Python Shell - Jupyter Notebook - IPython Magic Commands - NumPy Arrays-Universal Functions –
Aggregations – Computation on Arrays – Fancy Indexing – Sorting arrays – Structured data – Data
manipulation with Pandas – Data Indexing and Selection – Handling missing data – Hierarchical indexing –
Combining datasets – Aggregation and Grouping – String operations – Working with time series – High
performance

UNIT III MACHINE LEARNING 5

The modeling process - Types of machine learning - Supervised learning - Unsupervised learning - Semi-
supervised learning- Classification, regression - Clustering – Outliers and Outlier Analysis

UNIT IV DATA VISUALIZATION 5

Importing Matplotlib – Simple line plots – Simple scatter plots – visualizing errors – density and contour
plots – Histograms – legends – colors – subplots – text and annotation – customization – three dimensional
plotting - Geographic Data with Basemap - Visualization with Seaborn

UNIT V HANDLING LARGE DATA 5

Problems - techniques for handling large volumes of data - programming tips for dealing with large data
sets- Case studies: Predicting malicious URLs, Building a recommender system - Tools and techniques
needed - Research question - Data preparation - Model building – Presentation and automation.

30 PERIODS
PRACTICAL EXERCISES: 30 PERIODS
LAB EXERCISES
1. Download, install and explore the features of Python for data analytics.
2. Working with Numpy arrays
3. Working with Pandas data frames
4. Basic plots using Matplotlib
5. Statistical and Probability measures a) Frequency distributions b) Mean, Mode, Standard Deviation c)
Variability d) Normal curves e) Correlation and scatter plots f) Correlation coefficient g) Regression
6. Use the standard benchmark data set for performing the following: a) Univariate Analysis: Frequency,
Mean, Median, Mode, Variance, Standard Deviation, Skewness and Kurtosis. b) Bivariate Analysis: Linear
and logistic regression modelling.
7. Apply supervised learning algorithms and unsupervised learning algorithms on any data set.
8. Apply and explore various plotting functions on any data set. Note: Example data sets like: UCI, Iris,
Pima Indians Diabetes etc.

COURSE OUTCOMES:
At the end of this course, the students will be able to:
CO1: Gain knowledge on data science process.
CO2: Perform data manipulation functions using Numpy and Pandas.
CO3 Understand different types of machine learning approaches.
CO4: Perform data visualization using tools.
CO5: Handle large volumes of data in practical scenarios.
TOTAL:60 PERIODS

TEXT BOOKS
1. David Cielen, Arno D. B. Meysman, and Mohamed Ali, “Introducing Data
Science”, Manning Publications, 2016.
2. Jake VanderPlas, “Python Data Science Handbook”, O’Reilly, 2016.

REFERENCES
1. Robert S. Witte and John S. Witte, “Statistics”, Eleventh Edition, Wiley
Publications, 2017.
2. Allen B. Downey, “Think Stats: Exploratory Data Analysis in Python”, Green
Tea Press,2014.
UNIT I NOTES

UNIT I : Introduction

Syllabus

Data Science : Benefits and uses - facets of data Defining research goals - Retrieving data -
Data preparation - Exploratory Data analysis - build the model presenting findings and
building applications Warehousing - Basic Statistical descriptions of Data.

Data Science

• Data is measurable units of information gathered or captured from activity of people,

places and things.

• Data science is an interdisciplinary field that seeks to extract knowledge or insights from
various forms of data. At its core, Data Science aims to discover and extract actionable
knowledge from data that can be used to make sound business decisions and predictions.
Data science combines math and statistics, specialized programming, advanced analytics,
Artificial Intelligence (AI) and machine learning with specific subject matter expertise to
uncover actionable insights hidden in an organization's data.

• Data science uses advanced analytical theory and various methods such as time series
analysis for predicting future. From historical data, Instead of knowing how many products
sold in previous quarter, data science helps in forecasting future product sales and revenue
more accurately.

• Data science is devoted to the extraction of clean information from raw data to form
actionable insights. Data science practitioners apply machine learning algorithms to
numbers, text, images, video, audio and more to produce artificial intelligence systems to
perform tasks that ordinarily require human intelligence.

• The data science field is growing rapidly and revolutionizing so many industries. It has
incalculable benefits in business, research and our everyday lives.

• As a general rule, data scientists are skilled in detecting patterns hidden within large
volumes of data and they often use advanced algorithms and implement machine learning
models to help businesses and organizations make accurate assessments and predictions.
Data science and big data evolved from statistics and traditional data management but are
now considered to be distinct disciplines.

• Life cycle of data science:

1. Capture: Data acquisition, data entry, signal reception and data extraction.

2. Maintain Data warehousing, data cleansing, data staging, data processing and data
architecture.

3. Process Data mining, clustering and classification, data modeling and data
summarization.

4. Analyze : Data reporting, data visualization, business intelligence and decision making.

5. Communicate: Exploratory and confirmatory analysis, predictive analysis, regression, text

mining and qualitative analysis.

Big Data
• Big data can be defined as very large volumes of data available at various sources, in
varying degrees of complexity, generated at different speed i.e. velocities and varying
degrees of ambiguity, which cannot be processed using traditional technologies, processing
methods, algorithms or any commercial off-the-shelf solutions.

• 'Big data' is a term used to describe collection of data that is huge in size and yet growing
exponentially with time. In short, such a data is so large and complex that none of the
traditional data management tools are able to store it or process it efficiently.

Characteristics of Big Data

• Characteristics of big data are volume, velocity and variety. They are often referred to as
the three V's.

1. Volume Volumes of data are larger than that conventional relational database
infrastructure can cope with. It consisting of terabytes or petabytes of data.

2. Velocity: The term 'velocity' refers to the speed of generation of data. How fast the data
is generated and processed to meet the demands, determines real potential in the data. It
is being created in or near real-time.
3. Variety: It refers to heterogeneous sources and the nature of data, both structured and
unstructured.

• These three dimensions are also called as three V's of Big Data.

• Two other characteristics of big data is veracity and value.

a) Veracity:

• Veracity refers to source reliability, information credibility and content validity.

• Veracity refers to the trustworthiness of the data. Can the manager rely on the fact that
the data is representative? Every good manager knows that there are inherent
discrepancies in all the data collected.

• Spatial veracity: For vector data (imagery based on points, lines and polygons), the
quality varies. It depends on whether the points have been GPS determined or determined
by unknown origins or manually. Also, resolution and projection issues can alter veracity.

• For geo-coded points, there may be errors in the address tables and in the point location
algorithms associated with addresses.

• For raster data (imagery based on pixels), veracity depends on accuracy of recording
instruments in satellites or aerial devices and on timeliness.

b) Value :

• It represents the business value to be derived from big data.

• The ultimate objective of any big data project should be to generate some sort of value
for the company doing all the analysis. Otherwise, user just performing some technological
task for technology's sake.
• For real-time spatial big data, decisions can be enhance through visualization of dynamic
change in such spatial phenomena as climate, traffic, social-media-based attitudes and
massive inventory locations.

• Exploration of data trends can include spatial proximities and relationships.

• Once spatial big data are structured, formal spatial analytics can be applied, such as
spatial autocorrelation, overlays, buffering, spatial cluster techniques and location
quotients.

Difference between Data Science and Big Data

Comparison between Cloud Computing and Big Data

Benefits and Uses of Data Science

• Data science example and applications :

a) Anomaly detection: Fraud, disease and crime

b) Classification: Background checks; an email server classifying emails as "important"

c) Forecasting: Sales, revenue and customer retention

d) Pattern detection: Weather patterns, financial market patterns

e) Recognition : Facial, voice and text

f) Recommendation: Based on learned preferences, recommendation engines can refer

user to movies, restaurants and books

g) Regression: Predicting food delivery times, predicting home prices based on amenities
h) Optimization: Scheduling ride-share pickups and package deliveries

Benefits and Use of Big Data

• Benefits of Big Data :

1. Improved customer service

2. Businesses can utilize outside intelligence while taking decisions

3. Reducing maintenance costs

4. Re-develop our products : Big Data can also help us understand how others perceive our
products so that we can adapt them or our marketing, if need be.

5. Early identification of risk to the product/services, if any

6. Better operational efficiency

• Some of the examples of big data are:

1. Social media : Social media is one of the biggest contributors to the flood of data we
have today. Facebook generates around 500+ terabytes of data everyday in the form of
content generated by the users like status messages, photos and video uploads, messages,
comments etc.

2. Stock exchange : Data generated by stock exchanges is also in terabytes per day. Most of
this data is the trade data of users and companies.

3. Aviation industry: A single jet engine can generate around 10 terabytes of data during a
30 minute flight.

4. Survey data: Online or offline surveys conducted on various topics which typically has
hundreds and thousands of responses and needs to be processed for analysis and
visualization by creating a cluster of population and their associated responses.

5. Compliance data : Many organizations like healthcare, hospitals, life sciences, finance etc
has to file compliance reports.

Foundation of Data Science: Unit I: Introduction : Tag: : Definition, Characteristics, Comparison,

Benefits, Uses - Data Science and Big Data
Facets of Data

• Very large amount of data will generate in big data and data science. These data is various
types and main categories of data are as follows:

a) Structured

b) Natural language

c) Graph-based

d) Streaming

e) Unstructured

f) Machine-generated

g) Audio, video and images

Structured Data
• Structured data is arranged in rows and column format. It helps for application to retrieve
and process data easily. Database management system is used for storing structured data.

• The term structured data refers to data that is identifiable because it is organized in a
structure. The most common form of structured data or records is a database where
specific information is stored based on a methodology of columns and rows.

• Structured data is also searchable by data type within content. Structured data is
understood by computers and is also efficiently organized for human readers.

• An Excel table is an example of structured data.

Unstructured Data
• Unstructured data is data that does not follow a specified format. Row and columns are
not used for unstructured data. Therefore it is difficult to retrieve required information.
Unstructured data has no identifiable structure.

• The unstructured data can be in the form of Text: (Documents, email messages, customer
feedbacks), audio, video, images. Email is an example of unstructured data.
• Even today in most of the organizations more than 80 % of the data are in unstructured
form. This carries lots of information. But extracting information from these various sources
is a very big challenge.

• Characteristics of unstructured data:

1. There is no structural restriction or binding for the data.

2. Data can be of any type.

3. Unstructured data does not follow any structural rules.

4. There are no predefined formats, restriction or sequence for unstructured data.

5. Since there is no structural binding for unstructured data, it is unpredictable in nature.

Natural Language
• Natural language is a special type of unstructured data.

• Natural language processing enables machines to recognize characters, words and

sentences, then apply meaning and understanding to that information. This helps machines
to understand language as humans do.

• Natural language processing is the driving force behind machine intelligence in many
modern real-world applications. The natural language processing community has had
success in entity recognition, topic recognition, summarization, text completion and
sentiment analysis.

•For natural language processing to help machines understand human language, it must go
through speech recognition, natural language understanding and machine translation. It is
an iterative process comprised of several layers of text analysis.

Machine - Generated Data

• Machine-generated data is an information that is created without human interaction as a
result of a computer process or application activity. This means that data entered manually
by an end-user is not recognized to be machine-generated.

• Machine data contains a definitive record of all activity and behavior of our customers,
users, transactions, applications, servers, networks, factory machinery and so on.
• It's configuration data, data from APIs and message queues, change events, the output of
diagnostic commands and call detail records, sensor data from remote equipment and
more.

• Examples of machine data are web server logs, call detail records, network event logs and
telemetry.

• Both Machine-to-Machine (M2M) and Human-to-Machine (H2M) interactions generate

machine data. Machine data is generated continuously by every processor-based system, as
well as many consumer-oriented systems.

• It can be either structured or unstructured. In recent years, the increase of machine data
has surged. The expansion of mobile devices, virtual servers and desktops, as well as cloud-
based services and RFID technologies, is making IT infrastructures more complex.

Graph-based or Network Data

•Graphs are data structures to describe relationships and interactions between entities in
complex systems. In general, a graph contains a collection of entities called nodes and
another collection of interactions between a pair of nodes called edges.

• Nodes represent entities, which can be of any object type that is relevant to our problem
domain. By connecting nodes with edges, we will end up with a graph (network) of nodes.

• A graph database stores nodes and relationships instead of tables or documents. Data is
stored just like we might sketch ideas on a whiteboard. Our data is stored without
restricting it to a predefined model, allowing a very flexible way of thinking about and using
it.

• Graph databases are used to store graph-based data and are queried with specialized
query languages such as SPARQL.

• Graph databases are capable of sophisticated fraud prevention. With graph databases,
we can use relationships to process financial and purchase transactions in near-real time.
With fast graph queries, we are able to detect that, for example, a potential purchaser is
using the same email address and credit card as included in a known fraud case.

• Graph databases can also help user easily detect relationship patterns such as multiple
people associated with a personal email address or multiple people sharing the same IP
address but residing in different physical addresses.
• Graph databases are a good choice for recommendation applications. With graph
databases, we can store in a graph relationships between information categories such as
customer interests, friends and purchase history. We can use a highly available graph
database to make product recommendations to a user based on which products are
purchased by others who follow the same sport and have similar purchase history.

• Graph theory is probably the main method in social network analysis in the early history
of the social network concept. The approach is applied to social network analysis in order to
determine important features of the network such as the nodes and links (for example
influencers and the followers).

• Influencers on social network have been identified as users that have impact on the
activities or opinion of other users by way of followership or influence on decision made by
other users on the network as shown in Fig. 1.2.1.

• Graph theory has proved to be very effective on large-scale datasets such as social
network data. This is because it is capable of by-passing the building of an actual visual
representation of the data to run directly on data matrices.
Audio, Image and Video
• Audio, image and video are data types that pose specific challenges to a data scientist.
Tasks that are trivial for humans, such as recognizing objects in pictures, turn out to be
challenging for computers.

•The terms audio and video commonly refers to the time-based media storage format for
sound/music and moving pictures information. Audio and video digital recording, also
referred as audio and video codecs, can be uncompressed, lossless compressed or lossy
compressed depending on the desired quality and use cases.

• It is important to remark that multimedia data is one of the most important sources of
information and knowledge; the integration, transformation and indexing of multimedia
data bring significant challenges in data management and analysis. Many challenges have to
be addressed including big data, multidisciplinary nature of Data Science and
heterogeneity.

• Data Science is playing an important role to address these challenges in multimedia data.
Multimedia data usually contains various forms of media, such as text, image, video,
geographic coordinates and even pulse waveforms, which come from multiple sources.
Data Science can be a key instrument covering big data, machine learning and data mining
solutions to store, handle and analyze such heterogeneous data.

Streaming Data
Streaming data is data that is generated continuously by thousands of data sources, which
typically send in the data records simultaneously and in small sizes (order of Kilobytes).

• Streaming data includes a wide variety of data such as log files generated by customers
using your mobile or web applications, ecommerce purchases, in-game player activity,
information from social networks, financial trading floors or geospatial services and
telemetry from connected devices or instrumentation in data centers.
Difference between Structured and Unstructured Data

Data Science Process

Data science process consists of six stages :

1. Discovery or Setting the research goal

2. Retrieving data

3. Data preparation

4. Data exploration

5. Data modeling

6. Presentation and automation

• Fig. 1.3.1 shows data science design process.

• Step 1: Discovery or Defining research goal

This step involves acquiring data from all the identified internal and external sources, which
helps to answer the business question.

• Step 2: Retrieving data

It collection of data which required for project. This is the process of gaining a business
understanding of the data user have and deciphering what each piece of data means. This
could entail determining exactly what data is required and the best methods for obtaining it.
This also entails determining what each of the data points means in terms of the company. If
we have given a data set from a client, for example, we shall need to know what each column
and row represents.

• Step 3: Data preparation

Data can have many inconsistencies like missing values, blank columns, an incorrect data
format, which needs to be cleaned. We need to process, explore and condition data before
modeling. The cleandata, gives the better predictions.

• Step 4: Data exploration

Data exploration is related to deeper understanding of data. Try to understand how variables
interact with each other, the distribution of the data and whether there are outliers. To
achieve this use descriptive statistics, visual techniques and simple modeling. This steps is
also called as Exploratory Data Analysis.

• Step 5: Data modeling

In this step, the actual model building process starts. Here, Data scientist distributes datasets
for training and testing. Techniques like association, classification and clustering are applied
to the training data set. The model, once prepared, is tested against the "testing" dataset.

• Step 6: Presentation and automation

Deliver the final baselined model with reports, code and technical documents in this stage.
Model is deployed into a real-time production environment after thorough testing. In this
stage, the key findings are communicated to all stakeholders. This helps to decide if the
project results are a success or a failure based on the inputs from the model.

Defining Research Goals

• To understand the project, three concept must understand: what, why and how.

a) What is expectation of company or organization?

b) Why does a company's higher authority define such research value?

c) How is it part of a bigger strategic picture?

• Goal of first phase will be the answer of these three questions.

• In this phase, the data science team must learn and investigate the problem, develop context
and understanding and learn about the data sources needed and available for the project.

1. Learning the business domain :

• Understanding the domain area of the problem is essential. In many cases, data scientists
will have deep computational and quantitative knowledge that can be broadly applied across
many disciplines.

• Data scientists have deep knowledge of the methods, techniques and ways for applying
heuristics to a variety of business and conceptual problems.

2. Resources :
• As part of the discovery phase, the team needs to assess the resources available to support
the project. In this context, resources include technology, tools, systems, data and people.

3. Frame the problem :

• Framing is the process of stating the analytics problem to be solved. At this point, it is a
best practice to write down the problem statement and share it with the key stakeholders.

• Each team member may hear slightly different things related to the needs and the problem
and have somewhat different ideas of possible solutions.

4. Identifying key stakeholders:

• The team can identify the success criteria, key risks and stakeholders, which should include
anyone who will benefit from the project or will be significantly impacted by the project.

• When interviewing stakeholders, learn about the domain area and any relevant history from
similar analytics projects.

5. Interviewing the analytics sponsor:

• The team should plan to collaborate with the stakeholders to clarify and frame the analytics
problem.

• At the outset, project sponsors may have a predetermined solution that may not necessarily
realize the desired outcome.

• In these cases, the team must use its knowledge and expertise to identify the true
underlying problem and appropriate solution.

• When interviewing the main stakeholders, the team needs to take time to thoroughly
interview the project sponsor, who tends to be the one funding the project or providing the
high-level requirements.

• This person understands the problem and usually has an idea of a potential working
solution.

6. Developing initial hypotheses:

• This step involves forming ideas that the team can test with data. Generally, it is best to
come up with a few primary hypotheses to test and then be creative about developing several
more.
• These Initial Hypotheses form the basis of the analytical tests the team will use in later
phases and serve as the foundation for the findings in phase.

7. Identifying potential data sources:

• Consider the volume, type and time span of the data needed to test the hypotheses. Ensure
that the team can access more than simply aggregated data. In most cases, the team will need
the raw data to avoid introducing bias for the downstream analysis.

Retrieving Data

• Retrieving required data is second phase of data science project. Sometimes Data scientists
need to go into the field and design a data collection process. Many companies will have
already collected and stored the data and what they don't have can often be bought from third
parties.

• Most of the high quality data is freely available for public and commercial use. Data can be
stored in various format. It is in text file format and tables in database. Data may be internal
or external.

1. Start working on internal data, i.e. data stored within the company

• First step of data scientists is to verify the internal data. Assess the relevance and quality of
the data that's readily in company. Most companies have a program for maintaining key data,
so much of the cleaning work may already be done. This data can be stored in official data
repositories such as databases, data marts, data warehouses and data lakes maintained by a
team of IT professionals.

• Data repository is also known as a data library or data archive. This is a general term to
refer to a data set isolated to be mined for data reporting and analysis. The data repository is
a large database infrastructure, several databases that collect, manage and store data sets for
data analysis, sharing and reporting.

• Data repository can be used to describe several ways to collect and store data:

a) Data warehouse is a large data repository that aggregates data usually from multiple
sources or segments of a business, without the data being necessarily related.

b) Data lake is a large data repository that stores unstructured data that is classified and
tagged with metadata.
c) Data marts are subsets of the data repository. These data marts are more targeted to what
the data user needs and easier to use.

d) Metadata repositories store data about data and databases. The metadata explains where
the data source, how it was captured and what it represents.

e) Data cubes are lists of data with three or more dimensions stored as a table.

Advantages of data repositories:

i. Data is preserved and archived.

ii. Data isolation allows for easier and faster data reporting.

iii. Database administrators have easier time tracking problems.

iv. There is value to storing and analyzing data.

Disadvantages of data repositories :

i. Growing data sets could slow down systems.

ii. A system crash could affect all the data.

iii. Unauthorized users can access all sensitive data more easily than if it was distributed
across several locations.

2. Do not be afraid to shop around

• If required data is not available within the company, take the help of other company, which
provides such types of database. For example, Nielsen and GFK are provides data for retail
industry. Data scientists also take help of Twitter, LinkedIn and Facebook.

• Government's organizations share their data for free with the world. This data can be of
excellent quality; it depends on the institution that creates and manages it. The information
they share covers a broad range of topics such as the number of accidents or amount of drug
abuse in a certain region and its demographics.

3. Perform data quality checks to avoid later problem

• Allocate or spend some time for data correction and data cleaning. Collecting suitable,
error free data is success of the data science project.
• Most of the errors encounter during the data gathering phase are easy to spot, but being too
careless will make data scientists spend many hours solving data issues that could have been
prevented during data import.

• Data scientists must investigate the data during the import, data preparation and exploratory
phases. The difference is in the goal and the depth of the investigation.

• In data retrieval process, verify whether the data is right data type and data is same as in the
source document.

• With data preparation process, more elaborate checks performed. Check any shortcut
method is used. For example, check time and data format.

• During the exploratory phase, Data scientists focus shifts to what he/she can learn from the
data. Now Data scientists assume the data to be clean and look at the statistical properties
such as distributions, correlations and outliers.

Data Preparation

• Data preparation means data cleansing, Integrating and transforming data.

Data Cleaning
• Data is cleansed through processes such as filling in missing values, smoothing the noisy
data or resolving the inconsistencies in the data.

• Data cleaning tasks are as follows:

1. Data acquisition and metadata

2. Fill in missing values

3. Unified date format

4. Converting nominal to numeric

5. Identify outliers and smooth out noisy data

6. Correct inconsistent data

• Data cleaning is a first step in data pre-processing techniques which is used to find the
missing value, smooth noise data, recognize outliers and correct inconsistent.

• Missing value: These dirty data will affects on miming procedure and led to unreliable
and poor output. Therefore it is important for some data cleaning routines. For example,
suppose that the average salary of staff is Rs. 65000/-. Use this value to replace the missing
value for salary.

• Data entry errors: Data collection and data entry are error-prone processes. They often
require human intervention and because humans are only human, they make typos or lose
their concentration for a second and introduce an error into the chain. But data collected
by machines or computers isn't free from errors either. Errors can arise from human
sloppiness, whereas others are due to machine or hardware failure. Examples of errors
originating from machines are transmission errors or bugs in the extract, transform and
load phase (ETL).

• Whitespace error: Whitespaces tend to be hard to detect but cause errors like other
redundant characters would. To remove the spaces present at start and end of the string,
we can use strip() function on the string in Python.

• Fixing capital letter mismatches: Capital letter mismatches are common problem. Most
programming languages make a distinction between "Chennai" and "chennai".

• Python provides string conversion like to convert a string to lowercase, uppercase using
lower(), upper().

• The lower() Function in python converts the input string to lowercase. The upper()
Function in python converts the input string to uppercase.

Outlier
• Outlier detection is the process of detecting and subsequently excluding outliers from a
given set of data. The easiest way to find outliers is to use a plot or a table with the
minimum and maximum values.

• Fig. 1.6.1 shows outliers detection. Here O1 and O2 seem outliers from the rest.
• An outlier may be defined as a piece of data or observation that deviates drastically from
the given norm or average of the data set. An outlier may be caused simply by chance, but
it may also indicate measurement error or that the given data set has a heavy-tailed
distribution.

• Outlier analysis and detection has various applications in numerous fields such as fraud
detection, credit card, discovering computer intrusion and criminal behaviours, medical and
public health outlier detection, industrial damage detection.

• General idea of application is to find out data which deviates from normal behaviour of
data set.

Dealing with Missing Value

• These dirty data will affects on miming procedure and led to unreliable and poor output.
Therefore it is important for some data cleaning routines.

How to handle noisy data in data mining?

• Following methods are used for handling noisy data:

1. Ignore the tuple: Usually done when the class label is missing. This method is not good
unless the tuple contains several attributes with missing values.

2. Fill in the missing value manually : It is time-consuming and not suitable for a large data
set with many missing values.
3. Use a global constant to fill in the missing value: Replace all missing attribute values by
the same constant.

4. Use the attribute mean to fill in the missing value: For example, suppose that the
average salary of staff is Rs 65000/-. Use this value to replace the missing value for salary.

5. Use the attribute mean for all samples belonging to the same class as the given tuple.

6. Use the most probable value to fill in the missing value.

Correct Errors as Early as Possible

• If error is not corrected in early stage of project, then it create problem in latter stages.
Most of the time, we spend on finding and correcting error. Retrieving data is a difficult task
and organizations spend millions of dollars on it in the hope of making better decisions. The
data collection process is errorprone and in a big organization it involves many steps and
teams.

• Data should be cleansed when acquired for many reasons:

a) Not everyone spots the data anomalies. Decision-makers may make costly mistakes on
information based on incorrect data from applications that fail to correct for the faulty
data.

b) If errors are not corrected early on in the process, the cleansing will have to be done for
every project that uses that data.

c) Data errors may point to a business process that isn't working as designed.

d) Data errors may point to defective equipment, such as broken transmission lines and
defective sensors.

e) Data errors can point to bugs in software or in the integration of software that may be
critical to the company

Combining Data from Different Data Sources

1. Joining table
• Joining tables allows user to combine the information of one observation found in one
table with the information that we find in another table. The focus is on enriching a single
observation.

• A primary key is a value that cannot be duplicated within a table. This means that one
value can only be seen once within the primary key column. That same key can exist as a
foreign key in another table which creates the relationship. A foreign key can have
duplicate instances within a table.

• Fig. 1.6.2 shows Joining two tables on the CountryID and CountryName keys.

2. Appending tables

• Appending table is called stacking table. It effectively adding observations from one table
to another table. Fig. 1.6.3 shows Appending table. (See Fig. 1.6.3 on next page)
• Table 1 contains x3 value as 3 and Table 2 contains x3 value as 33.The result of appending
these tables is a larger one with the observations from Table 1 as well as Table 2. The
equivalent operation in set theory would be the union and this is also the command in SQL,
the common language of relational databases. Other set operators are also used in data
science, such as set difference and intersection.

3. Using views to simulate data joins and appends

• Duplication of data is avoided by using view and append. The append table requires more
space for storage. If table size is in terabytes of data, then it becomes problematic to
duplicate the data. For this reason, the concept of a view was invented.

• Fig. 1.6.4 shows how the sales data from the different months is combined virtually into a
yearly sales table instead of duplicating the data.
Transforming Data
• In data transformation, the data are transformed or consolidated into forms appropriate
for mining. Relationships between an input variable and an output variable aren't always
linear.

• Reducing the number of variables: Having too many variables in the model makes the
model difficult to handle and certain techniques don't perform well when user overload
them with too many input variables.

• All the techniques based on a Euclidean distance perform well only up to 10 variables.
Data scientists use special methods to reduce the number of variables but retain the
maximum amount of data.

Euclidean distance :

• Euclidean distance is used to measure the similarity between observations. It is calculated

as the square root of the sum of differences between each point.

Euclidean distance = √(X1-X2)2 + (Y1-Y2)2

Turning variable into dummies :

• Variables can be turned into dummy variables. Dummy variables canonly take two values:
true (1) or false√ (0). They're used to indicate the absence of acategorical effect that may
explain the observation.
Exploratory Data Analysis

• Exploratory Data Analysis (EDA) is a general approach to exploring datasets by means of

simple summary statistics and graphic visualizations in order to gain a deeper understanding
of data.

• EDA is used by data scientists to analyze and investigate data sets and summarize their
main characteristics, often employing data visualization methods. It helps determine how
best to manipulate data sources to get the answers user need, making it easier for data
scientists to discover patterns, spot anomalies, test a hypothesis or check assumptions.

• EDA is an approach/philosophy for data analysis that employs a variety of techniques to:

1. Maximize insight into a data set;

2. Uncover underlying structure;

 3. Extract important variables;

4. Detect outliers and anomalies;

5. Test underlying assumptions;

6. Develop parsimonious models; and

7. Determine optimal factor settings.

• With EDA, following functions are performed:

1. Describe of user data

2. Closely explore data distributions

3. Understand the relations between variables

4. Notice unusual or unexpected situations

5. Place the data into groups

6. Notice unexpected patterns within groups

7. Take note of group differences

• Box plots are an excellent tool for conveying location and variation information in data
sets, particularly for detecting and illustrating location and variation changes between
different groups of data.

• Exploratory data analysis is majorly performed using the following methods:

1. Univariate analysis: Provides summary statistics for each field in the raw data set
(or) summary only on one variable. Ex : CDF,PDF,Box plot

2. Bivariate analysis is performed to find the relationship between each variable in the
dataset and the target variable of interest (or) using two variables and finding relationship
between them. Ex: Boxplot, Violin plot.

3. Multivariate analysis is performed to understand interactions between different fields in

the dataset (or) finding interactions between variables more than 2.
• A box plot is a type of chart often used in explanatory data analysis to visually show the
distribution of numerical data and skewness through displaying the data quartiles or
percentile and averages.

1. Minimum score: The lowest score, exlcuding outliers.

2. Lower quartile : 25% of scores fall below the lower quartile value.

3. Median: The median marks the mid-point of the data and is shown by the line that divides
the box into two parts.

4. Upper quartile : 75 % of the scores fall below the upper quartiel value.

5. Maximum score: The highest score, excluding outliers.

6. Whiskers: The upper and lower whiskers represent scores outside the middle 50%.

7. The interquartile range: This is the box plot showing the middle 50% of scores.

• Boxplots are also extremely usefule for visually checking group differences. Suppose we
have four groups of scores and we want to compare them by teaching method. Teaching
method is our categorical grouping variable and score is the continuous outcomes variable
that the researchers measured.
Build the Models

• To build the model, data should be clean and understand the content properly. The
components of model building are as follows:

a) Selection of model and variable

b) Execution of model

c) Model diagnostic and model comparison

• Building a model is an iterative process. Most models consist of the following main steps:

1. Selection of a modeling technique and variables to enter in the model

2. Execution of the model

3. Diagnosis and model comparison

Model and Variable Selection
• For this phase, consider model performance and whether project meets all the
requirements to use model, as well as other factors:

1. Must the model be moved to a production environment and, if so, would it be easy to
implement?

2. How difficult is the maintenance on the model: how long will it remain relevantif left
untouched?

3. Does the model need to be easy to explain?

Model Execution
• Various programming language is used for implementing the model. For model execution,
Python provides libraries like StatsModels or Scikit-learn. These packages use several of the
most popular techniques.

• Coding a model is a nontrivial task in most cases, so having these libraries available can
speed up the process. Following are the remarks on output:

a) Model fit: R-squared or adjusted R-squared is used.

b) Predictor variables have a coefficient: For a linear model this is easy to interpret.

c) Predictor significance: Coefficients are great, but sometimes not enough evidence exists
to show that the influence is there.

• Linear regression works if we want to predict a value, but for classify something,
classification models are used. The k-nearest neighbors method is one of the best method.

• Following commercial tools are used :

1. SAS enterprise miner: This tool allows users to run predictive and descriptive models
based on large volumes of data from across the enterprise.

2. SPSS modeler: It offers methods to explore and analyze data through a GUI.

3. Matlab: Provides a high-level language for performing a variety of data analytics,

algorithms and data exploration.
4. Alpine miner: This tool provides a GUI front end for users to develop analytic workflows
and interact with Big Data tools and platforms on the back end.

• Open Source tools:

1. R and PL/R: PL/R is a procedural language for PostgreSQL with R.

2. Octave: A free software programming language for computational modeling, has some of
the functionality of Matlab.

3. WEKA: It is a free data mining software package with an analytic workbench. The
functions created in WEKA can be executed within Java code.

4. Python is a programming language that provides toolkits for machine learning and
analysis.

5. SQL in-database implementations, such as MADlib provide an alterative to in memory

desktop analytical tools.

Model Diagnostics and Model Comparison

Try to build multiple model and then select best one based on multiple criteria. Working
with a holdout sample helps user pick the best-performing model.

• In Holdout Method, the data is split into two different datasets labeled as a training and a
testing dataset. This can be a 60/40 or 70/30 or 80/20 split. This technique is called the
hold-out validation technique.

Suppose we have a database with house prices as the dependent variable and two
independent variables showing the square footage of the house and the number of rooms.
Now, imagine this dataset has 30 rows. The whole idea is that you build a model that can
predict house prices accurately.

• To 'train' our model or see how well it performs, we randomly subset 20 of those rows
and fit the model. The second step is to predict the values of those 10 rows that we
excluded and measure how well our predictions were.

• As a rule of thumb, experts suggest to randomly sample 80% of the data into the training
set and 20% into the test set.

• The holdout method has two, basic drawbacks :

1. It requires extra dataset.

2. It is a single train-and-test experiment, the holdout estimate of error rate will be

misleading if we happen to get an "unfortunate" split.

Presenting Findings and Building Applications

• The team delivers final reports, briefings, code and technical documents.

• In addition, team may run a pilot project to implement the models in a production
environment.

• The last stage of the data science process is where user soft skills will be most useful.

• Presenting your results to the stakeholders and industrializing your analysis process for
repetitive reuse and integration with other tools.

Data Mining

• Data mining refers to extracting or mining knowledge from large amounts of data. It is a
process of discovering interesting patterns or Knowledge from a large amount of data
stored either in databases, data warehouses or other information repositories.

Reasons for using data mining:

1. Knowledge discovery: To identify the invisible correlation, patterns in the database.

2. Data visualization: To find sensible way of displaying data.

3. Data correction: To identify and correct incomplete and inconsistent data.

Functions of Data Mining

• Different functions of data mining are characterization, association and correlation
analysis, classification, prediction, clustering analysis and evolution analysis.
1. Characterization is a summarization of the general characteristics or features of a target
class of data. For example, the characteristics of students can be produced, generating a
profile of all the University in first year engineering students.

2. Association is the discovery of association rules showing attribute-value conditions that

occur frequently together in a given set of data.

3. Classification differs from prediction. Classification constructs a set of models that

describe and distinguish data classes and prediction builds a model to predict some missing
data values.

4. Clustering can also support taxonomy formation. The organization of observations into a
hierarchy of classes that group similar events together.

5. Data evolution analysis describes and models' regularities for objects whose behaviour
changes over time. It may include characterization, discrimination, association,
classification or clustering of time-related data.

Data mining tasks can be classified into two categories: descriptive and predictive.

Predictive Mining Tasks

• To make prediction, predictive mining tasks performs inference on the current data.
Predictive analysis provides answers of the future queries that move across using historical
data as the chief principle for decisions

• It involves the supervised learning functions used for the prediction of the target value.
The methods fall under this mining category are the classification, time-series analysis and
regression.

• Data modeling is the necessity of the predictive analysis, which works by utilizing some
variables to anticipate the unknown future data values for other variables.

• It provides organizations with actionable insights based on data. It provides an estimation

regarding the likelihood of a future outcome.

• To do this, a variety of techniques are used, such as machine learning, data mining,
modeling and game theory.

• Predictive modeling can, for example, help to identify any risks or opportunities in the
future.
• Predictive analytics can be used in all departments, from predicting customer behaviour
in sales and marketing, to forecasting demand for operations or determining risk profiles
for finance.

• A very well-known application of predictive analytics is credit scoring used by financial

services to determine the likelihood of customers making future credit payments on time.
Determining such a risk profile requires a vast amount of data, including public and social
data.

• Historical and transactional data are used to identify patterns and statistical models and
algorithms are used to capture relationships in various datasets.

• Predictive analytics has taken off in the big data era and there are many tools available for
organisations to predict future outcomes.

Descriptive Mining Task

• Descriptive Analytics is the conventional form of business intelligence and data analysis,
seeks to provide a depiction or "summary view" of facts and figures in an understandable
format, to either inform or prepare data for further analysis.

• Two primary techniques are used for reporting past events : data aggregation and data
mining.

• It presents past data in an easily digestible format for the benefit of a wide business
audience.

• A set of techniques for reviewing and examining the data set to understand the data and
analyze business performance.

• Descriptive analytics helps organisations to understand what happened in the past. It

helps to understand the relationship between product and customers.

• The objective of this analysis is to understanding, what approach to take in the future. If
we learn from past behaviour, it helps us to influence future outcomes.

• It also helps to describe and present data in such format, which can be easily understood
by a wide variety of business readers.
Architecture of a Typical Data Mining System
• Data mining refers to extracting or mining knowledge from large amounts of data. It is a
process of discovering interesting patterns or knowledge from a large amount of data
stored either in databases, data warehouses.

• It is the computational process of discovering patterns in huge data sets involving

methods at the intersection of AI, machine learning, statistics and database systems.

• Fig. 1.10.1 (See on next page) shows typical architecture of data mining system.

• Components of data mining system are data source, data warehouse server, data mining
engine, pattern evaluation module, graphical user interface and knowledge base.

• Database, data warehouse, WWW or other information repository: This is set of

databases, data warehouses, spreadsheets or other kinds of data repositories. Data
cleaning and data integration techniques may be apply on the data.

• Data warehouse server based on the user's data request, data warehouse server is
responsible for fetching the relevant data.
• Knowledge base is helpful in the whole data mining process. It might be useful for guiding
the search or evaluating the interestingness of the result patterns. The knowledge base
might even contain user beliefs and data from user experiences that can be useful in the
process of data mining.

• The data mining engine is the core component of any data mining system. It consists of a
number of modules for performing data mining tasks including association, classification,
characterization, clustering, prediction, time-series analysis etc.

• The pattern evaluation module is mainly responsible for the measure of interestingness of
the pattern by using a threshold value. It interacts with the data mining engine to focus the
search towards interesting patterns.

• The graphical user interface module communicates between the user and the data mining
system. This module helps the user use the system easily and efficiently without knowing
the real complexity behind the process.

• When the user specifies a query or a task, this module interacts with the data mining
system and displays the result in an easily understandable manner.

Classification of DM System
• Data mining system can be categorized according to various parameters. These are
database technology, machine learning, statistics, information science, visualization and
other disciplines.

• Fig. 1.10.2 shows classification of DM system.

• Multi-dimensional view of data mining classification.

Data Warehousing

• Data warehousing is the process of constructing and using a data warehouse. A data
warehouse is constructed by integrating data from multiple heterogeneous sources that
support analytical reporting, structured and/or ad hoc queries and decision making. Data
warehousing involves data cleaning, data integration and data consolidations.

• A data warehouse is a subject-oriented, integrated, time-variant and non-volatile

collection of data in support of management's decision-making process. A data warehouse
stores historical data for purposes of decision support.

• A database an application-oriented collection of data that is organized, structured,

coherent, with minimum and controlled redundancy, which may be accessed by several
users in due time.

• Data warehousing provides architectures and tools for business executives to

systematically organize, understand and use their data to make strategic decisions.
• A data warehouse is a subject-oriented collection of data that is integrated, time-variant,
non-volatile, which may be used to support the decision-making process.

• Data warehouses are databases that store and maintain analytical data separately from
transaction-oriented databases for the purpose of decision support. Data warehouses
separate analysis workload from transaction workload and enable an organization to
consolidate data from several source.

• Data organization in data warehouses is based on areas of interest, on the major subjects
of the organization: Customers, products, activities etc. databases organize data based on
enterprise applications resulted from its functions.

• The main objective of a data warehouse is to support the decision-making system,

focusing on the subjects of the organization. The objective of a database is to support the
operational system and information is organized on applications and processes.

• A data warehouse usually stores many months or years of data to support historical
analysis. The data in a data warehouse is typically loaded through an extraction,
transformation and loading (ETL) process from multiple data sources.

• Databases and data warehouses are related but not the same.

• A database is a way to record and access information from a single source. A database is
often handling real-time data to support day-to-day business processes like transaction
processing.

• A data warehouse is a way to store historical information from multiple sources to allow
you to analyse and report on related data (e.g., your sales transaction data, mobile app
data and CRM data). Unlike a database, the information isn't updated in real-time and is
better for data analysis of broader trends.

• Modern data warehouses are moving toward an Extract, Load, Transformation (ELT)
architecture in which all or most data transformation is performed on the database that
hosts the data warehouse.

• Goals of data warehousing:

1. To help reporting as well as analysis.

2. Maintain the organization's historical information.

3. Be the foundation for decision making.

"How are organizations using the information from data warehouses ?"

• Most of the organizations makes use of this information for taking business decision like :

a) Increasing customer focus: It is possible by performing analysis of customer buying.

b) Repositioning products and managing product portfolios by comparing the performance

of last year sales.

c) Analysing operations and looking for sources of profit.

d) Managing customer relationships, making environmental corrections and managing the

cost of corporate assets.

Characteristics of Data Warehouse

1. Subject oriented Data are organized based on how the users refer to them. A data
warehouse can be used to analyse a particular subject area. For example, "sales" can be a
particular subject.

2. Integrated: All inconsistencies regarding naming convention and value representations

are removed. For example, source A and source B may have different ways of identifying a
product, but in a data warehouse, there will be only a single way of identifying a product.

3. Non-volatile: Data are stored in read-only format and do not change over time. Typical
activities such as deletes, inserts and changes that are performed in an operational
application environment are completely non-existent in a DW environment.

4. Time variant : Data are not current but normally time series. Historical information is
kept in a data warehouse. For example, one can retrieve files from 3 months, 6 months, 12
months or even previous data from a data warehouse.

Key characteristics of a Data Warehouse

1. Data is structured for simplicity of access and high-speed query performance.

2. End users are time-sensitive and desire speed-of-thought response times.

3. Large amounts of historical data are used.

4. Queries often retrieve large amounts of data, perhaps many thousands of rows.
5. Both predefined and ad hoc queries are common.

6. The data load involves multiple sources and transformations.

Multitier Architecture of Data Warehouse

• Data warehouse architecture is a data storage framework's design of an organization. A
data warehouse architecture takes information from raw sets of data and stores it in a
structured and easily digestible format.

• Data warehouse system is constructed in three ways. These approaches are classified the
number of tiers in the architecture.

a) Single-tier architecture.

b) Two-tier architecture.

c) Three-tier architecture (Multi-tier architecture).

• Single tier warehouse architecture focuses on creating a compact data set and minimizing
the amount of data stored. While it is useful for removing redundancies. It is not effective
for organizations with large data needs and multiple streams.

• Two-tier warehouse structures separate the resources physically available from the
warehouse itself. This is most commonly used in small organizations where a server is used
as a data mart. While it is more effective at storing and sorting data. Two-tier is not scalable
and it supports a minimal number of end-users.

Three tier (Multi-tier) architecture:

• Three tier architecture creates a more structured flow for data from raw sets to
actionable insights. It is the most widely used architecture for data warehouse systems.

• Fig. 1.11.1 shows three tier architecture. Three tier architecture sometimes called multi-
tier architecture.

• The bottom tier is the database of the warehouse, where the cleansed and transformed
data is loaded. The bottom tier is a warehouse database server.
• The middle tier is the application layer giving an abstracted view of the database. It
arranges the data to make it more suitable for analysis. This is done with an OLAP server,
implemented using the ROLAP or MOLAP model.

• OLAPS can interact with both relational databases and multidimensional databases, which
lets them collect data better based on broader parameters.

• The top tier is the front-end of an organization's overall business intelligence suite. The
top-tier is where the user accesses and interacts with data via queries, data visualizations
and data analytics tools.

• The top tier represents the front-end client layer. The client level which includes the tools
and Application Programming Interface (API) used for high-level data analysis, inquiring and
reporting. User can use reporting tools, query, analysis or data mining tools.
Needs of Data Warehouse
1) Business user: Business users require a data warehouse to view summarized data from
the past. Since these people are non-technical, the data may be presented to them in an
elementary form.

2) Store historical data: Data warehouse is required to store the time variable data from the
past. This input is made to be used for various purposes.

3) Make strategic decisions: Some strategies may be depending upon the data in the data
warehouse. So, data warehouse contributes to making strategic decisions.

4) For data consistency and quality Bringing the data from different sources at a
commonplace, the user can effectively undertake to bring the uniformity and consistency in
data.

5) High response time: Data warehouse has to be ready for somewhat unexpected loads
and types of queries, which demands a significant degree of flexibility and quick response
time.

Benefits of Data Warehouse

a) Understand business trends and make better forecasting decisions.

b) Data warehouses are designed to perform well enormous amounts of data.

c) The structure of data warehouses is more accessible for end-users to navigate,

understand and query.

d) Queries that would be complex in many normalized databases could be easier to build
and maintain in data warehouses.

e) Data warehousing is an efficient method to manage demand for lots of information from
lots of users.

f) Data warehousing provide the capabilities to analyze a large amount of historical data.
Difference between ODS and Data Warehouse

Metadata
• Metadata is simply defined as data about data. The data that is used to represent other
data is known as metadata. In data warehousing, metadata is one of the essential aspects.

• We can define metadata as follows:

a) Metadata is the road-map to a data warehouse.

b) Metadata in a data warehouse defines the warehouse objects.

c) Metadata acts as a directory. This directory helps the decision support system to locate
the contents of a data warehouse.

• In a data warehouse, we create metadata for the data names and definitions of a given
data warehouse. Along with this metadata, additional metadata is also created for time-
stamping any extracted data, the source of extracted data.

Why is metadata necessary in a data warehouse ?

a) First, it acts as the glue that links all parts of the data warehouses.

b) Next, it provides information about the contents and structures to the developers.

c) Finally, it opens the doors to the end-users and makes the contents recognizable in their
terms.
• Fig. 1.11.2 shows warehouse metadata.

Basic Statistical Descriptions of Data

• For data preprocessing to be successful, it is essential to have an overall picture of our

data. Basic statistical descriptions can be used to identify properties of the data and
highlight which data values should be treated as noise or outliers.

• Basic statistical descriptions can be used to identify properties of the data and highlight
which data values should be treated as noise or outliers.

• For data preprocessing tasks, we want to learn about data characteristics regarding both
central tendency and dispersion of the data.

• Measures of central tendency include mean, median, mode and midrange.

• Measures of data dispersion include quartiles, interquartile range (IQR) and variance.

• These descriptive statistics are of great help in understanding the distribution of the data.

Measuring the Central Tendency

• We look at various ways to measure the central tendency of data, include: Mean,
Weighted mean, Trimmed mean, Median, Mode and Midrange.
1. Mean :

• The mean of a data set is the average of all the data values. The sample mean x is the
point estimator of the population mean μ.

2. Median :

Sum of the values of then observations Number of observations in the sample

Sum of the values of the N observations Number of observations in the population

• The median of a data set is the value in the middle when the data items are arranged in
ascending order. Whenever a data set has extreme values, the median is the preferred
measure of central location.

• The median is the measure of location most often reported for annual income and
property value data. A few extremely large incomes of property values can inflate the
mean.

• For an off number of observations:

7 observations= 26, 18, 27, 12, 14, 29, 19.

Numbers in ascending order = 12, 14, 18, 19, 26, 27, 29

• The median is the middle value.

Median=19

• For an even number of observations :

8 observations = 26 18 29 12 14 27 30 19

Numbers in ascending order =12, 14, 18, 19, 26, 27, 29, 30

The median is the average of the middle two values.

3. Mode:

• The mode of a data set is the value that occurs with greatest frequency. The greatest
frequency can occur at two or more different values. If the data have exactly two modes,
the data have exactly two modes, the data are bimodal. If the data have more than two
modes, the data are multimodal.

• Weighted mean: Sometimes, each value in a set may be associated with a weight, the
weights reflect the significance, importance or occurrence frequency attached to their
respective values.

• Trimmed mean: A major problem with the mean is its sensitivity to extreme (e.g., outlier)
values. Even a small number of extreme values can corrupt the mean. The trimmed mean is
the mean obtained after cutting off values at the high and low extremes.

• For example, we can sort the values and remove the top and bottom 2 % before
computing the mean. We should avoid trimming too large a portion (such as 20 %) at both
ends as this can result in the loss of valuable information.

• Holistic measure is a measure that must be computed on the entire data set as a whole. It
cannot be computed by partitioning the given data into subsets and merging the values
obtained for the measure in each subset.

Measuring the Dispersion of Data

• An outlier is an observation that lies an abnormal distance from other values in a random
sample from a population.

• First quartile (Q1): The first quartile is the value, where 25% of the values are smaller than
Q1 and 75% are larger.

• Third quartile (Q3): The third quartile is the value, where 75 % of the values are smaller
than Q3 and 25% are larger.

• The box plot is a useful graphical display for describing the behavior of the data in the
middle as well as at the ends of the distributions. The box plot uses the median and the
lower and upper quartiles. If the lower quartile is Q1 and the upper quartile is Q3, then the
difference (Q3 - Q1) is called the interquartile range or IQ.

• Range: Difference between highest and lowest observed values

Variance :

• The variance is a measure of variability that utilizes all the data. It is based on the
difference between the value of each observation (x;) and the mean (x) for a sample, u for a
population).

• The variance is the average of the squared between each data value and the mean.

Standard Deviation :

• The standard deviation of a data set is the positive square root of the variance. It is
measured in the same in the same units as the data, making it more easily interpreted than
the variance.

• The standard deviation is computed as follows:

Difference between Standard Deviation and Variance

Graphic Displays of Basic Statistical Descriptions
• There are many types of graphs for the display of data summaries and distributions, such
as Bar charts, Pie charts, Line graphs, Boxplot, Histograms, Quantile plots and Scatter plots.

1. Scatter diagram

• Also called scatter plot, X-Y graph.

• While working with statistical data it is often observed that there are connections
between sets of data. For example the mass and height of persons are related, the taller
the person the greater his/her mass.

• To find out whether or not two sets of data are connected scatter diagrams can be used.
Scatter diagram shows the relationship between children's age and height.
• A scatter diagram is a tool for analyzing relationship between two variables. One variable
is plotted on the horizontal axis and the other is plotted on the vertical axis.

• The pattern of their intersecting points can graphically show relationship patterns.
Commonly a scatter diagram is used to prove or disprove cause-and-effect relationships.

• While scatter diagram shows relationships, it does not by itself prove that one variable
causes other. In addition to showing possible cause and effect relationships, a scatter
diagram can show that two variables are from a common cause that is unknown or that one
variable can be used as a surrogate for the other.

2. Histogram

• A histogram is used to summarize discrete or continuous data. In a histogram, the data

are grouped into ranges (e.g. 10-19, 20-29) and then plotted as connected bars. Each bar
represents a range of data.

• To construct a histogram from a continuous variable you first need to split the data into
intervals, called bins. Each bin contains the number of occurrences of scores in the data set
that are contained within that bin.

• The width of each bar is proportional to the width of each category and the height is
proportional to the frequency or percentage of that category.

3. Line graphs

• It is also called stick graphs. It gives relationships between variables.

• Line graphs are usually used to show time series data that is how one or more variables
vary over a continuous period of time. They can also be used to compare two different
variables over time.

• Typical examples of the types of data that can be presented using line graphs are monthly
rainfall and annual unemployment rates.

• Line graphs are particularly useful for identifying patterns and trends in the data such as
seasonal effects, large changes and turning points. Fig. 1.12.1 show line graph. (See Fig.
1.12.1 on next page)
• As well as time series data, line graphs can also be appropriate for displaying data that are
measured over other continuous variables such as distance.

• For example, a line graph could be used to show how pollution levels vary with increasing
distance from a source or how the level of a chemical varies with depth of soil.

• In a line graph the x-axis represents the continuous variable (for example year or distance
from the initial measurement) whilst the y-axis has a scale and indicated the measurement.

• Several data series can be plotted on the same line chart and this is particularly useful for
analysing and comparing the trends in different datasets.

• Line graph is often used to visualize rate of change of a quantity. It is more useful when
the given data has peaks and valleys. Line graphs are very simple to draw and quite
convenient to interpret.

4. Pie charts

• A type of graph is which a circle is divided into sectors that each represents a proportion
of whole. Each sector shows the relative size of each value.

• A pie chart displays data, information and statistics in an easy to read "pie slice" format
with varying slice sizes telling how much of one data element exists.

• Pie chart is also known as circle graph. The bigger the slice, the more of that particular
data was gathered. The main use of a pie chart is to show comparisons. Fig. 1.12.2 shows
pie chart. (See Fig. 1.12.2 on next page)
• Various applications of pie charts can be found in business, school and at home. For
business pie charts can be used to show the success or failure of certain products or
services.

• At school, pie chart applications include showing how much time is allotted to each
subject. At home pie charts can be useful to see expenditure of monthly income in different
needs.

• Reading of pie chart is as easy figuring out which slice of an actual pie is the biggest.

Limitation of pie chart:

• It is difficult to tell the difference between estimates of similar size.

Error bars or confidence limits cannot be shown on pie graph.

Legends and labels on pie graphs are hard to align and read.

• The human visual system is more efficient at perceiving and discriminating between lines
and line lengths rather than two-dimensional areas and angles.

• Pie graphs simply don't work when comparing data.

Two Marks Questions with Answers

Q.1 What is data science?

Ans;
• Data science is an interdisciplinary field that seeks to extract knowledge or insights from
various forms of data.

• At its core, data science aims to discover and extract actionable knowledge from data that
can be used to make sound business decisions and predictions.

• Data science uses advanced analytical theory and various methods such as time series
analysis for predicting future.

Q.2 Define structured data.

Ans. Structured data is arranged in rows and column format. It helps for application to
retrieve and process data easily. Database management system is used for storing structured
data. The term structured data refers to data that is identifiable because it is organized in a
structure.

Q.3 What is data?

Ans. Data set is collection of related records or information. The information may be on
some entity or some subject area.

Q.4 What is unstructured data ?

Ans. Unstructured data is data that does not follow a specified format. Row and columns are
not used for unstructured data. Therefore it is difficult to retrieve required information.
Unstructured data has no identifiable structure.

Q.5 What is machine - generated data ?

Ans. Machine-generated data is an information that is created without human interaction as a

result of a computer process or application activity. This means that data entered manually
by an end-user is not recognized to be machine-generated.

Q.6 Define streaming data.

Ans; Streaming data is data that is generated continuously by thousands of data sources,
which typically send in the data records simultaneously and in small sizes (order of
Kilobytes).

Q.7 List the stages of data science process.

Ans.: Stages of data science process are as follows:

1. Discovery or Setting the research goal

2. Retrieving data

3. Data preparation

4. Data exploration

5. Data modeling

6. Presentation and automation

Q.8 What are the advantages of data repositories?

Ans.: Advantages are as follows:

i. Data is preserved and archived.

ii. Data isolation allows for easier and faster data reporting.

iii. Database administrators have easier time tracking problems.

iv. There is value to storing and analyzing data.

Q.9 What is data cleaning?

Ans. Data cleaning means removing the inconsistent data or noise and collecting necessary
information of a collection of interrelated data.

Q.10 What is outlier detection?

Ans. : Outlier detection is the process of detecting and subsequently excluding outliers from
a given set of data. The easiest way to find outliers is to use a plot or a table with the
minimum and maximum values.

Q.11 Explain exploratory data analysis.

Ans. : Exploratory Data Analysis (EDA) is a general approach to exploring datasets by

means of simple summary statistics and graphic visualizations in order to gain a deeper
understanding of data. EDA is used by data scientists to analyze and investigate data sets and
summarize their main characteristics, often employing data visualization methods.

Q.12 Define data mining.

Ans. : Data mining refers to extracting or mining knowledge from large amounts of data. It is
a process of discovering interesting patterns or Knowledge from a large amount of data
stored either in databases, data warehouses, or other information repositories.

Q.13 What are the three challenges to data mining regarding data mining
methodology?

Ans. Challenges to data mining regarding data mining methodology include the following:

1. Mining different kinds of knowledge in databases,

2. Interactive mining of knowledge at multiple levels of abstraction,

3. Incorporation of background knowledge.

Q.14 What is predictive mining?

Ans. Predictive mining tasks perform inference on the current data in order to make
predictions. Predictive analysis provides answers of the future queries that move across using
historical data as the chief principle for decisions.

Q.15 What is data cleaning?

Ans. Data cleaning means removing the inconsistent data or noise and collecting necessary
information of a collection of interrelated data.

Q.16 List the five primitives for specifying a data mining task.

Ans. :

1. The set of task-relevant data to be mined

2. The kind of knowledge to be mined

3. The background knowledge to be used in the discovery process

4. The interestingness measures and thresholds for pattern evaluation

5. The expected representation for visualizing the discovered pattern.

Q.17 List the stages of data science process.

Ans. Data science process consists of six stages:

1. Discovery or Setting the research goal 2. Retrieving data 3. Data preparation

4. Data exploration 5. Data modeling 6. Presentation and automation

Q.18 What is data repository?

Ans. Data repository is also known as a data library or data archive. This is a general term to
refer to a data set isolated to be mined for data reporting and analysis. The data repository is
a large database infrastructure, several databases that collect, manage and store data sets for
data analysis, sharing and reporting.

Q.19 List the data cleaning tasks?

Ans. Data cleaning are as follows:

1. Data acquisition and metadata

2. Fill in missing values

3. Unified date format

4. Converting nominal to numeric

5. Identify outliers and smooth out noisy data

6. Correct inconsistent data

Q.20 What is Euclidean distance ?

Ans. Euclidean distance is used to measure the similarity between observations. It is

calculated as the square root of the sum of differences between each point.
 UNIT III MACHINE LEARNING

The modeling process in machine learning

The modeling process in machine learning typically involves several key steps, including data
preprocessing, model selection, training, evaluation, and deployment. Here's an overview of the general
modeling process:

1. Data Collection: Obtain a dataset that contains relevant information for the problem you want to solve. This
dataset should be representative of the real-world scenario you are interested in.

2. Data Preprocessing: Clean the dataset by handling missing values, encoding categorical variables, and
scaling numerical features. This step ensures that the data is in a suitable format for modeling.

3. Feature Selection/Engineering: Select relevant features (columns) from the dataset or create new features
based on domain knowledge. This step helps improve the performance of the model by focusing on the most
important information.

4. Splitting the Data: Split the dataset into training, validation, and test sets. The training set is used to train
the model, the validation set is used to tune hyperparameters, and the test set is used to evaluate the final
model.

5. Model Selection: Choose the appropriate machine learning model(s) for your problem. This decision is
based on factors such as the type of problem (classification, regression, clustering, etc.), the size of the
dataset, and the nature of the data.

6. Training the Model: Train the selected model(s) on the training data. During training, the model learns
patterns and relationships in the data that will allow it to make predictions on new, unseen data.

7. Hyperparameter Tuning: Use the validation set to tune the hyperparameters of the model.
Hyperparameters are parameters that control the learning process of the model (e.g., learning rate,
regularization strength) and can have a significant impact on performance.

8. Model Evaluation: Evaluate the model(s) using the test set. This step involves measuring performance
metrics such as accuracy, precision, recall, F1 score, or mean squared error, depending on the type of
problem.

9. Model Deployment: Once you are satisfied with the performance of the model, deploy it to a production
environment where it can make predictions on new data. This step may involve packaging the model into a
software application or integrating it into an existing system.

10. Monitoring and Maintenance: Continuously monitor the performance of the deployed model and update it
as needed to ensure that it remains accurate and reliable over time.

This is a high-level overview of the modeling process in machine learning. The specific details of each step
may vary depending on the problem you are working on and the tools and techniques you are using.
 Types of machine learning

Machine learning can be broadly categorized into three main types based on the nature of the
learning process and the availability of labeled data:

1. Supervised Learning: In supervised learning, the model is trained on a labeled dataset, where
each example is paired with a corresponding label or output. The goal of the model is to learn a
mapping from inputs to outputs so that it can predict the correct output for new, unseen inputs.
Examples of supervised learning algorithms include linear regression, logistic regression, decision
trees, random forests, support vector machines (SVM), and neural networks.

2. Unsupervised Learning: In unsupervised learning, the model is trained on an unlabeled dataset,

and the goal is to find hidden patterns or structures in the data. The model learns to group similar
data points together and identify underlying relationships without explicit guidance. Clustering
and dimensionality reduction are common tasks in unsupervised learning. Examples of
unsupervised learning algorithms include K-means clustering, hierarchical clustering, principal
component analysis (PCA), and t-distributed stochastic neighbor embedding (t-SNE).

3. Reinforcement Learning: Reinforcement learning is a type of machine learning where an agent

learns to make decisions by interacting with an environment. The agent receives feedback in the
form of rewards or penalties based on its actions, and the goal is to learn a policy that maximizes
the cumulative reward over time. Reinforcement learning is commonly used in applications such as
game playing, robotics, and autonomous driving. Examples of reinforcement learning algorithms
include Q-learning, deep Q-networks (DQN), and policy gradient methods.

These are the main types of machine learning, but there are also other subfields and specialized
approaches, such as semi-supervised learning, where the model is trained on a combination of
labeled and unlabeled data, and transfer learning, where knowledge gained from one task is
applied to another related task.

Supervised learning

Supervised learning is a type of machine learning where the model is trained on a labeled dataset,
meaning that each example in the dataset is paired with a corresponding label or output. The goal
of supervised learning is to learn a mapping from inputs to outputs so that the model can predict
the correct output for new, unseen inputs.

Supervised learning can be further divided into two main categories:

1. Classification: In classification tasks, the goal is to predict a categorical label or class for each
input. Examples of classification tasks include spam detection (classifying emails as spam or not
spam), image classification (classifying images into different categories), and sentiment analysis
(classifying text as positive, negative, or neutral).

2. Regression: In regression tasks, the goal is to predict a continuous value for each input. Examples
of regression tasks include predicting house prices based on features such as size, location, and
number of bedrooms, predicting stock prices based on historical data, and predicting the amount
of rainfall based on weather patterns.

Supervised learning algorithms learn from the labeled data by finding patterns and relationships
that allow them to make accurate predictions on new, unseen data. Some common supervised
learning algorithms include:

• Linear Regression: Used for regression tasks where the relationship between the input
features and the output is assumed to be linear.
• Logistic Regression: Used for binary classification tasks where the output is a binary label
(e.g., spam or not spam).
• Decision Trees: Used for both classification and regression tasks, decision trees make
decisions based on the values of input features.
• Random Forests: An ensemble method that uses multiple decision trees to improve
performance and reduce overfitting.
• Support Vector Machines (SVM): Used for both classification and regression tasks, SVMs
find a hyperplane that separates different classes or fits the data with the largest margin.
• Neural Networks: A versatile class of models inspired by the structure of the human brain,
neural networks can be used for a wide range of tasks including classification, regression,
and even reinforcement learning.

Overall, supervised learning is a powerful and widely used approach in machine learning, with
applications in areas such as healthcare, finance, marketing, and more.

Unsupervised learning in machine learning

Unsupervised learning is a type of machine learning where the model is trained on an unlabeled dataset,
meaning that the data does not have any corresponding output labels. The goal of unsupervised learning is to
find hidden patterns or structures in the data.

Unlike supervised learning, where the model learns from labeled examples to predict outputs for new inputs,
unsupervised learning focuses on discovering the underlying structure of the data without any guidance on
what the output should be. This makes unsupervised learning particularly useful for exploratory data
analysis and understanding the relationships between data points.

There are several key tasks in unsupervised learning:

1. Clustering: Clustering is the task of grouping similar data points together. The goal is to partition the data
into clusters such that data points within the same cluster are more similar to each other than to those in
other clusters. K-means clustering and hierarchical clustering are common clustering algorithms.

2. Dimensionality Reduction: Dimensionality reduction techniques are used to reduce the number of features
in the dataset while preserving as much information as possible. This can help in visualizing high-
dimensional data and reducing the computational complexity of models. Principal Component Analysis
(PCA) and t-distributed Stochastic Neighbor Embedding (t-SNE) are popular dimensionality reduction
techniques.

3. Anomaly Detection: Anomaly detection, also known as outlier detection, is the task of identifying data
points that deviate from the norm in a dataset. Anomalies may indicate errors in the data, fraudulent
behavior, or other unusual patterns. One-class SVM and Isolation Forest are common anomaly detection
algorithms.

4. Association Rule Learning: Association rule learning is the task of discovering interesting relationships
between variables in large datasets. It is often used in market basket analysis to identify patterns in consumer
behavior. Apriori and FP-growth are popular association rule learning algorithms.

Unsupervised learning is widely used in various fields such as data mining, pattern recognition, and
bioinformatics. It can help in gaining insights from data that may not be immediately apparent and can be a
valuable tool in exploratory data analysis and knowledge discovery.

Semi-supervised learning in Machine Learning

Semi-supervised learning is a type of machine learning that falls between supervised learning and
unsupervised learning. In semi-supervised learning, the model is trained on a dataset that contains
both labeled and unlabeled examples. The goal of semi-supervised learning is to leverage the
unlabeled data to improve the performance of the model on the task at hand.

The main idea behind semi-supervised learning is that labeled data is often expensive or time-
consuming to obtain, while unlabeled data is often abundant and easy to acquire. By using both
labeled and unlabeled data, semi-supervised learning algorithms aim to make better use of the
available data and improve the performance of the model.

There are several approaches to semi-supervised learning, including:

1. Self-training: In self-training, the model is initially trained on the labeled data. Then, it uses this
model to predict labels for the unlabeled data. The predictions with high confidence are added to
the labeled dataset, and the model is retrained on the expanded dataset. This process iterates until
convergence.

2. Co-training: In co-training, the model is trained on multiple views of the data, each of which
contains a different subset of features. The model is trained on the labeled data from each view
 and then used to predict labels for the unlabeled data in each view. The predictions from each
view are then combined to make a final prediction.

3. Semi-supervised Generative Adversarial Networks (GANs): GANs can be used for semi-
supervised learning by training a generator to produce realistic data samples and a discriminator
to distinguish between real and generated samples. The generator is trained using both labeled
and unlabeled data, while the discriminator is trained using only labeled data.

Semi-supervised learning is particularly useful in scenarios where labeled data is scarce but
unlabeled data is abundant, such as in medical imaging, speech recognition, and natural language
processing. By effectively leveraging both types of data, semi-supervised learning can improve the
performance of machine learning models and reduce the need for large amounts of labeled data.

Classification, regression in machine learning

Classification and regression are two fundamental types of supervised learning in machine
learning.

1. Classification:

• Classification is a supervised learning task where the goal is to predict the categorical label
of a new input based on past observations.
• In classification, the output variable is discrete and belongs to a specific class or category.
• Examples of classification tasks include spam detection (classifying emails as spam or not
spam), sentiment analysis (classifying movie reviews as positive or negative), and image
classification (classifying images into different categories).
• Common algorithms for classification include logistic regression, decision trees, random
forests, support vector machines (SVM), and neural networks.
• Evaluation metrics for classification include accuracy, precision, recall, F1 score, and area
under the receiver operating characteristic curve (ROC-AUC).

2. Regression:

• Regression is a supervised learning task where the goal is to predict a continuous value for
a new input based on past observations.
• In regression, the output variable is continuous and can take any value within a range.
• Examples of regression tasks include predicting house prices based on features such as size
and location, predicting stock prices based on historical data, and predicting the
temperature based on weather patterns.
• Common algorithms for regression include linear regression, polynomial regression,
decision trees, random forests, and neural networks.
 • Evaluation metrics for regression include mean squared error (MSE), root mean squared
error (RMSE), mean absolute error (MAE), and R-squared.

Both classification and regression are important tasks in machine learning and are used in a wide
range of applications. The choice between classification and regression depends on the nature of
the output variable and the specific problem being addressed.

Clustering in machine learning

Clustering is an unsupervised learning technique used to group similar data points together in
such a way that data points in the same group (or cluster) are more similar to each other than to
those in other groups. Clustering is commonly used in exploratory data analysis to identify
patterns, group similar objects together, and reduce the complexity of data.

There are several types of clustering algorithms, each with its own strengths and weaknesses:

1. K-means Clustering: K-means is one of the most commonly used clustering algorithms. It
partitions the data into K clusters, where each data point belongs to the cluster with the nearest
mean. K-means aims to minimize the sum of squared distances between data points and their
corresponding cluster centroids.

2. Hierarchical Clustering: Hierarchical clustering builds a hierarchy of clusters, where each data
point starts in its own cluster and clusters are successively merged or split based on their similarity.
Hierarchical clustering can be agglomerative (bottom-up) or divisive (top-down).

3. DBSCAN (Density-Based Spatial Clustering of Applications with Noise): DBSCAN is a density-

based clustering algorithm that groups together closely packed data points and identifies outliers
as noise. DBSCAN does not require the number of clusters to be specified in advance.

4. Mean Shift: Mean shift is a clustering algorithm that assigns each data point to the cluster
corresponding to the nearest peak in the density estimation of the data. Mean shift can
automatically determine the number of clusters based on the data.

5. Gaussian Mixture Models (GMM): GMM is a probabilistic model that assumes that the data is
generated from a mixture of several Gaussian distributions. GMM can be used for clustering by
fitting the model to the data and assigning each data point to the most likely cluster.

6. Agglomerative Clustering: Agglomerative clustering is a bottom-up hierarchical clustering

algorithm that starts with each data point as a singleton cluster and iteratively merges clusters
based on their similarity.
 Clustering is used in various applications such as customer segmentation, image segmentation,
anomaly detection, and recommender systems. The choice of clustering algorithm depends on the
nature of the data and the specific requirements of the problem.

Outliers are data points that significantly differ from other observations in a dataset. They can arise
due to errors in data collection, measurement variability, or genuine rare events. Outliers can have
a significant impact on the results of data analysis and machine learning models, as they can skew
statistical measures and distort the learning process.

Outlier analysis is the process of identifying and handling outliers in a dataset. There are several
approaches to outlier analysis:

1. Statistical Methods: Statistical methods such as Z-score, modified Z-score, and Tukey's method
(based on the interquartile range) can be used to detect outliers. These methods identify data
points that fall significantly far from the mean or median of the dataset.

2. Visualization: Visualization techniques such as box plots, scatter plots, and histograms can be
used to identify outliers visually. Outliers often appear as points that are far away from the main
cluster of data points.

3. Clustering: Clustering algorithms such as K-means can be used to cluster data points and identify
outliers as data points that do not belong to any cluster or belong to small clusters.

4. Distance-based Methods: Distance-based methods such as DBSCAN (Density-Based Spatial

Clustering of Applications with Noise) can be used to identify outliers as data points that are far
away from dense regions of the data.

Once outliers are identified, there are several approaches to handling them:

1. Removing Outliers: One approach is to remove outliers from the dataset. However, this approach
should be used with caution, as removing outliers can lead to loss of information and bias in the
data.

2. Transforming Variables: Another approach is to transform variables to make the distribution

more normal, which can reduce the impact of outliers.

3. Treating Outliers as Missing Values: Outliers can be treated as missing values and imputed using
techniques such as mean, median, or mode imputation.

4. Using Robust Statistical Methods: Robust statistical methods such as robust regression or robust
clustering can be used that are less sensitive to outliers.
 It's important to carefully analyze outliers and consider the context of the data before deciding on
the appropriate approach for handling them.

Here are some multiple-choice questions (MCQs) with answers related to machine learning:

1. What is machine learning? A) A technique to write algorithms B) A field of study that gives
computers the ability to learn without being explicitly programmed C) A method to automate tasks
D) A software application

Answer: B) A field of study that gives computers the ability to learn without being explicitly
programmed

2. Which of the following is a supervised learning technique? A) K-means clustering B) Principal

Component Analysis (PCA) C) Linear regression D) Apriori algorithm

Answer: C) Linear regression

3. What is the goal of unsupervised learning? A) To classify data into predefined classes B) To find
patterns in data without any labels C) To predict continuous values D) To learn from a teacher

Answer: B) To find patterns in data without any labels

4. Which algorithm is used for clustering in machine learning? A) Decision tree B) K-nearest
neighbors (KNN) C) Support Vector Machine (SVM) D) K-means

Answer: D) K-means

5. What is overfitting in machine learning? A) When a model is too simple to capture the underlying
structure of the data B) When a model performs well on the training data but fails to generalize to
new data C) When a model is too complex and memorizes the training data D) When a model is
not trained long enough

Answer: C) When a model is too complex and memorizes the training data

6. Which evaluation metric is commonly used for classification tasks? A) Mean Squared Error (MSE) B)
R-squared C) Accuracy D) Root Mean Squared Error (RMSE)

Answer: C) Accuracy
7. What is the purpose of feature scaling in machine learning? A) To make the data easier to visualize
B) To speed up the training process C) To ensure all features have the same scale D) To reduce the
number of features

Answer: C) To ensure all features have the same scale

8. Which of the following is an example of a reinforcement learning problem? A) Image classification

B) Spam detection C) Playing a game of chess D) Customer segmentation

Answer: C) Playing a game of chess

These are just a few examples of MCQs related to machine learning. Depending on the complexity
of the topic and the level of the audience, the questions can vary.

1. What is the first step in the machine learning modeling process? A) Data preprocessing B) Model selection
C) Data collection D) Model evaluation

Answer: C) Data collection

2. What is the purpose of data preprocessing in machine learning? A) To clean and prepare the data for
modeling B) To select the best model for the data C) To evaluate the performance of the model D) To
deploy the model in a production environment

Answer: A) To clean and prepare the data for modeling

3. What is the purpose of model selection in machine learning? A) To clean and prepare the data for modeling
B) To select the best model for the data C) To evaluate the performance of the model D) To deploy the
model in a production environment

Answer: B) To select the best model for the data

4. Which of the following is NOT a step in the machine learning modeling process? A) Data preprocessing B)
Model evaluation C) Model deployment D) Data visualization

Answer: D) Data visualization

5. What is the purpose of model evaluation in machine learning? A) To clean and prepare the data for
modeling B) To select the best model for the data C) To evaluate the performance of the model D) To
deploy the model in a production environment

Answer: C) To evaluate the performance of the model

6. What is the final step in the machine learning modeling process? A) Data preprocessing B) Model selection
C) Model evaluation D) Model deployment
 Answer: D) Model deployment

7. What is the goal of data preprocessing in machine learning? A) To create new features from existing data B)
To remove outliers from the data C) To scale the data to a standard range D) To clean and prepare the data
for modeling

Answer: D) To clean and prepare the data for modeling

8. Which of the following is NOT a common evaluation metric used in machine learning? A) Accuracy B)
Mean Squared Error (MSE) C) R-squared D) Principal Component Analysis (PCA)

Answer: D) Principal Component Analysis (PCA)

These questions cover the basic steps of the machine learning modeling process, including data
preprocessing, model selection, model evaluation, and model deployment.

1. What are the main types of machine learning? A) Supervised learning, unsupervised learning, and
reinforcement learning B) Classification, regression, and clustering C) Neural networks, decision
trees, and SVMs D) Linear regression, logistic regression, and K-means clustering

Answer: A) Supervised learning, unsupervised learning, and reinforcement learning

2. Which type of machine learning is used when the data is labeled? A) Supervised learning B)
Unsupervised learning C) Reinforcement learning D) Semi-supervised learning

Answer: A) Supervised learning

3. What is the goal of unsupervised learning? A) To predict a continuous value B) To classify data into
predefined classes C) To find patterns in data without any labels D) To learn from a teacher

Answer: C) To find patterns in data without any labels

4. Which type of machine learning is used when the data is not labeled? A) Supervised learning B)
Unsupervised learning C) Reinforcement learning D) Semi-supervised learning

Answer: B) Unsupervised learning

5. Which type of machine learning is used when the model learns from its own experience? A)
Supervised learning B) Unsupervised learning C) Reinforcement learning D) Semi-supervised
learning

Answer: C) Reinforcement learning

6. What is the goal of semi-supervised learning? A) To predict a continuous value B) To classify data
into predefined classes C) To find patterns in data without any labels D) To leverage both labeled
and unlabeled data for learning

Answer: D) To leverage both labeled and unlabeled data for learning

7. Which type of machine learning is used for anomaly detection? A) Supervised learning B)
Unsupervised learning C) Reinforcement learning D) Semi-supervised learning

Answer: B) Unsupervised learning

8. Which type of machine learning is used for customer segmentation? A) Supervised learning B)
Unsupervised learning C) Reinforcement learning D) Semi-supervised learning

Answer: B) Unsupervised learning

These questions cover the main types of machine learning, including supervised learning,
unsupervised learning, and reinforcement learning, as well as their goals and applications.

Supervised learning of machine learning MCQ with answers

Here are some multiple-choice questions (MCQs) with answers related to supervised learning in
machine learning:

1. What is supervised learning? A) A type of learning where the model learns from its own experience
B) A type of learning where the model learns from labeled data C) A type of learning where the
model learns without any labels D) A type of learning where the model learns from reinforcement

Answer: B) A type of learning where the model learns from labeled data

2. Which of the following is an example of a supervised learning task? A) Clustering B) Dimensionality

reduction C) Classification D) Anomaly detection

Answer: C) Classification

3. What is the goal of regression in supervised learning? A) To classify data into predefined classes B)
To predict a continuous value C) To find patterns in data without any labels D) To learn from a
teacher

Answer: B) To predict a continuous value

4. Which of the following is a common algorithm used for classification in supervised learning? A) K-
means clustering B) Decision tree C) Principal Component Analysis (PCA) D) Apriori algorithm

Answer: B) Decision tree

5. What is the purpose of the training data in supervised learning? A) To evaluate the performance of
the model B) To select the best model for the data C) To clean and prepare the data for modeling
D) To teach the model to make predictions

Answer: D) To teach the model to make predictions

6. Which of the following is NOT a common evaluation metric used in classification tasks? A)
Accuracy B) Mean Squared Error (MSE) C) Precision D) Recall

Answer: B) Mean Squared Error (MSE)

7. What is the goal of feature selection in supervised learning? A) To clean and prepare the data for
modeling B) To select the best model for the data C) To reduce the number of features to improve
model performance D) To ensure all features have the same scale

Answer: C) To reduce the number of features to improve model performance

8. Which of the following is an example of a regression task? A) Predicting whether an email is spam
or not B) Predicting house prices based on features such as size and location C) Clustering
customer data to identify segments D) Classifying images into different categories

Answer: B) Predicting house prices based on features such as size and location

These questions cover the basics of supervised learning in machine learning, including the goals,
algorithms, evaluation metrics, and applications of supervised learning.

Unsupervised learning of machine learning MCQ with answers

Here are some multiple-choice questions (MCQs) with answers related to unsupervised learning in machine
learning:

1. What is unsupervised learning? A) A type of learning where the model learns from labeled data B) A type of
learning where the model learns from its own experience C) A type of learning where the model learns
without any labels D) A type of learning where the model learns from reinforcement

Answer: C) A type of learning where the model learns without any labels
2. Which of the following is an example of an unsupervised learning task? A) Image classification B)
Clustering C) Spam detection D) Sentiment analysis

Answer: B) Clustering

3. What is the goal of clustering in unsupervised learning? A) To predict a continuous value B) To classify data
into predefined classes C) To find patterns in data without any labels D) To learn from a teacher

Answer: C) To find patterns in data without any labels

4. Which of the following is a common algorithm used for clustering in unsupervised learning? A) Decision
tree B) K-means C) Support Vector Machine (SVM) D) Linear regression

Answer: B) K-means

5. What is the purpose of dimensionality reduction in unsupervised learning? A) To reduce the number of
features to improve model performance B) To select the best model for the data C) To ensure all features
have the same scale D) To clean and prepare the data for modeling

Answer: A) To reduce the number of features to improve model performance

6. Which of the following is an example of an anomaly detection task? A) Predicting house prices based on
features such as size and location B) Classifying images into different categories C) Identifying fraudulent
transactions in financial data D) Clustering customer data to identify segments

Answer: C) Identifying fraudulent transactions in financial data

7. What is the goal of feature extraction in unsupervised learning? A) To clean and prepare the data for
modeling B) To reduce the number of features to improve model performance C) To select the best model
for the data D) To ensure all features have the same scale

Answer: B) To reduce the number of features to improve model performance

8. Which of the following is an example of a dimensionality reduction technique? A) K-means clustering B)

Decision tree C) Principal Component Analysis (PCA) D) Apriori algorithm

Answer: C) Principal Component Analysis (PCA)

These questions cover the basics of unsupervised learning in machine learning, including the goals,
algorithms, and applications of unsupervised learning.

Here are some multiple-choice questions (MCQs) with answers related to semi-supervised learning
in machine learning:

1. What is semi-supervised learning? A) A type of learning where the model learns from labeled data
B) A type of learning where the model learns from its own experience C) A type of learning where
 the model learns from both labeled and unlabeled data D) A type of learning where the model
learns without any labels

Answer: C) A type of learning where the model learns from both labeled and unlabeled data

2. Which of the following is an example of a semi-supervised learning task? A) Image classification B)

Clustering C) Sentiment analysis with a small labeled dataset and a large unlabeled dataset D)
Regression

Answer: C) Sentiment analysis with a small labeled dataset and a large unlabeled dataset

3. What is the goal of semi-supervised learning? A) To predict a continuous value B) To classify data
into predefined classes C) To leverage both labeled and unlabeled data for learning D) To learn
from a teacher

Answer: C) To leverage both labeled and unlabeled data for learning

4. Which of the following is a common approach used in semi-supervised learning? A) Self-training B)

K-means clustering C) Support Vector Machine (SVM) D) Principal Component Analysis (PCA)

Answer: A) Self-training

5. What is the purpose of self-training in semi-supervised learning? A) To clean and prepare the data
for modeling B) To select the best model for the data C) To predict labels for unlabeled data based
on a model trained on labeled data D) To ensure all features have the same scale

Answer: C) To predict labels for unlabeled data based on a model trained on labeled data

6. Which of the following is a benefit of using semi-supervised learning? A) It requires a large amount
of labeled data B) It can improve model performance by leveraging unlabeled data C) It is
computationally expensive D) It is only suitable for certain types of machine learning tasks

Answer: B) It can improve model performance by leveraging unlabeled data

7. What is the main challenge of using semi-supervised learning? A) It requires a large amount of
labeled data B) It can lead to overfitting C) It can be difficult to predict labels for unlabeled data
accurately D) It is not suitable for complex machine learning tasks

Answer: C) It can be difficult to predict labels for unlabeled data accurately

8. Which of the following is an example of a semi-supervised learning algorithm? A) K-means

clustering B) Decision tree C) Label Propagation D) Linear regression

Answer: C) Label Propagation

 These questions cover the basics of semi-supervised learning in machine learning, including its
goals, approaches, benefits, and challenges.

Classification, regression of machine learning MCQ with answers

Here are some multiple-choice questions (MCQs) with answers related to classification and regression in
machine learning:

1. What is the goal of classification in machine learning? A) To predict a continuous value B) To classify data
into predefined classes C) To find patterns in data without any labels D) To learn from a teacher

Answer: B) To classify data into predefined classes

2. Which of the following is an example of a classification task? A) Predicting house prices based on features
such as size and location B) Classifying emails as spam or not spam C) Clustering customer data to identify
segments D) Predicting a student's grade based on the number of hours studied

Answer: B) Classifying emails as spam or not spam

3. What is the goal of regression in machine learning? A) To classify data into predefined classes B) To predict
a continuous value C) To find patterns in data without any labels D) To learn from a teacher

Answer: B) To predict a continuous value

4. Which of the following is an example of a regression task? A) Classifying images into different categories
B) Predicting house prices based on features such as size and location C) Clustering customer data to
identify segments D) Predicting whether a customer will buy a product or not

Answer: B) Predicting house prices based on features such as size and location

5. Which evaluation metric is commonly used for classification tasks? A) Mean Squared Error (MSE) B) R-
squared C) Accuracy D) Root Mean Squared Error (RMSE)

Answer: C) Accuracy

6. Which evaluation metric is commonly used for regression tasks? A) Accuracy B) Mean Squared Error
(MSE) C) Precision D) Recall

Answer: B) Mean Squared Error (MSE)

7. What is the main difference between classification and regression? A) Classification predicts a continuous
value, while regression predicts a discrete class label B) Classification predicts a discrete class label, while
regression predicts a continuous value C) Classification uses labeled data, while regression uses unlabeled
data D) Regression uses labeled data, while classification uses unlabeled data
 Answer: B) Classification predicts a discrete class label, while regression predicts a continuous value

8. Which of the following algorithms is commonly used for classification tasks? A) Linear regression B)
Decision tree C) K-means clustering D) Principal Component Analysis (PCA)

Answer: B) Decision tree

These questions cover the basics of classification and regression in machine learning, including their goals,
examples, evaluation metrics, and algorithms.

Clustering of machine learning MCQ with answers

Here are some multiple-choice questions (MCQs) with answers related to clustering in machine
learning:

1. What is clustering in machine learning? A) A type of learning where the model learns from labeled
data B) A type of learning where the model learns from its own experience C) A type of learning
where the model learns without any labels D) A type of learning where the model learns from
reinforcement

Answer: C) A type of learning where the model learns without any labels

2. Which of the following is an example of a clustering task? A) Image classification B) Predicting

house prices based on features such as size and location C) Clustering customer data to identify
segments D) Classifying emails as spam or not spam

Answer: C) Clustering customer data to identify segments

3. What is the goal of clustering in machine learning? A) To predict a continuous value B) To classify
data into predefined classes C) To find patterns in data without any labels D) To learn from a
teacher

Answer: C) To find patterns in data without any labels

4. Which of the following is a common algorithm used for clustering in machine learning? A) Decision
tree B) K-means C) Support Vector Machine (SVM) D) Linear regression

Answer: B) K-means

5. What is the purpose of clustering in machine learning? A) To reduce the number of features to
improve model performance B) To select the best model for the data C) To find patterns in data
without any labels D) To ensure all features have the same scale
 Answer: C) To find patterns in data without any labels

6. Which of the following is an example of an evaluation metric used for clustering? A) Accuracy B)
Mean Squared Error (MSE) C) Silhouette score D) Precision

Answer: C) Silhouette score

7. Which of the following is NOT a common approach used in clustering? A) K-means clustering B)
Hierarchical clustering C) DBSCAN D) Linear regression

Answer: D) Linear regression

8. What is the main difference between clustering and classification? A) Clustering predicts a
continuous value, while classification predicts a discrete class label B) Clustering uses labeled data,
while classification uses unlabeled data C) Clustering predicts a discrete class label, while
classification predicts a continuous value D) Clustering is a type of unsupervised learning, while
classification is a type of supervised learning

Answer: D) Clustering is a type of unsupervised learning, while classification is a type of supervised

learning

These questions cover the basics of clustering in machine learning, including its goals, examples,
algorithms, and evaluation metrics.

Outliers and Outlier Analysis of machine learning MCQ with answers

Here are some multiple-choice questions (MCQs) with answers related to outliers and outlier
analysis in machine learning:

1. What is an outlier in a dataset? A) A data point that is missing a value B) A data point that is
significantly different from other observations C) A data point that is incorrectly labeled D) A data
point that is located at the center of the dataset

Answer: B) A data point that is significantly different from other observations

2. Why are outliers important in data analysis? A) They help to reduce the complexity of the dataset
B) They can provide valuable insights into the data C) They have no impact on the results of data
analysis D) They make the dataset more difficult to analyze

Answer: B) They can provide valuable insights into the data

3. Which of the following is a common method for detecting outliers? A) Z-score method B) Mean
Squared Error (MSE) C) Root Mean Squared Error (RMSE) D) Silhouette score

Answer: A) Z-score method

4. What is the Z-score method used for in outlier analysis? A) To calculate the mean of the dataset B)
To calculate the standard deviation of the dataset C) To identify data points that are significantly
different from the mean D) To calculate the range of the dataset

Answer: C) To identify data points that are significantly different from the mean

5. Which of the following is a common approach for handling outliers? A) Removing outliers from the
dataset B) Keeping outliers in the dataset C) Replacing outliers with the mean of the dataset D)
Ignoring outliers in the analysis

Answer: A) Removing outliers from the dataset

6. What is the impact of outliers on statistical measures such as mean and standard deviation? A)
Outliers have no impact on these measures B) Outliers increase the mean and standard deviation
C) Outliers decrease the mean and standard deviation D) The impact of outliers depends on their
value

Answer: B) Outliers increase the mean and standard deviation

7. Which of the following is a disadvantage of removing outliers from a dataset? A) It can lead to
biased results B) It can improve the accuracy of the analysis C) It can make the dataset easier to
analyze D) It can reduce the complexity of the dataset

Answer: A) It can lead to biased results

8. What is the purpose of outlier analysis in machine learning? A) To identify errors in the dataset B)
To improve the accuracy of machine learning models C) To reduce the complexity of the dataset D)
To increase the number of data points in the dataset

Answer: B) To improve the accuracy of machine learning models

These questions cover the basics of outliers and outlier analysis in machine learning, including
their detection, impact, and handling.

UNIT II DATA MANIPULATION

Python Shell
The Python Shell, also known as the Python interactive interpreter or Python REPL (Read-Eval-Print
Loop), is a command-line tool that allows you to interactively execute Python code. It provides a
convenient way to experiment with Python code, test small snippets, and learn about Python
features.

To start the Python Shell, you can open a terminal or command prompt and type python or
python3 depending on your Python installation. This will launch the Python interpreter, and you
will see a prompt (>>>) where you can start entering Python code.

Here is an example of using the Python Shell:

$ python

Python 3.8.5 (default, Jan 27 2021, 15:41:15)

[GCC 9.3.0] on linux

Type "help", "copyright", "credits" or "license" for more information.

>>> print("Hello, world!")

Hello, world!

>>> x = 5

>>> y = 10

>>> print(x + y)

15

>>> exit()

In this example, we start the Python interpreter, print a message, perform some basic arithmetic
operations, and then exit the Python interpreter using the exit() function.

Jupyter Notebook
 upyter Notebook is an open-source web application that allows you to create and share
documents that contain live code, equations, visualizations, and narrative text. It supports various
programming languages, including Python, R, and Julia, among others. Jupyter Notebook is widely
used for data cleaning, transformation, numerical simulation, statistical modeling, data
visualization, machine learning, and more.

To start using Jupyter Notebook, you first need to have Python installed on your computer. You
can then install Jupyter Notebook using pip, the Python package installer, by running the following
command in your terminal or command prompt:

pip install jupyterlab

Once Jupyter Notebook is installed, you can start it by running the following command in your terminal or
command prompt:

jupyter notebook

This will launch the Jupyter Notebook server and open a new tab in your web browser with the
Jupyter Notebook interface. From there, you can create a new notebook or open an existing one.
You can write and execute code in the notebook cells, add text and equations using Markdown,
and create visualizations using libraries like Matplotlib and Seaborn.

Jupyter Notebook is a powerful tool for interactive computing and is widely used in data science
and research communities.

IPython Magic Commands

IPython magic commands are special commands that allow you to perform various tasks in
IPython, the enhanced interactive Python shell. Magic commands are prefixed by one or two
percentage signs (% or %%) and provide additional functionality beyond what standard Python
syntax offers. Here are some commonly used IPython magic commands:

1. %run: Run a Python script inside the IPython session. Usage: %run script.py.

2. %time and %timeit: Measure the execution time of a single statement ( %time) or a Python
statement or expression (%timeit).

3. %load: Load code into the current IPython session. Usage: %load file.py.
4. %matplotlib: Enable inline plotting of graphs and figures in IPython. Usage: %matplotlib inline.

5. %reset: Reset the IPython namespace by removing all variables, functions, and imports. Usage:
%reset -f.

6. %who and %whos: List all variables in the current IPython session ( %who) or list all variables with
additional information such as type and value ( %whos).

7. %%time and %%timeit: Measure the execution time of a cell ( %%time) or a cell statement
(%%timeit) in IPython.

8. %magic: Display information about IPython magic commands and their usage. Usage: %magic.

9. %history: Display the command history for the current IPython session. Usage: %history.

10. %pdb: Activate the interactive debugger (Python debugger) for errors in the IPython session. Usage:
%pdb.

These are just a few examples of IPython magic commands. IPython provides many more magic
commands for various purposes, and you can explore them by typing %lsmagic to list all available
magic commands and %<command>? for help on a specific magic command (e.g., %time? for help
on the %time command).

NumPy Arrays

NumPy is a Python library that provides support for creating and manipulating arrays and matrices.
NumPy arrays are the core data structure used in NumPy to store and manipulate data efficiently.
Here's a brief overview of NumPy arrays:

1. Creating NumPy Arrays: NumPy arrays can be created using the numpy.array() function by
passing a Python list as an argument. For example:

import numpy as np
arr = np.array([1, 2, 3, 4, 5])

Array Attributes: NumPy arrays have several attributes that provide information about the array, such as its

shape, size, and data type. Some common attributes include shape, size, and dtype.

print(arr.shape) # (5,) - shape of the array

 print(arr.size) # 5 - number of elements in the array
print(arr.dtype) # int64 - data type of the array elements

Array Operations: NumPy arrays support element-wise operations, such as addition, subtraction,

multiplication, and division. These operations are performed on each element of the array.

arr1 = np.array([1, 2, 3])

arr2 = np.array([4, 5, 6])
result = arr1 + arr2 # [5, 7, 9]

Indexing and Slicing: NumPy arrays support indexing and slicing operations to access and modify

elements of the array.

print(arr[0]) # 1 - access the first element of the array

print(arr[1:3]) # [2, 3] - slice the array from index 1 to 2

Array Broadcasting: NumPy arrays support broadcasting, which allows operations to be performed on

arrays of different shapes.

arr = np.array([[1, 2, 3], [4, 5, 6]])
scalar = 2
result = arr * scalar # [[2, 4, 6], [8, 10, 12]]

1. Array Functions: NumPy provides a variety of functions for creating and manipulating arrays, such as
np.arange() , np.zeros(), np.ones(), np.linspace(), np.concatenate(), and more.

NumPy arrays are widely used in scientific computing, data analysis, and machine learning due to their
efficiency and versatility.

Universal Functions of datamanipulation

 Universal functions (ufuncs) in NumPy are functions that operate element-wise on arrays. They are
essential for numerical operations on arrays, as they provide a way to efficiently perform
operations without the need for explicit loops. Ufuncs support various mathematical operations
and can be used to manipulate data in arrays. Here are some common ufuncs used for data
manipulation in NumPy:

1. Mathematical Functions: NumPy provides ufuncs for basic mathematical operations such as
np.add(), np.subtract(), np.multiply(), np.divide(), np.power(), np.sqrt(), np.exp(),
np.log(), and more. These functions can be used to perform element-wise arithmetic operations
on arrays.

2. Trigonometric Functions: NumPy provides ufuncs for trigonometric functions such as np.sin(),
np.cos(), np.tan(), np.arcsin(), np.arccos(), np.arctan(), and more. These functions
operate element-wise on arrays and are useful for mathematical calculations involving angles.

3. Statistical Functions: NumPy provides ufuncs for statistical functions such as np.mean(),
np.median(), np.std(), np.var(), np.sum(), np.min(), np.max(), and more. These functions can
be used to calculate various statistical measures of arrays.

4. Logical Functions: NumPy provides ufuncs for logical operations such as np.logical_and(),
np.logical_or(), np.logical_not(), and more. These functions operate element-wise on
boolean arrays and are useful for logical operations.

5. Comparison Functions: NumPy provides ufuncs for comparison operations such as np.equal(),
np.not_equal(), np.greater(), np.greater_equal(), np.less(), np.less_equal(), and more.
These functions compare elements of arrays and return boolean arrays indicating the result of the
comparison.

6. Bitwise Functions: NumPy provides ufuncs for bitwise operations such as np.bitwise_and(),
np.bitwise_or(), np.bitwise_xor(), np.bitwise_not(), and more. These functions operate
element-wise on integer arrays and perform bitwise operations.

These are just a few examples of the many ufuncs available in NumPy for data manipulation.
Ufuncs are an important part of NumPy and are widely used for performing efficient and
vectorized operations on arrays.

Aggregations of datamanipulation

Aggregations in NumPy refer to the process of performing a computation on an array and

summarizing the result. NumPy provides several functions for aggregations, which can be used to
 calculate various statistical measures of an array. Some common aggregation functions in NumPy
include:

1. np.sum: Calculates the sum of all elements in the array or along a specified axis.

import numpy as np
arr = np.array([[1, 2, 3], [4, 5, 6]])
total_sum = np.sum(arr) # 21

np.mean: Calculates the mean (average) of all elements in the array or along a specified axis.

mean_value = np.mean(arr) # 3.5

np.median: Calculates the median of all elements in the array or along a specified axis.

median_value = np.median(arr) # 3.5

np.min and np.max: Calculate the minimum and maximum values in the array or along a specified axis.

min_value = np.min(arr) # 1
max_value = np.max(arr) # 6

np.std and np.var: Calculate the standard deviation and variance of the elements in the array or along a

specified axis.

std_value = np.std(arr) # 1.7078

var_value = np.var(arr) # 2.9167

np.sum(axis=0): Calculate the sum of elements along a specified axis (0 for columns, 1 for rows).

col_sum = np.sum(arr, axis=0) # array([5, 7, 9])

np.prod(): Calculate the product of all elements in the array or along a specified axis.

prod_value = np.prod(arr) # 720

 These aggregation functions are useful for summarizing and analyzing data in NumPy arrays. They provide
efficient ways to calculate various statistical measures and perform calculations on arrays.

Computation on Arrays

Computation on arrays in NumPy allows you to perform element-wise operations, broadcasting, and
vectorized computations efficiently. Here are some key concepts and examples:

1. Element-wise operations: NumPy allows you to perform arithmetic operations (addition, subtraction,
multiplication, division) on arrays of the same shape element-wise.

python

import numpy as np
x = np.array([1, 2, 3, 4])
y = np.array([5, 6, 7, 8])
z = x + y # [6, 8, 10, 12]

Broadcasting: Broadcasting is a powerful mechanism that allows NumPy to work with arrays of different

shapes when performing arithmetic operations.

x = np.array([[1, 2, 3], [4, 5, 6]])

y = np.array([10, 20, 30])
z = x + y # [[11, 22, 33], [14, 25, 36]]

Universal functions (ufuncs): NumPy provides a set of mathematical functions that operate element-wise

on arrays. These functions are called universal functions (ufuncs).

x = np.array([1, 2, 3, 4])
y = np.sqrt(x) # [1. 1.41421356 1.73205081 2. ]
 Aggregation functions: NumPy provides functions for aggregating data in arrays, such as sum, mean, min,

max, std, and var.

x = np.array([1, 2, 3, 4])
sum_x = np.sum(x) # 10
mean_x = np.mean(x) # 2.5

Vectorized computations: NumPy allows you to express batch operations on data without writing any for

loops, which can lead to more concise and readable code.

x = np.array([[1, 2], [3, 4]])

y = np.array([[5, 6], [7, 8]])
z = x * y # Element-wise multiplication: [[5, 12], [21, 32]]

NumPy's array operations are optimized and implemented in C, making them much faster than equivalent
Python operations using lists. This makes NumPy a powerful tool for numerical computation and data
manipulation in Python.

Fancy Indexing

Fancy indexing in NumPy refers to indexing using arrays of indices or boolean arrays. It allows you
to access and modify elements of an array in a more flexible way than simple indexing. Here are
some examples of fancy indexing:

1. Indexing with arrays of indices:

import numpy as np
x = np.array([10, 20, 30, 40, 50])
indices = np.array([1, 3, 4])
y = x[indices] # [20, 40, 50]

Indexing with boolean arrays:

x = np.array([10, 20, 30, 40, 50])
mask = np.array([False, True, False, True, True])
y = x[mask] # [20, 40, 50]

Combining multiple boolean conditions: x = np.array([10, 20, 30, 40, 50])

mask = (x > 20) & (x < 50)

y = x[mask] # [30, 40]

Assigning values using fancy indexing: x = np.array([10, 20, 30, 40, 50])

indices = np.array([1, 3, 4])

x[indices] = 0
# x is now [10, 0, 30, 0, 0]

Indexing multi-dimensional arrays:

x = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

row_indices = np.array([0, 2])
col_indices = np.array([1, 2])
y = x[row_indices, col_indices] # [2, 9]

Fancy indexing can be very useful for selecting and modifying specific elements of arrays based on complex
conditions. However, it is important to note that fancy indexing creates copies of the data, not views, so
modifying the result of fancy indexing will not affect the original array.

Sorting arrays

In NumPy, you can sort arrays using the np.sort() function or the sort() method of the array object.

Both functions return a sorted copy of the array without modifying the original array. Here are some
examples of sorting arrays in NumPy:

Sorting 1D arrays:

import numpy as np
x = np.array([3, 1, 2, 5, 4])
sorted_x = np.sort(x)
# sorted_x: [1, 2, 3, 4, 5]

Sorting 2D arrays by rows or columns:

x = np.array([[3, 1, 2], [6, 4, 5], [9, 7, 8]])

# Sort each row
sorted_rows = np.sort(x, axis=1)
# sorted_rows: [[1, 2, 3], [4, 5, 6], [7, 8, 9]]

# Sort each column

sorted_cols = np.sort(x, axis=0)
# sorted_cols: [[3, 1, 2], [6, 4, 5], [9, 7, 8]]

Sorting with argsort: NumPy's argsort() function returns the indices that would sort an array. This can

be useful for sorting one array based on the values in another array.

x = np.array([3, 1, 2, 5, 4])
indices = np.argsort(x)
sorted_x = x[indices]
# sorted_x: [1, 2, 3, 4, 5]

Sorting in-place: If you want to sort an array in-place (i.e., modify the original array), you can use the

sort() method of the array object.

x = np.array([3, 1, 2, 5, 4])
x.sort()
# x: [1, 2, 3, 4, 5]
Sorting with complex numbers: Sorting works with complex numbers as well, with the real part used for

sorting. If the real parts are equal, the imaginary parts are used.

x = np.array([3+1j, 1+2j, 2+3j, 5+4j, 4+5j])

sorted_x = np.sort(x)
# sorted_x: [1.+2.j, 2.+3.j, 3.+1.j, 4.+5.j, 5.+4.j]

Structured data

Structured data in NumPy refers to arrays where each element can contain multiple fields or columns,
similar to a table in a spreadsheet or a database table. NumPy provides the numpy.ndarray class to

represent structured data, and you can create structured arrays using the numpy.array() function with a

dtype parameter specifying the data type for each field. Here's an example:

import numpy as np

# Define the data type for the structured array

dtype = [('name', 'S10'), ('age', int), ('height', float)]

# Create a structured array

data = np.array([('Alice', 25, 5.6), ('Bob', 30, 6.0)], dtype=dtype)

# Accessing elements in a structured array

print(data['name']) # ['Alice' 'Bob']
print(data['age']) # [25 30]
print(data['height']) # [5.6 6. ]
 In this example, we define a dtype for the structured array with three fields: 'name' (string of
length 10), 'age' (integer), and 'height' (float). We then create a structured array data with two
elements, each containing values for the three fields.

You can also access and modify individual elements or slices of a structured array using the field
names. For example, to access the 'name' field of the first element, you can use data[0]['name'].

Structured arrays are useful for representing and manipulating tabular data in NumPy, and they
provide a way to work with heterogeneous data in a structured manner.

Data manipulation with Pandas

Pandas is a popular Python library for data manipulation and analysis. It provides powerful data
structures, such as Series and DataFrame, that allow you to work with structured data easily.
Here's an overview of how to perform common data manipulation tasks with Pandas:

1. Importing Pandas:

import pandas as pd
Creating a DataFrame: You can create a DataFrame from various data sources, such as lists, dictionaries,

NumPy arrays, or from a file (e.g., CSV, Excel).

data = {'Name': ['Alice', 'Bob', 'Charlie'],

'Age': [25, 30, 35],
'City': ['New York', 'Los Angeles', 'Chicago']}
df = pd.DataFrame(data)

Reading and Writing Data: Pandas provides functions to read data from and write data to various file

formats, such as CSV, Excel, SQL, and more.

# Read data from a CSV file

df = pd.read_csv('data.csv')

# Write data to a CSV file

df.to_csv('data.csv', index=False)
Viewing Data: Pandas provides functions to view the data in a DataFrame, such as head(), tail(), and

sample().

print(df.head()) # View the first few rows

print(df.tail()) # View the last few rows
print(df.sample(2)) # View a random sample of rows

Selecting Data: You can select columns or rows from a DataFrame using indexing and slicing.

# Select a single column

print(df['Name'])

# Select multiple columns

print(df[['Name', 'Age']])

# Select rows based on a condition

print(df[df['Age'] > 30])

Adding and Removing Columns: You can add new columns to a DataFrame or remove existing columns.

python

# Add a new column

df['Gender'] = ['Female', 'Male', 'Male']

# Remove a column
df = df.drop('City', axis=1)

Grouping and Aggregating Data: Pandas allows you to group data based on one or more columns and

perform aggregation

# Group data by 'City' and calculate the mean age in each city
print(df.groupby('City')['Age'].mean())
Handling Missing Data: Pandas provides functions to handle missing data, such as dropna(), fillna(),

and isnull().

# Drop rows with missing values

df = df.dropna()

# Fill missing values with a specific value

df = df.fillna(0)

# Check for missing values

print(df.isnull().sum())

Merging and Joining DataFrames: Pandas provides functions to merge or join multiple DataFrames based

on a common column.

df1 = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})

df2 = pd.DataFrame({'A': [3, 4, 5], 'C': [7, 8, 9]})

# Merge DataFrames based on the 'A' column

merged_df = pd.merge(df1, df2, on='A')

These are just a few examples of how you can manipulate data with Pandas. Pandas provides a wide range
of functions and methods for data cleaning, transformation, and analysis, making it a powerful tool for data
manipulation in Python.

Data Indexing and Selection of pandas

Data indexing and selection in pandas are fundamental operations for working with data frames
and series. Here's a brief overview:
1. Indexing with []: You can use square brackets to select columns of a DataFrame or specific
elements of a Series.
• DataFrame: df['column_name'] or df[['column_name1', 'column_name2']]
• Series: s[index]
2. Label-based indexing with .loc[]: Use .loc[] for label-based indexing, where you specify the
row and column labels.
• DataFrame: df.loc[row_label, column_label] or df.loc[row_label, 'column_name']
• Series: s.loc[label]
3. Position-based indexing with .iloc[]: Use .iloc[] for position-based indexing, where you
specify the row and column positions (0-based index).
• DataFrame: df.iloc[row_index, column_index]
• Series: s.iloc[index]
4. Boolean indexing: You can use boolean arrays for selection, which allows you to filter rows based
on conditions.
pythonCopy code
df[df[ 'column_name' ] > 0 ]
5. Attribute access: If your column names are valid Python identifiers, you can use attribute access
to select columns.

df.column_name

Callable indexing with .loc[] and .iloc[]: You can use callables with .loc[] and .iloc[] for more

advanced selection.

df.loc[lambda df: df['column_name'] > 0]

These are the basic ways to index and select data in pandas. Each method has its strengths, so choose the
one that best fits your use case.

Handling missing data in pandas

Handling missing data in pandas is crucial for data analysis and modeling. Pandas provides several
methods for dealing with missing data:

1. Identifying Missing Data:

• isna(), isnull(): Returns a boolean mask indicating missing values.
• notna(), notnull(): Returns the opposite of isna() and isnull().
2. Removing Missing Data:
• dropna(): Removes rows or columns with missing values.
 df.dropna(axis=0) # Remove rows with missing values
df.dropna(axis=1) # Remove columns with missing values
Filling Missing Data:
• fillna(): Fills missing values with a specified value or method.
df.replace(to_replace=np.nan, value=0) # Replace NaN with 0
Interpolating Missing Data:
• interpolate(): Performs linear interpolation to fill missing values.
df.interpolate() # Perform linear interpolation
1. Ignoring Missing Data:
• Many operations in pandas have an NA-aware counterpart that ignores missing values (e.g.,
sum(), mean(), min(), max()).
2. Filling Missing Data with Group-specific Values:
• groupby() with transform() : Fill missing values within groups based on group-specific
values.
df.groupby('group_column')['value_column'].transform(lambda x: x.fillna(x.mean()))
Using Sentinel Values:
• Sometimes, missing values are represented by sentinel values (e.g., -999).
df.replace(to_replace=-999, value=np.nan)

These methods provide flexibility in handling missing data in pandas, allowing you to choose the approach
that best suits your data and analysis needs.

Hierarchical indexing in pandas

Hierarchical indexing, also known as MultiIndexing, enables you to work with higher-dimensional
data in pandas by allowing you to have multiple index levels on an axis. This is particularly useful
for representing higher-dimensional data in a two-dimensional DataFrame. Here's a basic overview
of hierarchical indexing in pandas:

1. Creating a MultiIndex: You can create a MultiIndex by passing a list of index levels to the index
parameter when creating a DataFrame.
import pandas as pd

arrays = [
['A', 'A', 'B', 'B'],
[1, 2, 1, 2]
]
index = pd.MultiIndex.from_arrays(arrays, names=('first', 'second'))

df = pd.DataFrame({'data': [1, 2, 3, 4]}, index=index)

Indexing with a MultiIndex: You can use tuples to index into the DataFrame at multiple levels.

# Selecting a single value

df.loc[('A', 1)]

# Selecting a single level

df.loc['A']

# Selecting on both levels

df.loc[('A', 1):('B', 1)]

# Selecting on the second level only

df.loc[:, 1]

MultiIndex columns: You can also have a MultiIndex for columns.

columns = pd.MultiIndex.from_tuples([('A', 'one'), ('A', 'two'), ('B', 'three'), ('B',

'four')],
names=('first', 'second'))
df = pd.DataFrame([[1, 2, 3, 4], [5, 6, 7, 8]], index=['foo', 'bar'], columns=columns)

Indexing with MultiIndex columns: Indexing with MultiIndex columns is similar to indexing with

MultiIndex rows.

# Selecting a single column

df[('A', 'one')]
 # Selecting on the first level of columns
df['A']

# Selecting on both levels of columns

df.loc[:, ('A', 'one'):('B', 'three')]

Creating from a dictionary with tuples: You can also create a DataFrame with a MultiIndex from a

dictionary where keys are tuples representing the index levels.

data = {('A', 1): 1, ('A', 2): 2, ('B', 1): 3, ('B', 2): 4}

df = pd.Series(data)
Hierarchical indexing provides a powerful way to represent and manipulate higher-dimensional datasets in
pandas. It allows for more flexible data manipulation and analysis.

Combining datasets in pandas

Combining datasets in pandas typically involves operations like merging, joining, and
concatenating DataFrames. Here's an overview of each:

1. Concatenation:
• Use pd.concat() to concatenate two or more DataFrames along a particular axis (row or
column).
• By default, it concatenates along axis=0 (rows), but you can specify axis=1 to concatenate
columns.
df_concatenated = pd.concat([df1, df2], axis=0)

Merging:
• Use pd.merge() to merge two DataFrames based on a common column or index.
• Specify the on parameter to indicate the column to join on.
merged_df = pd.merge(df1, df2, on='common_column')

Joining:
• Use the .join() method to join two DataFrames on their indexes.
 • By default, it performs a left join (how='left'), but you can specify other types of joins.
joined_df = df1.join(df2, how='inner')

Appending:
• Use the .append() method to append rows of one DataFrame to another.
• This is similar to concatenation along axis=0, but with more concise syntax.
appended_df = df1.append(df2)

Merging on Index:
• You can merge DataFrames based on their index using left_index=True and
right_index=True.

merged_on_index = pd.merge(df1, df2, left_index=True, right_index=True)

Specifying Merge Keys:

• For more complex merges, you can specify multiple columns to merge on using the
left_on and right_on parameters.
merged_df = pd.merge(df1, df2, left_on=['key1', 'key2'], right_on=['key1', 'key2'])

Handling Overlapping Column Names:

• If the DataFrames have overlapping column names, you can specify suffixes to add to the
column names in the merged DataFrame.
merged_df = pd.merge(df1, df2, on='key', suffixes=('_left', '_right'))

These methods provide flexible ways to combine datasets in pandas, allowing you to perform various types
of joins and concatenations based on your data's structure and requirements.

Aggregation and Grouping in pandas

Aggregation and grouping are powerful features in pandas that allow you to perform operations
on groups of data. Here's an overview:

1. GroupBy:
• Use groupby() to group data based on one or more columns
grouped = df.groupby('column_name')
Aggregation Functions:
• Apply aggregation functions like sum(), mean() , count(), min(), max(), etc., to calculate
summary statistics for each group.
grouped.sum()
Custom Aggregation:
• You can also apply custom aggregation functions using agg() with a dictionary mapping
column names to functions.
grouped.agg({'column1': 'sum', 'column2': 'mean'})

Applying Multiple Aggregations:

• You can apply multiple aggregation functions to the same column or multiple columns.

grouped['column_name'].agg(['sum', 'mean', 'count'])

Grouping with Multiple Columns:

• You can group by multiple columns to create hierarchical groupings.
Grouping with Multiple Columns:
• You can group by multiple columns to create hierarchical groupings.
Iterating Over Groups:
• You can iterate over groups using groupby() to perform more complex operations.
for name, group in grouped:
print(name)
print(group)
Filtering Groups:
• You can filter groups based on group properties using filter().
grouped.filter(lambda x: x['column_name'].sum() > threshold)

Grouping with Time Series Data:

• For time series data, you can use resample() to group by a specified frequency.
df.resample('M').sum()
Grouping with Categorical Data:
• For categorical data, you can use groupby() directly on the categorical column.
pythonCo
df.groupby('category_column').mean()
These are some of the key concepts and techniques for aggregation and grouping in pandas. They allow
you to perform a wide range of operations on grouped data efficiently.
 String operations in pandas

String operations in pandas are used to manipulate string data in Series and DataFrame columns.
Pandas provides a wide range of string methods that are vectorized, meaning they can operate on
each element of a Series without the need for explicit looping. Here are some common string
operations in pandas:

1. Accessing String Methods:

• Use the .str accessor to access string methods.
df['column_name'].str.method_name()
Lowercasing/Uppercasing:
• Convert strings to lowercase or uppercase.
python

df['column_name'].str.lower()
df['column_name'].str.upper()
String Length:
• Get the length of each string.
df['column_name'].str.len()
String Concatenation:
• Concatenate strings with other strings or Series.
df['column_name'].str.cat(sep=',')
Substrings:
• Extract substrings using slicing or regular expressions.
df['column_name'].str.slice(start=0, stop=3)
df['column_name'].str.extract(r'(\d+)')
String Splitting:
• Split strings into lists using a delimiter.
python
df['column_name'].str.split(',')
String Stripping:
• Remove leading and trailing whitespace.
df['column_name'].str.strip()
String Replacement:
• Replace parts of strings with other strings.
df['column_name'].str.replace('old', 'new')
String Counting:
• Count occurrences of a substring.
 df['column_name'].str.count('substring')
Checking for Substrings:
• Check if a substring is contained in each string.
df['column_name'].str.contains('substring')
String Alignment:
• Left or right align strings.
df['column_name'].str.ljust(width)
df['column_name'].str.rjust(width)

String Padding:
• Pad strings with a specified character to reach a desired length.
df['column_name'].str.pad(width, side='left', fillchar='0')
These are just some of the string operations available in pandas. They are efficient for working with string
data and can be used to clean and transform text data in your DataFrame.

Working with time series in pandas

Working with time series data in pandas involves using the DateTime functionality provided by
pandas to manipulate, analyze, and visualize data that is indexed by dates or times. Here's a basic
overview of working with time series in pandas:

1. Creating a DateTimeIndex:
• Ensure your DataFrame has a DateTimeIndex, which can be set using the pd.to_datetime()
function.
df.index = pd.to_datetime(df.index)
Resampling:
• Use resample() to change the frequency of your time series data (e.g., from daily to
monthly).
df.resample('M').mean()
Indexing and Slicing:
• Use DateTimeIndex to index and slice your data based on dates.
df['2019-01-01':'2019-12-31']
Shifting:
• Use shift() to shift your time series data forward or backward in time.
df.shift(1)
Rolling Windows:
 • Use rolling() to calculate rolling statistics (e.g., rolling mean, sum) over a specified
window size.
df.rolling(window=3).mean()
Time Zone Handling:
• Use tz_localize() and tz_convert() to handle time zones in your data.
df.tz_localize('UTC').tz_convert('US/Eastern')
Date Arithmetic:
• Perform arithmetic operations with dates, like adding or subtracting time deltas.
df.index + pd.DateOffset(days=1)
Resampling with Custom Functions:
• Use apply() with resample() to apply custom aggregation functions.
df.resample('M').apply(lambda x: x.max() - x.min())
Handling Missing Data:
• Use fillna() or interpolate() to handle missing data in your time series.
df.fillna(method='ffill')
Time Series Plotting:
• Use plot() to easily visualize your time series data.
df.plot()

These are some common operations for working with time series data in pandas. The DateTime

functionality in pandas makes it easy to handle and analyze time series data efficiently.

UNIT IV DATA VISUALIZATION

Matplotlib
Matplotlib is a comprehensive library for creating static, animated, and interactive visualizations in Python. It
can be used to create a wide range of plots and charts, including line plots, bar plots, histograms, scatter
plots, and more. Here's a basic overview of using Matplotlib for plotting:

Installing Matplotlib:
• You can install Matplotlib using pip:
pip install matplotlib
Importing Matplotlib:
• Import the matplotlib.pyplot module, which provides a MATLAB-like plotting interface.
import matplotlib.pyplot as plt
Creating a Simple Plot:
• Use the plot() function to create a simple line plot.
x = [1, 2, 3, 4, 5]
y = [2, 3, 5, 7, 11]
plt.plot(x, y)
plt.show()

Adding Labels and Title:

• Use xlabel(), ylabel() , and title() to add labels and a title to your plot.
plt.xlabel('x-axis label')
plt.ylabel('y-axis label')
plt.title('Title')
Customizing Plot Appearance:
• Use various formatting options to customize the appearance of your plot.
plt.plot(x, y, color='red', linestyle='--', marker='o', label='data')
plt.legend()
Creating Multiple Plots:
• Use subplot() to create multiple plots in the same figure.
plt.subplot(2, 1, 1)
plt.plot(x, y)

plt.subplot(2, 1, 2)
plt.scatter(x, y)
Saving Plots:
• Use savefig() to save your plot as an image file (e.g., PNG, PDF, SVG).
plt.savefig('plot.png') Other Types of Plots:
• Matplotlib supports many other types of plots, including bar plots, histograms, scatter plots,
and more.
plt.bar(x, y)
plt.hist(data, bins=10)
plt.scatter(x, y)
Matplotlib provides a wide range of customization options and is highly flexible, making it a powerful tool
for creating publication-quality plots and visualizations in Python.

Simple line plots in Matplotlib

Creating a simple line plot in Matplotlib involves specifying the x-axis and y-axis values and then using the
plot() function to create the plot. Here's a basic example:

import matplotlib.pyplot as plt

# Data
x = [1, 2, 3, 4, 5]
y = [2, 3, 5, 7, 11]

# Create a simple line plot

plt.plot(x, y)

# Add labels and title

plt.xlabel('X-axis label')
plt.ylabel('Y-axis label')
plt.title('Simple Line Plot')

# Display the plot

plt.show()
This code will create a simple line plot with the given x and y values, and display it with labeled axes and a
title. You can customize the appearance of the plot further by using additional arguments in the plot()

function, such as color, linestyle, and marker.

Simple scatter plots in Matplotlib

Creating a simple scatter plot in Matplotlib involves specifying the x-axis and y-axis values and then using
the scatter() function to create the plot. Here's a basic example:

import matplotlib.pyplot as plt

# Data
x = [1, 2, 3, 4, 5]
y = [2, 3, 5, 7, 11]
 # Create a simple scatter plot
plt.scatter(x, y)

# Add labels and title

plt.xlabel('X-axis label')
plt.ylabel('Y-axis label')
plt.title('Simple Scatter Plot')

# Display the plot

plt.show()

This code will create a simple scatter plot with the given x and y values, and display it with labeled axes and
a title. You can customize the appearance of the plot further by using additional arguments in the
scatter() function, such as color, s (size of markers), and alpha (transparency).

visualizing errors in Matplotlib

Visualizing errors in Matplotlib can be done using error bars or shaded regions to represent
uncertainty or variability in your data. Here are two common ways to visualize errors:

1. Error Bars:
• Use the errorbar() function to plot data points with error bars.
import matplotlib.pyplot as plt

x = [1, 2, 3, 4, 5]
y = [2, 3, 5, 7, 11]
yerr = [0.5, 0.3, 0.7, 0.4, 0.8] # Error values

plt.errorbar(x, y, yerr=yerr, fmt='o', capsize=5)

plt.xlabel('X-axis label')
plt.ylabel('Y-axis label')
plt.title('Error Bar Plot')
plt.show()

Shaded Regions:
 • Use the fill_between() function to plot shaded regions representing errors or
uncertainties.
import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(0, 10, 100)

y = np.sin(x)
error = 0.1 # Error value

plt.plot(x, y)
plt.fill_between(x, y - error, y + error, alpha=0.2)
plt.xlabel('X-axis label')
plt.ylabel('Y-axis label')
plt.title('Shaded Error Region')
plt.show()

These examples demonstrate how to visualize errors in your data using Matplotlib. You can adjust the error
values and plot styles to suit your specific needs and data.

density and contour plots in Matplotlib

Density and contour plots are useful for visualizing the distribution and density of data points in a
2D space. Matplotlib provides several functions to create these plots, such as imshow() for density
plots and contour() for contour plots. Here's how you can create them:

1. Density Plot (imshow):

• Use the imshow() function to create a density plot. You can use a 2D histogram or a kernel
density estimation (KDE) to calculate the density.
import numpy as np
import matplotlib.pyplot as plt

# Generate random data

x = np.random.normal(size=1000)
y = np.random.normal(size=1000)

# Create density plot

plt.figure(figsize=(8, 6))
plt.hist2d(x, y, bins=30, cmap='Blues')
plt.colorbar(label='Density')
plt.xlabel('X-axis')
plt.ylabel('Y-axis')
plt.title('Density Plot')
plt.show()

Contour Plot (contour):

• Use the contour() function to create a contour plot. You can specify the number of
contour levels and the colormap.
import numpy as np
import matplotlib.pyplot as plt

# Generate random data

x = np.linspace(-3, 3, 100)
y = np.linspace(-3, 3, 100)
X, Y = np.meshgrid(x, y)
Z = np.sin(X**2 + Y**2)

# Create contour plot

plt.figure(figsize=(8, 6))
plt.contour(X, Y, Z, levels=20, cmap='RdGy')
plt.colorbar(label='Intensity')
plt.xlabel('X-axis')
plt.ylabel('Y-axis')
plt.title('Contour Plot')
plt.show()

These examples demonstrate how to create density and contour plots in Matplotlib. You can customize the
plots by adjusting parameters such as the number of bins, colormap, and contour levels to better visualize
your data.

Histograms in Matplotlib
 Histograms are a useful way to visualize the distribution of a single numerical variable. Matplotlib provides
the hist() function to create histograms. Here's a basic example:
import numpy as np
import matplotlib.pyplot as plt

# Generate random data

data = np.random.normal(loc=0, scale=1, size=1000)

# Create a histogram
plt.hist(data, bins=30, color='skyblue', edgecolor='black')

# Add labels and title

plt.xlabel('Value')
plt.ylabel('Frequency')
plt.title('Histogram of Random Data')

# Display the plot

plt.show()

In this example, data is a NumPy array containing random data sampled from a normal
distribution. The hist() function creates a histogram with 30 bins, colored in sky blue with black
edges. The x-axis represents the values, and the y-axis represents the frequency of each value.

You can customize the appearance of the histogram by adjusting parameters such as bins, color,
edgecolor, and adding labels and a title to make the plot more informative.

legends in Matplotlib

Legends in Matplotlib are used to identify different elements of a plot, such as lines, markers, or
colors, and associate them with labels. Here's how you can add legends to your plots:

1. Basic Legend:
• Use the legend() function to add a legend to your plot. You can specify the labels for each
element in the legend.
import matplotlib.pyplot as plt

x = [1, 2, 3, 4, 5]
y1 = [1, 2, 3, 4, 5]
y2 = [5, 4, 3, 2, 1]

plt.plot(x, y1, label='Line 1')

plt.plot(x, y2, label='Line 2')
plt.legend()
plt.show()

Customizing Legend Location:

• You can specify the location of the legend using the loc parameter. Common location
values are 'upper left', 'upper right', 'lower left', 'lower right'.
plt.legend(loc='upper left')

Adding Legend Title:

• You can add a title to the legend using the title parameter.
plt.legend(title='Legend Title')

Customizing Legend Labels:

• You can customize the labels in the legend by passing a list of labels to the labels
parameter.
plt.legend(labels=['Label 1', 'Label 2'])

Adding Legend to Specific Elements:

• You can add legends to specific plot elements by passing the label parameter to the plot
functions.
plt.plot(x, y1, label='Line 1')
plt.plot(x, y2, label='Line 2')

Multiple Legends:
• You can create multiple legends by calling the legend() function multiple times with
different labels.
plt.plot(x, y1)
plt.plot(x, y2)
plt.legend(['Line 1', 'Line 2'], loc='upper left')
 plt.legend(['Line 3', 'Line 4'], loc='lower right')

1. Removing Legend:
• You can remove the legend from your plot by calling plt.legend().remove() or
plt.gca().legend().remove() .

These are some common ways to add and customize legends in Matplotlib. Legends are useful for
explaining the components of your plot and making it easier for viewers to understand the data.

colors in Matplotlib

In Matplotlib, you can specify colors in several ways, including using predefined color names, RGB
or RGBA tuples, hexadecimal color codes, and more. Here's how you can specify colors in
Matplotlib:

1. Predefined Color Names:

• Matplotlib provides a set of predefined color names, such as 'red', 'blue', 'green', etc.
import matplotlib.pyplot as plt

plt.plot([1, 2, 3, 4], [1, 4, 9, 16], color='red') # Plot with red color

plt.show()

RGB or RGBA Tuples:

• You can specify colors using RGB or RGBA tuples, where each value ranges from 0 to 1.
plt.plot([1, 2, 3, 4], [1, 4, 9, 16], color=(0.1, 0.2, 0.5)) # Plot with RGB color
plt.show()

Hexadecimal Color Codes:

• You can also specify colors using hexadecimal color codes.
plt.plot([1, 2, 3, 4], [1, 4, 9, 16], color='#FF5733') # Plot with hexadecimal color
plt.show()

Short Color Codes:

• Matplotlib also supports short color codes, such as 'r' for red, 'b' for blue, 'g' for green, etc.
plt.plot([1, 2, 3, 4], [1, 4, 9, 16], color='g') # Plot with green color
plt.show()
Color Maps:
• You can use color maps (colormaps) to automatically assign colors based on a range of
values.
import numpy as np

x = np.linspace(0, 10, 100)

y = np.sin(x)

plt.scatter(x, y, c=x, cmap='viridis') # Scatter plot with colormap

plt.colorbar() # Add colorbar to show the mapping
plt.show()

These are some common ways to specify colors in Matplotlib. Using colors effectively can enhance the
readability and visual appeal of your plots.

subplots in Matplotlib

Subplots in Matplotlib allow you to create multiple plots within the same figure. You can arrange subplots
in a grid-like structure and customize each subplot independently. Here's a basic example of creating
subplots:
import matplotlib.pyplot as plt
import numpy as np

# Data for plotting

x = np.linspace(0, 2*np.pi, 100)
y1 = np.sin(x)
y2 = np.cos(x)

# Create a figure and a grid of subplots

fig, axs = plt.subplots(2, 1, figsize=(8, 6))

# Plot data on the first subplot

axs[0].plot(x, y1, label='sin(x)', color='blue')
 axs[0].set_title('Plot of sin(x)')
axs[0].legend()

# Plot data on the second subplot

axs[1].plot(x, y2, label='cos(x)', color='red')
axs[1].set_title('Plot of cos(x)')
axs[1].legend()

# Adjust layout and display the plot

plt.tight_layout()
plt.show()

In this example, plt.subplots(2, 1) creates a figure with 2 rows and 1 column of subplots. The
axs variable is a NumPy array containing the axes objects for each subplot. You can then use these
axes objects to plot data and customize each subplot independently.

You can customize the arrangement of subplots by changing the arguments to plt.subplots()
(e.g., plt.subplots(2, 2) for a 2x2 grid) and by adjusting the layout using plt.tight_layout()
to prevent overlapping subplots.

text and annotation in Matplotlib

Text and annotations in Matplotlib are used to add descriptive text, labels, and annotations to your
plots. Here's how you can add text and annotations:

1. Adding Text:
• Use the text() function to add text at a specific location on the plot.
import matplotlib.pyplot as plt

plt.plot([1, 2, 3, 4], [1, 4, 9, 16])

plt.text(2, 10, 'Example Text', fontsize=12, color='red')
plt.show()

Adding Annotations:
 • Use the annotate() function to add annotations with arrows pointing to specific points on
the plot.
import matplotlib.pyplot as plt

plt.plot([1, 2, 3, 4], [1, 4, 9, 16])

plt.annotate('Example Annotation', xy=(2, 4), xytext=(3, 8),
arrowprops=dict(facecolor='black', shrink=0.05))
plt.show()

Customizing Text Properties:

• You can customize the appearance of text and annotations using various properties like
fontsize, color, fontstyle, fontweight, etc.
plt.text(2, 10, 'Example Text', fontsize=12, color='red', fontstyle='italic', fontweight='bold')

Text Alignment:
• Use the ha and va parameters to specify horizontal and vertical alignment of text.
plt.text(2, 10, 'Example Text', ha='center', va='top')

Adding Mathematical Expressions:

• You can use LaTeX syntax to include mathematical expressions in text and annotations.
plt.text(2, 10, r'$\alpha > \beta$', fontsize=12)
Rotating Text:
• Use the rotation parameter to rotate text.
plt.text(2, 10, 'Example Text', rotation=45)
Adding Background Color:
• Use the bbox parameter to add a background color to text.
plt.text(2, 10, 'Example Text', bbox=dict(facecolor='red', alpha=0.5))
These are some common techniques for adding text and annotations to your plots in Matplotlib. They can
be useful for providing additional information and context to your visualizations.

customization in Matplotlib

Customization in Matplotlib allows you to control various aspects of your plots, such as colors, line
styles, markers, fonts, and more. Here are some common customization options:

1. Changing Figure Size:

• Use figsize in plt.subplots() or plt.figure() to set the size of the figure.
fig, ax = plt.subplots(figsize=(8, 6))

Changing Line Color, Style, and Width:

• Use color, linestyle , and linewidth parameters in plot functions to customize the lines.
plt.plot(x, y, color='red', linestyle='--', linewidth=2)

Changing Marker Style and Size:

• Use marker, markersize , and markerfacecolor parameters to customize markers in scatter
plots.
plt.scatter(x, y, marker='o', s=100, c='blue')

Setting Axis Limits:

• Use xlim() and ylim() to set the limits of the x and y axes.
plt.xlim(0, 10)
plt.ylim(0, 20)

Setting Axis Labels and Title:

• Use xlabel(), ylabel() , and title() to set axis labels and plot title.
plt.xlabel('X-axis Label', fontsize=12)
plt.ylabel('Y-axis Label', fontsize=12)
plt.title('Plot Title', fontsize=14)

Changing Tick Labels:

• Use xticks() and yticks() to set custom tick labels on the x and y axes.
plt.xticks([1, 2, 3, 4, 5], ['A', 'B', 'C', 'D', 'E'])

Adding Gridlines:
• Use grid() to add gridlines to the plot.
plt.grid(True)

Changing Font Properties:

• Use fontdict parameter in text functions to set font properties.
plt.text(2, 10, 'Example Text', fontdict={'family': 'serif', 'color': 'blue', 'size': 12})
Adding Legends:
• Use legend() to add a legend to the plot.
plt.legend(['Line 1', 'Line 2'], loc='upper left')
These are some common customization options in Matplotlib. You can combine these options to create
highly customized and visually appealing plots for your data.

three dimensional plotting in Matplotlib

Matplotlib provides a toolkit called mplot3d for creating 3D plots. You can create 3D scatter plots, surface

plots, wireframe plots, and more. Here's a basic example of creating a 3D scatter plot:

import matplotlib.pyplot as plt

from mpl_toolkits.mplot3d import Axes3D
import numpy as np

# Generate random data

x = np.random.normal(size=500)
y = np.random.normal(size=500)
z = np.random.normal(size=500)

# Create a 3D scatter plot

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.scatter(x, y, z, c='b', marker='o')

# Set labels and title

ax.set_xlabel('X Label')
ax.set_ylabel('Y Label')
ax.set_zlabel('Z Label')
ax.set_title('3D Scatter Plot')

# Show plot
plt.show()
In this example, fig.add_subplot(111, projection='3d') creates a 3D subplot, and
ax.scatter(x, y, z, c='b', marker='o') creates a scatter plot in 3D space. You can customize
the appearance of the plot by changing parameters such as c (color), marker, and adding labels
and a title.

You can also create surface plots and wireframe plots using the plot_surface() and
plot_wireframe() functions, respectively. Here's an example of a 3D surface plot:

import matplotlib.pyplot as plt

from mpl_toolkits.mplot3d import Axes3D
import numpy as np

# Generate data
x = np.linspace(-5, 5, 100)
y = np.linspace(-5, 5, 100)
x, y = np.meshgrid(x, y)
z = np.sin(np.sqrt(x**2 + y**2))

# Create a 3D surface plot

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.plot_surface(x, y, z, cmap='viridis')

# Set labels and title

ax.set_xlabel('X Label')
ax.set_ylabel('Y Label')
ax.set_zlabel('Z Label')
ax.set_title('3D Surface Plot')

# Show plot
plt.show()

These examples demonstrate how to create basic 3D plots in Matplotlib. You can explore the mplot3d

toolkit and its functions to create more advanced 3D visualizations.

Geographic Data with Basemap in Matplotlib

Basemap is a toolkit for Matplotlib that allows you to create maps and plot geographic data. It provides
various map projections and features for customizing maps. Here's a basic example of plotting geographic
data using Basemap:

import matplotlib.pyplot as plt

from mpl_toolkits.basemap import Basemap

# Create a map
plt.figure(figsize=(10, 6))
m = Basemap(projection='mill',llcrnrlat=-90,urcrnrlat=90,\
llcrnrlon=-180,urcrnrlon=180,resolution='c')
m.drawcoastlines()
m.drawcountries()
m.fillcontinents(color='lightgray',lake_color='aqua')
m.drawmapboundary(fill_color='aqua')

# Plot cities
lons = [-77.0369, -122.4194, 120.9660, -0.1276]
lats = [38.9072, 37.7749, 14.5995, 51.5074]
cities = ['Washington, D.C.', 'San Francisco', 'Manila', 'London']
x, y = m(lons, lats)
m.scatter(x, y, marker='o', color='r')

# Add city labels

for city, xpt, ypt in zip(cities, x, y):
plt.text(xpt+50000, ypt+50000, city, fontsize=10, color='blue')

# Add a title
plt.title('Cities Around the World')

# Show the map

plt.show()
 In this example, we first create a Basemap instance with the desired projection and map extent. We
then draw coastlines, countries, continents, and a map boundary. Next, we plot cities on the map
using the scatter() method and add labels for each city using plt.text(). Finally, we add a title
to the plot and display the map.

Basemap offers a wide range of features for working with geographic data, including support for
various map projections, drawing political boundaries, and plotting points, lines, and shapes on
maps. You can explore the Basemap documentation for more advanced features and
customization options.

Visualization with Seaborn

Seaborn is a Python visualization library based on Matplotlib that provides a high-level interface
for creating attractive and informative statistical graphics. It is particularly useful for visualizing
data from Pandas DataFrames and NumPy arrays. Seaborn simplifies the process of creating
complex visualizations such as categorical plots, distribution plots, and relational plots. Here's a
brief overview of some of the key features of Seaborn:

1. Installation:
• You can install Seaborn using pip:
pip install seaborn

Importing Seaborn:
• Import Seaborn as sns conventionally:
import seaborn as sns

Loading Example Datasets:

• Seaborn provides several built-in datasets for practice and exploration:
tips = sns.load_dataset('tips')

Categorical Plots:
• Seaborn provides several functions for visualizing categorical data, such as sns.catplot(),
sns.barplot(), sns.countplot(), and sns.boxplot().
sns.catplot(x='day', y='total_bill', data=tips, kind='box')

Distribution Plots:
• Seaborn offers various functions for visualizing distributions, including sns.distplot(),
sns.kdeplot(), and sns.histplot().
sns.distplot(tips['total_bill'])
 Relational Plots:
• Seaborn provides functions for visualizing relationships between variables, such as
sns.relplot(), sns.scatterplot(), and sns.lineplot().
sns.relplot(x='total_bill', y='tip', data=tips, kind='scatter')

Heatmaps:
• Seaborn can create heatmaps to visualize matrix-like data using sns.heatmap() .
flights = sns.load_dataset('flights').pivot('month', 'year', 'passengers')
sns.heatmap(flights, annot=True, fmt='d')

Pairplots:
• Pairplots are useful for visualizing pairwise relationships in a dataset using sns.pairplot().
sns.pairplot(tips, hue='sex')

1. Styling and Themes:

• Seaborn allows you to customize the appearance of plots using styling functions
(sns.set(), sns.set_style(), sns.set_context()) and themes (sns.set_theme()).
2. Other Plots:
• Seaborn offers many other types of plots and customization options. The official Seaborn
documentation provides detailed examples and explanations for each type of plot.

Seaborn is built on top of Matplotlib and integrates well with Pandas, making it a powerful tool for
visualizing data in Python.

UNIT V HANDLING LARGE DATA

techniques for handling large volumes of data

Handling large volumes of data requires a combination of techniques to efficiently process, store,
and analyze the data. Some common techniques include:

1. Distributed computing: Using frameworks like Apache Hadoop and Apache Spark to distribute
data processing tasks across multiple nodes in a cluster, allowing for parallel processing of large
datasets.
2. Data compression: Compressing data before storage or transmission to reduce the amount of
space required and improve processing speed.
3. Data partitioning: Dividing large datasets into smaller, more manageable partitions based on
certain criteria (e.g., range, hash value) to improve processing efficiency.
4. Data deduplication: Identifying and eliminating duplicate data to reduce storage requirements
and improve data processing efficiency.
5. Database sharding: Partitioning a database into smaller, more manageable parts called shards,
which can be distributed across multiple servers for improved scalability and performance.
6. Stream processing: Processing data in real-time as it is generated, allowing for immediate
analysis and decision-making.
7. In-memory computing: Storing data in memory instead of on disk to improve processing speed,
particularly for frequently accessed data.
8. Parallel processing: Using multiple processors or cores to simultaneously execute data processing
tasks, improving processing speed for large datasets.
9. Data indexing: Creating indexes on data fields to enable faster data retrieval, especially for
queries involving large datasets.
10. Data aggregation: Combining multiple data points into a single, summarized value to reduce the
overall volume of data while retaining important information.

These techniques can be used individually or in combination to handle large volumes of data
effectively and efficiently.

programming tips for dealing with large data sets

When dealing with large datasets in programming, it's important to use efficient techniques to
manage memory, optimize processing speed, and avoid common pitfalls. Here are some
programming tips for dealing with large datasets:

1. Use efficient data structures: Choose data structures that are optimized for the operations you
need to perform. For example, use hash maps for fast lookups, arrays for sequential access, and
trees for hierarchical data.
2. Lazy loading: Use lazy loading techniques to load data into memory only when it is needed,
rather than loading the entire dataset at once. This can help reduce memory usage and improve
performance.
3. Batch processing: Process data in batches rather than all at once, especially for operations like
data transformation or analysis. This can help avoid memory issues and improve processing speed.
4. Use streaming APIs: Use streaming APIs and libraries to process data in a streaming fashion,
which can be more memory-efficient than loading the entire dataset into memory.
5. Optimize data access: Use indexes and caching to optimize data access, especially for large
datasets. This can help reduce the time it takes to access and retrieve data.
6. Parallel processing: Use parallel processing techniques, such as multithreading or
multiprocessing, to process data concurrently and take advantage of multi-core processors.
7. Use efficient algorithms: Choose algorithms that are optimized for large datasets, such as sorting
algorithms that use divide and conquer techniques or algorithms that can be parallelized.
8. Optimize I/O operations: Minimize I/O operations and use buffered I/O where possible to reduce
the overhead of reading and writing data to disk.
9. Monitor memory usage: Keep an eye on memory usage and optimize your code to minimize
memory leaks and excessive memory consumption.
10. Use external storage solutions: For extremely large datasets that cannot fit into memory,
consider using external storage solutions such as databases or distributed file systems.

Case studies: Predicting malicious URLs,

Predicting malicious URLs is a critical task in cybersecurity to protect users from phishing attacks,
malware distribution, and other malicious activities. Machine learning models can be used to
classify URLs as either benign or malicious based on features such as URL length, domain age,
presence of certain keywords, and historical data. Here are two case studies that demonstrate how
machine learning can be used to predict malicious URLs:

1. Google Safe Browsing:

• Google Safe Browsing is a service that helps protect users from malicious websites by
identifying and flagging unsafe URLs.
• The service uses machine learning models to analyze URLs and classify them as safe or
unsafe.
• Features used in the model include URL length, domain reputation, presence of suspicious
keywords, and similarity to known malicious URLs.
• The model is continuously trained on new data to improve its accuracy and effectiveness.
2. Microsoft SmartScreen:
• Microsoft SmartScreen is a feature in Microsoft Edge and Internet Explorer browsers that
helps protect users from phishing attacks and malware.
• SmartScreen uses machine learning models to analyze URLs and determine their safety.
• The model looks at features such as domain reputation, presence of phishing keywords, and
similarity to known malicious URLs.
• SmartScreen also leverages data from the Microsoft Defender SmartScreen service to
improve its accuracy and coverage.

In both cases, machine learning is used to predict the likelihood that a given URL is malicious
based on various features and historical data. These models help protect users from online threats
and improve the overall security of the web browsing experience.
 Case studies: Building a recommender system
Building a recommender system involves predicting the "rating" or "preference" that a user would
give to an item. These systems are widely used in e-commerce, social media, and content
streaming platforms to personalize recommendations for users. Here are two case studies that
demonstrate how recommender systems can be built:

1. Netflix Recommendation System:

• Netflix uses a recommendation system to suggest movies and TV shows to its users.
• The system uses collaborative filtering, which involves analyzing user behavior (e.g., viewing
history, ratings) to identify patterns and make recommendations.
• Netflix also incorporates content-based filtering, which considers the characteristics of the
items (e.g., genre, cast, director) to make recommendations.
• The system uses machine learning algorithms such as matrix factorization and deep learning
to improve the accuracy of its recommendations.
• Netflix continuously collects data on user interactions and feedback to refine its
recommendation algorithms.
2. Amazon Product Recommendation System:
• Amazon uses a recommendation system to suggest products to its customers based on
their browsing and purchase history.
• The system uses collaborative filtering to identify products that are popular among similar
users.
• Amazon also uses item-to-item collaborative filtering, which recommends products that are
similar to those that a user has previously viewed or purchased.
• The system incorporates user feedback and ratings to improve the relevance of its
recommendations.
• Amazon's recommendation system is powered by machine learning algorithms that analyze
large amounts of data to make personalized recommendations.

In both cases, the recommendation systems use machine learning and data analysis techniques to
analyze user behavior and make personalized recommendations. These systems help improve user
engagement, increase sales, and enhance the overall user experience.

Tools and techniques needed for dealing with large data

Dealing with large datasets requires a combination of tools and techniques to manage, process,
and analyze the data efficiently. Here are some key tools and techniques:

1. Big Data Frameworks: Frameworks such as Apache Hadoop, Apache Spark, and Apache Flink
provide tools for distributed storage and processing of large datasets.
2. Data Storage: Use of distributed file systems like Hadoop Distributed File System (HDFS), cloud
storage services like Amazon S3, or NoSQL databases like Apache Cassandra or MongoDB for
storing large volumes of data.
3. Data Processing: Techniques such as MapReduce, Spark RDDs, and Spark DataFrames for parallel
processing of data across distributed computing clusters.
4. Data Streaming: Tools like Apache Kafka or Apache Flink for processing real-time streaming data.
5. Data Compression: Techniques like gzip, Snappy, or Parquet for compressing data to reduce
storage requirements and improve processing speed.
6. Data Partitioning: Divide large datasets into smaller, more manageable partitions based on
certain criteria to improve processing efficiency.
7. Distributed Computing: Use of cloud computing platforms like Amazon Web Services (AWS),
Google Cloud Platform (GCP), or Microsoft Azure for scalable and cost-effective processing of
large datasets.
8. Data Indexing: Create indexes on data fields to enable faster data retrieval, especially for queries
involving large datasets.
9. Machine Learning: Use of machine learning algorithms and libraries (e.g., scikit-learn, TensorFlow)
for analyzing and deriving insights from large datasets.
10. Data Visualization: Tools like Matplotlib, Seaborn, or Tableau for visualizing large datasets to gain
insights and make data-driven decisions.

By leveraging these tools and techniques, organizations can effectively manage and analyze large
volumes of data to extract valuable insights and drive informed decision-making.

Data preparation for dealing with large data

Data preparation is a crucial step in dealing with large datasets, as it ensures that the data is clean,
consistent, and ready for analysis. Here are some key steps involved in data preparation for large
datasets:

1. Data Cleaning: Remove or correct any errors or inconsistencies in the data, such as missing
values, duplicate records, or outliers.
2. Data Integration: Combine data from multiple sources into a single dataset, ensuring that the
data is consistent and can be analyzed together.
3. Data Transformation: Convert the data into a format that is suitable for analysis, such as
converting categorical variables into numerical ones or normalizing numerical variables.
4. Data Reduction: Reduce the size of the dataset by removing unnecessary features or aggregating
data to a higher level of granularity.
5. Data Sampling: If the dataset is too large to analyze in its entirety, use sampling techniques to
extract a representative subset of the data for analysis.
6. Feature Engineering: Create new features from existing ones to improve the performance of
machine learning models or better capture the underlying patterns in the data.
7. Data Splitting: Split the dataset into training, validation, and test sets to evaluate the performance
of machine learning models and avoid overfitting.
8. Data Visualization: Visualize the data to explore its characteristics and identify any patterns or
trends that may be present.
9. Data Security: Ensure that the data is secure and protected from unauthorized access or loss,
especially when dealing with sensitive information.

Model building for dealing with large data

When building models for large datasets, it's important to consider scalability, efficiency, and
performance. Here are some key techniques and considerations for model building with large data:

1. Use Distributed Computing: Utilize frameworks like Apache Spark or TensorFlow with distributed
computing capabilities to process large datasets in parallel across multiple nodes.
2. Feature Selection: Choose relevant features and reduce the dimensionality of the dataset to
improve model performance and reduce computation time.
3. Model Selection: Use models that are scalable and efficient for large datasets, such as gradient
boosting machines, random forests, or deep learning models.
4. Batch Processing: If real-time processing is not necessary, consider batch processing techniques
to handle large volumes of data in scheduled intervals.
5. Sampling: Use sampling techniques to create smaller subsets of the data for model building and
validation, especially if the entire dataset cannot fit into memory.
6. Incremental Learning: Implement models that can be updated incrementally as new data
becomes available, instead of retraining the entire model from scratch.
7. Feature Engineering: Create new features or transform existing features to better represent the
underlying patterns in the data and improve model performance.
8. Model Evaluation: Use appropriate metrics to evaluate model performance, considering the
trade-offs between accuracy, scalability, and computational resources.
9. Parallelization: Use parallel processing techniques within the model training process to speed up
computations, such as parallelizing gradient computations in deep learning models.
10. Data Partitioning: Partition the data into smaller subsets for training and validation to improve
efficiency and reduce memory requirements.

By employing these techniques, data scientists and machine learning engineers can build models
that are scalable, efficient, and capable of handling large datasets effectively.

Presentation and automation for dealing with large data

Presentation and automation are key aspects of dealing with large datasets to effectively
communicate insights and streamline data processing tasks. Here are some strategies for
presentation and automation:

1. Visualization: Use data visualization tools like Matplotlib, Seaborn, or Tableau to create
visualizations that help stakeholders understand complex patterns and trends in the data.
2. Dashboarding: Build interactive dashboards using tools like Power BI or Tableau that allow users
to explore the data and gain insights in real-time.
3. Automated Reporting: Use tools like Jupyter Notebooks or R Markdown to create automated
reports that can be generated regularly with updated data.
4. Data Pipelines: Implement data pipelines using tools like Apache Airflow or Luigi to automate
data ingestion, processing, and analysis tasks.
5. Model Deployment: Use containerization technologies like Docker to deploy machine learning
models as scalable and reusable components.
6. Monitoring and Alerting: Set up monitoring and alerting systems to track the performance of
data pipelines and models, and to be notified of any issues or anomalies.
7. Version Control: Use version control systems like Git to track changes to your data processing
scripts and models, enabling collaboration and reproducibility.
8. Cloud Services: Leverage cloud services like AWS, Google Cloud Platform, or Azure for scalable
storage, processing, and deployment of large datasets and models.

By incorporating these strategies, organizations can streamline their data processes, improve
decision-making, and derive more value from their large datasets.