||Jai Sri Gurudev ||
Sri AdichunchanagiriShikshana Trust®
SJB INSTITUTE OF TECHNOLOGY
Accredited by NBA & NAAC with ‘A+’ Grade
No. 67, BGS Health & Education City, Dr. Vishnuvardhan Road Kengeri,
Bangalore – 560 060
Department of Computer Science & Engineering
Python for Data Analytics [23CSE312]
MODULE - 1
An Introduction to Data Analysis
3rd SEMESTER – B. E
Academic Year: 2024 – 2025 (Odd)
Prepared By: Shilpashree S
Introduction to Data Analysis
● Data is everywhere: Produced by automatic systems, sensors, everyday actions (bank transactions,
social media).
● Data vs. Information: Data itself is not information; it becomes useful when processed.
● Data analysis: The process of extracting actionable insights from raw data.
Purpose of Data Analysis:
● Extract hidden information: Insights that aren't immediately obvious.
● Understand systems: Study mechanisms and predict system responses.
● Make predictions: Forecast future outcomes and evolutions based on data patterns.
Introduction to Data Analysis
Evolution of Data Analysis
● Started with simple data protection.
● Now a formal discipline: Development of methodologies and models.
● Modeling: Translate systems into mathematical or logical forms.
Importance of Models
● Goal of modeling: Make accurate predictions.
● Quality of predictions: Depends on the choice of dataset and modeling techniques.
● Data preparation: Data extraction, cleaning, and preparation are crucial parts of the analysis.
Data Visualization
● Importance: Visual representation aids in understanding data.
● Chart types: Various charts (bar, line, scatter, etc.) help visualize data patterns.
● Helps uncover hidden insights: Visualization can reveal relationships that raw data alone cannot.
Introduction to Data Analysis
Testing and Validation
● Model testing: Use a different dataset (not used in the model) to check predictions.
● Error calculation: Measure how well the model predicts actual outcomes.
● Assess model validity: Compare with other models to determine performance.
Deployment of Data Analysis
● Final step: Implement decisions based on the model's predictions.
● Risk prediction: Understand risks and impacts of decisions.
● Real-world application: Using data analysis in decision-making improves outcomes.
Conclusion
● Data analysis: A powerful tool to extract insights, make predictions, and support decision-making.
● Relevance: Applicable in various professions.
● Next step: Utilize these techniques to test hypotheses and understand complex systems.
Knowledge Domains of the Data Analyst
Introduction
● Data Analysis: Interdisciplinary field solving problems across various domains.
● Skills Needed: A data analyst must have proficiency in multiple disciplines such as computer
science, mathematics, and domain-specific knowledge.
● Interdisciplinary Teams: Larger projects often require collaboration across different expertise.
Core Knowledge Domains
1. Computer Science
2. Mathematics and Statistics
3. Machine Learning and AI
4. Domain-Specific Expertise
Knowledge Domains of the Data Analyst
Computer Science
● Why It’s Essential: Data analysis relies on computational tools for managing, processing, and
visualizing data.
● Tools & Skills:
○ Programming Languages: Python, C++, Java
○ Software: MATLAB, IDL
○ Data Formats: JSON, XML, CSV, XLS
○ SQL & Databases: Querying and extracting data efficiently
○ Web Scraping: Extracting data from websites (HTML tables, charts)
Knowledge Domains of the Data Analyst
Mathematics & Statistics
● Key to Data Processing: Provides the foundation for data analysis methodologies.
● Commonly used statistical techniques in data analysis are:
○ Bayesian Methods
○ Regression Analysis
○ Clustering Techniques
● Python Libraries: Simplify the application of complex mathematical and statistical models.
Knowledge Domains of the Data Analyst
Machine Learning & Artificial Intelligence
● Advanced Data Tools: Automate pattern recognition, trend analysis, and insights extraction.
● Key Concepts:
○ Algorithms to find patterns, clusters, and trends.
○ Importance: Speeds up and improves the accuracy of data insights.
● Python Libraries: Tools for implementing machine learning techniques.
Domain-Specific Expertise
● Importance of Field Knowledge: Understanding the data’s origin is critical for accurate analysis.
● Example Fields: Biology, finance, physics, social statistics.
● Collaboration: Work with experts when needed to understand the data's context.
Problem-Solving Approach
● For Smaller Problems: Analysts must be flexible, identifying new skills or knowledge needed.
● Solution: Learn new methods or consult domain experts to solve issues during analysis.
Understanding the Nature of the Data
Data in Data Analysis
● Data as the core: The key component in all processes of data analysis.
● Purpose: Extract insights to increase knowledge about the system under study.
When Data Becomes Information
● Definition of data: Measurable or categorizable events recorded in the world.
● Transformation: Through analysis, data help in understanding events or making predictions.
● Data leads to informed decisions: Processing raw data can guide future actions.
From Information to Knowledge
● Information: Provides details about specific events.
● Knowledge: Emerges when information forms rules, enabling predictions about future events.
● Key takeaway: Knowledge is the ultimate outcome of successful data analysis.
Understanding the Nature of the Data
Types of Data
● Two main categories:
1. Categorical
2. Numerical
Categorical Data
● Categorical Data: Values or observations divided into groups.
● Two types:
1. Nominal: No intrinsic order (e.g., colors, types of cars).
2. Ordinal: Has a specific, predetermined order (e.g., rankings, education levels).
Numerical Data
● Numerical Data: Derived from measurements.
● Two types:
1. Discrete: Countable, distinct, separated values (e.g., number of students).
2. Continuous: Can assume any value within a range (e.g., height, temperature).
Understanding the Nature of the Data
Summary
● Data: The raw material of data analysis.
● Transformation process: Data → Information → Knowledge.
● Understanding data types: Helps guide analysis and improve predictions.
The Data Analysis Process
● Data analysis is a multi-step process for transforming raw data into insights and produce data
visualizations, build predictive models, and derive actionable results.
● Data analysis is a series of interconnected stages where each plays a key role. Problem definition to
deployment, each step builds on the previous one.
● Data analysis is schematized as a process chain consisting of the following sequence of stages:
a. Problem definition
b. Data extraction
c. Data cleaning
d. Data transformation
e. Data exploration
f. Predictive modeling
g. Model validation/test
h. Visualization and interpretation of results
i. Deployment of the solution
Problem Definition in Data Analysis
● Data analysis begins with defining a problem to solve.
● The problem should relate to a specific system, mechanism, or process that requires
understanding or optimization.
● Focus on the system’s behavior to either:
○ Make predictions about its future behavior.
○ Make informed decisions to improve its functioning
● Properly document the scientific or business problem to:
○ Provide clarity and focus for the analysis.
○ Ensure the analysis aligns with desired outcomes.
● Once the problem is defined:
○ Begin project planning to identify needed resources.
○ Determine the professionals and tools required.
● Build a cross-disciplinary team for different perspectives.
● A good team is key to solving complex problems effectively.
Data Extraction
● Importance of Data:
○ Data selection is crucial for building a predictive model.
○ The data must reflect real-world behavior to ensure accurate analysis.
● Challenges in Data Collection:
○ Poorly chosen data can result in inaccurate models.
○ Unbalanced or unrepresentative datasets lead to poor predictions.
● Source of Data:
○ Laboratory Data: Experimental data are easier to identify.
○ Real-World Data: Can involve external experiments, surveys, or interviews.
○ Multiple data sources may be needed to create a comprehensive dataset.
● Data Search Techniques:
○ Web Scraping: Extracts data from HTML pages.
○ Specialized tools and software are used to gather unstructured web data.
● Goal:
○ To collect data that are reliable and representative for accurate predictions.
Data Preparation
1. Time and Resource Intensive:
○ Data preparation is one of the most resource- and time-consuming steps in data analysis.
○ Requires integrating data from multiple sources, each with different formats.
2. Key Activities in Data Preparation:
○ Data Cleaning: Removing invalid, ambiguous, or missing values.
○ Normalization: Ensuring data are in a consistent format.
○ Transformation: Converting data into an optimized, tabular format.
3. Challenges in Data Preparation:
○ Handling replicated fields.
○ Dealing with out-of-range or erroneous data.
4. Goal:
○ Prepare a clean, structured dataset that is suitable for the scheduled analysis methods.
Data Exploration/Visualization
Purpose of Data Exploration:
● A preliminary examination to understand patterns, relationships, and trends in the data.
● Guides in determining the most suitable analysis methods for model building.
Generally, this phase, in addition to a detailed study of charts through the visualization data, may consist
of one or more of the following activities:
● Summarizing data Grouping data: Reducing complexity without losing key insights.
● Exploration of the relationship between the various attributes: Finding common attributes and
organizing data into meaningful groups.
● Identification of patterns and trends: Identifying trends, correlations, and anomalies.
● Construction of regression models & Construction of classification models: Constructing
models to predict and categorize.
Importance of Data Visualization:
● Transforms raw data into easily understandable charts, graphs, and visual forms.
● Highlights patterns and relationships not easily seen in raw data.
● Tools like decision trees and association rules further enhance data interpretation.
Predictive Modeling
Predictive modeling is a data analysis process used to create or select a statistical model that predicts
the probability of future outcomes.
Main Objectives:
● Prediction: Using models to forecast future data values (Regression Models).
● Classification: Categorizing new data into pre-defined groups (Classification Models).
● Descriptive: Grouping data by shared characteristics (Clustering Models).
Types of Predictive Models:
● Classification Models: Results in categorical outcomes.
● Regression Models: Results in numeric predictions.
● Clustering Models: Provides descriptive groupings of data.
Predictive Modeling
Common Modeling Techniques:
● Linear Regression: Predicts continuous numeric outcomes.
● Logistic Regression: Predicts categorical outcomes.
● Decision Trees: Classifies or predicts based on feature splits.
● k-Nearest Neighbors (k-NN): Classifies or predicts based on proximity to known data points.
Model Selection:
a. Different models are suited for different data types and goals.
b. Some models provide transparent insights (e.g., linear regression), while others may act as a
"black box" (e.g., deep learning).
Model Validation
● Model validation is the process of testing the predictive model against new data to assess its
accuracy and generalization to unseen situations.
Training vs. Validation Sets:
● Training Set: Data used to build the model.
● Validation Set: Data used to test and validate the model's accuracy on new or unseen data.
Evaluation Through Comparison:
● Compare model predictions with actual system data to assess errors and limitations.
● Helps determine validity range: The model may perform well only within certain value ranges.
Model Validation
Key Techniques for Validation:
● Cross-Validation:
○ The training set is split into multiple parts.
○ Each part is used as a validation set while others are used for training.
○ Iterative process ensures refinement and minimizes overfitting.
● Outcome of Validation:
○ Quantitatively evaluates model effectiveness.
○ Enables comparison with other models to select the most accurate.
Deployment
Deployment is the final phase of the data analysis process, where the results are put into practice to
provide value.
Business and Technical Outcomes:
● In business, deployment delivers actionable insights for decision-making.
● In technical/scientific contexts, deployment results in design solutions or publications.
Types of Deployment:
● Report Creation: Data analysts provide a report summarizing:
○ Analysis Results
○ Decision Recommendations
○ Risk Assessments
○ Business Impact Measurement
Deployment
Predictive Models Deployment:
● Predictive models can be deployed as:
○ Standalone applications
○ Integrated into existing systems for automation or optimization.
● Key Focus:
○ Translating data insights into real-world benefits through client or management decisions
and solutions.
Quantitative and Qualitative Data Analysis
Quantitative Data Analysis:
● Focuses on numerical or categorical data.
● Involves structured data with logical order and categories.
● Mathematical models and statistics are used to derive objective conclusions.
● Commonly applied in scientific, financial, and technical analyses.
Qualitative Data Analysis:
● Deals with unstructured data (e.g., text, images, audio).
● Often relies on ad hoc methodologies to extract meaning.
● Conclusions can involve subjective interpretation.
● Frequently used in studying social phenomena or complex systems.
Quantitative and Qualitative Data Analysis
Key Differences:
● Quantitative Analysis:
○ Objective and data-driven.
○ Results in quantitative predictions (e.g., regression models).
● Qualitative Analysis:
○ Often subjective and exploratory.
○ Aims to understand complex systems with descriptive insights.
Applications:
● Quantitative: Measuring business performance, forecasting, engineering studies.
● Qualitative: Social research, user experience, content analysis.
Open Data
Here is a list of some Open Data available online. You can find a more complete list and details of the Open Data available
online in Appendix B.
● DataHub (http://datahub.io/dataset)
● World Health Organization (http://www.who.int/research/en/)
● Data.gov (http://data.gov)
● European Union Open Data Portal (http://open-data.europa.eu/en/data/)
● Amazon Web Service public datasets (http://aws.amazon.com/datasets)
● Facebook Graph (http://developers.facebook.com/docs/graph-api)
● Healthdata.gov (http://www.healthdata.gov)
● Google Trends (http://www.google.com/trends/explore)
● Google Finance (https://www.google.com/finance)
● Google Books Ngrams (http://storage.googleapis.com/books/ngrams/books/datasetsv2.html)
● Machine Learning Repository (http://archive.ics.uci.edu/ml/)
Python and Data Analysis
The focus here is on using Python to develop all data analysis concepts. Python has become a popular
programming language in scientific and data circles because it offers a wide array of tools for analysis and
data manipulation.
Why Python Over Other Languages?:
● While languages like R and Matlab are also used for data analysis, Python stands out because it's
not just a tool for data processing but offers unique advantages:
○ Python has a growing ecosystem of libraries that make advanced data analysis easier and
more efficient. Examples include NumPy, pandas, and Matplotlib.
○ It can interface with other languages like C and Fortran, meaning it can leverage even
more power for specific tasks.
○ .
Python and Data Analysis
More than Just Data Analysis:
● Unlike specialized languages that are solely used for data (like R), Python is versatile. You can use
it for general programming, creating scripts, interacting with databases, and even web
development (via frameworks like Django).
○ Example: You can build a data analysis project and integrate it into a web
application—something harder to do with languages like R
A Future-Proof Language:
○ Given its flexibility, expanding libraries, and powerful tools, Python is considered a smart
choice for anyone looking to dive into data analysis.
■ It’s not just a current trend—it’s likely to remain an essential tool for data analysts in the
future.
Reference:
Text Book 1 Chapter 1: Python for Data Analytics, Fabio Nelli.
