Applied Data Science

Module 1
Dr. Srabana Pramanik & Ms Divya Shamili
Assistant professor
Dept. of SCSE
Presidency University
 Data universe
The term "data universe" refers to the entirety of data that is relevant
to a particular context, problem, or domain. It encompasses all the
data available or potentially accessible for analysis, exploration, and
decision-making within a specific scope.
In a broader sense, the data universe can encompass a wide range of
data types, sources, and formats, including structured data (such as
databases and spreadsheets), unstructured data (like text documents
and images), semi-structured data (like JSON or XML files), and even
real-time streaming data.
By comprehensively exploring the data universe, data scientists and
analysts can derive insights, patterns, and trends that lead to
informed decision-making and valuable outcomes.
Definition of data science
Data Science is a combination of multiple disciplines
that uses statistics, data analysis, and machine
learning to analyze data and to extract knowledge and
insights from it.
Sources of Data
Data can be sourced from various places, both digital and physical. The
source of data depends on the type of information you're seeking and the
context of your analysis. Here are some common sources of data:
Databases: Structured data can be sourced from relational databases,
data warehouses, and NoSQL databases. These can include customer
records, sales transactions, inventory data, and more.
Files: Data can be obtained from various file formats, such as CSV, Excel
spreadsheets, JSON, XML, and more. These files might contain survey
responses, logs, or other structured data.
Web APIs: Many online platforms and services offer APIs (Application
Programming Interfaces) that allow you to access and retrieve data.
Examples include social media platforms, weather data providers, and
financial data services.
Web Scraping: You can extract data from websites using web scraping
techniques. This is useful for collecting data that might not be available
through APIs.
Sensor Data: In fields like IoT (Internet of Things), data can be collected
from sensors embedded in devices, vehicles, machinery, and other physical
objects.
Importance of Data Science
Data science is a rapidly growing and essential field that combines
various techniques, algorithms, processes, and systems to extract
knowledge and insights from structured and unstructured data. Its
importance has surged due to several factors:
Data Abundance: In the digital age, enormous amounts of data
are being generated daily, offering valuable insights that can drive
informed decision-making across industries.
Informed Decision-Making: Data science enables
organizations to make data-driven decisions, helping them optimize
processes, identify trends, forecast outcomes, and improve overall
business strategies.
Competitive Advantage: Companies that effectively use data
science gain a competitive edge by understanding customer
preferences, market trends, and their own operations better than their
competitors.
 Textual Data: Text data can come from sources like articles, books, social media posts, emails, and
more. Natural language processing (NLP) techniques are often used to analyze and extract insights from
textual data.
Images and Videos: Visual data, such as images and videos, can provide valuable information.
Computer vision techniques are used to analyze and interpret this type of data.
Surveys and Questionnaires: Data can be collected through surveys, questionnaires, and
feedback forms. This can provide insights into user preferences, opinions, and behaviors.
Public Data Repositories: Various organizations and governments provide publicly available
datasets for research and analysis. Examples include data.gov, Kaggle, and UCI Machine Learning
Repository.
Social Media: Social media platforms are a rich source of user-generated content, including text,
images, videos, and interactions.
Historical Records: Archives, historical documents, and records can provide insights into the past
and support historical research.
Physical Measurements: Scientific experiments, research studies, and industrial processes often
generate data through physical measurements and observations.
Transaction Logs: E-commerce platforms, financial institutions, and online services often maintain
transaction logs that can be used for analysis.
Private Data: Organizations may have internal databases and records that contain proprietary or
sensitive information.
It's important to note that data collection should always adhere to ethical and legal considerations, such as
privacy regulations and intellectual property rights.
Additionally, the quality and reliability of the data source are crucial factors for ensuring accurate and
meaningful analyses.
Personalization: Data science powers personalized
recommendations, marketing strategies, and user experiences,
leading to improved customer satisfaction and engagement.
Scientific Discovery: In fields such as medicine, genetics,
and environmental science, data science aids in analyzing
complex datasets, identifying patterns, and making breakthrough
discoveries.
Risk Management: Data science is crucial for assessing
and mitigating risks by modeling and predicting potential
challenges, be it in finance, insurance, or cybersecurity.
Healthcare and Medicine: Data science facilitates patient
diagnosis, treatment optimization, drug discovery, and
population health management through analysis of medical data.
Social Impact: Data science can be used to address societal
challenges, such as predicting disease outbreaks, optimizing urban
planning, and enhancing disaster response.

Innovation: Data-driven insights fuel innovation and research,

helping businesses create new products, services, and technologies
that cater to evolving customer needs.

Efficiency and Automation: Data science automates tasks,

streamlines processes, and optimizes workflows, leading to increased
efficiency and reduced operational costs.
 Resource Allocation: Organizations can make better use of
their resources by analyzing data to allocate budgets, optimize
supply chains, and manage inventory effectively.

Continuous Improvement: Through ongoing data analysis,

organizations can iteratively improve their processes, products, and
services, leading to long-term growth and sustainability.

Overall, data science is integral to understanding and leveraging the

wealth of information available in today's world. Its applications span
various sectors, offering insights that drive innovation, improve
decision-making, and create a better understanding of complex
systems.
 Application of Data Science
Data science has a wide range of applications across various industries and fields. Here are
some notable examples of how data science is applied:
Business and Marketing:
Customer Segmentation: Data science helps businesses identify distinct customer groups
based on behavior, preferences, and demographics for targeted marketing.
Market Analysis: Analyzing market trends, competitor data, and consumer sentiment to
inform business strategies and product development.
Recommender Systems: Creating personalized recommendations for products, services,
or content based on user behavior and preferences.
Pricing Optimization: Utilizing data to set optimal prices for products or services based on
market demand and competition.
Healthcare and Medicine:
Disease Prediction and Diagnosis: Applying data science to medical records, genetic data,
and other health data for early disease detection and accurate diagnosis.
Drug Discovery: Using data analysis and machine learning to identify potential drug
candidates and predict their effectiveness.
Medical Imaging Analysis: Enhancing medical image interpretation through image
recognition and analysis algorithms for improved diagnosis.
Health Monitoring: Developing wearable devices and apps that collect and analyze health
data for monitoring and early intervention.
Manufacturing and Supply Chain:
Demand Forecasting: Predicting future demand for products to optimize
inventory management and production planning.
Quality Control: Analyzing sensor data and production metrics to identify
defects and improve product quality.
Supply Chain Optimization: Optimizing the flow of goods, reducing costs, and
improving efficiency through data-driven insights.
Preventive Maintenance: Using data to predict equipment failures and schedule
maintenance to minimize downtime.
Energy and Utilities:
Energy Consumption Analysis: Monitoring and analyzing energy consumption
patterns to identify opportunities for energy efficiency.
Grid Management: Optimizing energy distribution and load balancing in smart
grids using real-time data.
Predictive Maintenance: Anticipating equipment failures in power plants and
infrastructure to minimize disruptions.
Finance and Banking:
Fraud Detection: Identifying unusual patterns in financial transactions to detect and prevent
fraudulent activities.
Risk Assessment: Evaluating credit risk, investment opportunities, and market trends to make
informed financial decisions.
Algorithmic Trading: Using data science to develop trading algorithms that leverage historical
data and market signals.
Personalized Financial Advice: Providing customized financial recommendations and
investment strategies based on individual goals and risk tolerance.
Transportation and Logistics:
Route Optimization: Utilizing data to find the most efficient routes for delivery vehicles, reducing fuel
consumption and delivery times.
Traffic Management: Analyzing traffic patterns to improve urban planning and alleviate congestion.
Ride-Sharing and Mobility Services: Matching riders with drivers, optimizing routes, and managing
vehicle fleets.
Social Sciences and Public Policy:
Social Media Analysis: Extracting insights from social media data to understand public sentiment,
trends, and opinions.
Crime Analysis: Using data to predict crime hotspots and allocate law enforcement resources
effectively.
Education Analytics: Analyzing student performance data to improve teaching methods and
educational outcomes.
These are just a few examples of how data science is applied in various domains. The
field continues to evolve, and new applications are constantly being explored as data
becomes more integral to decision-making and problem-solving in virtually every industry.
 Information Commons
The term "Information Commons" typically refers to a physical or
virtual space where people have access to various forms of
information, resources, and technologies for learning, research,
collaboration, and knowledge sharing. It's often associated with
libraries, academic institutions, and other public spaces that provide
access to a wide range of information and tools to support education
and research.
In case of medical field, Information Commons is a platform for data-
driven research that provides access to shared multi-modal patient-
and population-level biomedical data and a wide range of exploration
and analysis tools via secure computational environments.
Information Commons can be found in various settings, including
universities, public libraries, research institutions, and community
centers. The concept highlights the importance of providing equitable
access to information and technology, fostering collaborative learning
environments, and supporting lifelong learning and research. As
technology continues to advance, the role of Information Commons in
facilitating information sharing and knowledge creation remains
significant.
Data Science Project Life Cycle: OSEMN Framework
The OSEMN framework is a data science methodology used for the end-
to-end process of working with data to extract valuable insights and make
informed decisions. Each letter in the acronym "OSEMN" stands for a key
step in the data science workflow:
1. Obtain: In this phase, you gather the data needed for your analysis.
This could involve data collection from various sources, such as
databases, APIs, files, or even web scraping.
2. Scrub (or Scrubbing): Here, the collected data is cleaned,
preprocessed, and transformed. This step involves handling missing
values, removing duplicates, and dealing with outliers. The goal is to
ensure the data is accurate and ready for analysis.
3. Explore: In the exploration phase, you perform exploratory data
analysis (EDA) to understand the data's characteristics, relationships,
and patterns. Visualization and summary statistics are often used to gain
insights into the data's structure and potential trends.
OSEMN framework

Model: In this step, you build and train machine learning models or statistical models
depending on the analysis goals. This involves selecting appropriate algorithms, feature
engineering, model training, and evaluation.

Interpret (or Interpretation): After obtaining model results, you interpret

them to draw meaningful insights and conclusions. This step helps answer the original
questions or objectives of the analysis. It may also involve refining models, exploring
feature importance, and understanding how well the models perform.

Communicate (or Communication): The final step involves communicating

the findings and insights to stakeholders. This could be in the form of reports,
dashboards, presentations, or any other means that effectively convey the results of the
analysis. Clear communication is crucial for making informed decisions based on the
analysis.

The OSEMN framework provides a structured approach to tackling data science projects,
ensuring that important steps like data preparation and model evaluation are not
overlooked. It helps data scientists and analysts manage the complexities of working with
data and extracting meaningful information from it.
Difference between data analysis and data analytics.
Data analysis and data analytics are related terms that often overlap,
but they have distinct differences in their scope, focus, and
methodologies.
Let's explore the key differences between the two:
Data Analysis: Data analysis involves the process of inspecting,
cleaning, transforming, and organizing raw data to extract meaningful
insights, discover patterns, and draw conclusions.
It is a fundamental step in understanding and interpreting data. Data
analysis is typically more focused on examining historical data and
identifying trends, correlations, and relationships within the data set.
It often includes descriptive statistics and visualization techniques to
communicate findings effectively.
Data analysis is important for making informed decisions, but it might
not always involve advanced statistical or predictive techniques.
Data Analytics: Data analytics encompasses a broader range of activities
that go beyond simple data analysis.
It involves the application of various techniques, tools, and algorithms to
explore data, uncover hidden patterns, and derive actionable insights.
Data analytics includes not only descriptive analysis but also diagnostic,
predictive, and prescriptive analysis.
The goal of data analytics is to answer specific business questions, make
predictions about future trends, and guide decision-making.
It often involves more sophisticated statistical modeling, machine learning,
and data mining techniques to provide deeper insights and predictive
capabilities.
In summary, while both data analysis and data analytics involve working with
data to gain insights, data analysis is a subset of data analytics. Data analysis
focuses on understanding the data's past and present characteristics, while
data analytics encompasses a wider range of activities, including advanced
techniques and predictive modeling, to inform future decisions and strategies.
Python Programming