Unit II: Applications of machine learning in Data science, role of ML in DS, Python tools

like sklearn, modelling process for feature engineering, model selection, validation and
prediction, types of ML, semi-supervised learning
Handling large data: problems and general techniques for handling large data, programming
tips for dealing large data, case studies on DS projects for predicting malicious URLs, for
building recommender systems

Applications of machine learning in Data science:

1. Predictive Analytics
ML models predict future outcomes based on historical data.
Applications:
Sales Forecasting: Predicting future sales based on past trends.
Stock Market Predictions: Analyzing stock price movements using historical financial data.
Weather Forecasting: Predicting temperature, rainfall, and extreme weather conditions.
2. Natural Language Processing (NLP)
ML enables computers to understand, interpret, and generate human language.
Applications:
Sentiment Analysis: Determining customer sentiment in reviews or social media posts.
Chatbots and Virtual Assistants: Automating customer support with tools like ChatGPT or
Alexa.
Text Summarization: Creating concise summaries of large documents or articles.
Language Translation: Applications like Google Translate.
3. Image and Video Analysis
ML can process and interpret visual data.
Applications:
Facial Recognition: Used in security systems, unlocking devices, and surveillance.
Medical Imaging: Detecting diseases like cancer from X-rays or MRIs.
Object Detection: Self-driving cars detecting pedestrians, road signs, and obstacles.
Video Content Moderation: Filtering inappropriate content on platforms like YouTube.
4. Anomaly Detection

QISCET[AIML] 1
ML identifies unusual patterns that deviate from the norm.
Applications:
Fraud Detection: Identifying suspicious transactions in banking and e-commerce.
Network Security: Detecting malware and potential security breaches.
Industrial Monitoring: Spotting equipment faults or failures before they occur.
5. Recommender Systems
ML algorithms suggest products, services, or content based on user preferences.
Applications:
E-commerce: Personalized product recommendations (e.g., Amazon).
Streaming Platforms: Suggesting movies or songs based on user history (e.g., Netflix,
Spotify).
Online Learning: Recommending courses or resources based on past activity.
6. Clustering and Segmentation
ML groups data into clusters or segments based on similarity.
Applications:
Customer Segmentation: Grouping customers for targeted marketing campaigns.
Document Clustering: Organizing large collections of documents by topics.
Image Segmentation: Segmenting images for object detection or medical imaging.
7. Time Series Analysis
ML models analyze data over time to identify trends and patterns.
Applications:
Demand Forecasting: Predicting inventory needs in supply chain management.
Energy Consumption Analysis: Forecasting power usage in smart grids.
Healthcare Monitoring: Analyzing patient vitals over time to detect anomalies.
8. Automation and Robotics
ML drives automation and decision-making in physical and digital environments.
Applications:
Self-Driving Cars: Autonomous navigation and decision-making.
Manufacturing Robots: Automating repetitive tasks on production lines.
Process Automation: Streamlining workflows in industries like insurance and finance.

QISCET[AIML] 2
9. Healthcare
ML improves diagnostics, treatment planning, and personalized medicine.
Applications:
Disease Prediction: Identifying high-risk patients for specific diseases.
Drug Discovery: Accelerating the development of new medications.
Patient Monitoring: Real-time analysis of health data from wearable devices.
10. Gaming and Entertainmen
ML enhances user experiences and gameplay mechanics.
Applications:
Game AI: Creating intelligent and adaptive non-player characters (NPCs).
Content Generation: Generating new levels or challenges in games.
Interactive Storytelling: Personalizing narratives based on user choices.
11. Personalization and User Profiling
ML tailors experiences based on individual preferences.
Applications:
Marketing Campaigns: Crafting targeted ads and offers for users.
Social Media Feeds: Optimizing content delivery based on user interactions.
Personalized Education: Adapting learning materials to suit individual student needs.
12. Supply Chain and Logistics
ML optimizes routes, inventory, and operations.
Applications:
Route Optimization: Reducing delivery times in logistics.
Inventory Management: Predicting stock levels to prevent overstocking or shortages.
Demand Forecasting: Anticipating customer demand for efficient supply chain
management.
13. Environmental Monitoring
ML helps address global challenges related to the environment.
Applications:
Climate Modeling: Predicting climate changes and their impact.
Wildlife Conservation: Monitoring animal populations and habitats.

QISCET[AIML] 3
Pollution Detection: Analyzing air or water quality data for environmental protection.
14. Finance and Insurance
ML supports risk assessment, fraud detection, and decision-making.
Applications:
Credit Scoring: Evaluating loan eligibility based on user profiles.
Insurance Underwriting: Assessing risk and setting premiums.
Stock Trading: Automated trading systems using market data.
15. Advanced Research
ML accelerates breakthroughs in scientific and technological fields.
Applications:
Physics Simulations: Modeling complex systems like particle collisions.
Genomics: Analyzing DNA sequences for insights into diseases.
Astronomy: Detecting exoplanets or identifying celestial patterns.
Role of Machine Learning in Data Science:

What is Machine Learning?

Machine Learning, often abbreviated as ML, is a subset of artificial intelligence (AI) that
focuses on the development of computer algorithms that improve automatically through
experience and by the use of data. In simpler terms, machine learning enables computers to
learn from data and make decisions or predictions without being explicitly programmed to do
so.Machine learning (ML) plays a transformative role in data science by providing powerful
tools to automatically learn patterns from data and make predictions or decisions without
being explicitly programmed. Here's a breakdown of its role:

1. Data Analysis and Pattern Discovery

Machine learning algorithms can analyze vast datasets to discover hidden patterns and
relationships.
These insights can be used to make data-driven business decisions.
Example: Clustering customer data to find distinct customer segments.

2. Predictive Modeling
ML enables data scientists to build predictive models that forecast future outcomes.
These models are trained on historical data and applied to unseen data.

QISCET[AIML] 4
Example: Predicting customer churn rates or stock market trends.
3. Automating Data Processing
ML models automate complex and repetitive data tasks like cleaning, transformation, and
anomaly detection.
Example: Automatic detection and correction of missing values in datasets.
4. Real-time Analytics
ML models can process and analyze streaming data in real-time.
This is valuable for applications like fraud detection and network monitoring.
Example: Flagging fraudulent credit card transactions as they occur.
5. Feature Selection and Engineering
ML helps identify important features from a dataset that contribute the most to the prediction
or decision-making process.
It can also generate new features to enhance model accuracy.
Example: Identifying critical health indicators from patient data for disease prediction.
6. Optimization of Business Processes
Data scientists use ML models to optimize various business processes, such as supply chain
logistics and marketing strategies.
Example: Route optimization for delivery services like Amazon or FedEx.
7. Personalization and Recommendation Systems
ML enables personalized experiences by analyzing user behavior and preferences.
Example: Recommending products on e-commerce platforms or content on Netflix.
8. Enhanced Decision-Making
ML models support decision-making by providing data-driven insights and automating
routine decisions.
Example: AI-driven analytics dashboards for financial decision-making.
9. Data Visualization and Communication
Advanced ML models, combined with visualization techniques, help present complex data in
understandable ways.
Example: Visualizing prediction models' outcomes for stakeholders.
10. Handling Unstructured Data
ML models can process and analyze unstructured data like text, images, and videos,
expanding the scope of data science beyond structured databases.

QISCET[AIML] 5
Python tools like sklearn:
Scikit-learn has emerged as a powerful and user-friendly Python library. Its simplicity
and versatility make it an excellent choice for both beginners and experienced data
scientists looking to build and implement machine learning models.
1. Scikit-Learn (sklearn)
 Purpose: Machine learning library for classification, regression, and clustering.
 Features:
o Provides algorithms like linear regression, decision trees, and support vector
machines.
o Tools for model selection, evaluation, and data preprocessing.
 Example Use: Building predictive models and tuning hyperparameters.
python
CopyEdit
from sklearn.ensemble import RandomForestClassifier
model = RandomForestClassifier()
model.fit(X_train, y_train)
predictions = model.predict(X_test)

2. Pandas
 Purpose: Data manipulation and analysis.
 Features:
o Handling structured data with DataFrames.
o Data cleaning, filtering, and aggregation.
python
CopyEdit
import pandas as pd
data = pd.read_csv('data.csv')
print(data.head())
3. NumPy
 Purpose: Numerical computing and array operations.

QISCET[AIML] 6
  Features:
o High-performance operations on arrays and matrices.
o Useful for linear algebra and statistical operations.
python
CopyEdit
import numpy as np
arr = np.array([1, 2, 3])
print(arr.mean())
4. Matplotlib
 Purpose: Data visualization library.
 Features:
o Create static, interactive, and publication-quality plots.
python
CopyEdit
import matplotlib.pyplot as plt
plt.plot([1, 2, 3], [4, 5, 6])
plt.show()
5. Seaborn
 Purpose: Statistical data visualization.
 Features:
o Built on Matplotlib; makes complex visualizations easy.
python
CopyEdit
import seaborn as sns
sns.histplot(data['column_name'])
6. TensorFlow & Keras
 Purpose: Deep learning frameworks.
 Features:
o Build neural networks for complex data science problems like image
classification.

QISCET[AIML] 7
python
CopyEdit
import tensorflow as tf
from tensorflow import keras
model = keras.Sequential([keras.layers.Dense(units=1)])
7. PyTorch
 Purpose: Deep learning framework.
 Features:
o Dynamic computation graph and easier debugging for neural networks.
python
CopyEdit
import torch
x = torch.tensor(2.0)
y = torch.tensor(3.0)
print(x + y)
8. Statsmodels
 Purpose: Statistical modeling and hypothesis testing.
 Features:
o Useful for regression models and time-series analysis.
python
CopyEdit
import statsmodels.api as sm
model = sm.OLS(y, X).fit()
print(model.summary())
9. SciPy
 Purpose: Scientific and technical computing.
 Features:
o Provides functions for optimization, integration, and signal processing.
python
CopyEdit

QISCET[AIML] 8
from scipy import stats
print(stats.ttest_ind(data1, data2))
10. XGBoost
 Purpose: Gradient boosting algorithm for structured data.
 Features:
o High performance for classification and regression tasks.
python
CopyEdit
import xgboost as xgb
model = xgb.XGBClassifier()
model.fit(X_train, y_train)
11. LightGBM
 Purpose: Gradient boosting framework optimized for speed.
 Features:
o Handles large datasets efficiently.
python
CopyEdit
import lightgbm as lgb
model = lgb.LGBMClassifier()
model.fit(X_train, y_train)
12. Plotly
 Purpose: Interactive data visualization.
 Features:
o Ideal for web-based dashboards.
python
CopyEdit
import plotly.express as px
fig = px.scatter(data, x="x_value", y="y_value")
fig.show()

QISCET[AIML] 9
Modeling Process for Feature Engineering in Data Science:
Feature engineering is a crucial step in building robust machine learning models. It involves
creating new features or transforming existing ones to improve model accuracy and
predictive power. Below is a structured modeling process for feature engineering in data
science:
1. Understand the Problem
 Clearly define the business problem and target variable.
 Understand the context and key factors influencing the outcome.
Example: In a fraud detection problem, time of transaction and transaction amount might be
crucial features.
2. Data Collection and Exploration
 Gather relevant data from multiple sources (databases, APIs, CSV files).
 Perform Exploratory Data Analysis (EDA) to:
o Understand distributions
o Detect outliers
o Identify correlations between features
python
CopyEdit
import pandas as pd
import seaborn as sns
data = pd.read_csv('transactions.csv')
sns.pairplot(data)
3. Data Cleaning
 Handle missing values using imputation or deletion strategies.
 Deal with outliers by capping, transformation, or removal.
 Standardize or normalize data when necessary.
python
CopyEdit
data.fillna(data.median(), inplace=True)
4. Feature Selection
Identify the most relevant features using the following methods:

QISCET[AIML] 10
  Statistical Tests: ANOVA, Chi-Square Test
 Correlation Analysis: Pearson or Spearman correlation
 Model-based Selection: Feature importances from decision trees or regularized
regression models.
python
CopyEdit
from sklearn.ensemble import RandomForestClassifier
model = RandomForestClassifier()
model.fit(X_train, y_train)
print(model.feature_importances_)
5. Feature Engineering Techniques
a. Transformation Techniques
 Logarithm Transformation: Handle skewed data.
 Standardization & Normalization: Scale features to a common range.
python
CopyEdit
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
b. Interaction Features
 Create new features by combining existing ones (e.g., product or ratio of two
features).
python
CopyEdit
data['transaction_per_day'] = data['total_transactions'] / data['days_active']
c. Encoding Categorical Variables
 One-hot encoding or target encoding for categorical features.
python
CopyEdit
from sklearn.preprocessing import OneHotEncoder

QISCET[AIML] 11
encoder = OneHotEncoder()
X_encoded = encoder.fit_transform(X_categorical)
d. Time-based Features
 Extract day, month, or season from timestamps.
python
CopyEdit
data['hour'] = pd.to_datetime(data['timestamp']).dt.hour
e. Text Features (for NLP)
 Tokenization, TF-IDF, or word embeddings.
f. Dimensionality Reduction
 PCA, t-SNE, or feature elimination for high-dimensional datasets.
python
CopyEdit
from sklearn.decomposition import PCA
pca = PCA(n_components=2)
X_reduced = pca.fit_transform(X)
6. Model Building and Evaluation
 Use multiple algorithms to test the effectiveness of engineered features.
 Evaluate using cross-validation and metrics like RMSE, accuracy, or F1-score.
7. Feature Tuning and Optimization
 Fine-tune feature transformations or add/remove features iteratively.
 Use hyperparameter optimization techniques (Grid Search or Random Search).
8. Model Validation
 Test the model on unseen data to validate feature effectiveness.
9. Deploy and Monitor
 Deploy the model and continuously monitor the impact of features.
 Re-engineer features based on changing data trends.
Model Selection, Validation, and Prediction in Data Science:
A structured approach to selecting, validating, and making predictions using machine
learning models ensures better performance and generalization to unseen data.

QISCET[AIML] 12
1. Model Selection
Model selection involves choosing the best algorithm for your data and problem.
Steps for Model Selection:
a. Define Problem Type:
 Regression: Predicting continuous values (e.g., house prices)
 Classification: Predicting categories (e.g., spam detection)
 Clustering: Grouping data points (e.g., customer segmentation)
b. Choose Candidate Models:
Consider models based on problem complexity and data size.
Examples:

Problem Algorithms

Classification Logistic Regression, Random Forest, SVM

Regression Linear Regression, XGBoost, Neural Networks

Clustering K-Means, DBSCAN, Hierarchical Clustering

c. Hyperparameter Optimization:
Use grid search or random search to tune model hyperparameters.
python
CopyEdit
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV
params = {'n_estimators': [100, 200], 'max_depth': [5, 10]}
model = RandomForestClassifier()
grid_search = GridSearchCV(model, param_grid=params, cv=3)
grid_search.fit(X_train, y_train)
print("Best Parameters:", grid_search.best_params_)
2. Model Validation
Model validation assesses how well the model generalizes to unseen data.
a. Train-Test Split
Split the dataset into training and testing sets.
python
CopyEdit

QISCET[AIML] 13
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
b. Cross-Validation
Split the data into K folds and train/test the model K times.
python
CopyEdit
from sklearn.model_selection import cross_val_score
scores = cross_val_score(model, X_train, y_train, cv=5)
print("Cross-validation Scores:", scores)
c. Performance Metrics
Evaluate the model using appropriate metrics:
 Regression: RMSE, MAE, R²
 Classification: Accuracy, Precision, Recall, F1-score, ROC-AUC
python
CopyEdit
from sklearn.metrics import accuracy_score, f1_score
predictions = model.predict(X_test)
print("Accuracy:", accuracy_score(y_test, predictions))
print("F1 Score:", f1_score(y_test, predictions, average='weighted'))
3. Model Prediction
Once validated, the model is used to predict outcomes on new or test data.
python
CopyEdit
final_predictions = model.predict(new_data)
print("Predicted Values:", final_predictions)
Saving and Loading the Model
Save the trained model for reuse.
python
CopyEdit
import joblib

QISCET[AIML] 14
# Save model
joblib.dump(model, 'model.pkl')
# Load model
loaded_model = joblib.load('model.pkl')
print(loaded_model.predict(new_data))
Types of Machine Learning (ML)
Machine learning can be categorized into different types based on how the models learn from
data. Below are the major types:
1. Supervised Learning
Definition:
The model is trained on labeled data, where each input has a corresponding output (target).
The goal is to learn a mapping from inputs to outputs.
Common Algorithms:
 Linear Regression
 Decision Trees
 Support Vector Machines (SVM)
 Neural Networks
Use Cases:
 Spam detection in emails (classification)
 Predicting house prices (regression)
2. Unsupervised Learning
Definition:
The model is trained on unlabeled data and must discover hidden patterns or relationships in
the data.
Common Algorithms:
 K-Means Clustering
 Principal Component Analysis (PCA)
 Hierarchical Clustering
 Autoencoders
Use Cases:
 Customer segmentation

QISCET[AIML] 15
  Anomaly detection in manufacturing
3. Semi-Supervised Learning
Definition:
A combination of supervised and unsupervised learning, where the model is trained on a
small amount of labeled data and a large amount of unlabeled data.
Common Algorithms:
 Self-training models
 Graph-based models
Use Cases:
 Medical image classification (where labeled data is expensive)
 Speech analysis
4. Reinforcement Learning
Definition:
The model learns by interacting with an environment and receiving feedback in the form of
rewards or penalties. The objective is to learn an optimal strategy.
Common Algorithms:
 Q-Learning
 Deep Q-Networks (DQN)
 Policy Gradient Methods
Use Cases:
 Robotics (learning to navigate)
 Game AI (like AlphaGo)
 Self-driving cars
5. Self-Supervised Learning
A form of unsupervised learning where the model generates its own labels by understanding
patterns in the input data.
Common Algorithms:
 Contrastive Learning (SimCLR)
 Transformers (used in models like BERT and GPT)
Use Cases:
 Natural Language Processing (NLP)

QISCET[AIML] 16
  Image recognition in computer vision
Semi-Supervised Learning (SSL)
Definition:
Semi-supervised learning is a type of machine learning that combines a small amount of
labeled data with a large amount of unlabeled data. The idea is to use the labeled data to
guide the learning process, making better use of the vast quantities of unlabeled data, which
are often cheaper and easier to obtain.
Why Use Semi-Supervised Learning?
 Cost-Effective: Labeling large datasets can be time-consuming and expensive. SSL
leverages a small amount of labeled data.
 Improved Performance: Often outperforms purely supervised learning when labeled
data is limited.
 Effective for Complex Tasks: Useful for applications like image and speech
recognition, where labeled data is scarce.
How Semi-Supervised Learning Works:
1. Initial Model Training: Train a base model on the small labeled dataset.
2. Label Propagation: Use the base model to predict labels for the unlabeled data.
3. Re-Training: Incorporate confident predictions from the unlabeled data to improve
the model iteratively.
Techniques in Semi-Supervised Learning:
1. Self-Training
 The model is trained on labeled data, and then predictions on the unlabeled data are
added back as pseudo-labels for retraining.
2. Graph-Based Methods
 Represent data points as nodes in a graph. Use relationships between points to
propagate labels.
3. Generative Models
 Train models to learn the joint distribution of input features and labels (e.g.,
Variational Autoencoders).
4. Consistency Regularization
 Encourage the model to produce consistent predictions for perturbed versions of the
same input.
Real-World Applications:

QISCET[AIML] 17
  Speech Recognition: Labeling speech data is costly, so SSL improves automatic
speech transcription.
 Medical Imaging: Limited labeled medical images can be supplemented with
unlabeled scans for better diagnostics.
 Fraud Detection: Few fraud cases labeled; SSL helps in classifying normal and
suspicious transactions.
 Text Classification: Label propagation in NLP tasks like sentiment analysis.
Example of Self-Training with Semi-Supervised Learning
Here’s a simple approach using sklearn for a semi-supervised learning task:
python
CopyEdit
from sklearn.semi_supervised import SelfTrainingClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
# Create a dataset
X, y = make_classification(n_samples=200, n_features=5, random_state=42)
y[:150] = -1 # Assume 150 samples are unlabeled
# Split labeled and unlabeled data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
# Create a Self-Training model using Random Forest
base_model = RandomForestClassifier()
ssl_model = SelfTrainingClassifier(base_model)
# Train the model
ssl_model.fit(X_train, y_train)
# Evaluate the model
print("Accuracy on Test Data:", ssl_model.score(X_test, y_test))
Advantages:
 Reduces dependency on labeled data.
 Improves learning when labeled data is sparse.
 Effective for complex tasks where unlabeled data is abundant.

QISCET[AIML] 18
Challenges:
 Label noise from pseudo-labels can degrade model performance.
 Model selection can be complex for SSL tasks.
Handling large data: problems and general techniques for handling large data
Working with large datasets presents unique challenges that require specific
techniques and tools to manage efficiently. Below are common problems and practical
techniques for handling large data.
Common Problems with Large Data
1. Memory Limitations:
o Loading entire datasets into memory may cause the system to crash.
2. Slow Processing Speed:
o Complex computations can take a long time, especially for large datasets.
3. Data I/O Bottlenecks:
o Reading and writing large files can slow down the data pipeline.
4. Limited Storage:
o Storing massive datasets locally becomes challenging.
5. Scalability Issues:
o Models and algorithms designed for small datasets may not scale well to big
data.
6. Data Quality Issues:
o Large datasets often have missing, inconsistent, or noisy data that needs
cleaning.
General Techniques for Handling Large Data
1. Data Sampling
 Work with a smaller subset of data that represents the whole dataset.
 Useful for prototyping models quickly.
python
CopyEdit
sample_data = data.sample(frac=0.1, random_state=42)
2. Data Chunking (Batch Processing)
 Process data in smaller chunks instead of loading it all at once.

QISCET[AIML] 19
 python
CopyEdit
import pandas as pd
chunk_size = 10000
for chunk in pd.read_csv('large_file.csv', chunksize=chunk_size):
process_chunk(chunk)
3. Distributed Computing
 Use distributed frameworks to handle large data by dividing tasks across multiple
nodes.
Tools:
 Apache Spark (via PySpark)
 Dask (for parallel computing)
python
CopyEdit
from dask import dataframe as dd
df = dd.read_csv('large_dataset.csv')
result = df.groupby('column_name').sum().compute()
4. Compression
 Store data in compressed formats (e.g., .parquet, .gzip) to reduce file size.
python
CopyEdit
data.to_parquet('data.parquet', compression='gzip')
5. Cloud Storage and Computing
 Offload data storage and processing to cloud services (AWS, Google Cloud, Azure).
6. Database Systems for Large Data
 Use databases instead of flat files for efficient data querying and storage.
Examples:
 SQL Databases (PostgreSQL, MySQL)
 NoSQL Databases (MongoDB, Cassandra)
7. Efficient Data Structures

QISCET[AIML] 20
  Use memory-efficient libraries such as NumPy and Pandas optimized types
(category for strings).
python
CopyEdit
data['column'] = data['column'].astype('category')
8. Parallel Processing
 Leverage multi-core CPUs for parallel processing using libraries like Joblib or
Multiprocessing.
python
CopyEdit
from joblib import Parallel, delayed
results = Parallel(n_jobs=4)(delayed(function)(item) for item in data_list)
9. Feature Engineering with Sparse Matrices
 Use sparse matrix representations to save memory for sparse datasets.
python
CopyEdit
from scipy.sparse import csr_matrix
sparse_matrix = csr_matrix(large_data)
10. Model Optimization for Big Data
 Use incremental learning models that can process data in batches.
python
CopyEdit
from sklearn.linear_model import SGDClassifier
model = SGDClassifier()
for X_batch, y_batch in data_batches:
model.partial_fit(X_batch, y_batch, classes=[0, 1])

QISCET[AIML] 21
 Summary of Tools and Techniques

Problem Solution Tools/Techniques

Memory
Chunking, Compression Dask, Parquet
Limitations

Slow Processing Distributed Computing PySpark, Dask

Compression, Cloud
Storage Issues GCP, AWS, Azure
Storage

Data I/O PostgreSQL,

Database Usage
Bottlenecks MongoDB

Scalability Parallel Processing Joblib, Multiprocessing

Programming Tips for Handling Large Data :

Here are practical programming strategies for efficiently managing large datasets in data
science projects:
1. Use Efficient Data Structures
 Use NumPy Arrays: NumPy arrays are more memory-efficient than lists.
 Use Pandas Optimized Data Types: Convert data types to category, float32, or int32
to save memory.
python
CopyEdit
import pandas as pd
df['category_col'] = df['category_col'].astype('category')
df['numeric_col'] = pd.to_numeric(df['numeric_col'], downcast='float')
2. Load Data in Chunks (Chunking)
 Read large datasets in manageable pieces to avoid memory overload.
python
CopyEdit
chunk_size = 10000
for chunk in pd.read_csv('large_file.csv', chunksize=chunk_size):
process_chunk(chunk)
3. Use Data Compression

QISCET[AIML] 22
  Store and read compressed data formats such as Parquet, HDF5, or GZIP to save
space.
python
CopyEdit
df.to_parquet('data.parquet', compression='snappy')
4. Vectorize Operations
 Replace loops with vectorized operations using NumPy or Pandas.
python
CopyEdit
# Slow loop operation
df['new_col'] = [x**2 for x in df['existing_col']]
# Fast vectorized operation
df['new_col'] = df['existing_col'] ** 2
5. Parallel Processing
 Use multiple cores for faster computation with Joblib or Multiprocessing.
python
CopyEdit
from joblib import Parallel, delayed
def square(x):
return x ** 2
results = Parallel(n_jobs=4)(delayed(square)(i) for i in range(10000))
6. Distributed Computing with Dask
 Dask allows scalable data processing by splitting operations across multiple
machines.
python
CopyEdit
import dask.dataframe as dd
df = dd.read_csv('large_dataset.csv')
df.groupby('column_name').sum().compute()
7. Use Generators for Data Processing
 Generators process data lazily, reducing memory usage.

QISCET[AIML] 23
python
CopyEdit
def data_generator(file_path):
with open(file_path) as file:
for line in file:
yield line.strip()
8. Incremental Learning Models
 Use models that support partial fitting, like SGDClassifier from scikit-learn.
python
CopyEdit
from sklearn.linear_model import SGDClassifier
model = SGDClassifier()
for X_batch, y_batch in data_batches:
model.partial_fit(X_batch, y_batch, classes=[0, 1])
9. Memory Mapping with NumPy
 Use memory-mapped arrays for handling massive datasets.
python
CopyEdit
import numpy as np
data = np.memmap('large_data.dat', dtype='float32', mode='r', shape=(10000, 10000))
10. Optimize Data Storage and Querying
 Use databases like SQLite, PostgreSQL, or NoSQL databases (e.g., MongoDB)
instead of flat files for faster queries.
python
CopyEdit
import sqlite3
conn = sqlite3.connect('data.db')
df.to_sql('table_name', conn, if_exists='replace')
Summary Table

QISCET[AIML] 24
Problem Solution Tool/Technique

Memory Constraints Data Chunking Pandas chunksize

Slow Processing Vectorization NumPy, Pandas

Scalability Distributed Computing Dask, PySpark

Storage Issues Compression Parquet, GZIP

Model Efficiency Incremental Learning SGDClassifier

Data Query Performance Database Integration SQLite, PostgreSQL

Below are some notable case studies and project ideas for using data science to detect
malicious URLs.
Case Study 1: Malicious URL Detection Using Machine Learning
Objective:
Develop a system that classifies URLs as either malicious or benign using machine learning
models.
Dataset:
 Source: OpenPhish, PhishTank, or Kaggle datasets.
 Data Features:
o URL Length
o Presence of special characters (@, //)
o Domain age
o WHOIS information
Approach:
1. Data Collection:
o Scrape datasets containing both benign and malicious URLs.
2. Feature Engineering:
o Extract features like:
 Number of dots in the URL
 Use of IP addresses
 Presence of HTTP vs. HTTPS
 Domain entropy

QISCET[AIML] 25
 3. Model Selection:
o Tested models:
 Random Forest
 Gradient Boosting (XGBoost)
 Neural Networks
4. Results:
o Achieved 95% accuracy using a Gradient Boosting classifier with
hyperparameter tuning.
Tools Used:
 Scikit-Learn, XGBoost, Python Requests, Pandas, and Matplotlib.
Case Study 2: Real-Time URL Classification using Deep Learning
Objective:
Detect phishing URLs using a character-level convolutional neural network (CNN).
Dataset:
 Dataset from Alexa top websites (for benign URLs) and Spamhaus (for malicious
URLs).
Key Features:
 Character sequences of URLs were directly used as input without manual feature
extraction.
Model Architecture:
 1D Convolutional Neural Network (CNN) trained on character embeddings.
Results:
 Achieved 98% accuracy with a recall of 97% for malicious URLs.
Tools Used:
 TensorFlow, Keras, and Matplotlib.
Case Study 3: Predicting Malicious URLs Using Natural Language Processing (NLP)
Objective:
Predict malicious URLs by analyzing domain names and path strings using NLP techniques.
Dataset:
 Dataset with URLs and associated labels collected from VirusTotal.
Key Steps:

QISCET[AIML] 26
 1. Text Preprocessing:
o Remove special characters and extract meaningful segments from the URLs.
2. Feature Extraction:
o Used TF-IDF and Word2Vec embeddings for URL tokenization.
3. Modeling:
o Logistic Regression and LSTM networks.
4. Results:
o LSTM achieved a 96% F1-score, outperforming Logistic Regression.
Tools Used:
 NLTK, Gensim, TensorFlow, Scikit-Learn.
Case Study 4: Ensemble Learning for Malicious URL Detection
Objective:
Create a robust classifier using ensemble methods for malicious URL detection.
Dataset:
 Mixed data from Kaggle and URLhaus.
Modeling Approach:
 Combined Decision Trees, Random Forest, and Gradient Boosting using
VotingClassifier.
Results:
 Ensemble approach achieved a 97% accuracy, outperforming individual classifiers.
Tools Used:
 Scikit-Learn, Matplotlib, Seaborn.
General Key Insights from Case Studies
1. Feature Engineering Matters:
URL-based features (length, special characters) significantly improve model accuracy.
2. Deep Learning Works Well:
Character-level models (CNN, LSTM) are effective for sequential data like URLs.
3. NLP Techniques Help:
Embeddings and tokenization provide insights beyond simple features.
4. Ensemble Models Perform Best:
Voting and stacking classifiers improve robustness and accuracy.
For building recommender systems:

QISCET[AIML] 27
Case Studies on Building Recommender Systems in Data Science
Below are detailed case studies and project ideas for building recommender systems across
various domains:
Case Study 1: Personalized Movie Recommendation System
Objective:
Develop a system to recommend personalized movies to users based on their viewing history
and ratings.
Dataset:
 Source: MovieLens Dataset
Key Steps:
1. Data Preprocessing:
o Handle missing user ratings
o Normalize movie genres
2. Techniques:
o Collaborative Filtering (Matrix Factorization) using Singular Value
Decomposition (SVD)
o Content-Based Filtering: Using metadata such as genres, actors, and
directors
3. Evaluation Metrics:
o Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE)
4. Results:
o Achieved a 10% improvement in RMSE over baseline collaborative filtering
models by hybridizing content-based and collaborative models.
Tools Used:
 Surprise Library, Pandas, Scikit-Learn, Matplotlib
Case Study 2: E-commerce Product Recommendation System
Objective:
Provide product recommendations based on user purchase history and browsing patterns.
Dataset:
 Source: Amazon Product Review Dataset
Approach:
 Collaborative Filtering: Using user-item interaction data

QISCET[AIML] 28
  Association Rule Mining: Applying Apriori for frequently co-purchased products
 Session-Based Recommendations: Using Recurrent Neural Networks (RNNs)
Results:
 Enhanced click-through rates by 15% after integrating session-based
recommendations.
Tools Used:
 Scikit-Learn, Apriori Algorithm (MLxtend), TensorFlow for RNNs
Case Study 3: Music Recommendation System using Deep Learning
Objective:
Build a deep learning-based music recommender system.
Dataset:
 Source: Million Song Dataset
Approach:
 Audio Feature Extraction: Mel-frequency cepstral coefficients (MFCCs)
 Deep Learning Model: Autoencoders for feature compression and similarity search
Results:
 The system provided recommendations with a 92% match to user preferences.
Tools Used:
 Librosa, Keras, Matplotlib
Case Study 4: News Recommendation System Using NLP
Objective:
Recommend news articles based on user reading history and textual content.
Dataset:
 Source: MIND News Dataset
Approach:
1. Text Processing:
o Extracted TF-IDF features from news content
o Applied Word2Vec for semantic similarity
2. Modeling:
o Used Content-Based Filtering

QISCET[AIML] 29
 o Integrated Collaborative Filtering for user preferences
3. Results:
o Achieved a 20% increase in user engagement by combining NLP-based
recommendations.
Tools Used:
 NLTK, Gensim, Scikit-Learn, LightFM
General Techniques for Building Recommender Systems

Technique Description Use Case

Collaborative Recommends based on similar user Movie and product

Filtering preferences recommendations

Content-Based
Recommends based on item features News and job recommendations
Filtering

Combines collaborative and content-

Hybrid Models E-commerce platforms
based

Music and media

Deep Learning Learns complex user-item interactions
recommendations

Association Rules Finds product co-occurrence patterns Market basket analysis

Key Insights from Case Studies

1. Hybrid Models Work Best:
Combining collaborative and content-based methods often yields superior results.
2. Deep Learning for Complex Scenarios:
Use deep neural networks when traditional models don't capture complex interactions.
3. Evaluation Metrics:
Use precision, recall, NDCG, and RMSE to evaluate the performance of
recommendation models.
4. Data Preprocessing Matters:
Clean and normalize data for optimal model performance
Reference:
https://livebook.manning.com/book/introducing-data-science/chapter-4/9
https://hevodata.com/learn/machine-learning-in-data-science-2/

QISCET[AIML] 30
QISCET[AIML] 31