Unit 4 Machine Learning
🟩 1. Sparse Modeling and Estimation
🔹 Definition:
Sparse modeling refers to techniques that assume the underlying data or model can be
represented using only a small number of non-zero parameters (i.e., sparsity). This helps
in reducing complexity, avoiding overfitting, and improving interpretability.
🔹 Algorithms/Procedures:
✅ Lasso Regression (L1 Regularization):
Adds an L1 penalty to the regression coefficients:
Encourages sparse coefficient vectors by shrinking some coefficients to zero.
✅ Basis Pursuit / L1 Minimization:
Used in compressed sensing.
Solves min∥β∥1 s.t. y=Aβ
✅ Matching Pursuit / Orthogonal Matching Pursuit (OMP):
Greedy algorithms that iteratively select features contributing most to the signal.
✅ Compressed Sensing:
Recovers sparse signals from incomplete linear measurements using optimization
techniques.
🔹 Applications & Examples:
Signal processing : Denoising, image reconstruction
Genomics : Identifying key genes influencing disease
Finance : Portfolio selection with few assets
Example: Recovering images from limited sensor data in MRI scans.
🟩 2. Modeling Sequence/Time-Series Data
🔹 Definition:
Sequence modeling involves analyzing ordered data where temporal or sequential
dependencies are important. Time-series modeling focuses specifically on sequences
indexed by time.
🔹 Algorithms/Procedures:
✅ Autoregressive Models (AR, ARIMA):
Predict future values based on past values.
ARIMA also includes differencing to make time series stationary.
✅ Hidden Markov Models (HMMs):
Probabilistic models for sequences with hidden states.
Used in speech recognition and part-of-speech tagging.
✅ Recurrent Neural Networks (RNNs), LSTM, GRU:
RNNs process sequences step-by-step, maintaining memory.
LSTMs and GRUs overcome vanishing gradient issues in standard RNNs.
✅ Transformers / Self-Attention Mechanism:
Use attention to model long-range dependencies without recurrence.
Widely used in NLP and sequence-to-sequence tasks.
✅ Dynamic Time Warping (DTW):
Measures similarity between two time series that may vary in speed.
🔹 Applications & Examples:
Speech recognition
Stock price prediction
Weather forecasting
Activity recognition from wearable sensors
Example: Forecasting sales using historical data with seasonal trends.
🟩 3. Deep Learning and Feature Representation Learning
🔹 Definition:
Deep learning uses multi-layered neural networks to automatically learn hierarchical
representations of data. Feature representation learning refers to the process of
transforming raw input into more meaningful, high-level features.
🔹 Algorithms/Procedures:
✅ Artificial Neural Networks (ANNs):
Multi-layer perceptrons that map inputs to outputs through weighted connections
and activation functions.
✅ Convolutional Neural Networks (CNNs):
Use convolutional layers to detect local patterns (e.g., edges, textures).
Ideal for images and grid-like data.
✅ Autoencoders:
Unsupervised models that compress input into latent code and reconstruct it.
Used for dimensionality reduction and feature extraction.
✅ Variational Autoencoders (VAEs):
Probabilistic autoencoders that enforce smooth latent space distributions.
✅ Generative Adversarial Networks (GANs):
Two networks compete: generator creates fake data; discriminator tries to
distinguish real vs fake.
✅ Pretrained Models and Transfer Learning:
Use models like ResNet, BERT, etc., trained on large datasets, and fine-tune them
for specific tasks.
🔹 Applications & Examples:
Image classification (e.g., ImageNet)
Natural Language Processing (e.g., translation, sentiment analysis)
Anomaly detection
Medical imaging and diagnostics
Example: Using BERT for question answering or GANs for generating synthetic
faces.
✅ Summary Table
TOPIC TYPE KEY IDEA ALGORITHM(S) APPLICATION
Sparse Signal Learn models with Lasso, OMP, Genomics, Image
Modeling Processing few non-zero Compressed Reconstruction
parameters Sensing
Sequence/ Temporal Capture order and HMM, RNN, Speech Recognition,
Time-Series Modeling dependencies in LSTM, Transformer Stock Prediction
Modeling sequences
Deep Learning Representatio Automatically learn CNN, Autoencoder, Image
n Learning hierarchical features GAN, VAE, Classification, NLP,
Transfer Learning Anomaly Detection
