Scribd
Upload
11 views2 pages

ML Assignment3

This lab assignment focuses on linear regression concepts including basis function expansion, bias, variance, and regularization techniques such as Lasso and Ridge regression. It involves generating data from a true function, addressing overfitting, and analyzing the impact of regularization on model performance. Additionally, it covers the application of scikit-learn for real estate price prediction, feature importance assessment, and optimal feature selection through cross-correlation.

Uploaded by

rishitpd2526
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
11 views2 pages

ML Assignment3

This lab assignment focuses on linear regression concepts including basis function expansion, bias, variance, and regularization techniques such as Lasso and Ridge regression. It involves generating data from a true function, addressing overfitting, and analyzing the impact of regularization on model performance. Additionally, it covers the application of scikit-learn for real estate price prediction, feature importance assessment, and optimal feature selection through cross-correlation.

Uploaded by

rishitpd2526
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Machine Learning

Lab Assignment 3
(Linear Regression)

Note: This exercise is to solidify your understanding of basis function expansion, bias, variance, underfitting,
overfitting, model capacity, regularization, ridge regression, and lasso regression.

Question 1
Let f (x) = x + 2 · sin(0.5 · x) be your true function.

1. Generate the data: Generate 50 data points for x ∈ [−10, 10] by adding noise drawn from a Gaussian
distribution with mean 0 and variance 0.5 to the true function.
2. Optimal model capacity: Use the learnt w to predict labels on test data and measure squared error on the
test set. Plot the training and generalization error against the model capacity for each case.
3. Addressing overfitting and Regularization: Perform regularization on the curve with maximum overfit-
ting.
• Use more data: For the overfitting case, sample more training data points from the function, say 100,
and now plot the true function, originally learnt curve, and the curve learnt on more data. What is the
observation?
• Lasso Regression: Perform Lasso regression on the cost function as follows, vary λ = 0 to 5 in steps
of 0.5:
N
1 X
J(w) = (yi − (w0 + wT ϕn (xi )))2 + λ|w|
N i=1

– Plot both training and test error for different values of λ or (ln λ) on the x-axis. What value of
hyperparameter λ is most suitable? Is w sparse? Plot the number of non-zero components of w
against λ.
– Plot the true function, and curves learnt with and without regularization. Compare and comment.
– Plot the L1 norm of w against λ.
• Ridge Regression: Perform Ridge regression on the cost function as follows, vary regularization coef-
ficient λ = 0 to 5 in steps of 0.5:
N
1 X
J(w) = (yi − (w0 + wT ϕn (xi )))2 + λ||w||2
N i=1

– Plot both training and test error for different values of λ or (ln λ) on the x-axis. What value of
hyperparameter λ is most suitable?
– Plot the L2 norm of w against λ.
– Plot the true function, and curves learnt with and without regularization. Compare and comment.
• Does Lasso regression give sparser solution than Ridge regression (more number of zero components in
w)?

1
Question 2
a) Show the usage of scikit-learn’s linear regression module for the real estate price prediction regression problem.
What is the RMS error on the test set?
b) Based on the regression coefficients, what can you comment about the importance of different features? Is it
correct to assume that larger coefficients mean more important features?
c) Now, standardize the dataset to have all features on a scale of 0 to 1. Re-learn the regression coefficients and
now comment on the importance of different features.
d) What is the distribution of the residuals?
e) Use cross-correlation to find the optimal set of features to use for regression. Using all possible feature sets of
length 1, 2, 3, or 4, what is the optimal feature set as per the validation set and how does this set of features
perform on the test set wrt the model learned on the entire feature set?

You might also like