Logistic regression
and regularization
LINEAR CLASSIFIERS IN PYTHON
Michael (Mike) Gelbart
Instructor, The University of British
Columbia
Regularized logistic regression
LINEAR CLASSIFIERS IN PYTHON
Regularized logistic regression
LINEAR CLASSIFIERS IN PYTHON
How does regularization affect training accuracy?
lr_weak_reg = LogisticRegression(C=100)
lr_strong_reg = LogisticRegression(C=0.01)
lr_weak_reg.fit(X_train, y_train)
lr_strong_reg.fit(X_train, y_train)
lr_weak_reg.score(X_train, y_train)
lr_strong_reg.score(X_train, y_train)
1.0
0.92
regularized loss = original loss + large coefficient penalty
more regularization: lower training accuracy
LINEAR CLASSIFIERS IN PYTHON
How does regularization affect test accuracy?
lr_weak_reg.score(X_test, y_test)
0.86
lr_strong_reg.score(X_test, y_test)
0.88
regularized loss = original loss + large coefficient penalty
more regularization: lower training accuracy
more regularization: (almost always) higher test accuracy
LINEAR CLASSIFIERS IN PYTHON
L1 vs. L2 regularization
Lasso = linear regression with L1 regularization
Ridge = linear regression with L2 regularization
For other models like logistic regression we just say L1, L2, etc.
lr_L1 = LogisticRegression(solver='liblinear', penalty='l1')
lr_L2 = LogisticRegression() # penalty='l2' by default
lr_L1.fit(X_train, y_train)
lr_L2.fit(X_train, y_train)
plt.plot(lr_L1.coef_.flatten())
plt.plot(lr_L2.coef_.flatten())
LINEAR CLASSIFIERS IN PYTHON
L2 vs. L1 regularization
LINEAR CLASSIFIERS IN PYTHON
Let's practice!
LINEAR CLASSIFIERS IN PYTHON
Logistic regression
and probabilities
LINEAR CLASSIFIERS IN PYTHON
Michael (Mike) Gelbart
Instructor, The University of British
Columbia
Logistic regression probabilities
Without regularization model coefficients:
(C = 108 ): [[1.55 1.57]]
model intercept: [-0.64]
LINEAR CLASSIFIERS IN PYTHON
Logistic regression probabilities
Without regularization
(C = 108 ):
model coefficients:
[[1.55 1.57]]
model intercept: [-0.64]
LINEAR CLASSIFIERS IN PYTHON
Logistic regression probabilities
Without regularization With regularization (C = 1):
(C = 108 ):
model coefficients: model coefficients:
[[1.55 1.57]] [[0.45 0.64]]
model intercept: [-0.64] model intercept: [-0.26]
LINEAR CLASSIFIERS IN PYTHON
How are these probabilities computed?
logistic regression predictions: sign of raw model output
logistic regression probabilities: "squashed" raw model output
LINEAR CLASSIFIERS IN PYTHON
Let's practice!
LINEAR CLASSIFIERS IN PYTHON
Multi-class logistic
regression
LINEAR CLASSIFIERS IN PYTHON
Michael (Mike) Gelbart
Instructor, The University of British
Columbia
Combining binary classifiers with one-vs-rest
lr0.fit(X, y==0) lr2.decision_function(X)[0]
lr1.fit(X, y==1)
-7.532
lr2.fit(X, y==2)
lr = LogisticRegression(multi_class='ovr')
lr.fit(X, y)
# get raw model output
lr.predict(X)[0]
lr0.decision_function(X)[0]
0
6.124
lr1.decision_function(X)[0]
-5.429
LINEAR CLASSIFIERS IN PYTHON
One-vs-rest: "Multinomial" or "softmax":
fit a binary classifier for fit a single classifier for all
each class classes
predict with all, take largest prediction directly outputs
output best class
pro: simple, modular con: more complicated, new
code
con: not directly optimizing
accuracy pro: tackle the problem
common for SVMs as well directly
can produce probabilities possible for SVMs, but less
common
can produce probabilities
LINEAR CLASSIFIERS IN PYTHON
Model coefficients for multi-class
lr_ovr = LogisticRegression(multi_class='ovr') lr_mn = LogisticRegression(multi_class="multinomial")
lr_mn.fit(X,y)
lr_ovr.fit(X,y)
lr_mn.coef_.shape
lr_ovr.coef_.shape
(3,13)
(3,13)
lr_mn.intercept_.shape
lr_ovr.intercept_.shape
(3,)
(3,)
LINEAR CLASSIFIERS IN PYTHON
Let's practice!
LINEAR CLASSIFIERS IN PYTHON
