Scribd
Upload
76 views9 pages

Overfitting Solutions for Moons and Circles Datasets

This document discusses various techniques for preventing neural networks from overfitting, including adding regularization, using early stopping, batch normalization, and dropout. It provides code examples of multilayer perceptrons applied to classification tasks on synthetic datasets like moons and circles. The models are trained and their learning curves are plotted to analyze effects of different regularization approaches on reducing overfitting.

Uploaded by

Meenakshi
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
76 views9 pages

Overfitting Solutions for Moons and Circles Datasets

This document discusses various techniques for preventing neural networks from overfitting, including adding regularization, using early stopping, batch normalization, and dropout. It provides code examples of multilayer perceptrons applied to classification tasks on synthetic datasets like moons and circles. The models are trained and their learning curves are plotted to analyze effects of different regularization approaches on reducing overfitting.

Uploaded by

Meenakshi
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

10/04/2020 improving the neuralnets .

ipynb - Colaboratory

1 # overfit mlp for the moons dataset
2 from [Link] import make_moons
3 from tensorfl[Link] import Sequential
4 from tensorfl[Link] import Dense
5 from matplotlib import pyplot
6 # generate 2d classification dataset
7 X, y = make_moons(n_samples=100, noise=0.2, random_state=1)
8 # split into train and test setsi
9 n_train = 30
10 trainX, testX = X[:n_train, :], X[n_train:, :]
11 trainy, testy = y[:n_train], y[n_train:]
12 # define model
13 model = Sequential()
14 [Link](Dense(500, input_dim=2, activation='relu'))
15 [Link](Dense(1, activation='sigmoid'))
16 [Link](loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
17 # fit model
18 history = model.fit(trainX, trainy, epochs=4000, validation_data=(testX, testy), verbose=2)
19 # evaluate the model
20 _, train_acc = [Link](trainX, trainy, verbose=0)
21 _, test_acc = [Link](testX, testy, verbose=0)
22 print('Train: %.3f, Test: %.3f' % (train_acc, test_acc))
23  

This is completly over tting Problems

1  

1 # plot loss learning curves
2 [Link](211)
3 [Link]('Cross-Entropy Loss', pad=-40)
4 [Link]([Link]['loss'], label='train')
5 [Link]([Link]['val_loss'], label='test')
6 l t l d()
[Link] 1/9
10/04/2020 improving the neuralnets .ipynb - Colaboratory
6 [Link]()
7 # plot accuracy learning curves
8 [Link](212)
9 [Link]('Accuracy', pad=-40)
10 [Link]([Link]['accuracy'], label='train')
11 [Link]([Link]['val_accuracy'], label='test')
12 [Link]()
13 [Link]()p
14  

1 # mlp overfit on the two circles dataset with activation regularization before activation
2 from [Link] import make_circles
3 from tensorfl[Link] import Dense
4 from tensorfl[Link] import Sequential
5 from tensorfl[Link] import l2
6 from tensorfl[Link] import Activation
7 from matplotlib import pyplot
8 # generate 2d classification dataset
9 X, y = make_circles(n_samples=100, noise=0.1, random_state=1)
10 # split into train and test
11 n_train = 30
12 trainX, testX = X[:n_train, :], X[n_train:, :]
13 trainy, testy = y[:n_train], y[n_train:]
[Link] 2/9
10/04/2020 improving the neuralnets .ipynb - Colaboratory
14 # define model
15 model = Sequential()
16 [Link](Dense(500, input_dim=2, activation='linear', activity_regularizer=l2(0.0001)))
17 [Link](Activation('relu'))
18 [Link](Dense(1, activation='sigmoid'))
19 [Link](loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
20 # fit model
21 history = model.fit(trainX, trainy, validation_data=(testX, testy), epochs=500, verbose=2)
22 # evaluate the model
23 _, train_acc = [Link](trainX, trainy, verbose=0)
24 _, test_acc = [Link](testX, testy, verbose=0)
25 print('Train: %.3f, Test: %.3f' % (train_acc, test_acc))
26 # plot loss learning curves
27 [Link](211)
28 [Link]('Cross-Entropy Loss', pad=-40)
29 [Link]([Link]['loss'], label='train')
30 [Link]([Link]['val_loss'], label='test')
31 [Link]()
32 # plot accuracy learning curves
33 [Link](212)
34 [Link]('Accuracy', pad=-40)
35 [Link]([Link]['accuracy'], label='train')
36 [Link]([Link]['val_accuracy'], label='test')
37 [Link]()
38 [Link]()
39  

1 # mlp overfit on the moons dataset with simple early stopping
2 from [Link] import make_moons
3 from tensorfl[Link] import Sequential
4 from tensorfl[Link] import Dense
5 from tensorfl[Link] import EarlyStopping
6 from matplotlib import pyplot
7 # generate 2d classification dataset
8 X, y = make_moons(n_samples=100, noise=0.2, random_state=1)
9 # split into train and test
10 n_train = 30
11 trainX testX = X[:n train :] X[n train: :]
[Link] 3/9
10/04/2020 improving the neuralnets .ipynb - Colaboratory
11 trainX, testX = X[:n_train, :], X[n_train:, :]
12 trainy, testy = y[:n_train], y[n_train:]
13 # define model
14 model = Sequential()
15 [Link](Dense(500, input_dim=2, activation='relu'))
16 [Link](Dense(1, activation='sigmoid'))
17 [Link](loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
18 # simple early stopping
19 es = EarlyStopping(monitor='val_loss', mode='min', verbose=1)
20 # fit model
21 history = model.fit(trainX, trainy, validation_data=(testX, testy), epochs=4000, verbose=2, callbacks=[es])
22 # evaluate the model
23 _, train_acc = [Link](trainX, trainy, verbose=0)
24 _, test_acc = [Link](testX, testy, verbose=0)
25 print('Train: %.3f, Test: %.3f' % (train_acc, test_acc))
26 # plot loss learning curves
27 [Link](211)
28 [Link]('Cross-Entropy Loss', pad=-40)
29 [Link]([Link]['loss'], label='train')
30 [Link]([Link]['val_loss'], label='test')
31 [Link]()
32 # plot accuracy learning curves
33 [Link](212)
34 [Link]('Accuracy', pad=-40)
35 [Link]([Link]['accuracy'], label='train')
36 [Link]([Link]['val_accuracy'], label='test')
37 [Link]()
38 [Link]()

1 # mlp for the two circles problem with batchnorm after activation function
2 from [Link] import make_circles
3 from tensorfl[Link] import Sequential
4 from tensorfl[Link] import Dense
5 from tensorfl[Link] import BatchNormalization
6 from tensorfl[Link] import SGD
7 from matplotlib import pyplot
8 # generate 2d classification dataset
9 X, y = make_circles(n_samples=1000, noise=0.1, random_state=1)
[Link] 4/9
10/04/2020 improving the neuralnets .ipynb - Colaboratory

10 # split into train and test
11 n_train = 500
12 trainX, testX = X[:n_train, :], X[n_train:, :]
13 trainy, testy = y[:n_train], y[n_train:]
14 # define model
15 model = Sequential()
16 [Link](Dense(50, input_dim=2, activation='relu', kernel_initializer='he_uniform'))
17 [Link](BatchNormalization())
18 [Link](Dense(1, activation='sigmoid'))
19 opt = SGD(lr=0.01, momentum=0.9)
20 [Link](loss='binary_crossentropy', optimizer=opt, metrics=['accuracy'])
21 # fit model
22 history = model.fit(trainX, trainy, validation_data=(testX, testy), epochs=100, verbose=2)
23 # evaluate the model
24 _, train_acc = [Link](trainX, trainy, verbose=0)
25 _, test_acc = [Link](testX, testy, verbose=0)
26 print('Train: %.3f, Test: %.3f' % (train_acc, test_acc))
27 # plot loss learning curves
28 [Link](211)
29 [Link]('Cross-Entropy Loss', pad=-40)
30 [Link]([Link]['loss'], label='train')
31 [Link]([Link]['val_loss'], label='test')
32 [Link]()
33 # plot accuracy learning curves
34 [Link](212)
35 [Link]('Accuracy', pad=-40)
36 [Link]([Link]['accuracy'], label='train')
37 [Link]([Link]['val_accuracy'], label='test')
38 [Link]()
39 [Link]()

Batch normalization will helps in over tting

Drop out

[Link] 5/9
10/04/2020 improving the neuralnets .ipynb - Colaboratory

1 # mlp overfit on the two circles dataset
2 from [Link] import make_circles
3 from tensorfl[Link] import Dense
4 from tensorfl[Link] import Sequential
5 from matplotlib import pyplot
6 # generate 2d classification dataset
7 X, y = make_circles(n_samples=100, noise=0.1, random_state=1)
8 # split into train and test
9 n_train = 30
10 trainX, testX = X[:n_train, :], X[n_train:, :]
11 trainy, testy = y[:n_train], y[n_train:]
12 # define model
13 model = Sequential()
14 [Link](Dense(500, input_dim=2, activation='relu'))
15 [Link](Dense(1, activation='sigmoid'))
16 [Link](loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
17 # fit model
18 history = model.fit(trainX, trainy, validation_data=(testX, testy), epochs=1000, verbose=2)
19 # evaluate the model
20 _, train_acc = [Link](trainX, trainy, verbose=0)
21 _, test_acc = [Link](testX, testy, verbose=0)
22 print('Train: %.3f, Test: %.3f' % (train_acc, test_acc))
23 # plot loss learning curves
24 [Link](211)
25 [Link]('Cross-Entropy Loss', pad=-40)
26 [Link]([Link]['loss'], label='train')
27 [Link]([Link]['val_loss'], label='test')
28 [Link]()
29 # plot accuracy learning curves
30 [Link](212)
31 [Link]('Accuracy', pad=-40)
32 [Link]([Link]['accuracy'], label='train')
33 [Link]([Link]['val_accuracy'], label='test')
34 [Link]()
35 [Link]()

1 [Link]()
[Link] 6/9
10/04/2020 improving the neuralnets .ipynb - Colaboratory

Model: "sequential_6"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_12 (Dense) (None, 500) 1500
_________________________________________________________________
dense_13 (Dense) (None, 1) 501
=================================================================
Total params: 2,001
Trainable params: 2,001
Non-trainable params: 0
_________________________________________________________________

1 # mlp with dropout on the two circles dataset
2 from [Link] import make_circles
3 from tensorfl[Link] import Sequential
a
4 from tensorfl[Link] import Dense
a
5 from tensorfl[Link] import Dropout
6  
7 from matplotlib import pyplot
8 # generate 2d classification dataset
9 X, y = make_circles(n_samples=100, noise=0.1, random_state=1)
10 # split into train and test
11 n_train = 30
12 trainX, testX = X[:n_train, :], X[n_train:, :]
13 trainy, testy = y[:n_train], y[n_train:]
14 # define model
15 model = Sequential()
16 [Link](Dense(500, input_dim=2, activation='relu'))
17 [Link](Dropout(0.3)) # dropping of 40% of conncections
18 [Link](Dense(1, activation='sigmoid'))
19 [Link](loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
20 # fit model
21 history = model.fit(trainX, trainy, validation_data=(testX, testy), epochs=1000, verbose=2)
22 # evaluate the model
23 _, train_acc = [Link](trainX, trainy, verbose=0)
24 _, test_acc = [Link](testX, testy, verbose=0)
25 print('Train: % 3f Test: % 3f' % (train acc test acc))
[Link] 7/9
10/04/2020 improving the neuralnets .ipynb - Colaboratory
25 print( Train: %.3f, Test: %.3f  % (train_acc, test_acc))
26 # plot loss learning curves
27 [Link](211)
28 [Link]('Cross-Entropy Loss', pad=-40)
29 [Link]([Link]['loss'], label='train')
30 [Link]([Link]['val_loss'], label='test')
31 [Link]()
32 # plot accuracy learning curves
33 [Link](212)
34 [Link]('Accuracy', pad=-40)
35 [Link]([Link]['accuracy'], label='train')
36 [Link]([Link]['val_accuracy'], label='test')
37 [Link]()
38 [Link]()

1 [Link]()

Model: "sequential_7"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_14 (Dense) (None, 500) 1500
_________________________________________________________________
dropout (Dropout) (None, 500) 0
_________________________________________________________________
dense_15 (Dense) (None, 1) 501
=================================================================
Total params: 2,001
Trainable params: 2,001
Non-trainable params: 0
_________________________________________________________________

[Link] 8/9
10/04/2020 improving the neuralnets .ipynb - Colaboratory

[Link] 9/9

You might also like