A Gentle Introduction
to Neural Networks
(with Python)
Tariq Rashid @postenterprise
PyCon Italy April 2017
Background
Ideas
DIY … and a live demo!
Handwriting
Thoughts
Background
Start With Two Questions
locate people in this photo add these numbers
2403343781289312
+ 2843033712837981
+ 2362142787897881
+ 3256541312323213
+ 9864479802118978
+ 8976677987987897
+ 8981257890087988
= ?
AI is Huge!
Google’s and Go
Ideas
Simple Predicting Machine
Simple Predicting Machine
Kilometres to Miles
random starting parameter
try a model - this one is linear
Kilometres to Miles
not great
Kilometres to Miles
better
Kilometres to Miles
worse !
Kilometres to Miles
best yet !
Key Points
a
g w o rk s e xactly? Try
in
’t k n ow how someth .
1. D o n
d ju s ta b le parameters
a
model with
.
re fi n e th e parameters
or to
2. Use the err
Garden Bugs
Classifying Bugs
Classifying Bugs
Classifying Bugs
Classifying Bugs
Key Points
things.
k in d a lik e predicting
thing s is
1. Classifying
Learning from Data
Example Width Length Bug
1 3.0 1.0 ladybird
2 1.0 3.0 caterpillar
Learning from Data
Learning from Data
not a good separator
Learning from Data
shift the line up
just above the training data point
Learning from Data
How Do We Update The Parameter?
error = target - actual
E = (A + ΔA)x - Ax
ΔA = E / x
Hang On!
Oh no!
each update ignores
previous examples
Calm Down the Learning
ΔA = L · (E / x)
learning rate
Calm Down the Learning
learning rate = 0.5
Key Points
you
ing is g o o d - ensures
your learn pact of
1. Moderating nd re d u c e s im
rn fro m all your data, a
lea .
o r n o is y training data
outliers
Boolean Logic
IF I have eaten my vegetables AND I am still hungry
THEN I can have ice cream.
IF it’s the weekend OR I am on annual leave THEN I’ll
go to the park.
Input A Input B AND OR
0 0 0 0
0 1 0 1
1 0 0 1
1 1 1 1
Boolean Logic
Boolean Logic
XOR Puzzle!
Input A Input B XOR
0 0 0
0 1 1
1 0 1
1 1 0
XOR Solution!
… Use more than one node!
Key Points
gle
e s o lv e d w ith just a sin
n’t b
So m e p roblems ca
1.
r classifier.
simple linea
e s w o rk in g together to
nod
o u c a n u s e multiple
2. Y th e se problem
s.
a n y o f
solve m
Brains in Nature
Brains in Nature
brain 0.4 grams 11,000 neurons
nature’s brains can eat, fly, navigate, fight,
communicate, play, learn …
.. and they’re resilient
302 neurons
37 billion neurons
(humans 20 billion)
https://en.wikipedia.org/wiki/List_of_animals_by_number_of_neurons
https://faculty.washington.edu/chudler/facts.html
Brains in Nature
Brains in Nature
logistic function
y = 1 / (1 + e-x)
Brains in Nature
Artificial Neuron
Artificial Neural Network .. finally!
Pause.
...
Where Does The Learning Happen?
link weight? sigmoid function slope?
Key Points
is ti c a te d things, and
do sop h ct
N a tu ra l brains can m a g e and imperfe
1. to d a
ly resilient
are incredib it io n a l c omputing.
like tr a d
signals .. un
d
a l b ra in s p artly inspire
logic
T ry in g to copy bio
2 .
l n e u r a l n etworks.
artificia
it’s
ju s ta b le p arameter -
e ad
L in k w e ights are th
3. s.
th e le a rn ing happen
where
Feeding Signals Forward
Feeding Signals Forward
Feeding Signals Forward
Matrix Multiplication
Matrix Multiplication
weights incoming signals
W·I = X
dot product
Key Points
e
rd c a lc u la tions can b
a no
. T he m any feedforw a tr ix m u lt iplication,
1 as m
x p re s s e d concisely
e
s h a p e th e network.
t
matter wha
g u a g e s c a n do matrix
lan
S o m e p rogramming tl y a nd quickly
.
2. ic ie n
n really eff
multiplicatio
Network Error
Network Error
Internal Error
Internal Error
Matrices Again!
Key Points
we
e e r r o r to guide how
th
R e m e m ber we use - li n k weights.
1 . m e te r
del’s para
refine a mo
e
u t n o d e s is easy - th
tp
or at the ou nd actual
2. The err d e s ir e d a
etween the
difference b
outputs.
o d e s is n ’t obvious. A
l n
. T h e e rr or at interna lit it in pr o portion to
3 to s p
e u ris ti c a pproach is
h
hts.
the link weig
be
a g a ti n g th e error can
back prop too!
4. … and m a trix m ultiplication
as a
expressed
Yes, But How Do We Actually Update The Weights?
Aaarrrggghhh !!
Perfect is the Enemy of Good
landscape is a complicated difficult mathematical function ..
… with all kinds of lumps, bumps, kinks …
Gradient Descent
smaller gradient .. you’re closer to the
bottom … take smaller steps?
Key Points
e
cti c a l w a y of finding th
a pra
G r ad ie n t descent is
1.
d if fic u lt fu nctions.
f
minimum o
by
n c e o f o v e rshooting
ha
Y o u ca n avoid the c d ie n t g e ts shallower.
2. eg ra
in g s m a ll e r steps if th
tak
tion
o rk is a d ifficult func
etw
h e e rr o r o f a neural n g ra dient desce
nt
3. T m a y b e
eights … so
of the link w
will help ...
Climbing Down the Network Error Landscape
We need to find this gradient
Error Gradient
E = (desired - actual)2
school level calculus (chain rule)
dE/dwij = - ej . oj . (1 - oj) . oi
previous node
A gentle intro to calculus
http://makeyourownneuralnetwork.blogspot.co.uk/2016/01/a-gentle-introduction-to-calculus.html
Updating the Weights
move wjk in the opposite direction to
the slope
remember that learning rate
DIY
Python Class and Functions
Neural Network
Class
Initialise Train Query
set size, initial weights do the learning query for answers
Python has Cool Tools
matrix maths
numpy
scipy
matplotlib
notebook
Function - Initialise
# initialise the neural network
def __init__(self, inputnodes, hiddennodes, outputnodes, learningrate):
# set number of nodes in each input, hidden, output layer
self.inodes = inputnodes
self.hnodes = hiddennodes
self.onodes = outputnodes
# link weight matrices, wih and who
# weights inside the arrays are w_i_j, where link is from node i to node j in the next layer
# w11 w21
# w12 w22 etc
self.wih = numpy.random.normal(0.0, pow(self.hnodes, -0.5), (self.hnodes, self.inodes))
self.who = numpy.random.normal(0.0, pow(self.onodes, -0.5), (self.onodes, self.hnodes))
# learning rate
self.lr = learningrate
# activation function is the sigmoid function
self.activation_function = lambda x: scipy.special.expit(x)
pass
numpy.random.normal()
random initial weights
Function - Query
combined weighted signals into hidden
layer
then sigmoid applied
# query the neural network
def query(self, inputs_list):
# convert inputs list to 2d array
inputs = numpy.array(inputs_list, ndmin=2).T
# calculate signals into hidden layer
hidden_inputs = numpy.dot(self.wih, inputs)
# calculate the signals emerging from hidden layer
hidden_outputs = self.activation_function(hidden_inputs)
# calculate signals into final output layer
final_inputs = numpy.dot(self.who, hidden_outputs)
# calculate the signals emerging from final output layer
final_outputs = self.activation_function(final_inputs)
return final_outputs
numpy.dot() similar for output layer
Function - Train
# train the neural network
def train(self, inputs_list, targets_list):
# convert inputs list to 2d array same feed forward as before
inputs = numpy.array(inputs_list, ndmin=2).T
targets = numpy.array(targets_list, ndmin=2).T
# calculate signals into hidden layer
hidden_inputs = numpy.dot(self.wih, inputs)
# calculate the signals emerging from hidden layer
hidden_outputs = self.activation_function(hidden_inputs)
# calculate signals into final output layer output layer errors
final_inputs = numpy.dot(self.who, hidden_outputs)
# calculate the signals emerging from final output layer
final_outputs = self.activation_function(final_inputs) hidden layer errors
# output layer error is the (target - actual)
output_errors = targets - final_outputs
# hidden layer error is the output_errors, split by weights, recombined at hidden nodes
hidden_errors = numpy.dot(self.who.T, output_errors)
# update the weights for the links between the hidden and output layers
self.who += self.lr * numpy.dot((output_errors * final_outputs * (1.0 - final_outputs)),
numpy.transpose(hidden_outputs))
# update the weights for the links between the input and hidden layers
self.wih += self.lr * numpy.dot((hidden_errors * hidden_outputs * (1.0 - hidden_outputs)),
numpy.transpose(inputs))
pass
update weights
Handwriting
Handwritten Numbers Challenge
MNIST Datasets
MNIST dataset:
60,000 training data examples
10,000 test data examples
MNIST Datasets
label
784 pixels
values
28 by 28 pixel image
Output Layer Values
Experiments
96% is very good!
we’ve only used simple ideas
and code
random processes
do go wonky!
More Experiments
98% is amazing!
Thoughts
Peek Inside The Mind Of a Neural Network?
Peek Inside The Mind Of a Neural Network?
this isn’t done very
often
Thanks!
live demo!
Finding Out More
makeyourownneuralnetwork.blogspot.co.uk
github.com/makeyourownneuralnetwork
www.amazon.co.uk/dp/B01EER4Z4G
twitter.com/myoneuralnet
slides goo.gl/JKsb62
Raspberry Pi Zero
It all works on a Raspberry Pi Zero
… and it only costs £4 / $5 !!
