Artificial Neural
Networks
part one
Artificial Intelligence for
Control and Identification
Dr. Wilbert G. Aguilar
Ph.D. in Automatic Control, Robotics and Computer Vision
Mster AR
2007 Dr. X. Parra & Dr. C. Angulo
Outline
1. Why Artificial Neural Networks?
Artificial Neural Networks
2. Model of a neuron
1. Why Artificial Neural Networks?
Von Neumanns Computer
Artificial Neural Networks
determinism
Human Brain
fuzzy behaviour
sequence of instructions
parallelism
high speed
slow speed
repetitive tasks
programming
uniqueness of solutions
ex. matrix product
adaptation to situations
learning
different solutions
ex. face recognition
1. Why Artificial Neural Networks?
[Human] Brain Operation
When humans recognize a face or take an object, they
do not solve equations
Artificial Neural Networks
Brain works in an associative way
Each sensorial state evokes a brain state (an
electro-chemical activity) which is memorized
depending on the necessities
1. Why Artificial Neural Networks?
Playing tennis
Ball trajectory depends on many different factors:
Artificial Neural Networks
shot strength, initial angle, racket trajectory, ball spin, wind
speed,
A desired trajectory requires:
accurate measure of all the variables
simultaneous solution of many complex equations, which
must be solved for each data acquisition (fast dynamics)
How makes a player to manage all that?
1. Why Artificial Neural Networks?
Playing tennis
In a learning phase, a human player tries and experiments with
different actions and memorizes the good ones:
Artificial Neural Networks
if the racket is in the right side and the ball comes from right to left
then move a step backward and cross the racket to left side
if racket is in the right side and the ball comes from left to right then
move a step forward
if racket is
1. Why Artificial Neural Networks?
Playing tennis
Artificial Neural Networks
In an operative phase, the brain controls the actions without
thinking, on the base of the learned associations
A similar mechanism is
used for speech recognition
or motion control
Outline
1. Why Artificial Neural Networks?
Artificial Neural Networks
2. Model of a neuron
2. Model of a neuron
Biological inspiration
Artificial Neural Networks
The basic scheme of a biological neuron is
soma
nucleus
axon
synapse
dendrites
2. Model of a neuron
Biological inspiration
Artificial Neural Networks
Incoming signals from other neurons determine if the neuron
shall fire
The contribution of the signals to the output depends on the
strength of the synaptic connection. The output depends on
the attenuation/amplification in the synapses.
Dendrites receive the amplified signals through the synapses
and send them to the cellular body, where they are added.
When the sum exceeds a threshold the neuron output is
active (firing)
2. Model of a neuron
Biological inspiration
Artificial Neural Networks
Some of the brain properties are:
Neuron speed
milliseconds
Number of neurons
1011 1012
Number of connections
103 104 per neuron
Number of synapses
1014 1016 nervous system
Distributed control
any CPU
Fault tolerant
graceful degradation
Low power consumption
no batteries (but food)
10
2. Model of a neuron
McCulloch & Pitts Model (1943)
Artificial Neural Networks
The McCulloch-Pitts neuron is a binary neuron with
only two states: active (fired/excited) and inactive
(not fired/excited)
The neuron worked by inputting either a 1 or 0 for
each of the inputs (binary inputs), where 1 represented true and 0 false. Likewise, the threshold was
given a real value (), say 1, which would allow for a 0
or 1 output if the threshold was met or exceeded
(binary output)
11
2. Model of a neuron
McCulloch & Pitts Model (1943)
Artificial Neural Networks
W. S. McCulloch and W. Pitts. A logical calculus of ideas immanent in nervous
activity. Bulletin of Mathematical Biophysics, 5:115--133, 1943.
inputs
synaptic
weights
x1
w1
x2
xm
xi{-1,1}
wi{-1,1}
y{-1,1}
summing
junction
w2
activation
function
output
wm
1
1 if
0
-1
-2
y=
-1
+1
Sign Function
+2
w x
i
i=1
-1 otherwise
12
2. Model of a neuron
Artificial Neural Networks
McCulloch & Pitts Model (1943)
inputs
synaptic
weights
x1
w1
summing
junction
x2
activation
function
output
w2
determine the parameters (w1, w2 and ) so that
the neuron represents a logical OR function
13
2. Model of a neuron
McCulloch & Pitts Model (1943)
Some interesting points:
Artificial Neural Networks
it is the relative magnitudes that are important and not their
absolute magnitudes
in the previous example we could change the weights and threshold
by a factor of 2 and the output would be the same
a realization of the AND function may be achieved by simply
changing the threshold to -1
we can specify weights and then determine the binary logic output
The next important step is the ability to specify the desired
output and adjust the weights to achieve that output
14
2. Model of a neuron
Hebbs Rule (1949)
Hebb, D.O. The organization of behavior; a neuropsychological theory. WileyInterscience, New York, 1949
Artificial Neural Networks
Biological Rule: the synapse resistance to the incoming signal
can be changed during a learning process
If an input of a neuron is repeatedly and
persistently causing the neuron to fire, a
metabolic change happens in the synapse of
that particular input to reduce its resistance
learning is not an inherently property of the
neuron but is due to synapses modification
15
2. Model of a neuron
Hebbs Rule (1949)
Artificial Neural Networks
Hebbian Learning Rule: a change in the strength of a connection
is a function of the pre- and postsynaptic neural activities
It is a method of determining how to alter the weights between
model neurons
If xi is the output of the presynaptic neuron, yj the output of the
postsynaptic neuron, and wji the strength of the connection
between them, and learning rate, the form of the learning rule is:
w ji = xi y j
16
2. Model of a neuron
The Perceptron (1958)
Rosenblatt, Frank (1958), The Perceptron: A Probabilistic Model for Information Storage and
Organization in the Brain, Cornell Aeronautical Lab, Psychological Review, v65, N.6, pp.386-408.
Artificial Neural Networks
Combines the McCulloch-Pitts model of an artificial neuron and
the Hebbian learning rule of adjusting weights
In addition to the variable weight values, the perceptron model
added an extra input that represents bias
bias
x1
w1
x2
w2
summing
junction
xm
wm
inputs
synaptic
weights
y
output
activation
function
17
2. Model of a neuron
The Perceptron (1958)
It produces associations between inputs and outputs:
\ {0,1}
Artificial Neural Networks
Pk Tk
k = 1... p
where Pk are the inputs or patterns and Tk are the outputs or
targets
The set of patterns and targets is the learning set or training set
{Pk , Tk }k =1... p
18
2. Model of a neuron
The Perceptron (1958)
Artificial Neural Networks
Perceptron Learning Rule: change the weight by an amount
proportional to the difference between the desired output and the
actual output
w ji = ( Tkj y j ) Pk i
learning rate
desired output
actual output
w ji (t + 1) = w ji (t ) + ( Tkj y j (t ) ) Pk i
i = 1...m
j = 1...n
k = 1... p
19
2. Model of a neuron
The Perceptron (1958)
Novikoff, A. B. (1962). On convergence proofs on perceptrons. Symposium on the
Mathematical Theory of Automata, 12, 615-622. Polytechnic Institute of Brooklyn.
Artificial Neural Networks
Perceptron Theorem: if this process is
repeated cyclical, ends up converging to the
weights looked for in finite time
for
Pk
inputs,
t < | y j (t ) = Tkj
20
2. Model of a neuron
The Perceptron (1958)
Artificial Neural Networks
Vector notation:
x0
w0
x1
w1
x2
w2
xm
wm
x = [x0 x1 x2 ... xm]T
w = [w0 w1 w2 ... wm]T
y = F(wTx) = F(wxT)
21
2. Model of a neuron
Artificial Neural Networks
The Perceptron (1958)
The equation for wxT, can be
viewed as an equation of a line.
Depending on the values of the
weights, this line will separate
the four possible inputs into two
categories
x2
C
x1
Equation of the line becomes
w2 =
w1
w2
x1
w0
w2
w0=-1
w1=-1 A
w2=1 w =1
0
w1=-1
w2=1
B
w0=1
w1=1
w2=1
w0=-1
w1=1
w2=1
22
2. Model of a neuron
The Perceptron (1958)
Example
Artificial Neural Networks
(0,1)
(1,1)
(-1,1)
(1,0)
(-1,0)
(0,-1)
\2
{0,1}
(-1,1)
(-1,0)
(0,-1)
(1,0)
(0,1)
(1,1)
23
2. Model of a neuron
The Perceptron (1958)
Minsky M L and Papert S A 1969 Perceptrons (Cambridge, MA: MIT Press)
The XOR problem Minsky & Papert (1969)
Artificial Neural Networks
(0,1)
(0,0)
(1,1)
(1,0)
\2
{0,1}
(0,0)
(0,1)
(1,0)
(1,1)
24
2. Model of a neuron
The Perceptron (1958)
The XOR problem Minsky & Papert (1969)
Artificial Neural Networks
(0,1)
(0,0)
(1,1)
(1,0)
\2
{0,1}
(0,0)
(0,1)
(1,0)
(1,1)
Functions such as XOR, require two lines to separate the
points into the appropriate classes. Hence we would require
multiple layers of neural units to represent this function
25
2. Model of a neuron
The Perceptron (1958)
Artificial Neural Networks
Synthesis :
Architecture
single layer feedforward
Transfer function
Hardlim
Associations
\ m {0,1}
Learning rule
w ji (t + 1) = w ji (t ) + ( Tkj y j (t ) ) Pk i
i = 1...m
j = 1...n
k = 1... p
ERROR
26
2. Model of a neuron
Learning
The aim of the learning is to find an association between the
patterns and the targets of the training set
Artificial Neural Networks
actual
output
p(1,1)
P= #
p(p,1)
y(1,1)
Y= #
y(p,1)
" p(1,m)
%
"
" y(1,n)
%
"
p(p,m)
t(1,1)
T= #
t(p,1)
y(p,n)
t(1,1) - y(1,1)
E=T-Y= #
t(p,1) - y(p,1)
desired
output
" t(1,n)
%
"
t(p,n)
" t(1,n) - y(1,n)
%
"
= [ 0]
t(p,n) - y(p,n)
27
2. Model of a neuron
Learning
so the aim of the learning is to find the weights that
minimizes the error, but
Artificial Neural Networks
how can we measure the error E = [0]?
Objective function
p
1 p n 2
e = e(w ) = E ( k,j) = diag ( ET E )
2 k =1 j=1
k =1
LEAST
SQUARE
ERROR
28
2. Model of a neuron
Learning
Artificial Neural Networks
It is possible to define the problem as a function approximation
2
1 p n 2
1 p n
min e = e(w ) = E ( k,j) = ( T ( k,j) Y ( k,j) )
2 k =1 j=1
2 k =1 j=1
where
Y ( k,j) = F w ( j,i ) P ( k,i )
i =0
Classification: classifier function
error
surface
Regression: approximation function
29
