Supervised Learning for Decorrelated Gaussian Networks

The Generalized Hebbian Algorithm has been proposed for training linear feedforward neural networks and has been proven to cause the weights to converge to the eigenvectors of the input distribution (Sanger 1989a, b). For an input distribution given by 2D Gaussian smoothed white noise inside a Gaussian window, some of the masks learned by the Generalized Hebbian Algorithm resemble edge and bar detectors. Since these do not match the form of the actual eigenvectors of this distribution , we seek an explanation of the development of the masks prior to complete convergence to the correct solution. Analysis in the spatial and spatial frequency domains sheds light on this development, and shows that the masks which occur tend to be localized in the spatial frequency domain, reminiscent of one of the properties of 2D Gabor filters proposed by as a model for the receptive fields of cells in primate visual cortex.

arXiv (Cornell University), 2022

We consider dense, associative neural-networks trained with no supervision and we investigate their computational capabilities analytically, via statistical-mechanics tools, and numerically, via Monte Carlo simulations. In particular, we obtain a phase diagram summarizing their performance as a function of the control parameters (e.g. quality and quantity of the training dataset, network storage, noise) that is valid in the limit of large network size and structureless datasets. Moreover, we establish a bridge between macroscopic observables standardly used in statistical mechanics and loss functions typically used in the machine learning. As technical remarks, from the analytical side, we extend Guerra's interpolation to tackle the non-Gaussian distributions involved in the post-synaptic potentials while, from the computational counterpart, we insert Plefka's approximation in the Monte Carlo scheme, to speed up the evaluation of the synaptic tensor, overall obtaining a novel and broad approach to investigate unsupervised learning in neural networks, beyond the shallow limit.

Biological Cybernetics, 1990

How does the brain form a useful representation of its environment? It is shown here that a layer of simple Hebbian units connected by modifiable anti-Hebbian feed-back connections can learn to code a set of patterns in such a way that statistical dependency between the elements of the representation is reduced, while information is preserved. The resulting code is sparse, which is favourable if it is to be used as input to a subsequent supervised associative layer. The operation of the network is demonstrated on two simple problems.

Journal of Intelligent and Fuzzy Systems, 2007

A new local (Hebbian) learning algorithm for artificial neurons is presented. It is shown that, in spite of its implementation simplicity, this new algorithm, applied to neurons with sigmoidal activation function, performs data clustering by finding valleys of the probability density function (PDF) of the multivariate random variables that model incoming data. Some interesting features of this new algorithm are illustrated by some experiments based on both artificial data and real world data.

Systems and Computers in Japan, 1996

Hebbian rule might be the most popular one as an unsupervised learning model of neural nets. Recently, the opposite of the Hebbian rule, i.e., the socalled anti-Hebbian rule, has drawn attention as a new learning paradigm. This paper first clarifies some fundamental properties of the anti-Hebbian rule, and then shows that a variety of networks can be acquired by some anti-Hebbian rules.

The paper consists of two parts, each of them describing a learning neural network with the same modular architecture and with a similar set of functioning algorithms. Both networks are artificially partitioned into several equal modules according to the number of classes that the network has to recognize. Hebbian learning rule is used for network training. In the first network, learning process is concentrated inside the modules so that a system of intersecting neural assemblies is formed in each module. Unlike that, in the second network, learning connections link only neurons of different modules. Computer simulation of the networks is performed. Testing of the networks is executed on the MNIST database. Both networks directly use brightness values of image pixels as features. The second network has a better performance than the first one and demonstrates the recognition rate of 98.15%.

Systems and Computers in Japan, 1999

Self-organizing neural networks using sparse coding have been studied recently, and these studies have proposed various learning rules for the formation of interlayer connections: Hebbian, reconstructive, and synapticcompetition rules. This paper discusses the synaptic-competition learning rule proposed by Maekawa and colleagues. The equilibrium state of the learning rule is analyzed, and the mean square reconstruction error, which is a potential function of reconstructive learning, is shown to be a Lyapunov function of the synaptic-competition learning rule. The result obtained is that the objective of the synaptic-competition learning rule is almost the same as that of reconstructive learning: reconstruction of input signals. Computer simulations performed to verify the theoretical results show in addition that the synaptic-competition learning rule can form local receptive fields more stably than other methods considered. ©1999 Scripta Technica, Syst Comp Jpn, 30(8): 110, 1999

Formal Aspects of Computing, 1996

In this paper we address the question of how interactions affect the formation and organization of receptive fields in a network composed of interacting neurons with Hebbian-type learning. We show how to partially decouple single cell effects from network effects, and how some phenomenological models can be seen as approximations to these learning networks. We show that the interaction affects the structure of receptive fields. We also demonstrate how the organization of different receptive fields across the cortex is influenced by the interaction term, and that the type of singularities depends on the symmetries of the receptive fields.

Neural Computation, 2012

We show how a Hopfield network with modifiable recurrent connections undergoing slow Hebbian learning can extract the underlying geometry of an input space. First, we use a slow and fast analysis to derive an averaged system whose dynamics derives from an energy function and therefore always converges to equilibrium points. The equilibria reflect the correlation structure of the inputs, a global object extracted through local recurrent interactions only. Second, we use numerical methods to illustrate how learning extracts the hidden geometrical structure of the inputs. Indeed, multidimensional scaling methods make it possible to project the final connectivity matrix onto a Euclidean distance matrix in a high-dimensional space, with the neurons labeled by spatial position within this space. The resulting network structure turns out to be roughly convolutional. The residual of the projection defines the nonconvolutional part of the connectivity, which is minimized in the process. Fina...

Frontiers in Neuroscience, 2019

Neurons in the dorsal pathway of the visual cortex are thought to be involved in motion processing. The first site of motion processing is the primary visual cortex (V1), encoding the direction of motion in local receptive fields, with higher order motion processing happening in the middle temporal area (MT). Complex motion properties like optic flow are processed in higher cortical areas of the Medial Superior Temporal area (MST). In this study, a hierarchical neural field network model of motion processing is presented. The model architecture has an input layer followed by either one or cascade of two neural fields (NF): the first of these, NF1, represents V1, while the second, NF2, represents MT. A special feature of the model is that lateral connections used in the neural fields are trained by asymmetric Hebbian learning, imparting to the neural field the ability to process sequential information in motion stimuli. The model was trained using various traditional moving patterns such as bars, squares, gratings, plaids, and random dot stimulus. In the case of bar stimuli, the model had only a single NF, the neurons of which developed a direction map of the moving bar stimuli. Training a network with two NFs on moving square and moving plaids stimuli, we show that, while the neurons in NF1 respond to the direction of the component (such as gratings and edges) motion, the neurons in NF2 (analogous to MT) responding to the direction of the pattern (plaids, square object) motion. In the third study, a network with 2 NFs was simulated using random dot stimuli (RDS) with translational motion, and show that the NF2 neurons can encode the direction of the concurrent dot motion (also called translational flow motion), independent of the dot configuration. This translational RDS flow motion is decoded by a simple perceptron network (a layer above NF2) with an accuracy of 100% on train set and 90% on the test set, thereby demonstrating that the proposed network can generalize to new dot configurations. Also, the response properties of the model on different input stimuli closely resembled many of the known features of the neurons found in electrophysiological studies.

Lecture Notes in Computer Science, 1997

In previous work it was shown that a mechanism competing for a presynaptic factor enables the self-organizing formation of local receptive elds with orientation selectivity, even though the synapses between the input and output layers are all nonlocal, i.e., fully connected. The previous model, however, assumed a priori competitive systems, called hypercolumns, that may not appropriately represent the inherent structure of the input, which is a hierarchy of low-to high-level features. In this paper we propose to use a self-organizing competitive system, rather than an a priori determined system. Self-organization is implemented by including F oldi ak's anti-Hebbian learning rule in out system. Computer simulations show that this model allows for the formation of local receptive elds with orientation selectivity, and that selforganization successfully structures the competitive system.

1994

Neurons learning under an unsupervised Hebbian learning rule can perform a nonlinear generalization of principal component analysis. This relationship between nonlinear PCA and nonlinear neurons is reviewed. The stable fixed points of the neuron learning dynamics correspond to the maxima of the statist,ic optimized under nonlinear PCA. However, in order to predict. what the neuron learns, knowledge of the basins of attractions of the neuron dynamics is required. Here the correspondence between nonlinear PCA and neural networks breaks down. This is shown for a simple model. Methods of statistical mechanics can be used to find the optima of the objective function of non-linear PCA. This determines what the neurons can learn. In order to find how the solutions are partitioned amoung the neurons, however, one must solve the dynamics.

In a physical neural system, where storage and processing are intimately intertwined, the rules for adjusting the synaptic weights can only depend on variables that are available locally, such as the activity of the pre-and post-synaptic neurons, resulting in local learning rules. A systematic framework for studying the space of local learning rules is obtained by first specifying the nature of the local variables, and then the functional form that ties them together into each learning rule. Such a framework enables also the systematic discovery of new learning rules and exploration of relationships between learning rules and group symmetries. We study polynomial local learning rules stratified by their degree and analyze their behavior and capabilities in both linear and non-linear units and networks. Stacking local learning rules in deep feedforward networks leads to deep local learning. While deep local learning can learn interesting representations, it cannot learn complex input-output functions, even when targets are available for the top layer. Learning complex input-output functions requires local deep learning where target information is communicated to the deep layers through a backward learning channel. The nature of the communicated information about the targets and the structure of the learning channel partition the space of learning algorithms. For any learning algorithm, the capacity of the learning channel can be defined as the number of bits provided about the error gradient per weight, divided by the number of required operations per weight. We estimate the capacity associated with several learning algorithms and show that backpropagation outperforms them by simultaneously maximizing the information rate and minimizing the computational cost. This result is also shown to be true for recurrent networks, by unfolding them in time. The theory clarifies the concept of Hebbian learning, establishes the power and limitations of local learning rules, introduces the learning channel which enables a formal analysis of the optimality of backpropagation, and explains the sparsity of the space of learning rules discovered so far.

Journal of Physics A: Mathematical and General, 1993

A model describing the dynamics of the synaptic weights of a single neuron performing Hebbian learning is described. The neuron is repeatedly excited by a set of input patterns. Its response is modeled as a continuous, nonlinear function of its excitation. We study how the model forms a self-organized representation of the set of input patterns. The dynamical equations are solved directly in a few simple cases. The model is studied for random patterns by a signal to noise analysis, and by introducing a partition function and applying the replica approach. As the number of patterns is increased a rst order phase transitions occurs where the neuron becomes unable to remember one pattern but rather learns to a mixture of very many patterns. The critical number of patterns for this transition scales as N b , where N is the number of synapses and b is the degree of nonlinearity. The leading order nite size corrections are calculated and compared with numerical simulations. It is shown how the representation of the input patterns learned by the neuron depends upon the nonlinearity in the neuron's response. Two types of behaviour can be identi ed depending on the degree of nonlinearity: either the neuron learns to discriminate one pattern from all the others, or it will learn to a complex mixture of many of the patterns.

Advances in Artificial Life, ECAL 2013, 2013

Hebbian learning is a classical non-supervised learning algorithm used in neural networks. Its particularity is to transcribe the correlations between couple of neurons within their connecting synapse. From this idea, we created a robotic task where 2 sensory modalities indicate the same target in order to find out if a neural network equipped with Hebbian learning could naturally exploit the relation between those modalities. Another question we explored is the difference in terms of learning between a feedforward neural network(FNN) and spiking neural network(SNN). Our results indicate that a FNN can partially exploit the relation between the modalities and the task when receiving a feedback from a teacher. We also found out that a SNN could not complete the task because of the nature of the Hebbian learning modeled.