Recurrent Neural Networks for Temporal Sequences Recognition

2011

The RNNs (Recurrent Neural Networks) are a general case of arti cial neural networks where the connections are not feed-forward ones only. In RNNs, connections between units form directed cycles, providing an implicit internal memory. Those RNNs are adapted to problems dealing with signals evolving through time. Their internal memory gives them the ability to naturally take time into account. Valuable approximation results have been obtained for dynamical systems. During the last few years, several interesting neural networks developments have emerged such as spike nets and deep networks. This book will show that a lot of improvement and results are also present in the active  eld of RNNs. In the  rst chapter, we will see that many diff erent algorithms have been applied to prediction in time series. ARIMA, one of the models studied, combines three models (AR, MA and ARMA). It is compared to Elman-RNN with four diff erent architectures. The second chapter gives an overview of RNN for time series prediction. The algorithm BPTT is detailed then delayed connections are added resulting into two new algorithms: EBPTT and CBPTT. BPTT is also upgraded through boosting thus giving much bett er results especially on multi-step ahead prediction. The third chapter presents the application of RNN to the diagnosis of Carpal Tunnel Syndrome. The RNN used in this study is Elman-RNN and the Levenberg-Marquardt learning algorithm is detailed. The fourth chapter describes the use of neural networks to model the hysteresis phenomena encountered on human meridian systems. Models using extreme learning machine (ELM) with a non-recurrent neural network and a RNN are compared. The  ft h chapter shows the use of a dynamic RNN to model the dynamic control of human movement. From multiple signals (EMG and EEG), the goal is to  nd the mapping with the movement of the diff erent parts of the body. Some relations found by the RNN help for a bett er understanding of motor organization in the human brain. The sixth chapter proposes a paradigm of how the brain deals with active interaction with environment. It is based on Compact Internal Representation (CIR). The RNNs are used here to learn and retrieve these CIRs and also to predict the trajectories of moving obstacles.

Neural Networks, 1990

A recurrent, synchronous neural network is treated as a collection of independent perceptrons. The dynamics of the network can be described by a mapping: a finite set of transitions in the state space of the network. We define legal mapping as a mapping that a synchronous neural network is able to perform, and state the necessary and sufficient conditions for a mapping to be legal. A learning algorithm for the network, based on the perceptron's learning algorithm, is guaranteed to converge to a solution when the network is trained to realize a legal mapping. It is shown that the algorithm performs a gradient descent search for a minimum of a cost function that is a certain error measure in the weight space. Performance of the algorithm for the associative memory application and for temporal sequences production is illustrated by numerical simulations. A method is proposed for legalizing any given mapping at the expense of adding a finite number of neurons to the network. It is also shown that when the number of transitions in a random mapping is less than the number of neurons in the network, the probability that such a mapping is legal approaches unity.

Neural Networks, IEEE …, 1997

1998

Recurrent Self-Organizing Map (RSOM) is studied in temporal sequence processing. RSOM includes a recurrent difference vector in each unit of the map, which allows storing temporal context from consecutive input vectors fed to the map. RSOM is a modification of the Temporal Kohonen Map (TKM). It is shown that RSOM learns a correct mapping from temporal sequences of a simple synthetic data, while TKM fails to learn this mapping. In addition, two case studies are presented, in which RSOM is applied to EEG based epileptic activity detection and to time series prediction with local models. Results suggest that RSOM can be efficiently used in temporal sequence processing.

1990

We design neural networks to learn, recognize, and reproduce complex temporal sequence, with short-term memory (STM) modeled by units comprising recurrent excitatory connections between two neurons (a dual neuron model). The output of a neuron has graded values instead of binary ones. By applying the Hebbian learning rule at each synapse and a normalization rule among all synaptic weights of a neuron, we show that a certain quantity, called the input potential, increases monotonically with sequence presentation, and that the neuron can only be fired when its input signals are arranged in a specific sequence. These sequencedetecting neurons form the basis for our model of complex sequence recognition, which can tolerate distortions of the learned sequences. A recurrent network of two layers is provided for reproducing complex sequences.

Logic Journal of IGPL, 2010

Recurrent neural networks (RNNs) offer flexible machine learning tools which share the learning abilities of feedforward networks and which extend their expression abilities based on dynamical equations. Hence, they can directly process complex spatiotemporal data and model complex dynamic systems. Since temporal and spatial data are present in many domains such as processing environmental time series, modelling the financial market, speech and language processing, robotics, bioinformatics, medical informatics, etc., RNNs constitute promising candidates for a variety of applications. Further, their rich dynamic repertoire as time dependent systems makes them suitable candidates for modelling brain phenomena or mimicking large-scale distributed computations and argumentations. Thus, RNNs carry the promise of efficient biologically plausible signal processing models optimally suited for a wide area of industrial applications on the one hand and an explanation of cognitive phenomena of the human brain on the other hand.

Neural computation, 1989

2012 IEEE 27th Convention of Electrical and Electronics Engineers in Israel, 2012

We show that real valued continuous functions can be recognized in a reliable way, with good generalization ability using an adapted version of the Liquid State Machine (LSM) that receives direct real valued input. Furthermore this system works without the necessity of preliminary extraction of signal processing features. This avoids the necessity of discretization and encoding that has plagued earlier attempts on this process. We show this is effective on a simulated signal designed to have the properties of a physical trace of human speech. The main changes to the basic liquid state machine paradigm are (i) external stimulation to neurons by normalized real values and (ii) adaptation of the integrate and fire neurons in the liquid to have a history dependent sliding threshold (iii) topological constraints on the network connectivity.

2016

Recurrent Neural Networks (RNNs) are powerful architectures for sequence learning. Recent advances on the vanishing gradient problem have led to improved results and an increased research interest. Among recent proposals are architectural innovations that allow the emergence of multiple timescales during training. This paper explores a number of architectures for sequence generation and prediction tasks with long-term relationships. We compare the Simple Recurrent Network (SRN) and Long Short-Term Memory (LSTM) with the recently proposed Clockwork RNN (CWRNN), Structurally Constrained Recurrent Network (SCRN), and Recurrent Plausibility Network (RPN) with regard to their capabilities of learning multiple timescales. Our results show that partitioning hidden layers under distinct temporal constraints enables the learning of multiple timescales, which contributes to the understanding of the fundamental conditions that allow RNNs to self-organize to accurate temporal abstractions.