Neural Information Processing Systems, Nov 29, 1993
Most theoretical investigations of large recurrent networks focus on the properties of the macros... more Most theoretical investigations of large recurrent networks focus on the properties of the macroscopic order parameters such as population averaged activities or average overlaps with memories. However, the statistics of the fluctuations in the local activities may be an important testing ground for comparison between models and observed cortical dynamics. We evaluated the neuronal correlation functions in a stochastic network comprising of excitatory and inhibitory populations. We show that when the network is in a stationary state, the cross-correlations are relatively weak, i.e., their amplitude relative to that of the auto-correlations are of order of 1/ N, N being the size of the interacting population. This holds except in the neighborhoods of bifurcations to nonstationary states. As a bifurcation point is approached the amplitude of the cross-correlations grows and becomes of order 1 and the decay timeconstant diverges. This behavior is analogous to the phenomenon of critical slowing down in systems at thermal equilibrium near a critical point. Near a Hopf bifurcation the cross-correlations exhibit damped oscillations.
Various Aspects Of Artificial And Biological Neural Networks
The field of neural networks is a wide and diverse field which spans a variety of interests, mode... more The field of neural networks is a wide and diverse field which spans a variety of interests, modelling methods, and potential uses. In my thesis, I discuss two different topics. The first is the use of feed-forward networks for time-series analysis and prediction. A new criterion for the quality of predictions of neural networks is presented. The criterion is based on the performance of networks for long term predictions, where one compares the distribution of the predicted values with that of the original data. Using this criterion as a weak constraint, one is able to choose the best networks from a population of trained networks that are similar in their short term predictions. This criterion may be also used to create a new predictor in which the various networks are weighted according to their long term behavior. In addition, a method for improving both short and long term predictions is proposed. Since quite often the consecutive errors of prediction are correlated in a systema...
Physical variables such as the orientation of a line in the visual field or the location of the b... more Physical variables such as the orientation of a line in the visual field or the location of the body in space are coded as activity levels in populations of neurons. Reconstruction or decoding is an inverse problem in which the physical variables are estimated from observed neural activity. Reconstruction is useful first in quantifying how much information about the physical variables is present in the population, and second, in providing insight into how the brain might use distributed representations in solving related computational problems such as visual object recognition and spatial navigation. Two classes of reconstruction methods, namely, probabilistic or Bayesian methods and basis function methods, are discussed. They include important existing methods as special cases, such as population vector coding, optimal linear estimation and template matching. As a representative example for the reconstruction problem, different methods were applied to multi-electrode spike train da...
One of the main experimental tools in probing the interactions between neurons has been the measu... more One of the main experimental tools in probing the interactions between neurons has been the measurement of the correlations in their activity. In general, however the interpretation of the observed correlations is difficult, since the correlation between a pair of neurons is influenced not only by the direct interaction between them but also by the dynamic state of the entire network to which they belong. Thus, a comparison between the observed correlations and the predictions from specific model networks is needed. In this paper we develop the theory of neuronal correlation functions in large networks comprising of several highly connected subpopulations, and obey stochastic dynamic rules. When the networks are in asynchronous states, the crosscorrelations are relatively weak, i.e., their amplitude relative to that of the auto-correlations is of order of 1=N , N being the size of the interacting populations. Using the weakness of the cross-correlations, general equations whi...
e-mail: The induction of rules by making queries is a dynamical process based on seeking informat... more e-mail: The induction of rules by making queries is a dynamical process based on seeking information. Experimenters typically look for one dominant strategy that is used by subjects, which may or may not agree with normative models of this psychological process. In this study we approach this problem from a different perspective, related to work in learning theory (see for example Baum 1991, Freund et al. 1995). Using information theory in a Bayesian framework, we estimated the information gained by queries when the task is to find a specific rule in a hypothesis space. Assuming that at each point subjects have a preferred working hypothesis, we considered several possible strategies, and determined the best one
such as the orientation of a line in the visual field or the location of Two main goals for recon... more such as the orientation of a line in the visual field or the location of Two main goals for reconstruction are approached in this the body in space are coded as activity levels in populations of neurons. Reconstruction or decoding is an inverse problem in which paper. The first goal is technical and is exemplified by the the physical variables are estimated from observed neural activity. population vector method applied to motor cortical activities Reconstruction is useful first in quantifying how much information during various reaching tasks (Georgopoulos et al. 1986, 1989; about the physical variables is present in the population and, second, Schwartz 1994) and the template matching method applied to in providing insight into how the brain might use distributed represen- disparity selective cells in the visual cortex (Lehky and Sejnowtations in solving related computational problems such as visual ob- ski 1990) and hippocampal place cells during rapid learning of ject recognition...
Averaged and decorrelated neural networks as a time-series predictor
Proceedings of the 12th IAPR International Conference on Pattern Recognition (Cat. No.94CH3440-5)
We study the effect of removing temporal structure in the prediction error. We observe that netwo... more We study the effect of removing temporal structure in the prediction error. We observe that networks which are not optimally trained, exhibit strong temporal structure in their prediction error, which can be eliminated using linear regression. This elimination improves performance significantly, but does not lead to the best performance which is achieved by training networks until they do not exhibit any such temporal structure. The improvement in performance of ensemble net averaging does not affect possible temporal structure of the error, thus averaging can be performed before or after temporal structure removal. We demonstrate these findings on the sunspot data set.
We propose a method for improving the performance of any network designed to predict the next val... more We propose a method for improving the performance of any network designed to predict the next value of a time series. We advocate analyzing the deviations of the network's predictions from the data in the training set. This can be carried out by a secondary network trained on the time series of these residuals. The combined system of the two networks is viewed as the new predictor. We demonstrate the simplicity and success of this method, by applying it to the sunspots data. The small corrections of the secondary network can be regarded as resulting from a Taylor expansion of a complex network which includes the combined system. We find that the complex network is more difficult to train and performs worse than the two-step procedure of the combined system.
We propose a method for improving the performance of any network designed to predict the next val... more We propose a method for improving the performance of any network designed to predict the next value of a time series. Vve advocate analyzing the deviations of the network's predictions from the data in the training set. This can be carried out by a secondary network trained on the time series of these residuals. The combined system of the two networks is viewed as the new predictor. We demonstrate the simplicity and success of this method, by applying it to the sunspots data. The small corrections of the secondary network can be regarded as resulting from a Taylor expansion of a complex network which includes the combined system. \\Te find that the complex network is more difficult to train and performs worse than the two-step procedure of the combined system.
Most theoretical investigations of large recurrent networks focus on the properties of the macros... more Most theoretical investigations of large recurrent networks focus on the properties of the macroscopic order parameters such as population averaged activities or average overlaps with memories. However, the statistics of the fluctuations in the local activities may be an important testing ground for comparison between models and observed cortical dynamics. We evaluated the neuronal correlation functions in a stochastic network comprising of excitatory and inhibitory populations. We show that when the network is in a stationary state, the cross-correlations are relatively weak, i.e., their amplitude relative to that of the auto-correlations are of order of 1/ N, N being the size of the interacting population. This holds except in the neighborhoods of bifurcations to nonstationary states. As a bifurcation point is approached the amplitude of the cross-correlations grows and becomes of order 1 and the decay timeconstant diverges. This behavior is analogous to the phenomenon of critical slowing down in systems at thermal equilibrium near a critical point. Near a Hopf bifurcation the cross-correlations exhibit damped oscillations.
Zhang, K., Ginzburg, I., McNaughton, B.L. & Sejnowski, T.J. Interpreting neuronal population activity by reconstruction: unified framework with application to hippocampal place cells. J. Neurophysiol. 79, 1017−1044
Physical variables such as the orientation of a line in the visual field or the location of the b... more Physical variables such as the orientation of a line in the visual field or the location of the body in space are coded as activity levels in populations of neurons. Reconstruction or decoding is an inverse problem in which the physical variables are estimated from observed neural activity. Reconstruction is useful first in quantifying how much information about the physical variables is present in the population and, second, in providing insight into how the brain might use distributed representations in solving related computational problems such as visual object recognition and spatial navigation. Two classes of reconstruction methods, namely, probabilistic or Bayesian methods and basis function methods, are discussed. They include important existing methods as special cases, such as population vector coding, optimal linear estimation, and template matching. As a representative example for the reconstruction problem, different methods were applied to multi-electrode spike train d...
We present a distributed model of the bronchial tree which simulates the global dynamic character... more We present a distributed model of the bronchial tree which simulates the global dynamic characteristics of the lung. Local mechanical characteristics of each airway are represented by RCL circuits and parameters of the electrical components are determined from local physiological data. The bronchi geometry is described by Weibel's symmetric model, the flow in each airway is assumed laminar and mixing effects at the bifurcations are neglected; the transpulmonary pressure is assumed to be sinusoidal. In simulations of quiet breathing the resistance to airflow is found to be dominant, the flow amplitude decreasing as breathing frequency increases, but remaining almost constant in all the generations. Simulations of ventilation through obstructed lungs show frequency dependence of the dynamic characteristics in very compliant lungs. The global resistance to airflow and the dynamic compliance of the bronchi decrease as the forced oscillation frequency increases in a pattern similar t...
We present a distributed model of the bronchial tree which simulates the global dynamic character... more We present a distributed model of the bronchial tree which simulates the global dynamic characteristics of the lung. Local mechanical characteristics of each airway are represented by RCL circuits and parameters of the electrical components are determined from local physiological data. The bronchi geometry is described by Weibel's symmetric model, the flow in each airway is assumed laminar and mixing effects at the bifurcations are neglected; the transpulmonary pressure is assumed to be sinusoidal. In simulations of quiet breathing the resistance to airflow is found to be dominant, the flow amplitude decreasing as breathing frequency increases, but remaining almost constant in all the generations. Simulations of ventilation through obstructed lungs show frequency dependence of the dynamic characteristics in very compliant lungs. The global resistance to airflow and the dynamic compliance of the bronchi decrease as the forced oscillation frequency increases in a pattern similar to in vivo measurements in diseased lungs. This may be an outcome of the RCL properties of the network rather than due to uneven distribution of mechanical properties of the lung.
such as the orientation of a line in the visual field or the location of Two main goals for recon... more such as the orientation of a line in the visual field or the location of Two main goals for reconstruction are approached in this the body in space are coded as activity levels in populations of paper. The first goal is technical and is exemplified by the neurons. Reconstruction or decoding is an inverse problem in which population vector method applied to motor cortical activities the physical variables are estimated from observed neural activity. during various reaching tasks (Georgopoulos et al. 1986, 1989; Reconstruction is useful first in quantifying how much information Schwartz 1994) and the template matching method applied to about the physical variables is present in the population and, second, in providing insight into how the brain might use distributed represen-disparity selective cells in the visual cortex (Lehky and Sejnowtations in solving related computational problems such as visual obski 1990) and hippocampal place cells during rapid learning of ject recognition and spatial navigation. Two classes of reconstruction place fields in a novel environment (Wilson and McNaughton methods, namely, probabilistic or Bayesian methods and basis func-1993). In these examples, reconstruction extracts information tion methods, are discussed. They include important existing methods from noisy neuronal population activity and transforms it to a as special cases, such as population vector coding, optimal linear more explicit and convenient representation of movement and estimation, and template matching. As a representative example for position. In this paper, various reconstruction methods that are the reconstruction problem, different methods were applied to multitheoretically optimal under different frameworks are considelectrode spike train data from hippocampal place cells in freely ered; the ultimate theoretical limits on the best achievable accumoving rats. The reconstruction accuracy of the trajectories of the racy for all possible methods also are derived. rats was compared for the different methods. Bayesian methods were especially accurate when a continuity constraint was enforced, and
Neural Information Processing Systems, Nov 29, 1993
Most theoretical investigations of large recurrent networks focus on the properties of the macros... more Most theoretical investigations of large recurrent networks focus on the properties of the macroscopic order parameters such as population averaged activities or average overlaps with memories. However, the statistics of the fluctuations in the local activities may be an important testing ground for comparison between models and observed cortical dynamics. We evaluated the neuronal correlation functions in a stochastic network comprising of excitatory and inhibitory populations. We show that when the network is in a stationary state, the cross-correlations are relatively weak, i.e., their amplitude relative to that of the auto-correlations are of order of 1/ N, N being the size of the interacting population. This holds except in the neighborhoods of bifurcations to nonstationary states. As a bifurcation point is approached the amplitude of the cross-correlations grows and becomes of order 1 and the decay timeconstant diverges. This behavior is analogous to the phenomenon of critical slowing down in systems at thermal equilibrium near a critical point. Near a Hopf bifurcation the cross-correlations exhibit damped oscillations.
Various Aspects Of Artificial And Biological Neural Networks
The field of neural networks is a wide and diverse field which spans a variety of interests, mode... more The field of neural networks is a wide and diverse field which spans a variety of interests, modelling methods, and potential uses. In my thesis, I discuss two different topics. The first is the use of feed-forward networks for time-series analysis and prediction. A new criterion for the quality of predictions of neural networks is presented. The criterion is based on the performance of networks for long term predictions, where one compares the distribution of the predicted values with that of the original data. Using this criterion as a weak constraint, one is able to choose the best networks from a population of trained networks that are similar in their short term predictions. This criterion may be also used to create a new predictor in which the various networks are weighted according to their long term behavior. In addition, a method for improving both short and long term predictions is proposed. Since quite often the consecutive errors of prediction are correlated in a systema...
Physical variables such as the orientation of a line in the visual field or the location of the b... more Physical variables such as the orientation of a line in the visual field or the location of the body in space are coded as activity levels in populations of neurons. Reconstruction or decoding is an inverse problem in which the physical variables are estimated from observed neural activity. Reconstruction is useful first in quantifying how much information about the physical variables is present in the population, and second, in providing insight into how the brain might use distributed representations in solving related computational problems such as visual object recognition and spatial navigation. Two classes of reconstruction methods, namely, probabilistic or Bayesian methods and basis function methods, are discussed. They include important existing methods as special cases, such as population vector coding, optimal linear estimation and template matching. As a representative example for the reconstruction problem, different methods were applied to multi-electrode spike train da...
One of the main experimental tools in probing the interactions between neurons has been the measu... more One of the main experimental tools in probing the interactions between neurons has been the measurement of the correlations in their activity. In general, however the interpretation of the observed correlations is difficult, since the correlation between a pair of neurons is influenced not only by the direct interaction between them but also by the dynamic state of the entire network to which they belong. Thus, a comparison between the observed correlations and the predictions from specific model networks is needed. In this paper we develop the theory of neuronal correlation functions in large networks comprising of several highly connected subpopulations, and obey stochastic dynamic rules. When the networks are in asynchronous states, the crosscorrelations are relatively weak, i.e., their amplitude relative to that of the auto-correlations is of order of 1=N , N being the size of the interacting populations. Using the weakness of the cross-correlations, general equations whi...
e-mail: The induction of rules by making queries is a dynamical process based on seeking informat... more e-mail: The induction of rules by making queries is a dynamical process based on seeking information. Experimenters typically look for one dominant strategy that is used by subjects, which may or may not agree with normative models of this psychological process. In this study we approach this problem from a different perspective, related to work in learning theory (see for example Baum 1991, Freund et al. 1995). Using information theory in a Bayesian framework, we estimated the information gained by queries when the task is to find a specific rule in a hypothesis space. Assuming that at each point subjects have a preferred working hypothesis, we considered several possible strategies, and determined the best one
such as the orientation of a line in the visual field or the location of Two main goals for recon... more such as the orientation of a line in the visual field or the location of Two main goals for reconstruction are approached in this the body in space are coded as activity levels in populations of neurons. Reconstruction or decoding is an inverse problem in which paper. The first goal is technical and is exemplified by the the physical variables are estimated from observed neural activity. population vector method applied to motor cortical activities Reconstruction is useful first in quantifying how much information during various reaching tasks (Georgopoulos et al. 1986, 1989; about the physical variables is present in the population and, second, Schwartz 1994) and the template matching method applied to in providing insight into how the brain might use distributed represen- disparity selective cells in the visual cortex (Lehky and Sejnowtations in solving related computational problems such as visual ob- ski 1990) and hippocampal place cells during rapid learning of ject recognition...
Averaged and decorrelated neural networks as a time-series predictor
Proceedings of the 12th IAPR International Conference on Pattern Recognition (Cat. No.94CH3440-5)
We study the effect of removing temporal structure in the prediction error. We observe that netwo... more We study the effect of removing temporal structure in the prediction error. We observe that networks which are not optimally trained, exhibit strong temporal structure in their prediction error, which can be eliminated using linear regression. This elimination improves performance significantly, but does not lead to the best performance which is achieved by training networks until they do not exhibit any such temporal structure. The improvement in performance of ensemble net averaging does not affect possible temporal structure of the error, thus averaging can be performed before or after temporal structure removal. We demonstrate these findings on the sunspot data set.
We propose a method for improving the performance of any network designed to predict the next val... more We propose a method for improving the performance of any network designed to predict the next value of a time series. We advocate analyzing the deviations of the network's predictions from the data in the training set. This can be carried out by a secondary network trained on the time series of these residuals. The combined system of the two networks is viewed as the new predictor. We demonstrate the simplicity and success of this method, by applying it to the sunspots data. The small corrections of the secondary network can be regarded as resulting from a Taylor expansion of a complex network which includes the combined system. We find that the complex network is more difficult to train and performs worse than the two-step procedure of the combined system.
We propose a method for improving the performance of any network designed to predict the next val... more We propose a method for improving the performance of any network designed to predict the next value of a time series. Vve advocate analyzing the deviations of the network's predictions from the data in the training set. This can be carried out by a secondary network trained on the time series of these residuals. The combined system of the two networks is viewed as the new predictor. We demonstrate the simplicity and success of this method, by applying it to the sunspots data. The small corrections of the secondary network can be regarded as resulting from a Taylor expansion of a complex network which includes the combined system. \\Te find that the complex network is more difficult to train and performs worse than the two-step procedure of the combined system.
Most theoretical investigations of large recurrent networks focus on the properties of the macros... more Most theoretical investigations of large recurrent networks focus on the properties of the macroscopic order parameters such as population averaged activities or average overlaps with memories. However, the statistics of the fluctuations in the local activities may be an important testing ground for comparison between models and observed cortical dynamics. We evaluated the neuronal correlation functions in a stochastic network comprising of excitatory and inhibitory populations. We show that when the network is in a stationary state, the cross-correlations are relatively weak, i.e., their amplitude relative to that of the auto-correlations are of order of 1/ N, N being the size of the interacting population. This holds except in the neighborhoods of bifurcations to nonstationary states. As a bifurcation point is approached the amplitude of the cross-correlations grows and becomes of order 1 and the decay timeconstant diverges. This behavior is analogous to the phenomenon of critical slowing down in systems at thermal equilibrium near a critical point. Near a Hopf bifurcation the cross-correlations exhibit damped oscillations.
Zhang, K., Ginzburg, I., McNaughton, B.L. & Sejnowski, T.J. Interpreting neuronal population activity by reconstruction: unified framework with application to hippocampal place cells. J. Neurophysiol. 79, 1017−1044
Physical variables such as the orientation of a line in the visual field or the location of the b... more Physical variables such as the orientation of a line in the visual field or the location of the body in space are coded as activity levels in populations of neurons. Reconstruction or decoding is an inverse problem in which the physical variables are estimated from observed neural activity. Reconstruction is useful first in quantifying how much information about the physical variables is present in the population and, second, in providing insight into how the brain might use distributed representations in solving related computational problems such as visual object recognition and spatial navigation. Two classes of reconstruction methods, namely, probabilistic or Bayesian methods and basis function methods, are discussed. They include important existing methods as special cases, such as population vector coding, optimal linear estimation, and template matching. As a representative example for the reconstruction problem, different methods were applied to multi-electrode spike train d...
We present a distributed model of the bronchial tree which simulates the global dynamic character... more We present a distributed model of the bronchial tree which simulates the global dynamic characteristics of the lung. Local mechanical characteristics of each airway are represented by RCL circuits and parameters of the electrical components are determined from local physiological data. The bronchi geometry is described by Weibel's symmetric model, the flow in each airway is assumed laminar and mixing effects at the bifurcations are neglected; the transpulmonary pressure is assumed to be sinusoidal. In simulations of quiet breathing the resistance to airflow is found to be dominant, the flow amplitude decreasing as breathing frequency increases, but remaining almost constant in all the generations. Simulations of ventilation through obstructed lungs show frequency dependence of the dynamic characteristics in very compliant lungs. The global resistance to airflow and the dynamic compliance of the bronchi decrease as the forced oscillation frequency increases in a pattern similar t...
We present a distributed model of the bronchial tree which simulates the global dynamic character... more We present a distributed model of the bronchial tree which simulates the global dynamic characteristics of the lung. Local mechanical characteristics of each airway are represented by RCL circuits and parameters of the electrical components are determined from local physiological data. The bronchi geometry is described by Weibel's symmetric model, the flow in each airway is assumed laminar and mixing effects at the bifurcations are neglected; the transpulmonary pressure is assumed to be sinusoidal. In simulations of quiet breathing the resistance to airflow is found to be dominant, the flow amplitude decreasing as breathing frequency increases, but remaining almost constant in all the generations. Simulations of ventilation through obstructed lungs show frequency dependence of the dynamic characteristics in very compliant lungs. The global resistance to airflow and the dynamic compliance of the bronchi decrease as the forced oscillation frequency increases in a pattern similar to in vivo measurements in diseased lungs. This may be an outcome of the RCL properties of the network rather than due to uneven distribution of mechanical properties of the lung.
such as the orientation of a line in the visual field or the location of Two main goals for recon... more such as the orientation of a line in the visual field or the location of Two main goals for reconstruction are approached in this the body in space are coded as activity levels in populations of paper. The first goal is technical and is exemplified by the neurons. Reconstruction or decoding is an inverse problem in which population vector method applied to motor cortical activities the physical variables are estimated from observed neural activity. during various reaching tasks (Georgopoulos et al. 1986, 1989; Reconstruction is useful first in quantifying how much information Schwartz 1994) and the template matching method applied to about the physical variables is present in the population and, second, in providing insight into how the brain might use distributed represen-disparity selective cells in the visual cortex (Lehky and Sejnowtations in solving related computational problems such as visual obski 1990) and hippocampal place cells during rapid learning of ject recognition and spatial navigation. Two classes of reconstruction place fields in a novel environment (Wilson and McNaughton methods, namely, probabilistic or Bayesian methods and basis func-1993). In these examples, reconstruction extracts information tion methods, are discussed. They include important existing methods from noisy neuronal population activity and transforms it to a as special cases, such as population vector coding, optimal linear more explicit and convenient representation of movement and estimation, and template matching. As a representative example for position. In this paper, various reconstruction methods that are the reconstruction problem, different methods were applied to multitheoretically optimal under different frameworks are considelectrode spike train data from hippocampal place cells in freely ered; the ultimate theoretical limits on the best achievable accumoving rats. The reconstruction accuracy of the trajectories of the racy for all possible methods also are derived. rats was compared for the different methods. Bayesian methods were especially accurate when a continuity constraint was enforced, and
Uploads
Papers by Iris Ginzburg