In developing algorithms that dynamically changes the structure and weights of ANN (Artificial Ne... more In developing algorithms that dynamically changes the structure and weights of ANN (Artificial Neural Networks), there must be a proper balance between network complexity and its generalization capability. SEPA addresses these issues using an encoding scheme where network weights and connections are encoded in matrices of real numbers. Network parameters are locally encoded and locally adapted with fitness evaluation consisting mainly of fast feed-forward operations. Experimental results in some well-known classification problems demonstrate SEPA’s high consistency performance in classification, fast convergence, and good optimality of structure.
Proceedings of the International Joint Conference on Neural Networks, 2003., 2003
ABSTRACT One important issue in developing dynamic algorithms that changes the structure and weig... more ABSTRACT One important issue in developing dynamic algorithms that changes the structure and weights of ANN (artificial neural networks) is how to achieve a proper balance between network complexity and its generalization capability. Typical hybrid approaches to address this problem incorporates EA strategy using a population of backpropagation networks. Since individuals undergo backpropagation networks. Since individuals undergo backpropagation training, this approach is inefficient and inherits the pitfalls of gradient learning. SEPA (structure evolution and parameter adaptation) addresses these issues using an encoding scheme where network weights and connections are encoded in matrices of real numbers. Network parameters are locally encoded and undergo local adaptation with fitness evaluation consisting mainly of fast feed-forward matrix operations that can be implemented in parallel or distributed environment. Experimental results show that SEPA's strategy produces optimal network structure with fast convergence, high consistency, and good generalization capability.
In neural network regression problems, often referred to as additive noise models, NIC (Network I... more In neural network regression problems, often referred to as additive noise models, NIC (Network Information Criterion) has been proposed as a general model selection criterion to determine the optimal network size with high generalization performance. Although NIC has been derived using asymptotic expansion, it has been pointed out that this technique cannot be applied under the assumption that a target function is in a family of assumed networks and the family is not minimal for representing the target true function, i.e. the overrealizable case, in which NIC reduces to the well-known AIC (Akaike Information Criterion) and others depending on a loss function. Because NIC is the unbiased estimator of generalization error based on training error, it is required to derive the expectations of errors for neural networks for such cases. This paper gives upper bounds of the expectations of training errors with respect to the distribution of training data, which we call the expected training error, for some types of networks under the squared error loss. In the overrealizable case, because the errors are determined by fitting properties of networks to noise components, including in data, the target set of data is taken to be a Gaussian noise sequence. For radial basis function networks and 3-layered neural networks with bell shaped activation function in the hidden layer, the expected training error is bounded above by sigma2* - 2nsigma2*logT/T, where sigma2* is the variance of noise, n is the number of basis functions or the number of hidden units and T is the number of data. Furthermore, for 3-layered neural networks with sigmoidal activation function in the hidden layer, we obtained the upper bound of sigma2* - O(log T/T) when n > 2. If the number of data is large enough, these bounds of the expected training error are smaller than sigma2* - N(n)sigma2*/T as evaluated in NIC, where N(n) is the number of all network parameters.
In order to analyze the stochastic property of multilayered perceptrons or other learning machine... more In order to analyze the stochastic property of multilayered perceptrons or other learning machines, we deal with simpler models and derive the asymptotic distribution of the least-squares estimators of their parameters. In the case where a model is unidentified, we show different results from traditional linear models: the well-known property of asymptotic normality never holds for the estimates of redundant parameters.
Evolving gradient-learning artificial neural networks (ANNs) using an evolutionary algorithm (EA)... more Evolving gradient-learning artificial neural networks (ANNs) using an evolutionary algorithm (EA) is a popular approach to address the local optima and design problems of ANN. The typical approach is to combine the strength of backpropagation (BP) in weight learning and EA's capability of searching the architecture space. However, the BP's "gradient descent" approach requires a highly computer-intensive operation that relatively restricts the search coverage of EA by compelling it to use a small population size. To address this problem, we utilized mutation-based genetic neural network (MGNN) to replace BP by using the mutation strategy of local adaptation of evolutionary programming (EP) to effect weight learning. The MGNN's mutation enables the network to dynamically evolve its structure and adapt its weights at the same time. Moreover, MGNN's EP-based encoding scheme allows for a flexible and less restricted formulation of the fitness function and makes fitness computation fast and efficient. This makes it feasible to use larger population sizes and allows MGNN to have a relatively wide search coverage of the architecture space. MGNN implements a stopping criterion where overfitness occurrences are monitored through "sliding-windows" to avoid premature learning and overlearning. Statistical analysis of its performance to some well-known classification problems demonstrate its good generalization capability. It also reveals that locally adapting or scheduling the strategy parameters embedded in each individual network may provide a proper balance between the local and global searching capabilities of MGNN.
A model of the luminosity-type (L-type) horizontal cell layer in the retina was constructed in te... more A model of the luminosity-type (L-type) horizontal cell layer in the retina was constructed in terms of the soma, axon terminal, and gap junction to understand the spatiotemporal properties. The soma and axon terminal were expressed by the Hodgkin-Huxley-type equations based on the relevant ionic currents. The L-type horizontal cell layer was simulated by connecting each L-type horizontal cell model with a linear gap junction. While the model response explained the experimental data well, the dynamics features were not fully realized without a feedback path from the horizontal cells to the photoreceptors.
... Lucy Hadden. Pages: 29 - 33. A computational model of retinogeniculate development. Gary L. H... more ... Lucy Hadden. Pages: 29 - 33. A computational model of retinogeniculate development. Gary L. Haith, David Heeger. Pages: 35 - 40. ... Pages: 47 - 51. Encoding context in spatial navigation: one role of dentate gyrus. Karl Kilborn, Gary Lynch, Richard Granger. Pages: 53 - 58. ...
ABSTRACT Neuroinformatics is a new discipline which combines neuroscience with information techno... more ABSTRACT Neuroinformatics is a new discipline which combines neuroscience with information technology. The Japan-Node of INCF was established at NIJC of RIKEN Brain Science Institute to address the task of integrating outstanding neuroscience researches in Japan. Each platform subcommittee from selected research areas develops a platform on the base-platform XooNIps. NIJC operates the J-Node portal to make platform resources open accessible in public. We introduce our concepts and the scheme of J-Node including nine platforms.
In developing algorithms that dynamically changes the structure and weights of ANN (Artificial Ne... more In developing algorithms that dynamically changes the structure and weights of ANN (Artificial Neural Networks), there must be a proper balance between network complexity and its generalization capability. SEPA addresses these issues using an encoding scheme where network weights and connections are encoded in matrices of real numbers. Network parameters are locally encoded and locally adapted with fitness evaluation consisting mainly of fast feed-forward operations. Experimental results in some well-known classification problems demonstrate SEPA’s high consistency performance in classification, fast convergence, and good optimality of structure.
Proceedings of the International Joint Conference on Neural Networks, 2003., 2003
ABSTRACT One important issue in developing dynamic algorithms that changes the structure and weig... more ABSTRACT One important issue in developing dynamic algorithms that changes the structure and weights of ANN (artificial neural networks) is how to achieve a proper balance between network complexity and its generalization capability. Typical hybrid approaches to address this problem incorporates EA strategy using a population of backpropagation networks. Since individuals undergo backpropagation networks. Since individuals undergo backpropagation training, this approach is inefficient and inherits the pitfalls of gradient learning. SEPA (structure evolution and parameter adaptation) addresses these issues using an encoding scheme where network weights and connections are encoded in matrices of real numbers. Network parameters are locally encoded and undergo local adaptation with fitness evaluation consisting mainly of fast feed-forward matrix operations that can be implemented in parallel or distributed environment. Experimental results show that SEPA's strategy produces optimal network structure with fast convergence, high consistency, and good generalization capability.
In neural network regression problems, often referred to as additive noise models, NIC (Network I... more In neural network regression problems, often referred to as additive noise models, NIC (Network Information Criterion) has been proposed as a general model selection criterion to determine the optimal network size with high generalization performance. Although NIC has been derived using asymptotic expansion, it has been pointed out that this technique cannot be applied under the assumption that a target function is in a family of assumed networks and the family is not minimal for representing the target true function, i.e. the overrealizable case, in which NIC reduces to the well-known AIC (Akaike Information Criterion) and others depending on a loss function. Because NIC is the unbiased estimator of generalization error based on training error, it is required to derive the expectations of errors for neural networks for such cases. This paper gives upper bounds of the expectations of training errors with respect to the distribution of training data, which we call the expected training error, for some types of networks under the squared error loss. In the overrealizable case, because the errors are determined by fitting properties of networks to noise components, including in data, the target set of data is taken to be a Gaussian noise sequence. For radial basis function networks and 3-layered neural networks with bell shaped activation function in the hidden layer, the expected training error is bounded above by sigma2* - 2nsigma2*logT/T, where sigma2* is the variance of noise, n is the number of basis functions or the number of hidden units and T is the number of data. Furthermore, for 3-layered neural networks with sigmoidal activation function in the hidden layer, we obtained the upper bound of sigma2* - O(log T/T) when n > 2. If the number of data is large enough, these bounds of the expected training error are smaller than sigma2* - N(n)sigma2*/T as evaluated in NIC, where N(n) is the number of all network parameters.
In order to analyze the stochastic property of multilayered perceptrons or other learning machine... more In order to analyze the stochastic property of multilayered perceptrons or other learning machines, we deal with simpler models and derive the asymptotic distribution of the least-squares estimators of their parameters. In the case where a model is unidentified, we show different results from traditional linear models: the well-known property of asymptotic normality never holds for the estimates of redundant parameters.
Evolving gradient-learning artificial neural networks (ANNs) using an evolutionary algorithm (EA)... more Evolving gradient-learning artificial neural networks (ANNs) using an evolutionary algorithm (EA) is a popular approach to address the local optima and design problems of ANN. The typical approach is to combine the strength of backpropagation (BP) in weight learning and EA's capability of searching the architecture space. However, the BP's "gradient descent" approach requires a highly computer-intensive operation that relatively restricts the search coverage of EA by compelling it to use a small population size. To address this problem, we utilized mutation-based genetic neural network (MGNN) to replace BP by using the mutation strategy of local adaptation of evolutionary programming (EP) to effect weight learning. The MGNN's mutation enables the network to dynamically evolve its structure and adapt its weights at the same time. Moreover, MGNN's EP-based encoding scheme allows for a flexible and less restricted formulation of the fitness function and makes fitness computation fast and efficient. This makes it feasible to use larger population sizes and allows MGNN to have a relatively wide search coverage of the architecture space. MGNN implements a stopping criterion where overfitness occurrences are monitored through "sliding-windows" to avoid premature learning and overlearning. Statistical analysis of its performance to some well-known classification problems demonstrate its good generalization capability. It also reveals that locally adapting or scheduling the strategy parameters embedded in each individual network may provide a proper balance between the local and global searching capabilities of MGNN.
A model of the luminosity-type (L-type) horizontal cell layer in the retina was constructed in te... more A model of the luminosity-type (L-type) horizontal cell layer in the retina was constructed in terms of the soma, axon terminal, and gap junction to understand the spatiotemporal properties. The soma and axon terminal were expressed by the Hodgkin-Huxley-type equations based on the relevant ionic currents. The L-type horizontal cell layer was simulated by connecting each L-type horizontal cell model with a linear gap junction. While the model response explained the experimental data well, the dynamics features were not fully realized without a feedback path from the horizontal cells to the photoreceptors.
... Lucy Hadden. Pages: 29 - 33. A computational model of retinogeniculate development. Gary L. H... more ... Lucy Hadden. Pages: 29 - 33. A computational model of retinogeniculate development. Gary L. Haith, David Heeger. Pages: 35 - 40. ... Pages: 47 - 51. Encoding context in spatial navigation: one role of dentate gyrus. Karl Kilborn, Gary Lynch, Richard Granger. Pages: 53 - 58. ...
ABSTRACT Neuroinformatics is a new discipline which combines neuroscience with information techno... more ABSTRACT Neuroinformatics is a new discipline which combines neuroscience with information technology. The Japan-Node of INCF was established at NIJC of RIKEN Brain Science Institute to address the task of integrating outstanding neuroscience researches in Japan. Each platform subcommittee from selected research areas develops a platform on the base-platform XooNIps. NIJC operates the J-Node portal to make platform resources open accessible in public. We introduce our concepts and the scheme of J-Node including nine platforms.
Uploads
Papers by Shiro Usui