Error Loss Networks
Pengju Ren2021
Sign up for access to the world's latest research
checkGet notified about relevant papers
checkSave papers to use in your research
checkJoin the discussion with peers
checkTrack your impact
Abstract
A novel model called error loss network (ELN) is proposed to build an error loss function for supervised learning. The ELN is in structure similar to a radial basis function (RBF) neural network, but its input is an error sample and output is a loss corresponding to that error sample. That means the nonlinear input-output mapper of ELN creates an error loss function. The proposed ELN provides a unified model for a large class of error loss functions, which includes some information theoretic learning (ITL) loss functions as special cases. The activation function, weight parameters and network size of the ELN can be predetermined or learned from the error samples. On this basis, we propose a new machine learning paradigm where the learning process is divided into two stages: first, learning a loss function using an ELN; second, using the learned loss function to continue to perform the learning. Experimental results are presented to demonstrate the desirable performance of the new me...
Related papers
An Improved Radial Basis Function Networks in Networks Weights Adjustment for Training Real-World Nonlinear Datasets
IAES International Journal of Artificial Intelligence (IJ-AI), 2019
In neural networks, the accuracies of its networks are mainly relying on two important factors which are the centers and the networks weight. The gradient descent algorithm is a widely used weight adjustment algorithm in most of neural networks training algorithm. However, the method is known for its weakness for easily trap in local minima. It suffers from a random weight generated for the networks during initial stage of training at input layer to hidden layer networks. The performance of radial basis function networks (RBFN) has been improved from different perspectives, including centroid initialization problem to weight correction stage over the years. Unfortunately, the solution does not provide a good trade-off between quality and efficiency of the weight produces by the algorithm. To solve this problem, an improved gradient descent algorithm for finding initial weight and improve the overall networks weight is proposed. This improved version algorithm is incorporated into RB...
A minimum error neural network (MNN
Neural Networks, 1993
Abstraet--A minimum error neural network ( MNN) model is presented and applied to a network of the appropriate architecture. The associated one-pass learning rule involves the estimation of input densities. This is accomplished by utilizing local Gaussian functions. A major distinction between this network and other Gaussian based estimators is in the selection of covariance matrices. In MNN, every single local function has its own covariance matrix. The Gram-Schmidt orthogonalization process is used to obtain these matrices. In comparison with the well known probabilistic neural network ( PNN), the proposed network has shown improved performance.
CB3: An Adaptive Error Function for Backpropagation Training
Neural Processing Letters, 2006
Effective backpropagation training of multi-layer perceptrons depends on the incorporation of an appropriate error or objective function. Classification-based (CB) error functions are heuristic approaches that attempt to guide the network directly to correct pattern classification rather than using common error minimization heuristics, such as sum-squared error and cross-entropy, which do not explicitly minimize classification error. This work presents CB3, a novel CB approach that learns the error function to be used while training. This is accomplished by learning pattern confidence margins during training, which are used to dynamically set output target values for each training pattern. On eleven applications, CB3 significantly outperforms previous CB error functions, and also reduces average test error over conventional error metrics using 0-1 targets without weight decay by 1.8%, and by 1.3% over metrics with weight decay. CB3 also exhibits lower model variance and tighter mean confidence interval.
Reformulated radial basis neural networks trained by gradient descent
IEEE Transactions on Neural Networks, 1999
This paper presents an axiomatic approach for constructing radial basis function (RBF) neural networks. This approach results in a broad variety of admissible RBF models, including those employing Gaussian RBF's. The form of the RBF's is determined by a generator function. New RBF models can be developed according to the proposed approach by selecting generator functions other than exponential ones, which lead to Gaussian RBF's. This paper also proposes a supervised learning algorithm based on gradient descent for training reformulated RBF neural networks constructed using the proposed approach. A sensitivity analysis of the proposed algorithm relates the properties of RBF's with the convergence of gradient descent learning. Experiments involving a variety of reformulated RBF networks generated by linear and exponential generator functions indicate that gradient descent learning is simple, easily implementable, and produces RBF networks that perform considerably better than conventional RBF models trained by existing algorithms.
Neural network classification using error entropy minimization
Biological and Artificial Intelligence Environments, 2005
One way of using the entropy criteria in learning systems is to minimize the entropy of the error between two variables: typically, one is the output of the learning system and the other is the target. This framework has been used for regression. In this paper we show how to use the minimization of the entropy of the error for classification. The minimization of the entropy of the error implies a constant value for the errors. This, in general, does not imply that the value of the errors is zero. In regression, this problem is solved by making a shift of the final result such that it's average equals the average value of the desired target. We prove that, under mild conditions, this algorithm, when used in a classification problem, makes the error converge to zero and can thus be used in classification.
On the construction and training of reformulated radial basis function neural networks
IEEE Transactions on Neural Networks, 2003
This paper presents a systematic approach for constructing reformulated radial basis function (RBF) neural networks, which was developed to facilitate their training by supervised learning algorithms based on gradient descent. This approach reduces the construction of radial basis function models to the selection of admissible generator functions. The selection of generator functions relies on the concept of the blind spot, which is introduced in this paper. This paper also introduces a new family of reformulated radial basis function neural networks, which are referred to as cosine radial basis functions. Cosine radial basis functions are constructed by linear generator functions of a special form and their use as similarity measures in radial basis function models is justified by their geometric interpretation. A set of experiments on a variety of datasets indicate that cosine radial basis functions outperform considerably conventional radial basis function neural networks with Gaussian radial basis functions. Cosine radial basis functions are also strong competitors to existing reformulated radial basis function models trained by gradient descent and feedforward neural networks with sigmoid hidden units.
Special issue on machine learning for nonlinear processing
Neurocomputing, 2014
Radial basis function neural network learning with modified backpropagation algorithm
2014
Radial Basis Function Neural Network (RBFNN) is a class of Artificial Neural Network (ANN) widely used in science and engineering for classification problems with Backpropagation (BP) algorithm. However, major disadvantages of BP are due to the relatively slow convergence rate and always being trapped at the local minima. To overcome this problem, an improved Backpropagation (MBP) algorithm using modified cost function was developed to enhance RBFNN learning with discretized data to enhance the performance of classification accuracy and error rate convergence of the network. In RBFNN learning with Standard Backpropagation (SBP), there are many elements to be considered such as the number of input nodes, number of hidden nodes, number of output nodes, learning rate, bias rate, minimum error and activation functions. These parameters affect the speed of RBFNN learning. In this study, the proposed MBP algorithm was applied to RBFNN to enhance the learning process in terms of classifica...
A modified error function for the backpropagation algorithm
Neurocomputing, 2004
We have noted that the local minima problem in the backpropagation algorithm is usually caused by update disharmony between weights connected to the hidden layer and the output layer. To solve this problem, we propose a modiÿed error function. It can harmonize the update of weights connected to the hidden layer and those connected to the output layer by adding one term to the conventional error function. It can thus avoid the local minima problem caused by such disharmony. Simulations on a benchmark problem and a real classiÿcation task have been performed to test the validity of the modiÿed error function.
Error-Correcting Neural Network
ArXiv, 2019
Though deep learning has been applied successfully in many scenarios, malicious inputs with human-imperceptible perturbations can make it vulnerable in real applications. This paper proposes an error-correcting neural network (ECNN) that combines a set of binary classifiers to combat adversarial examples in the multi class classification problem. To build an ECNN, we propose to design a code matrix so that the minimum Hamming distance between any two rows (i.e., two codewords) and the minimum shared information distance between any two columns (i.e., two partitions of class labels) are simultaneously maximized. Maximizing row distances can increase the system fault tolerance while maximizing column distances helps increase the diversity between binary classifiers. We propose an end-to-end training method for our ECNN, which allows further improvement of the diversity between binary classifiers. The end-to-end training renders our proposed ECNN different from the traditional error-co...
Principle of Neural Network and Its Main Types: Review
2020
In this paper, an overview of the artificial neural networks is presented. Their main and popular types such as the multilayer feedforward neural network (MLFFNN), the recurrent neural network (RNN), and the radial basis function (RBF) are investigated. Furthermore, the main advantages and disadvantages of each type are included as well as the training process.
A New Method for Constructing Radial Basis Function Neural Networks
Proceedings on Intelligent Systems and Knowledge Engineering (ISKE2007), 2007
Ignoring the samples far away from the training samples, our study team gives a new norm-based derivative process of localized generalization error boundary. Enlightened by the above research, this paper proposes a new method to construct radial basis function neural networks, which minimizes the sum of training error and stochastic sensitivity. Experimental results show that the new method can lead to simple and better network architecture.
Fast neural network algorithm for solving classification tasks: Batch error back-propagation algorithm
2013 Proceedings of IEEE Southeastcon, 2013
Optimization in an Error Backpropagation NeuralNetwork Environment with a Performance Teston a Pattern Classification Problem
1998
Various techniques of optimizing the multiple class cross-entropy error function to train single hidden layer neural network classi ers with softmax output transfer functions are investigated on a real-world multispectral pixel-by-pixel classi cation problem that is of fundamental importance in remote sensing. These techniques include epoch-based and batchv ersions of backpropagation of gradient descent, PR-conjugate gradient and BFGS quasi-Newton errors. The method of choice depends upon the nature of the learning task and whether one wants to optimize learning for speed or generalization performance. It was found that, comparatively considered, gradient descent error backpropagation provided the best and most stable out-of-sample performance results across batch and epoch-based modes of operation. If the goal is to maximize learning speed and a sacri ce in generalisation is acceptable, then PR-conjugate gradient error backpropagation tends to be superior. If the training set is very large, stochastic epoch-based versions of local optimizers should be chosen utilizing a larger rather than a smaller epoch size to avoid inacceptable instabilities in the generalization results.
High performance learning by modified error backpropagation
Neural Computing & Applications, 1993
In using a neural network for an application, data representation and network structure are critical to performance. While most improvements to networks focus on these aspects, we have found that modification of the error function based on current performance can result in significant advantages. We consider here a multilayered network trained by the backpropagation error reduction rule. We also consider a specific task, namely that of direct recognition of handwriting patterns, without any feature extraction to optimise the representation used. We show that the relaxation of the definition of error improves the final performance and accelerates learning. Since the application used in this study has generic qualities, we believe that the results of this numerical experiment are pertinent to a wide class of applications.
An Effective Solution to Regression Problem by RBF Neuron Network
International Journal of Operations Research and Information Systems, 2015
Radial Basis Function (RBF) neuron network is being applied widely in multivariate function regression. However, selection of neuron number for hidden layer and definition of suitable centre in order to produce a good regression network are still open problems which have been researched by many people. This article proposes to apply grid equally space nodes as the centre of hidden layer. Then, the authors use k-nearest neighbour method to define the value of regression function at the center and an interpolation RBF network training algorithm with equally spaced nodes to train the network. The experiments show the outstanding efficiency of regression function when the training data has Gauss white noise.
Neural Network Principles and Applications
Digital Systems, 2018
Due to the recent trend of intelligent systems and their ability to adapt with varying conditions, deep learning becomes very attractive for many researchers. In general, neural network is used to implement different stages of processing systems based on learning algorithms by controlling their weights and biases. This chapter introduces the neural network concepts, with a description of major elements consisting of the network. It also describes different types of learning algorithms and activation functions with the examples. These concepts are detailed in standard applications. The chapter will be useful for undergraduate students and even for postgraduate students who have simple background on neural networks.
Loading Preview
Sorry, preview is currently unavailable. You can download the paper by clicking the button above.