Regression estimation with support vector learning machines

2001

2000

Support Vector Machines Regression (SVMR) is a regression technique which has been recently introduced by V. Vapnik and his collaborators (Vapnik, 1995; Vapnik, Golowich and Smola, 1996). In SVMR the goodness of fit is measured not by the usual quadratic loss function (the mean square error), but by a different loss function called Vapnik's - insensitive loss function, which is

Department of Computer Science, University of …, 2004

1999

In this report we show some consequences of the work done by Pontil et al. in 1]. In particular we show that in the same hypotheses of the theorem proved in their paper, the optimal approximating hyperplane f R found by SVM regression classi es the data. This means that y i f R (x i ) > 0 for points which live externally to the margin between the two classes or points which live internally to the margin but correctly classi ed by SVM classi cation. Moreover y i f R (x i ) < 0 for incorrectly classi ed points. Finally, the zero level curve of the optimal approximating hyperplane determined by SVMR and the optimal separating hyperplane determined by SVMC coincide.

Applied Soft Computing, 2020

We propose a novel convex loss function termed as 'ϵ-penalty loss function', to be used in Support Vector Regression (SVR) model. The proposed ϵ-penalty loss function is shown to be optimal for a more general noise distribution. The popular ϵ-insensitive loss function and the Laplace loss function are particular cases of the proposed loss function. Making the use of the proposed loss function, we have proposed two new Support Vector Regression models in this paper. The first model which we have termed with 'ϵ-Penalty Support Vector Regression' (ϵ-PSVR) model minimizes the proposed loss function with L 2-norm regularization. The second model minimizes the proposed loss function with L 1-Norm regularization and has been termed as 'L 1-Norm Penalty Support Vector Regression' (L 1-Norm PSVR) model. The proposed loss function can offer different rates of penalization inside and outside of the ϵ-tube. This strategy enables the proposed SVR models to use the full information of the training set which make them to generalize well. Further, the numerical results obtained from the experiments carried out on various artificial, benchmark datasets and financial time series datasets show that the proposed SVR models own better generalization ability than existing SVR models.

2013

Support Vector-based learning methods are an important part of Computational Intelligence techniques. Recent efforts have been dealing with the problem of learning from very large datasets. This paper reviews the most commonly used formulations of support vector machines for regression (SVRs) aiming to emphasize its usability on large-scale applications. We review the general concept of support vector machines (SVMs), address the state-of-the-art on training methods SVMs, and explain the fundamental principle of SVRs. The most common learning methods for SVRs are introduced and linear programming-based SVR formulations are explained emphasizing its suitability for large-scale learning. Finally, this paper also discusses some open problems and current trends.

2002

We discuss the relation betweenϵ-support vector regression (ϵ-SVR) and v-support vector regression (v-SVR). In particular, we focus on properties that are different from those of C-support vector classification (C-SVC) and v-support vector classification (v-SVC). We then discuss some issues that do not occur in the case of classification: the possible range of ϵ and the scaling of target values. A practical decomposition method for v-SVR is implemented, and computational experiments are conducted.

Machine Learning, 2002

In this paper, we elaborate on the well-known relationship between Gaussian Processes (GP) and Support Vector Machines (SVM) under some convex assumptions for the loss functions. This paper concentrates on the derivation of the evidence and error bar approximation for regression problems. An error bar formula is derived based on the -insensitive loss function. P (a(z)|D) = 1 P (D) P (D|a(X))P (a(X), a(z))da(X)

Neurocomputing, 2003

We develop an intuitive geometric framework for support vector regression (SVR). By examining when-tubes exist, we show that SVR can be regarded as a classiÿcation problem in the dual space. Hard and soft-tubes are constructed by separating the convex or reduced convex hulls, respectively, of the training data with the response variable shifted up and down by. A novel SVR model is proposed based on choosing the max-margin plane between the two shifted data sets. Maximizing the margin corresponds to shrinking the e ective-tube. In the proposed approach, the e ects of the choices of all parameters become clear geometrically. The kernelized model corresponds to separating the convex or reduced convex hulls in feature space. Generalization bounds for classiÿcation can be extended to characterize the generalization performance of the proposed approach. We propose a simple iterative nearest-point algorithm that can be directly applied to the reduced convex hull case in order to construct soft-tubes. Computational comparisons with other SVR formulations are also included.

1999

In this report we show some simple properties of SVM for regression. In particular we show that for close to zero, minimizing the norm of w is equivalent to maximizing the distance between the optimal approximating hyperplane solution of SVMR and the closest points in the data set. So, in this case, there exists a complete analogy between SVM for regression and classi cation, and the -tube plays the same role as the margin between classes. Moreover we show that for every the set of support vectors found by SVMR is linearly separable in the feature space and the optimal approximating hyperplane is a separator for this set. As a consequence, we show that for every regression problem there exists a classi cation problem which is linearly separable in the feature space. This is due to the fact that the solution of SVMR separates the set of support vectors in two classes: the support vectors living above and the one living below the optimal approximating hyperplane solution of SVMR. The position of the support vectors with respect to the hyperplane is given by the sign of ( i ? i ). Finally, we present a simple algorithm for obtaining a sparser representation of the optimal approximating hyperplane by using SVM for classi cation.

2004

Abstract Support vector regression (SVR) has been popular in the past decade, but it provides only an estimated target value instead of predictive probability intervals. Many work have addressed this issue but sometimes the SVR formula must be modified. This paper presents a rather simple and direct approach to construct such intervals. We assume that the conditional distribution of the target value depends on its input only through the predicted value, and propose to model this distribution by simple functions.

Computational Statistics, 2014

According to the Statistical Learning Theory, the support vectors represent the most informative data points and compress the information contained in training set. However, a basic problem in the standard support vector machine is that when the data is noisy, there exists no guaranteed scheme in support vector machines' formulation to dissuade the machine from learning noise. Therefore, the noise which is typically presents in financial time series data may be taken into account as support vectors. In turn, noisy support vectors are modeled into the estimated function. As such, the inclusion of noise in support vectors may lead to an over-fitting and in turn to a poor generalization. The standard support vector regression (SVR) is reformulated in this article in such a way that the large errors which correspond to noise are restricted by a new parameter E. The simulation and real world experiments indicate that the novel SVR machine meaningfully performs better than the standard SVR in terms of accuracy and precision especially where the data is noisy, but in expense of a longer computation time.

Perspectives in Neural Computing, 1998

A new algorithm for Support Vector regression is proposed. For a priori chosen , it automatically adjusts a exible tube of minimal radius to the data such that at most a fraction of the data points lie outside. The algorithm is analysed theoretically and experimentally.

ArXiv, 2020

The insensitive parameter in support vector regression determines the set of support vectors that greatly impacts the prediction. A data-driven approach is proposed to determine an approximate value for this insensitive parameter by minimizing a generalized loss function originating from the likelihood principle. This data-driven support vector regression also statistically standardizes samples using the scale of noises. Nonlinear and linear numerical simulations with three types of noises ($\epsilon$-Laplacian distribution, normal distribution, and uniform distribution), and in addition, five real benchmark data sets, are used to test the capacity of the proposed method. Based on all of the simulations and the five case studies, the proposed support vector regression using a working likelihood, data-driven insensitive parameter is superior and has lower computational costs.