A geometric approach to support vector regression
Kristin Bennett2003, Neurocomputing
Sign up for access to the world's latest research
checkGet notified about relevant papers
checkSave papers to use in your research
checkJoin the discussion with peers
checkTrack your impact
Abstract
We develop an intuitive geometric framework for support vector regression (SVR). By examining when-tubes exist, we show that SVR can be regarded as a classiÿcation problem in the dual space. Hard and soft-tubes are constructed by separating the convex or reduced convex hulls, respectively, of the training data with the response variable shifted up and down by. A novel SVR model is proposed based on choosing the max-margin plane between the two shifted data sets. Maximizing the margin corresponds to shrinking the e ective-tube. In the proposed approach, the e ects of the choices of all parameters become clear geometrically. The kernelized model corresponds to separating the convex or reduced convex hulls in feature space. Generalization bounds for classiÿcation can be extended to characterize the generalization performance of the proposed approach. We propose a simple iterative nearest-point algorithm that can be directly applied to the reduced convex hull case in order to construct soft-tubes. Computational comparisons with other SVR formulations are also included.
Related papers
Duality, geometry, and support vector regression
2002
We develop an intuitive geometric framework for support vector regression (SVR). By examining when-tubes exist, we show that SVR can be regarded as a classification problem in the dual space. Hard and soft-tubes are constructed by separating the convex or reduced convex hulls respectively of the training data with the response variable shifted up and down by. A novel SVR model is proposed based on choosing the max-margin plane between the two shifted datasets. Maximizing the margin corresponds to shrinking the effective-tube. In the proposed approach the effects of the choices of all parameters become clear geometrically.
Shrinking the tube: a new support vector regression algorithm
1999
A new algorithm for Support Vector regression is described. For a priori chosen 1/, it automatically adjusts a flexible tube of minimal radius to the data such that at most a fraction 1/ of the data points lie outside. Moreover, it is shown how to use parametric tube shapes with non-constant radius. The algorithm is analysed theoretically and experimentally.
A Geometric Approach to Support Vector Machine (SVM) Classification
IEEE Transactions on Neural Networks, 2006
The geometric framework for the support vector machine (SVM) classification problem provides an intuitive ground for the understanding and the application of geometric optimization algorithms, leading to practical solutions of real world classification problems. In this work, the notion of "reduced convex hull" is employed and supported by a set of new theoretical results. These results allow existing geometric algorithms to be directly and practically applied to solve not only separable, but also nonseparable classification problems both accurately and efficiently. As a practical application of the new theoretical results, a known geometric algorithm has been employed and transformed accordingly to solve nonseparable problems successfully.
Duality and Geometry in SVM Classifiers
Proceedings of the Seventeenth International Conference on Machine Learning, 2000
We develop an intuitive geometric interpretation of the standard support vector machine (SVM) for classification of both linearly separable and inseparable data and provide a rigorous derivation of the concepts behind the geometry. For the separable case finding the maximum margin between the two sets is equivalent to finding the closest points in the smallest convex sets that contain each class (the convex hulls). We now extend this argument to the inseparable case by using a reduced convex hull reduced away from outliers. We prove that solving the reduced convex hull formulation is exactly equivalent to solving the standard inseparable SVM for appropriate choices of parameters. Some additional advantages of the new formulation are that the effect of the choice of parameters becomes geometrically clear and that the formulation may be solved by fast nearest point algorithms. By changing norms these arguments hold for both the standard 2-norm and 1-norm SVM.
On margin and support vector separability in Support Vector Machines for Regression
1999
In this report we show some simple properties of SVM for regression. In particular we show that for close to zero, minimizing the norm of w is equivalent to maximizing the distance between the optimal approximating hyperplane solution of SVMR and the closest points in the data set. So, in this case, there exists a complete analogy between SVM for regression and classi cation, and the -tube plays the same role as the margin between classes. Moreover we show that for every the set of support vectors found by SVMR is linearly separable in the feature space and the optimal approximating hyperplane is a separator for this set. As a consequence, we show that for every regression problem there exists a classi cation problem which is linearly separable in the feature space. This is due to the fact that the solution of SVMR separates the set of support vectors in two classes: the support vectors living above and the one living below the optimal approximating hyperplane solution of SVMR. The position of the support vectors with respect to the hyperplane is given by the sign of ( i ? i ). Finally, we present a simple algorithm for obtaining a sparser representation of the optimal approximating hyperplane by using SVM for classi cation.
Asymmetric -tube support vector regression
Computational Statistics & Data Analysis, 2014
Support Vector Machines for Regression: A Succinct Review of Large-Scale and Linear Programming Formulations
2013
Support Vector-based learning methods are an important part of Computational Intelligence techniques. Recent efforts have been dealing with the problem of learning from very large datasets. This paper reviews the most commonly used formulations of support vector machines for regression (SVRs) aiming to emphasize its usability on large-scale applications. We review the general concept of support vector machines (SVMs), address the state-of-the-art on training methods SVMs, and explain the fundamental principle of SVRs. The most common learning methods for SVRs are introduced and linear programming-based SVR formulations are explained emphasizing its suitability for large-scale learning. Finally, this paper also discusses some open problems and current trends.
Support Vector Machines, SVMs, and the
2014
Support Vector Machines, SVMs, and the Large Margin Nearest Neighbor algorithm, LMNN, are two very popular learning algorithms with quite different learning biases. In this paper we bring them into a unified view and show that they have a much stronger relation than what is commonly thought. We analyze SVMs from a metric learning perspective and cast them as a metric learning problem, a view which helps us uncover the relations of the two algorithms. We show that LMNN can be seen as learning a set of local SVM-like models in a quadratic space. Along the way and inspired by the metric-based interpretation of SVMs we derive a novel variant of SVMs,-SVM, to which LMNN is even more similar. We give a unified view of LMNN and the different SVM variants. Finally we provide some preliminary experiments on a number of benchmark datasets in which show that-SVM compares favorably both with respect to LMNN and SVM.
Support Vector Machine: A practical Implementation
Support Vector Machines is a very attractive and useful tool for classification and regression; however, since they rely on subtle and complex algebraic notions of optimization theory, lose their elegance and simplicity when implementation is concerned. It has been shown that the SVM solution, for the case of separate classes, corresponds to the minimum distance between the respective convex hulls. For the non-separable case, this is true for the Reduced Convex Hulls (RCH). In this paper a new geometric algorithm is presented, applied, and compared with other non-geometric algorithms for the non-separable case. In this we implemented SVM on PIMA Indian Diabetes dataset and achieved an accuracy of seventy four percent.
Feature selection for support vector regression via Kernel penalization
2010
his paper presents a novel feature selection approach (KP-SVR) that determines a non-linear regression function with minimal error and simultaneously minimizes the number of features by penalizing their use in the dual formulation of SVR. The approach optimizes the width of an anisotropic RBF Kernel using an iterative algorithm based on the gradient descent method, eliminating features that have low relevance for the regression model. Our approach presents an explicit stopping criterion, indicating clearly when eliminating further features begins to affect negatively the model's performance. Experiments with two real-world benchmark problems demonstrate that our approach accomplishes the best performance compared to well-known feature selection methods using consistently a small number of features.
Related papers
Properties of support vector machines for regression
1999
In this report we show that the -tube size in Support Vector Machine (SVM) for regression is 2 = p 1 + jjwjj 2 . By using this result we show that, in the case all the data points are inside the -tube, minimizing jjwjj 2 in SVM for regression is equivalent to maximizing the distance between the approximating hyperplane and the farest points in the training set. Moreover, in the most general setting in which the data points live also outside the -tube, we show that, for a xed value of , minimizing jjwjj 2 is equivalent to maximizing the sparsity of the representation of the optimal approximating hyperplane, that is equivalent to minimizing the number of coe cients di erent from zero in the expression of the optimal w. Then, the solution found by SVM for regression is a tradeo between sparsity of the representation and closeness to the data. We also include a complete derivation of SVM for regression in the case of linear approximation.
A Geometric Nearest Point Algorithm for the Efficient Solution of the SVM Classification Task
IEEE Transactions on Neural Networks, 2000
Geometric methods are very intuitive and provide a theoretically solid approach to many optimization problems. One such optimization task is the Support Vector Machine (SVM) classification, which has been the focus of intense theoretical as well as application oriented research in Machine Learning. In this work, the incorporation of recent results in Reduced Convex Hulls (RCH) to a Nearest Point Algorithm (NPA) leads to an elegant and efficient solution to the SVM classification task, with encouraging practical results to real world classification problems, i.e., linear or non-linear, separable or non-separable.
A Geometric Viewpoint of the Selection of the Regularization Parameter in Some Support Vector Machines
The regularization parameter of support vector machines is intended to improve their generalization performance. Since the feasible region of binary class support vector machines with finite dimensional feature space is a polytope, we note that classifiers at vertices of this unbounded polytope correspond to certain ranges of the regularization parameter. This reduces the search for a suitable regularization parameter to a search of (finite number of) vertices of this polytope. We propose an algorithm that identifies neighbouring vertices of a given vertex and thereby identifies the classifiers corresponding to the set of vertices of this polytope. A classifier can then be chosen from them based on a suitable test error criterion. We illustrate our results with an example which demonstrates that this path can be complicated. A portion of the path is sandwiched between two finite intervals of path, each generated by separate sets of vertices and edges.
Multivariate convex support vector regression with semidefinite programming
Knowledge-Based Systems, 2012
As one of important nonparametric regression method, support vector regression can achieve nonlinear capability by kernel trick. This paper discusses multivariate support vector regression when its regression function is restricted to be convex. This paper approximates this convex shape restriction with a series of linear matrix inequality constraints and transforms its training to a semidefinite programming problem, which is computationally tractable. Extensions to multivariate concave case, ' 2-norm Regularization, ' 1 and ' 2-norm loss functions, are also studied in this paper. Experimental results on both toy data sets and a real data set clearly show that, by exploiting this prior shape knowledge, this method can achieve better performance than the classical support vector regression.
A class of new Support Vector Regression models
Applied Soft Computing, 2020
We propose a novel convex loss function termed as 'ϵ-penalty loss function', to be used in Support Vector Regression (SVR) model. The proposed ϵ-penalty loss function is shown to be optimal for a more general noise distribution. The popular ϵ-insensitive loss function and the Laplace loss function are particular cases of the proposed loss function. Making the use of the proposed loss function, we have proposed two new Support Vector Regression models in this paper. The first model which we have termed with 'ϵ-Penalty Support Vector Regression' (ϵ-PSVR) model minimizes the proposed loss function with L 2-norm regularization. The second model minimizes the proposed loss function with L 1-Norm regularization and has been termed as 'L 1-Norm Penalty Support Vector Regression' (L 1-Norm PSVR) model. The proposed loss function can offer different rates of penalization inside and outside of the ϵ-tube. This strategy enables the proposed SVR models to use the full information of the training set which make them to generalize well. Further, the numerical results obtained from the experiments carried out on various artificial, benchmark datasets and financial time series datasets show that the proposed SVR models own better generalization ability than existing SVR models.
Support Vector Regression
Department of Computer Science, University of …, 2004
Support vector machines
Wiley Interdisciplinary Reviews: Computational Statistics, 2009
Support vector machines (SVMs) are a family of machine learning methods, originally introduced for the problem of classification and later generalized to various other situations. They are based on principles of statistical learning theory and convex optimization, and are currently used in various domains of application, including bioinformatics, text categorization, and computer vision. 2009 John Wiley & Sons, Inc. WIREs Comp Stat 2009 1 283-289 S upport vector machines (SVMs), introduced by
A novel SVM Geometric Algorithm based on Reduced Convex Hulls
18th International Conference on Pattern Recognition (ICPR'06), 2006
Geometric methods are very intuitive and provide a theoretically solid viewpoint to many optimization problems. SVM is a typical optimization task that has attracted a lot of attention over the recent years in many Pattern Recognition and Machine Learning tasks. In this work, we exploit recent results in Reduced Convex Hulls (RCH) and apply them to a Nearest Point Algorithm (NPA) leading to an elegant and efficient solution to the general (linear and nonlinear, separable and non-separable) SVM classification task.
Loading Preview
Sorry, preview is currently unavailable. You can download the paper by clicking the button above.