Deep Support Vector Machines for Regression Problems
Marco A. Wiering
Lambert Schomaker
Arnold Meijster2013
Sign up for access to the world's latest research
checkGet notified about relevant papers
checkSave papers to use in your research
checkJoin the discussion with peers
checkTrack your impact
Abstract
In this paper we describe a novel extension of the support vector machine, called the deep support vector machine (DSVM). The original SVM has a single layer with kernel functions and is therefore a shallow model. The DSVM can use an arbitrary number of layers, in which lower-level layers contain support vector machines that learn to extract relevant features from the input patterns or from the extracted features of one layer below. The highest level SVM performs the actual prediction using the highest-level extracted features as inputs. The system is trained by a simple gradient ascent learning rule on a min-max formulation of the optimization problem. A two-layer DSVM is compared to the regular SVM on ten regression datasets and the results show that the DSVM outperforms the SVM.
Figures (1)
Related papers
Deep Support Vector Classification and Regression
2019
Support Vector Machines, SVM, are one of the most popular machine learning models for supervised problems and have proved to achieve great performance in a wide broad of predicting tasks. However, they can suffer from scalability issues when working with large sample sizes, a common situation in the big data era. On the other hand, Deep Neural Networks (DNNs) can handle large datasets with greater ease and in this paper we propose Deep SVM models that combine the highly non-linear feature processing of DNNs with SVM loss functions. As we will show, these models can achieve performances similar to those of standard SVM while having a greater sample scalability.
Deep support vector neural networks
Integrated Computer-Aided Engineering, 2020
Kernel based Support Vector Machines, SVM, one of the most popular machine learning models, usually achieve top performances in two-class classification and regression problems. However, their training cost is at least quadratic on sample size, making them thus unsuitable for large sample problems. However, Deep Neural Networks (DNNs), with a cost linear on sample size, are able to solve big data problems relatively easily. In this work we propose to combine the advanced representations that DNNs can achieve in their last hidden layers with the hinge and ϵ insensitive losses that are used in two-class SVM classification and regression. We can thus have much better scalability while achieving performances comparable to those of SVMs. Moreover, we will also show that the resulting Deep SVM models are competitive with standard DNNs in two-class classification problems but have an edge in regression ones.
The Neural Support Vector Machine
This paper describes a new machine learning algorithm for regression and dimensionality reduction tasks. The Neural Support Vector Machine (NSVM) is a hybrid learning algorithm consisting of neural networks and support vector machines (SVMs). The output of the NSVM is given by SVMs that take a central feature layer as their input. The feature-layer representation is the output of a number of neural networks that are trained to minimize the dual objectives of the SVMs. Because the NSVM uses a shared feature layer, the learning architecture is able to handle multiple outputs and therefore it can also be used as a dimensionality reduction method. The results on 7 regression datasets show that the NSVM in general outperforms a standard SVM and a multi-layer perceptron. Furthermore, experiments on eye images show that the NSVM autoencoder outperforms state-of-the-art dimensionality reduction methods.
Support Vector Machines for Regression: A Succinct Review of Large-Scale and Linear Programming Formulations
2013
Support Vector-based learning methods are an important part of Computational Intelligence techniques. Recent efforts have been dealing with the problem of learning from very large datasets. This paper reviews the most commonly used formulations of support vector machines for regression (SVRs) aiming to emphasize its usability on large-scale applications. We review the general concept of support vector machines (SVMs), address the state-of-the-art on training methods SVMs, and explain the fundamental principle of SVRs. The most common learning methods for SVRs are introduced and linear programming-based SVR formulations are explained emphasizing its suitability for large-scale learning. Finally, this paper also discusses some open problems and current trends.
Image Classification using Deep Learning with Support Vector Machines
Convolutional Neural Networks (CNNs) are a subset of Supervised Learning class of algorithms that are very similar to regular Neural Networks and aim to find an optimal predictive model that assigns the input variable to the correct label. In contrast to the Multilayer Perceptron Architecture (MLP) that uses fully connected network layers, a CNN does not need to provide information of the entire feature space to all hidden layer nodes, but instead it breaks the input matrix into regions and then connects each region to a single hidden node. With this regional breakdown and assignment of small local groups of features to different hidden nodes, CNNs are performing very well for image recognition tasks. On the other hand, a Support Vector Machine classifier tries to separate the data into K classes by maximizing the distance between the differently labeled data. If the data are not linearly separable, then by using an appropriate kernel function we can map the data into a higher dimension where they happen to be linearly separable and we find the linear boundary there. Finally, we transform that linear boundary back to the original lower dimensions and we get a non-linear separator. In this project we are going to replace the standard sigmoid activation function of the penultimate layer of the network with a linear Support Vector Machine classifier and investigate performance differences. We are going to implement the standard CNN architecture as benchmark model and see how it compares with a Deep Learning SVC so that we choose the best model to implement the final solution.
Deep recurrent support vector machine for online regression
2017 International Artificial Intelligence and Data Processing Symposium (IDAP), 2017
This paper introduces a novel deep recurrent support vector regressor (DRSVR) model for online regression. DRSVR model is constructed by a state equation followed by an output construction. The inner layer is actually a least squares support vector regressor (LS-SVR) of the states with an adaptive kernel function. In addition, an infinite impulse response (UR) filter is adopted in the model. LS-SVR and UR filter together constitute an intermediate layer which performs the recursive state update. Each internal state has a recurrency which is a function of the observed input-output data and the previous states. Hence, internal states track the temporal dependencies in the feature space. The outer layer is a linear combination of the states. The model parameters, including the Gaussian kernel width parameter, are updated simultaneously, that provides the model to capture the time-varying dynamics of the data quickly. Parameters are adaptively tuned using error-square minimization via c...
Support vector regression machines
Advances in Neural Information Processing Systems-9, 1997
Support Vector Regression Machines Harris Drucker* Chris JC Burges** Linda Kaufman** Alex Smola** Vladimir Vapnik+ * Bell Labs and Monmouth University Department of Electronic Engineering West Long Branch, NJ 07764 ** Bell ...
Pattern Based Multivariable Regression using Deep Learning (PBMR-DP)
arXiv (Cornell University), 2022
We propose a deep learning methodology for multivariable regression that is based on pattern recognition that triggers fast learning over sensor data. We used a conversion of sensors-to-image which enables us to take advantage of Computer Vision architectures and training processes. In addition to this data preparation methodology, we explore the use of state-of-the-art architectures to generate regression outputs to predict agricultural crop continuous yield information. Finally, we compare with some of the top models reported in MLCAS2021. We found that using a straightforward training process, we were able to accomplish an MAE of 4.394, RMSE of 5.945, and R 2 of 0.861.
Related papers
A neural support vector machine
Neural Networks, 2010
This paper describes a new machine learning algorithm for regression and dimensionality reduction tasks. The Neural Support Vector Machine (NSVM) is a hybrid learning algorithm consisting of neural networks and support vector machines (SVMs). The output of the NSVM is given by SVMs that take a central feature layer as their input. The feature-layer representation is the output of a number of neural networks that are trained to minimize the dual objectives of the SVMs. Because the NSVM uses a shared feature layer, the learning architecture is able to handle multiple outputs and therefore it can also be used as a dimensionality reduction method. The results on 7 regression datasets show that the NSVM in general outperforms a standard SVM and a multi-layer perceptron. Furthermore, experiments on eye images show that the NSVM autoencoder outperforms state-of-the-art dimensionality reduction methods.
Deep Learning using Linear Support Vector Machines
Recently, fully-connected and convolutional neural networks have been trained to achieve state-of-the-art performance on a wide variety of tasks such as speech recognition, image classification, natural language processing , and bioinformatics. For classification tasks, most of these " deep learning " models employ the softmax activation function for prediction and minimize cross-entropy loss. In this paper, we demonstrate a small but consistent advantage of replacing the soft-max layer with a linear support vector machine. Learning minimizes a margin-based loss instead of the cross-entropy loss. While there have been various combinations of neu-ral nets and SVMs in prior art, our results using L2-SVMs show that by simply replacing softmax with linear SVMs gives significant gains on popular deep learning datasets MNIST, CIFAR-10, and the ICML 2013 Representation Learning Workshop's face expression recognition challenge.
Deep Features for Training Support Vector Machines
Journal of Imaging, 2021
Features play a crucial role in computer vision. Initially designed to detect salient elements by means of handcrafted algorithms, features now are often learned using different layers in convolutional neural networks (CNNs). This paper develops a generic computer vision system based on features extracted from trained CNNs. Multiple learned features are combined into a single structure to work on different image classification tasks. The proposed system was derived by testing several approaches for extracting features from the inner layers of CNNs and using them as inputs to support vector machines that are then combined by sum rule. Several dimensionality reduction techniques were tested for reducing the high dimensionality of the inner layers so that they can work with SVMs. The empirically derived generic vision system based on applying a discrete cosine transform (DCT) separately to each channel is shown to significantly boost the performance of standard CNNs across a large and ...
Support Feature Machines: Support vectors are not enough
The 2010 International Joint Conference on Neural Networks (IJCNN), 2010
Support Vector Machines (SVMs) with various kernels have played dominant role in machine learning for many years, finding numerous applications. Although they have many attractive features interpretation of their solutions is quite difficult, the use of a single kernel type may not be appropriate in all areas of the input space, convergence problems for some kernels are not uncommon, the standard quadratic programming solution has O(m 3 ) time and O(m 2 ) space complexity for m training patterns. Kernel methods work because they implicitly provide new, useful features. Such features, derived from various kernels and other vector transformations, may be used directly in any machine learning algorithm, facilitating multiresolution, heterogeneous models of data. Therefore Support Feature Machines (SFM) based on linear models in the extended feature spaces, enabling control over selection of support features, give at least as good results as any kernel-based SVMs, removing all problems related to interpretation, scaling and convergence. This is demonstrated for a number of benchmark datasets analyzed with linear discrimination, SVM, decision trees and nearest neighbor methods.
A Deep Level Understanding of Support Vector Machine with Practical Implementation in Scikit Learn
Support Vector Machine, 2019
A Support Vector Machine (SVM) is a discriminative classifier that can be used for both classification and regression problems. The goal of SVM is to identify an optimal separating hyperplane which maximizes the margin between different classes of the training data. In other words, given labeled training data (supervised learning), the algorithm outputs an optimal hyperplane which categorizes new examples to create the largest possible distance to reduce an upper bound. Supports Vectors are simply the coordinates of data points that are nearest to the optimal separating hyperplane provide the most useful information for SVM classification. In addition, an appropriate kernel function is used to transform the data into a high-dimension to use linear discriminate functions.
Properties of support vector machines for regression
1999
In this report we show that the -tube size in Support Vector Machine (SVM) for regression is 2 = p 1 + jjwjj 2 . By using this result we show that, in the case all the data points are inside the -tube, minimizing jjwjj 2 in SVM for regression is equivalent to maximizing the distance between the approximating hyperplane and the farest points in the training set. Moreover, in the most general setting in which the data points live also outside the -tube, we show that, for a xed value of , minimizing jjwjj 2 is equivalent to maximizing the sparsity of the representation of the optimal approximating hyperplane, that is equivalent to minimizing the number of coe cients di erent from zero in the expression of the optimal w. Then, the solution found by SVM for regression is a tradeo between sparsity of the representation and closeness to the data. We also include a complete derivation of SVM for regression in the case of linear approximation.
Loading Preview
Sorry, preview is currently unavailable. You can download the paper by clicking the button above.