An efficient and speeded-up tree for multi-class classification

2007

In this paper we propose a new learning architecture that we call Unbalanced Decision Tree (UDT), attempting to improve existing methods based on Directed Acyclic Graph (DAG) [1] and One-versus-All (OVA) [2] approaches to multi-class pattern classification tasks. Several standard techniques, namely One-versus-One (OVO) [3], OVA, and DAG, are compared against UDT by some benchmark datasets from the University of California, Irvine (UCI) repository of machine learning databases [4].

2010

We present an improved version of One-Against-All (OAA) method for multiclass SVM classification based on a decision tree approach. The proposed decision tree based OAA (DT-OAA) is aimed at increasing the classification speed of OAA by using posterior probability estimates of binary SVM outputs. DT-OAA decreases the average number of binary SVM tests required in testing phase to a greater extent when compared to OAA and other multiclass SVM methods. For a balanced multiclass dataset with K classes, under best situation, DT-OAA requires only (K + 1)/2 binary tests on an average as opposed to K binary tests in OAA; however, on imbalanced multiclass datasets we observed DT-OAA to be much faster with proper selection of order in which the binary SVMs are arranged in the decision tree. Computational comparisons on publicly available datasets indicate that the proposed method can achieve almost the same classification accuracy as that of OAA, but is much faster in decision making.

Information Sciences, 2013

One approach to multi-class classification consists in decomposing the original problem into a collection of binary classification tasks. The outputs of these binary classifiers are combined to produce a single prediction. Winner-takes-all, max-wins and tree voting schemes are the most popular methods for this purpose. However, tree schemes can deliver faster predictions because they need to evaluate less binary models. Despite previous conclusions reported in the literature, this paper shows that their performance depends on the organization of the tree scheme, i.e. the positions where each pairwise classifier is placed on the graph. Different metrics are studied for this purpose, proposing a new one that considers the precision and the complexity of each pairwise model, what makes the method to be classifier-dependent. The study is performed using Support Vector Machines (SVMs) as base classifiers, but it could be extended to other kind of binary classifiers. The proposed method, tested on benchmark data sets and on one real-world application, is able to improve the accuracy of other decomposition multi-class classifiers, producing even faster predictions.

In this paper a novel architecture of Support Vector Machine classifiers utilizing binary decision tree (SVMBDT) for solving multiclass problems is presented. The hierarchy of binary decision subtasks using SVMs is designed with clustering algorithm. For consistency between the clustering model and SVM the clustering model utilizes distance measures at the kernel space, not at the input space. The proposed SVM based Binary Decision Tree architecture takes advantage of both the efficient computation of the decision tree architecture and the high classification accuracy of SVMs. The performance of the proposed SVMBDT architecture was measured on a problem of recognition of handwritten digits and letters. The experiments were conducted with samples from MNIST, Pendigit, Optdigit and Statlog databases of segmented digits and letters. The results of the experiments indicate that maintaining comparable accuracy, SVM-BDT is faster to be trained than the other methods. Especially in classification, due to its Log complexity, it is much faster than the widely used multi-class SVM methods like “one-against-one” and “oneagainst- all” for multiclass problems. The experiments showed that this method becomes more favorable as the number of classes in the recognition problem increases.

In this paper, decision tree SVMs architecture is constructed to solve multi-class problems. To maintain high generalization ability, the optimal structure of decision tree is determined using statistical measures for obtaining class separability. The proposed optimal decision tree SVM (ODT-SVM) takes advantage of both the efficient computation of the decision tree architecture and the high classification accuracy of SVM. A robust non-parametric test is carried out for statistical comparison of proposed ODT-SVM with other classifiers over multiple data sets. Performance is evaluated in terms of classification accuracy and computation time. The statistical analysis on UCI repository datasets indicate that ten cross validation accuracy of our proposed framework is significantly better than widely used multi-class classifiers. Experimental results and statistical tests have shown that the proposed ODT-SVM is significantly better in comparison to conventional OvO and OAA in terms of both training and testing time.

A novel architecture of Support Vector Machine classifiers utilizing binary decision tree (SVMDTA) for solving multiclass problems is proposed in this paper. A clustering algorithm was used to determine the hierarchy of binary decision subtasks performed by the SVM binary classifiers. The applied clustering model utilizes Mahalanobis distance measures at the kernel space for better consistency with the used SVM kernel. The proposed SVM based Binary Decision Tree architecture takes advantage of both the efficient computation of the decision tree architecture and the high classification accuracy of SVMs. The performance of the proposed SVM-DTA was estimated on a problem of recognition of handwritten digits and letters. The experiments were conducted with samples from Pendigit and Statlog databases of segmented digits and letters. The results of the experiments indicate that the proposed method is faster to be trained than the other methods. Also, due to its Log complexity, the proposed SVM-DTA is much faster than the widely used multi-class SVM methods like “one-against-one” and “one-against-all”, maintaining comparable accuracy. The experiments also showed that this method becomes more favorable as the number of classes in the recognition problem increases.

This paper presents architecture of Support Vector Machine classifiers arranged in a binary tree structure for solving multi-class classification problems with increased efficiency. The proposed SVM based Binary Tree Architecture (SVM-BTA) takes advantage of both the efficient computation of the tree architecture and the high classification accuracy of SVMs. Clustering algorithm is used to convert the multi-class problem into binary tree, in which the binary decisions are made by the SVMs. The proposed clustering model utilizes distance measures at the kernel space, not at the input space. The performance of this method was measured on the problem of recognition of handwritten digits and letters using samples from MNIST, Pendigit, Optdigit and Statlog database of segmented digits and letters. The results of the experiments indicate that this method has much faster training and testing times than the widely used multi-class SVM methods like “one-against-one” and “one-against-all” while keeping comparable recognition rates. The experiments showed that this method becomes more favorable as the number of classes in the recognition problem increases.

International Journal of Applied Mathematics and Computer Science

We propose new methods for support vector machines using a tree architecture for multi-class classification. In each node of the tree, we select an appropriate binary classifier, using entropy and generalization error estimation, then group the examples into positive and negative classes based on the selected classifier, and train a new classifier for use in the classification phase. The proposed methods can work in time complexity between O(log2 N) and O(N), where N is the number of classes. We compare the performance of our methods with traditional techniques on the UCI machine learning repository using 10-fold cross-validation. The experimental results show that the methods are very useful for problems that need fast classification time or those with a large number of classes, since the proposed methods run much faster than the traditional techniques but still provide comparable accuracy.

Pattern Analysis and Applications, 2004

The support vector machine (SVM) has a high generalisation ability to solve binary classification problems, but its extension to multi-class problems is still an ongoing research issue. Among the existing multi-class SVM methods, the one-against-one method is one of the most suitable methods for practical use. This paper presents a new multi-class SVM method that can reduce the number of hyperplanes of the oneagainst-one method and thus it returns fewer support vectors. The proposed algorithm works as follows. While producing the boundary of a class, no more hyperplanes are constructed if the discriminating hyperplanes of neighbouring classes happen to separate the rest of the classes. We present a large number of experiments that show that the training time of the proposed method is the least among the existing multiclass SVM methods. The experimental results also show that the testing time of the proposed method is less than that of the one-against-one method because of the reduction of hyperplanes and support vectors. The proposed method can resolve unclassifiable regions and alleviate the over-fitting problem in a much better way than the one-against-one method by reducing the number of hyperplanes. We also present a direct acyclic graph SVM (DAGSVM) based testing methodology that improves the testing time of the DAGSVM method.

Engineering Journal, 2022

We propose a new technique for support vector machines (SVMs) in tree structures for multiclass classification. For each tree node, we select an appropriate binary classifier using data class centroids and their in-between distances, categorize the training examples into positive and negative groups of classes and train a new classifier. The proposed technique is fast-trained and can classify an output class data with a complexity between O(log 2 N) and O(N) where N is the number of classes. The 10-fold cross-validation experimental results show that the performance of our methods is comparable to that of traditional techniques and required less decision times. Our proposed technique is suitable for problems with a large number of classes due to its advantages of requiring less training time and computational complexity.