Most of the current metric learning methods are proposed for point-to-point distance (PPD) based ... more Most of the current metric learning methods are proposed for point-to-point distance (PPD) based classification. In many computer vision tasks, however, we need to measure the point-to-set distance (PSD) and even set-to-set distance (SSD) for classification. In this paper, we extend the PPD based Mahalanobis distance metric learning to PSD and SSD based ones, namely point-to-set distance metric learning (PSDML) and set-to-set distance metric learning (SSDML), and solve them under a unified optimization framework. First, we generate positive and negative sample pairs by computing the PSD and SSD between training samples. Then, we characterize each sample pair by its covariance matrix, and propose a covariance kernel based discriminative function. Finally, we tackle the PSDML and SSDML problems by using standard support vector machine solvers, making the metric learning very efficient for multiclass visual classification tasks. Experiments on gender classification, digit recognition, object categorization and face recognition show that the proposed metric learning methods can effectively enhance the performance of PSD and SSD based classification.
With the rapid development of digital imaging and communication technologies, image set based fac... more With the rapid development of digital imaging and communication technologies, image set based face recognition (ISFR) is becoming increasingly important. One key issue of ISFR is how to effectively and efficiently represent the query face image set by using the gallery face image sets. The set-to-set distance based methods ignore the relationship between gallery sets, while representing the query set images individually over the gallery sets ignores the correlation between query set images. In this paper, we propose a novel image set based collaborative representation and classification method for ISFR. By modeling the query set as a convex or regularized hull, we represent this hull collaboratively over all the gallery sets. With the resolved representation coefficients, the distance between the query set and each gallery set can then be calculated for classification. The proposed model naturally and effectively extends the image based collaborative representation to an image set based one, and our extensive experiments on benchmark ISFR databases show the superiority of the proposed method to state-of-the-art ISFR methods under different set sizes in terms of both recognition rate and efficiency.
a b s t r a c t 22 Due to good performance in classification and regression, support vector machi... more a b s t r a c t 22 Due to good performance in classification and regression, support vector machines have attracted much 23 attention and become one of the most popular learning machines in last decade. As a black box, the sup-24 port vector machine is difficult for users' understanding and explanation. In many application domains 25 including medical diagnosis or credit scoring, understandability and interpretability are very important 26 for the practicability of the learned models. To improve the comprehensibility of SVMs, we propose a rule 27 extraction technique from support vector machines via analyzing the distribution of samples. We define 28 the consistent region of samples in terms of classification boundary, and form a consistent region cover-29 ing of the sample space. Then a covering reduction algorithm is developed for extracting compact repre-30 sentation of classes, thus a minimal set of decision rules is derived. Experiment analysis shows that the 31 extracted models perform well in comparison with decision tree algorithms and other support vector 32 machine rule extraction methods.
In this paper, a novel regularized nea method is proposed for image sets based face modeling an i... more In this paper, a novel regularized nea method is proposed for image sets based face modeling an image set as a regularized affine regularized nearest points (RNP), one on each are automatically determined by an efficient ite between-set distance of RNP is then defined by the distance between the RNPs and the structu Compared with the recently developed spar nearest points (SANP) method, RNP has formulation, less variables and lower time com experiments on benchmark databases (e.g., Ho Mobo and YouTube databases) clearly show th RNP consistently outperforms state-of-the-art accuracy and efficiency.
Improving accuracy, robustness and understandability is the objective of classification modeling.... more Improving accuracy, robustness and understandability is the objective of classification modeling. Regarding instability and performance limitation of existing rule learning techniques, we introduce an ensemble classifier based on randomized neighborhood reduction and neighborhood covering reduction. A set of reducts are obtained with randomized attribute reduction. A collection of rule sets are derived from the reducts based on neighborhood covering reduction. And then the classification result is output by combining the classification decision of different rule sets. The experiment result shows that the proposed technique is better than or equal to other classifiers, and is more stable when deals with no isy data.
Bagging is a simple and effective technique for generating an ensemble of classifiers. It is foun... more Bagging is a simple and effective technique for generating an ensemble of classifiers. It is found there are a lot of redundant base classifiers in the original Bagging. We design a pruning approach to bagging for improving its generalization power. The proposed technique introduces the margin distribution based classification loss as the optimization objective and minimizes the loss on training samples, which leads to an optimal margin distribution. Meanwhile, in order to derive a sparse ensemble, l 1 regularization is introduced to control the size of ensembles. By this way, we can obtain a sparse weight vector of base classifiers. Then we rank the base classifiers with respect to their weights and combine the base classifiers with large weights. We call this technique MArgin Distribution base Bagging pruning (MAD-Bagging). Simple voting and weighted voting are tried to combine the outputs of selected base classifiers. The performance of this pruned ensemble is evaluated with several UCI benchmark tasks, where base classifiers are trained with SVM, CART, and the nearest neighbor (1NN) rule, respectively. The results show that margin distribution based CART pruned Bagging can significantly improve classification accuracies. However, SVM and 1NN pruned Bagging improve little compared with single classifiers.
Attribute reduction is one of the most meaningful research topics in the existing fuzzy rough set... more Attribute reduction is one of the most meaningful research topics in the existing fuzzy rough sets, and the approach of discernibility matrix is the mathematical foundation of computing reducts. When computing reducts with discernibility matrix, we find that only the minimal elements in a discernibility matrix are sufficient and necessary. This fact motivates our idea in this paper to develop a novel algorithm to find reducts that are based on the minimal elements in the discernibility matrix. Relative discernibility relations of conditional attributes are defined and minimal elements in the fuzzy discernibility matrix are characterized by the relative discernibility relations. Then, the algorithms to compute minimal elements and reducts are developed in the framework of fuzzy rough sets. Experimental comparison shows that the proposed algorithms are effective.
Small sample size is one of the most challenging problems in face recognition due to the difficul... more Small sample size is one of the most challenging problems in face recognition due to the difficulty of sample collection in many real-world applications. By representing the query sample as a linear combination of training samples from all classes, the so-called collaborative representation based classification (CRC) shows very effective face recognition performance with low computational cost. However, the recognition rate of CRC will drop dramatically when the available training samples per subject are very limited. One intuitive solution to this problem is operating CRC on patches and combining the recognition outputs of all patches. Nonetheless, the setting of patch size is a nontrivial task. Considering the fact that patches on different scales can have complementary information for classification, we propose a multiscale patch based CRC method, while the ensemble of multi-scale outputs is achieved by regularized margin distribution optimization. Our extensive experiments validated that the proposed method outperforms many state-of-the-art patch based face recognition algorithms.
The nearest neighbor classification is a simple and yet effective technique for pattern recogniti... more The nearest neighbor classification is a simple and yet effective technique for pattern recognition. Performance of this technique depends significantly on the distance function used to compute similarity between examples. Some techniques were developed to learn weights of features for changing the distance structure of samples in nearest neighbor classification. In this paper, we propose an approach to learning sample weights for enlarging margin by using a gradient descent algorithm to minimize margin based classification loss. Experimental analysis shows that the distances trained in this way reduce the loss of the margin and enlarge the hypothesis margin on several datasets. Moreover, the proposed approach consistently outperforms nearest neighbor classification and some other state-of-the-art methods.
Rough set theory has been extensively discussed in the domain of machine learning and data mining... more Rough set theory has been extensively discussed in the domain of machine learning and data mining. Pawlak's rough set theory offers a formal theoretical framework for attribute reduction and rule learning from nominal data. However, this model is not applicable to numerical data, which widely exist in real-world applications. In this work, we extend this framework to numerical feature spaces by replacing partition of universe with neighborhood covering and derive a neighborhood covering reduction based approach to extracting rules from numerical data. We first analyze the definition of covering reduction and point out its advantages and disadvantages. Then we introduce the definition of relative covering reduction and develop an algorithm to compute it. Given a feature space, we compute the neighborhood of each sample and form a neighborhood covering of the universe, and then employ the algorithm of relative covering reduction to the neighborhood covering, thus derive a minimal covering rule set. Some numerical experiments are presented to show the effectiveness of the proposed technique.
Feature selection is viewed as an important preprocessing step for pattern recognition, machine l... more Feature selection is viewed as an important preprocessing step for pattern recognition, machine learning and data mining. It is used to find an optimal subset to reduce computational cost, increase the classification accuracy and improve result comprehensibility. In this paper, a weighted distance learning approach is introduced to minimize Leaving-One-Out classification error using a gradient descent algorithm. The quality of features is evaluated with the learned weight and the features with great weights are considered to be useful for classification. Experimental analysis shows that the proposed approach has better performance than several state-of-the-art methods.
The nearest neighbor classification is a simple and effective technique for pattern recognition. ... more The nearest neighbor classification is a simple and effective technique for pattern recognition. The performance of this technique is known to be sensitive to the distance function used in classifying a test instance. In this paper, we propose a technique to learn sample weights via maximizing classification consistency. Experimental analysis shows that the distance trained in this way enlarges the classification consistency on several datasets and has a strong ability to tolerate noise. Moreover, the proposed approach has better performance than nearest neighbor classification and several state-of-the-art methods.
Feature selection is an important preprocessing step in pattern analysis and machine learning. Th... more Feature selection is an important preprocessing step in pattern analysis and machine learning. The key issue in feature selection is to evaluate quality of candidate features. In this work, we introduce a weighted distance learning algorithm for feature selection via maximizing fuzzy dependency. We maximize fuzzy dependency between features and decision by distance learning and then evaluate the quality of features with the learned weight vector. The features deriving great weights are considered to be useful for classification learning. We test the proposed technique with some classical methods and the experimental results show the proposed algorithm is effective.
Most of the current metric learning methods are proposed for point-to-point distance (PPD) based ... more Most of the current metric learning methods are proposed for point-to-point distance (PPD) based classification. In many computer vision tasks, however, we need to measure the point-to-set distance (PSD) and even set-to-set distance (SSD) for classification. In this paper, we extend the PPD based Mahalanobis distance metric learning to PSD and SSD based ones, namely point-to-set distance metric learning (PSDML) and set-to-set distance metric learning (SSDML), and solve them under a unified optimization framework. First, we generate positive and negative sample pairs by computing the PSD and SSD between training samples. Then, we characterize each sample pair by its covariance matrix, and propose a covariance kernel based discriminative function. Finally, we tackle the PSDML and SSDML problems by using standard support vector machine solvers, making the metric learning very efficient for multiclass visual classification tasks. Experiments on gender classification, digit recognition, object categorization and face recognition show that the proposed metric learning methods can effectively enhance the performance of PSD and SSD based classification.
With the rapid development of digital imaging and communication technologies, image set based fac... more With the rapid development of digital imaging and communication technologies, image set based face recognition (ISFR) is becoming increasingly important. One key issue of ISFR is how to effectively and efficiently represent the query face image set by using the gallery face image sets. The set-to-set distance based methods ignore the relationship between gallery sets, while representing the query set images individually over the gallery sets ignores the correlation between query set images. In this paper, we propose a novel image set based collaborative representation and classification method for ISFR. By modeling the query set as a convex or regularized hull, we represent this hull collaboratively over all the gallery sets. With the resolved representation coefficients, the distance between the query set and each gallery set can then be calculated for classification. The proposed model naturally and effectively extends the image based collaborative representation to an image set based one, and our extensive experiments on benchmark ISFR databases show the superiority of the proposed method to state-of-the-art ISFR methods under different set sizes in terms of both recognition rate and efficiency.
a b s t r a c t 22 Due to good performance in classification and regression, support vector machi... more a b s t r a c t 22 Due to good performance in classification and regression, support vector machines have attracted much 23 attention and become one of the most popular learning machines in last decade. As a black box, the sup-24 port vector machine is difficult for users' understanding and explanation. In many application domains 25 including medical diagnosis or credit scoring, understandability and interpretability are very important 26 for the practicability of the learned models. To improve the comprehensibility of SVMs, we propose a rule 27 extraction technique from support vector machines via analyzing the distribution of samples. We define 28 the consistent region of samples in terms of classification boundary, and form a consistent region cover-29 ing of the sample space. Then a covering reduction algorithm is developed for extracting compact repre-30 sentation of classes, thus a minimal set of decision rules is derived. Experiment analysis shows that the 31 extracted models perform well in comparison with decision tree algorithms and other support vector 32 machine rule extraction methods.
In this paper, a novel regularized nea method is proposed for image sets based face modeling an i... more In this paper, a novel regularized nea method is proposed for image sets based face modeling an image set as a regularized affine regularized nearest points (RNP), one on each are automatically determined by an efficient ite between-set distance of RNP is then defined by the distance between the RNPs and the structu Compared with the recently developed spar nearest points (SANP) method, RNP has formulation, less variables and lower time com experiments on benchmark databases (e.g., Ho Mobo and YouTube databases) clearly show th RNP consistently outperforms state-of-the-art accuracy and efficiency.
Improving accuracy, robustness and understandability is the objective of classification modeling.... more Improving accuracy, robustness and understandability is the objective of classification modeling. Regarding instability and performance limitation of existing rule learning techniques, we introduce an ensemble classifier based on randomized neighborhood reduction and neighborhood covering reduction. A set of reducts are obtained with randomized attribute reduction. A collection of rule sets are derived from the reducts based on neighborhood covering reduction. And then the classification result is output by combining the classification decision of different rule sets. The experiment result shows that the proposed technique is better than or equal to other classifiers, and is more stable when deals with no isy data.
Bagging is a simple and effective technique for generating an ensemble of classifiers. It is foun... more Bagging is a simple and effective technique for generating an ensemble of classifiers. It is found there are a lot of redundant base classifiers in the original Bagging. We design a pruning approach to bagging for improving its generalization power. The proposed technique introduces the margin distribution based classification loss as the optimization objective and minimizes the loss on training samples, which leads to an optimal margin distribution. Meanwhile, in order to derive a sparse ensemble, l 1 regularization is introduced to control the size of ensembles. By this way, we can obtain a sparse weight vector of base classifiers. Then we rank the base classifiers with respect to their weights and combine the base classifiers with large weights. We call this technique MArgin Distribution base Bagging pruning (MAD-Bagging). Simple voting and weighted voting are tried to combine the outputs of selected base classifiers. The performance of this pruned ensemble is evaluated with several UCI benchmark tasks, where base classifiers are trained with SVM, CART, and the nearest neighbor (1NN) rule, respectively. The results show that margin distribution based CART pruned Bagging can significantly improve classification accuracies. However, SVM and 1NN pruned Bagging improve little compared with single classifiers.
Attribute reduction is one of the most meaningful research topics in the existing fuzzy rough set... more Attribute reduction is one of the most meaningful research topics in the existing fuzzy rough sets, and the approach of discernibility matrix is the mathematical foundation of computing reducts. When computing reducts with discernibility matrix, we find that only the minimal elements in a discernibility matrix are sufficient and necessary. This fact motivates our idea in this paper to develop a novel algorithm to find reducts that are based on the minimal elements in the discernibility matrix. Relative discernibility relations of conditional attributes are defined and minimal elements in the fuzzy discernibility matrix are characterized by the relative discernibility relations. Then, the algorithms to compute minimal elements and reducts are developed in the framework of fuzzy rough sets. Experimental comparison shows that the proposed algorithms are effective.
Small sample size is one of the most challenging problems in face recognition due to the difficul... more Small sample size is one of the most challenging problems in face recognition due to the difficulty of sample collection in many real-world applications. By representing the query sample as a linear combination of training samples from all classes, the so-called collaborative representation based classification (CRC) shows very effective face recognition performance with low computational cost. However, the recognition rate of CRC will drop dramatically when the available training samples per subject are very limited. One intuitive solution to this problem is operating CRC on patches and combining the recognition outputs of all patches. Nonetheless, the setting of patch size is a nontrivial task. Considering the fact that patches on different scales can have complementary information for classification, we propose a multiscale patch based CRC method, while the ensemble of multi-scale outputs is achieved by regularized margin distribution optimization. Our extensive experiments validated that the proposed method outperforms many state-of-the-art patch based face recognition algorithms.
The nearest neighbor classification is a simple and yet effective technique for pattern recogniti... more The nearest neighbor classification is a simple and yet effective technique for pattern recognition. Performance of this technique depends significantly on the distance function used to compute similarity between examples. Some techniques were developed to learn weights of features for changing the distance structure of samples in nearest neighbor classification. In this paper, we propose an approach to learning sample weights for enlarging margin by using a gradient descent algorithm to minimize margin based classification loss. Experimental analysis shows that the distances trained in this way reduce the loss of the margin and enlarge the hypothesis margin on several datasets. Moreover, the proposed approach consistently outperforms nearest neighbor classification and some other state-of-the-art methods.
Rough set theory has been extensively discussed in the domain of machine learning and data mining... more Rough set theory has been extensively discussed in the domain of machine learning and data mining. Pawlak's rough set theory offers a formal theoretical framework for attribute reduction and rule learning from nominal data. However, this model is not applicable to numerical data, which widely exist in real-world applications. In this work, we extend this framework to numerical feature spaces by replacing partition of universe with neighborhood covering and derive a neighborhood covering reduction based approach to extracting rules from numerical data. We first analyze the definition of covering reduction and point out its advantages and disadvantages. Then we introduce the definition of relative covering reduction and develop an algorithm to compute it. Given a feature space, we compute the neighborhood of each sample and form a neighborhood covering of the universe, and then employ the algorithm of relative covering reduction to the neighborhood covering, thus derive a minimal covering rule set. Some numerical experiments are presented to show the effectiveness of the proposed technique.
Feature selection is viewed as an important preprocessing step for pattern recognition, machine l... more Feature selection is viewed as an important preprocessing step for pattern recognition, machine learning and data mining. It is used to find an optimal subset to reduce computational cost, increase the classification accuracy and improve result comprehensibility. In this paper, a weighted distance learning approach is introduced to minimize Leaving-One-Out classification error using a gradient descent algorithm. The quality of features is evaluated with the learned weight and the features with great weights are considered to be useful for classification. Experimental analysis shows that the proposed approach has better performance than several state-of-the-art methods.
The nearest neighbor classification is a simple and effective technique for pattern recognition. ... more The nearest neighbor classification is a simple and effective technique for pattern recognition. The performance of this technique is known to be sensitive to the distance function used in classifying a test instance. In this paper, we propose a technique to learn sample weights via maximizing classification consistency. Experimental analysis shows that the distance trained in this way enlarges the classification consistency on several datasets and has a strong ability to tolerate noise. Moreover, the proposed approach has better performance than nearest neighbor classification and several state-of-the-art methods.
Feature selection is an important preprocessing step in pattern analysis and machine learning. Th... more Feature selection is an important preprocessing step in pattern analysis and machine learning. The key issue in feature selection is to evaluate quality of candidate features. In this work, we introduce a weighted distance learning algorithm for feature selection via maximizing fuzzy dependency. We maximize fuzzy dependency between features and decision by distance learning and then evaluate the quality of features with the learned weight vector. The features deriving great weights are considered to be useful for classification learning. We test the proposed technique with some classical methods and the experimental results show the proposed algorithm is effective.
Uploads
Papers by 鹏飞 朱
Books by 鹏飞 朱