Fixed-size kernel logistic regression for phoneme classification

2006

For the classical statistical classification algorithms the probability distribution models are known. However, in many real life applications, such as speech recognition, there is not enough information about the probability distribution function. This is a very common scenario and poses a very serious restriction in classification. Support Vector Machines (SVMs) can help in such situations because they are distribution free algorithms that originated from statistical learning theory and Structural Risk Minimization (SRM). In the most basic approach SVMs use linearly separating Hyperplanes to create classification with maximal margins. However in application, the classification problem requires a constrained nonlinear approach to be taken during the learning stages, and a quadratic problem has to be solved. For the case where the classes cannot be linearly separable due to overlap, the SVM algorithm will transform the original input space into a higher dimensional feature space, wh...

IEEE Transactions on Audio, Speech, and Language Processing, 2000

This work proposes methods for combining cepstral and acoustic waveform representations for a front-end of support vector machine (SVM) based speech recognition systems that are robust to additive noise. The key issue of kernel design and noise adaptation for the acoustic waveform representation is addressed first. Cepstral and acoustic waveform representations are then compared on a phoneme classification task. Experiments show that the cepstral features achieve very good performance in low noise conditions, but suffer severe performance degradation already at moderate noise levels. Classification in the acoustic waveform domain, on the other hand, is less accurate in low noise but exhibits a more robust behavior in high noise conditions. A combination of the cepstral and acoustic waveform representations achieves better classification performance than either of the individual representations over the entire range of noise levels tested, down to −18dB SNR.

2005

In this paper we use kernel-based Fisher Discriminants (KFD) for classification by integrating this method in a HMM-based speech recognition system. We translate the outputs of the KFD-classifier into conditional probabilities and use them as production probabilities of a HMM-based decoder for speech recognition. To obtain a good performance also in terms of computational complexity the Recursive Least Squares Algorithm (RLS-Algorithm) is enforced. We train and test the described hybrid structure on the Resource Management Corpus (RM1).

Interspeech 2009

This work focuses on the robustness of phoneme classification to additive noise in the acoustic waveform domain using support vector machines (SVMs). We address the issue of designing kernels for acoustic waveforms which imitate the state-ofthe-art representations such as PLP and MFCC and are tuned to the physical properties of speech. For comparison, classification results in the PLP representation domain with cepstral mean-and-variance normalization (CMVN) using standard kernels are also reported. It is shown that our custom-designed kernels achieve better classification performance at high noise. Finally, we combine the PLP and acoustic waveform representations to attain better classification than either of the individual representations over the entire range of noise levels tested, from quiet condition up to −18dB SNR.

2000

We describe a new method for phoneme sequence recognition given a speech utterance. In contrast to HMM-based approaches, our method uses a kernel-based discriminative training procedure in which the learning process is tailored to the goal of minimizing the Levenshtein distance between the predicted phoneme sequence and the correct sequence. The phoneme sequence predictor is devised by mapping the speech utterance along with a proposed phoneme sequence to a vector-space endowed with an inner-product that is realized by a Mercer kernel. Building on large margin techniques for predicting whole sequences, we are able to devise a learning algorithm which distills to separating the correct phoneme sequence from all other sequences. We describe an iterative algorithm for learning the phoneme sequence recognizer and further describe an efficient implementation of it. We present initial encouraging experimental results with the TIMIT and compare the proposed method to an HMM-based approach.

2006 IEEE International Conference on Acoustics Speed and Signal Processing Proceedings, 2006

In this paper, we examine the problem of kernel selection for oneversus-all (OVA) classification of multiclass data with support vector machines (SVMs). We focus specifically on the problem of training what we refer to as generalized linear kernels-that is, kernels of the form, ´Ü½ ܾµ Ü Ì ½ Êܾ, where Ê is a positive semidefinite matrix. Our approach for training ´Ü½ ܾµ involves first constructing a set of upper bounds on the rates of false positives and false negatives at a given score threshold. Under various conditions, minimizing these bounds leads to the closed-form solution, Ê Ï ½ , where Ï is the expected within-class covariance matrix of the data. We tested various parameterizations of Ê, including a diagonal parameterization that simply performs perfeature variance normalization, on the 1-conversation training condition of the SRE-2003 and SRE-2004 speaker recognition tasks. In experiments on a state-of-the-art MLLR-SVM speaker recognition system [1], the parameterization, Ê Ï ½ × , where Ï× is a smoothed estimate of Ï, achieves relative reductions in the minimum decision cost function (DCF) [2] of up to 22% below the results obtained when Ê does per-feature variance normalization.

We consider pattern classification using a weighted sum of normalized kernel functions. Such schemes can be viewed as estimates of class a posteriori probabilities. We apply this regression method successfully to two real life pattern recognition problems.

Intelligenza Artificiale, 2017

Expressive but complex kernel functions, such as Sequence or Tree kernels, are usually underemployed in NLP tasks as for their significant computational cost in both learning and classification stages. Recently, the Nyström methodology for data embedding has been proposed as a viable solution to scalability problems. It improves scalability of learning processes acting over highly structured data, by mapping data into low-dimensional compact linear representations of kernel spaces. In this paper, a stratification of the model corresponding to the embedding space is proposed as a further highly flexible optimization. Nyström embedding spaces of increasing sizes are combined in an efficient ensemble strategy: upper layers, providing higher dimensional representations, are invoked on input instances only when the adoption of smaller (i.e., less expressive) embeddings provides uncertain outcomes. Experimental results using different models of such an uncertainty show that state-of-the-art accuracy on three semantic inference tasks can be obtained even when one order of magnitude fewer kernel computations is carried out.

The least-squares probabilistic classifier (LSPC) is a computationally efficient alternative to kernel logistic regression (KLR). A key idea for the speedup is that, unlike KLR that uses maximum likelihood estimation for a log-linear model, LSPC uses least-squares estimation for a linear model. This allows us to obtain a global solution analytically in a classwise manner. In exchange for the speedup, however, this linear least-squares formulation does not necessarily produce a non-negative estimate. Nevertheless, consistency of LSPC is guaranteed in the large sample limit, and rounding up a negative estimate to zero in finite sample cases was demonstrated not to degrade the classification performance in experiments. Thus, LSPC is a practically useful probabilistic classifier. In this paper, we give an overview of LSPC and its extentions to covariate shift, multi-task, and multi-label scenarios. A MATLAB implementation of LSPC is available from '

Logistic regression is a linear binary classification algorithm frequently used for classification problems. In this paper we present its kernel version which is used for classification of non-linearly separable problems. We briefly introduce the concept of multiple kernel learning and apply it to kernel logistic regression. We elaborate the performance differences between classical, kernel logistic regression and its stochas-tic variant (both classical and kernel logistic regression).

Lecture Notes in Computer Science, 2001

Logistic regression is presumably the most popular representative of probabilistic discriminative classifiers. In this paper, a kernel variant of logistic regression is introduced as an iteratively re-weighted least-squares algorithm in kernel-induced feature spaces. This formulation allows us to apply highly efficient approximation methods that are capable of dealing with large-scale problems. For multi-class problems, a pairwise coupling procedure is proposed. Pairwise coupling for "kernelized" logistic regression effectively overcomes conceptual and numerical problems of standard multi-class kernel classifiers.

1999

Support Vector Machines (SVMs) represent a new approach to pattern classification which has recently attracted a great deal of interest in the machine learning community. Their appeal lies in their strong connection to the underlying statistical learning theory, in particular the theory of Structural Risk Minimization. SVMs have been shown to be particularly successful in fields such as image identification and face recognition; in many problems SVM classifiers have been shown to perform much better than other nonlinear classifiers such as artificial neural networks and ¡ -nearest neighbors.

2005

While the temporal dynamic of speech can be handled very efficiently by Hidden Markov Models (HMMs), the classification of the single speech units (phonemes) is usually done with Gaussian probability density functions which are not discriminative. In this paper we use the Kernel Fisher Discriminant (KFD) for classification by integrating this method in a HMM-based speech recognition system. In this structure we translate the outputs of the KFD into class-conditional probabilities and use them as production probabilities in an HMM-based speech decoder. The KFD has already shown good classification results in other fields (e. g. pattern recognition). To obtain good performance also in terms of computational complexity the KFD is implemented iteratively with a sparse greedy approach. We train and test the described hybrid structure on the Resource Management (RM1) task.

2006 14th European Signal Processing Conference, 2006

The robustness of phoneme recognition using support vector machines to additive noise is investigated for three kinds of speech representation. The representations considered are PLP, PLP with RASTA processing, and a high-dimensional principal component approximation of acoustic waveforms. While the classification in the PLP and PLP/RASTA domains attains superb accuracy on clean data, the classification in the high-dimensional space proves to be much more robust to additive noise.

Proceedings of the 8th …, 2009

In this paper, we are proposing a new classifier called support vector machines to classify the speech signals. We have achieved very good generalization performance by implementing the support vector machines with various kernel functions. The use One-vs-One Classifier with voting algorithm improves speech signal classification systems efficiency.