We consider the problem of bounding large deviations for noni.i.d. random variables that are allo... more We consider the problem of bounding large deviations for noni.i.d. random variables that are allowed to have arbitrary dependencies. Previous works typically assumed a specific dependence structure, namely the existence of independent components. Bounds that depend on the degree of dependence between the observations have only been studied in the theory of mixing processes, where variables are time-ordered. Here, we introduce a new way of measuring dependences within an unordered set of variables. We prove concentration inequalities, that apply to any set of random variables, but benefit from the presence of weak dependencies. We also discuss applications and extensions of our results to related problems of machine learning and large deviations.
Pac-Bayes bounds are among the most accurate generalization bounds for classifiers learned from i... more Pac-Bayes bounds are among the most accurate generalization bounds for classifiers learned from independently and identically distributed (IID) data, and it is particularly so for margin classifiers: there have been recent contributions showing how practical these bounds can be either to perform model selection (Ambroladze et al., 2007) or even to directly guide the learning of linear classifiers (Germain et al., 2009). However, there are many practical situations where the training data show some dependencies and where the traditional IID assumption does not hold. Stating generalization bounds for such frameworks is therefore of the utmost interest, both from theoretical and practical standpoints. In this work, we propose the first-to the best of our knowledge-Pac-Bayes generalization bounds for classifiers trained on data exhibiting interdependencies. The approach undertaken to establish our results is based on the decomposition of a so-called dependency graph that encodes the dependencies within the data, in sets of independent data, thanks to graph fractional covers. Our bounds are very general, since being able to find an upper bound on the fractional chromatic number of the dependency graph is sufficient to get new Pac-Bayes bounds for specific settings. We show how our results can be used to derive bounds for ranking statistics (such as Auc) and classifiers trained on data distributed according to a stationary β-mixing process. In the way, we show how our approach seemlessly allows us to deal with U-processes. As a side note, we also provide a Pac-Bayes generalization bound for classifiers learned on data from stationary ϕ-mixing distributions.
We study the restless bandit problem where arms are associated with stationary φ-mixing processes... more We study the restless bandit problem where arms are associated with stationary φ-mixing processes and where rewards are therefore dependent: the question that arises from this setting is that of carefully recovering some independence by 'ignoring' the values of some rewards. As we shall see, the bandit problem we tackle requires us to address the exploration/exploitation/independence trade-off, which we do by considering the idea of a waiting arm in the new Remix-UCB algorithm, a generalization of Improved-UCB for the problem at hand, that we introduce. We provide a regret analysis for this bandit strategy; two noticeable features of Remix-UCB are that i) it reduces to the regular Improved-UCB when the φ-mixing coefficients are all 0, i.e. when the i.i.d scenario is recovered, and ii) when φ(n) = O(n-α), it is able to ensure a controlled regret of order Θ~ (Δ*(α-2)/ α log1/α T), where Δ* encodes the distance between the best arm and the best suboptimal arm, even in the case ...
The trace regression model, a direct extension of the well-studied linear regression model, allow... more The trace regression model, a direct extension of the well-studied linear regression model, allows one to map matrices to real-valued outputs. We here introduce an even more general model, namely the partial-trace regression model, a family of linear mappings from matrix-valued inputs to matrix-valued outputs; this model subsumes the trace regression model and thus the linear regression model. Borrowing tools from quantum information theory, where partial trace operators have been extensively studied, we propose a framework for learning partial trace regression models from data by taking advantage of the so-called low-rank Kraus representation of completely positive maps. We show the relevance of our framework with synthetic and real-world experiments conducted for both i) matrix-to-matrix regression and ii) positive semidefinite matrix completion, two tasks which can be formulated as partial trace regression problems.
We provide new concentration inequalities for functions of dependent variables. The work extends ... more We provide new concentration inequalities for functions of dependent variables. The work extends that of Janson (2004), which proposes concentration inequalities using a combination of the Laplace transform and the idea of fractional graph coloring, as well as many works that derive concentration inequalities using the entropy method (see, e.g., (Boucheron et al., 2003)). We give inequalities for fractionally sub-additive and fractionally self-bounding functions. In the way, we prove a new Talagrand concentration inequality for fractionally sub-additive functions of dependent variables. The results allow us to envision the derivation of generalization bounds for various applications where dependent variables naturally appear, such as in bipartite ranking.
We present a new local strategy to solve incremental learning tasks. Applied to Support Vector Ma... more We present a new local strategy to solve incremental learning tasks. Applied to Support Vector Machines based on local kernel, it allows to avoid re-learning of all the parameters by selecting a working subset where the incremental learning is performed. Automatic selection procedure is based on the estimation of generalization error by using theoretical bounds that involve the margin notion. Experimental simulation on three typical datasets of machine learning give promising results.
In the context of large-scale problems, traditional multiclass classification approaches have to ... more In the context of large-scale problems, traditional multiclass classification approaches have to deal with class imbalancement and complexity issues which make them inoperative in some extreme cases. In this paper we study a transformation that reduces the initial multiclass classification of examples into a binary classification of pairs of examples and classes. We present generalization error bounds that exhibit the interdependency between the pairs of examples and which recover known results on binary classification with i.i.d. data. We show the efficiency of the deduced algorithm compared to state-of-the-art multiclass classification strategies on two large-scale document collections especially in the interesting case where the number of classes becomes very large.
2015 International Joint Conference on Neural Networks (IJCNN), 2015
Cutting-plane methods are well-studied localization (and optimization) algorithms. We show that t... more Cutting-plane methods are well-studied localization (and optimization) algorithms. We show that they provide a natural framework to perform machine learning-and not just to solve optimization problems posed by machine learning-in addition to their intended optimization use. In particular, they allow one to learn sparse classifiers and provide good compression schemes. Moreover, we show that very little effort is required to turn them into effective active learning methods. This last property provides a generic way to design a whole family of active learning algorithms from existing passive methods. We present numerical simulations testifying of the relevance of cutting-plane methods for passive and active learning tasks.
We study the bandit problem where arms are associated with stationary ϕ-mixing processes and wher... more We study the bandit problem where arms are associated with stationary ϕ-mixing processes and where rewards are therefore dependent: the question that arises from this setting is that of recovering some independence by ignoring the value of some rewards. As we shall see, the bandit problem we tackle requires us to address the exploration/exploitation/independence trade-off. To do so, we provide a UCB strategy together with a general regret analysis for the case where the size of the independence blocks (the ignored rewards) is fixed and we go a step beyond by providing an algorithm that is able to compute the size of the independence blocks from the data. Finally, we give an analysis of our bandit problem in the restless case, i.e., in the situation where the time counters for all mixing processes simultaneously evolve.
In this paper, we address the issue of learning nonlinearly separable concepts with a kernel clas... more In this paper, we address the issue of learning nonlinearly separable concepts with a kernel classifier in the situation where the data at hand are altered by a uniform classification noise. Our proposed approach relies on the combination of the technique of random or deterministic projections with a classification noise tolerant perceptron learning algorithm that assumes distributions defined over finitedimensional spaces. Provided a sufficient separation margin characterizes the problem, this strategy makes it possible to envision the learning from a noisy distribution in any separable Hilbert space, regardless of its dimension; learning with any appropriate Mercer kernel is therefore possible. We prove that the required sample complexity and running time of our algorithm is polynomial in the classical PAC learning parameters. Numerical simulations on toy datasets and on data from the UCI repository support the validity of our approach.
We present a new classification algorithm capable of learning from data corrupted by a class depe... more We present a new classification algorithm capable of learning from data corrupted by a class dependent uniform classification noise. The produced classifier is a linear classifier, and the algorithm works seamlessly when using kernels. The algorithm relies on the sampling of random hyperplanes that help the building of new training examples of which the correct classes are known; a linear classifier (e.g. an SVM) is learned from these examples and output by the algorithm. The produced examples are sample averages computed from the data at hand with respect to areas of the space defined by the random hyperplanes and the target hyperplane. A statistical analysis of the properties of these sample averages is provided as well as results from numerical simulations conducted on synthetic datasets. These simulations show that the linear and kernelized versions of our algorithm are effective for learning from both noise-free and noisy data.
We provide new theoretical results on the generalization properties of learning algorithms for mu... more We provide new theoretical results on the generalization properties of learning algorithms for multiclass classification problems. The originality of our work is that we propose to use the confusion matrix of a classifier as a measure of its quality; our contribution is in the line of work which attempts to set up and study the statistical properties of new evaluation measures such as, e.g. ROC curves. In the confusion-based learning framework we propose, we claim that a targetted objective is to minimize the size of the confusion matrix C, measured through its operator norm C. We derive generalization bounds on the (size of the) confusion matrix in an extended framework of uniform stability, adapted to the case of matrix valued loss. Pivotal to our study is a very recent matrix concentration inequality that generalizes McDiarmid's inequality. As an illustration of the relevance of our theoretical results, we show how two SVM learning procedures can be proved to be confusion-friendly. To the best of our knowledge, the present paper is the first that focuses on the confusion matrix from a theoretical point of view.
This paper provides the first-to the best of our knowledge-analysis of online learning algorithms... more This paper provides the first-to the best of our knowledge-analysis of online learning algorithms for multiclass problems when the confusion matrix is taken as a performance measure. The work builds upon recent and elegant results on noncommutative concentration inequalities, i.e. concentration inequalities that apply to matrices, and, more precisely, to matrix martingales. We do establish generalization bounds for online learning algorithms and show how the theoretical study motivates the proposition of a new confusion-friendly learning procedure. This learning algorithm, called COPA (for COnfusion Passive-Aggressive) is a passive-aggressive learning algorithm; it is shown that the update equations for COPA can be computed analytically and, henceforth, there is no need to recourse to any optimization package to implement it.
The C-bound, introduced in Lacasse et al. [1], gives a tight upper bound on the risk of a binary ... more The C-bound, introduced in Lacasse et al. [1], gives a tight upper bound on the risk of a binary majority vote classifier. In this work, we present a first step towards extending this work to more complex outputs, by providing generalizations of the C-bound to the multiclass and multi-label settings.
Dans ce travail, nous proposons une méthode originale permettant d'appréhender des séries tempore... more Dans ce travail, nous proposons une méthode originale permettant d'appréhender des séries temporelles aux caractéristiques non linéaires. Cette méthode repose sur une extension du filtrage de Kalman à l'aide de l'astuce du noyau (kernel trick). L'usage d'une fonction noyau permet d'employer la formulation classique du filtre Kalman dans l'espace des caractéristiques défini par la fonction noyau.
The Lasso is an optimization problem devoted to find a sparse representation of some signal with ... more The Lasso is an optimization problem devoted to find a sparse representation of some signal with respect to some predefined dictionary. We propose an original and computationally efficient method to solve the Lasso problem. Our method rests upon the idea of dynamic screening test and allows us to accelerate first-order algorithms. At each iteration we take advantage of the computation done for the optimization process to reduce the size of the dictionary by discarding elements that will surely not enter the sparse representation. As this screening step is not expensive, the computational cost of the algorithm using the dynamically screened dictionary is cheaper than the standard algorithm. Numerical simulations on synthetic and real data support the relevance of our approach.
Apres avoir exhibe un exemple basique montrant que les SVMa marges douces (CSVM) ne sont pas tol´... more Apres avoir exhibe un exemple basique montrant que les SVMa marges douces (CSVM) ne sont pas tol´ erantes au bruit de classification uniforme, nous proposons une version modifi´ ee de CSVM bas´ ee sur une fonction objectif utilisant un estimateur des slack variables du probl` eme non bruite. Les bonnes propri´ etes de cet estimateur sont appuyees par une analyse
In this paper, we address the issue of learning nonlinearly separable concepts with a kernel clas... more In this paper, we address the issue of learning nonlinearly separable concepts with a kernel classifier in the situation where the data at hand are altered by a uniform classification noise. Our proposed approach relies on the combination of the technique of random or deterministic projections with a classification noise tolerant perceptron learning algorithm that assumes distributions defined over finitedimensional spaces. Provided a sufficient separation margin characterizes the problem, this strategy makes it possible to envision the learning from a noisy distribution in any separable Hilbert space, regardless of its dimension; learning with any appropriate Mercer kernel is therefore possible. We prove that the required sample complexity and running time of our algorithm is polynomial in the classical PAC learning parameters. Numerical simulations on toy datasets and on data from the UCI repository support the validity of our approach.
We consider the problem of bounding large deviations for noni.i.d. random variables that are allo... more We consider the problem of bounding large deviations for noni.i.d. random variables that are allowed to have arbitrary dependencies. Previous works typically assumed a specific dependence structure, namely the existence of independent components. Bounds that depend on the degree of dependence between the observations have only been studied in the theory of mixing processes, where variables are time-ordered. Here, we introduce a new way of measuring dependences within an unordered set of variables. We prove concentration inequalities, that apply to any set of random variables, but benefit from the presence of weak dependencies. We also discuss applications and extensions of our results to related problems of machine learning and large deviations.
Pac-Bayes bounds are among the most accurate generalization bounds for classifiers learned from i... more Pac-Bayes bounds are among the most accurate generalization bounds for classifiers learned from independently and identically distributed (IID) data, and it is particularly so for margin classifiers: there have been recent contributions showing how practical these bounds can be either to perform model selection (Ambroladze et al., 2007) or even to directly guide the learning of linear classifiers (Germain et al., 2009). However, there are many practical situations where the training data show some dependencies and where the traditional IID assumption does not hold. Stating generalization bounds for such frameworks is therefore of the utmost interest, both from theoretical and practical standpoints. In this work, we propose the first-to the best of our knowledge-Pac-Bayes generalization bounds for classifiers trained on data exhibiting interdependencies. The approach undertaken to establish our results is based on the decomposition of a so-called dependency graph that encodes the dependencies within the data, in sets of independent data, thanks to graph fractional covers. Our bounds are very general, since being able to find an upper bound on the fractional chromatic number of the dependency graph is sufficient to get new Pac-Bayes bounds for specific settings. We show how our results can be used to derive bounds for ranking statistics (such as Auc) and classifiers trained on data distributed according to a stationary β-mixing process. In the way, we show how our approach seemlessly allows us to deal with U-processes. As a side note, we also provide a Pac-Bayes generalization bound for classifiers learned on data from stationary ϕ-mixing distributions.
We study the restless bandit problem where arms are associated with stationary φ-mixing processes... more We study the restless bandit problem where arms are associated with stationary φ-mixing processes and where rewards are therefore dependent: the question that arises from this setting is that of carefully recovering some independence by 'ignoring' the values of some rewards. As we shall see, the bandit problem we tackle requires us to address the exploration/exploitation/independence trade-off, which we do by considering the idea of a waiting arm in the new Remix-UCB algorithm, a generalization of Improved-UCB for the problem at hand, that we introduce. We provide a regret analysis for this bandit strategy; two noticeable features of Remix-UCB are that i) it reduces to the regular Improved-UCB when the φ-mixing coefficients are all 0, i.e. when the i.i.d scenario is recovered, and ii) when φ(n) = O(n-α), it is able to ensure a controlled regret of order Θ~ (Δ*(α-2)/ α log1/α T), where Δ* encodes the distance between the best arm and the best suboptimal arm, even in the case ...
The trace regression model, a direct extension of the well-studied linear regression model, allow... more The trace regression model, a direct extension of the well-studied linear regression model, allows one to map matrices to real-valued outputs. We here introduce an even more general model, namely the partial-trace regression model, a family of linear mappings from matrix-valued inputs to matrix-valued outputs; this model subsumes the trace regression model and thus the linear regression model. Borrowing tools from quantum information theory, where partial trace operators have been extensively studied, we propose a framework for learning partial trace regression models from data by taking advantage of the so-called low-rank Kraus representation of completely positive maps. We show the relevance of our framework with synthetic and real-world experiments conducted for both i) matrix-to-matrix regression and ii) positive semidefinite matrix completion, two tasks which can be formulated as partial trace regression problems.
We provide new concentration inequalities for functions of dependent variables. The work extends ... more We provide new concentration inequalities for functions of dependent variables. The work extends that of Janson (2004), which proposes concentration inequalities using a combination of the Laplace transform and the idea of fractional graph coloring, as well as many works that derive concentration inequalities using the entropy method (see, e.g., (Boucheron et al., 2003)). We give inequalities for fractionally sub-additive and fractionally self-bounding functions. In the way, we prove a new Talagrand concentration inequality for fractionally sub-additive functions of dependent variables. The results allow us to envision the derivation of generalization bounds for various applications where dependent variables naturally appear, such as in bipartite ranking.
We present a new local strategy to solve incremental learning tasks. Applied to Support Vector Ma... more We present a new local strategy to solve incremental learning tasks. Applied to Support Vector Machines based on local kernel, it allows to avoid re-learning of all the parameters by selecting a working subset where the incremental learning is performed. Automatic selection procedure is based on the estimation of generalization error by using theoretical bounds that involve the margin notion. Experimental simulation on three typical datasets of machine learning give promising results.
In the context of large-scale problems, traditional multiclass classification approaches have to ... more In the context of large-scale problems, traditional multiclass classification approaches have to deal with class imbalancement and complexity issues which make them inoperative in some extreme cases. In this paper we study a transformation that reduces the initial multiclass classification of examples into a binary classification of pairs of examples and classes. We present generalization error bounds that exhibit the interdependency between the pairs of examples and which recover known results on binary classification with i.i.d. data. We show the efficiency of the deduced algorithm compared to state-of-the-art multiclass classification strategies on two large-scale document collections especially in the interesting case where the number of classes becomes very large.
2015 International Joint Conference on Neural Networks (IJCNN), 2015
Cutting-plane methods are well-studied localization (and optimization) algorithms. We show that t... more Cutting-plane methods are well-studied localization (and optimization) algorithms. We show that they provide a natural framework to perform machine learning-and not just to solve optimization problems posed by machine learning-in addition to their intended optimization use. In particular, they allow one to learn sparse classifiers and provide good compression schemes. Moreover, we show that very little effort is required to turn them into effective active learning methods. This last property provides a generic way to design a whole family of active learning algorithms from existing passive methods. We present numerical simulations testifying of the relevance of cutting-plane methods for passive and active learning tasks.
We study the bandit problem where arms are associated with stationary ϕ-mixing processes and wher... more We study the bandit problem where arms are associated with stationary ϕ-mixing processes and where rewards are therefore dependent: the question that arises from this setting is that of recovering some independence by ignoring the value of some rewards. As we shall see, the bandit problem we tackle requires us to address the exploration/exploitation/independence trade-off. To do so, we provide a UCB strategy together with a general regret analysis for the case where the size of the independence blocks (the ignored rewards) is fixed and we go a step beyond by providing an algorithm that is able to compute the size of the independence blocks from the data. Finally, we give an analysis of our bandit problem in the restless case, i.e., in the situation where the time counters for all mixing processes simultaneously evolve.
In this paper, we address the issue of learning nonlinearly separable concepts with a kernel clas... more In this paper, we address the issue of learning nonlinearly separable concepts with a kernel classifier in the situation where the data at hand are altered by a uniform classification noise. Our proposed approach relies on the combination of the technique of random or deterministic projections with a classification noise tolerant perceptron learning algorithm that assumes distributions defined over finitedimensional spaces. Provided a sufficient separation margin characterizes the problem, this strategy makes it possible to envision the learning from a noisy distribution in any separable Hilbert space, regardless of its dimension; learning with any appropriate Mercer kernel is therefore possible. We prove that the required sample complexity and running time of our algorithm is polynomial in the classical PAC learning parameters. Numerical simulations on toy datasets and on data from the UCI repository support the validity of our approach.
We present a new classification algorithm capable of learning from data corrupted by a class depe... more We present a new classification algorithm capable of learning from data corrupted by a class dependent uniform classification noise. The produced classifier is a linear classifier, and the algorithm works seamlessly when using kernels. The algorithm relies on the sampling of random hyperplanes that help the building of new training examples of which the correct classes are known; a linear classifier (e.g. an SVM) is learned from these examples and output by the algorithm. The produced examples are sample averages computed from the data at hand with respect to areas of the space defined by the random hyperplanes and the target hyperplane. A statistical analysis of the properties of these sample averages is provided as well as results from numerical simulations conducted on synthetic datasets. These simulations show that the linear and kernelized versions of our algorithm are effective for learning from both noise-free and noisy data.
We provide new theoretical results on the generalization properties of learning algorithms for mu... more We provide new theoretical results on the generalization properties of learning algorithms for multiclass classification problems. The originality of our work is that we propose to use the confusion matrix of a classifier as a measure of its quality; our contribution is in the line of work which attempts to set up and study the statistical properties of new evaluation measures such as, e.g. ROC curves. In the confusion-based learning framework we propose, we claim that a targetted objective is to minimize the size of the confusion matrix C, measured through its operator norm C. We derive generalization bounds on the (size of the) confusion matrix in an extended framework of uniform stability, adapted to the case of matrix valued loss. Pivotal to our study is a very recent matrix concentration inequality that generalizes McDiarmid's inequality. As an illustration of the relevance of our theoretical results, we show how two SVM learning procedures can be proved to be confusion-friendly. To the best of our knowledge, the present paper is the first that focuses on the confusion matrix from a theoretical point of view.
This paper provides the first-to the best of our knowledge-analysis of online learning algorithms... more This paper provides the first-to the best of our knowledge-analysis of online learning algorithms for multiclass problems when the confusion matrix is taken as a performance measure. The work builds upon recent and elegant results on noncommutative concentration inequalities, i.e. concentration inequalities that apply to matrices, and, more precisely, to matrix martingales. We do establish generalization bounds for online learning algorithms and show how the theoretical study motivates the proposition of a new confusion-friendly learning procedure. This learning algorithm, called COPA (for COnfusion Passive-Aggressive) is a passive-aggressive learning algorithm; it is shown that the update equations for COPA can be computed analytically and, henceforth, there is no need to recourse to any optimization package to implement it.
The C-bound, introduced in Lacasse et al. [1], gives a tight upper bound on the risk of a binary ... more The C-bound, introduced in Lacasse et al. [1], gives a tight upper bound on the risk of a binary majority vote classifier. In this work, we present a first step towards extending this work to more complex outputs, by providing generalizations of the C-bound to the multiclass and multi-label settings.
Dans ce travail, nous proposons une méthode originale permettant d'appréhender des séries tempore... more Dans ce travail, nous proposons une méthode originale permettant d'appréhender des séries temporelles aux caractéristiques non linéaires. Cette méthode repose sur une extension du filtrage de Kalman à l'aide de l'astuce du noyau (kernel trick). L'usage d'une fonction noyau permet d'employer la formulation classique du filtre Kalman dans l'espace des caractéristiques défini par la fonction noyau.
The Lasso is an optimization problem devoted to find a sparse representation of some signal with ... more The Lasso is an optimization problem devoted to find a sparse representation of some signal with respect to some predefined dictionary. We propose an original and computationally efficient method to solve the Lasso problem. Our method rests upon the idea of dynamic screening test and allows us to accelerate first-order algorithms. At each iteration we take advantage of the computation done for the optimization process to reduce the size of the dictionary by discarding elements that will surely not enter the sparse representation. As this screening step is not expensive, the computational cost of the algorithm using the dynamically screened dictionary is cheaper than the standard algorithm. Numerical simulations on synthetic and real data support the relevance of our approach.
Apres avoir exhibe un exemple basique montrant que les SVMa marges douces (CSVM) ne sont pas tol´... more Apres avoir exhibe un exemple basique montrant que les SVMa marges douces (CSVM) ne sont pas tol´ erantes au bruit de classification uniforme, nous proposons une version modifi´ ee de CSVM bas´ ee sur une fonction objectif utilisant un estimateur des slack variables du probl` eme non bruite. Les bonnes propri´ etes de cet estimateur sont appuyees par une analyse
In this paper, we address the issue of learning nonlinearly separable concepts with a kernel clas... more In this paper, we address the issue of learning nonlinearly separable concepts with a kernel classifier in the situation where the data at hand are altered by a uniform classification noise. Our proposed approach relies on the combination of the technique of random or deterministic projections with a classification noise tolerant perceptron learning algorithm that assumes distributions defined over finitedimensional spaces. Provided a sufficient separation margin characterizes the problem, this strategy makes it possible to envision the learning from a noisy distribution in any separable Hilbert space, regardless of its dimension; learning with any appropriate Mercer kernel is therefore possible. We prove that the required sample complexity and running time of our algorithm is polynomial in the classical PAC learning parameters. Numerical simulations on toy datasets and on data from the UCI repository support the validity of our approach.
Uploads
Papers by Liva Ralaivola