Interpretable and Specialized Conformal Predictors Ulf
Cecilia Sönströd2019
Sign up for access to the world's latest research
checkGet notified about relevant papers
checkSave papers to use in your research
checkJoin the discussion with peers
checkTrack your impact
Abstract
In real-world scenarios, interpretable models are often required to explain predictions, and to allow for inspection and analysis of the model. The overall purpose of oracle coaching is to produce highly accurate, but interpretable, models optimized for a specific test set. Oracle coaching is applicable to the very common scenario where explanations and insights are needed for a specific batch of predictions, and the input vectors for this test set are available when building the predictive model. In this paper, oracle coaching is used for generating underlying classifiers for conformal prediction. The resulting conformal classifiers output valid label sets, i.e., the error rate on the test data is bounded by a preset significance level, as long as the labeled data used for calibration is exchangeable with the test set. Since validity is guaranteed for all conformal predictors, the key performance metric is efficiency, i.e., the size of the label sets, where smaller sets are more in...
Related papers
Exact Optimization of Conformal Predictors via Incremental and Decremental Learning
2021
Conformal Predictors (CP) are wrappers around ML models, providing error guarantees under weak assumptions on the data distribution. They are suitable for a wide range of problems, from classification and regression to anomaly detection. Unfortunately, their very high computational complexity limits their applicability to large datasets. In this work, we show that it is possible to speed up a CP classifier considerably, by studying it in conjunction with the underlying ML method, and by exploiting incremental&decremental learning. For methods such as k-NN, KDE, and kernel LSSVM, our approach reduces the running time by one order of magnitude, whilst producing exact solutions. With similar ideas, we also achieve a linear speed up for the harder case of bootstrapping. Finally, we extend these techniques to improve upon an optimization of k-NN CP for regression. We evaluate our findings empirically, and discuss when methods are suitable for CP optimization.
Excape WP 1 . Conformal Predictors
2015
The report summarises some preliminary findings of WP1.4: Confidence Estimation and feature significance. It presents an application of conformal predictors in transductive and inductive modes to the large, high-dimensional, sparse and imbalanced data sets found in Compound Activity Prediction from PubChem public repository. The report describes a version of conformal predictors called Mondrian Predictor that keeps validity guarantees for each class. The experiments were conducted using several non-conformity measures extracted from underlying algorithms such as SVM, Nearest Neighbours and Näıve Bayes. The results show (1) that Inductive Conformal Mondrian Prediction framework is quick and effective for large imbalanced data and (2) that its less strict i.i.d. requirements combine well with training set editing algorithms such as Cascade SVM. Among the algorithms tested with the Mondrian ICP framework, Cascade SVM with Tanimoto+RBF kernel appeared to be best performing one, if the q...
How Nonconformity Functions and Difficulty of Datasets Impact the Efficiency of Conformal Classifiers
ArXiv, 2021
The property of conformal predictors to guarantee the required accuracy rate makes this framework attractive in various practical applications. However, this property is achieved at a price of reduction in precision. In the case of conformal classification, the systems can output multiple class labels instead of one. It is also known from the literature, that the choice of nonconformity function has a major impact on the efficiency of conformal classifiers. Recently, it was shown that different model-agnostic nonconformity functions result in conformal classifiers with different characteristics. For a Neural Network-based conformal classifier, the inverse probability (or hinge loss) allows minimizing the average number of predicted labels, and margin results in a larger fraction of singleton predictions. In this work, we aim to further extend this study. We perform an experimental evaluation using 8 different classification algorithms and discuss when the previously observed relatio...
Learning by Conformal Predictors with Additional Information
In many supervised learning applications, the existence of additional information in training data is very common. Recently, Vapnik introduced a new method called LUPI which provides a learning paradigm under privileged (or additional) information. It describes the SVM+ technique to process this information in batch mode. Following this method, we apply the approach to deal with additional information by conformal predictors. An application to a medical diagnostic problem is considered and the results are reported.
Impact of model-agnostic nonconformity functions on efficiency of conformal classifiers: an extensive study
2021
The property of conformal predictors to guarantee the required accuracy rate makes this framework attractive in various practical applications. However, this property is achieved at a price of reduction in precision. In the case of conformal classification, the system can output multiple class labels instead of one. It is also known, that the choice of nonconformity function has a major impact on the efficiency of conformal classifiers. Recently, it was shown that different model-agnostic nonconformity functions result in conformal classifiers with different characteristics. For a Neural Network-based conformal classifier, the inverse probability (or hinge loss) allows minimizing the average number of predicted labels, and margin results in a larger fraction of singleton predictions. In this work, we aim to further extend this study. We perform an experimental evaluation using 8 different classification algorithms and discuss when the previously observed relationship holds or not. A...
Preliminary numerical investigations of conformal predictors based on fuzzy logic classifiers
Annals of Mathematics and Artificial Intelligence, 2014
A new family of techniques, called conformal predictors, have very recently been developed to hedge the estimates of machine learning methods, by providing two parameters, credibility and confidence, which can assess the level of trust that can be attributed to their outputs. In this paper, the main steps required to extend this approach to fuzzy logic classifiers are reported. The more delicate aspect is the definition of an appropriate nonconformity score, which has to be based on the membership function to preserve the specificities of Fuzzy Logic. Various examples of increasing complexity are introduced, to describe the main properties of fuzzy logic based conformal predictors and to compare their performance with alternative approaches. The obtained results are quite promising, since conformal predictors based on fuzzy classifiers outperform solutions based on the nearest neighbour in terms of ambiguity, robustness and interpretability.
Regression Conformal Prediction with Nearest Neighbours
Journal of Artificial Intelligence Research, 2011
In this paper we apply Conformal Prediction (CP) to the k -Nearest Neighbours Regression (k -NNR) algorithm and propose ways of extending the typical nonconformity measure used for regression so far. Unlike traditional regression methods which produce point predictions, Conformal Predictors output predictive regions that satisfy a given confidence level. The regions produced by any Conformal Predictor are automatically valid, however their tightness and therefore usefulness depends on the nonconformity measure used by each CP. In effect a nonconformity measure evaluates how strange a given example is compared to a set of other examples based on some traditional machine learning algorithm. We define six novel nonconformity measures based on the k -Nearest Neighbours Regression algorithm and develop the corresponding CPs following both the original (transductive) and the inductive CP approaches. A comparison of the predictive regions produced by our measures with those of the typical regression measure suggests that a major improvement in terms of predictive region tightness is achieved by the new measures.
Localized Conformal Prediction: A Generalized Inference Framework for Conformal Prediction
2021
We propose a new inference framework called localized conformal prediction. It generalizes the framework of conformal prediction and offers a single-test-sample adaptive construction by emphasizing a local region around it. Although there have been methods constructing heterogeneous prediction intervals for Y by designing better conformal score functions, to our knowledge, this is the first work that introduces an adaptive nature to the inference framework itself. We prove that our proposal leads to an assumption-free and finite sample marginal coverage guarantee, as well as an approximate conditional coverage guarantee. Our proposal achieves asymptotic conditional coverage under suitable assumptions. The localized conformal prediction can be combined with many existing works in conformal prediction, including different types of conformal score constructions. We will demonstrate how to change from conformal prediction to localized conformal prediction in these related works and a po...
Constructing normalized nonconformity measures based on maximizing predictive efficiency
Proceedings of Machine Learning Research, 2020
The problem of regression in the inductive conformal prediction framework is addressedto provide prediction intervals that are optimized by predictive efficiency. A differentiablefunction is used to approximate the exact optimization problem of minimizing predictive inefficiency on a training data set using a conformal predictor based on a parametric normalized nonconformity measure. Gradient descent is then used to find a solution. Sincethe optimization approximates the conformal predictor, this method is called surrogate conformal predictor optimization. Experiments are reported that show that it results in conformal predictors that provide improved predictive efficiency for regression problems on several data sets, whilst remaining reliable. It is also shown that the optimal parameter values typically differ for different confidence levels. Using house price data, alternative measures of inefficiency are explored to address different application requirements.
Cross-conformal predictive distributions
Proceedings of Machine Learning Research , 2018
Conformal predictive systems are a recent modification of conformal predictors that output, in regression problems, probability distributions for labels of test observations rather than set predictions. The extra information provided by conformal predictive systems may be useful, e.g., in decision making problems. Conformal predictive systems inherit the relative computational ine ciency of conformal predictors. In this paper we discuss two computationally efficient versions of conformal predictive systems, which we call split conformal predictive systems and cross-conformal predictive systems, and discuss their advantages and limitations.
Related topics
Related papers
A tutorial on conformal prediction
Journal of Machine Learning Research, 2007
praktiqeskie vyvody teorii vero tnoste mogut byt obosnovany v kaqestve sledstvi gipotez o predel no pri dannyh ograniqeni h slo nosti izuqaemyh vleni Abstract Conformal prediction uses past experience to determine precise levels of confidence in new predictions. Given an error probability , together with a method that makes a predictionŷ of a label y, it produces a set of labels, typically containingŷ, that also contains y with probability 1 − . Conformal prediction can be applied to any method for producingŷ: a nearest-neighbor method, a support-vector machine, ridge regression, etc.
Obtaining accurate and comprehensible classifiers using oracle coaching
Intelligent Data Analysis, 2012
While ensemble classifiers often reach high levels of predictive performance, the resulting models are opaque and hence do not allow direct interpretation. When employing methods that do generate transparent models, predictive performance typically has to be sacrificed. This paper presents a method of improving predictive performance of transparent models in the very common situation where instances to be classified, i.e., the production data, are known at the time of model building. This approach, named oracle coaching, employs a strong classifier, called an oracle, to guide the generation of a weaker, but transparent model. This is accomplished by using the oracle to predict class labels for the production data, and then applying the weaker method on this data, possibly in conjunction with the original training set. Evaluation on 30 data sets from the UCI repository shows that oracle coaching significantly improves predictive performance, measured by both accuracy and area under ROC curve, compared to using training data only. This result is shown to be robust for a variety of methods for generating the oracles and transparent models. More specifically, random forests and bagged radial basis function networks are used as oracles, while J48 and JRip are used for generating transparent models. The evaluation further shows that significantly better results are obtained when using the oracle-classified production data together with the original training data, instead of using only oracle data. An analysis of the fidelity of the transparent models to the oracles shows that performance gains can be expected from increasing oracle performance rather than from increasing fidelity. Finally, it is shown that further performance gains can be achieved by adjusting the relative weights of training data and oracle data.
Majority vote ensembles of conformal predictors
Machine Learning
We study majority vote ensembles of ε-valid conformal predictors (CP). We show that the prediction set Γ η produced as the majority vote among the prediction sets Γ ε i of k independent ε-valid CPs is also valid, for some significance level η; we provide a method to compute ε to achieve a desired η. We further indicate an error upper bound for an ensemble of correlated CPs, and derive a value ε for which such an ensemble guarantees η conservative validity. We evaluate empirically our findings, and compare them with alternative strategies for combining CPs' predictions. Keywords Conformal prediction • Ensembles • Majority vote • Error bounds 1 More precisely, a nonconformity measure is a scoring function (e.g, a classifier with probabilistic output).
Special Issue on Conformal and Probabilistic Prediction with Applications: Preface
Pattern Recognition, 2022
Feature Selection by Conformal Predictor
IFIP Advances in Information and Communication Technology, 2011
HAL is a multidisciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.
Multiple Kernel Learning for Efficient Conformal Predictions
2010
The Conformal Predictions framework is a recent development in machine learning to associate reliable measures of confidence with results in classification and regression. This framework is founded on the principles of algorithmic randomness (Kolmogorov complexity), transductive inference and hypothesis testing. While the formulation of the framework guarantees validity, the efficiency of the framework depends greatly on the choice of the classifier and appropriate kernel functions or parameters. While this framework has ...
A Cross-Conformal Predictor for Multi-label Classification
IFIP Advances in Information and Communication Technology, 2014
Unlike the typical classification setting where each instance is associated with a single class, in multi-label learning each instance is associated with multiple classes simultaneously. Therefore the learning task in this setting is to predict the subset of classes to which each instance belongs. This work examines the application of a recently developed framework called Conformal Prediction (CP) to the multi-label learning setting. CP complements the predictions of machine learning algorithms with reliable measures of confidence. As a result the proposed approach instead of just predicting the most likely subset of classes for a new unseen instance, also indicates the likelihood of each predicted subset being correct. This additional information is especially valuable in the multi-label setting where the overall uncertainty is extremely high.
Introduction to conformal predictors based on fuzzy logic classifiers
2012
In this paper, an introduction to the main steps required to develop conformal predictors based on fuzzy logic classifiers is provided. The more delicate aspect is the definition of an appropriate nonconformity score, which has to be based on the membership function to preserve the specificities of Fuzzy Logic. Various examples are introduced, to describe the main properties of fuzzy logic based conformal predictors and to compare their performance with alternative approaches. The obtained results are quite promising, since conformal predictors based on fuzzy classifiers show the potential to outperform solutions based on the nearest neighbour in terms of ambiguity, robustness and interpretability
Accurate and interpretable regression trees using oracle coaching
2014 IEEE Symposium on Computational Intelligence and Data Mining (CIDM), 2014
In many real-world scenarios, predictive models need to be interpretable, thus ruling out many machine learning techniques known to produce very accurate models, e.g., neural networks, support vector machines and all ensemble schemes. Most often, tree models or rule sets are used instead, typically resulting in significantly lower predictive performance. The overall purpose of oracle coaching is to reduce this accuracy vs. comprehensibility trade-off by producing interpretable models optimized for the specific production set at hand. The method requires production set inputs to be present when generating the predictive model, a demand fulfilled in most, but not all, predictive modeling scenarios. In oracle coaching, a highly accurate, but opaque, model is first induced from the training data. This model ("the oracle") is then used to label both the training instances and the production instances. Finally, interpretable models are trained using different combinations of the resulting data sets. In this paper, the oracle coaching produces regression trees, using neural networks and random forests as oracles. The experiments, using 32 publicly available data sets, show that the oracle coaching leads to significantly improved predictive performance, compared to standard induction. In addition, it is also shown that a highly accurate opaque model can be successfully used as a preprocessing step to reduce the noise typically present in data, even in situations where production inputs are not available. In fact, just augmenting or replacing training data with another copy of the training set, but with the predictions from the opaque model as targets, produced significantly more accurate and/or more compact regression trees.
Approximating Full Conformal Prediction at Scale via Influence Functions
Proceedings of the ... AAAI Conference on Artificial Intelligence, 2023
Conformal prediction (CP) is a wrapper around traditional machine learning models, giving coverage guarantees under the sole assumption of exchangeability; in classification problems, a CP guarantees that the error rate is at most a chosen significance level ε, irrespective of whether the underlying model is misspecified. However, the prohibitive computational costs of full CP led researchers to design scalable alternatives, which alas do not attain the same guarantees or statistical power of full CP. In this paper, we use influence functions to efficiently approximate full CP. We prove that our method is a consistent approximation of full CP, and empirically show that the approximation error becomes smaller as the training set increases; e.g., for 1, 000 training points the two methods output p-values that are < 0.001 apart: a negligible error for any practical application. Our methods enable scaling full CP to large real-world datasets. We compare our full CP approximation (ACP) to mainstream CP alternatives, and observe that our method is computationally competitive whilst enjoying the statistical predictive power of full CP. Method Prediction set ACP bird, cat, deer, frog SCP bird, deer, frog RAPS bird, cat, deer, dog, frog CV+ bird, cat, deer, dog, frog Method Prediction set ACP auto, cat, frog, horse, truck SCP auto, deer, frog, truck RAPS plane, auto, bird, deer, frog, ship, truck CV+ plane, auto, deer, frog, horse, truck Method Prediction set ACP cat, deer, frog, horse SCP cat, deer, dog, frog, horse, truck RAPS cat, deer, dog, frog, horse, truck CV+ cat, deer, dog, frog, horse
Loading Preview
Sorry, preview is currently unavailable. You can download the paper by clicking the button above.