Several parametric and nonparametric tests of independence between two random vectors are availab... more Several parametric and nonparametric tests of independence between two random vectors are available in the literature. But, many of them perform poorly for high dimensional data and are not applicable when the dimension exceeds the sample size. In this article, we propose some tests based on ranks of nearest neighbors, which can be conveniently used in high dimension, low sample size situations. Several simulated and real data sets are analyzed to show the utility of the proposed tests. Codes for implementation of the proposed tests are available as supplementary materials.
Journal of the Royal Statistical Society Series B: Statistical Methodology
Covariance estimation is ubiquitous in functional data analysis. Yet, the case of functional obse... more Covariance estimation is ubiquitous in functional data analysis. Yet, the case of functional observations over multidimensional domains introduces computational and statistical challenges, rendering the standard methods effectively inapplicable. To address this problem, we introduce Covariance Networks (CovNet) as a modelling and estimation tool. The CovNet model is universal—it can be used to approximate any covariance up to desired precision. Moreover, the model can be fitted efficiently to the data and its neural network architecture allows us to employ modern computational tools in the implementation. The CovNet model also admits a closed-form eigendecomposition, which can be computed efficiently, without constructing the covariance itself. This facilitates easy storage and subsequent manipulation of a covariance in the context of the CovNet. We establish consistency of the proposed estimator and derive its rate of convergence. The usefulness of the proposed method is demonstrat...
In high dimension, low sample size (HDLSS) settings, classifiers based on Euclidean distances lik... more In high dimension, low sample size (HDLSS) settings, classifiers based on Euclidean distances like the nearest neighbor classifier and the average distance classifier perform quite poorly if differences between locations of the underlying populations get masked by scale differences. To rectify this problem, several modifications of these classifiers have been proposed in the literature. However, existing methods are confined to location and scale differences only, and they often fail to discriminate among populations differing outside of the first two moments. In this article, we propose some simple transformations of these classifiers resulting in improved performance even when the underlying populations have the same location and scale. We further propose a generalization of these classifiers based on the idea of grouping of variables. High-dimensional behavior of the proposed classifiers is studied theoretically. Numerical experiments with a variety of simulated examples as well as an extensive analysis of benchmark data sets from three different databases exhibit advantages of the proposed methods.
When the competing classes in a classification problem are not of comparable size, many popular c... more When the competing classes in a classification problem are not of comparable size, many popular classifiers exhibit a bias towards larger classes, and the nearest neighbor classifier is no exception. To take care of this problem, we develop a statistical method for nearest neighbor classification based on such imbalanced data sets. First, we construct a classifier for the binary classification problem and then extend it for classification problems involving more than two classes. Unlike the existing oversampling or undersampling methods, our proposed classifiers do not need to generate any pseudo observations or remove any existing observations, hence the results are exactly reproducible. We establish the Bayes risk consistency of these classifiers under appropriate regularity conditions. Their superior performance over the existing methods is amply demonstrated by analyzing several benchmark data sets.
We propose a new model-free feature screening method based on energy distances for ultrahigh-dime... more We propose a new model-free feature screening method based on energy distances for ultrahigh-dimensional binary classification problems. Unlike existing methods, the cut-off involved in our procedure is data adaptive. With a high probability, the proposed method retains only relevant features after discarding all the noise variables. The proposed screening method is also extended to identify pairs of variables that are marginally undetectable, but have differences in their joint distributions. Finally, we build a classifier which maintains coherence between the proposed feature selection criteria and discrimination method, and also establish its risk consistency. An extensive numerical study with simulated data sets and real benchmark data sets show clear and convincing advantages of our classifier over the state-of-the-art methods.
Journal of Statistical Computation and Simulation, 2021
Testing mutual independence among several random vectors of arbitrary dimensions is a challenging... more Testing mutual independence among several random vectors of arbitrary dimensions is a challenging problem in Statistics, and it has gained considerable interest in recent years. In this article, we propose some nonparametric tests based on different notions of ranks of nearest neighbour. These proposed tests can be conveniently used for high dimensional data, even when the dimensions of the random vectors are larger than the sample size. We investigate the performance of these tests on several simulated and real data sets and also use them in identifying causal relationships among the random vectors. Our numerical results show that they can outperform state-of-the-art tests in a wide variety of examples.
In high dimension, low sample size (HDLSS) settings, the nearest neighbor classifier based on the... more In high dimension, low sample size (HDLSS) settings, the nearest neighbor classifier based on the Euclidean distance yields poor performance if differences between the locations get masked by the scale differences. To rectify this problem, several modifications of the nearest neighbor classifier have been proposed in the literature. However, these existing methods often fail to discriminate among populations having same locations and scales. In this article, we propose some simple modifications of the nearest neighbor classifier using a new class of dissimilarity measures. The resulting classifiers perform quite well even when the underlying populations have no differences in their locations and scales. Some of these classifiers can also discriminate among populations having the same one-dimensional marginal distributions. High-dimensional behavior of the proposed classifiers are studied theoretically. Numerical experiments with a variety of simulated as well as real data sets clear...
In high dimension, low sample size (HDLSS) settings, Euclidean distance based classifiers suffer ... more In high dimension, low sample size (HDLSS) settings, Euclidean distance based classifiers suffer from curse of dimensionality if the competing populations are similar in terms of their location and scale parameters. In this article, we propose a classifier which can discriminate between populations having different marginal distributions. A generalization of the proposed classifier has been developed to differentiate between populations having different joint distributions for the component variables. Performance of the proposed classifiers are demonstrated using a variety of simulated data sets.
In this article, we develop and investigate a new classifier based on features extracted using sp... more In this article, we develop and investigate a new classifier based on features extracted using spatial depth. Our construction is based on fitting a generalized additive model to the posterior probabilities of the different competing classes. To cope with possible multi-modal as well as non-elliptic population distributions, we develop a localized version of spatial depth and use that with varying degrees of localization to build the classifier. Final classification is done by aggregating several posterior probability estimates each of which is obtained using localized spatial depth with a fixed scale of localization. The proposed classifier can be conveniently used even when the dimension is larger than the sample size, and its good discriminatory power for such data has been established using theoretical as well as numerical results.
Popular clustering algorithms based on usual distance functions (e.g., Euclidean distance) often ... more Popular clustering algorithms based on usual distance functions (e.g., Euclidean distance) often suffer in high dimension, low sample size (HDLSS) situations, where concentration of pairwise distances has adverse effects on their performance. In this article, we use a dissimilarity measure based on the data cloud, called MADD, which takes care of this problem. MADD uses the distance concentration property to its advantage and, as a result, clustering algorithms based on MADD usually perform better for high dimensional data. We also address the problem of estimating the number of clusters. This is a very challenging problem in cluster analysis, and several algorithms have been proposed for it. We show that many of these existing algorithms have superior performance in high dimensions when MADD is used instead of Euclidean distance. We also construct a new estimator based on penalized Dunn index and prove its consistency in HDLSS asymptotic regime, where the sample size remains fixed ...
Testing for equality of two high-dimensional distributions is a challenging problem, and this bec... more Testing for equality of two high-dimensional distributions is a challenging problem, and this becomes even more challenging when the sample size is small. Over the last few decades, several graph-based two-sample tests have been proposed in the literature, which can be used for data of arbitrary dimensions. Most of these test statistics are computed using pairwise Euclidean distances among the observations. But, due to concentration of pairwise Euclidean distances, these tests have poor performance in many high-dimensional problems. Some of them can have powers even below the nominal level when the scale-difference between two distributions dominates the location-difference. To overcome these limitations, we introduce some new dissimilarity indices and use them to modify some popular graph-based tests. These modified tests use the distance concentration phenomenon to their advantage, and as a result, they outperform the corresponding tests based on the Euclidean distance in a wide variety of examples. We establish the high-dimensional consistency of these modified tests under fairly general conditions. Analyzing several simulated as well as real data sets, we demonstrate their usefulness in high dimension, low sample size situations.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019
Popular clustering algorithms based on usual distance functions (e.g., Euclidean distance) often ... more Popular clustering algorithms based on usual distance functions (e.g., Euclidean distance) often suffer in high dimension, low sample size (HDLSS) situations, where concentration of pairwise distances has adverse effects on their performance. In this article, we use a dissimilarity measure based on the data cloud, called MADD, which takes care of this problem. MADD uses the distance concentration phenomenon to its advantage, and as a result, clustering algorithms based on MADD usually perform better for high dimensional data. Using theoretical and numerical results, we amply demonstrate it in this article. We also address the problem of estimating the number of clusters. This is a very challenging problem in cluster analysis, and several algorithms have been proposed for it. We show that many of these existing algorithms have superior performance in high dimensions when MADD is used instead of the Euclidean distance. We also construct a new estimator based on penalized Dunn index and prove its consistency in the HDLSS asymptotic regime, where the sample size remains fixed and the dimension grows to infinity. Several simulated and real data sets are analyzed to demonstrate the importance of MADD for cluster analysis of high dimensional data.
Several parametric and nonparametric tests of independence between two random vectors are availab... more Several parametric and nonparametric tests of independence between two random vectors are available in the literature. But, many of them perform poorly for high dimensional data and are not applicable when the dimension exceeds the sample size. In this article, we propose some tests based on ranks of nearest neighbors, which can be conveniently used in high dimension, low sample size situations. Several simulated and real data sets are analyzed to show the utility of the proposed tests. Codes for implementation of the proposed tests are available as supplementary materials.
The multivariate one-sample location problem is well studied in the literature, and several tests... more The multivariate one-sample location problem is well studied in the literature, and several tests are available for it. But most of the existing tests perform poorly for high dimensional data, and many of them are not applicable when the dimension exceeds the sample size. In this article, we develop and investigate some nonparametric one-sample tests based on functions of interpoint distances. These proposed tests can be conveniently used in high dimension low sample size (HDLSS) situations, and good power properties of these tests for HDLSS data have been established using both theoretical as well as numerical results.
Journal of Statistical Planning and Inference, 2016
Abstract Several nonparametric methods are available in the literature to test the independence b... more Abstract Several nonparametric methods are available in the literature to test the independence between two random vectors. But, many of them perform poorly for high dimensional data and are not applicable when the dimension of one of these vectors exceeds the sample size. Moreover, most of these tests are not distribution-free in the general multivariate set up. Recently, Heller et al. (2012) proposed a test of independence, which is distribution-free and can be conveniently used even when the dimensions are larger than the sample size. In this article, we point out some limitations of this test and propose some modifications to overcome them retaining its distribution-free property. Some simulated and real data sets are analyzed to demonstrate the utility of our proposed modifications.
Testing for mutual independence among several random vectors is a challenging problem, and in rec... more Testing for mutual independence among several random vectors is a challenging problem, and in recent years, it has gained significant attention in statistics and machine learning literature. Most of the existing tests of independence deal with only two random vectors, and they do not have straightforward generalizations for testing mutual independence among more than two random vectors of arbitrary dimensions. On the other hand, there are various tests for mutual independence among several random variables, but these univariate tests do not have natural multivariate extensions. In this article, we propose two general recipes, one based on inter-point distances and the other based on linear projections, for multivariate extensions of these univariate tests. Under appropriate regularity conditions, these resulting tests turn out to be consistent whenever we have consistency for the corresponding univariate tests. We carry out extensive numerical studies to compare the empirical performance of these proposed methods with the state-of-the-art methods.
Journal of Statistical Computation and Simulation, Feb 4, 2021
Testing mutual independence among several random vectors of arbitrary dimensions is a challenging... more Testing mutual independence among several random vectors of arbitrary dimensions is a challenging problem in Statistics, and it has gained considerable interest in recent years. In this article, we propose some nonparametric tests based on different notions of ranks of nearest neighbour. These proposed tests can be conveniently used for high dimensional data, even when the dimensions of the random vectors are larger than the sample size. We investigate the performance of these tests on several simulated and real data sets and also use them in identifying causal relationships among the random vectors. Our numerical results show that they can outperform state-of-the-art tests in a wide variety of examples.
Several parametric and nonparametric tests of independence between two random vectors are availab... more Several parametric and nonparametric tests of independence between two random vectors are available in the literature. But, many of them perform poorly for high dimensional data and are not applicable when the dimension exceeds the sample size. In this article, we propose some tests based on ranks of nearest neighbors, which can be conveniently used in high dimension, low sample size situations. Several simulated and real data sets are analyzed to show the utility of the proposed tests. Codes for implementation of the proposed tests are available as supplementary materials.
Journal of the Royal Statistical Society Series B: Statistical Methodology
Covariance estimation is ubiquitous in functional data analysis. Yet, the case of functional obse... more Covariance estimation is ubiquitous in functional data analysis. Yet, the case of functional observations over multidimensional domains introduces computational and statistical challenges, rendering the standard methods effectively inapplicable. To address this problem, we introduce Covariance Networks (CovNet) as a modelling and estimation tool. The CovNet model is universal—it can be used to approximate any covariance up to desired precision. Moreover, the model can be fitted efficiently to the data and its neural network architecture allows us to employ modern computational tools in the implementation. The CovNet model also admits a closed-form eigendecomposition, which can be computed efficiently, without constructing the covariance itself. This facilitates easy storage and subsequent manipulation of a covariance in the context of the CovNet. We establish consistency of the proposed estimator and derive its rate of convergence. The usefulness of the proposed method is demonstrat...
In high dimension, low sample size (HDLSS) settings, classifiers based on Euclidean distances lik... more In high dimension, low sample size (HDLSS) settings, classifiers based on Euclidean distances like the nearest neighbor classifier and the average distance classifier perform quite poorly if differences between locations of the underlying populations get masked by scale differences. To rectify this problem, several modifications of these classifiers have been proposed in the literature. However, existing methods are confined to location and scale differences only, and they often fail to discriminate among populations differing outside of the first two moments. In this article, we propose some simple transformations of these classifiers resulting in improved performance even when the underlying populations have the same location and scale. We further propose a generalization of these classifiers based on the idea of grouping of variables. High-dimensional behavior of the proposed classifiers is studied theoretically. Numerical experiments with a variety of simulated examples as well as an extensive analysis of benchmark data sets from three different databases exhibit advantages of the proposed methods.
When the competing classes in a classification problem are not of comparable size, many popular c... more When the competing classes in a classification problem are not of comparable size, many popular classifiers exhibit a bias towards larger classes, and the nearest neighbor classifier is no exception. To take care of this problem, we develop a statistical method for nearest neighbor classification based on such imbalanced data sets. First, we construct a classifier for the binary classification problem and then extend it for classification problems involving more than two classes. Unlike the existing oversampling or undersampling methods, our proposed classifiers do not need to generate any pseudo observations or remove any existing observations, hence the results are exactly reproducible. We establish the Bayes risk consistency of these classifiers under appropriate regularity conditions. Their superior performance over the existing methods is amply demonstrated by analyzing several benchmark data sets.
We propose a new model-free feature screening method based on energy distances for ultrahigh-dime... more We propose a new model-free feature screening method based on energy distances for ultrahigh-dimensional binary classification problems. Unlike existing methods, the cut-off involved in our procedure is data adaptive. With a high probability, the proposed method retains only relevant features after discarding all the noise variables. The proposed screening method is also extended to identify pairs of variables that are marginally undetectable, but have differences in their joint distributions. Finally, we build a classifier which maintains coherence between the proposed feature selection criteria and discrimination method, and also establish its risk consistency. An extensive numerical study with simulated data sets and real benchmark data sets show clear and convincing advantages of our classifier over the state-of-the-art methods.
Journal of Statistical Computation and Simulation, 2021
Testing mutual independence among several random vectors of arbitrary dimensions is a challenging... more Testing mutual independence among several random vectors of arbitrary dimensions is a challenging problem in Statistics, and it has gained considerable interest in recent years. In this article, we propose some nonparametric tests based on different notions of ranks of nearest neighbour. These proposed tests can be conveniently used for high dimensional data, even when the dimensions of the random vectors are larger than the sample size. We investigate the performance of these tests on several simulated and real data sets and also use them in identifying causal relationships among the random vectors. Our numerical results show that they can outperform state-of-the-art tests in a wide variety of examples.
In high dimension, low sample size (HDLSS) settings, the nearest neighbor classifier based on the... more In high dimension, low sample size (HDLSS) settings, the nearest neighbor classifier based on the Euclidean distance yields poor performance if differences between the locations get masked by the scale differences. To rectify this problem, several modifications of the nearest neighbor classifier have been proposed in the literature. However, these existing methods often fail to discriminate among populations having same locations and scales. In this article, we propose some simple modifications of the nearest neighbor classifier using a new class of dissimilarity measures. The resulting classifiers perform quite well even when the underlying populations have no differences in their locations and scales. Some of these classifiers can also discriminate among populations having the same one-dimensional marginal distributions. High-dimensional behavior of the proposed classifiers are studied theoretically. Numerical experiments with a variety of simulated as well as real data sets clear...
In high dimension, low sample size (HDLSS) settings, Euclidean distance based classifiers suffer ... more In high dimension, low sample size (HDLSS) settings, Euclidean distance based classifiers suffer from curse of dimensionality if the competing populations are similar in terms of their location and scale parameters. In this article, we propose a classifier which can discriminate between populations having different marginal distributions. A generalization of the proposed classifier has been developed to differentiate between populations having different joint distributions for the component variables. Performance of the proposed classifiers are demonstrated using a variety of simulated data sets.
In this article, we develop and investigate a new classifier based on features extracted using sp... more In this article, we develop and investigate a new classifier based on features extracted using spatial depth. Our construction is based on fitting a generalized additive model to the posterior probabilities of the different competing classes. To cope with possible multi-modal as well as non-elliptic population distributions, we develop a localized version of spatial depth and use that with varying degrees of localization to build the classifier. Final classification is done by aggregating several posterior probability estimates each of which is obtained using localized spatial depth with a fixed scale of localization. The proposed classifier can be conveniently used even when the dimension is larger than the sample size, and its good discriminatory power for such data has been established using theoretical as well as numerical results.
Popular clustering algorithms based on usual distance functions (e.g., Euclidean distance) often ... more Popular clustering algorithms based on usual distance functions (e.g., Euclidean distance) often suffer in high dimension, low sample size (HDLSS) situations, where concentration of pairwise distances has adverse effects on their performance. In this article, we use a dissimilarity measure based on the data cloud, called MADD, which takes care of this problem. MADD uses the distance concentration property to its advantage and, as a result, clustering algorithms based on MADD usually perform better for high dimensional data. We also address the problem of estimating the number of clusters. This is a very challenging problem in cluster analysis, and several algorithms have been proposed for it. We show that many of these existing algorithms have superior performance in high dimensions when MADD is used instead of Euclidean distance. We also construct a new estimator based on penalized Dunn index and prove its consistency in HDLSS asymptotic regime, where the sample size remains fixed ...
Testing for equality of two high-dimensional distributions is a challenging problem, and this bec... more Testing for equality of two high-dimensional distributions is a challenging problem, and this becomes even more challenging when the sample size is small. Over the last few decades, several graph-based two-sample tests have been proposed in the literature, which can be used for data of arbitrary dimensions. Most of these test statistics are computed using pairwise Euclidean distances among the observations. But, due to concentration of pairwise Euclidean distances, these tests have poor performance in many high-dimensional problems. Some of them can have powers even below the nominal level when the scale-difference between two distributions dominates the location-difference. To overcome these limitations, we introduce some new dissimilarity indices and use them to modify some popular graph-based tests. These modified tests use the distance concentration phenomenon to their advantage, and as a result, they outperform the corresponding tests based on the Euclidean distance in a wide variety of examples. We establish the high-dimensional consistency of these modified tests under fairly general conditions. Analyzing several simulated as well as real data sets, we demonstrate their usefulness in high dimension, low sample size situations.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019
Popular clustering algorithms based on usual distance functions (e.g., Euclidean distance) often ... more Popular clustering algorithms based on usual distance functions (e.g., Euclidean distance) often suffer in high dimension, low sample size (HDLSS) situations, where concentration of pairwise distances has adverse effects on their performance. In this article, we use a dissimilarity measure based on the data cloud, called MADD, which takes care of this problem. MADD uses the distance concentration phenomenon to its advantage, and as a result, clustering algorithms based on MADD usually perform better for high dimensional data. Using theoretical and numerical results, we amply demonstrate it in this article. We also address the problem of estimating the number of clusters. This is a very challenging problem in cluster analysis, and several algorithms have been proposed for it. We show that many of these existing algorithms have superior performance in high dimensions when MADD is used instead of the Euclidean distance. We also construct a new estimator based on penalized Dunn index and prove its consistency in the HDLSS asymptotic regime, where the sample size remains fixed and the dimension grows to infinity. Several simulated and real data sets are analyzed to demonstrate the importance of MADD for cluster analysis of high dimensional data.
Several parametric and nonparametric tests of independence between two random vectors are availab... more Several parametric and nonparametric tests of independence between two random vectors are available in the literature. But, many of them perform poorly for high dimensional data and are not applicable when the dimension exceeds the sample size. In this article, we propose some tests based on ranks of nearest neighbors, which can be conveniently used in high dimension, low sample size situations. Several simulated and real data sets are analyzed to show the utility of the proposed tests. Codes for implementation of the proposed tests are available as supplementary materials.
The multivariate one-sample location problem is well studied in the literature, and several tests... more The multivariate one-sample location problem is well studied in the literature, and several tests are available for it. But most of the existing tests perform poorly for high dimensional data, and many of them are not applicable when the dimension exceeds the sample size. In this article, we develop and investigate some nonparametric one-sample tests based on functions of interpoint distances. These proposed tests can be conveniently used in high dimension low sample size (HDLSS) situations, and good power properties of these tests for HDLSS data have been established using both theoretical as well as numerical results.
Journal of Statistical Planning and Inference, 2016
Abstract Several nonparametric methods are available in the literature to test the independence b... more Abstract Several nonparametric methods are available in the literature to test the independence between two random vectors. But, many of them perform poorly for high dimensional data and are not applicable when the dimension of one of these vectors exceeds the sample size. Moreover, most of these tests are not distribution-free in the general multivariate set up. Recently, Heller et al. (2012) proposed a test of independence, which is distribution-free and can be conveniently used even when the dimensions are larger than the sample size. In this article, we point out some limitations of this test and propose some modifications to overcome them retaining its distribution-free property. Some simulated and real data sets are analyzed to demonstrate the utility of our proposed modifications.
Testing for mutual independence among several random vectors is a challenging problem, and in rec... more Testing for mutual independence among several random vectors is a challenging problem, and in recent years, it has gained significant attention in statistics and machine learning literature. Most of the existing tests of independence deal with only two random vectors, and they do not have straightforward generalizations for testing mutual independence among more than two random vectors of arbitrary dimensions. On the other hand, there are various tests for mutual independence among several random variables, but these univariate tests do not have natural multivariate extensions. In this article, we propose two general recipes, one based on inter-point distances and the other based on linear projections, for multivariate extensions of these univariate tests. Under appropriate regularity conditions, these resulting tests turn out to be consistent whenever we have consistency for the corresponding univariate tests. We carry out extensive numerical studies to compare the empirical performance of these proposed methods with the state-of-the-art methods.
Journal of Statistical Computation and Simulation, Feb 4, 2021
Testing mutual independence among several random vectors of arbitrary dimensions is a challenging... more Testing mutual independence among several random vectors of arbitrary dimensions is a challenging problem in Statistics, and it has gained considerable interest in recent years. In this article, we propose some nonparametric tests based on different notions of ranks of nearest neighbour. These proposed tests can be conveniently used for high dimensional data, even when the dimensions of the random vectors are larger than the sample size. We investigate the performance of these tests on several simulated and real data sets and also use them in identifying causal relationships among the random vectors. Our numerical results show that they can outperform state-of-the-art tests in a wide variety of examples.
Uploads
Papers by Soham Sarkar