Missing Data Imputation

This paper proposed the combination of two statistical techniques for the detection and imputation of outliers in time series data. An autoregressive integrated moving average with exogenous inputs (ARIMAX) model is used to extract the characteristics of the time series and to find the residuals. The outliers are detected by performing hypothesis testing on the extrema of the residuals and the anomalous data are imputed using another ARIMAX model. The process is performed in an iterative way because at the beginning the process, the residuals are contaminated by the anomalies and therefore, the ARIMAX model needs to be re-learned on “cleaner” data at every step. We test the algorithm using both synthetic and real data sets and we present the analysis and comments on those results.

Given the paucity of specimens available, it is necessary to extract as much information as possible for each specimen, even when only partial remains are present due to taphonomic or other destructive processes.  While different methods function differently, the goal of missing data imputation methods is to accurately estimate the missing values using the other observed values.  We selected four data imputation techniques: Hot Deck, Iterative robust model (IRMI), K nearest-neighbor (kNN=5) and the variable means, to examine which of the imputation methods performed best. A subset of Howells’ craniometric database was used. The full sample consisted of 352 individuals from four population groups (Ainu = 86; Arikara = 69; Dogon = 99; Zalavar = 98).  Twenty variables were selected to represent the entire cranium and facial skeleton. Two versions of the dataset were then created wherein values were randomly deleted from each variable so that 25% and 50% of the data were considered missing.  The same data subsets were used for each of the imputation techniques, and the efficacy of each technique was based on the smallest difference between the imputed measures and actual measures.  Correct classification rates and Mahalanobis D2 values were calculated for the original dataset with actual measures and each of the imputed datasets in order to examine the effects of imputed data on biodistance and classification. Results suggest that kNN imputation is the most accurate method. Additionally, it was found that midface variables are more accurately imputed as opposed to cranial variables.

This practical introduction is geared towards scientists who wish to employ Bayesian networks for applied research using the BayesiaLab software platform. Through numerous examples, this book illustrates how implementing Bayesian networks involves concepts from many disciplines, including computer science, probability theory, information theory, machine learning, and statistics. Each chapter explores a real-world problem domain, exploring aspects of Bayesian networks and simultaneously introducing functions of BayesiaLab. The book can serve as a self-study guide for learners and as a reference manual for advanced practitioners.

"An important step in building a wind farm is to choose the most suitable turbine. The selection of
the turbine involves a careful analysis of costs and technical parameters, among which stands out
the capacity factor, whose calculation is performed with a statistical analysis of anemometric
records. Based on wind speed records collected in São Martinho da Serra - RS, this paper
describes an approach traditionally used in that capacity factor is the criterion of identifying the
most appropriate turbine."

The imputation of missing data is often a crucial step in the analysis of survey data.This study reviews typical problems with missing data and discusses a method for the imputation of missing survey data with a large number of categorical variables which do not have a monotone missing pattern. We develop a method for constructing a monotone missing pattern that allows for imputation of categorical data in data sets with a large number of variables using a model-based MCMC approach. We report the results of imputing the missing data from a case study, using educational, sociopsychological, and socioeconomic data from the National Latino and Asian American Study (NLAAS). We report the results of multiply imputed data on a substantive logistic regression analysis predicting socioeconomic success from several educational, sociopsychological, and familial variables. We compare the results of conducting inference using a single imputed data set to those using a combined test over several imputations. Findings indicate that, for all variables in the model, all of the single tests were consistent with the combined test.

Özet: Bu araştırmada, farklı oranlarda (%15 ve %25) ve yapılarda (TROK ve ROK) oluşturulan kayıp veriler yerine farklı yöntemlerle yaklaşık değer atanması sonucu elde edilen veri setlerinin tam veri setleriyle karşılaştırılarak incelenmesi amaçlanmıştır. Bu araştırma, PISA'ya (2012) Türkiye'den katılan 15 yaş grubundaki 4848 öğrenci arasından matematik özyeterliği anketine katılan ve eksiksiz bir şekilde yanıtlayan 3129 öğrencinin puanlarından oluşan veri seti üzerinde yürütülmüştür. Söz konusu veri seti içerisinden farklı yapılar oluşturulacak şekilde farklı oranlarda veri silinerek eksik veri setleri oluşturulmuştur. Bu eksik veri setleri BM, BVA, ESE, MUA, MZMC ve RA olmak üzere altı farklı gelişmiş değer atama yöntemiyle tamamlanmıştır. Söz konusu yöntemlerle yapılan yaklaşık değer atamaları sonucu elde edilen ölçek puanları ile tam veri ölçek puanları arasındaki korelasyon değerlerinin yüksek olduğu görülmüştür. Benzer şekilde farklı yöntemlerle tamamlanmış veri setlerinden elde edilen ölçek puanları arasındaki korelasyon değerleri de yüksek bulunmuştur. Tam veri seti ile tamamlanmış veri setlerinden hesaplanan ölçek puanları arası farkların mutlak değer toplamları ve ortalamaları göz önünde bulundurulduğunda belirlenen koşullar altında en iyi çalışan yaklaşık değer atama yöntemlerinin MZMC ve BM olduğu sonucuna ulaşılmıştır.


Abstract: In this study, it is aimed to comparatively research of data sets obtained imputation for missing values that is formed by different ratios (%15 and %25) and in different structures (MCAR and MAR) with different methods. This study has been conducted on data set formed by points of 3129 students who participated in mathematics self-efficacy survey and answered it completely among 4848 students-age group of 15-who participated in PISA 2012 from Turkey. Missing data sets have been constituted by deleting data in different ratios to be constitute different structures in the data set. These data sets have been completed by six different nearby value imputation including EM, BIM, PSM, MCMC, MDIM, and RIM. Obtained data sets have been compared with full data sets by scale points of students. In the scope of the research, correlation between obtained scale points and scale points of real data has been seen quite high. Similarly, when scale points is considered, correlation of missing data imputation methods with each other have also been found quite high. Considering the difference between the totals and avarages of student scores calculated from the full data set and imputed data sets EM and MCMC is founded that the best missing data imputation methods under all conditions.

Statistics Netherlands participated in the EUREDIT project, a large international research and development project on statistical data editing and imputation that lasted from March 2000 till February 2003. The main goals of this project were the development and evaluation of new and currently used methods for data editing and imputation. In this article we describe the general approach applied by Statistics Netherlands on the two business surveys used in the EUREDIT project. In the EUREDIT project data for only one year were available. In our edit and imputation methods we therefore could not use data from a previous year and had to restrict ourselves to using only data from the data set to be edited and imputed itself. We also describe the development of our edit and imputation strategy and give results supporting the choices we have made. Finally, we provide results of our approach on the two evaluation data sets, and compare these results to the results of the other institutes participating in EUREDIT.

Most of hydrological studies are based on
statistical science. The first step in water project engineering
studies, agricultural development plans and such like is to
use of correct data and information. However, because of
various reasons, there are several gaps in available data. In
this study, for finding missing value (545 records) in our
data, we used the daily data from five pluviometry stations in
the Klang river basin between 2005 to 2015. Statistical
methods used in this study were the linear interpolation,
linear regression, trendline command in Excel software and
EM method in SPSS. The results obtained from this research
showed that the best method for singular missing data is a
linear interpolation method and the fastest way to fill the
missing gaps was the use of the SPSS software. Also, if the
aim is to find the relationship between variables and
determine the priorities, the regression method is
recommended.

EIA Seminar slides to explain ‘prediction’ to other statisticians and data analysts. Graphics are illustrative. The 3-D scatterplots (Joel Douglas) would be rotated to 2-D views for analyses, but are shown in partially rotated form here, for purposes of qualitative illustration.
[Note that a random sample can easily be drawn which is substantially "unrepresentative" of the population, perhaps especially with continuous data that have a few outsized members of a finite population. That is why model-assisted design-based sampling can be so useful. (Sometimes the model may be more important.)]
EIA Seminar, Washington, DC, USA; 06/2010

Effective control of energy storage system (ESS), supplying an ancillary service to a grid, requires effective and critical calculation of state-of-charge (SoC). Charging and discharging values from battery operations are essential in calculating the efficiency and performance of a storage system. This information can also be a key to understand and forecast peak demand performance. Missing data is a real problem in any operations system, and it appears to be more common within powers systems due to sensor and/or network malfunctioning problems. Missing data imputation techniques have evolved in power systems research using smart meter data, but little research has gone into understanding how missing data can be best handled within storage management systems. This paper builds on a year's worth of charging and discharging data collected from a real 6MW/10MWh lithium-ion storage battery deployed on the distribution network at Leighton Buzzard, UK. Using R Studio version (1.3.959-1) open-source software, eight selected imputation techniques were applied in identifying the best suited technique in replacing various missing data amounts and patterns. Findings from the study open up avenues for discussion and debate in identifying an appropriate imputation technique within the storage management context. The study also provides a pioneering lead in understanding the importance of decomposition in evaluating the right imputation technique.