Generalizability theory

Mesure Et Evaluation En Education, 2008

2018

WOS:000449986600010Purpose of the Study: Missing data are a common problem encountered while implementing measurement instruments. Yet the extent to which reliability, validity, average discrimination and difficulty of the test results are affected by the missing data has not been studied much. Since it is inevitable that missing data have an impact on the psychometric properties of measurement instruments, it was considered necessary to investigate this topic. Depending on the identified need, a simulative study was conducted on the effects of missing data on reliability. The reliability estimates were discussed in terms of generalizability theory (G theory). Research Methods: Depending on the research questions, complete data sets having different sample sizes (100, 200, 400, 1000) in weak and strong one-dimensional structures under normal distribution were produced. Missing data sets were created by deleting data at different rates (5%, 10%, 20%, 30%) randomly from the complete s...

2005

Abstract:" Given a set of random variables, it is often the case that their associations can be explained by hidden common causes. We present a set of well-defined assumptions and a provably correct algorithm that allow us to identify some of such hidden common causes.

Multivariate behavioral research, 2018

Over the years, research in the social sciences has been dominated by reporting of reliability coefficients that fail to account for key sources of measurement error. Use of these coefficients, in turn, to correct for measurement error can hinder scientific progress by misrepresenting true relationships among the underlying constructs being investigated. In the research reported here, we addressed these issues using generalizability theory (G-theory) in both traditional and new ways to account for the three key sources of measurement error (random-response, specific-factor, and transient) that affect scores from objectively scored measures. Results from 20 widely used measures of personality, self-concept, and socially desirable responding showed that conventional indices consistently misrepresented reliability and relationships among psychological constructs by failing to account for key sources of measurement error and correlated transient errors within occasions. The results furt...

Psychological Methods, 2017

Although widely recognized as a comprehensive framework for representing score reliability, generalizability theory (G-theory), despite its potential benefits, has been used sparingly in reporting of results for measures of individual differences. In this article, we highlight many valuable ways that G-theory can be used to quantify, evaluate, and improve psychometric properties of scores. Our illustrations encompass assessment of overall reliability, percentages of score variation accounted for by individual sources of measurement error, dependability of cut-scores for decision making, estimation of reliability and dependability for changes made to measurement procedures, disattenuation of validity coefficients for measurement error, and linkages of G-theory with classical test theory and structural equation modeling. We also identify computer packages for performing G-theory analyses, most of which can be obtained free of charge, and describe how they compare with regard to data input requirements, ease of use, complexity of designs supported, and output produced.

Frontiers in Psychology, 2015

Accountability and Educational Improvement

In educational research, comparisons are often made of groups or of the development of various (latent) constructs over time (e.g. teaching quality in different countries or different groups’ (girls vs. boys) perceptions of teaching quality). However, before the results of such comparisons can be accurately interpreted, measurement invariance (MI) of the constructs under investigation needs to be established to ensure their meaning remains consistent across groups, subjects, or assessment points. Thus, if mean level changes are to be compared between groups, scalar factorial invariance needs to be established. In this chapter, we investigate and discuss how results of MI analyses should be interpreted and whether they should be reported on with regard to contents. Using data from the well-known Programme for International Student Assessment (PISA) study on teaching quality, we introduce an approach to examining the conditions under which comparison among cultural groups is possible ...

1986

Animal Behaviour, 2006

Psychometrika, 1978

Four approximate tests are considered for repeated measurement designs in which observations are multivariate normal with arbitrary covariance matrices. In these tests traditional withinsubject mean square ratios are compared with critical values derived from F distributions with adjusted degrees of freedom. Two of them--the g approximate and the improved general approximate (IGA) tests--behave adequately in terms of Type I error. Generally, the IGA test functions better than the/" approximate test, however the latter involves less computations. In regards to power, the IGA test may compete with one multivariate procedure when the assumptions of the latter are tenable.