Skip to main content

Joni Lakin

Auburn University, College of Education, Faculty Member

Followers

134

Following

12

Co-author

1

Public Views

Christine Schnittka

Auburn University

Southern University

Antonella Carrozzini

Università Del Salento

saeedeh ziaeefard

Michigan Technological University

Danielle Boulden

North Carolina State University

Jacob Noel-Storr

Arizona State University

Kathleen Schenkel

Angela Calabrese Barton

University of Michigan

Hong Kong Polytechnic University

InterestsView All (16)

Uploads

Papers by Joni Lakin

Science STUFF (Science, Society and Technology for Underrepresented Future Fabricators): Curriculum for future makers and designers

by Christine Schnittka and Joni Lakin

This paper documents the year-long process of developing one curriculum module out of several use... more This paper documents the year-long process of developing one curriculum module out of several used in formal and informal settings in a rural southern US state. The goal was to develop engineering design-infused STEM activities that used everyday materials, related to students' lives, had a pro-social focus, met state science standards, and helped youth develop STEM literacy and skills. This process of curriculum design falls into the category of " design experiments " as described by Brown (1992), Collins (1992) and others. Over 375 students were exposed to the Solar Landmine Detectors curriculum over the course of a year and the module was continuously revised based on observations of the students.

Test directions as a critical component of test design: Best practices and the impact of examinee characteristics

The purpose of test directions is to familiarize examinees with a test so that they respond to it... more The purpose of test directions is to familiarize examinees with a test so that they respond to items in the manner intended. However, changes in educational measurement as well as the U.S. student population present new challenges to test directions and increase the impact that differential familiarity could have on the validity of test score interpretations. This article reviews the literature on best practices for the development of test directions as well as documenting differences in test familiarity for culturally and linguistically diverse students that could be addressed with test directions and practice. The literature indicates that choice of practice items and feedback are critical in the design of test directions and that more extensive practice opportunities may be required to reduce group differences in test familiarity. As increasingly complex and rich item formats are introduced in next-generation assessments, test directions become a critical part of test design and validity.

Sex Differences in Fluid Reasoning: Manifest and Latent Estimates from the Cognitive Abilities Test

Assessing peer problematic behavior and related training experiences

Sex differences in reasoning abilities: Surprising evidence that male-female ratios in the tails of the quantitative reasoning distribution have increased

Sex differences in cognitive abilities, particularly at the extremes of ability distributions, ha... more Sex differences in cognitive abilities, particularly at the extremes of ability distributions, have important implications for the participation of men and women in highly valued and technical career fields. Although negligible mean differences have been found in many domains, differences in variability and high ratios of males to females in the tails of the ability distribution have been found in a number of studies and across domains. A few studies have also observed trends over time, most noting decreasing ratios of boys to girls in the highest levels of mathematics test performance. In this study, sex differences in means, variances, and ratios were evaluated in four cohorts (1984, 1992, 2000, and 2011) in verbal, quantitative, and nonverbal/figural reasoning domains as measured by the Cognitive Abilities Test. Samples included U.S. students in grades 3-11. Overall, the results were consistent with previous research, showing small mean differences in the three domains, but considerably greater variability for males. The most surprising finding was that, contrary to related research, the ratio of males to females in the upper tail of the quantitative reasoning distribution seemed to increase over time. Explanations for this finding are explored.

Evaluating growth for ELL students: Implications for accountability policies

In recent years, many U.S. states have introduced growth models as part of their educational acco... more In recent years, many U.S. states have introduced growth models as part of their educational accountability systems. Although the validity and fairness of growth-based accountability models have been evaluated for the general population, the impact of those models for English language learner (ELL) students, a growing segment of the student population, has not received sufficient attention. We evaluated three commonly used growth models: value tables or transition matrices, projection models, and student growth percentiles (SGP). The value table model identified more ELL students as on-track to proficiency, but with lower accuracy for ELL students. The projection and SGP models were more accurate overall, but classified the fewest ELL students as on-track and were less likely to identify ELL students who would later be proficient. We found that each model had significant trade-offs in terms of the decisions made for ELL students. These findings should be replicated in additional state contexts and considered in the development of future growth-based accountability policies.

Assessing the cognitive abilities of culturally and linguistically diverse students: Predictive validity of verbal, quantitative, and nonverbal batteries

Verbal and quantitative reasoning tests provide valuable information about cognitive abilities th... more Verbal and quantitative reasoning tests provide valuable information about cognitive abilities that are important to academic success. Information about these abilities may be particularly valuable to teachers of students who are English-language learners (ELL), because leveraging reasoning skills to support comprehension is a critical aptitude for their academic success.

Multidimensional ability tests and culturally and linguistically diverse students: Evidence of measurement invariance

Ability tests are used by teachers to provide additional context for interpreting student achieve... more Ability tests are used by teachers to provide additional context for interpreting student achievement and as a tool for differentiating instruction to the cognitive strengths and weaknesses of students. Tests that provide the most useful information for this purpose measure school-related content domains including verbal and quantitative reasoning. However, there is concern that verbal content affects validity for culturally and linguistically diverse students. In this study, the structure of a multidimensional ability test of verbal, quantitative, and nonverbal reasoning skills was explored in three groups of students who varied in language and cultural background. Configural invariance and invariant factor loadings were supported, but the verbal battery showed weaker relationships to the other batteries and reduced variability for English learners. Results indicate that battery-level scores are appropriate for all students, but that accounting for educational opportunity may be required for interpreting scores.

Investigating ESL students’ performance on outcomes assessments in higher education

Outcomes assessments are gaining great attention in higher education due to increased demand for ... more Outcomes assessments are gaining great attention in higher education due to increased demand for accountability. These assessments are widely used by U.S. higher education institutions to measure students' college-level knowledge and skills, including students who speak English as a second language (ESL). For the past decade, the increasing number of ESL students has changed the landscape of U.S. higher education. However, little research exists documenting how ESL students perform on outcomes assessments. In this study, we investigated ESL students' performance on the ETS Proficiency Profile in terms of factor structure, predictive validity, and differential item functioning (DIF). The test showed partial measurement invariance between ESL and non-ESL students, consistent predictive validity, and few examples of DIF. The results suggest the critical need for consideration of language background in outcomes assessment research in higher education.

Distinguishing verbal, quantitative, and nonverbal facets of fluid intelligence in young students

Measures of broad fluid abilities including verbal, quantitative, and figural reasoning are commo... more Measures of broad fluid abilities including verbal, quantitative, and figural reasoning are commonly used in the K-12 school context for a variety of purposes. However, it is difficult to differentiate these domains for young children (grades K-2) who lack language and mathematics literacy. This study evaluated the latent factor structure of a picture-based test of broad fluid reasoning abilities using a bifactor model to separate general and broad domain factors in a large representative sample of U.S. school children. Results showed substantial evidence that picturebased item formats can distinguish between general and domain-specific fluid reasoning abilities.

Multigroup Generalizability Analysis of Verbal, Quantitative, and Nonverbal Ability Tests for Culturally and Linguistically Diverse Students

Educational and Psychological Measurement, 2012

... Joni M. Lakin1 and Emily R. Lai2 ... of such students in the US school system (American Feder... more

The Predictive Accuracy of Verbal, Quantitative, and Nonverbal Reasoning Tests: Consequences for Talent Identification and Program Diversity

Journal for The Education of The Gifted, 2011

Verbal and quantitative reasoning tests provide valuable information about cognitive abilities th... more Verbal and quantitative reasoning tests provide valuable information about cognitive abilities that are important to academic success. Information about these abilities may be particularly valuable to teachers of students who are English-language learners (ELL), because leveraging reasoning skills to support comprehension is a critical aptitude for their academic success.

Nonverbal test scores as one component of an identification system: Integrating ability, achievement, and teacher ratings

Identifying Academically Gifted English- Language Learners Using Nonverbal Tests A Comparison of the Raven, NNAT, and CogAT

In this study, the authors compare the validity of three nonverbal tests for the purpose of ident... more In this study, the authors compare the validity of three nonverbal tests for the purpose of identifying academically gifted English-language learners (ELLs). Participants were 1,198 elementary children (approximately 40% ELLs). All were administered the Raven Standard Progressive Matrices (Raven), the Naglieri Nonverbal Ability Test (NNAT), and Form 6 of the Cognitive Abilities Test (CogAT). Results show that the U.S. national norms for the Raven substantially overestimate the number of high-scoring children; that because of errors in norming, the NNAT overestimates the number of both high-scoring and low-scoring children; that primary-level ELL children score especially poorly on the NNAT; that the standard error of measurement was twice as large for the NNAT as for the Raven or the CogAT; that ELL children scored .5 to .67 standard deviations lower than non-ELL children on the three nonverbal tests; and that none of the nonverbal tests predict achievement for ELL students very well.

Assessing the Cognitive Abilities of Culturally and Linguistically Diverse Students: Predictive Validity of Verbal, Quantitative, and Nonverbal Test

Verbal and quantitative reasoning tests provide valuable information about cognitive abilities th... more Verbal and quantitative reasoning tests provide valuable information about cognitive abilities that are important to academic success. Information about these abilities may be particularly valuable to teachers of students who are English-language learners (ELL), because leveraging reasoning skills to support comprehension is a critical aptitude for their academic success.

The commonality of extreme discrepancies in the ability profiles of academically gifted students

Extreme discrepancies in abilities are more common among the most and least able students than am... more Extreme discrepancies in abilities are more common among the most and least able students than among average ability children. Therefore, procedures for identifying gifted children that deliberately or inadvertently rely on a composite score that averages across ability domains will exclude many children who reason exceptionally well in particular symbol systems. In this article, we first discuss general issues in the measurement of ability profiles. We then introduce a method for categorizing score profiles and finally document the reliability and stability of score profiles using the 2000 standardization data of the Cognitive Abilities Test .

Consistencies in Sex Differences on the Cognitive Abilities Test across Countries, Grades, and Cohorts

Recently, Strand, Deary, and Smith (2006) reported sex differences for over 320,000 students 11-1... more Recently, Strand, Deary, and Smith (2006) reported sex differences for over 320,000 students 11-12 years old on the Cognitive Abilities Test. Although mean differences were small, score distributions for males were more variable. Males were also overrepresented at the upper and lower extremes of the score distributions on the Quantitative and Nonverbal batteries and at the lower extreme of the Verbal Battery. However, it is unclear whether these results are unique to the U.K. or whether they would be seen at other grades, cohorts, or forms of the same test. To investigate this, we replicated and expanded their results by comparing the distributions of scores for males and females from three standardizations of the U.S. version of the test. Changes in the proportions of males and females at each score level were examined across countries (U.K. versus U.S.), grades (3-11) and cohorts/test forms (1984, 1992, and 2000). The results showed an astonishing consistency in sex differences across countries, age cohorts, and test forms.

Test Directions as a Critical Component of Test Design: Best Practices and the Impact of Examinee Characteristic

The purpose of test directions is to familiarize examinees with a test so that they respond to it... more The purpose of test directions is to familiarize examinees with a test so that they respond to items in the manner intended. However, changes in educational measurement as well as the U.S. student population present new challenges to test directions and increase the impact that differential familiarity could have on the validity of test score interpretations. This article reviews the literature on best practices for the development of test directions as well as documenting differences in test familiarity for culturally and linguistically diverse students that could be addressed with test directions and practice. The literature indicates that choice of practice items and feedback are critical in the design of test directions and that more extensive practice opportunities may be required to reduce group differences in test familiarity. As increasingly complex and rich item formats are introduced in next-generation assessments, test directions become a critical part of test design and validity.

Science STUFF (Science, Society and Technology for Underrepresented Future Fabricators): Curriculum for future makers and designers

by Christine Schnittka and Joni Lakin

This paper documents the year-long process of developing one curriculum module out of several use... more This paper documents the year-long process of developing one curriculum module out of several used in formal and informal settings in a rural southern US state. The goal was to develop engineering design-infused STEM activities that used everyday materials, related to students' lives, had a pro-social focus, met state science standards, and helped youth develop STEM literacy and skills. This process of curriculum design falls into the category of " design experiments " as described by Brown (1992), Collins (1992) and others. Over 375 students were exposed to the Solar Landmine Detectors curriculum over the course of a year and the module was continuously revised based on observations of the students.

Test directions as a critical component of test design: Best practices and the impact of examinee characteristics

The purpose of test directions is to familiarize examinees with a test so that they respond to it... more The purpose of test directions is to familiarize examinees with a test so that they respond to items in the manner intended. However, changes in educational measurement as well as the U.S. student population present new challenges to test directions and increase the impact that differential familiarity could have on the validity of test score interpretations. This article reviews the literature on best practices for the development of test directions as well as documenting differences in test familiarity for culturally and linguistically diverse students that could be addressed with test directions and practice. The literature indicates that choice of practice items and feedback are critical in the design of test directions and that more extensive practice opportunities may be required to reduce group differences in test familiarity. As increasingly complex and rich item formats are introduced in next-generation assessments, test directions become a critical part of test design and validity.

Sex Differences in Fluid Reasoning: Manifest and Latent Estimates from the Cognitive Abilities Test

Assessing peer problematic behavior and related training experiences

Sex differences in reasoning abilities: Surprising evidence that male-female ratios in the tails of the quantitative reasoning distribution have increased

Sex differences in cognitive abilities, particularly at the extremes of ability distributions, ha... more Sex differences in cognitive abilities, particularly at the extremes of ability distributions, have important implications for the participation of men and women in highly valued and technical career fields. Although negligible mean differences have been found in many domains, differences in variability and high ratios of males to females in the tails of the ability distribution have been found in a number of studies and across domains. A few studies have also observed trends over time, most noting decreasing ratios of boys to girls in the highest levels of mathematics test performance. In this study, sex differences in means, variances, and ratios were evaluated in four cohorts (1984, 1992, 2000, and 2011) in verbal, quantitative, and nonverbal/figural reasoning domains as measured by the Cognitive Abilities Test. Samples included U.S. students in grades 3-11. Overall, the results were consistent with previous research, showing small mean differences in the three domains, but considerably greater variability for males. The most surprising finding was that, contrary to related research, the ratio of males to females in the upper tail of the quantitative reasoning distribution seemed to increase over time. Explanations for this finding are explored.

Evaluating growth for ELL students: Implications for accountability policies

In recent years, many U.S. states have introduced growth models as part of their educational acco... more In recent years, many U.S. states have introduced growth models as part of their educational accountability systems. Although the validity and fairness of growth-based accountability models have been evaluated for the general population, the impact of those models for English language learner (ELL) students, a growing segment of the student population, has not received sufficient attention. We evaluated three commonly used growth models: value tables or transition matrices, projection models, and student growth percentiles (SGP). The value table model identified more ELL students as on-track to proficiency, but with lower accuracy for ELL students. The projection and SGP models were more accurate overall, but classified the fewest ELL students as on-track and were less likely to identify ELL students who would later be proficient. We found that each model had significant trade-offs in terms of the decisions made for ELL students. These findings should be replicated in additional state contexts and considered in the development of future growth-based accountability policies.

Assessing the cognitive abilities of culturally and linguistically diverse students: Predictive validity of verbal, quantitative, and nonverbal batteries

Verbal and quantitative reasoning tests provide valuable information about cognitive abilities th... more Verbal and quantitative reasoning tests provide valuable information about cognitive abilities that are important to academic success. Information about these abilities may be particularly valuable to teachers of students who are English-language learners (ELL), because leveraging reasoning skills to support comprehension is a critical aptitude for their academic success.

Multidimensional ability tests and culturally and linguistically diverse students: Evidence of measurement invariance

Ability tests are used by teachers to provide additional context for interpreting student achieve... more Ability tests are used by teachers to provide additional context for interpreting student achievement and as a tool for differentiating instruction to the cognitive strengths and weaknesses of students. Tests that provide the most useful information for this purpose measure school-related content domains including verbal and quantitative reasoning. However, there is concern that verbal content affects validity for culturally and linguistically diverse students. In this study, the structure of a multidimensional ability test of verbal, quantitative, and nonverbal reasoning skills was explored in three groups of students who varied in language and cultural background. Configural invariance and invariant factor loadings were supported, but the verbal battery showed weaker relationships to the other batteries and reduced variability for English learners. Results indicate that battery-level scores are appropriate for all students, but that accounting for educational opportunity may be required for interpreting scores.

Investigating ESL students’ performance on outcomes assessments in higher education

Outcomes assessments are gaining great attention in higher education due to increased demand for ... more Outcomes assessments are gaining great attention in higher education due to increased demand for accountability. These assessments are widely used by U.S. higher education institutions to measure students' college-level knowledge and skills, including students who speak English as a second language (ESL). For the past decade, the increasing number of ESL students has changed the landscape of U.S. higher education. However, little research exists documenting how ESL students perform on outcomes assessments. In this study, we investigated ESL students' performance on the ETS Proficiency Profile in terms of factor structure, predictive validity, and differential item functioning (DIF). The test showed partial measurement invariance between ESL and non-ESL students, consistent predictive validity, and few examples of DIF. The results suggest the critical need for consideration of language background in outcomes assessment research in higher education.

Distinguishing verbal, quantitative, and nonverbal facets of fluid intelligence in young students

Measures of broad fluid abilities including verbal, quantitative, and figural reasoning are commo... more Measures of broad fluid abilities including verbal, quantitative, and figural reasoning are commonly used in the K-12 school context for a variety of purposes. However, it is difficult to differentiate these domains for young children (grades K-2) who lack language and mathematics literacy. This study evaluated the latent factor structure of a picture-based test of broad fluid reasoning abilities using a bifactor model to separate general and broad domain factors in a large representative sample of U.S. school children. Results showed substantial evidence that picturebased item formats can distinguish between general and domain-specific fluid reasoning abilities.

Multigroup Generalizability Analysis of Verbal, Quantitative, and Nonverbal Ability Tests for Culturally and Linguistically Diverse Students

Educational and Psychological Measurement, 2012

... Joni M. Lakin1 and Emily R. Lai2 ... of such students in the US school system (American Feder... more

The Predictive Accuracy of Verbal, Quantitative, and Nonverbal Reasoning Tests: Consequences for Talent Identification and Program Diversity

Journal for The Education of The Gifted, 2011

Verbal and quantitative reasoning tests provide valuable information about cognitive abilities th... more Verbal and quantitative reasoning tests provide valuable information about cognitive abilities that are important to academic success. Information about these abilities may be particularly valuable to teachers of students who are English-language learners (ELL), because leveraging reasoning skills to support comprehension is a critical aptitude for their academic success.

Nonverbal test scores as one component of an identification system: Integrating ability, achievement, and teacher ratings

Identifying Academically Gifted English- Language Learners Using Nonverbal Tests A Comparison of the Raven, NNAT, and CogAT

In this study, the authors compare the validity of three nonverbal tests for the purpose of ident... more In this study, the authors compare the validity of three nonverbal tests for the purpose of identifying academically gifted English-language learners (ELLs). Participants were 1,198 elementary children (approximately 40% ELLs). All were administered the Raven Standard Progressive Matrices (Raven), the Naglieri Nonverbal Ability Test (NNAT), and Form 6 of the Cognitive Abilities Test (CogAT). Results show that the U.S. national norms for the Raven substantially overestimate the number of high-scoring children; that because of errors in norming, the NNAT overestimates the number of both high-scoring and low-scoring children; that primary-level ELL children score especially poorly on the NNAT; that the standard error of measurement was twice as large for the NNAT as for the Raven or the CogAT; that ELL children scored .5 to .67 standard deviations lower than non-ELL children on the three nonverbal tests; and that none of the nonverbal tests predict achievement for ELL students very well.

Assessing the Cognitive Abilities of Culturally and Linguistically Diverse Students: Predictive Validity of Verbal, Quantitative, and Nonverbal Test

Verbal and quantitative reasoning tests provide valuable information about cognitive abilities th... more Verbal and quantitative reasoning tests provide valuable information about cognitive abilities that are important to academic success. Information about these abilities may be particularly valuable to teachers of students who are English-language learners (ELL), because leveraging reasoning skills to support comprehension is a critical aptitude for their academic success.

The commonality of extreme discrepancies in the ability profiles of academically gifted students

Extreme discrepancies in abilities are more common among the most and least able students than am... more Extreme discrepancies in abilities are more common among the most and least able students than among average ability children. Therefore, procedures for identifying gifted children that deliberately or inadvertently rely on a composite score that averages across ability domains will exclude many children who reason exceptionally well in particular symbol systems. In this article, we first discuss general issues in the measurement of ability profiles. We then introduce a method for categorizing score profiles and finally document the reliability and stability of score profiles using the 2000 standardization data of the Cognitive Abilities Test .

Consistencies in Sex Differences on the Cognitive Abilities Test across Countries, Grades, and Cohorts

Recently, Strand, Deary, and Smith (2006) reported sex differences for over 320,000 students 11-1... more Recently, Strand, Deary, and Smith (2006) reported sex differences for over 320,000 students 11-12 years old on the Cognitive Abilities Test. Although mean differences were small, score distributions for males were more variable. Males were also overrepresented at the upper and lower extremes of the score distributions on the Quantitative and Nonverbal batteries and at the lower extreme of the Verbal Battery. However, it is unclear whether these results are unique to the U.K. or whether they would be seen at other grades, cohorts, or forms of the same test. To investigate this, we replicated and expanded their results by comparing the distributions of scores for males and females from three standardizations of the U.S. version of the test. Changes in the proportions of males and females at each score level were examined across countries (U.K. versus U.S.), grades (3-11) and cohorts/test forms (1984, 1992, and 2000). The results showed an astonishing consistency in sex differences across countries, age cohorts, and test forms.

Test Directions as a Critical Component of Test Design: Best Practices and the Impact of Examinee Characteristic

The purpose of test directions is to familiarize examinees with a test so that they respond to it... more The purpose of test directions is to familiarize examinees with a test so that they respond to items in the manner intended. However, changes in educational measurement as well as the U.S. student population present new challenges to test directions and increase the impact that differential familiarity could have on the validity of test score interpretations. This article reviews the literature on best practices for the development of test directions as well as documenting differences in test familiarity for culturally and linguistically diverse students that could be addressed with test directions and practice. The literature indicates that choice of practice items and feedback are critical in the design of test directions and that more extensive practice opportunities may be required to reduce group differences in test familiarity. As increasingly complex and rich item formats are introduced in next-generation assessments, test directions become a critical part of test design and validity.