Papers by Christopher Rhoads
The Journal of Experimental Education
Educational research quarterly, 2013
The struggles of students in the United States to comprehend non-fiction science text are well do... more The struggles of students in the United States to comprehend non-fiction science text are well documented Middle school students, in particular, have minimal instruction in comprehending nonfktion and flounder on assessments. This article describes the development process of the Readorium software, an interactive web-basedprogram being developed to assist students with comprehension of science text. The program incorporates research-based recommendations for effective reading comprehension suggested by the Institute of Education Science (IES). Efforts to turn the IES recommendations into a coherent software product that motivates learners are described, as is the process of incorporating student and teacherfeedback to improve the usability of the product. Preliminary results suggest that the program operates efficiently, motivates students, and may substantially impact student comprehension of science text.IntroductionNational measures of student achievement in reading - particularl...
Journal of Educational and Behavioral Statistics, 2010
Experimental designs that randomly assign entire clusters of individuals (e.g., schools and class... more Experimental designs that randomly assign entire clusters of individuals (e.g., schools and classrooms) to treatments are frequently advocated as a way of guarding against contamination of the estimated average causal effect of treatment. However, in the absence of contamination, experimental designs that randomly assign intact clusters to treatments are less efficient than designs that randomly assign individual units within clusters. The current article considers the case of contamination processes that tend to make experimental and control subjects appear more similar than they truly are. The article demonstrates that, for most parameter values of practical interest, the statistical power of a randomized block (RB) design remains higher than the power of a cluster randomized (CR) design even when contamination causes the effect size to decrease by as much as 10%–60%. Furthermore, from the standpoint of point estimation, RB designs will tend to be preferred when true effect sizes ...
While peer mentor programs for students with disabilities in higher education are increasing in p... more While peer mentor programs for students with disabilities in higher education are increasing in prevalence, the research examining the effectiveness of these programs remains limited. In this study, the effect of one college peer mentoring program on academic performance at a small, private four-year university was examined. The sample was drawn from all students registered with the Office of Disability Services (n = 287), some of whom participated in a peer mentoring intervention as well as a comparison group comprised of non-participants. In light of the observational nature of the data, propensity score weighting and matching were used to adjust for possible confounding variables and to explore robustness to different methodological approaches. Logistic and linear regression methods were used to examine the effect of peer mentoring on academic probation status and grade point average (GPA), respectively, while incorporating propensity score-based adjustments. There were no signif...
The use of cluster randomized experiments to study the effects of treatments on groups of subject... more The use of cluster randomized experiments to study the effects of treatments on groups of subjects has increased in recent years. Many of these experiments lack the necessary Table of Contents ABSTRACT Acknowledgments List of Tables List of Figures Chapter 1. Introduction R (GLS)-CM test, max, min as ftn. of ρ, n = n ex = 100, δ = 0.3, 0.5, 0.8, α = .025 • Due to some instability in the numerical integral involved for very high values of ρ, the maximum and minimum over rho was evaluated only for ρ ∈ [0, .96]. • The value of δ = 0.3 was chosen rather than than Cohen's "small" effect size of 0.2 due to a concern that for these very small values of m the power of all tests at δ = 0.2 would be too small to show any differences.
This paper provides a guide to calculating statistical power for the complex multilevel designs t... more This paper provides a guide to calculating statistical power for the complex multilevel designs that are used in most field studies in education research. For multilevel evaluation studies in the field of education, it is important to account for the impact of clustering on the standard errors of estimates of treatment effects. Using ideas from survey research, the paper explains how sample design induces random variation in the quantities observed in a randomized experiment, and how this random variation relates to statistical power. The manner in which statistical power depends upon the values of intraclass correlations, sample sizes at the various levels, the standardized average treatment effect (effect size), the multiple correlation between covariates and the outcome at different levels, and the heterogeneity of treatment effects across sampling units is illustrated. Both hierarchical and randomized block designs are considered. The paper demonstrates that statistical power in...
Education. We would like to thank the staff, teachers, and students at Fullerton School District ... more Education. We would like to thank the staff, teachers, and students at Fullerton School District for their contributions to this research. We also thank Charles Harding, Lyn Bopp, and Kyle Luke for their contributions to this report. We are also grateful to Helen Apthorp and Richard Mayer for their comments on an earlier version of this report.
Society for Research on Educational Effectiveness, 2014
Limit 4 pages single-spaced. Background / Context: Description of prior research and its intellec... more Limit 4 pages single-spaced. Background / Context: Description of prior research and its intellectual context. Recent years have seen an increased interest in quantitative educational research studies that use random assignment (RA) to evaluate the causal impacts of educational interventions (Angrist, 2004). The multi-level structure of the public education system in the United States often leads to experimental designs where naturally occurring clusters (eg. schools) are utilized to recruit participants (eg. students) into a study and/or as units that are randomized into one of two experimental conditions.
International Association for Development of the Information Society, 2016
Problem-based learning (PBL) is an instructional design approach for promoting student learning, ... more Problem-based learning (PBL) is an instructional design approach for promoting student learning, in context-rich settings. GlobalEd 2 (GE2) is PBL intervention that combines face-to-face and online environments into a 12-week simulation of international negotiations of science advisors on global water resource issues. The GE2 environment is described examining the impact it has had on middle school students’ written scientific argumentation during the intervention. Analyses using HLM on treatment and comparison groups demonstrated a significant positive impact on the written scientific argumentation scores of 1818 middle-grade students from two states with an effect size of 0.257 (p<.001).
Reading Research Quarterly, 2020
Motivation for reading is important to comprehension and has been studied extensively in offline ... more Motivation for reading is important to comprehension and has been studied extensively in offline reading contexts. However, little is known about the role of motivation in online reading, a new and increasingly important context for reading, largely because of a lack of valid and reliable instruments to estimate a student's motivation for online reading. The authors report on the development of the Motivations for Online Reading Questionnaire (MORQ) among 1,798 seventh-grade students in two states. Results from confirmatory factor analysis revealed a three-factor solution for the MORQ: curiosity/ value, self-efficacy, and self-improvement beliefs. Additionally, measurement invariance across female and male students was established. Predictive validity of the MORQ was supported by the positive and significant contribution of the MORQ to the Online Research and Comprehension Assessment, an established measure of online reading comprehension. Results help establish the MORQ as a well-validated instrument for measuring online reading motivation. Results are discussed in relation to theory, research, and practice. M any factors affect a reader's ability to comprehend and learn during reading, such as prior knowledge (Bråten, Anmarkrud,
Journal of Applied Developmental Psychology, 2019
A classroom randomized efficacy trial conducted over four years in 7 community-based preschool an... more A classroom randomized efficacy trial conducted over four years in 7 community-based preschool and 6 Head Start programs investigated effects of the Second Step Early Learning (SSEL) curriculum on end of preschool executive functioning (EF) and social-emotional (SE) skills in low-income children. Outcomes are reported for n = 770 four-year-olds independently assessed for EF and SE by study staff in fall and spring of the prekindergarten year. Main outcomes were analyzed using two, three-level hierarchical linear models, one each for EF and SE skills. A significant effect (effect size of 0.15) for EF and a nonsignificant effect for SE were found. Secondary analyses found no significant differences on pre-academic skills. SSEL appears to have a meaningful impact on at-risk children's EF skills that supports its continued dissemination.
Computers in Human Behavior, 2017
Problem-based learning (PBL) is an instructional design approach for promoting student learning i... more Problem-based learning (PBL) is an instructional design approach for promoting student learning in context-rich settings. GlobalEd 2 (GE2) is PBL intervention that combines face-to-face and online environments into a 12-week simulation of international negotiations of science delegates on global science issues. Although GE2 focuses on science, it is implemented in a social studies classroom. This manuscript describes the GE2 environment and evaluates its impact on middle school students' scientific literacy compared to a comparison group receiving normal educational practice. Hierarchical Linear Modeling (HLM) analyses on GE2 and comparison groups demonstrates the significant positive impact of GE2 on two measures of scientific literacy (Socio-scientific Literacy and Scientific Inquiry), among middle-grade students from two states. Implications regarding instructional practice and future research are discussed.
Computers & Education, 2016
Previous studies of one-to-one computing programs have suffered from several methodological limit... more Previous studies of one-to-one computing programs have suffered from several methodological limitations and produced mixed results. Especially surprising is that previous studies have not evaluated students' ability to use the Internet to learn from online information. This study evaluated the ability of 1129 seventh grade students, in two different states, to conduct online research and learn in science. One state, Maine, had implemented a one-to-one program for several years, beginning in seventh grade. It ranked 33rd (out of 50 states) in median family income (U.S. Census Bureau, 2012). The other, Connecticut, had yet to implement a similar policy and ranked 4th among all states in median family income. Stratified random sampling was used to select participating students who were representative of each state. The primary measure was a performance-based assessment of online research and learning, with demonstrated validity and reliability. Students completed two online research projects in science on two different days. Results for students in the two states were compared using regression models that conditioned on student prior knowledge and school level indicators of socioeconomic status and teacher experience. The adjusted mean scores for online research and learning were generally significantly higher for Maine, the state that provided laptops to middle and high school students. The overall effect size was comparable to about a half a year of annual growth, the same amount of time that students had access to one-to-one laptops in their classrooms. These results are discussed in terms of their implications for technology investments by state and local educational agencies, especially in light of the new educational standards beginning to appear in Australia, the United States, and other nations. Limitations are also explored. A low SES state (1:1 laptops) did better in science than a high SES state without.Students in both states succeeded on only 50% of online research and learning tasks.The greatest difference between these two states appeared in online locating tasks.
American Journal of Evaluation, 2016
Interest in the regression discontinuity (RD) design as an alternative to randomized control tria... more Interest in the regression discontinuity (RD) design as an alternative to randomized control trials (RCTs) has grown in recent years. There is little practical guidance, however, on conditions that would lead to a successful RD evaluation or the utility of studies with underpowered RD designs. This article describes the use of RD design to evaluate the impact of a supplemental algebra-readiness curriculum, Transition to Algebra, on students’ mathematics outcomes. Lessons learned highlight the need for evaluators to understand important data requirements for strong RD evaluation studies, the need to collaborate with informed and committed partners to ensure successful RD design implementation, the value of embedding an RCT within an RD design whenever possible, and the need for caution when contemplating an RD design with a small sample. Underpowered RD studies—unlike underpowered RCTs—may not produce useful evaluation results, particularly if other RD data requirements are not met.
National Center For Special Education Research, Apr 1, 2010
This paper provides a guide to calculating statistical power for the complex multilevel designs t... more This paper provides a guide to calculating statistical power for the complex multilevel designs that are used in most field studies in education research. For multilevel evaluation studies in the field of education, it is important to account for the impact of clustering on the standard errors of estimates of treatment effects. Using ideas from survey research, the paper explains how sample design induces random variation in the quantities observed in a randomized experiment, and how this random variation relates to statistical power. The manner in which statistical power depends upon the values of intraclass correlations, sample sizes at the various levels, the standardized average treatment effect (effect size), the multiple correlation between covariates and the outcome at different levels, and the heterogeneity of treatment effects across sampling units is illustrated. Both hierarchical and randomized block designs are considered. The paper demonstrates that statistical power in complex designs involving clustered sampling can be computed simply from standard power tables using the idea of operational effect sizes: effect sizes multiplied by a design effect that depends on features of the complex experimental design. These concepts are applied to provide methods for computing power for each of the research designs most frequently used in education research.
Uploads
Papers by Christopher Rhoads