Decision-Making Accuracy of CBM Progress-Monitoring Data
Amanda Marcotte2017, Journal of Psychoeducational Assessment
Sign up for access to the world's latest research
checkGet notified about relevant papers
checkSave papers to use in your research
checkJoin the discussion with peers
checkTrack your impact
Abstract
This study examined the diagnostic accuracy associated with decision making as is typically conducted with curriculum-based measurement (CBM) approaches to progress monitoring. Using previously published estimates of the standard errors of estimate associated with CBM, 20,000 progress-monitoring data sets were simulated to model student reading growth of twoword increase per week across 15 consecutive weeks. Results indicated that an unacceptably high proportion of cases were falsely identified as nonresponsive to intervention when a common 4-point decision rule was applied, under the context of typical levels of probe reliability. As reliability and stringency of the decision-making rule increased, such errors decreased. Findings are particularly relevant to those who use a multi-tiered response-to-intervention model for evaluating formative changes associated with instructional intervention and evaluating responsiveness to intervention across multiple tiers of intervention. Keywords assessment of interventions/outcomes, diagnostic classification models, measurement, response to intervention (RTI)/multi-tiered system of supports (MTSS), statistical analyses, measurement Curriculum-based measurement (CBM) is a set of standardized and specific measurement procedures that can be used to index student performance in the basic skill areas of early literacy and reading, early numeracy and mathematics computation and application, spelling, and written expression (
Related papers
Development and Validation of Curriculum-Based Measurement (CBM) for Identifying Students with Reading Difficulties
2015
The role of classroom teachers in the early detection of learning difficulty/disability in school children cannot be ignored. When it comes to young children's literacy learning, there is substantial consensus that the teacher is the primary assessment agent (Johnston & Rogers, 2002). But classroom teachers also have a lot of responsibilities in school. As such, they need an assessment and identification approach that they can easily employ. Progress monitoring is one of the feasible and practicable methods in identifying students at-risk. Progress monitoring is a set of assessment procedures for determining the extent to which students are benefiting from classroom instruction and for monitoring effectiveness of curriculum (Johnson, Mellard, Fuchs, & McKnight, 2006). It makes use of Curriculum-Based Measurement (CBM), which has been demonstrated by research to effectively gather student performance data to support a wide range of educational decisions, such as screening to iden...
Progress Monitoring in Reading: Comparison of Weekly, Bimonthly, and Monthly Assessments for Students at Risk for Reading Difficulties in Grades 2–4
School Psychology Review, 2018
The present study examined the utility of two progress monitoring assessment schedules (bimonthly and monthly) as alternatives to monitoring once weekly with curriculum-based measurement in reading (CBM-R). General education students (N = 93) in Grades 2-4 who were at risk for reading difficulties but not yet receiving special education services had their progress monitored via three assessment schedules across 1 academic year. Four mixedfactorial analyses of variance tested the effect of progress monitoring schedule (weekly, bimonthly, monthly), grade (2, 3, and 4), and the interaction effect between schedule and grade on four progress monitoring outcomes: intercept, slope, standard error of the estimate, and standard error of the slope. Results indicated that (a) progress monitoring schedule significantly predicted each outcome, (b) grade predicted each progress monitoring outcome except the standard error of the slope, and (c) the effect of schedule on each outcome did not depend on students' grade levels. Overall, findings from this study reveal that collecting CBM-R data less frequently than weekly may be a viable option for educators monitoring the progress of students in Grades 2-4 who are at risk for reading difficulties. Data-based decision making is a key component of effective multitiered systems of support (MTSS). Within MTSS, all students are screened three times per year. Students who do not meet norm-referenced expectations are identified as being "at risk," and have their progress monitored frequently. Scores from frequent progress monitoring are used to make decisions about the effectiveness of core instruction and/or supplemental intervention for individual students. Despite a nearly 40-year history, research investigating the interpretations and use of scores from curriculum-based
Technical Adequacy and Acceptability of Curriculum-Based Measurement and the Measures of Academic Progress
Assessment for Effective Intervention
Curriculum-based measurement in reading (CBM-R) and the Measures of Academic Progress (MAP) are assessment tools widely employed for universal screening in schools. Although a large body of research supports the validity of CBM-R, limited empirical evidence exists supporting the technical adequacy of MAP or the acceptability of either measure for universal screening. Purposes of the current study were to replicate and extend prior research by (a) examining the extent to which CBM-R performance measures more than word reading skills, (b) evaluating the concurrent validity of MAP with CBM-R, (c) determining the potential benefit of administering MAP with CBM-R for universal screening, and (d) examining teachers’ acceptability of MAP and CBM-R. Participants included 802 students in Grades 1 to 5 who were administered three CBM-R probes and the MAP during universal screening and 86 elementary teachers who completed a universal screening assessments survey. Results provide evidence of th...
Using Curriculum-Derived Progress Monitoring Data as Part of a Response-to-Intervention Strategy: A Case Study
The California School Psychologist, 2006
The revised Individuals with Disabilities Improvement Education Act (2004) and subsequent Federal Regulations promote the use of alternative process of identifying students with specific learning disabilities based on how well a student responds to researched-based interventions. As these strategies are implemented, school psychologists have the opportunity to expand their roles and to assume leadership positions in implementing a response-to-intervention (RtI) model. A central element of all RtI approaches is the universal monitoring of students' academic progress. As part of a general effort to implement a data-driven system, multiple sources of information may be used. This article contributes to these efforts by presenting a case study demonstrating how a school psychologist took the first steps to implement a low-cost, continuous progress monitoring procedure in one urban school. This was accomplished by using data readily available at the school site (reading probes included with the district reading curriculum) to develop a systematic way to monitor progress by creating local school norms and using existing reading benchmarks. Every year school psychologists in the U.S. conduct approximately 816,000 initial evaluations as part of the process to determine eligibility for special education for students suspected of having a specific learning disability (Federal Register, 2005). This would not be surprising to most school psychologists, because assessing students for special education is one of their primary job duties. School psychologists report that typically half of their day is spent on tasks related to individual assessment, such as administering intelligence tests (Reschly & Ysseldyke, 2002). For over 25 years, since the passage of the Education for All Handicapped Children Act of 1975 (P.L. 94-142), school psychologists have been key participants in the multidisciplinary teams that identify students with learning disabilities, primarily using a model based on finding a discrepancy between IQ and achievement (Fuchs, Mock, Morgan, & Young, 2003; Lyon et al., 2001; Vaughn & Fuchs, 2003). However, due to recent changes in federal law, school psychologists may no longer use as much of their time administering intelligence tests. On December 3, 2004 the Individuals with Disabilities Education Improvement Act (IDEIA) was signed into law as Public Law 108-446. The new law allows alternative procedures for evaluating students suspected of having Specific Learning Disabilities. States are no longer required to use a discrepancy between intellectual abilities and achievement as part of the LD eligibility process. States now "…must permit a process that examines whether the child responds to scientific, research-based interventions as part of the evaluation procedures" (Federal Register, p. 35802). This change in federal law was expected. For the past few years several influential groups such as the National Association of School Psychologists, the Office of Special Education Program in the U.S. Department of Education, and the National Center of Learning Disabilities have expressed concerns about the continued use of an IQ-Achievement discrepancy model for diagnosing learning disabilities, and instead have proposed a model that is known as response-to-intervention (RtI; Fuchs et al., 2003; Jimerson, Burns, & VanDerHeyden, in press; Lyon et al., 2001). With the recent release of the new Federal Regulations, school psychologists and other educators now await guidance from their educational agencies on the practical questions regarding how RtI will be
Predicting reading outcomes with progress monitoring slopes among middle grade students
Learning and Individual Differences, 2014
Effective implementation of response-to-intervention (RTI) frameworks depends on efficient tools for monitoring progress. Evaluations of growth (i.e., slope) may be less efficient than evaluations of status at a single time point, especially if slopes do not add to predictions of outcomes over status. We examined progress monitoring slope validity for predicting reading outcomes among middle school students by evaluating latent growth models for different progress monitoring measure-outcome combinations. We used multi-group modeling to evaluate the effects of reading ability, reading intervention, and progress monitoring administration condition on slope validity. Slope validity was greatest when progress monitoring was aligned with the outcome (i.e., word reading fluency slope was used to predict fluency outcomes in contrast to comprehension outcomes), but effects varied across administration conditions (viz., repeated reading of familiar vs. novel passages). Unless the progress monitoring measure is highly aligned with outcome, slope may be an inefficient method for evaluating progress in an RTI context.
Evaluating Curriculum-Based Measurement From a Behavioral Assessment Perspective.
Behavior Analyst Today, 2008
Identification of Reading Problems in First Grade Within a Response-To-Intervention Framework
2011
Models of Response to Intervention (RTI) include parameters of assessment and instruction. This study focuses on assessment with the purpose of developing a screening battery that validly and efficiently identifies first-grade children at risk for reading problems. In an RTI model, these children would be candidates for early intervention. We examined accuracy, fluency, growth, and teacher rating measures as predictors of child status (at risk, not at risk) at the end of the school year based on an unselected sample of 243 children. The prediction model that best fit our selection criteria included 2-word fluency measures and a teacher rating of reading problems. Word-fluency growth was an equally plausible choice statistically, but, because the measure would require an additional data point, it was not the most efficient choice. The receiver-operator characteristic curve analysis yielded an area-under-the-curve index of .96, which indicates the selected 3-variable model is highly accurate. The most current reauthorization of the Individuals with Disabilities Education Improvement Act (IDEIA, 2004) allows for the incorporation of Response to Intervention (RTI) to identify children with specific learning disabilities. This approach may be used instead of a discrepancy model in which students' scores on intelligence and achievement tests are compared. The typical RTI model includes three to four stages, or tiers, of assessment and instruction. In the domain of reading, Tier 1 usually involves (a) universal screening (i.e., screening of all students) to identify those at risk for reading problems, (b) general education instruction that is founded on scientifically based reading research, and (c) progress monitoring to identify any students who are not growing in reading skills at an expected rate (Berkeley, Bender, Peaster, & Saunders, 2009; McKenzie, 2009; Shinn, 2007). Students who are at risk for reading problems and do not show a positive response to general education instruction, including growth at expected rates, are selected to receive increasingly intense and focused instruction in subsequent tiers.
Reliability of Decision- Making Frameworks for Response to Intervention for Reading
The current study examines the consistency of two response-to-intervention (RTI) decisionmaking models. Weekly progress monitoring data for 30 students participating in a Tier II intervention were collected for 30 weeks. The data were examined by comparing them to an aimline with a yearly goal and by computing a dual discrepancy (DD) using numerical slope and postintervention reading level. A k coefficient was computed between the two models regarding student growth using three categorical decisions of making sufficient progress, not making sufficient progress, and exceeding expected progress. The resulting coefficient of .29 represents fair agreement. Data were also assigned to one of two sets in an every-other order, with resulting new aimlines and new DDs. The consistency between the two sets of data resulted in coefficients of k = .29 for aimline and k = .58 for DD. Finally, the standard error of measurement for the baseline data point affected the decision made (k = .13) for aimline but not for DD (k = .93). Implications for research, practice, and special education decision making are included.
Is it working?: An overview of curriculum based measurement and its uses for assessing instructional, intervention, or program effectiveness
The Behavior Analyst Today, 2007
Presented here is an overview of Curriculum Based Measurement (CBM)-a standardized process of obtaining data regarding a student's acquisition of skills in reading, writing, math, and spelling. A review of research that has utilized CBM in its efforts to define effective teaching strategies also is offered. Overall, the goal of this discussion is to orient professionals working with students in and outside of the school setting to a model of assessment that is gaining wide acceptance in data based decision making.
A systematic review and summarization of the recommendations and research surrounding Curriculum-Based Measurement of oral reading fluency (CBM-R) decision rules
Journal of School Psychology, 2013
Research and policy have established that data are necessary to guide decisions within education. Many of these decisions are made within problem solving and response to intervention frameworks for service delivery. Curriculum-Based Measurement in Reading (CBM-R) is a widely used data collection procedure within those models of service delivery. Although the evidence for CBM-R as a screening and benchmarking procedure has been summarized multiple times in the literature, there is no comprehensive review of the evidence for its application to monitor and evaluate individual student progress. The purpose of this study was to identify and summarize the psychometric and empirical evidence for CBM-R as it is used to monitor and evaluate student progress. There was an emphasis on the recommended number of data points collected during progress monitoring and interpretive guidelines. The review identified 171 journal articles, chapters, and instructional manuals using online search engines and research databases. Recommendations and evidence from 102 documents that met the study criteria were evaluated and summarized. Results indicate that most decision-making practices are based on expert opinion and that there is very limited psychometric or empirical support for such practices. There is a lack of published evidence to support program evaluation and progress monitoring with CBM-R. More research is required to inform data collection procedures and interpretive guidelines.
Curriculum-Based Measurement in Writing: Predicting the Success of High-School Students on State Standards Tests
Exceptional Children, 2008
Using Curriculum-Based Measurement to Monitor Reading Progress in Inclusive Elementary Settings
Reading & Writing Quarterly, 2006
The purpose of this study was to examine technical and instructional features of a kindergarten curriculum-based measurement (CBM) tool designed to track students' mathematics progress in terms of computational concepts, procedures, and counting strategies. Students in 10 kindergarten classrooms in three elementary schools completed alternate forms of the CBM measure twice per month from January to May. Mathematics development was indexed on a standardized mathematics achievement test in May. Findings indicate strong reliability and validity of the CBM system, with coefficients exceeding .80 and .60, respectively. Technical features of the CBM system's skills analysis suggest implications for teachers' instructional decision-making.
Early Response-to-Intervention Measures and Criteria as Predictors of Reading Disability in the Beginning of Third Grade
Journal of Learning Disabilities, 2013
We explored the usefulness of first and second grade reading measures and responsiveness criteria collected within a response-to-intervention (RtI) framework for predicting reading disability (RD) in third grade. We used existing data from 387 linguistically diverse students who had participated in a longitudinal RtI study. Model-based predictors of RD were analyzed using logistic regression; isolated measure/criteria combinations for predicting RD were analyzed using classification analysis. Models yielded superior classification rates compared to single measure approaches and did not systematically misclassify English learners. However, particular first and second grade measure/criteria combinations also showed promise as isolated predictors of RD in word reading/text fluency. Model-based approaches were required for acceptable classification of students with RD in comprehension. Although the former finding is promising for early identification of students in need of more intensive instruction in lexical or fluency-based skills, the latter finding reaffirms literature attesting to the complexity of RD in comprehension and difficulty of predicting deficits using early measures of reading, which primarily assess word reading skill. Results replicated well with an independent sample, thus enhancing confidence in study conclusions. Implications regarding the use of RtI for predicting RD are discussed.
Using curriculum-based measurement to prevent failure and assess learning in the content areas. Assessment for Effective Intervention
2003
Many students continue to experience reading difficulties into their secondary-school years. Difficulties include poor word decoding, limited vocabulary knowledge, and deficits in working memory capacity. Such difficulties limit the ability of students to acquire new knowledge and be successful in content-area classes. In this article, we describe a curriculum-based measure, vocabulary matching, that teachers can use to assess and monitor the performance of students with reading difficulties in the content areas. Vocabulary matching, has been found to be a valid and reliable measure of performance and progress in content areas. A case study is provided to illustrate the use of vocabulary matching in content-area instruction. At the secondary-school level, students are expected to read to leam rather than to learn to read (Alley & Deshler, 1979). Unfortunately, many students reach sec-ondary school not yet proficient in reading. These students experience difficulties in many areas of...
Measuring Response to Early Literacy Intervention With Preschoolers at Risk
Topics in Early Childhood Special Education, 2008
Response to intervention (RTI) is characterized as a logical science of decision making that has applicability for early childhood, particularly in the context of multitiered intervention models. This study examined the utility of using curriculum-based early literacy measures as screening tools and for evaluating whether growth in early literacy skills was altered following brief interventions. Thirty-five preschool-age children at risk for learning difficulties participated in a 5-week intervention focused on key phonemic awareness skills delivered in classwide and individual formats. Curriculum-based measurement probes were administered each week to all children. Results showed the curriculum-based measures led to enhanced decision accuracy about children at risk for learning difficulties, particularly when combined with brief classwide interventions designed to systematically address opportunities to learn. For children who were initially low performers, weekly progress monitoring combined with classwide early literacy interventions appeared to be related to accelerated growth toward early literacy targets. Implications for the expansion of RTI decision-making frameworks in early childhood are discussed.
Examining the Effects of Linking Student Performance and Progression in a Tier 2 Kindergarten Reading Intervention
Journal of Learning Disabilities, 2013
Despite the emerging evidence base on response to intervention, there is limited research regarding how to effectively use progress-monitoring data to adjust instruction for students in Tier 2 intervention. In this study, we analyzed extant data from a series of randomized experimental studies of a kindergarten supplemental reading intervention to determine whether linking performance on formative assessments to curriculum progression improved kindergarten reading outcomes over standard implementation. We were interested in whether specific progression adjustments would enhance the effects of supplemental reading intervention. Growth-mixture modeling using data from kindergarteners ( n = 136) whose intervention progression (e.g. repeat lessons, skip lessons) was adjusted every 4 weeks based on mastery data identified four latent classes characterized by unique profiles of curriculum progression adjustments. Multilevel analyses comparing the performance of students in the four classe...
Measuring response to early literacy intervention with preschoolers at risk.(Report)
Topics in Early Childhood Special Education, 2007
Response to intervention (RTI) is characterized as a logical science of decision making that has applicability for early childhood, particularly in the context of multitiered intervention models. This study examined the utility of using curriculum-based early literacy measures as screening tools and for evaluating whether growth in early literacy skills was altered following brief interventions. Thirty-five preschool-age children at risk for learning difficulties participated in a 5-week intervention focused on key phonemic awareness skills delivered in classwide and individual formats. Curriculum-based measurement probes were administered each week to all children. Results showed the curriculum-based measures led to enhanced decision accuracy about children at risk for learning difficulties, particularly when combined with brief classwide interventions designed to systematically address opportunities to learn. For children who were initially low performers, weekly progress monitoring combined with classwide early literacy interventions appeared to be related to accelerated growth toward early literacy targets. Implications for the expansion of RTI decision-making frameworks in early childhood are discussed.
Loading Preview
Sorry, preview is currently unavailable. You can download the paper by clicking the button above.