When college personnel need information about student achievement and growth, cognitive and nonco... more When college personnel need information about student achievement and growth, cognitive and noncognitive tests are commonly utilized. Th e ensuing analysis of test score results often leads to scrutiny of score interpretations. It is quite reasonable to speculate that in testing situations for which no personal consequences exist, low examinee motivation may result in test performances that underestimate true student ability. Th us, it becomes quite easy for stakeholders, generally relying on anecdotal evidence, to conclude that belowexpected performance levels are due to lack of motivation, whether that be the true state of aff airs or not. It is common for faculty members and committees when confronted with disappointing results to attribute low performance to a lack of motivation. Th is may or may not refl ect reality. Said another way, we gather assessment results to help us make sound decisions and interpretations about our academic programs. Every accrediting body, as well as ...
Institutions of higher education are encountering new and increasingly demanding expectations to ... more Institutions of higher education are encountering new and increasingly demanding expectations to address accountability and transparency of student learning and development efforts. Changes in federal administration have not dampened interest in the assessment and reporting of student learning (Jaschik, 2009), yet the centrality of general education to every educational mission will render it a problematic assessment domain. When assessment programs are implemented, how will we know that our students are engaged in completing the assigned tasks? The Student Opinion Scale (SOS; Sundre & Moore, 2002) was developed to assess examinee motivation during testing conditions. Based on the expectancy-value model of achievement motivation (Eccles, et al., 1983; Pintrich, 1989; Pintrich & DeGroot, 1990), the two subscores of the SOS are Effort and Importance. The 10-item questionnaire is administered at the close of testing and asks students to report the level of effort they invested and thei...
Advancing Assessment of Quantitative and Scientific Reasoning is a four-year NSF Project (DUE-061... more Advancing Assessment of Quantitative and Scientific Reasoning is a four-year NSF Project (DUE-0618599) in part designed to evaluate the generalizability of quantitative (QR) and scientific reasoning (SR) assessment instruments created at James Madison University to four other fouryear institutions with very distinct missions and student demographics. This article describes the methods, results, and findings we obtained in our studies. More specifically, we describe how to conduct content-alignment exercises in which faculty members map each item from a prospective test to the student learning objectives taught at the institution. Our results indicated that 92-100 % of the QR and SR items were successfully mapped to each of the partner institutions’ learning objectives. We also guided the partner institutions on assessing the balance of test items across the intended student learning objectives to assure greater content validity and coverage. The reliability (internal consistency) re...
Effective mentoring programs help to recruit new teachers and improve teacher retention rates (e.... more Effective mentoring programs help to recruit new teachers and improve teacher retention rates (e.g. Ingersoll & Strong, 2011, Rideout & Windle, 2010). Many school districts require mentoring programs for new teachers; however, they do not always have the resources to prepare their teachers to mentor beginning teachers. Clinical faculty (CF), who are teachers trained to mentor student teachers, are a resource with existing expertise in mentorship. A curriculum for CF was designed, piloted, and implemented using a best practice utilization-focused approach (Patton, 2008). Its focus is on meeting school division needs to equip CF to better mentor beginning teachers, particularly in identified critical shortage areas. The curriculum deepens and expands CF capabilities to more effectively model, observe, and explore data collaboratively with first-year teachers, moving them from mentoring pre-service to in-service teachers. This paper describes the design of and findings from the impleme...
This article describes a seven-phase model of personal conflict over worldview, frame of irony, a... more This article describes a seven-phase model of personal conflict over worldview, frame of irony, and styles in organizational settings that is tested in the context of church organizations. Finally, adversarial and collaborative approaches for resolving conflict are examined.
Assessment, a key component of program evaluation, is often accomplished through multiple-choice ... more Assessment, a key component of program evaluation, is often accomplished through multiple-choice testing. This method has many advantages (mostly related to efficiency) but one significant drawback is the lack of information about how items are interpreted, reacted to and solved by the examinee. Greater understanding of the test-taker experience provides greater
Differential Item Functioning (DIF) occurs when there is a greater probability of solving an item... more Differential Item Functioning (DIF) occurs when there is a greater probability of solving an item based on group membership after controlling for ability. Following administration of a 50-item scientific and quantitative reasoning exam to 286 two-year and 1174 four-year students, items were evaluated for DIF. Two-year students performed better-than-expected on 13 items and worse than expected on 10 items. Reasons for DIF are explored, along with the importance of conducting this type of study.
Advancing Assessment of Quantitative and Scientific Reasoning is a four-year NSF Project (DUE-061... more Advancing Assessment of Quantitative and Scientific Reasoning is a four-year NSF Project (DUE-0618599) in part designed to evaluate the generalizability of quantitative (QR) and scientific reasoning (SR) assessment instruments created at James Madison University to four other fouryear institutions with very distinct missions and student demographics. This article describes the methods, results, and findings we obtained in our studies. More specifically, we describe how to conduct content-alignment exercises in which faculty members map each item from a prospective test to the student learning objectives taught at the institution. Our results indicated that 92-100% of the QR and SR items were successfully mapped to each of the partner institutions' learning objectives. We also guided the partner institutions on assessing the balance of test items across the intended student learning objectives to assure greater content validity and coverage. The reliability (internal consistency) results from the partner institutions for the learning objectives and major subtests are strikingly similar across very different student populations. We interpret lower reliabilities from one institution to be the result of test administration and student motivation factors, the latter being a serious threat to the health and vigor of any assessment program. Validity study results at the partner institutions add to the evidence of construct validity of the QR and SR instruments. While our studies focus on QR and SR instruments, the methods will apply to other instruments and other institutions as they attempt to answer important questions about student learning outcomes.
... Robin D. Anderson Robin D. Anderson is the associate director of the Center for Assessment an... more ... Robin D. Anderson Robin D. Anderson is the associate director of the Center for Assessment and Research Studies at James Madison University. ... The Test Anxiety Profile (tap) manual. Fort Collins: Colorado Rocky Mountain Behavioral Sciences Institute. Salinas, M. (1998). ...
... Robin D. Anderson Robin D. Anderson is the associate director of the Center for Assessment an... more ... Robin D. Anderson Robin D. Anderson is the associate director of the Center for Assessment and Research Studies at James Madison University. ... The Test Anxiety Profile (tap) manual. Fort Collins: Colorado Rocky Mountain Behavioral Sciences Institute. Salinas, M. (1998). ...
Institutions of higher education are encountering new and increasingly demanding expectations to ... more Institutions of higher education are encountering new and increasingly demanding expectations to address accountability and transparency of student learning and development efforts. Changes in federal administration have not dampened interest in the assessment and reporting of student learning ), yet the centrality of general education to every educational mission will render it a problematic assessment domain. When assessment programs are implemented, how will we know that our students are engaged in completing the assigned tasks? The Student Opinion Scale (SOS;
When college personnel need information about student achievement and growth, cognitive and nonco... more When college personnel need information about student achievement and growth, cognitive and noncognitive tests are commonly utilized. Th e ensuing analysis of test score results often leads to scrutiny of score interpretations. It is quite reasonable to speculate that in testing situations for which no personal consequences exist, low examinee motivation may result in test performances that underestimate true student ability. Th us, it becomes quite easy for stakeholders, generally relying on anecdotal evidence, to conclude that belowexpected performance levels are due to lack of motivation, whether that be the true state of aff airs or not. It is common for faculty members and committees when confronted with disappointing results to attribute low performance to a lack of motivation. Th is may or may not refl ect reality.
Advancing Assessment of Quantitative and Scientific Reasoning is a four-year NSF Project (DUE-061... more Advancing Assessment of Quantitative and Scientific Reasoning is a four-year NSF Project (DUE-0618599) in part designed to evaluate the generalizability of quantitative (QR) and scientific reasoning (SR) assessment instruments created at James Madison University to four other fouryear institutions with very distinct missions and student demographics. This article describes the methods, results, and findings we obtained in our studies. More specifically, we describe how to conduct content-alignment exercises in which faculty members map each item from a prospective test to the student learning objectives taught at the institution. Our results indicated that 92-100% of the QR and SR items were successfully mapped to each of the partner institutions' learning objectives. We also guided the partner institutions on assessing the balance of test items across the intended student learning objectives to assure greater content validity and coverage. The reliability (internal consistency) results from the partner institutions for the learning objectives and major subtests are strikingly similar across very different student populations. We interpret lower reliabilities from one institution to be the result of test administration and student motivation factors, the latter being a serious threat to the health and vigor of any assessment program. Validity study results at the partner institutions add to the evidence of construct validity of the QR and SR instruments. While our studies focus on QR and SR instruments, the methods will apply to other instruments and other institutions as they attempt to answer important questions about student learning outcomes.
Assessment, a key component of program evaluation, is often accomplished through multiple-choice ... more Assessment, a key component of program evaluation, is often accomplished through multiple-choice testing. This method has many advantages (mostly related to efficiency), but one significant drawback is the lack of information about how items are interpreted, reacted to, and solved by the examinee. Deeper understanding of the test-taker experience provides greater evidence for the validity of the scores produced by
When college personnel need information about student achievement and growth, cognitive and nonco... more When college personnel need information about student achievement and growth, cognitive and noncognitive tests are commonly utilized. Th e ensuing analysis of test score results often leads to scrutiny of score interpretations. It is quite reasonable to speculate that in testing situations for which no personal consequences exist, low examinee motivation may result in test performances that underestimate true student ability. Th us, it becomes quite easy for stakeholders, generally relying on anecdotal evidence, to conclude that belowexpected performance levels are due to lack of motivation, whether that be the true state of aff airs or not. It is common for faculty members and committees when confronted with disappointing results to attribute low performance to a lack of motivation. Th is may or may not refl ect reality. Said another way, we gather assessment results to help us make sound decisions and interpretations about our academic programs. Every accrediting body, as well as ...
Institutions of higher education are encountering new and increasingly demanding expectations to ... more Institutions of higher education are encountering new and increasingly demanding expectations to address accountability and transparency of student learning and development efforts. Changes in federal administration have not dampened interest in the assessment and reporting of student learning (Jaschik, 2009), yet the centrality of general education to every educational mission will render it a problematic assessment domain. When assessment programs are implemented, how will we know that our students are engaged in completing the assigned tasks? The Student Opinion Scale (SOS; Sundre & Moore, 2002) was developed to assess examinee motivation during testing conditions. Based on the expectancy-value model of achievement motivation (Eccles, et al., 1983; Pintrich, 1989; Pintrich & DeGroot, 1990), the two subscores of the SOS are Effort and Importance. The 10-item questionnaire is administered at the close of testing and asks students to report the level of effort they invested and thei...
Advancing Assessment of Quantitative and Scientific Reasoning is a four-year NSF Project (DUE-061... more Advancing Assessment of Quantitative and Scientific Reasoning is a four-year NSF Project (DUE-0618599) in part designed to evaluate the generalizability of quantitative (QR) and scientific reasoning (SR) assessment instruments created at James Madison University to four other fouryear institutions with very distinct missions and student demographics. This article describes the methods, results, and findings we obtained in our studies. More specifically, we describe how to conduct content-alignment exercises in which faculty members map each item from a prospective test to the student learning objectives taught at the institution. Our results indicated that 92-100 % of the QR and SR items were successfully mapped to each of the partner institutions’ learning objectives. We also guided the partner institutions on assessing the balance of test items across the intended student learning objectives to assure greater content validity and coverage. The reliability (internal consistency) re...
Effective mentoring programs help to recruit new teachers and improve teacher retention rates (e.... more Effective mentoring programs help to recruit new teachers and improve teacher retention rates (e.g. Ingersoll & Strong, 2011, Rideout & Windle, 2010). Many school districts require mentoring programs for new teachers; however, they do not always have the resources to prepare their teachers to mentor beginning teachers. Clinical faculty (CF), who are teachers trained to mentor student teachers, are a resource with existing expertise in mentorship. A curriculum for CF was designed, piloted, and implemented using a best practice utilization-focused approach (Patton, 2008). Its focus is on meeting school division needs to equip CF to better mentor beginning teachers, particularly in identified critical shortage areas. The curriculum deepens and expands CF capabilities to more effectively model, observe, and explore data collaboratively with first-year teachers, moving them from mentoring pre-service to in-service teachers. This paper describes the design of and findings from the impleme...
This article describes a seven-phase model of personal conflict over worldview, frame of irony, a... more This article describes a seven-phase model of personal conflict over worldview, frame of irony, and styles in organizational settings that is tested in the context of church organizations. Finally, adversarial and collaborative approaches for resolving conflict are examined.
Assessment, a key component of program evaluation, is often accomplished through multiple-choice ... more Assessment, a key component of program evaluation, is often accomplished through multiple-choice testing. This method has many advantages (mostly related to efficiency) but one significant drawback is the lack of information about how items are interpreted, reacted to and solved by the examinee. Greater understanding of the test-taker experience provides greater
Differential Item Functioning (DIF) occurs when there is a greater probability of solving an item... more Differential Item Functioning (DIF) occurs when there is a greater probability of solving an item based on group membership after controlling for ability. Following administration of a 50-item scientific and quantitative reasoning exam to 286 two-year and 1174 four-year students, items were evaluated for DIF. Two-year students performed better-than-expected on 13 items and worse than expected on 10 items. Reasons for DIF are explored, along with the importance of conducting this type of study.
Advancing Assessment of Quantitative and Scientific Reasoning is a four-year NSF Project (DUE-061... more Advancing Assessment of Quantitative and Scientific Reasoning is a four-year NSF Project (DUE-0618599) in part designed to evaluate the generalizability of quantitative (QR) and scientific reasoning (SR) assessment instruments created at James Madison University to four other fouryear institutions with very distinct missions and student demographics. This article describes the methods, results, and findings we obtained in our studies. More specifically, we describe how to conduct content-alignment exercises in which faculty members map each item from a prospective test to the student learning objectives taught at the institution. Our results indicated that 92-100% of the QR and SR items were successfully mapped to each of the partner institutions' learning objectives. We also guided the partner institutions on assessing the balance of test items across the intended student learning objectives to assure greater content validity and coverage. The reliability (internal consistency) results from the partner institutions for the learning objectives and major subtests are strikingly similar across very different student populations. We interpret lower reliabilities from one institution to be the result of test administration and student motivation factors, the latter being a serious threat to the health and vigor of any assessment program. Validity study results at the partner institutions add to the evidence of construct validity of the QR and SR instruments. While our studies focus on QR and SR instruments, the methods will apply to other instruments and other institutions as they attempt to answer important questions about student learning outcomes.
... Robin D. Anderson Robin D. Anderson is the associate director of the Center for Assessment an... more ... Robin D. Anderson Robin D. Anderson is the associate director of the Center for Assessment and Research Studies at James Madison University. ... The Test Anxiety Profile (tap) manual. Fort Collins: Colorado Rocky Mountain Behavioral Sciences Institute. Salinas, M. (1998). ...
... Robin D. Anderson Robin D. Anderson is the associate director of the Center for Assessment an... more ... Robin D. Anderson Robin D. Anderson is the associate director of the Center for Assessment and Research Studies at James Madison University. ... The Test Anxiety Profile (tap) manual. Fort Collins: Colorado Rocky Mountain Behavioral Sciences Institute. Salinas, M. (1998). ...
Institutions of higher education are encountering new and increasingly demanding expectations to ... more Institutions of higher education are encountering new and increasingly demanding expectations to address accountability and transparency of student learning and development efforts. Changes in federal administration have not dampened interest in the assessment and reporting of student learning ), yet the centrality of general education to every educational mission will render it a problematic assessment domain. When assessment programs are implemented, how will we know that our students are engaged in completing the assigned tasks? The Student Opinion Scale (SOS;
When college personnel need information about student achievement and growth, cognitive and nonco... more When college personnel need information about student achievement and growth, cognitive and noncognitive tests are commonly utilized. Th e ensuing analysis of test score results often leads to scrutiny of score interpretations. It is quite reasonable to speculate that in testing situations for which no personal consequences exist, low examinee motivation may result in test performances that underestimate true student ability. Th us, it becomes quite easy for stakeholders, generally relying on anecdotal evidence, to conclude that belowexpected performance levels are due to lack of motivation, whether that be the true state of aff airs or not. It is common for faculty members and committees when confronted with disappointing results to attribute low performance to a lack of motivation. Th is may or may not refl ect reality.
Advancing Assessment of Quantitative and Scientific Reasoning is a four-year NSF Project (DUE-061... more Advancing Assessment of Quantitative and Scientific Reasoning is a four-year NSF Project (DUE-0618599) in part designed to evaluate the generalizability of quantitative (QR) and scientific reasoning (SR) assessment instruments created at James Madison University to four other fouryear institutions with very distinct missions and student demographics. This article describes the methods, results, and findings we obtained in our studies. More specifically, we describe how to conduct content-alignment exercises in which faculty members map each item from a prospective test to the student learning objectives taught at the institution. Our results indicated that 92-100% of the QR and SR items were successfully mapped to each of the partner institutions' learning objectives. We also guided the partner institutions on assessing the balance of test items across the intended student learning objectives to assure greater content validity and coverage. The reliability (internal consistency) results from the partner institutions for the learning objectives and major subtests are strikingly similar across very different student populations. We interpret lower reliabilities from one institution to be the result of test administration and student motivation factors, the latter being a serious threat to the health and vigor of any assessment program. Validity study results at the partner institutions add to the evidence of construct validity of the QR and SR instruments. While our studies focus on QR and SR instruments, the methods will apply to other instruments and other institutions as they attempt to answer important questions about student learning outcomes.
Assessment, a key component of program evaluation, is often accomplished through multiple-choice ... more Assessment, a key component of program evaluation, is often accomplished through multiple-choice testing. This method has many advantages (mostly related to efficiency), but one significant drawback is the lack of information about how items are interpreted, reacted to, and solved by the examinee. Deeper understanding of the test-taker experience provides greater evidence for the validity of the scores produced by
Uploads
Papers by Amy Thelk