Assessment & Evaluation in Higher Education, Aug 7, 2024
Feedback can be powerful, but its effects are dependent on what students do. There has been inten... more Feedback can be powerful, but its effects are dependent on what students do. There has been intensive research in recent years under the banner of 'feedback literacy' to understand how to help students make the most of feedback. Although there are instruments to measure feedback literacy, they largely measure perceptions and orientations rather than what learners actually do. This paper documents the development and validation of the Feedback Literacy Behaviour Scale (FLBS), which is a self-report instrument intended to measure students' feedback behaviours. A framework for feedback literacy was constructed with five factors: Seek Feedback information (SF), Make Sense of information (MS), Use Feedback information (UF), Provide Feedback information (PF), and Manage Affect (MA). An initial set of 45 questions were reviewed in an iterative process by feedback experts, resulting in 39 questions that were trialled with 350 student participants from four countries. Our final survey of 24 questions was generally supported by confirmatory factor and Rasch analyses, and has acceptable test-retest reliability. The FLBS provides a more robust way for educators and researchers to capture behavioural indicators of feedback literacy and the impact of interventions to improve it.
This series heralds the idea that new times call for new and different thinking about assessment ... more This series heralds the idea that new times call for new and different thinking about assessment and learning, the identities of teachers and students, and what is involved in using and creating new knowledge. Its scope is consistent with a view of assessment as inherently connected with cultural, social practices and contexts. Assessment is a shared enterprise where teachers and students come together to not only develop knowledge and skills, but also to use and create knowledge and identities. Working from this position, the series confronts some of the major educational assessment issues of our times.
Feedback involves much more than providing potentially useful information to learners about their... more Feedback involves much more than providing potentially useful information to learners about their work. The challenge is now on how to engage with learners in ways likely to result in discernable change and the nature of the conditions necessary to ensure that feedback discussions are likely to be acted upon. The main aspects of assessment and feedback drawn on here are those that focus on how assessment and feedback can contribute to the ongoing learning of students (and indeed all health professionals) and the need for feedback to actually influence what students do. Higher education programmes are more frequently criticised in student surveys for deficiencies in assessment and feedback than any other aspect. This has resulted in renewed interest in what feedback is and how it can work effectively. In particular, it has led to recognition that feedback in clinical settings, just like feedback in any other context, must be characterised not solely in terms of inputs that are made, but effects that result. The definition of feedback 1 in Box 1 highlights this change in thinking. Teachers, or others offering feedback information, can therefore only confirm that learning has resulted from information provided if students act on it, to complete a feedback loop. 2
Although there have been increasing attempts to involve undergraduate students in conducting rese... more Although there have been increasing attempts to involve undergraduate students in conducting research, a pivotal moment when students engage in knowledge production is during honours programmes. Honours programmes, particularly those in Australia, seek to develop students' capacity to engage in higher order thinking that may lead to knowledge production. This transition is facilitated through advanced disciplinary knowledge, research training and a research project. However, there is a pedagogical tension between requiring students to engage in this deeper level of inquiry at the same time as they complete a heavy knowledge acquisition load. This paper explores how a number of disciplines in Australia balance these elements of the honours curricula. It argues that the combination of these curriculum goals can make it difficult for students to apply the knowledge they have gained in advanced disciplinary and research training courses to their research project work. This has serious implications for honours programmes.
The paper discusses a framework for the analysis oflearning designs using new technologies. It ta... more The paper discusses a framework for the analysis oflearning designs using new technologies. It takes a learner-centred view derived from literature in higher, professional and adult education. The process of developing guidelines for applying this framework to particular learning activities is outlined and the strengths and limitations of this approach considered. L'article discute d'un cadre pour l'analyse de schemas d'enseignement faisant usage des nouvelles technologies. Il prend en compte un enseignement centre sur l'apprenant a partir d'articles et de livres sur l'education superieure, professionnelle et des adultes. Le processus de developpement des lignes directrices pour l'application de ce cadre a des activites d'enseignement particuliers est mis en valeur et les forces et les limitations de cette approche sont prises en consideration. In diesem Artikel werden Grundlagen zur Analyse von Lernforrnen besprochen, die sich auf neue Technologien stutzen, Man konzentriert sich auf die Sicht des Lernenden und stutzt sich auf die Auswertung der in der hoheren Schulbildung, Berufs-und Erwachsenenbildung verwendeten Literatur. Es wird die Entwicklung von Richtlinien bei der Anwendung dieser Grundlagen bei bestimmten Lernvorgangen besprochen und die Starken und Grenzen ihrer Anwendung diskutiert.
International Journal of Training and Development, Sep 27, 2020
Self-assessment is a fundamental skill for professionals because self-assessment can promote self... more Self-assessment is a fundamental skill for professionals because self-assessment can promote self-regulated learning and professional development. However, studies reporting the use of self-assessment instruments in the professional training context are scarce. This study aimed to re-evaluate the psychometric properties of the Self-assessment Practice Scale (SaPS), which was originally developed in the school context, and extend its use to the professional training context. A sample of 200 investigative interviewer trainees from Australia and North America were invited to complete the modified SaPS. After removing three items, the confirmatory factor analysis results confirmed a first-order four-factor solution. The multidimensional Rasch analysis demonstrated that the resultant 16 items had satisfactory fit to the Rasch model. In general, the results supported the use of the 16-item modified SaPS as a valid measure for the sample in this study. The potential of using the SaPS in professional training contexts is discussed.
The Honours undergraduate degree in Australia is unlike that in most other countries. It has take... more The Honours undergraduate degree in Australia is unlike that in most other countries. It has taken on a particular significance as a qualification, as a pathway to and a prerequisite for direct entry into doctoral programs. This paper explores the outcomes of a study that suggests that the aims, outcomes, curriculum, pedagogical practices, purposes and enrolment patterns of Honours vary substantially across disciplines and university types. It addresses the questions about the diverse nature of Honours programs and questions what this diversity means for Australian higher education in the context where global standardisation of awards is rapidly occurring. Honours is seen variously as a qualification, an experience, or a program. These variations are discussed and it is demonstrated that Honours globally has not one, but many meanings. These meanings are often poorly understood within, and outside the academy. These multiple meanings create confusion about what Honours stands for and inhibit communication about the role and purpose of Honours both within Australia and in a global context.
Higher education research and development, Nov 21, 2014
Can extended opportunities for self-assessment over time help students develop the capacity to ma... more Can extended opportunities for self-assessment over time help students develop the capacity to make better judgements about their work? Using evidence gathered through students' voluntary self-assessment of their performance with respect to assessment tasks in two different disciplines at two Australian universities, the paper focuses on the effects of sequences of units of study and the use of different types of assessment task (written, oral, analysis, project) in the development of student judgement. Convergence between student criteria-based gradings of their own performance in units of study and those allocated by tutors was analysed to explore the calibration of students' judgement over time. First it seeks to replicate analyses from an earlier smaller-scale study to confirm that students' judgements can be calibrated through continuing opportunities for self-assessment and feedback. Second, it extends the analysis to coherently designed sequences of units of study and explores the effects of different types of assessment. It finds that disruptive patterns of assessment within a sequence of subjects can reduce convergence between student and tutor judgements.
Despite common assumptions that assessment practices in higher education are enduring, the past f... more Despite common assumptions that assessment practices in higher education are enduring, the past forty years has seen remarkable changes. A key change has been from the dominance of unseen end-of-year examinations, through the move to 'continuous assessment' and on to a range of diverse assessment approaches. Another notable change has been from assessment weightings being regarded as confidential, to the transparency of assessment standards and criteria of today. Assessment has thus shifted in some positive directions. Unfortunately, during the same period the emphasis on what we now call formative assessment has lessened: from a general acceptance that all courses involved the production of considerable written work that was used purely to aid learning, we now have regimes based on the assumption that all assessment must 'count' towards final marks or a grade point average.
Many universities advertise that their courses prepare students for work. Indeed, the main ration... more Many universities advertise that their courses prepare students for work. Indeed, the main rationale for courses for the professions is that they contribute to preparing students to become effective practitioners. While the more established professions recognise that there is a transitional period following graduation needed in this process, and indeed structured postgraduate training, newer professions have yet to embrace this feature. For both there is an assumption that whatever additional elements may be needed, the course itself is the main foundation. There is no shortage of features of courses claimed to prepare students for practice: various kinds of work--integrated learning, placements and practical work, authentic tasks and assessment activities . In some cases there are approaches to restructuring the entire curriculum to focus students' attention on the kinds of issues that practitioners deal with and their ways of thinking, for example, problem--based learning. But, can it be reasonably claimed that such approaches recognise the nature of practice, the nature of what it is to engage in deliberate practice and thus prepare students accordingly? This chapter suggests that many courses are far from exemplars of good educational practice for the professions. This is not primarily because of teaching quality, but because they tend to have a poorly conceptualised view of what it is that their graduates do in their professional practice. They are too often governed by what is involved in teaching within academic disciplines rather than on how learning occurs within professional work. There is a continuing risk that students will be trapped in current knowledge without the capacity to move beyond what they have been taught. And they are often not characterised by a strong sense that courses need to be actively designed and redesigned to produce graduates that will be deliberate professionals. The approach taken here will be to view professional practice and what is needed to become a deliberate practitioner, through the lens of research on workplace learning and of emerging practice theories. This involves starting with what practitioners do when they practise, and work from this to examine the implications for the courses that precede it. In particular the chapter focuses on what professionals require for learning in practice, one of the key aims of a pedagogy of deliberateness. It suggests that similar considerations apply to what students need to do in order to learn within courses. It starts by thinking about professional
Pre-assessment interventions aimed at helping students to understand assessment criteria and stan... more Pre-assessment interventions aimed at helping students to understand assessment criteria and standards have led to significant improvements in performance (Rust et al., 2003). However, broad adoption of these interventions has not been widespread, perhaps due to a perception that too much additional time may need to be spent in conducting these interventions. This presentation analyses a tightly-focused intervention to improve students‘ and tutors‘ understanding of assessment criteria and their judgement of required standards. It was integrated within an honours degree subject in Design using the web-based marking tool ReView, The intervention engaged students with the assessment criteria and standards of a design task exemplar using the same criteria that would be used to assess their work on this task. The tutor used 20 minutes of a normal lecture period working with a class of 80 students to grade the exemplar piece prior to the submission date. Onscreen grading sliders in ReView...
Assessment & Evaluation in Higher Education, Aug 7, 2024
Feedback can be powerful, but its effects are dependent on what students do. There has been inten... more Feedback can be powerful, but its effects are dependent on what students do. There has been intensive research in recent years under the banner of 'feedback literacy' to understand how to help students make the most of feedback. Although there are instruments to measure feedback literacy, they largely measure perceptions and orientations rather than what learners actually do. This paper documents the development and validation of the Feedback Literacy Behaviour Scale (FLBS), which is a self-report instrument intended to measure students' feedback behaviours. A framework for feedback literacy was constructed with five factors: Seek Feedback information (SF), Make Sense of information (MS), Use Feedback information (UF), Provide Feedback information (PF), and Manage Affect (MA). An initial set of 45 questions were reviewed in an iterative process by feedback experts, resulting in 39 questions that were trialled with 350 student participants from four countries. Our final survey of 24 questions was generally supported by confirmatory factor and Rasch analyses, and has acceptable test-retest reliability. The FLBS provides a more robust way for educators and researchers to capture behavioural indicators of feedback literacy and the impact of interventions to improve it.
This series heralds the idea that new times call for new and different thinking about assessment ... more This series heralds the idea that new times call for new and different thinking about assessment and learning, the identities of teachers and students, and what is involved in using and creating new knowledge. Its scope is consistent with a view of assessment as inherently connected with cultural, social practices and contexts. Assessment is a shared enterprise where teachers and students come together to not only develop knowledge and skills, but also to use and create knowledge and identities. Working from this position, the series confronts some of the major educational assessment issues of our times.
Feedback involves much more than providing potentially useful information to learners about their... more Feedback involves much more than providing potentially useful information to learners about their work. The challenge is now on how to engage with learners in ways likely to result in discernable change and the nature of the conditions necessary to ensure that feedback discussions are likely to be acted upon. The main aspects of assessment and feedback drawn on here are those that focus on how assessment and feedback can contribute to the ongoing learning of students (and indeed all health professionals) and the need for feedback to actually influence what students do. Higher education programmes are more frequently criticised in student surveys for deficiencies in assessment and feedback than any other aspect. This has resulted in renewed interest in what feedback is and how it can work effectively. In particular, it has led to recognition that feedback in clinical settings, just like feedback in any other context, must be characterised not solely in terms of inputs that are made, but effects that result. The definition of feedback 1 in Box 1 highlights this change in thinking. Teachers, or others offering feedback information, can therefore only confirm that learning has resulted from information provided if students act on it, to complete a feedback loop. 2
Although there have been increasing attempts to involve undergraduate students in conducting rese... more Although there have been increasing attempts to involve undergraduate students in conducting research, a pivotal moment when students engage in knowledge production is during honours programmes. Honours programmes, particularly those in Australia, seek to develop students' capacity to engage in higher order thinking that may lead to knowledge production. This transition is facilitated through advanced disciplinary knowledge, research training and a research project. However, there is a pedagogical tension between requiring students to engage in this deeper level of inquiry at the same time as they complete a heavy knowledge acquisition load. This paper explores how a number of disciplines in Australia balance these elements of the honours curricula. It argues that the combination of these curriculum goals can make it difficult for students to apply the knowledge they have gained in advanced disciplinary and research training courses to their research project work. This has serious implications for honours programmes.
The paper discusses a framework for the analysis oflearning designs using new technologies. It ta... more The paper discusses a framework for the analysis oflearning designs using new technologies. It takes a learner-centred view derived from literature in higher, professional and adult education. The process of developing guidelines for applying this framework to particular learning activities is outlined and the strengths and limitations of this approach considered. L'article discute d'un cadre pour l'analyse de schemas d'enseignement faisant usage des nouvelles technologies. Il prend en compte un enseignement centre sur l'apprenant a partir d'articles et de livres sur l'education superieure, professionnelle et des adultes. Le processus de developpement des lignes directrices pour l'application de ce cadre a des activites d'enseignement particuliers est mis en valeur et les forces et les limitations de cette approche sont prises en consideration. In diesem Artikel werden Grundlagen zur Analyse von Lernforrnen besprochen, die sich auf neue Technologien stutzen, Man konzentriert sich auf die Sicht des Lernenden und stutzt sich auf die Auswertung der in der hoheren Schulbildung, Berufs-und Erwachsenenbildung verwendeten Literatur. Es wird die Entwicklung von Richtlinien bei der Anwendung dieser Grundlagen bei bestimmten Lernvorgangen besprochen und die Starken und Grenzen ihrer Anwendung diskutiert.
International Journal of Training and Development, Sep 27, 2020
Self-assessment is a fundamental skill for professionals because self-assessment can promote self... more Self-assessment is a fundamental skill for professionals because self-assessment can promote self-regulated learning and professional development. However, studies reporting the use of self-assessment instruments in the professional training context are scarce. This study aimed to re-evaluate the psychometric properties of the Self-assessment Practice Scale (SaPS), which was originally developed in the school context, and extend its use to the professional training context. A sample of 200 investigative interviewer trainees from Australia and North America were invited to complete the modified SaPS. After removing three items, the confirmatory factor analysis results confirmed a first-order four-factor solution. The multidimensional Rasch analysis demonstrated that the resultant 16 items had satisfactory fit to the Rasch model. In general, the results supported the use of the 16-item modified SaPS as a valid measure for the sample in this study. The potential of using the SaPS in professional training contexts is discussed.
The Honours undergraduate degree in Australia is unlike that in most other countries. It has take... more The Honours undergraduate degree in Australia is unlike that in most other countries. It has taken on a particular significance as a qualification, as a pathway to and a prerequisite for direct entry into doctoral programs. This paper explores the outcomes of a study that suggests that the aims, outcomes, curriculum, pedagogical practices, purposes and enrolment patterns of Honours vary substantially across disciplines and university types. It addresses the questions about the diverse nature of Honours programs and questions what this diversity means for Australian higher education in the context where global standardisation of awards is rapidly occurring. Honours is seen variously as a qualification, an experience, or a program. These variations are discussed and it is demonstrated that Honours globally has not one, but many meanings. These meanings are often poorly understood within, and outside the academy. These multiple meanings create confusion about what Honours stands for and inhibit communication about the role and purpose of Honours both within Australia and in a global context.
Higher education research and development, Nov 21, 2014
Can extended opportunities for self-assessment over time help students develop the capacity to ma... more Can extended opportunities for self-assessment over time help students develop the capacity to make better judgements about their work? Using evidence gathered through students' voluntary self-assessment of their performance with respect to assessment tasks in two different disciplines at two Australian universities, the paper focuses on the effects of sequences of units of study and the use of different types of assessment task (written, oral, analysis, project) in the development of student judgement. Convergence between student criteria-based gradings of their own performance in units of study and those allocated by tutors was analysed to explore the calibration of students' judgement over time. First it seeks to replicate analyses from an earlier smaller-scale study to confirm that students' judgements can be calibrated through continuing opportunities for self-assessment and feedback. Second, it extends the analysis to coherently designed sequences of units of study and explores the effects of different types of assessment. It finds that disruptive patterns of assessment within a sequence of subjects can reduce convergence between student and tutor judgements.
Despite common assumptions that assessment practices in higher education are enduring, the past f... more Despite common assumptions that assessment practices in higher education are enduring, the past forty years has seen remarkable changes. A key change has been from the dominance of unseen end-of-year examinations, through the move to 'continuous assessment' and on to a range of diverse assessment approaches. Another notable change has been from assessment weightings being regarded as confidential, to the transparency of assessment standards and criteria of today. Assessment has thus shifted in some positive directions. Unfortunately, during the same period the emphasis on what we now call formative assessment has lessened: from a general acceptance that all courses involved the production of considerable written work that was used purely to aid learning, we now have regimes based on the assumption that all assessment must 'count' towards final marks or a grade point average.
Many universities advertise that their courses prepare students for work. Indeed, the main ration... more Many universities advertise that their courses prepare students for work. Indeed, the main rationale for courses for the professions is that they contribute to preparing students to become effective practitioners. While the more established professions recognise that there is a transitional period following graduation needed in this process, and indeed structured postgraduate training, newer professions have yet to embrace this feature. For both there is an assumption that whatever additional elements may be needed, the course itself is the main foundation. There is no shortage of features of courses claimed to prepare students for practice: various kinds of work--integrated learning, placements and practical work, authentic tasks and assessment activities . In some cases there are approaches to restructuring the entire curriculum to focus students' attention on the kinds of issues that practitioners deal with and their ways of thinking, for example, problem--based learning. But, can it be reasonably claimed that such approaches recognise the nature of practice, the nature of what it is to engage in deliberate practice and thus prepare students accordingly? This chapter suggests that many courses are far from exemplars of good educational practice for the professions. This is not primarily because of teaching quality, but because they tend to have a poorly conceptualised view of what it is that their graduates do in their professional practice. They are too often governed by what is involved in teaching within academic disciplines rather than on how learning occurs within professional work. There is a continuing risk that students will be trapped in current knowledge without the capacity to move beyond what they have been taught. And they are often not characterised by a strong sense that courses need to be actively designed and redesigned to produce graduates that will be deliberate professionals. The approach taken here will be to view professional practice and what is needed to become a deliberate practitioner, through the lens of research on workplace learning and of emerging practice theories. This involves starting with what practitioners do when they practise, and work from this to examine the implications for the courses that precede it. In particular the chapter focuses on what professionals require for learning in practice, one of the key aims of a pedagogy of deliberateness. It suggests that similar considerations apply to what students need to do in order to learn within courses. It starts by thinking about professional
Pre-assessment interventions aimed at helping students to understand assessment criteria and stan... more Pre-assessment interventions aimed at helping students to understand assessment criteria and standards have led to significant improvements in performance (Rust et al., 2003). However, broad adoption of these interventions has not been widespread, perhaps due to a perception that too much additional time may need to be spent in conducting these interventions. This presentation analyses a tightly-focused intervention to improve students‘ and tutors‘ understanding of assessment criteria and their judgement of required standards. It was integrated within an honours degree subject in Design using the web-based marking tool ReView, The intervention engaged students with the assessment criteria and standards of a design task exemplar using the same criteria that would be used to assess their work on this task. The tutor used 20 minutes of a normal lecture period working with a class of 80 students to grade the exemplar piece prior to the submission date. Onscreen grading sliders in ReView...
With the increasing challenges facing professional engineers working in more complex, global and ... more With the increasing challenges facing professional engineers working in more complex, global and inter-disciplinary contexts, different approaches to understanding how engineers practice and learn are necessary. This paper draws on recent research in the social sciences from the field of workplace learning, to suggest that a practice theory perspective on engineers' professional learning is fruitful. It shifts the focus from the attributes of the individual learner (knowledge, skills and attitudes) to the attributes of the practice (interactions, materiality, opportunities, challenges). Learning is thus more than the technical acquisition and transfer of knowledge, but a complex bundle of activities that is social, material, embodied and emerging. The paper is illustrated with examples from a research study of the learning of experienced engineers in the construction industry to demonstrate common practices—site walks and design review meetings—in which learning takes place.
Uploads
Papers by David Boud