Learning and Instruction 25 (2013) 24e34

Contents lists available at SciVerse ScienceDirect

Learning and Instruction

journal homepage: www.elsevier.com/locate/learninstruc

Using example problems to improve student learning in algebra: Differentiating

between correct and incorrect examples
Julie L. Booth a, *, Karin E. Lange a, Kenneth R. Koedinger b, Kristie J. Newton a
a
Temple University, 1301 W. Cecil B. Moore Avenue, Philadelphia, PA 19122, USA
b
Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213, USA

a r t i c l e i n f o a b s t r a c t

Article history: In a series of two in vivo experiments, we examine whether correct and incorrect examples with prompts
Received 30 April 2012 for self-explanation can be effective for improving students’ conceptual understanding and procedural
Received in revised form skill in Algebra when combined with guided practice. In Experiment 1, students working with the
5 November 2012
Algebra I Cognitive Tutor were randomly assigned to complete their unit on solving two-step linear
Accepted 11 November 2012
equations with the traditional Tutor program (control) or one of three versions which incorporated
examples; results indicate that explaining worked examples during guided practice leads to improved
Keywords:
conceptual understanding compared with guided practice alone. In Experiment 2, a more comprehensive
Worked examples
Self-explanation
battery of conceptual and procedural tests was used to determine which type of examples is most
Learning from errors beneficial for improving different facets of student learning. Results suggest that incorrect examples,
Conceptual understanding either alone or in combination with correct examples, may be especially beneficial for fostering
Algebra conceptual understanding.
Ó 2012 Elsevier Ltd. All rights reserved.

1. Introduction practicing rather than doing double the practice. This is thought to
occur because viewing the examples reduces cognitive load,
Numerous studies have demonstrated that learning is improved allowing students to focus on learning and understanding the steps
when students study worked examples while they practice solving of a problem’s solution (Sweller, 1999; Zhu & Simon, 1987).
problems in a laboratory setting (e.g., Sweller, 1999; Sweller & However, variations on the traditional format of worked examples
Cooper, 1985). Several studies have extended this work to may lead to different types of learning benefits. In the following
demonstrate that the use of worked examples can be beneficial for sections, we describe two such variations: Combination with self-
learning in real world classrooms over short-term lessons (Carroll, explanation prompts and the study of correct vs. incorrect
1994; Ward & Sweller, 1990), over longer-term computerized solutions.
lessons (Kalyuga, Chandler, Tuovinen, & Sweller, 2001; Kim, Weitz,
Heffernan, & Krach, 2009; Schwonke et al., 2007), and, most
recently, over long-term traditional lessons (Booth, Koedinger, 1.1. Worked examples with self-explanation
& Paré-Blagoev, submitted for publication).
The most robust finding regarding the use of worked examples One common variant is to couple worked examples with
is that replacing approximately half of the practice problems in prompts for students to explain the information to themselves.
a session with fully worked-out examples to study leads to Explaining instructional material has been shown to improve
improved procedural knowledge (e.g., Sweller & Cooper, 1985, learning by forcing students to make their new knowledge explicit
Ward & Sweller, 1990, etc.). That is, after studying examples, indi- (Chi, 2000; Roy & Chi, 2005). Logically, it then follows that asking
viduals learn to solve problems faster by viewing examples and students to explain examples could further improve their learning
over having them simply study examples. Indeed, Renkl, Stark,
Gruber, and Mandl (1998) found that including self-explanation
prompts with examples of interest calculation problems fosters
* Corresponding author. Tel.: þ1 215 204 6223; fax: þ1 215 204 6013.
E-mail addresses: [email protected] (J.L. Booth), [email protected]
both near transfer of problem solving skills (i.e., solving the type of
(K.E. Lange), [email protected] (K.R. Koedinger), [email protected] problem they practiced) and far transfer (i.e., solving problems that
(K.J. Newton). are related, but not isomorphic to those practiced (Haskell, 2001));

0959-4752/$ e see front matter Ó 2012 Elsevier Ltd. All rights reserved.
http://dx.doi.org/10.1016/j.learninstruc.2012.11.002
 J.L. Booth et al. / Learning and Instruction 25 (2013) 24e34 25

Catrambone and Yuasa (2006) also demonstrated that prompting implicitly) between possible actions when solving a problem, but
self-explanations yielded greater success at locating the relevant that as initial knowledge is often overly general, beginning learners
information needed to perform transfer tasks when utilizing often choose incorrect options. Ohlsson suggests that in order to
computerized databases. Aleven and Koedinger (2002) also improve their task knowledge, learners must first detect an error,
demonstrated that adding self-explanation prompts to a comput- identify the overgeneral knowledge that caused the error, and
erized tutor leads to increased declarative knowledge in Geometry, explain what additional conditions or features must be added to the
and Hilbert, Renkl, Kessler, and Reiss (2008) further suggest that overly general knowledge in order to make it correct.
adding self-explanation to worked examples improves students’ These ideas are also consistent with overlapping waves theory
conceptual knowledge of Geometry. These findings suggest that, at (Siegler, 1996), which maintains that individuals know and use
least for mathematical content, the addition of self-explanation a variety of strategies which compete with each other for use in any
allows worked examples to improve students’ understanding of given situation. With improved or increased knowledge, good
the underlying concepts inherent in the problems as well as their strategies gradually replace ineffective ones. However, for more
ability to carry out the steps they were shown. efficient change to occur, learners must reject their ineffective
strategies, which can only happen if they understand both that the
1.2. Studying correct vs. incorrect examples procedure is wrong and why it is wrong (i.e., which problem
features make the strategy inappropriate (Siegler, 2002)). Work in
Another dimension on which examples can differ which has the domain of science on conceptual change is based on a similar
received far less research attention is the nature of the solution in premise: inducing cognitive conflict along with providing accurate
the examples. While most worked-example research focuses on the information can help students to build correct conceptions (e.g.,
use of correct examples, recent work suggests that asking children Diakidoy, Kendeou, & Ioannides, 2003; Eryilmaz, 2002).
to explain a combination of correct and incorrect examples can be Collectively, these assertions suggest that the combination of
even more effective (Durkin & Rittle-Johnson, 2012; Rittle-Johnson, correct and incorrect examples is beneficial because the incorrect
2006; Siegler, 2002; Siegler & Chen, 2008); similar results have also examples help to weaken faulty knowledge and force students to
been found with older students (Huang, Liu, & Shiu, 2008) and attend to critical problem features (which helps them not only to
adults (Curry, 2004). The benefit of explaining errors is twofold. detect and correct errors, but also to consider correct concepts),
First, it can help students to recognize and accept when they have while the correct examples provide support for constructing correct
chosen incorrect procedures, leading to improved procedural concepts and procedures, beyond that embedded in traditional
knowledge over practice alone or correct examples plus practice instruction. It seems clear that both types of support are necessary,
(Siegler, 2002). Second, and perhaps more important, it can draw but what if extra support for knowledge construction is achieved
students’ attention to the particular features in a problem that through other types of innovative classroom practice? In that case,
make the procedure inappropriate. For instance, consider an would it still be optimal to provide a combination of correct and
example in which the equation 3x  4 ¼ 5 is incorrectly solved by incorrect examples, or would providing incorrect examples alone
subtracting 4 from both sides and resulting in a next problem state suffice for improving student learning? In the present study, we test
of 3x ¼ 1. By explaining how the procedure led to an incorrect the contribution of correct vs. incorrect examples in the context of
answer, students are forced to both accept that the procedure is such support for knowledge construction-guided problem-solving
wrong, and to notice that the negative sign that precedes the 4 practice with the Cognitive Tutor, a self-paced intelligent tutor
makes it inappropriate to apply the strategy. This can help the system which provides students with feedback and hints as they
student replace faulty conceptual knowledge they have about the practice (Koedinger, Anderson, Hadley, & Mark, 1997).
meaning of the problem features with correct conceptual knowl- In a series of two experiments, we examine the relative benefit
edge about those features; the acquisition of accurate, deep of explaining correct and incorrect examples alone and in combi-
features with which to represent problem situations is key to nation for improving conceptual and procedural learning for
building expertise (Chi, Feltovich, & Glaser, 1981). Consistent with students beginning Algebra. Algebra 1 is a particularly useful test-
this assertion, the combination of correct and incorrect examples bed, because it is considered to be a gatekeeper course in which
has been shown to lead to improvements in both conceptual success is thought to be crucial for access to advanced opportunities
understanding and procedural skill in Algebra compared with in mathematics and science. Despite its importance, many students
procedural practice alone (Booth et al., submitted for publication). entering Algebra 1 do not have accurate conceptual knowledge
In this study, the approach was shown to be especially beneficial for about the critical features found in equations, which hinders their
minority students (Booth et al., submitted for publication). ability to succeed in the course. For example, students tend to think
However, the inclusion of incorrect examples does not always lead that the equals sign is an indicator of where the answer belongs,
to increased benefit for all learners. For example, Große and Renkl rather than of balance between the two sides (Baroody & Ginsburg,
(2007) found that relatively novice learners cannot benefit from 1983; Kieran, 1981), that negative signs represent subtraction but
incorrect examples when they are expected to locate and identify do not modify the terms they precede (Vlassis, 2004), and that
the error in the example themselves. It makes sense that novice variables represent a single value (Booth, 1984; Knuth, Stephens,
students would have difficulty with this component, given that McNeil, & Alibali, 2006; Küchemann, 1978). Such misunderstand-
they likely make many of the mistakes themselves and may not ings have been found to have a detrimental effect on students’
recognize them as incorrect. However, further work is needed to ability to solve equations and to hinder students’ ability to learn
determine if removing this requirement and instead prompting new Algebraic content (Booth & Koedinger, 2008), and, unfortu-
students to explain what happened in the example and why it is nately, tend to persist even after targeted classroom instruction
incorrect may be beneficial. Further direct testing of correct (Booth, Koedinger, & Siegler, 2007; Vlassis, 2004). Thus, in order to
examples vs. a combination of correct and incorrect examples is increase student success in Algebra, it is imperative to utilize more
thus necessary for varied domains and student populations. effective methods of helping students to build stronger and more
Though research focusing specifically on incorrect examples has accurate conceptual knowledge, without sacrificing attention to
only emerged in the past decade, the idea that errors can be procedural skills. These goals may be accomplished by using
effective learning tools is not new. Ohlsson’s (1996) theory of a combination of worked examples and self-explanation in the
learning from errors maintains that individuals choose (perhaps domain of equation-solving, which is fundamental in Algebra 1.
26 J.L. Booth et al. / Learning and Instruction 25 (2013) 24e34

2. The present study tutor with students working individually on guided practice
problems.
The present study was conducted in collaboration with the Participating classrooms were recruited from three high schools.
Pittsburgh Science of Learning Center (PSLC), one of six research We were not permitted to collect demographic data for individual
centers created by the National Science Foundation (NSF) for the students, but we were able to retrieve ethnicity and SES data at the
purpose of promoting progress in the learning sciences. The main school level. One school was a west-coast high school in which 16%
goals of the PSLC include establishing connections between of students were economically disadvantaged; the ethnic break-
researchers, educators, and curriculum developers in order to down of attendees was: 43% Caucasian, 8% Black, 37% Hispanic, and
conduct high quality research studies to establish the benefit of 11% Asian. The other two schools were located in the Mid-Atlantic
instructional practices in real educational settings (Koedinger, region: a career and trade center for high school students (96%
Corbett, & Perfetti, 2012; see also http://learnlab.org). Teachers Caucasian, 4% Black), and a suburban high school (94% Caucasian,
whose classes take part in these in vivo experiments are encour- 5% Black, and 1% Asian). In both of these schools, 30% of attendees
aged to attend meetings with researchers to review research were economically disadvantaged. In all cases, students taking
project plans and evaluate the potential benefit a particular study regular Algebra 1 (rather than honors or remedial) were tested,
may have on student learning or potential difficulties that may arise thus we expect the demographics of the tested samples to reflect
in classroom implementation. In this setting, researchers, educa- those of the larger school populations.
tors, and curriculum developers collaborate to improve study ideas, Students were randomly assigned to one of four conditions:
and in ideal cases, devise new ideas for studies. three example-based conditions (1 with correct examples only, 1
There are two research questions for the present study. First, do with incorrect examples only, and 1 with correct and incorrect
worked examples with self-explanation improve student learning examples) and one control condition. Eighteen students were
in Algebra when combined with scaffolded practice solving prob- excluded from analysis because they did not complete the posttest,
lems? Previous work in geometry suggests that there is added leaving a final sample of 116 students: 86 example-based
benefit to explaining examples when embedded within Tutor (30 correct examples only, 31 incorrect examples only, and 25
activities, however, there is no evidence to date on whether correct and incorrect examples) and 30 control.
students using the Algebra 1 tutor, who are generally younger than
geometry students, would benefit similarly. The Cognitive Tutor has 3.1.2. Intervention
already been shown to be more effective for middle school students All conditions were situated within the solving two-step equa-
than traditional instruction (Ritter, Anderson, Koedinger, & Corbett, tions unit of the Algebra 1 Cognitive Tutor. Each of the students in
2007; What Works Clearinghouse, 2009), thus we are comparing the example-based conditions received a total of eight examples
the worked example approach against a more stringent, ecologi- interspersed into their practice session, with each example
cally valid control group (Corbett & Anderson, 2001). Our specific replacing a scheduled guided practice problem. The control
hypotheses for this research question are that (a) students who condition completed the original tutor unit, which contained only
explain worked examples will improve more than students who do guided practice problems with no examples.
not (control condition) (Hypothesis 1a), but that these effects will Each of the example-based problems illustrated either a correct
be stronger on the measure of conceptual knowledge (Hypothesis or an incorrect example of solving an equation. Students were then
1b), given that procedural performance is already supported asked to explain both what was done in the example and why the
and improved through the use of the Cognitive Tutor (Ritter strategy was either correct or incorrect; both steps are thought to
et al., 2007). be necessary for improving conceptual understanding (Rittle-
Second, are there differential effects on learning when students Johnson, 2006).
explain correct examples, incorrect examples, or a combination See Fig. 1 for sample screenshots of the interface for the
thereof? Typical use of examples in classrooms is largely focused on examples. In both correct and incorrect examples, students were
the use of correct examples to show students what they are to be shown the beginning state of a problem and the state of the
learning how to do. However, as mentioned in Section 1 above, problem after the first step; for incorrect examples, the mistakes
there are many reasons to believe that there is value added in were always real student errors collected in previous work about
explaining incorrect examples (e.g., Siegler & Chen, 2008). Results the particular features of the problem or about the goal of
from such studies support the use of incorrect examples in problem-solving in general (Booth & Koedinger, 2008). Students
combination with correct examples; the present study extends this were told that the step was either correct or incorrect, and were
work by evaluating the effectiveness of incorrect examples when asked to explain first “what” was done in the step, and then “why”
they are accompanied by guided practice, but not coupled with it is correct or incorrect. Students built their “what” and “why”
correct examples. Our specific hypothesis is that students who explanations from choices of sentence fragments. Students indi-
explain incorrect examples will improve more than students who cated “what” was done by building a sentence from a series of
explain only correct examples, again particularly for the measure of three menus. The choices in the first menu were operations
conceptual understanding (Hypothesis 2). (Added, Subtracted, Multiplied, Divided, and Moved) and once the
student selected an operation, the choices in the 2nd and 3rd boxes
3. Experiment 1 were dependent on that first choice (until a choice is made in the
first box, the other boxes are inactive). Students selected a choice
3.1. Methods from all three boxes to build an explanation, for instance, “Added 3
to both sides”, and then submitted their answer. They had to
3.1.1. Participants complete the “what” phase correctly before they could move on to
Participating in Experiment 1 were 134 high school students in the “why” phase to ensure that they were explaining the appro-
nine Algebra 1 classrooms using the Algebra 1 Cognitive Tutor priate step. For incorrect examples, students indicated “why” the
(Koedinger, Anderson, Hadley, & Mark, 1997). This curriculum is step was wrong by building a sentence from a series of two menus
commercially available and used in over 2000 school districts where the choices in the first box indicated whether the overall
across the United States. Classes, which use this curriculum, spend reason was because it was “illegal” or “legal but not helpful,” and
a portion of their instructional time in the computer lab using the the choices in the second menu were dependent on the choice
 J.L. Booth et al. / Learning and Instruction 25 (2013) 24e34 27

Fig. 1. Screenshots of the interface for correct and incorrect examples in Experiment 1.

made in the first menu (e.g., “It was illegal because it combined 3.1.3.2. Procedural knowledge. To assess students’ procedural
terms that were not like terms”; “It was legal but not helpful knowledge, we used 8 items that required students to carry out
because it did not reduce the number of terms”). For correct procedures to solve problems. Four of the items were isomorphic to
examples, students indicated “why” the step was right by using those taught during the tutor unit, and four were transfer problems,
a single menu box to complete the sentences “It is legal because.” which included features that students had not yet encountered
and “It is helpful because.”. These sorts of menu-based self- (e.g., two variable terms in the equation). All items were repre-
explanations have previously been shown to improve learning and sentative of the types of problems found in Algebra 1 textbooks and
transfer in other mathematical contexts (Aleven & Koedinger, taught in Algebra 1 courses; sample items can be found in the
2002). As with the guided practice problems, feedback and hints middle and right columns of Table 1. Student responses were coded
were available for all examples. as correct or incorrect. To establish reliability, 30% of the data were
coded by a second rater. Inter-rater reliability was sufficient for
3.1.3. Measures both isomorphic (k ¼ .87) and transfer problems (k ¼ .93).
To distinguish meaningful differences in knowledge growth
based on the intervention, specific items were designed to measure 3.1.4. Procedure
conceptual knowledge and procedural knowledge. Procedural Participating students were administered a paper-and-pencil
knowledge, for the purpose of this study, is identified as the how, test assessing their conceptual and procedural knowledge of
while conceptual knowledge is identified as the what and why. algebra. Two parallel forms of the test were created, in which the
Procedural knowledge encompasses knowing how to complete the structure of the problems was held constant across the two tests,
task, while conceptual knowledge identifies the important features but the surface features of the problems (e.g., the numbers and
and their meaning (the what), taken together to then understand letters) were changed. In order to counterbalance presentation, half
the appropriateness of the procedures (the why) (e.g., Booth, 2011). of the students in each class were randomly assigned to receive
Version A, and the other half took Version B as their pretest. All
3.1.3.1. Conceptual knowledge. To assess students’ conceptual students completed the pretest at the same time, but worked
knowledge, we used 54 items that measured understanding of individually; they took approximately 25 min to complete the test.
concepts identified in previous research as crucial for algebraic Following completion of the pretest, students logged on to the
equation solving (e.g., the meaning of the equals sign, the signifi- Cognitive Tutor and automatically received their assigned version
cance of negatives in terms, identification of like terms, etc.; Booth of the two-step linear equations unit, which they worked on during
& Koedinger, 2008; Kieran, 1981; Knuth et al., 2006; Vlassis, 2004; all of the scheduled computer lab sessions for their class until the
a ¼ .72). The measure was designed to assess a wide variety of unit was complete. The Tutor provided guided procedural practice
concepts related to the content, and combined experimenter- solving two-step equations; students received immediate feedback
designed items with well-established conceptual items that have about any errors, and could ask the Tutor for hints if they were
been developed by other top researchers in the field (e.g., Alibali, unsure of what to do; if assistance was requested of the teacher, it
Knuth, Hattikudur, McNeil, & Stephens, 2007; Asquith, Stephens, was only given after the student had asked the Tutor for hints,
Knuth, & Alibali, 2007). Sample conceptual items can be found in ensuring that the Tutor registered the student’s confusion. Students
the left-most column of Table 1. were told to work through the material individually, and at their
28 J.L. Booth et al. / Learning and Instruction 25 (2013) 24e34

Table 1
Sample assessment items for Experiment 1.

Conceptual Procedural isomorphic Procedural transfer

State whether each of the following is equal to 4x þ 3: 4x þ 5 ¼ 8 6 þ 3y ¼ 5y þ 8
a. 4x þ 3 Yes No
b. 3  4x Yes No
c. 4x  3 Yes No
d. 3 þ (4x) Yes No
e. 3 þ 4x Yes No
6 8
If 10x  12 ¼ 17 is true, state whether each of the following must also be true: ¼ 9 5 ¼ 1
b t
a. 10x  12 þ 12 ¼ 17 þ 12 Yes No
b. x  2 ¼ 17 Yes No
c. 10x ¼ 29 Yes No
d. 10x ¼ 17 Yes No
e. 10x  10  12  10 ¼ 17 Yes No
f. 10x  12 þ 12 ¼ 17 Yes No
d v
State whether each of the following is a like term for 8c: 3 ¼  7 ¼ 2
7 8
a. 5 Yes No
b. 8 Yes No
c. 6c Yes No
d. 3d Yes No
e. 5(c þ 1) Yes No
f. 4c Yes No

own pace, until the Tutor had determined that mastery was knowledge, and transfer procedural knowledge, with composite
reached on all requisite skills. This was done in order to maintain pretest scores used as a covariate. The multivariate effect did not
the ecological validity of a Cognitive Tutor classroom; had we reach significance, F(6,162) ¼ .96, p ¼ .46, h2p ¼ :03, indicating that
instead controlled for time on task, students would have gotten there were no differences detected among the three example-
varying levels of benefit from the Tutor itself. As soon as possible based conditions.
after each student completed the unit, he or she was administered
a posttest1, which again took approximately 25 min. Because 3.3. Discussion
students were allowed to complete the unit to mastery, and thus
take as long as they needed, the total study time varied for each Results from this experiment indicate that a combination of
student; however, most of the students had completed the study examples and guided practice problems is more beneficial for
within 4 weeks (eight class sessions). There were no significant student learning than guided practice alone (Hypothesis 1a). This
differences between conditions at pretest (F(3,115) ¼ 1.75, ns). experience fostered students’ conceptual understanding of the
features in equations (Hypothesis 1b) without sacrificing their
3.2. Results procedural knowledge, which was as good as or perhaps better than
that of students in the control group. This study provides evidence
Mean pretest and posttest scores for each condition on each of that the benefit of worked examples persists, even when combined
the three measures can be found in Table 2. To test whether with a successful, research-based curriculum. It also provides
explaining examples provided additional benefit to students when a conceptual replication of the findings of Booth et al. (submitted for
combined with the Cognitive Tutor, we first conducted a MANCOVA publication), demonstrating that the laboratory-proven worked
for condition (examples vs. no examples) on students’ posttest example principle does indeed transfer to classrooms.
conceptual knowledge and posttest isomorphic and transfer Results regarding the relative benefit of correct vs. incorrect
procedural knowledge; composite pretest scores were used as examples (Hypothesis 2), however, were not conclusive. In Exper-
a covariate (see Fig. 2). The multivariate effect of condition was iment 1, students in all three example-based conditions performed
significant F(1,111) ¼ 2.74, p < .05, h2p ¼ :07). Even though students equally, suggesting that perhaps it doesn’t matter what kind of
in the experimental condition solved 8 fewer practice problems examples a student gets, as long as examples are studied. However,
than those in the control, they performed just as well on isomor- it remains possible that such differences in benefit do exist for
phic post-test problems (42% correct vs. 41% in control group, experience with different types of examples, but they were not
F(1,113) ¼ .03, p ¼ .86, and trended toward better performance on detected in Experiment 1.
a measure of procedural transfer skill (24% vs. 16% correct, A more important possibility is that our measures of algebraic
F(1,113) ¼ 3.43, p ¼ .07, h2p ¼ :03). More importantly, students who competence in Experiment 1 may not have been sufficiently
received example-based assignments outscored control students comprehensive. We measured broad conceptual knowledge,
on a posttest measure of conceptual knowledge (63% vs. 58%
correct, F(1,113) ¼ 5.28, p < .05, h2p ¼ :05). Table 2
Next, to determine whether there were any differences among Mean pretest and posttest scores on each measure in Experiment 1 for students in
the three example-based conditions, we conducted a MANCOVA for each condition
condition (3: correct only, incorrect only, correct þ incorrect) on Conceptual knowledge Isomorphic procedural Transfer procedural
students’ posttest conceptual knowledge, isomorphic procedural
Pre Post Pre Post Pre Post
Control .62 (.09) .60 (.11) .43 (.27) .44 (.27) .16 (.19) .18 (.22)
1 Correct .53 (.15) .62 (.14) .33 (.27) .45 (.32) .23 (.26) .28 (.30)
Students in eight of the nine classrooms were administered the alternate
only
version of the paper-and-pencil test as a posttest (e.g., if they took version A for the
Incorrect .59 (.13) .65 (.10) .32 (.26) .40 (.29) .18 (.26) .24 (.23)
pretest, they took version B for the posttest); in one classroom, students were given
only
the same version of the test at pretest and posttest due to teacher error. Scores from
Combined .59 (.11) .60 (.14) .35 (.25) .39 (.34) .20 (.23) .24 (.30)
this class did not differ significantly from those in the other eight classes, so data
are collapsed across all nine classes. Note: Mean (SD).
 J.L. Booth et al. / Learning and Instruction 25 (2013) 24e34 29

feature knowledge in Experiment 1 was too broad. In designing the

measure, we included items to evaluate student knowledge of each
feature that is found in two-step algebraic equations. However, the
intervention was focused on improving student knowledge of three
features in particulardnegative signs, the equals sign, and like
terms. A more appropriate conceptual measure that would be more
sensitive to changes due to different types of examples may be one
that focuses on those three targeted features.
Thus, Experiment 2 was designed to specifically address one
overarching research question: What differences in learning exist
when students are exposed to the three types of experience with
examples? In this experiment, we include more examples in the
intervention, balanced across problem type. We also use a more
targeted measure of conceptual feature knowledge, a new measure
of skill in encoding problem features, and a new measure of flexi-
bility in problem solving, along with measures of isomorphic and
Fig. 2. Conceptual, procedural isomorphic, and procedural transfer scores for example-
transfer equation-solving problems. We again test the hypothesis
based and control students in Experiment 1. that students who explain incorrect examples (Combined and
Incorrect only conditions) will improve more on measures of
conceptual understanding than those who only explain correct
isomorphic procedural knowledge, and transfer procedural knowl- examples (Correct only condition) (Hypothesis 2); we also
edge, however there are specific facets of each of these types of hypothesize that students who explain traditional and unconven-
mathematical knowledge for which the intervention might have had tional correct examples (Correct only condition) will improve more
a larger impact. For example, perhaps students in different condi- in flexibility of problem-solving skills than those who do not
tions learn to represent problem features more accurately. Previous explain unconventional correct examples (Combined and Incorrect
work has shown that students do not accurately encode equations, only conditions) (Hypothesis 3).
often ignoring or changing the placement of the negative signs,
equals signs, and variables in the problem (Booth & Davenport, 4. Experiment 2
submitted for publication). In particular, we expect that viewing
incorrect examples might lead students to be more aware of the 4.1. Methods
features in the problem as they try to use them to describe why
a procedure is inappropriate, even if it does not lead to differences in 4.1.1. Participants
their responses on the conceptual knowledge measures of those Participants in Experiment 2 were recruited from eighth-grade
features. We thus included an additional conceptual knowledge Algebra I classrooms that used the Cognitive Tutor software in an
measure of students’ encoding of problem features. inner-ring suburban school (i.e., a suburb that is located close to
Another possibility that we had not previously considered is that a major city) in the mid-western United States, in which 29% of
the value added of viewing multiple correct examples of the same students are classified as low-income. Sixty-four students partici-
type (as is done in the correct-only condition) may not lead to pated in this study: 29 females (45%) and 35 males (55%). Fifty two
greater gains in equation-solving compared with viewing just one percent of the participants were Black, 37% were Caucasian, and
example of each type (as is done in the combined condition). 11% identified with multiple races. Students were randomly
Consistent with this assertion, viewing and explaining only one assigned to one of three example-based conditions (1 with correct
instance has been shown to be sufficient when students’ conceptual examples only (N ¼ 22), 1 with incorrect examples only (N ¼ 21),
knowledge is engaged during the instruction (Ahn, Brewer, & and 1 with correct and incorrect examples (N ¼ 21)).
Mooney, 1992). However, comparing multiple, varied instances of
correct examples (Rittle-Johnson & Star, 2007) and generating 4.1.2. Intervention
multiple solution methods (Star & Siefert, 2006) have both previ- The intervention was largely the same as in Experiment 1, with
ously been shown to affect the strategies students choose to use the following changes: First, each condition incorporated 12
when solving problems on their own, allowing them to become examples rather than 8. This was done in order to make sure
more adept at choosing more efficient strategies when available (e.g., students had the opportunity to work with the same number of
Rittle-Johnson & Star, 2007). We thus included an additional proce- examples in each of the problem types presented by the Tutor (i.e.,
dural knowledge measure of this problem-solving flexibility, in order 4 while solving two-step equations with addition and subtraction,
to extend this work to providing self-explanations of multiple correct 4 while solving two-step equations with multiplication, and 4
examples. To capitalize on this potential benefit, we also expanded while solving two-step equations with division).
the correct-only condition to include examples of both standard and Second, because exposure to more of the same type of correct
more unconventional but effective strategies (Star & Seifert, 2006). examples may not be necessary for learning when students are
Two more minor possibilities are that students didn’t receive providing explanations (Ahn et al., 1992), but exposure to varied
enough exposure to the different types of examples for there to be solution strategies for solving the same type of problem may
a differential effect; in Experiment 1, students received 8 examples promote flexibility in problem solving (Rittle-Johnson & Star, 2007),
over three problem types, meaning that for one of the problem the Correct only condition in Experiment 2 gave students 6
types, students received only two examples, regardless of condi- examples solved in the traditional correct method and 6 solved in
tion. Exposure to more examples, with better balance across unconventional but effective ways. To illustrate an unconventional
problem types, may be more optimal (recall that in all of the orig- correct example, consider the problem x/2  ½ ¼ 3/2. One could
inal worked-example studies, Sweller and colleagues utilize a one- solve the problem traditionally, by adding ½ to both sides and then
to-one ratio of problems to examples (e.g., Sweller & Cooper, 1985)). multiplying both sides by 2 to isolate the variable (x/2 ¼ 4/2; x ¼ 4).
Another minor possibility may be that the measure of conceptual However, it is also feasible to first multiply the whole problem by 2
30 J.L. Booth et al. / Learning and Instruction 25 (2013) 24e34

and then add 1 to find the value of x (x  1 ¼ 3; x ¼ 4). This method sufficient for both isomorphic (k ¼ .97) and transfer problems
may not be explicitly taught in school, but is an effective method to (k ¼ .92).
make the numbers in the problem easier to manipulate. Thus, for
each of the problem types presented by the Tutor (e.g., solving 4.1.3.4. Flexibility. We evaluated students’ ability to recognize and
equations with addition and subtraction), students in the Correct choose an effective strategy when solving an equation. Students
only condition received 2 examples using the traditional method were asked to solve three equations using two different strategies;
for solution and 2 using more innovative methods to find the for these problems, they received a flexibility score that was
correct solutions. All correct examples presented to students in the computed as either a 0, .5, or 1 for each problem (a ¼ .70). Receiving
combined group were conventionally solved. a 1 indicated that the student used two different methods to solve
Two minor changes were also made to the interface for the the problem, getting the correct answer both times. A score of .5
intervention. After analysis of both the problem tasks as well as indicated that the student did attempt two different methods, but
grade-level vocabulary, two additional options were added to the was not able to come to a correct answer using one of the two
first pull-down menu to allow students to select “Removed” and methods. A score of 0 indicated that the student was not able to
“Took the reciprocal of” to describe the mathematics occurring in identify two different methods to solve the equation. All items were
the steps to solve the equation. The vocabulary of describing why averaged into a flexibility score, with higher scores indicating that
the step was right or wrong was also changed, exchanging the words students were able to recognize and use more effective strategies
“legal” and “illegal” for “valid” and “invalid,” respectively, to more when solving equations.
accurately represent language used by students in the classroom.
4.1.3.5. Composite scores. For each student, we also computed
4.1.3. Measures three composite scores: Overall score (percent correct across all
We utilized two measures of conceptual understanding (feature feature knowledge, encoding, isomorphic equation-solving, trans-
knowledge and encoding) and two measures of procedural fluency fer equation-solving, and flexibility items), Conceptual Under-
(equation-solving and flexibility). We describe each of the standing score (percent correct across feature knowledge and
measures in more detail below. encoding items), and Procedural Fluency score (percent correct
across isomorphic equation-solving, transfer equation-solving, and
4.1.3.1. Feature knowledge. Feature knowledge questions measured flexibility items). The conceptual understanding and procedural
students’ understanding of concepts that have been identified in fluency measures were positively correlated (R(85) ¼ .50, p < .001),
previous research as crucial for success in Algebra; it was and the internal consistency for the composite measure was suffi-
comprised of 37 items from the conceptual test in Experiment 1 cient (a ¼ .83).
that measured student knowledge of three critical features: the
meaning of the equals sign, negative signs, and like terms (a ¼ .77). 4.1.4. Procedure
The percentage of feature knowledge items answered correctly was All procedures were identical to those in Experiment 1 except
computed for each student. that to avoid confusion about the version of the pretest and posttest
that should be given to each student there was only one version of
4.1.3.2. Encoding. Students’ encoding of problem features was the pencil and paper test; all students received that version as both
measured using a reconstruction task (e.g., Matthews & Rittle- their pretest and their posttest. There were no significant differ-
Johnson, 2009; McNeil & Alibali, 2004) in which students were ences between conditions at pretest (F(2,83) ¼ .04, ns), and pretest
presented with an equation for six seconds and then asked to and posttest composite scores were positively correlated
reconstruct the problem from memory immediately after it dis- (R(65) ¼ .58, p < .001).
appeared from view. Students completed this task for a series of six
equations with different structural formats and different place- 4.2. Results
ments of key problem features (e.g., 4x ¼ 9 þ 7x  6;
p  5 ¼ 2p þ 3). Student answers were coded for overall correct- First, to determine whether there was any effect of condition on
ness, and also in terms of the number and types of errors made on students’ scores in general, we conducted a 3-level (Condition)
the items (e.g., how many times did they move, drop, or insert ANCOVA on overall scores, controlling for pretest scores (see Fig. 3).
a negative sign from an equation; how many times did they switch A trend toward a main effect of condition was found for overall
one numeral with another). For each individual student we performance (F(2,64) ¼ 3.063, p < .10, h2p ¼ :009). Follow-up pair-
computed the percent of equations reconstructed correctly, the wise comparisons with Bonferroni correction revealed that
total number of errors made on the conceptual features (i.e., those students in the Combined condition (M ¼ 63%, SD ¼ 16%) scored
involving the negative sign, equals sign, or variable), and the total higher than students in the Correct only condition (M ¼ 51%,
number of non-conceptual errors (i.e., those involving letters or SD ¼ 22%; p ¼ .05); no other comparisons were significant.
numbers). To establish reliability, 30% of the data were coded by Mean pretest and posttest scores for each condition on each of
a second rater. Inter-rater reliability was sufficient for correct the five measures can be found in Table 3. To determine whether
reconstructions (k ¼ 1.00), conceptual errors (k ¼ .80) and non- there was any effect of condition on the conceptual understanding
conceptual errors (k ¼ .92). measures, we conducted a 3-level (Condition) MANCOVA on post-
test feature knowledge scores and posttest encoding errors on
4.1.3.3. Equation-solving. As in Experiment 1, we measured conceptual features, and posttest encoding errors on non-
students’ ability to effectively carry out procedures to solve three conceptual features, with overall pretest conceptual under-
isomorphic (i.e., problems that are identical in structure to those standing scores included as the covariate (see Fig. 4, left 2 columns).
they have been trained on; a ¼ .84) and three transfer problems The multivariate effect of condition was significant, F(6, 120) ¼ 3.11,
(i.e., problems that include additional features or alternate struc- p < .01, h2p ¼ :13. Significant univariate main effects of condition
tures from those they have been trained to solve; a ¼ .84). For each were found for conceptual features (F(2,64) ¼ 5.30, p < .01,
student, we computed the percent of isomorphic problems h2p ¼ :15) and conceptual encoding errors (F(2,64) ¼ 3.56, p < .05,
answered correctly and the percent of transfer problems answered h2p ¼ :10). Follow up pairwise comparisons with Bonferroni
correctly. Inter-rater reliability (computed on 30% of the data) was correction indicated that for the conceptual features measure,
 J.L. Booth et al. / Learning and Instruction 25 (2013) 24e34 31

Fig. 3. Overall posttest scores (adjusted for pretest scores) for students in each
condition in Experiment 2.

Fig. 4. Encoding, feature knowledge, isomorphic equation-solving, transfer equation-

students in the Combined condition (M ¼ 72%, SD ¼ 15%) scored solving, and flexibility posttest scores (adjusted for pretest performance) by condi-
tion in Experiment 2.
significantly higher than students who received the Correct only
condition (M ¼ 53%, SD ¼ 26%, p < .01). In addition, students in the
Incorrect only condition made fewer conceptual encoding errors
(M ¼ 2.0, SD ¼ 2.1) than students in the Correct only condition provided may not have much influence on procedural learning when
(M ¼ 4.4, SD ¼ 4.2; p < .05); the comparison between the Combined combined with guided practice. In general, the level of performance
and Incorrect only conditions was not significant. The univariate on the isomorphic and transfer problems was much lower than
main effect of condition on non-conceptual encoding errors was expected in both studies, especially given that prior research has
marginally significant (F(2, 64) ¼ 2.55, p < .10, h2p ¼ :08), however, shown the potential for significant improvement in procedural skill
follow-up pairwise tests revealed no significant differences when using the Cognitive Tutor (Ritter et al., 2007). Though all
between individual conditions. problems were of the types included in the students’ textbooks for
A parallel 3-level (Condition) MANCOVA was conducted on solving two-step (isomorphic) and multi-step equations (transfer),
posttest isomorphic equation-solving, transfer equation-solving, the easiest possible problems (e.g., ones with all positive numbers
and flexibility scores, with overall pretest procedural fluency and/or numbers that divided or combined neatly) were not repre-
scores entered as a covariate (see Fig. 4, right 3 columns). The sented in the measure. Future work should include simpler prob-
multivariate effect of condition did not reach significance for the lems to more accurately measure the range of learning that occurred
procedural fluency measures, F(6, 120) ¼ 1.38, p ¼ .23, h2p ¼ :06. in the lesson and determine whether differences in condition
emerge for students’ improvement in solving such simple problems.
Even more surprising, however, is that students in the Correct
5. General discussion only conditiondthe only condition to include unconventionally
solved problemsddid not see any significant improvement in
Experiment 1 indicated that combining guided practice with procedural fluency when compared to their peers. In particular, we
worked example problems benefited students’ conceptual knowl- expected that viewing unconventionally solved problems would be
edge (Hypotheses 1a and 1b); Experiment 2 measured the impacts crucial for improved flexibility (Hypothesis 3). However, all three
of particular types of examples. Results indicated that students groups performed equally on the procedural fluency tasks. One
performed best after explaining incorrect examples; in particular, possible reason for this finding is that improved conceptual
students in the Combined condition gained more knowledge than understanding (as occurred in the Combined and the Incorrect only
those in the Correct only condition about the conceptual features in conditions) may be sufficient to prompt improved flexibility; if
the equation, while students who studied only incorrect examples students have a better understanding of what the features in the
displayed improved encoding of conceptual features in the equa- equation actually mean, then they may be more likely to construct
tions compared with those who only received correct examples non-traditional ways of manipulating those features to solve prob-
(Hypothesis 2). lems. Another possible explanation is that the measure of flexibility
No differences were found between any of the conditions on any in Experiment 2 was focused on students’ recognition of multiple
of the procedural measures. For the isomorphic and transfer prob- effective methods for solving equations rather than their identifi-
lems, the combination of guided practice with the Cognitive Tutor cation of the most effective strategy. A further concern of note is that
and examples yields equal or better performance than the Cognitive since only one condition included unconventional examples, it is not
Tutor alone (Experiment 1), but the specific types of examples clear whether receiving them is not effective or whether receiving

Table 3
Mean pretest and posttest scores on each measure in Experiment 2 for students in each condition

Encoding conc errs Conceptual features Isomorphic procedural Transfer procedural Flexibility

Pre Post Pre Post Pre Post Pre Post Pre Post
Correct only 5.04 (3.18) 4.48 (4.20) .67 (.16) .52* (.26) .30 (.28) .28 (.26) .13 (.22) .19 (.30) .15 (.26) .13 (.29)
Incorrect only 3.67 (2.59) 1.90* (2.10) .67 (.19) .63 (.26) .51 (.39) .40 (.33) .32 (.32) .30 (.38) .22 (.30) .15 (.31)
Combined 4.38 (2.52) 2.90* (2.55) .66 (.16) .71* (.17) .30 (.35) .29 (.30) .22 (.34) .32 (.32) .14 (.33) .18 (.32)

Note: Mean (SD). *Posttest score significantly different from pretest score at p < .05.
32 J.L. Booth et al. / Learning and Instruction 25 (2013) 24e34

them in that particular context is not useful; it is also unknown how targeted a different possible misunderstanding about the equation
much flexibility improves just with the standard Cognitive Tutor features or solution process. Perhaps as long as students think
curriculum. Future research is necessary to investigate what types through ‘enough’ incorrect solutions, there is no added benefit to
and combinations of instruction are sufficient for improving seeing more. However, then it is strange that the Combined
recognition of alternate methods, and whether any of those conditiondwhich received ‘enough’ incorrect examples plus the
methods are superior for improving students’ strategy choice. same number of correct examplesddid not outperform the
In general, the two conditions that included incorrect examples Incorrect-only condition, even on procedural tasks for which
appear to be more beneficial improving conceptual knowledge than correct examples are frequently shown to improve learning.
the Correct only condition. This supports and extends previous work Perhaps the potential benefit of this combination was minimized
in several important ways. First, it provides evidence that the because students did not directly compare the two types of
combination of correct and incorrect examples is more beneficial examples (Rittle-Johnson & Star, 2007). Further work is needed to
than correct examples alone in the domain of Algebraic equation- investigate these questions, and to determine whether incorrect
solving; previous work in Algebra has focused on correct and examples might be more beneficial alone or in combination when
incorrect examples of word problems (Curry, 2004). It also confirms they are not also combined with the guided practice.
the effectiveness of this combination for middle school students Another limitation of the present study was that we do not know
(Huang et al., 2008). The present study was also the first to test whether students felt they were learning better from the interven-
whether a combination of correct and incorrect examples is more tion compared to their typical experience with the tutor, or whether
beneficial than incorrect examples alone, and suggests that receiving the teacher believed it to be the case. Observations that were con-
incorrect examples can be beneficial regardless of whether it is ducted in the classrooms suggest that students were able to work
paired with correct examples. This finding is especially important to well with the intervention and that the lesson went smoothly,
note because when examples are used in classrooms and in text- however it would be useful to know if the advantages of the
books, they are most frequently correctly solved examples. In fact, in example-based tutors are manifest solely in cognitive benefits, or if
our experience, teachers generally seem uncomfortable with the students’ self-efficacydor the belief that they can succeed (Bandura,
idea of presenting incorrect examples, as they are concerned their 1986)dchanged as a result of working with any of the conditions.
students would be confused by them and/or would adopt the Future studies should also examine process data to determine how
demonstrated incorrect strategies for solving problems. Our results explaining the examples impacted students’ work with the guided
strongly suggest that this is not the case, and that students should practice problems. For example, did it take them fewer problems to
work with incorrect examples as part of their classroom activities. reach mastery? Did they make fewer mistakes when solving prob-
Our results do not suggest, however, that students can learn lems after explaining correct vs. incorrect examples? It would also be
solely from explaining incorrect examples. It is important to note useful to know whether improvements in conceptual understanding
that all students saw correct examples, regardless of condition, not as demonstrated on our measures actually yielded noticeable
only because they are regularly included in textbooks and class- differences in students’ performance in their traditional lessons. All
room instruction, but because the correctly completed problems of these factors should be examined in future work.
the students produced with the help of the Cognitive Tutor could Nonetheless, though inadequate conceptual knowledge of
also be considered correct examples of sorts. We maintain that features in algebraic equations hinders students’ success in Algebra
students clearly need support for building correct knowledge, (Booth & Koedinger, 2008), the present study demonstrates
however, if that support is coming from another source (e.g., guided a successful plan for intervention: Provide examples along with
practice with feedback), spending additional time on correct guided practice problems to improve conceptual understanding.
examples may not be as important as exposing students to incor- Specifically, exposure to incorrect examples, which target typical
rect examples. student misconceptions about problem features and problem-
Future work should continue to investigate how the structure of solving approaches, may be crucial for developing students’
incorrect examples influences their benefit for students of varying conceptual knowledge. Viewing and explaining such incorrect
ability levels. For example, it seems likely that it would be critical to examples may help students to both confront their own miscon-
combine incorrect examples with self-explanation rather than ceptions as well as refine their understanding of the features in the
promoting aimless examination of errors. Further, the way in which problem; this may be especially so when students are guided to
students provide that self-explanation may change their level of notice critical features in the problems, either by forcing them to
benefit from incorrect examples; lower-achieving students in explain what changed between steps or by prompting them to
Große and Renkl (2007) were not successful with incorrect exam- answer specific questions about the mistakes in the problem.
ples, but such students in the present study did benefit, perhaps in Future research should also examine whether the optimal
part because they were supported in constructing reasonable self- combination of examples differs for students with varied levels of
explanations through use of the drop-down menu choices. As the background knowledge; such results would not be unprecedented
benefits of self-explanation are reliant on the learner generating the (Große & Renkl, 2007; Kalyuga, Chandler, & Sweller, 2001).
explanation (rather than simply reading a provided explanation; Systematic exploration of such individual differences in future
Chi, De Leeuw, Chiu, & Lavancher, 1994) this scaffolding may have studies is necessary to identify exactly when each type of examples
allowed students who would not normally be able to construct self- are most beneficial for individual students. This knowledge would
explanations to succeed in the process; affording them benefits for have the potential to improve instruction on equation-solving for
constructing new knowledge (Chi, 2000) and representing the students in the Cognitive Tutor curriculum, as well as to inform all
algebra problems (Neuman & Schwarz, 2000). teachers of example-based strategies that can be used to differen-
While results from the present study appear conclusive in tiate algebra instruction for their students.
supporting the use of incorrect examples in classroom activities,
several other important questions are raised from the intricacies of Acknowledgments
the results. For instance, why did no differences emerge between
students in the Combined and the Incorrect-only conditions on any Funding for this research was provided by the National Science
of the tasks? It seems odd that an increased number of incorrect Foundation, Grant Number SBE-0354420 to the Pittsburgh Science
examples wouldn’t influence performance, especially since each of Learning Center (PSLC, http://www.learnlab.org). The research
 J.L. Booth et al. / Learning and Instruction 25 (2013) 24e34 33

reported here was also supported by the Institute of Education Eryılmaz, A. (2002). Effects of conceptual assignments and conceptual change
discussions on students’ misconceptions and achievement regarding force and
Sciences, U.S. Department of Education, through Grant
motion. Journal of Research In Science Teaching, 39(10), 1001e1015. http://
R305A100074 to Temple University. The opinions expressed are dx.doi.org/10.1002/tea.10054.
those of the authors and do not represent views of the Institute or Große, C. S., & Renkl, A. (2007). Finding and fixing errors in worked examples: can
the U.S. Department of Education. this foster learning outcomes? Learning & Instruction, 17, 617e634. http://
dx.doi.org/10.1016/j.learninstruc.2007.09.008.
Portions of this work were presented at meetings of the Haskell, R. E. (2001). Transfer of learning. Cognition, instruction, and reasoning. San
Cognitive Science Society, the Eastern Psychological Association, Diego: Academic Press.
the Cognitive Development Society, and the American Education Hilbert, T. S., Renkl, A., Kessler, S., & Reiss, K. (2008). Learning to prove in geometry:
learning from heuristic examples and how it can be supported. Learning and
Research Association. Thanks are due to Golden Valley High School, Instruction, 18, 54e65. http://dx.doi.org/10.1016/j.learninstruc.2006.10.008.
Central Westmoreland Career and Trade Center, and Shaker Heights Huang, T.-H., Liu, Y.-C., & Shiu, C.-Y. (2008). Construction of an online learning system
Middle School. for decimal numbers through the use of cognitive conflict strategy. Computers &
Education, 50, 61e76. http://dx.doi.org/10.1016/j.compedu.2006.03.007.
Kalyuga, S., Chandler, P., & Sweller, J. (2001). Learner experience and efficiency of
instructional guidance. Educational Psychology, 21(1), 5e23. http://dx.doi.org/
References 10.1080/01443410124681.
Kalyuga, S., Chandler, P., Tuovinen, J., & Sweller, J. (2001). When problem solving is
Ahn, W. K., Brewer, W. F., & Mooney, R. J. (1992). Schema acquisition from a single superior to studying worked examples. Journal of Educational Psychology, 93(3),
example. Journal of Experimental Psychology: Learning, Memory, & Cognition, 579e588. http://dx.doi.org/10.1037/0022-0663.93.3.579.
18(2), 391e412. http://dx.doi.org/10.1037/0278-7393.18.2.391. Kieran, C. (1981). Concepts associated with the equality symbol. Educational Studies
Aleven, V., & Koedinger, K. R. (2002). An effective metacognitive strategy: learning in Mathematics, 12, 317e326.
by doing and explaining with a computer-based Cognitive Tutor. Cognitive Kim, R., Weitz, R., Heffernan, N., & Krach, N. (2009). Tutored problem solving vs.
Science, 26, 147e179. http://dx.doi.org/10.1016/S0364-0213(02)00061-7. “pure”: worked examples. In N. A. Taatgen, & H. van Rijn (Eds.), Proceedings of
Alibali, M. W., Knuth, E. J., Hattikudur, S., McNeil, N. M., & Stephens, A. C. (2007). the 31st Annual Conference of the Cognitive Science Society (pp. 3121e3126).
A longitudinal look at middle-school students’ understanding of the equal sign Austin, TX: Cognitive Science Society.
and equivalent equations. Mathematical Thinking and Learning, 9, 221e247. Knuth, E. J., Stephens, A. C., McNeil, N. M., & Alibali, M. W. (2006). Does under-
http://dx.doi.org/10.1080/10986060701360902. standing the equal sign matter? Evidence from solving equations. Journal for
Asquith, P. S., Stephens, A. C., Knuth, E. J., & Alibali, M. W. (2007). Middle-school Research in Mathematics Education, 37, 297e312.
mathematics teachers’ knowledge of students’ understanding of core algebraic Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. A. (1997). Intelligent
concepts: equal sign and variable. Mathematical Thinking and Learning, 9, 247e tutoring goes to school in the big city. International Journal of Artificial Intelli-
270. http://dx.doi.org/10.1080/10986060701360910. gence in Education, 8, 30e43.
Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory. Koedinger, K. R., Corbett, A. C., & Perfetti, C. (2012). The Knowledge-learning-
Englewood Cliffs, NJ: Prentice-Hall. instruction (kli) framework: bridging the science-practice chasm to enhance
Baroody, A., & Ginsburg, H. (1983). The effects of instructions on children’s robust student learning. Cognitive Science, 36, 757e798. http://dx.doi.org/
understanding of the equals sign. The Elementary School Journal, 84, 199e212. 10.1111/j.1551-6709.2012.01245.x.
http://dx.doi.org/10.1086/461356. Küchemann, D. (1978). Children’s understanding of numerical variables. Mathe-
Booth, J. L. (2011). Why can’t students get the concept of math? Perspectives on matics in School, 7, 23e26.
Language and Literacy, Spring, 2011, 31e35. Matthews, P. G., & Rittle-Johnson, B. (2009). In pursuit of knowledge: comparing
Booth, J. L., & Davenport, J. L. (submitted for publication). The role of problem self-explanations, concepts, and procedures as pedagogical tools. Journal of
representation and feature knowledge in algebraic equation-solving. Journal of Experimental Child Psychology, 104, 1e21. http://dx.doi.org/10.1016/
Mathematical Behavior. j.jecp.2008.08.004.
Booth, J. L., & Koedinger, K. R. (2008). Key misconceptions in algebraic problem McNeil, N. M., & Alibali, M. W. (2004). You’ll see what you mean: students encode
solving. In B. C. Love, K. McRae, & V. M. Sloutsky (Eds.), Proceedings of the 30th equations based on their knowledge of arithmetic. Cognitive Science, 28, 451e
Annual Cognitive Science Society (pp. 571e576). Austin, TX: Cognitive Science 466. http://dx.doi.org/10.1016/j.cogsci.2003.11.002.
Society. Neuman, Y., & Schwarz, B. (2000). Substituting one mystery for another: the role of
Booth, J. L., Koedinger, K. R., & Paré-Blagoev, E. J. (submitted for publication). Does self-explanations in solving algebra word-problems. Learning and Instruction,
practice alone make perfect? Taking example-based assignments for a test drive 10, 203e220. http://dx.doi.org/10.1016/S0959-4752(99)00027-4.
in real-world classrooms. Journal of Educational Psychology. Ohlsson, S. (1996). Learning from error and the design of task environments.
Booth, J. L., Koedinger, K. R., & Siegler, R. S. (2007). The effect of prior conceptual International Journal of Educational Research, 25(5), 419e448. http://dx.doi.org/
knowledge on procedural performance and learning in algebra. Poster presented 10.1016/S0883-0355(97)81236-0.
at the 29th Annual Meeting of the Cognitive Science Society in Nashville, TN. Renkl, A., Stark, R., Gruber, H., & Mandl, H. (1998). Learning from worked-out examples:
Booth, L. R. (1984). Algebra: Children’s strategies and errors. Windsor, UK: NFER- the effects of example variability and elicited self-explanations. Contemporary
Nelson. Educational Psychology, 23, 90e108. http://dx.doi.org/10.1006/ceps.1997.0959.
Carroll, W. M. (1994). Using worked examples as an instructional support in the Ritter, S., Anderson, J. R., Koedinger, K. R., & Corbett, A. (2007). Cognitive tutor:
algebra classroom. Journal of Educational Psychology, 86(3), 360e367. http:// supplied research in mathematics education. Psychonomic Bulletin & Review, 14,
dx.doi.org/10.1037/0022-0663.86.3.360. 249e255. http://dx.doi.org/10.3758/BF03194060.
Catrambone, R., & Yuasa, M. (2006). Acquisition of procedures: the effects of Rittle-Johnson, B. (2006). Promoting transfer: effects of self-explanation and direct
example elaborations and active learning exercises. Learning and Instruction, 16, instruction. Child Development, 77, 1e29. http://dx.doi.org/10.1111/j.1467-
139e153. http://dx.doi.org/10.1016/j.learninstruc.2006.02.002. 8624.2006.00852.x.
Chi, M. T. H. (2000). Self-explaining expository texts: the dual processes of generating Rittle-Johnson, B., & Star, J. R. (2007). Does comparing solution methods facilitate
inferences and repairing mental models. In R. Glaser (Ed.), Advances in instruc- conceptual and procedural knowledge? An experimental study on learning to
tional psychology (pp. 161e238). Mahwah, NJ: Lawrence Erlbaum Associates. solve equations. Journal of Educational Psychology, 99(3), 561e574. http://
Chi, M. T. H., De Leeuw, N., Chiu, M., & Lavancher, C. (1994). Eliciting self- dx.doi.org/10.1037/0022-0663.99.3.561.
explanations improves understanding. Cognitive Science, 18, 439e477. http:// Roy, M., & Chi, M. T. H. (2005). Self-explanation in a multi-media context. In
dx.doi.org/10.1207/s15516709cog1803_3. R. Mayer (Ed.), Cambridge handbook of multimedia learning (pp. 271e286).
Chi, M. T. H., Feltovich, P. J., & Glaser, R. (1981). Categorization and representation of Cambridge Press.
physics problems by experts and novices. Cognitive Science, 5(2), 121e152, Schwonke, R., Wittwer, J., Aleven, V., Salden, R. J. C. M., Krieg, C., and Renkl, A. (2007).
http://dx.doi.org/10.1207/s15516709cog0502_2. Can tutored problem solving benefit from faded worked-out examples? Paper
Corbett, A. T., & Anderson, J. R.. (2001). Locus of feedback control in computer-based presented at The European Cognitive Science Conference, Delphi, Greece.
tutoring: impact on learning rate, achievement, and attitudes. In Proceedings of Siegler, R. S. (1996). Emerging minds: The process of change in children’s thinking. New
ACM CHI’2001 Conference on Human Factors in Computing Systems, 245e252. York: Oxford University Press.
Curry, L. (2004). The effects of self-explanations of correct and incorrect solutions Siegler, R. S. (2002). Microgenetic studies of self-explanations. In N. Granott, &
on algebra problem-solving performance. In K. Forbus, D. Gentner, & T. Regier J. Parziale (Eds.), Microdevelopment: Transition processes in development and
(Eds.), Proceedings of the 26th annual conference of the cognitive science society learning (pp. 31e58). New York: Cambridge University.
(pp. 1548). Mahwah, NJ: Erlbaum. Siegler, R. S., & Chen, Z. (2008). Differentiation and integration: guiding principles
Diakidoy, I. A. N., Kendeou, P., & Ioannides, C. (2003). Reading about energy: the for analyzing cognitive change. Developmental Science, 11, 433e448. http://
effects of text structure in science learning and conceptual change. Contempo- dx.doi.org/10.1111/j.1467-7687.2008.00689.x.
rary Educational Psychology, 28(3), 335e356. http://dx.doi.org/10.1016/S0361- Star, J. R., & Seifert, C. (2006). The development of flexibility in equation solving.
476X(02)00039-5. Contemporary Educational Psychology, 31, 280e300. http://dx.doi.org/10.1016/
Durkin, K., & Rittle-Johnson, B. (2012). The effectiveness of using incorrect examples j.cedpsych.2005.08.001.
to support learning about decimal magnitude. Learning and Instruction, 22, Sweller, J. (1999). Instructional design in technical areas. Victoria, Australia: Austra-
206e214. http://dx.doi.org/10.1016/j.learninstruc.2011.11.001. lian Council for Education Press.
34 J.L. Booth et al. / Learning and Instruction 25 (2013) 24e34

Sweller, J., & Cooper, G. (1985). The use of worked examples as a substitute for Ward, M., & Sweller, J. (1990). Structuring effective worked examples. Cognition and
problem solving in learning algebra. Cognition and Instruction, 2(1), 59e89. Instruction, 7(1), 1e39. http://dx.doi.org/10.1207/s1532690xci0701_1.
http://dx.doi.org/10.1207/s1532690xci0201_3. What Works Clearinghouse (2009). Middle School Math: Cognitive Tutor Algebra 1.
Vlassis, J. (2004). Making sense of the minus sign or becoming flexible in ‘nega- Zhu, X., & Simon, H. A. (1987). Learning mathematics from examples and by
tivity’. Learning and Instruction, 14, 469e484. http://dx.doi.org/10.1016/ doing. Cognition and Instruction, 4, 137e166. http://dx.doi.org/10.1207/
j.learninstruc.2004.06.012. s1532690xci0403_1.