Stark-Inbar A, Raza M, Taylor JA, Ivry RB. Individual differences in implicit motor learning: tas... more Stark-Inbar A, Raza M, Taylor JA, Ivry RB. Individual differences in implicit motor learning: task specificity in sensorimotor adaptation and sequence learning..—In standard taxono-mies, motor skills are typically treated as representative of implicit or procedural memory. We examined two emblematic tasks of implicit motor learning, sensorimotor adaptation and sequence learning, asking whether individual differences in learning are correlated between these tasks, as well as how individual differences within each task are related to different performance variables. As a prerequisite, it was essential to establish the reliability of learning measures for each task. Participants were tested twice on a visuomotor adaptation task and on a sequence learning task, either the serial reaction time task or the alternating reaction time task. Learning was evident in all tasks at the group level and reliable at the individual level in visuomotor adaptation and the alternating reaction time task but not in the serial reaction time task. Performance variability was predictive of learning in both domains, yet the relationship was in the opposite direction for adaptation and sequence learning. For the former, faster learning was associated with lower variability, consistent with models of sensorimotor adaptation in which learning rates are sensitive to noise. For the latter, greater learning was associated with higher variability and slower reaction times, factors that may facilitate the spread of activation required to form predictive, sequential associations. Interestingly, learning measures of the different tasks were not correlated. Together, these results oppose a shared process for implicit learning in sensorimotor adaptation and sequence learning and provide insight into the factors that account for individual differences in learning within each task domain. NEW & NOTEWORTHY We investigated individual differences in the ability to implicitly learn motor skills. As a prerequisite, we assessed whether individual differences were reliable across test sessions. We found that two commonly used tasks of implicit learning , visuomotor adaptation and the alternating serial reaction time task, exhibited good test-retest reliability in measures of learning and performance. However, the learning measures did not correlate between the two tasks, arguing against a shared process for implicit motor learning.
When a person fails to obtain an expected reward from an object in the environment, they face a c... more When a person fails to obtain an expected reward from an object in the environment, they face a credit assignment problem: Did the absence of reward reflect an extrinsic property of the environment or an intrinsic error in motor execution? To explore this problem, we modified a popular decision-making task used in studies of reinforcement learning, the two-armed bandit task. We compared a version in which choices were indicated by key presses, the standard response in such tasks, to a version in which the choices were indicated by reaching movements, which affords execution failures. In the key press condition, participants exhibited a strong risk aversion bias; strikingly, this bias reversed in the reaching condition. This result can be explained by a reinforcement model wherein movement errors influence decision-making, either by gating reward prediction errors or by modifying an implicit representation of motor competence. Two further experiments support the gating hypothesis. First, we used a condition in which we provided visual cues indicative of movement errors but informed the participants that trial outcomes were independent of their actual movements. The main result was replicated, indicating that the gating process is independent of participants' explicit sense of control. Second, individuals with cerebellar degeneration failed to modulate their behavior between the key press and reach conditions, providing converging evidence of an implicit influence of movement error signals on reinforcement learning. These results provide a mechanistically tracta-ble solution to the credit assignment problem. decision-making | reinforcement learning | sensory prediction error | reward prediction error | cerebellum W hen a diner reaches across the table and knocks over her coffee, the absence of anticipated reward should be attributed to a failure of coordination rather than diminish her love of coffee. Although this attribution is intuitive, current models of decision-making lack a mechanistic explanation for this seemingly simple computation. We set out to ask if, and how, selection processes in decision-making incorporate information specific to action execution and thus solve the credit assignment problem that arises when an expected reward is not obtained because of a failure in motor execution. Humans are highly capable of tracking the value of stimuli, varying their behavior on the basis of reinforcement history (1, 2), and exhibiting sensitivity to intrinsic motor noise when reward outcomes depend on movement accuracy (3–5). In real-world behavior, the underlying cause of unrewarded events is often ambiguous: A lost point in tennis could occur because the player made a poor choice about where to hit the ball or failed to properly execute the stroke. However, in laboratory studies of reinforcement learning, the underlying cause of unrewarded events is typically unambiguous, either solely dependent on properties of the stimulus or on motor noise. Thus, it remains unclear how people assign credit to either extrinsic or intrinsic causes during reward learning. We hypothesized that, during reinforcement learning, sensorimotor error signals could indicate when negative outcomes should be attributed to failures of the motor system. To test this idea, we developed a task in which outcomes could be assigned to properties of the environment or intrinsic motor error. We find that the presence of signals associated with movement errors has a marked effect on choice behavior, and does so in a way consistent with the operation of an implicit learning mechanism that modulates credit assignment. This process appears to be impaired in individuals with cerebellar degeneration, consistent with a computational model in which movement errors modulate reinforcement learning. Results Participants performed a two-armed " bandit task " (ref. 1, Fig. 1A), seeking to maximize points that were later exchanged for money. For all participants, the outcome of each trial was predetermined by two functions: One function defined if a target yielded a reward for that trial (" hit " or " miss "), and the other specified the magnitude of reward on hit trials (Fig. 1B). The expected value was equivalent for the two targets on all trials; however, risk, defined in terms of hit probability, was not. Under such conditions, people tend to be risk-averse (2, 6). We manipulated three variables: The manner in which participants made their choices, the feedback on " miss trials, " and the instructions. In experiment 1, participants were assigned to one of three conditions (n = 20/group). In the Standard condition, choices were indicated by pressing one of two keys, the typical response method in bandit tasks (1, 2). Points were only earned on hit trials Significance Thorndike's Law of Effect states that when an action leads to a desirable outcome, that action is likely to be repeated. However , when an action is not rewarded, the brain must solve a credit assignment problem: Was the lack of reward attributable to a bad decision or poor action execution? In a series of experiments , we find that salient motor error signals modulate biases in a simple decision-making task. This effect is independent of the participant's sense of control, suggesting that the error information impacts behavior in an implicit and automatic manner. We describe computational models of reinforcement learning in which execution error signals influence , or gate, the updating of value representations, providing a novel solution to the credit assignment problem.
Sensorimotor adaptation tasks have been used to characterize processes responsible for calibratin... more Sensorimotor adaptation tasks have been used to characterize processes responsible for calibrating the mapping between desired outcomes and motor commands. Research has focused on how this form of error-based learning takes place in an implicit and automatic manner. However, recent work has revealed the operation of multiple learning processes, even in this simple form of learning. This review focuses on the contribution of cognitive strategies and heuristics to sensorimotor learning, and how these processes enable humans to rapidly explore and evaluate novel solutions to enable flexible, goal-oriented behavior. This new work points to limitations in current computational models, and how these must be updated to describe the conjoint impact of multiple processes in sensorimotor learning.
Traditionally, motor learning has been studied as an implicit learning process, one in which move... more Traditionally, motor learning has been studied as an implicit learning process, one in which movement errors are used to improve performance in a continuous, gradual manner. The cerebellum figures prominently in this literature given well-established ideas about the role of this system in error-based learning and the production of automatized skills. Recent developments have brought into focus the relevance of multiple learning mechanisms for sensorimotor learning. These include processes involving repetition, reinforcement learning, and strategy utilization. We examine these developments, considering their implications for understanding cerebellar function and how this structure interacts with other neural systems to support motor learning. Converging lines of evidence from behavioral, computational, and neuropsychological studies suggest a fundamental distinction between processes that use error information to improve action execution or action selection. While the cerebellum is clearly linked to the former, its role in the latter remains an open question.
■ Sensorimotor adaptation occurs when there is a discrepancy between the expected and actual sens... more ■ Sensorimotor adaptation occurs when there is a discrepancy between the expected and actual sensory consequences of a movement. This learning can be precisely measured, but its source has been hard to pin down because standard adaptation tasks introduce two potential learning signals: task performance errors and sensory prediction errors. Here we employed a new method that induces sensory prediction errors without task performance errors. This method combines the use of clamped visual feedback that is angularly offset from the target and independent of the direction of motion, along with instructions to ignore this feedback while reaching to targets. Despite these instructions, participants unknowingly showed robust adaptation of their movements. This adaptation was similar to that observed with standard methods, showing sign dependence, local generalization, and cerebellar dependency. Surprisingly, adaptation rate and magnitude were invariant across a large range of offsets. Collectively, our results challenge current models of adaptation and demonstrate that behavior observed in many studies of adaptation reflect the composite effects of task performance and sensory prediction errors. ■
Stark-Inbar A, Raza M, Taylor JA, Ivry RB. Individual differences in implicit motor learning: tas... more Stark-Inbar A, Raza M, Taylor JA, Ivry RB. Individual differences in implicit motor learning: task specificity in sensorimotor adaptation and sequence learning..—In standard taxono-mies, motor skills are typically treated as representative of implicit or procedural memory. We examined two emblematic tasks of implicit motor learning, sensorimotor adaptation and sequence learning, asking whether individual differences in learning are correlated between these tasks, as well as how individual differences within each task are related to different performance variables. As a prerequisite, it was essential to establish the reliability of learning measures for each task. Participants were tested twice on a visuomotor adaptation task and on a sequence learning task, either the serial reaction time task or the alternating reaction time task. Learning was evident in all tasks at the group level and reliable at the individual level in visuomotor adaptation and the alternating reaction time task but not in the serial reaction time task. Performance variability was predictive of learning in both domains, yet the relationship was in the opposite direction for adaptation and sequence learning. For the former, faster learning was associated with lower variability, consistent with models of sensorimotor adaptation in which learning rates are sensitive to noise. For the latter, greater learning was associated with higher variability and slower reaction times, factors that may facilitate the spread of activation required to form predictive, sequential associations. Interestingly, learning measures of the different tasks were not correlated. Together, these results oppose a shared process for implicit learning in sensorimotor adaptation and sequence learning and provide insight into the factors that account for individual differences in learning within each task domain. NEW & NOTEWORTHY We investigated individual differences in the ability to implicitly learn motor skills. As a prerequisite, we assessed whether individual differences were reliable across test sessions. We found that two commonly used tasks of implicit learning , visuomotor adaptation and the alternating serial reaction time task, exhibited good test-retest reliability in measures of learning and performance. However, the learning measures did not correlate between the two tasks, arguing against a shared process for implicit motor learning.
When a person fails to obtain an expected reward from an object in the environment, they face a c... more When a person fails to obtain an expected reward from an object in the environment, they face a credit assignment problem: Did the absence of reward reflect an extrinsic property of the environment or an intrinsic error in motor execution? To explore this problem, we modified a popular decision-making task used in studies of reinforcement learning, the two-armed bandit task. We compared a version in which choices were indicated by key presses, the standard response in such tasks, to a version in which the choices were indicated by reaching movements, which affords execution failures. In the key press condition, participants exhibited a strong risk aversion bias; strikingly, this bias reversed in the reaching condition. This result can be explained by a reinforcement model wherein movement errors influence decision-making, either by gating reward prediction errors or by modifying an implicit representation of motor competence. Two further experiments support the gating hypothesis. First, we used a condition in which we provided visual cues indicative of movement errors but informed the participants that trial outcomes were independent of their actual movements. The main result was replicated, indicating that the gating process is independent of participants' explicit sense of control. Second, individuals with cerebellar degeneration failed to modulate their behavior between the key press and reach conditions, providing converging evidence of an implicit influence of movement error signals on reinforcement learning. These results provide a mechanistically tracta-ble solution to the credit assignment problem. decision-making | reinforcement learning | sensory prediction error | reward prediction error | cerebellum W hen a diner reaches across the table and knocks over her coffee, the absence of anticipated reward should be attributed to a failure of coordination rather than diminish her love of coffee. Although this attribution is intuitive, current models of decision-making lack a mechanistic explanation for this seemingly simple computation. We set out to ask if, and how, selection processes in decision-making incorporate information specific to action execution and thus solve the credit assignment problem that arises when an expected reward is not obtained because of a failure in motor execution. Humans are highly capable of tracking the value of stimuli, varying their behavior on the basis of reinforcement history (1, 2), and exhibiting sensitivity to intrinsic motor noise when reward outcomes depend on movement accuracy (3–5). In real-world behavior, the underlying cause of unrewarded events is often ambiguous: A lost point in tennis could occur because the player made a poor choice about where to hit the ball or failed to properly execute the stroke. However, in laboratory studies of reinforcement learning, the underlying cause of unrewarded events is typically unambiguous, either solely dependent on properties of the stimulus or on motor noise. Thus, it remains unclear how people assign credit to either extrinsic or intrinsic causes during reward learning. We hypothesized that, during reinforcement learning, sensorimotor error signals could indicate when negative outcomes should be attributed to failures of the motor system. To test this idea, we developed a task in which outcomes could be assigned to properties of the environment or intrinsic motor error. We find that the presence of signals associated with movement errors has a marked effect on choice behavior, and does so in a way consistent with the operation of an implicit learning mechanism that modulates credit assignment. This process appears to be impaired in individuals with cerebellar degeneration, consistent with a computational model in which movement errors modulate reinforcement learning. Results Participants performed a two-armed " bandit task " (ref. 1, Fig. 1A), seeking to maximize points that were later exchanged for money. For all participants, the outcome of each trial was predetermined by two functions: One function defined if a target yielded a reward for that trial (" hit " or " miss "), and the other specified the magnitude of reward on hit trials (Fig. 1B). The expected value was equivalent for the two targets on all trials; however, risk, defined in terms of hit probability, was not. Under such conditions, people tend to be risk-averse (2, 6). We manipulated three variables: The manner in which participants made their choices, the feedback on " miss trials, " and the instructions. In experiment 1, participants were assigned to one of three conditions (n = 20/group). In the Standard condition, choices were indicated by pressing one of two keys, the typical response method in bandit tasks (1, 2). Points were only earned on hit trials Significance Thorndike's Law of Effect states that when an action leads to a desirable outcome, that action is likely to be repeated. However , when an action is not rewarded, the brain must solve a credit assignment problem: Was the lack of reward attributable to a bad decision or poor action execution? In a series of experiments , we find that salient motor error signals modulate biases in a simple decision-making task. This effect is independent of the participant's sense of control, suggesting that the error information impacts behavior in an implicit and automatic manner. We describe computational models of reinforcement learning in which execution error signals influence , or gate, the updating of value representations, providing a novel solution to the credit assignment problem.
Sensorimotor adaptation tasks have been used to characterize processes responsible for calibratin... more Sensorimotor adaptation tasks have been used to characterize processes responsible for calibrating the mapping between desired outcomes and motor commands. Research has focused on how this form of error-based learning takes place in an implicit and automatic manner. However, recent work has revealed the operation of multiple learning processes, even in this simple form of learning. This review focuses on the contribution of cognitive strategies and heuristics to sensorimotor learning, and how these processes enable humans to rapidly explore and evaluate novel solutions to enable flexible, goal-oriented behavior. This new work points to limitations in current computational models, and how these must be updated to describe the conjoint impact of multiple processes in sensorimotor learning.
Traditionally, motor learning has been studied as an implicit learning process, one in which move... more Traditionally, motor learning has been studied as an implicit learning process, one in which movement errors are used to improve performance in a continuous, gradual manner. The cerebellum figures prominently in this literature given well-established ideas about the role of this system in error-based learning and the production of automatized skills. Recent developments have brought into focus the relevance of multiple learning mechanisms for sensorimotor learning. These include processes involving repetition, reinforcement learning, and strategy utilization. We examine these developments, considering their implications for understanding cerebellar function and how this structure interacts with other neural systems to support motor learning. Converging lines of evidence from behavioral, computational, and neuropsychological studies suggest a fundamental distinction between processes that use error information to improve action execution or action selection. While the cerebellum is clearly linked to the former, its role in the latter remains an open question.
■ Sensorimotor adaptation occurs when there is a discrepancy between the expected and actual sens... more ■ Sensorimotor adaptation occurs when there is a discrepancy between the expected and actual sensory consequences of a movement. This learning can be precisely measured, but its source has been hard to pin down because standard adaptation tasks introduce two potential learning signals: task performance errors and sensory prediction errors. Here we employed a new method that induces sensory prediction errors without task performance errors. This method combines the use of clamped visual feedback that is angularly offset from the target and independent of the direction of motion, along with instructions to ignore this feedback while reaching to targets. Despite these instructions, participants unknowingly showed robust adaptation of their movements. This adaptation was similar to that observed with standard methods, showing sign dependence, local generalization, and cerebellar dependency. Surprisingly, adaptation rate and magnitude were invariant across a large range of offsets. Collectively, our results challenge current models of adaptation and demonstrate that behavior observed in many studies of adaptation reflect the composite effects of task performance and sensory prediction errors. ■
Uploads
Papers by Richard Ivry