Available online at www.sciencedirect.

com

ScienceDirect

Oscillations, neural computations and learning during

wake and sleep
Hector Penagos, Carmen Varela and Matthew A Wilson

Learning and memory theories consider sleep and the reactivation of exploration to discover different courses when familiar routes
waking hippocampal neural patterns to be crucial for the long-term are not accessible. By contrast, using a map as a model,
consolidation of memories. Here we propose that precisely coordinated subjects could gain in generalization and flexibil- ity because
representations across brain regions allow the inference and evaluation of they would be able to devise, evaluate and plan alternative
causal relationships to train an internal generative model of the world. routes without previously having experi- enced them, as
This training starts during wakefulness and strongly benefits from sleep predicted by the cognitive map hypothe- sis [3,4]. Importantly,
because its recurring nested oscillations may reflect compositional knowledge of the spatial layout can have limited value when
operations that facilitate a hierarchical processing of information, planning and executing a plan, because a map does not
potentially including behavioral policy evaluations. This suggests that an consider conditions and rules that may govern navigation in a
important function of sleep activity is to provide conditions conducive to given environment. Answer- ing questions such as what paths
general inference, prediction and insight, which contribute to a more are available and why they are accessible or not is necessary
robust internal model that underlies generalization and adaptive for an individual to decide how to execute a plan to reach a
behavior. goal. Hence, incorporating information to answer these what,
why and how questions can lead to a more robust model that
Address generates appropriate actions under varying requirements and
Center for Brains, Minds and Machines, Picower Institute for Learning and contexts. Training such a generative model relies on extracting
Memory, Department of Brain and Cognitive Sciences, Massachusetts
meaningful structure from its inputs to learn statistical
Institute of Technology, Cambridge, MA, USA
representations that can account for the broad set of conditions
Corresponding author: Wilson, Matthew A ([email protected]) associated with them [5●●]. For neural systems this training
could start with the encoding of external information during
awake behavior and continue during periods of sleep, resulting
Current Opinion in Neurobiology 2017, 44:193–201 in more robust repre- sentations that increase behavioral
This review comes from a themed issue on Neurobiology of sleep flexibility [6]. Indeed, several studies demonstrate that sleep,
Edited by Yang Dan and Thomas Kilduff in addition to promoting memory consolidation, enables the
discovery of implicit rules and insights, which are essential
For a complete overview see the Issue and the Editorial
ele- ments for generalization and learning [7]. What compu-
http://dx.doi.org/10.1016/j.conb.2017.05.009 tational principles are at work during sleep to facilitate the
0959-4388/ã 2017 Elsevier Ltd. All rights reserved. consolidation of memories while also promoting generali-
zation as well as the inference of causal relationships?
While a dialogue between the neocortex and hippocam- pus
has been thought to mediate the systems consolida- tion of
memory and the slow incorporation of statistical regularities
into general cortical schemas [8●●,9,10], it is unclear whether
Introduction or how it could also contribute to the training of a generative
The challenge faced by the brain in perceiving and model that infers causal relation- ships. In the following
interpreting the external world is to map a high-dimen- sional sections, we explore the idea that, in the case of spatial
input into neural representations in the form of distributed navigation, the coordinated neural representations in the
spiking activity across brain regions, and to then infer causal hippocampus, neocortex and thal- amus during sleep, train a
relationships behind this sensory-driven code. A consequence of generative model that infers contextual and spatial
this inference is the generation of actions that allow an optimal contingencies, and which can be used during navigation to
interaction with the envi- ronment in different contexts. flexibly select actions to meet contextual conditions.
Computationally, the com- plexity of this challenge renders
solutions that rely on discriminative methods and lookup Predictive coding and neural network
tables as rigid and inefficient. Instead, solutions based on representations
probabilistic gen- erative models aim to learn the underlying
Machine learning methods can provide helpful theoreti- cal
rules behind external world observations, allowing more general frameworks for the implementation and training of flexible
appli- cability [1,2]. For instance, in the case of spatial
generative models in the brain. Although deep
explora- tion, using a lookup table constructed on past
navigational experiences, subjects would rely on trial and
error

www.sciencedirect.com Current Opinion in Neurobiology 2017, 44:193–201

194 Neurobiology of sleep

architectures have enjoyed a recent wave of success [11,12], that could be used for flexible behavior. For example,
they largely require explicit external feedback during training, during goal-directed navigation, a subject would benefit from
which contrasts with how the brain learns despite the lack of knowing the upcoming spatial layout of the envi- ronment
external feedback. Generative net- works, by contrast, while and the rules affecting potential choices. In rodents, the
typically not structurally deep, can extract statistical patterns in spatial component is given by the anticipa- tory firing of CA1
an unsupervised manner [13]. These types of models cast the place cells within cycles of the ongoing theta rhythm (8–12
brain as an inference device that compares predictions generated Hz) in which place cells with partially overlapping receptive
by an internal model of the world against the ongoing spike fields fire in a temporal sequence that reflects their relative
train code. Perception, for example, is then a constructive position on the maze [19,20]. These theta sequences represent
process by which the brain continuously tries to account for the upcoming position of the animal and, at decision points,
its sensations in terms of internally generated expectations or, reveal sweeps in the direction of all available options,
when a mismatch occurs, updates the model that generates consistent with an active, constructive evaluation of potential
the predictions [14●●,15●]. Stochastic recurrent neural networks choices [21]. Neocortical areas, including prefrontal and
offer a formal implementation that is consistent with this type retrosplenial cortices, could represent the rules and actions that
of predictive coding because they allow the sequential ulti- mately impact the decision of the subject [22–24].
estimation of probabilistic relationships between time- Con- sistent with this cooperative interaction, prefrontal neu-
dependent random variables through generative models. The rons are selectively phased locked to the hippocampal theta
temporal restricted Boltzmann machine (TRBM) [16●,17], rhythm as subjects approach decision points [25,26]. An
a forerunner to modern recurrent neural networks, offers an additional structure that could assist in coordinating consistent
intuition of how this process unfolds. The TRBM is expectations and representations across brain areas is the
composed of a sequence of individual RBMs [18] (Box 1) thalamus, given its role in the expression of intended
each of which contains two sets of stochastic binary units. navigational trajectories in prefrontal cortex and hippocampus
A layer of visible units receives the input to the RBM and is [27], its projections to multiple neocortical regions and its
linked to a set of hidden units through connection weights potential role in modulating the hippo- campal theta
in such a way that the state of the units and the strength of oscillation [28,29]. How is this complex network, involving
their connections can, through training, extract and encode the multiple brain regions, trained to support predictive
statistical regularities of the inputs. Using this computational coding?
design as a conceptual framework, even without a specific
correspondence with detailed brain anatomy, can provide Training a generative model during awake
insights about how neocortical, hippocampal and thalamic behavior
networks act together to implement and train a generative Predictive coding relies on correcting errors resulting from
model of the world comparisons between internal predictions and actual
observations. This error, estimated through a process

Box 1 Restricted Boltzmann Machine as a computational model for learning across brain states

The stochastic nature of RBMs allows the estimation of the statistical structure implicit in its inputs in an unsupervised way mimicking the challenge faced by the brain in interpreting
external stimulus, and it provides the initial conditions for the hidden units in the immediately following time step. Physiologically, this could correspond to the encoding and retrieval
The recurrent TRBM (RTRBM, (b) bottom) provides a mathematical advantage over the TRBM because, at any given time step, the state of the hidden units is obtained by a determin
 represented by the hidden layer of the RTRBM. As explored in the main text, the interconnected anatomical relationships between the thalamus, neocortex and hippocampus, along w

(a)General RBM training algorithm Step 1Step

2.- Update hidden Units1.- Fix Hidden Units
H

V
1.- Fix Visible Units (input data)
2.- Sample Visible Un

(b) RBM variants for sequential data

TRBM
init
H0 H1 H2 H3 H4

V0 V1 V2 V3 V4

time
RTRBM
H2
init
H0 H1 H3

H’0 H’1 H’2 H’3 H’4

V0 V1 V2 V3V4
Current Opin

known as contrastive divergence [30] in the RBM formu- model updates as evidenced by the finding that, in response
lation, drives an update in the connection weights between to changes in reward amounts at spatial loca- tions,
visible and hidden layers (Box 1). In models of the hippocampal place cell activity changes from sig- naling the
neocortex, this is postulated to occur through the interaction animal’s current position to depicting, in reverse order, the
of lower and higher cortical layers, but training could also start at trajectory that led the subject to its current rewarded location
earlier processing stages if the neocortex sent its expectations to [32,33]. Coupling this reverse trajectory replay with bursting
the thalamus [15●]. The error signal could then be generated dopaminergic activity [34] would produce a location-value
through the inhibitory effect of the thalamic reticular nucleus, gradient (highest at the reward location) that could serve two
which receives inputs from both the neocortex and the dorsal purposes: first, in the hippocampus, establish location-based
thalamus, on thalamocortical cells. In the hippocampus, error reward expecta- tions and second, in neocortical sites, attribute
esti- mation can be calculated by comparing spiking activities values to the actions that led to reward. Combining these
during the encoding and retrieval phases of the theta rhythm new associations would effectively yield a more abstract
[26,31]. The outcome of this operation can lead to state-action representation that combines relevant
information such as space, reward and additional contin- structure that promotes the integration of information in a
gencies of a particular experience that can alter the subject’s hierarchical manner. In its first proposed role, this func- tional
behavioral policy to maximize reward collection. Note that this unit mediates specific computations depending on its internal
reverse representation of spatial trajectories is accompanied by temporal microstructure, which is supported by neocortical
fast oscillations in the hippocampal local field potential (LFP), cells preferentially spiking at different phases of the up
known as sharp wave ripples (100–300 Hz), which are state [36]. In addition, spindles (7– 15 Hz) may be present
otherwise predominantly found during non-rapid eye within the SO, typically following the K-complex, and their
movement (NREM) sleep. Because neocortical activity during peak amplitude can happen at different phases of the up state.
this state is temporally related to hippocampal ripples, sleep A related empirical observation is that spindles peaking in
could be an ideal state for model training that generalizes the late phase of the SO are more predictive of subsequent
these state-action representations to include additional learning [37]. In addition to spindles, delta waves (1–4 Hz)
scenarios and environments. may also be present at various phases of the SO [38,39]. Non-
neocor- tical areas such as the thalamus and hippocampus con-
Training a generative model during sleep tribute to the microstructure of the SO [36,40]. Thalamic
If training through error correction occurs as behavior takes activity is involved in spindle pacemaking and could also be
place, what is the role of sleep? If no additional involved in the initiation of an up state [41,42].
modifications occurred during sleep, each internal activity pattern Hippocampal ripples (50–100 ms) preferentially occur before
produced during wakefulness would be almost exclusively and after K-complexes, delta waves and spindles [43]; when
tied to the set of conditions directly experi- enced by an they happen in multi-ripple bursts (including up to eight
individual. Although for a few navigational experiences this ripples) [44], they can also be phased-locked to individual
might seem like a pragmatic solution, the brain would rapidly cycles of a spindle [45●]. Overall, the nested nature of these
face an impossible challenge: to uniquely encode an oscillations suggests a means by which complex
exponentially large number of states with a finite number of computations can be decomposed into elemen- tary operations
internal spiking patterns. This would hinder the ability of across brain areas. These operations can be used recursively
the brain to generalize and infer causal relationships beyond in a sequential manner to extract fundamental associations
those experienced by the subject. Because during sleep the between neural representations potentially revealing causal
external input that normally drives the update of the model is relationships between them.
not present, we propose that sleep provides an opportunity for
the model to evaluate novel representations that could not Artificial recurrent networks and sleep dynamics Current
otherwise be considered, given the strong constraints imposed learning models posit that statistical regularities of the
by external input during wakefulness. environment are slowly consolidated over time through
cortico-hippocampal interactions during sleep, resulting in
In the following sections we will consider how the tem- more robust internal representations that can facilitate
poral organization of sleep and the computations across brain subsequent learning [8●●,10]. We propose that the extraction
regions can contribute to this extended training. of the statistics of the world during sleep is one of a more
extensive set of computations whose ultimate goal is to
Nested oscillations during NREM sleep infer causal dependencies between rules, actions and states
The recurring organization of NREM sleep provides an that produce favorable outcomes during behavior. Therefore,
appealing temporal setting for model training that could favor a as statistical regu- larities emerge, they are sampled and
hierarchical and compositional processing of infor- mation where combined proba- bilistically with rules and action representations
increasing levels of abstraction are identi- fied at to reveal alternative means to solve previously encountered
correspondingly increasing timescales and incor- porated into pro- blems. In the case of navigation, the process of sampling
the model. Specifically, neuronal activity during NREM sleep and combining representations of spatial paths with rules and
is characterized by alternations of suppressed (down) and outcomes could result in the partition of behavioral trajectories
elevated (up) spiking that corre- spond to the cortical slow into shorter paths that can reveal intermediate positions as key
oscillation (SO, 0.1–1 Hz), where a large-amplitude nodes, or sub-goals, in reaching a desired goal location. Paths
biphasic wave known as K- complex marks the down state. that link several of these nodes can then be recombined as
Because the SO is present throughout NREM sleep and modules that produce novel routes to be evaluated as potential
because it groups additional sleep rhythms [35], it could navigational policies in sub- sequent behavior [46,47].
represent a functional unit that serves two functions: first, to
trigger the recursive execution of elementary operations across Consistent with this proposal, a recent experiment in which
brain areas within each up state and, second, to link the rats learned odor-place associations highlights the co-
outcomes of these operations in a sequential fashion over occurrence of these processes during sleep: rats learned the
an extended period of time, leading to a temporally nested rules and spatial mapping of the task over
several days and, once performance was asymptotic, they were observation that neurons in the medial entorhinal cortex can
able to extrapolate the rules to new odor-location pairs in a sustain elevated firing activity across several neocor- tical up
single training session. Critically, the ability to perform the states [53] suggests an effective means by which the brain
task successfully with the newly paired items was eliminated if could implement a dilated causal convolution- like operation
the hippocampus was lesioned before rats had the opportunity to link temporal relationships across cycles of the SO.
to sleep following training. In other words, when the task
required the extrapolation of rules beyond purely spatial In summary, while several artificial network architectures can
changes to the layout of the environment, an intact discover causal relationships across representations, the RTRBM
hippocampus was required for sleep to have a positive impact provides guiding principles that are consis- tent with the
on behavior [48]. temporal structure of sleep oscillations.
Intuitively, establishing causal relationships requires that several Learning through simulations during sleep
neural representations interact over time to infer their correct An important difference between wake and sleep is that,
temporal dependencies. Several recurrent networks can during the latter, the generative model is exclusively
successfully learn and generate sequential data and suggest producing and comparing internal predictions across layers.
potential implementations of model training during sleep. Physiologically, this means that the neocortex and
What these networks have in com- mon is that operations hippocampus are, in principle, not constrained to encode
between visible and hidden layers are performed iteratively representations that match identical navigational experiences.
over time and the outcome at one time step influences the Instead, each area can produce variations of those
immediately following com- putation, consistent with the representations that can be combined in a compo- sitional
recurring structure of sleep. They differ, however, in how manner to produce simulations that yield increas- ing levels of
they incorporate longer- range influences of one abstraction in the internal network. At the most basic level,
computation over upcoming representations and whether neocortical and hippocampal patterns that are not linked to
their operations can be broadly mapped onto physiological identical spatial experiences can help reveal equivalent
events. For example, the long short-term memory formulation features of an environment, lead- ing to the reconstruction of
adds an internal state or memory cell to the hidden layer at each a map that can be used for navigation. Next, including non-
time step [49]. The content of the memory cell remains spatial sensory dimen- sions into these internal simulations can
unchanged for, and can influence, an arbitrary number of bring about rules associated with this map. Lastly, combining
computa- tion time steps. While effective in enabling long- these inferred rules and locations with representations of
range interactions across representations, it is unclear what the actions could assess their value in different contexts and
physiological equivalent of this memory cell would be. By improve the predictive power of the model. Ulti- mately,
contrast, the recurrent TRBM (RTRBM, Box 1) does not these hierarchical processes can lead to richer and more flexible
explicitly include long-range relationships, while empha- sizing representations because they can answer the what, why and
dependencies between adjacent computations [16●]. Despite how questions that allow for generaliza- tion and appropriate
this shortcoming, the RTRBM is reminis- cent of the cortico- behavioral policies within and across environments.
hippocampal temporal dynamics observed within and
across individual SOs (Figure 1) suggesting a broad These tiered processes can correspond to the multiple
correspondence to how the neocortex and hippocampus scales of temporal dynamics of the SO reflected by its
influence each other’s representations within individual nested oscillations (Figure 1A). For example, recall that multi-
periods of the SO, in line with recent electrophysiological ripple bursts can be phase-locked to spindles resulting in a
findings [50]. repeating pattern of alternating cortico- hippocampal activity.
Because representations of extended trajectories in a multi-
An interesting consideration is that sleep representations could ripple burst arise from the sequential expression of shorter
violate the temporal order in which they are pro- duced segments within single ripples [44], individual spindle cycles
during wake. This notion is captured in the bidi- rectional provide an opportunity for cortical influence on the content of
network formulation, which adds a second hid- den layer so imme- diately upcoming ripple-encoded spatial snippets. A
that at each computation step, one hidden layer receives potential consequence is the expression of novel trajec- tories,
information from previous states and the other hidden layer like shortcuts [54], that link segments of the environment
receives information from future states [51]. This configuration over a multi-ripple burst. Likewise, indi- vidual ripples could
emphasizes the notion that the role of one representation may trigger the neocortical expression of additional dimensions that
change upon its specific position within a sequence of events. allow the inference of rules and actions relevant to the
Contrasting with bidirectional networks, models using dilated environment. Importantly, ripples can also be tied to K-
causal con- volutions, although not recurrent, can effectively link complexes and delta waves;
past computations over long intervals preserving the temporal
order with which the network models the data [52]. The
Figure 1

(a)
Delta Spindle K-complex
Neocortex

Multi-ripple burst
Hippocampus

Slow Oscillation
(b)

V0 V1 V2 V3 V4 V5

H’5
H’0 H’1 H’2 H’3 H’4

H0 H1 H2 H3 H4 H5

Current Opinion in Neurobiology

Nested sleep oscillations under the RTRBM framework.

Sleep is characterized by thalamocortical and hippocampal oscillations whose temporal evolution is reminiscent of the sequential processing in the RTRBM. (a) Diagram
showing three schematic consecutive cycles of neocortical slow oscillation (SO) and the nested accompanying rhythms in neocortex and hippocampus. The
neocortical slow oscillation is a recurring pattern that groups the major thalamocortical sleep rhythms including K-complexes, spindles and delta waves.
Hippocampal ripples are also grouped by the SO and tend to occur in temporal proximity to neocortical sleep waves. Each up state in the SO may serve as a
functional unit to process information and extract patterns that train different aspects of an internal generative model. (b) RTRBM-like interactions may happen within
individual up states in which different neocortical regions assume the role of the visible layer, Vt, and the hippocampus that of the intermediate hidden layer, H’t. The network
update realized within a given SO functional unit depends on its internal temporal structure, the neocortical areas engaged dominantly with the hippocampus and the
state of the network in the previous block of computations. For example, during a delta-dominated SO, interactions (red lines) between neocortical (V) and hippocampal
(H’) representations could result in the incorporation of rules to the internal generative model (H). This, in turn, could bias (green lines) cortical and hippocampal
representations to be evaluated iteratively in individual cycles (dotted box) of an upcoming spindle-multi-ripple- burst event. The result can provide a means to infer
spatial layouts or to simulate actions (e.g., a novel trajectory) in these inferred virtual environments. Ultimately, these sequential processing chains can lead to more
abstract representations that support generalization and casual inference.

the finding that elevated prefrontal-hippocampal spike [24]. Together, these observations suggest two possibili- ties:
correlations were observed in the absence of spindles the extraction of rules could take place either during the late,
within up states [43] suggests further computations linked post-spindle, phase of an SO, or within SOs dominated by
to these additional neocortical oscillations. In particular, delta oscillations alone. Besides inferring relations between
spiking patterns encoding rules in prefrontal cortex are rules and space, sleep may allow the opportunity to improve
subsequently replayed toward the end of neocortical up behavioral policies through the evaluation of actions in
states, coinciding with ripples in the hip- pocampus as well as simulated contexts. The retro- splenial cortex lies at the
delta oscillations in prefrontal cortex intersection of allocentric and
egocentric representation streams and its spiking activity also of sleep could enable the identification of modules that can
encodes action-related variables [55,56]. Hence, its be flexibly combined in a compositional fashion to yield
interactions with the hippocampus and ventral striatum novel behavioral policy representations that can be evaluated
[57] could carry out this final state-action evaluation for through sleep simulations [47,60]. To this end, the RBM and
network training during SOs. In this stage, the coordi- nated its recurrent variants provide a general framework of
activities of retrosplenial cortex and hippocampus could computational principles across different brain states that lead
simulate multiple route options in order to assess their value to the establishment of robust generative models of the
for efficient navigation, with multi-ripple bursts potentially world. The observation that humans are able to imagine and
mediating this simulation, given their representation of anticipate scenarios, even in the absence of external cues,
extended behavioral sequences, and ventral striatum provides intuitive proof of the existence of this type of
providing reward associations. models in the brain, which can identify and establish causal
relationships between contexts, actions and outcomes, and not
Importantly, the recurrent operations of the RTRBM do not merely recall past events. In addition to providing a useful
change across time steps. Instead the representations of the framework to interpret physiological observations, the RBM
intermediate and visible layers dictate what aspects of the formula- tion raises important questions that require further
generative model can be learned (Figure 1B). theo- retical and experimental work. For instance, experiments in
Physiologically, this implies that multiple brain structures can which generalization within and across environments can be
differentially interact with the hippocampus to fulfill different assessed before and after sleep at the electrophysi- ological and
training needs. For example, sensory cortices would behavioral level, can examine the contribution of sleep to our
preferentially contribute to extraction of map features, proposed hypotheses of model updating during this state. In
prefrontal cortex would enable rule identifica- tion and addition, manipulations that target the temporal relationships
retrosplenial cortex would be favored for policy and action across areas by altering phase rela- tionships between specific
evaluation. The thalamus, with its extensive projections and oscillatory events and/or the representations associated with
gating mechanisms, could bias cortical activation and them, can shed light into the actual computations that take place
representation patterns that engage with the hippocampus for in neural circuits. For example, experiments directed at
different aspects of model training. For instance, it could breaking multi- ripple bursts into individual ripples, changing the
modulate the duration, structure and content of ripple-spindle duration of spindles, or altering the phase relationships with
computations and it could provide a link across SOs to one another, could provide opportunities to examine neural
implement an effective strategy that prioritizes meaningful representations in these altered states. Moreover, directing these
representations to update the generative model while avoiding manipulations to retrosplenial, prefrontal or hippo- campal areas
exhaustive comparisons that may not add value to it. could provide further assessments into their contributions to
inferring maps, rules and actions, and lead to improved
In summary, sleep network training can be thought of as a theoretical architectures that better account for the structure
simulation involving coordinated activity between tha- lamic, and function of the brain.
cortical and hippocampal areas. During this state,
representations of past experiences are combined to infer the Conflict of interest statement
spatial structure and valid rules in an environment. Together
Nothing declared.
with the evaluation of simulated actions, these inferences
effectively answer the what, why and how questions that
lead to a flexible model of the external world. In the end Acknowledgements
We thank members of the Wilson lab for comments on an earlier version of this
this model may enable the brain to compose unexplored manuscript. This material is based upon work supported by the Center for Brains,
scenarios, something that could not otherwise happen during Minds and Machines (CBMM), funded by National Science Foundation Science
awake behavior given the strong constrains imposed by and Technology Center award CCF-1231216 and by a NARSAD Young
Investigator Award from the Brain & Behavior Research Foundation to C.V.
external input.
References and recommended reading
Conclusions Papers of particular interest, published within the period of review, have been
Previous work has emphasized the contribution of sleep to the highlighted as:
formation of episodic and semantic knowledge and to the ● of special interest
identification of similar contexts in different envir- onments ●● of outstanding interest
[8●●,10]. However, the discovery of insights or implicit rules
for problem solving during sleep has been largely 1. Tenenbaum JB, Kemp C, Griffiths TL, Goodman ND: How to grow a
mind: statistics, structure, and abstraction. Science 2011, 331:1279-
unaccounted for [7,58,59]. Here, we explored 1285.
computational principles during sleep that endow the brain
2. Chater N, Tenenbaum JB, Yuille A: Probabilistic models of
with the ability to find new solutions to familiar problems cognition: conceptual foundations. Trends Cogn. Sci. 2006, 10:287-291.
or to extrapolate previous strategies to new scenarios. We
discussed how the temporal architecture
3. O’Keefe J: A computational theory of the hippocampal
cognitive map. Prog. Brain Res. 1990, 83:301-312. and conclude that such recurrent networks can be trained to implement generative
models that can be used for temporal prediction, e.g., moving stimuli, in
4. Pfeiffer BE, Foster DJ: Hippocampal place-cell sequences neocortical physiological networks.
depict future paths to remembered goals. Nature 2013, 16. Sutskever I, Hinton GE, Taylor GW: In Advances in Neural
497:74-79. ● Information Processing Systems 21. Edited by Koller D, Schuurmans
D, Bengio Y, Bottou L. Curran Associates, Inc.; 2009:1601-1608.
5. Hinton G: Where do features come from? Cogn. Sci. 2014,
●● 38:1078-1101. This paper introduces the mathematical formulation for the Recurrent Temporal
This paper offers a brief and informative review of the evolution of two major Restricted Boltzmann Machine (RTRBM) and it shows the mathematical
efforts in machine learning: deep networks and the Restricted Boltzmann equivalence between deep architectures and the RTRBM.
Machine. Using intuitive descriptions, the author contrasts the differences between
these types of networks. The author also outlines how stacking RBMs can lead to 17. Sutskever I, Hinton GE: Learning multilevel distributed
deep architectures that can learn statistical features in an unsupervised fashion and representations for high-dimensional sequences. AISTATS 2007,
can then be fine-tuned with external feedback. This paper is relevant because the 2:548-555.
Recurrent Temporal Restricted Boltzmann Machine is composed of sequential
layers of RBMs that are stacked in time. 18. Colin McDonnell: Lecture 12.3 — Restricted Boltzmann Machines [Neural
Networks for Machine Learning].
6. Hinton GE, Dayan P, Frey BJ, Neal RM: The ‘wake-sleep’
algorithm for unsupervised neural networks. Science 1995, 19. Skaggs WE, McNaughton BL, Wilson MA, Barnes CA: Theta phase
268:1158-1161. precession in hippocampal neuronal populations and the
compression of temporal sequences. Hippocampus 1996, 6:149-172.
7. Wagner U, Gais S, Haider H, Verleger R, Born J: Sleep inspires
insight. Nature 2004, 427:352-355. 20. Foster DJ, Wilson MA: Hippocampal theta sequences.
Hippocampus 2007, 17:1093-1099.
8. Kumaran D, Hassabis D, McClelland JL: What learning systems
●● do intelligent agents need? Complementary learning systems 21. Redish AD: Vicarious trial and error. Nat. Rev. Neurosci. 2016,
theory updated. Trends Cogn. Sci. 2016, 20:512-534. 17:147-159.
This paper reviews and updates the complementary learning systems (CLS) theory
to better account for behavioral and theoretical results. In its original form, the CLS 22. Smith DM, Barredo J, Mizumori SJY: Complimentary roles of the
theory proposed that the hippocampus can quickly store detailed experiences, while hippocampus and retrosplenial cortex in behavioral context
neocortex updates synaptic weights slowly to eventually store knowledge discrimination. Hippocampus 2012, 22:1121-1133.
representations that are less detailed but reflect environment statistics. Two
updates to the CLS theory are discussed: first, results from machine learning in 23. Vedder LC, Miller AMP, Harrison MB, Smith DM: Retrosplenial
multilayer networks suggest that the neocortex can also learn fast, provided that cortical neurons encode navigational cues, trajectories and
the new information is consistent with the model already stored in neocortical reward locations during goal directed navigation. Cereb.
networks; and second, the hippocampus can ensure that certain experi- ences are Cortex 2016 http://dx.doi.org/10.1093/cercor/bhw192.
remembered in detail by tagging salient experiences during their hippocampal [Epub ahead of print].
reactivation.
24. Peyrache A, Khamassi M, Benchenane K, Wiener SI, Battaglia FP:
9. Marr D: Simple memory: a theory for archicortex. Philos. Trans. Replay of rule-learning related neural patterns in the prefrontal
R. Soc. Lond. B Biol. Sci. 1971, 262:23-81. cortex during sleep. Nat. Neurosci. 2009, 12:919-926.
10. McClelland JL, McNaughton BL, O’Reilly RC: Why there are
25. Jones MW, Wilson MA: Theta rhythms coordinate hippocampal–
complementary learning systems in the hippocampus and
prefrontal interactions in a spatial memory task. PLoS Biol. 2005,
neocortex: insights from the successes and failures of 3(12):2187-2199.
connectionist models of learning and memory. Psychol. Rev.
1995, 102:419-457. 26. Hyman JM, Hasselmo ME, Seamans JK: What is the functional
11. Mnih V et al.: Human-level control through deep reinforcement relevance of prefrontal cortex entrainment to hippocampal
learning. Nature 2015, 518:529-533. theta rhythms? Front. Neurosci. 2011, 5:24.

12. Silver D et al.: Mastering the game of Go with deep neural 27. Ito HT, Zhang S-J, Witter MP, Moser EI, Moser M-B: A
networks and tree search. Nature 2016, 529:484-489. prefrontal-thalamo-hippocampal circuit for goal-directed
spatial navigation. Nature 2015, 522:50-55.
13. Freund Y, Haussler D: In Advances in Neural Information
Processing Systems 4. Edited by Moody JE, Hanson SJ, Lippmann RP. 28. Descheˆ nes M, Veinante P, Zhang ZW: The organization of
Morgan-Kaufmann; 1992:912-919. corticothalamic projections: reciprocity versus parity. Brain
Res. Brain Res. Rev. 1998, 28:286-308.
14. Vertes RP, Hoover WB, Viana Di Prisco G: Theta rhythm of the hippocampus: subcortical
●● Clark A: Whatever next? Predictive brains, situated agents, and control and functional significance. Behav. Cogn. Neurosci. Rev. 2004, 3:173-200.
the future of cognitive science. Behav. Brain Sci. 2013, 36:181- 204.
29.
The author reviews evidence in support of a unified hypothesis of perception, Hinton GE: Training products of experts by minimizing contrastive divergence. Neural Comput.
cognition and action that presents predictive coding, imple- mented through 2002, 14:1771-1800.
probabilistic generative models, as a fundamental com- putational tool in
neocortical networks. Under this hypothesis, sensory, cognitive and motor systems 31. Ketz N, Morkonda SG, O’Reilly RC: Theta coordinated error- driven
rely on estimated probability-density dis- tributions of external world variables to learning in the hippocampus. PLoS Comput. Biol. 2013, 9:e1003067.
generate top-down predictions that can be compared with ongoing sensory input.
The resulting error signal is sent forward along the cortical hierarchy to drive 32. Foster DJ, Wilson MA: Reverse replay of behavioural
perception and allow further tuning of the internal model. In humans, the top- sequences in hippocampal place cells during the awake state.
down predictions and feedforward error signals are proposed to also incorpo- rate Nature 2006, 440:680-683.
the statistics of sociocultural constrains, and to account for dysfunc- tional states
such as hallucinations and delusions. 33. Ambrose RE, Pfeiffer BE, Foster DJ: Reverse replay of
hippocampal place cells is uniquely modulated by changing
15. Rao RPN, Sejnowski TJ: Probabilistic Models of the Brain: reward. Neuron 2016, 91:1124-1136.
● Perception and Neural Function. The MIT Press; 2002:297-315.
The authors discuss and test the hypothesis that recurrent connectivity in neocortex 34. Gomperts SN, Kloosterman F, Wilson MA: VTA neurons
implements temporal predictive coding through a generative model that can be coordinate with the hippocampal reactivation of spatial
trained using the rules of spike-timing dependent synaptic plasticity (STDP). experience. eLife 2015, 4:e05360.
They test this hypothesis using a compart- mental model of interconnected
neurons with synaptic weights that change according to STDP when presented 35. Steriade M: Grouping of brain rhythms in corticothalamic
with sequential stimuli, systems. Neuroscience 2006, 137:1087-1106.
30.
36. Neske GT: The slow oscillation in cortical and thalamic Botvinick M, Toussaint M: Planning as inference. Trends Cogn. Sci. 2012, 16:485-488.
networks: mechanisms and functions. Front. Neural Circuits
2015, 9:88. 47. Donnarumma F, Maisto D, Pezzulo G: Problem solving as
probabilistic inference with subgoaling: explaining human
37. Demanuele C et al.: Coordination of slow waves with sleep successes and pitfalls in the Tower of Hanoi. PLoS Comput.
spindles predicts sleep-dependent memory consolidation in Biol. 2016, 12:e1004864.
schizophrenia. Sleep 2016.
48. Tse D et al.: Schemas and memory consolidation. Science 2007,
38. Steriade M, Dossi RC, Nun˜ ez A: Network modulation of a slow 316:76-82.
intrinsic oscillation of cat thalamocortical neurons implicated
in sleep delta waves: cortically induced synchronization and 49. Graves A: Generating Sequences With Recurrent Neural
brainstem cholinergic suppression. J. Neurosci. 1991, 11:3200-3217. Networks. ArXiv13080850 Cs (2013).
39. Nun˜ ez A, Curro´ Dossi R, Contreras D, Steriade M: Intracellular 50. Rothschild G, Eban E, Frank LM: A cortical-hippocampal-
evidence for incompatibility between spindle and delta cortical loop of information processing during memory
oscillations in thalamocortical neurons of cat. Neuroscience consolidation. Nat. Neurosci. 2017, 20:251-259.
1992, 48:75-85.
51. Schuster M, Paliwal KK: Bidirectional recurrent neural
40. Gardner RJ, Hughes SW, Jones MW: Differential spike timing and networks. IEEE Trans. Signal Process. 1997, 45:2673-2681.
phase dynamics of reticular thalamic and prefrontal cortical
neuronal populations during sleep spindles. J. Neurosci. 2013, 52. van den Oord A et al.: WaveNet: A Generative Model for Raw
33:18469-18480. Audio. ArXiv160903499 Cs; 2016.
41. Steriade M, Domich L, Oakson G, Descheˆ nes M: The deafferented 53. Hahn TTG, McFarland JM, Berberich S, Sakmann B, Mehta MR:
reticular thalamic nucleus generates spindle rhythmicity. J. Spontaneous persistent activity in entorhinal cortex
Neurophysiol. 1987, 57:260-273. modulates cortico-hippocampal interaction in vivo. Nat.
Neurosci. 2012, 15:1531-1538.
42. MacLean JN, Watson BO, Aaron GB, Yuste R: Internal dynamics
determine the cortical response to thalamic stimulation. Neuron 54. Gupta AS, van der Meer MAA, Touretzky DS, Redish AD:
2005, 48:811-823. Hippocampal replay is not a simple function of experience. Neuron
2010, 65:695-705.
43. Peyrache A, Battaglia FP, Destexhe A: Inhibition recruitment in prefrontal
cortex during sleep spindles and gating of hippocampal inputs. 55. Alexander AS, Nitz DA: Retrosplenial cortex maps the
Proc. Natl. Acad. Sci. U. S. A. 2011, 108:17207-17212. conjunction of internal and external spaces. Nat. Neurosci.
2015, 18:1143-1151.
44. Davidson TJ, Kloosterman F, Wilson MA: Hippocampal replay of
extended experience. Neuron 2009, 63:497-507. 56. Jacob P-Y et al.: An independent, landmark-dominated
46. head- direction signal in dysgranular retrosplenial cortex.
Nat.
45. Staresina BP et al.: Hierarchical nesting of slow oscillations, Neurosci. 2017, 20:173-175 http://dx.doi.org/10.1038/nn.4465.
● spindles and ripples in the human hippocampus during sleep. Nat.
Neurosci. 2015, 18:1679-1686. 57. Lansink CS, Goltstein PM, Lankelma JV, McNaughton BL, Pennartz
CMA: Hippocampus leads ventral striatum in replay of
This study explores the relationship between the slow oscillation, spindles place-reward information. PLoS Biol. 2009, 7:e1000173.
and ripples in the human hippocampus. The authors obtained intra- cranial
electroencephalogram (iEEG) recordings from 12 subjects and assessed the phase- 58. Mirkovic´ J, Gaskell MG: Does sleep improve your grammar?
amplitude relationship between sleep oscillations. They report a nested structure in Preferential consolidation of arbitrary components of new
which hippocampal ripples occur at the troughs of spindles, which in turn, take linguistic knowledge. PLoS One 2016, 11.
place during the negative phase of the hippocampal slow oscillation. Importantly,
they find that ripples occur in bursts with a repetition rate of 14.5 Hz, reinforcing 59. Monaghan P et al.: Sleep promotes analogical transfer in
the observation that multi-ripple bursts can also have a tight timing relationship with problem solving. Cognition 2015, 143:25-30.
spindles. These observations are significant because they provide a framework like
that of the neocortical slow oscillation through which the hippocampus and 60. Arnold AEGF, Iaria G, Ekstrom AD: Mental simulation of routes
neocortex can communicate with precise timing during sleep. during navigation involves adaptive temporal compression. Cognition
2016, 157:14-23.