Internally generated hippocampal sequences as a vantage ...rnel.rice.edu/pubs/assets/Pezzulo2017.pdfPezzulo et al. Hippocampal sequences and future-oriented cognition Figure 1. Three

Ann. N.Y. Acad. Sci. ISSN 0077-8923

ANNALS OF THE NEW YORK ACADEMY OF SCIENCESIssue: The Year in Cognitive NeuroscienceREVIEW

Internally generated hippocampal sequences as a vantagepoint to probe future-oriented cognition

Giovanni Pezzulo,1 Caleb Kemere,2 and Matthijs A.A. van der Meer3

1Institute of Cognitive Sciences and Technologies, National Research Council, Rome, Italy. 2Electrical and ComputerEngineering, Rice University, Houston, Texas. 3Department of Psychological and Brain Sciences, Dartmouth College,Hanover, New Hampshire

Address for correspondence: Giovanni Pezzulo, Institute of Cognitive Sciences and Technologies, National Research Council,Via S. Martino della Battaglia 44, 00185 Rome, Italy. [email protected]; Matthijs A.A. van der Meer, Department ofPsychological and Brain Sciences, Dartmouth College, Hanover, NH 03755. [email protected]

Information processing in the rodent hippocampus is fundamentally shaped by internally generated sequences(IGSs), expressed during two different network states: theta sequences, which repeat and reset at the �8 Hz thetarhythm associated with active behavior, and punctate sharp wave-ripple (SWR) sequences associated with wakefulrest or slow-wave sleep. A potpourri of diverse functional roles has been proposed for these IGSs, resulting in afragmented conceptual landscape. Here, we advance a unitary view of IGSs, proposing that they reflect an inferentialprocess that samples a policy from the animal’s generative model, supported by hippocampus-specific priors. Thesame inference affords different cognitive functions when the animal is in distinct dynamical modes, associated withspecific functional networks. Theta sequences arise when inference is coupled to the animal’s action–perceptioncycle, supporting online spatial decisions, predictive processing, and episode encoding. SWR sequences arise whenthe animal is decoupled from the action–perception cycle and may support offline cognitive processing, such asmemory consolidation, the prospective simulation of spatial trajectories, and imagination. We discuss the empiricalbases of this proposal in relation to rodent studies and highlight how the proposed computational principles canshed light on the mechanisms of future-oriented cognition in humans.

Keywords: future-oriented cognition; internally generated hippocampal sequences; generative model; prediction;

prospection

Introduction

It’s a poor sort of memory that only works backwards.–Lewis Carroll

A fascinating open question in cognitive neuro-science is how we escape the present,1 or how wetemporarily detach from the here and now of thecurrent sensorimotor cycle to engage in forms ofcognition such as prospection, imagination, andmental time travel. The classic example of reducedsensory awareness is sleep, which enables the inter-nal generation of rich patterns of neural activityonly weakly constrained by sensory input. However,awake brains have periods of unconstrained, spon-taneous patterns, as activity in the default mode net-work (DMN) during off-task periods illustrates.2–5

When quizzed about their thoughts during these

periods, subjects often report thinking about thefuture.6 Crucially, the DMN is also engaged dur-ing on-task processing; as in the case of sleep,shared brain circuits support both online situatedaction coupled to the action–perception cycle anddetached, offline cognitive processing that escapesthe present. To understand cognition, we mustunderstand the principles that govern neural cir-cuits during and between these stimulus-driven orinternally generated “dynamical modes.”7,8

A model system for exploring how neural circuitsswitch between stimulus-driven (also stimulus-evoked) and internally generated (spontaneous orself-organized) modes is the hippocampus, and itsrole in spatial navigation as well as the encodingand retrieval of episodic memories in support ofadaptive action. Studies in rodents have provided

doi: 10.1111/nyas.13329

144 Ann. N.Y. Acad. Sci. 1396 (2017) 144–165 C© 2017 New York Academy of Sciences.

Pezzulo et al. Hippocampal sequences and future-oriented cognition

Figure 1. Three kinds of sequences in an ensemble of place cells recorded from hippocampal subfield CA1: behavioral sequence(blue), theta sequences (red), and sharp wave-ripple (SWR) sequence (green). (A) Activity of an ensemble of hippocampal placecells as an animal runs a spatial trajectory (in this case, a left turn on a T-maze); a spatiotemporal sequence of place cells is activated,starting with the place cell with its field at the start of the maze (bottom row; each row shows spikes from a single place cell), andending with the place cell with its field at the end of the maze (top row). Inset: Zooming in on the activity in the gray square showsrepeating, compressed theta sequences (red lines). (B) Activity from the same ensemble as the animal rests away from the track(but in the same room). Two synchronous bursts of activity, associated with network events called SWR complexes, punctuatean otherwise quiet (non-theta) activity regime. The second SWR activates cells in an order similar to that seen during behavior,forming an SWR sequence (green line). Data are adapted from Ref. 191.

access to the content of hippocampal dynamics atfast time scales and allow for temporally preciseexperimental interventions, providing an ideal van-tage point to probe detached and future-orientedcognition.9–19 Internally generated activity occursin different brain states (awake and attentive, awakerest, different sleep stages), with most experimentalstudies focusing on one state only. Similarly, indi-vidual laboratories typically study such activity in anarrow behavioral setting, different from that stud-ied in other laboratories. These factors have cre-ated a fragmented conceptual landscape containinga number of interpretations. Here, we aim to pro-vide a unified theoretical viewpoint on the differentforms of internally generated activity in the rodenthippocampus, consider the functions proposed forthese phenomena, and finally provide the bones of aunifying account, whose implications may extend tothe study of human future-oriented cognition, too.

Internally generated sequencesin the hippocampus and their rolesin detached cognition

Studies of information processing in the rodent hip-pocampus have historically focused on “place cells.”These neurons tend to be active in specific, spa-

tially restricted areas of a given environment, suchthat the animal’s location can be accurately decodedfrom an ensemble of simultaneously recorded placecells.20,21 It is now well established that the activityof place cells encodes other variables beyond loca-tion alone;22,23 moreover, hippocampal cells can tilenonspatial experience with restricted firing fields(“time cells” or “episode cells”;24,25) consistent withthe view that hippocampal activity encodes a “con-tinuous record of attended experience.”26 Never-theless, examining the activity of place cells from apurely spatial viewpoint continues to provide a use-ful access point into information processing in thehippocampus, as we discuss next.

By moving around in space, an animal will tra-verse sequentially the firing fields of a number ofplace cells, resulting in a spatiotemporal sequenceof neural activity at the time scale of behavior.This sequence is readily apparent from a raster-plot in which the rows, corresponding to neu-rons, are ordered according to the location of theirplace field (Fig. 1A). Note that approximately 7 sec-onds elapse between the activation of the first placecell, at the bottom of the plot, and the last placecell (at the top). Interestingly, there is additionalstructure in this sequential activation of place cells:

145Ann. N.Y. Acad. Sci. 1396 (2017) 144–165 C© 2017 New York Academy of Sciences.

Hippocampal sequences and future-oriented cognition Pezzulo et al.

at the time scale of the theta rhythm (�8 Hz inmoving rodents), repeating, temporally compressedsequences are formed (Fig. 1A, inset). In these thetasequences, place cells are active in the same order asin the slower time scale behavioral sequence. Thetasequences are compressed because the time betweenthe peak firing rates of a given pair of cells withina theta cycle is shorter than the time between peakfiring rates of the same cells at the overall behavioraltime scale (estimates of this compression factor vary,but are in the 5–10× range;27–30 note, however, thatthese sequences do not necessarily unfold at a con-stant speed, so any compression factor is an approx-imation). They are repeating because during eachtheta cycle, a new sequence is initiated and termi-nated. As a general rule, theta sequences are local(i.e., include the animal’s current location) and for-ward (i.e., proceed in the same direction the animalis moving). The period of the theta cycle limits thespace that can be spanned by a single theta sequence,although it should be noted that there is substantialdiversity between theta sequences, and the late phaseof the theta cycle, in particular, is associated with aless precise spatial representation (and perhaps evenreverse order31,32).

Theta sequences are of interest because theirstructure appears to be internally generated: sensoryinput as it presents to the animal does not start outwith repeating, compressed structure at the thetatime scale. Determining the mechanistic basis oftheta sequence generation is a topic of active experi-mental and computational work, discussed in detailelsewhere (including the connection with the single-cell phenomenon of theta phase precession33,34);here, we focus on the possible functional benefitsof organizing activity in this striking way.

The main proposals in the literature tend toemphasize two quite different ideas: (1) that thetasequences reflect online, short time scale predictionsof possible courses of action and their outcomes,useful in decision making,35 and (2) that thetasequences facilitate the initial storage of episodic-like memories by arranging spike times to beconducive to spike timing–dependent plasticity.32

Both of these ideas have proven to be difficult totest directly, although there is indirect support forboth. Striking experimental observations of thetasequences extending alternately down one arm andthen another as animals appear to deliberate atchoice points are suggestive of a role in planning,36

but in that particular experiment, animals’ choicescould not be predicted from the content of thetasequences. A subsequent study11 found a relation-ship between the length of theta sequences and thegoal animals subsequently ran to, although in theseand many other recording studies it is unclear ifthese activity patterns were in fact contributingto behavior. Similarly, the link to plasticity is alsoindirect. In a limited number of studies in whichpharmacological manipulations have disruptedtheta sequences,37,38 memory performance wasalso disrupted, although it remains unclear to whatextent this deficit results from encoding, retrieval,or nonsequence-specific sources. In general, thereis a large body of correlative evidence linking theproperties of the theta rhythm to both memoryencoding and retrieval,39–41 but relating these tocontent (i.e., decoded theta sequences) continues tobe an experimental challenge that precludes preciseinterpretations.

A second type of internally generated activity inthe hippocampus occurs during sharp wave-ripple(SWR) complexes, punctate events associated withhighly synchronous spiking.42 SWRs generally occurwhen the animal is at rest but awake, or in slow-wave sleep, and not when moving. However, partic-ularly in novel environments, the boundary betweenhigh-gamma oscillations and SWR is blurred, andit has been reported that SWRs occur in animalsduring exploration.43–46 During SWRs, place cellscan fire in similar order to that seen during behav-ior, prompting the familiar term “replay.” As withtheta sequences, the timing of these SWR sequencesis compressed relative to behavior, with estimatesof the compression factor ranging 10–40×;28,47 butnote here too that the speed is likely not constant.19

Unlike theta sequences, however, SWR sequencescan be forward and backward,48–50 even when theanimal has not actually run the behavioral sequencein the backward direction. SWR sequences caninclude the animal’s current location, but often donot, instead spanning a trajectory on the oppositeside of a large T-maze51 or a trajectory in a differ-ent maze entirely.52 Multiple SWRs can be chainedtogether, with the next SWR sequence starting wherethe previous one left off, to span long distances.47

While early studies emphasized the content of SWRsequences as replay, several studies have shownthat SWR sequences do not simply reflect recentexperience, but can include trajectories toward an



upcoming goal, be biased away from the currentlyrewarded trajectory, be novel combinations of pre-viously experienced paths, and be even trajectoriesthat have not been experienced at all9,19,43,51,53,54 (butsee Ref. 55).

Early theories about SWRs proposed that theycontribute to systems consolidation (i.e., a way fortime-limited memories stored in the hippocampusto be consolidated into neocortical memorystructures56). This idea is supported by studiesinterrupting SWRs during offline processing(sleep) that found impairments in the acquisitionof hippocampus-dependent spatial tasks57,58 (seealso Ref. 59) and activity in cortical and subcorticalstructures temporally aligned to the SWR.60–62

However, this view does not explain why SWRs arefrequently observed during pauses in task perfor-mance and appear to signal trajectories leading to agoal location (prospective) rather than replaying therecently taken trajectory (retrospective).12,63 Inter-rupting SWRs in the awake state, during task per-formance, led to a performance impairment on theoutbound leg of an alternation task (which requiredthe animals to remember the identity of the previoustrial) but not the inbound leg (which required noworking memory).14 These observations are morein line with an interpretation of awake SWRs asmemory retrieval or behavioral planning.42,64,65,192

From the above review of the properties of thetasequences and SWR sequences, the main modesof internally generated activity in the hippocam-pus, a number of open issues are apparent. Whatis the relationship between theta sequences andSWR sequences? How can we reconcile the differ-ences in content of SWR sequences observed ondifferent tasks? How can the underspecified ideas ofconsolidation and planning be made more specificas to produce testable predictions and new ideasfor experiments and interpretation? To address theabove issues, we consider both kinds of internallygenerated sequences (IGSs) as process componentsof an overall generative model architecture that sup-ports both adaptive action control and detachedcognition, by operating at different dynamicalmodes (stimulus tied and internally generated) andby forming functional networks that include differ-ent brain areas, depending on task demands (seeFig. 2 for a schematic of our overall proposal). Thiswould imply that detached cognition taps the sameneurocomputational resources (e.g., those produc-

ing sequential neuronal activity) that afford situ-ated action within action–perception cycles.171 Forthis to be possible, the same underlying mecha-nisms must be flexible enough to operate in differ-ent (stimulus-tied and internally generated) modesand at different time scales (i.e., time compression)while also engaging different brain networks in atask-dependent way. To understand how this may bepossible, below we discuss computational principlesthat may underlie sequential neuronal activity andits role across action–perception, memory function,and prospection within our architecture.

A computational perspective on IGSs

The starting point of this proposal is that the hip-pocampus forms internal generative models jointlywith other brain areas, which can be engaged (ornot) depending on task demands, including theentorhinal cortex (EC), the ventral striatum (VS),and the prefrontal cortex (PFC).62 The notion ofgenerative models is central in leading neurophys-iological theories of brain structure and function,such as predictive coding, the free energy prin-ciple, and the Bayesian brain,66,67 and is congru-ent with proposals emphasizing the constructivenature of hippocampal memories and its role inimagination.5,68 According to these converging per-spectives, brains are statistical inference devicesthat gradually acquire generative models, whichencode the statistics of the environment and ofagent–environment interactions (i.e., contingenciesbetween actions, sensations, and rewards). The lat-ter are especially important, as ultimately the brainuses generative models to perform inference in theservice of adaptive action.

As regularities in the exteroceptive, interocep-tive, and proprioceptive domains unfold at differenttime scales, generative models include various hier-archical levels, which influence one another con-tinuously and reciprocally, in both bottom-up andtop-down directions.69 In predictive coding par-lance, higher hierarchical levels propagate predic-tions (e.g., about an expected sensory stimulus)downward through the hierarchy, while lower hier-archical levels propagate prediction errors upward(e.g., difference between expected and sensed stim-ulus), and the impact of the “messages” is weightedby their (inverse) uncertainty or precision. Thisprocess can be cast with reference to Bayesianinference, if one considers that prior beliefs are



Figure 2. Schematic illustration of different modes within an overall generative model architecture for prefrontal cortex–hippocampus interactions. The different modes, illustrated in the rounded rectangular panels, are organized along a continuumof constraints from sensory input and task demands. At one extreme, corresponding to full engagement of the hippocampus inthe task (bottom left panel), the theta rhythm coordinates an action-perception cycle in which theta sequences create short-termpredictions, based on the generative model, that are integrated with sensory input and prediction errors propagated back up to themodel. In this theta mode, time-limited encoding of ongoing experience, consisting of episodic-like memory traces, is facilitated byrepeating, compressed theta sequences and the associated predictive processing. Top left panel: SWRs shaped by current sensoryinput are samples from the generative model relevant to the current task and can be used for planning and inference. Their contentwill reflect current task demands, and will tend to be prospective (e.g., tracing paths to a goal). Top right panel: During offline states,SWR sequences are the vehicles for signaling the content of previously encoded memory episodes (bottom left panel) to corticalregions for model updating (systems consolidation). This consolidation process occurs in the absence of sensory input and is insteaddriven by short-term plasticity on intrahippocampal synapses. In this mode, SWR content reflects recent experience. Bottom rightpanel: In the absence of recent short-term plasticity in the hippocampus, SWRs are generated that do not reflect recent experience,but instead serve to improve the generative model by enforcing self-consistency, finding simpler models to prevent overfitting andavoiding catastrophic interference.

iteratively revised as novel information acquiredthat is (in)compatible with expectations, thus form-ing posterior beliefs that are more informed. Thegeneral objective of a predictive coding system isto minimize prediction error (or free energy66)—in which case, predictions are compatible with theunfolding of sensory events.

The notion of Bayesian inference in (hierarchi-cal) generative models has been used to explain bothperception70–72 and action.73 In perceptual process-

ing, the higher hierarchical levels encode percep-tual hypotheses (e.g., I am looking a dog) that arerevised on the basis of bottom-up prediction errors(e.g., hearing a meow, not a bark, as I was expecting)until prediction error (or free energy) is minimizedand the “correct” hypothesis is identified (e.g., thisis a cat not a dog). However, rather than minimizeprediction error by revising my hypothesis, I canalso minimize it by acting so as to make the hypoth-esis true. For example, I can make my original (dog)



hypothesis true by searching for a dog in the visualscene and foveating it. In this case, the predictionspropagated by higher hierarchical levels act as goalstates, and engaging arc reflexes minimizes the ensu-ing prediction errors. This scheme can be extendedto the planning of sequences of actions (or poli-cies) if the agent can predictively compare the (inte-gral of) free energy or surprise conditioned on aseries of successive actions (e.g., whether turningtwice right or twice left will bring to the expectedgoal state).74,75 This example requires engaging thegenerative model to internally generate and evalu-ate sequences of predictions. A closely related ideais called planning as inference, in which the cur-rent and desired (goal) states are fed to the system(“clamped”) and Bayesian inference is used to “fillgaps” (i.e., to find a trajectory between the cur-rent and desired goal location).76–78 These planningmethods that rest on Bayesian inference over gener-ative models (that encode transitions between statesand probabilistic relations between states and obser-vations) will become relevant when we next discusspossible neuronal implementations of prospection.

Various modeling studies have applied the ideaof probabilistic (Bayesian) inference in generativemodels to hippocampal function (and surround-ing regions).79–83 One study highlights that thehippocampus (areas CA1 and CA3), the EC, andprefrontal areas have the right connectivity toencode the agent’s generative model of its envi-ronment and to support state estimation, sensoryimagery, and other functions, depending on theflow of the information between the areas.79 In thismodel, areas CA3–CA1 support state estimation bycombining path integration and sensory input fromthe lateral and medial EC, respectively. The samearchitecture can also afford sensory imagery whenthe medial EC receives (virtual) motor commandsfrom the PFC, uses path integration to update the(virtual) position, and communicates it to areasCA3–CA1, which in turn uses its projections tolateral EC to produce predictions of sensory states.

Another proposal posits that a prediction-matching mechanism is implemented within a hier-archical architecture that includes the EC, CA1,and CA3 at increasingly higher hierarchical levels.27

In this perspective, the most active cell assembliesin CA3 encode predicted positions, whereas ECencodes current inputs. It is when top-down (pre-dictive) and bottom-up (sensory) streams “match”

that CA1 cells fire, thus encoding the updated ani-mal location. This view uses predictive coding topredict current inputs, but it can be expandedto also cover predictions about future stimuli orlocations. Accordingly, it has been proposed thatthe first half of each theta cycle may be stimulusdriven and compute the current position (usingsensory and path integration information fromthe lateral and medial EC, respectively), whereasthe second half may depend on intrinsic networkdynamics involving both grid cells in the medial ECand the hippocampus and compute future spatialpositions.84

Yet another proposal focuses on a wider brainnetwork formed by the hippocampus and theVS, which may encode two components of agenerative model—transitions between locationsand contingencies between locations and rewards,respectively—affording Bayesian inference of thebest policy or plan.80 When uncertain, this systemsequentially samples from the (combined) inter-nal model to update the “value” of different poli-cies (say, for turning right or left at a junction),which results in the coordinated elicitation of longtheta sequences in the hippocampus and of covertreward expectations in the striatum.64 After suffi-cient experience, this system can select actions basedon “cached” action values without engaging the gen-erative model (hence in a model-free way85)—atleast while task contingencies remain stable. Thislatter aspect is consistent with evidence that spatialdecisions become hippocampus independent aftersufficient experience.86

Finally, a series of computational studies exploredthe idea that various aspects of spatial cognition,including spatial decision making, route planning,model selection, vicarious trial and error (VTE),and the covert evaluation of future spatial tra-jectories, may be based on probabilistic inferenceand a common generative model (implementedin the hippocampus and surrounding structures),also discussing various neuronal implementationsof (approximate) Bayesian inference.7,75,79,87,88

By leveraging and extending these and other mod-els, we discuss below what the notions of statis-tical inference and generative models can offer tothe study of prospective cognition and the rela-tions between stimulus-evoked and internally gen-erated brain activity as applied to theta and SWRsequences.



InferenceIt emerges from the aforementioned models thatstatistical (Bayesian) inference within generativemodels affords a homogeneous—yet very flexible—form of computation. We argue that hippocampalIGSs may reflect an inferential process that sam-ples a sequence from the animal’s generative model(or sequence generator). During spatial navigation,sequences code for spatial trajectories, which canbe evaluated (e.g., to check whether it leads to agoal location) to support use as a plan (or pol-icy) for navigation. In the absence of active nav-igation, sequences contribute to the constructionand maintenance of the generative model (imple-menting a number of processes in support of mem-ory consolidation). Below, we expand on this idea,discussing how the same generative scheme affordsvarious cognitive functions, including fasterprospective processes (e.g., policy selection and con-trol within action–perception cycles) and slowerprocesses (e.g., learning and memory consolidationover longer time scales). The key distinguishing ele-ment between these two alternatives is the “dynam-ical mode” under which the animal operates. Thedynamical mode is reflected in the temporal charac-teristics of brain rhythms: during theta, hippocam-pal processing is coupled to the action–perceptioncycle, whereas, during SWRs, there is weaker or eventotal uncoupling from external events.

Theta sequences. How can we represent naviga-tion to a known goal location as an inference prob-lem supported via IGSs? Imagine that the animalknows (i.e., has a model of) the environment andmust form and execute a plan to reach a goal loca-tion, such as when escaping from a water maze.It can initially select one (e.g., the best a priori)among multiple possible policies maintained in itsinternal model, by “scoring” them according to theirusefulness to reach the goal. Then, as it navigates,the animal has to continuously refine this initialplan (e.g., by filling initially incomplete segmentsand reducing uncertainty about current and goallocations and improving the accuracy of the transi-tions represented in the internal model) while alsoremaining flexible enough to exploit novel oppor-tunities or reconsider the situation (e.g., considerother policies) in the light of new evidence. In thisperspective, the core process in goal-directed nav-igation is an inference (e.g., about the spatial tra-

jectory to reach a goal) performed on the basis ofthe internal model of policies (or plans), and themain role of sensory stimuli is keeping the inference“in register”—not triggering actions directly as instimulus–response systems. The inference processis coupled to the animal’s action–perception cycle,and combines stimulus-tied and internally gener-ated aspects, the latter crucially including prospec-tive aspects, such as the covert evaluation of possibleroutes.

In Bayesian inference terms, one can think ofthe policies as hypotheses (on future trajectories,or “where to go next”), which continuously com-pete and are updated in the light of new evidence,optimized to minimize a measure of distance fromthe desired goal state and additional constraints,such as energy consumption. In principle, this canbe done by testing all policies in parallel before tak-ing any action, but this can be approximated witha more parsimonious scheme in which one or afew candidate policies are considered serially.73–75

In this scheme, a to-be-tested policy (e.g., the pol-icy having the highest a priori “score”) is sampledfrom the generative model and it generates pre-dictions, which are evaluated and “kept in regis-ter” using external inputs (Fig. 3). To formulate astrong hypothesis, a policy is sampled and evalu-ated within each theta cycle and the theta sequenceis read out of this process. The first elements of atheta sequence may be those that are kept in registerusing external stimuli (like in state estimation); thiscan be done with a predictive-coding style match-ing of top-down predictions (from the policy) andbottom-up sensations, such as cues or path integra-tion signals in the hippocampal–entorhinal system.The successive elements in the theta sequence repre-sent predictions propagated forward in time usingthe model, reflecting successive inferential updatesof the plan (i.e., predictions about the next states);89

they participate in the plan evaluation by engagingprefrontal or ventral striatal mechanisms that con-sider predicted states in relation to goals or rewardexpectancies.7,64 According to this hypothesis, pol-icy evaluation would be serial, and different policiesmay be evaluated within different (e.g., consecutive)theta cycles, as occurs in theta flickering or perhapsalso during VTE behavior.36,90

This hypothesis assigns theta sequences a moreproactive role compared to self-localization andshort-term prediction35,91 by assuming that the span



Figure 3. Schematic illustration of theta sequences within an inferential framework that evaluates (and uses) a candidate policyduring action–perception cycles. One policy is evaluated for each theta cycle. Potentially, the sequence of states predicted by thepolicy spans an entire trajectory from start to goal location; however, only a part of it can be expressed within a single theta cycle(e.g., a portion of the plan, from the current location to a goal or another relevant location). Within each theta cycle, it is possibleto evaluate the quality of the proximal sensory predictions of a policy (early in the theta cycle) and/or the relation between its distalaspects and current goals (later in the theta cycle). The former part roughly corresponds to state estimation and uses active sensingto temporally coordinate descending predictions and incoming stimuli, such as cues from the entorhinal cortex. The latter parttaps goal-related information (or prior preferences) in areas such as the ventral striatum and the prefrontal cortex. This evaluationresults in a “score” for the policy that increases (or decreases) the probability that it will be used for action selection or the nextevaluation (in successive theta cycles).

of prediction is not limited a priori but can “stretch”(to cover, e.g., long distances) when necessary. In thisperspective, what is decoded as a theta sequence maybe the limited readout of a more sophisticated (andfar-looking) inferential process that runs covertlyto continuously evaluate and select policies—animplicit trajectory through the plan’s state space,which can be “read” sequentially by downstreamregions.92 The readout is limited, given that infer-ence occurs within theta cycles, which allow fora small number of elements to be encoded (e.g.,5–10 future locations predicted under the currentpolicy93). Nevertheless, the mechanism can be flexi-ble and cover different portions of the plan depend-ing on task demands. Theta sequences are usu-ally short, possibly reflecting a current focus of theinference on short-term predictions; in some cases,

they can stretch to cover trajectories toward distalgoals11 or (when the goal is uncertain) to sequen-tially evaluate and select between alternative spatialplans, as in VTE.36 Theta sequences may temporar-ily be centered behind or ahead of the animal,17

suggesting that different parts of the plan may beinferred or updated (predicted or postdicted) overtime depending on task demands—possibly relyingon a theta-based segmentation of long streams ofexperience into meaningful subparts.17

Importantly, rhythms and other dynamical phe-nomena are part and parcel of the inference, inparticular, for its temporal aspects and predictivetiming94,95 (i.e., predicting when something willoccur as opposed to what will occur). The tempo-ral alignment of top-down predictions and bottom-up sensory streams within each theta cycle may



be coordinated by active sensing mechanisms.96

Internally generated theta oscillations may becomecoordinated (or even phase locked) with externalrhythmic sensory stimuli gathered by active sensingroutines, such as whisking and sniffing, which alsooccur at theta frequency,97,98 if the onset of theseevents “phase-resets” hippocampal oscillations99—coherent with the idea that action contributes tocouple the intrinsic brain rhythms to the dynam-ics of external events.100 This implies that thetaphase resets may occur to bring specific theta phasesin alignment with expected timing of externalevents101—a phenomenon that may be more visiblein conditioning experiments (where cues are pre-sented with predictable timing) than in typical spa-tial navigation tasks. The (rhythmic) entrainmentof slow cortical oscillations to the stimulation ratewould also enhances perceptual sensitivity,102 withthe theta cycle acting as a filter that increases the gainof signals collected at the right phase and suppressesthe other (as noise). Finally, dynamical phenomena,such as synchronization between rhythms in differ-ent brain areas, may regulate the routing of informa-tion across different brain areas.92 For example, thetemporal alignment of hippocampal activity withVS reward signals during spatial navigation103,104

(but also during replay,105 see below) may indexcommunication between these two areas, as if theyjointly encode a generative model linking place andreward information.7,64,106 Coupling between thehippocampus and the PFC may be beneficial insteadwhen memory demands are high.107 Thus, we sug-gest that theta sequences reflect ongoing predictionsof a generative model synchronized to the action–perception cycle.

SWR sequences. Inferential processes may alsounderlie SWR sequences, the second dynamicalmode of the hippocampus (Fig. 4). When the ani-mal pauses during a task or sleeps, inference can betemporarily decoupled from the action–perceptioncycle and its constraints, such as the necessity toalign predictions to external stimuli collected viaactive sensing routines. The ensuing spontaneousneuronal dynamics—many of which are detectedas SWR sequences—can be interpreted as consecu-tive samples from a (prior) distribution over poli-cies or trajectories encoded in the internal model,which are (relatively) unaffected by external sensoryevents, as in “mind wandering.”108 Modeling stud-

ies have shown that, if left free from constraints,the relative frequency of sampled states or trajec-tories converges on a stationary distribution that isindependent from the initial state of the system.109

This would imply that measuring spontaneous acti-vations should in the long run reveal a statisticallyoptimal model (i.e., a model that is tuned to thestatistics of the environment), as has been observedin other brain areas.110

However, several studies have shown that the con-tent of SWRs can diverge in interesting ways fromuniform distributions, suggesting that SWR con-tent is not completely decoupled from task demandsand/or experience. It is important to separate thesestudies according to the level of task engagement,starting with the distinction between awake andsleep SWRs. Early studies of sleep SWRs focusedon sleep directly following behavior and found aclear contribution of recent experience.111–113 Therate of SWR emission is increased during sleep thatfollows novel experience compared with familiarexperience,114 consistent with the generative modelbeing updated with recently acquired information.Although a strict interpretation of the term “replay”implies that SWR content accurately reflects thestatistics of recent experience, the efficient learn-ing of generative models instead favors preferentialreplay of particularly informative or behaviorallyrelevant experience115,116 circumventing the “real”statistics of the environment, so that behaviorallyrelevant events are over-resampled and thus pref-erentially encoded in the generative model. Thedynamics (a large increase followed by a slowdecrease) of both the rate of SWRs and whatfraction of SWRs contains detectable sequenceswould then reflect meta-control of the resamplingprocess.114,117,118 In turn, the coupling of fast-rippleoscillations may regulate what information is trans-mitted to other brain areas, such as to communi-cate with the PFC for the formation of declarativememories,119 to keep episodic memories in regis-ter with changing neocortical representations,120 orto train a behavioral controller (offline) before achoice, thus saving resources during the decision-making process proper.121 Nevertheless, these viewsof SWR content and function are compatible witha classical “consolidation” account in which recentexperience is used to update a generative model.

Beyond recent experience, the generative modelview suggests that other factors should contribute



Figure 4. Schematic illustration of SWR sequences within an inferential framework while the animal is decoupled from theaction–perception cycle (e.g., while it sleeps). Here, SWR sequences correspond to resamples from the policy distribution, as codedin an internal model. In the absence of biases, the resampling would be from the prior policy distribution and can play rolesin structuring the hippocampal internal model (see main text for explanation) or internal models in other brain areas, such asthe prefrontal cortex. Other brain areas can bias this process. For example, prefrontal goal information can bias the process topreferentially resample policies that reach a known goal site, thus producing SRW sequences that are relevant for planning.

to sleep SWR content as well. These may include theneed to keep the model coherent or self-consistentas learning progresses, ways to prevent overfitting,and preventing destructive interference (well-known ideas in machine learning and connectionistmodels56,122). For example, the risk of overfitting(i.e., the tuning of the generative model to noise oridiosyncratic features of experience that do not gen-eralize) suggests a process to get rid of experiencesthat complicate the model without improving theirpredictive power—or model reduction (indeed,there have been suggestions that, under certain con-ditions, SWRs may elicit long-term depression123).Destructive interference refers to the possibilitythat updating one memory trace may adversely

affect overlapping traces, which may be preventedby active maintenance of relevant traces that havenot been experienced for some time. Thus, SWRcontent during sleep may not be limited to recentexperience content for building generative models,but may also include processes for maintaining,tuning, and optimizing such models in a mannerindependent of specific recent experience.

Awake SWR activity is more constrained bysensory input and task demands. Strong constraintsare expected on tasks that require hippocampalfunction to perform, such as delayed alternationand some place-navigation tasks.12,14 On suchtasks, SWR content can reflect operations like recalland planning that directly support performance



of the task. Although SWR activity is temporarilydecoupled from the action–perception cycle, SWRactivity can still interact with other brain areas,such as the PFC,124,125 from which it can receive,for instance, goal information.126 In turn, thisinformation acts as a constraint to the spontaneousdynamics, which then “drift away” from the (prior)stationary distribution to converge, for example,to a plan to the goal,12 with the same inferentialmechanisms described before. In other words,during performance of hippocampal-dependenttasks, awake SWR sequences may reflect inferentialupdates for planning in the same way we describedfor the first dynamical mode, except that it istemporarily decoupled from the action–perceptioncycle and only uses self-generated information(e.g., about goal locations). If this hypothesis holds,then awake SWR and theta sequences may supportplanning in a coordinated manner at two differenttime scales. Awake SWR sequences may be used toselect an initial policy (to form a rough plan to aknown goal location12,14,127) before taking action,and—if needed—theta sequences may evaluate (orrevise) the policy during action–perception cyclesas the animal approaches the goal site; see Ref. 127for data consistent with such a cooperative view.

Not all awake SWR content necessarily supportsimmediate, upcoming behavior. When generativemodel output is not needed for task performance,such as may occur on tasks or task phases that donot depend on the hippocampus, SWR activitymay uncouple more from task demands. In suchsituations, SWR sequences can include contentexpected from more offline generative modelupdating and maintenance, as discussed for sleepabove. In addition, SWRs may be important toidentify actions that are most informative inupdating the generative model.128 Thus, awakeSWR content may emphasize recent experience fol-lowing rewarding or unexpected outcomes,49,63,129

upcoming constructive trajectories when the hip-pocampus is needed to perform the task,12,14 andtrajectories that do not lead to a rewarded goal butto the maximally informative outcome instead51,192

(see below). In sum, SWR sequences may reflectinferential processes that are temporarily disen-gaged from the demands of perception–actionloops (and hence more spontaneous comparedwith theta sequences) but nevertheless influencedby different sources of self-generated information,

thus potentially playing multiple roles in planning,memory consolidation, and the generation of novel(imaginary) experiences.

Interestingly, a recent study46 of the primate hip-pocampus during visual exploration reports fre-quent SWRs. This suggests that, just as in rodentsexploring novel mazes, there may be task charac-teristics that can cause the primate hippocampusto enter into an SWR-dominated state. The mech-anisms regulating the selection of theta to SWR-dominated modes, and the transitions betweenthem, are incompletely known. While we haveemphasized their differences, our computationalframework suggests various ways theta and SWRsequences may work synergistically in the serviceof online behavior. For example, some tasks mayrequire theta and SWR sequences to play comple-mentary functions, such as short-term (theta) andlong-term prediction and planning (SWR) or, alter-natively, encoding (theta) and simulation (SWR),at fine time scales.14 These tasks may thus requirefrequent transitions between dynamical modes andengage a cyclical relationship between theta andSWR sequences, with rapid consecutive cycles ofengagement (theta) and detachment (SWR) fromthe action–perception cycle while the animal is stillengaged in the task. Understanding the interplaybetween theta and SWR sequences remains an excit-ing avenue for future research.

Generative modelsThe notion of generative models has been widelyused in the context of brain computations andespecially cortical processing;66,67 thus, one mayask what is special about generative models in thehippocampus. An observation that dates back toTolman130 is that the hippocampus may supportthe rapid encoding of a cognitive map (a notionthat is not limited to spatial maps, but encompassesstructured information in nonspatial domains41).From a computational perspective, there are at leastfour (interconnected) aspects of this notion, whichwe discuss in order.

Encoding arbitrary statistical relations. Onewould expect generative models to be organizeddifferently for, say, perceptual processing (e.g.,predicting a ball trajectory) or the encoding of arbi-trary associations (e.g., digits of a phone number),because the environmental statistics underlyingthese phenomena are different. In perceptual



processing, predictions can be often considered tobe extrapolations, at least in the sense that changesfrom one visual scene to the next one are small.From a computational perspective, extrapolationmay benefit from “averaging” and generalizingfrom numerous previous experiences, and modernmachine learning algorithms excel at this usingbig data.131 Rather, the internal model of thehippocampus may excel at a fast form of encodingof more arbitrary, memory-based predictions (e.g.,for phone numbers). These arbitrary sequenceshave a more episodic nature and may require keep-ing different events separated, not merging themto form (semantic) averages, because generalizingfrom “average” representations may be ineffectiveif one has to predict a specific future episode. Thecoding of place information may not eschew thisrule, as manifested by the strong pattern separationof place fields in dentate gyrus and CA3 cellassemblies.132 Orthogonal representations of thiskind may provide a sufficiently large basis to learnseparated and specific representations for a hugevariety of environments and events, even beyondspatial codes. The idea that hippocampal generativemodels are specialized for the fast encoding (andrecall) of arbitrary sequences would explain thewell-recognized importance of this brain structurefor episodic memory and one-trial learning,133

even outside spatial domains. As such, it has beensuggested that the spatial and mnemonic functionsare manifestations of a more general role of the hip-pocampus in representing the relationships betweenobjects and events in both space and time.134

Encoding sequences, not just associations. Giventhe frequent decoding of sequences (rather than justisolated place cells or pairs) in the hippocampus,one may speculate that the hippocampal internalmodel may be preconfigured to learn sequences ortransitions—or, in other words, that sequences (notjust single elements like place cells) may be first-order objects in hippocampal coding. Sequentialcoding is key across spatial navigation and episodicmemory studies. In spatial navigation, sequentialorganization may stem from the fact that the animaltrajectories trace sequences of place cells in space;but the fact that time-compressed sequences are alsoexpressed in internally generated activity suggeststhat this may be an important dimension of neuralcoding, or perhaps an adaptation of neural cod-

ing to the sequential nature of spatial navigation.In the episodic memory domain, it has been oftenargued that episodes may be coded, organized, andrecalled as sequences having a limited number ofelements.135,136 This perspective may help in rec-onciling two main ideas in the literature regard-ing theta sequences, which emphasize their rolein the storage of episodic-like memories137 or inonline prediction.35 These ideas may be connectedif one considers that they focus on two differentphases (encoding or storage and retrieval or recall,respectively), but these two processes may operatesynergistically. Theta sequences may be primarilyconfigured to develop episodic-like memories; but,once they are developed, these memories may beflexibly used in the service of multiple cognitive pro-cesses, such as navigation and decision making.138

Sequences as dynamical templates for experience.Several aspects of hippocampal processing—fastlearning of arbitrary elements, fast remappingacross episodes (which implies that previous asso-ciations between place cells are no longer valid),and the possibility to express sequences for never-experienced routes (preplay)—all suggest that hip-pocampal sequences may be largely preconfigured inthe model as “dynamical templates”139 to organizeexperience, possibly from a limited set of buildingblocks.140 Dynamical templates may be preconfig-ured to represent the temporal structure of multi-item events and be recruited immediately without(at least initially) committing to the “content” ofeach item in the sequence; but they also permit toquickly “bind” their items to the sensory stimuliexperienced one after the other.141 In this perspec-tive, the coding of a sequence of place cells wouldnot be due to synaptic learning between consecutiveplace cells but largely to the binding of each item ofa preconfigured sequence to a place cell (althoughstimuli may fine-tune sequential information139).This may be possible using computational schemesthat form temporal sequences through a state spaceof attractors or attractor-like states142–144 or tran-sient trajectories109,142 possibly corresponding torecurrent computations in CA3/CA1 circuits (seealso early models of sequential processing suchas Ref. 145). In this perspective, sequences wouldnot be fully represented in the hippocampus, butin the coupling of the hippocampus and otherareas, such as the EC. One proposal is that the



Table 1. Predictions

Theta sequences

• In common with other models that stress a probabilistic inference interpretation, we expect that theta sequences contain

signals coding for precision (or its inverse, uncertainty). Preliminary support for this idea comes from the observation that

passive transport disrupts theta sequences185 and that experience with a novel environment rapidly improves theta sequence

quality.13 Virtual reality environments, in which the mapping from actions to sensory changes, as well as the transitions in

the sensory world itself, can be systematically manipulated and offer a promising avenue for identifying such signals. As

pointed out by Penny et al.,79 such a signal would also allow for optimal multisensory integration.

• The proposal that theta sequences encode precision implies that the impact of prediction errors will depend on the degree of

precision at the time of the error. Episodic memory traces are only encoded in the hippocampus when a sufficiently large

prediction error occurs and therefore requires a certain amount of precision in the prediction. Consistent with this idea,

animals require a minimum amount of experience in a novel context before they will associate that context with an

unpredicted shock.186

• Manipulations that disrupt theta sequences affect both “encoding” and “retrieval” functions of hippocampal-dependent

memory. For instance, Robbe et al.187 found that systemic cannabinoids reduced animals’ performance on a delayed T-maze

alternation task; if such disruption would be applied specifically during a sample (encoding) lap and subsequent retrieval laps

(using, for instance, optogenetic manipulation of inputs from the medial septum), impairments would be observed for both.

SWR sequences

• The content of SWRs emitted during sleep will reflect experience according to (1) its recency and (2) its informativeness,

consistent with a consolidation process in which experience stored in the hippocampus can be resampled to update a

cognitive map and cortical knowledge structures. In addition, sleep SWRs contain content that reflects a role in other aspects

of generative model learning, such as improving self-consistency, pruning of model features unlikely to generalize, and

prevention of catastrophic interference between memory traces. As an example of self-consistency, suppose that an animal

has learned that A is connected to B and B is connected to C. Unexpected reward is that received at C after the animal was

placed there, causing it to be associated with a reward value. Replay of the B–C connection could be one way in which B

could also acquire (discounted) value, as can be shown behaviorally.188,189

• The content of SWRs emitted during waking will reflect those actions that lead to maximally informative outcomes.

Consistent with Gupta et al.,51 this means that, when an animal is rewarded for choosing only the left arm on a T-maze, the

right arm is the most informative choice and may be more likely to be replayed. In the maze version of spontaneous object

recognition tasks, in which animals are biased to seek out an object they have not seen recently,128,190 SWR content will be

biased toward the most informative object. The sensitivity of SWRs to information content can be probed by experiments

that manipulate the informativeness or epistemic value74,75 of places or cues independent of their reward (prediction) value.

Overall

• If both theta and SWR sequences stem from the same generative model, manipulations that affect it (e.g., in CA3 or the

medial prefrontal cortex) should affect the content of both kinds of sequences in a coherent manner.

• If theta and SWR sequences are parts of a coordinated planning mechanism, suppressing SWR should enhance theta

sequences (and vice versa); see Ref. 127 for preliminary evidence.

• If IGSs have a truly causal role in behavior and cognition, rather than (for example) being only functional to the readout of

cognitive processing in other brain areas such as the PFC, it should be able to target specific cognitive processes (e.g.,

memory consolidation and planning) by manipulating them.

coding of place cell sequences can be part of afactorized representation in which the hippocam-pus supports temporal sequencing (when compu-tations) while other areas, such as the EC, code forthe content (the what) of sequences.141 In this per-spective, the hippocampus would support integra-tive (what–where–when or what–where–which146)functions by acting as a hub (and sequential pro-cessor) of information distributed in various branareas. Interestingly, retrieving a single element (e.g.,

the fifth element) from a sequence requires “re-playing” the sequence until the element is tapped,as there are no stand-alone indexes or “pointers”to isolated items of the sequence. This may explainwhy experience needs to be replayed sequentially formemory retrieval or consolidation.56,121

Structural priors to form maps. Learningsequences (e.g., for trajectories) does not automati-cally produce good maps. Rather, one needs also to



discover that, for example, the starting point of twosequences is the same as the end point of anothersequence. From a statistical perspective, if thehippocampus has to rapidly form cognitive maps,as in the original notion of Tolman,130 it wouldgreatly benefit from strong structural priors,147

which reflect (formal) aspects about inputs, suchas the fact that sequences of trajectories in the sameenvironment should not have gaps (unless thereis a wall or obstacle) or that paths can usually betraversed in both directions. These priors wouldenable the filling of gaps in experience, such asforming a complete spatial map from a limited setof temporally separated episodes. In this perspec-tive, resampling sequences during SWR activitysupports the generation of novel experiences thatrecombine other ones in constructive ways that areconsistent with the priors, which would explainwhy SWR sequences can reflect novel trajectoriesor shortcuts.51 This would resemble the usualinferential process of predictive coding if one has tominimize differences (or prediction errors) betweenthe hypotheses expressed in the priors (e.g., thatsequences must be connected) and current evidence(e.g., that recently experienced sequences are notconnected). When this is not possible (e.g., whenthe experienced trajectories cannot be merged), themost parsimonious explanation from a statisticalviewpoint would be that these trajectories belongto two (or more) distinct environments, and this inturn may help the animal in building multiple maps.Furthermore, resampling SWR sequences may helprestructuring the internal model (e.g., removingredundant parameters, biasing its content,116,148

and forming state space representations thatorganize experience within a spatial context69,149).

The role of the hippocampus in informationseeking. An important feature of generativemodels is that, to the extent that they are adaptive,acquiring information for the purpose of construct-ing effective models or reducing uncertainty beforea choice has behavioral utility. Many organisms,including humans, nonhuman primates, androdents, are willing to work to obtain informationeven in the absence of association with reward(“curiosity”150,151), thereby following an epistemicdrive.74 Johnson et al.128 pointed out that, inrodents, active information seeking may dependon hippocampal function; specifically, the hip-

pocampus is required for inferring which action ismaximally informative. Extending their proposal tohippocampal sequences, it is striking to note that thecontent of awake SWR sequences may reflect sucha function, including counterintuitive cases such asthat shown in Ref. 51, where animals preferentiallyreplayed the opposite side of a T-maze. Similarly,in the case of an aversive task,192 SWR sequencestravel to the location to be avoided, which maybe significantly more relevant to the animal thanthe alternative, behaviorally neutral path taken. Insome cases, the trajectory toward a (rewarded) goalmay also be the most informative,12,14,63 such thatthe planning and information-seeking accountsoverlap, but the two can be explicitly dissociated;see Table 1 for specific predictions.

Conclusions

Hippocampal IGSs have been extensively studiedduring goal-directed navigation (in rodents), andthey may offer a vantage point to understand howanimal brains temporarily detach from the here andnow to engage in spontaneous or internally directedactions, which have adaptive value (e.g., for mem-ory consolidation or planning). We proposed thatstimulus-tied and internally generated modes ofactivity may engage the same sequential “inferen-tial machine” (on the basis of generative modelsthat the animal maintains of its environment) tosupport overt goal-directed action and cover men-tal activities, including forms of future-orientedand prospective (but also retrospective) cognitionthat share resemblances with the more sophisti-cated forms—imagination and mental time travel—studied in humans. Although we have focused onhippocampal IGSs, sequential neuronal activity hasbeen reported also in other brain areas, such as theparietal cortex,152 the PFC,153,154 the medial EC,155

and the striatum156 of rodents and sensory (olfac-tory) areas of insects,157,158 which may speak forthe generality of the inferential principles describedhere.

Several aspects of our proposals remain to befully tested empirically. One fundamental questionregards the validity of the inferential scheme pro-posed here. The computations we have describedcan be formalized using probabilistic computa-tions in active inference,74,75 model-based rein-forcement learning,159 or other methods, such asMonte Carlo tree search,160 planning as inference,76



and generative (deep) networks.131,161 At theneuronal level, statistical inference has viable bio-logical implementations and can be performed viavariational inference66 or by iteratively samplingindividual elements or sequences from the inter-nal model (e.g., using sampling-based probabilisticinference in recurrent networks, similar to Markovchain Monte Carlo methods in statistics.109,162–164)For example, theta sequences emerge under the vari-ational scheme of active inference if one considersthat at each (theta) cycle successive elements of thepolicy state space are inferred, and the inference pro-ceeds more rapidly for locations that the animal isapproaching,88 which in turn produces a “phase pre-cession” within this inferential scheme. These andother formal schemes may be used to test empir-ical predictions on how rodents solve challengingnavigation tasks or even how humans solve abstracttasks.165–167 Compared with other computationalproposals that also discuss hippocampal functionin relation to generative models,7,75,79,87,88 we havestressed the idea that hippocampal coding and pro-cessing may be organized around sequences. Hip-pocampal generative models may be preconfiguredfor sequential processing and the rapid encoding ofarbitrary sequences of events139 in ways that currentmachine learning techniques fail to model. Further-more, we have stressed the importance of tempo-ral dynamics of probabilistic inference (and predic-tive timing) to explain various functions that wehave associated with theta and SWR sequences. Twoexamples are active sensing and the formation ofdeclarative memories, which require a fine tempo-ral coordination within the action–perception cycle(active sensing) or during the interaction with otherbrain areas, such as the PFC (memory consolida-tion). The validity of the inferential scheme pro-posed here to address sequential processing andtemporal dynamics remains to be fully tested.

Yet another open question regards the specificroles and interplay of theta and SWR sequences dur-ing behavior and cognition. We have proposed thatthey may stem from the same generative processand that their different dynamical signatures (andfunctions) may depend on the different dynamicalmodes under which they operate, the interactionwith other brain areas, and the input of the gener-ative model. The extent to which the disruption oftheta or SWR sequences affects behavior may thusdepend on task demands—for example, whether the

animal is facing a decision on the basis of exter-nal, episodic input versus prior experiences—oreven disengaged from active tasks (e.g., sleeping)—because these tasks would imply different con-straints and inputs for the generative model. Fur-thermore, several lines of evidence reviewed abovesuggest that theta and SWR sequences may interact(e.g., jointly support planning function or encode-then-simulate function) and also be nested withineach other at fine time scales (e.g., SWR sequencesoccurring between consecutive action–perceptioncycles), but these interactions and their behavioralconsequences remain to be completely charted.

Another crucial aspect of this proposal is assessingthe width of the brain networks implied in spatialnavigation and planning. Here, we have focused on arestricted brain network that includes only the hip-pocampus, the VS, and the PFC. Clearly, the brainnetworks implied in goal-directed navigation aremuch wider, and some fundamental aspects of theirfunctioning and interactions remain unclear (e.g.,what are the specific roles of different brain areas andwhich aspects of sequential processing are intrinsicto the hippocampus (e.g., through recurrent con-nections in CA3) and which originate in other areas,such as the PFC 168). Simultaneous recordings frommultiple brain areas and disruption of brain activity(e.g., through optogenetics) may help in sheddinglight on these and other fundamental questions (seeTable 1 for a list of predictions stemming from theproposed framework).

A corollary of our proposal is that, althoughthe cognitive requirements of a specific cognitivetask (e.g., goal-directed navigation) are usuallycompartmentalized into separate processes (e.g.,active sensory processing, attention, state and con-text estimation, working memory of goal location,route planning, and spatial decisions), there isa possibility that a single inferential mechanismsimultaneously implements many or all of theseprocesses, as well as others that we have not dis-cussed (e.g., context estimation79,169). The varietyof functions attributed to SWRs can be reconciled ifone considers that they may stem from a commongenerative model, which can flexibly interact withother brain areas in task-specific ways. Similarly,sequences of neuronal activity found in the rodentparietal cortex may simultaneously play multipleroles in decision making, working memory, and thecoordination of sensory processing.152



Even more intriguingly, our proposal suggeststhat the brain may implement distinct cognitiveprocesses by switching its dynamical mode ofoperation and the coupling with other brain areas.This would imply that the mechanisms supportingdetached cognition may not be fundamentallydifferent from those operating during the action–perception cycle; the key differences may lie inthe ways the same underlying computations (e.g.,inference using a generative model) are triggeredand which information they can access. One canalso speculate that the latter (detached) dynamicsresult from the internalization of processes thatoriginally supported situated action—or thatcognition stems from action.3,8,170,171

If this perspective is correct, then evidenceon hippocampal IGSs can be used to exposefundamental mechanisms permitting humans to“escape from the present” and afford prospective(future-oriented) or retrospective (past-oriented)forms of cognition, despite the large conceptualand methodological distance between the fieldsof rodent spatial navigation and human detachedcognition. Our proposal implies that memory recalland future-oriented processes, such as planningand imagination, are based on the same inferentialmechanisms (e.g., resampling of a policy, covertlyevaluating it) and tap into internal generativemodels in the hippocampus (but also in other brainareas) in the same way. In short, episodic memory(or prediction) is not just recall (or extrapolation)but probabilistic inference using a generativemodel. The involvement of a common core (infer-ential) of mechanisms across multiple domains ofdetached and higher cognition—such as episodicmemory, imagination, counterfactual thinking,mind wandering, prospection, and “time travel”into the past and the future—may explain whythese abilities recruit shared neuronal circuits.2,5,172

In this perspective, mind wandering may be asso-ciated with spontaneous thought173,174 or the sam-pling from (the prior distribution of) an inter-nal generative model that is minimally constrainedfrom external events, analogous to SWR sequences.This process may have adaptive roles in memoryconsolidation or model reduction. Furthermore, asfor SWRs, the same potentially unbiased (mindwandering) process can be engaged in various otherways. It can become oriented toward future (goal)states, as in the case of planning, when the resam-

pling process is biased by goal information, ortoward the past, when a cue engages episodic mem-ory retrieval—as in Proust’s madeleine example. Inother words, the common core (inferential) mecha-nisms remain the same across multiple domains ofdetached cognition, while the orientation (towardthe future or past) and the content (e.g., moreepisodic or semantic) depends on the situation orthe ways the resampling is biased.

It is worth noting that the inferential processesdescribed here in relation to theta and SWRsequences—but also possibly human future-oriented cognition—are not (or not necessarily)conscious. The internal inferential updates ofgenerative models that implement planning ordeliberation can operate at a time scale below thatof cognition, in the same way that the time scale ofvisual saccades is shorter than the time scale of per-ception. For example, in the computational schemefor planning introduced in the section on inference,evaluating each single policy requires multipleinferential updates, which then need to operateat a much faster time scale compared with theaction–perception cycle.73–75,193,194 It thus remainsto be established under which conditions one canhave conscious access to inferential processes thatimplement planning and mind wandering.

The hippocampus has been consistently reportedas an important hub in detached cognitiveprocesses175–180 and the brain default network181—a dynamical mode that epitomizes the “detach-ment” from action–perception cycles in humans.This has raised questions about the necessity fordetached operations to be episodic or autobio-graphical, requiring projecting oneself into the pastor future, rather than more mundanely predictingthem.133 From the computational viewpoint dis-cussed here, the engagement of the hippocampusis not mandatory, but it is beneficial when it is nec-essary to construct events (in the past or the future)that depend on specific circumstances or when aver-aging across all evidence (as semantic models do)would not be accurate.182

Still, this constructive process would require“meshing” content in semantically coherent waysand hence would require a combination of seman-tic and episodic processing. We still lack adetailed understanding of such mixed semantic–episodic operations. Perhaps understanding howSWR sequences mesh episodes coherently (e.g.,



adjacent trajectories but not separated ones51) mayshed light on how human episodic constructivememory works (e.g., meshing memories of oneselfat in the grandmother’s house and of oneself at a pastchildren’s party to imagine how a children’s partyat the grandmother’s house would be). Progressin the field may come from studies that directlyprobe sequential activity resembling SWR or thetasequences during human cognitive processing,183,184

possibly designed to test (and disentangle) alterna-tive computational models.

Acknowledgments

The authors want to thank David Redish, KarlFriston, Paul Verschure, Cyriel Pennartz, ThomasAkam, and the other participants in the work-shop on “Internally generated sequences in thehippocampus” held in Ariccia, Rome, on Septem-ber 27–30, 2016, as well as two anonymousreviewers, for useful comments and suggestions.G.P., M.V.D.M., and C.K. gratefully acknowledgethe support of HFSP (Young Investigator GrantRGY0088/2014). CK additionally acknowledgessupport from the NSF IOS-1550994 and CBET-1351692.

Competing interests

The authors declare no competing interests.

References

1. Corballis, M.C. 2014. Mental time travel: how the mindescapes from the present. Cosmology 18: 139–145.

2. Hassabis, D. & E.A. Maguire. 2009. The construction systemof the brain. Philos. Trans. R. Soc. Lond. B Biol. Sci. 364:1263–1271.

3. Pezzulo, G. & C. Castelfranchi. 2009. Thinking as thecontrol of imagination: a conceptual framework for goal-directed systems. Psychol. Res. 73: 559–577.

4. Pezzulo, G. & P. Cisek. 2016. Navigating the affordancelandscape: feedback control as a process model of behaviorand cognition. Trends Cogn. Sci. 20: 414–424.

5. Schacter, D.L., D.R. Addis, D. Hassabis, et al. 2012. Thefuture of memory: remembering, imagining, and the brain.Neuron 76: 677–694.

6. Andrews-Hanna, J.R., J. Smallwood & R.N. Spreng. 2014.The default network and self-generated thought: compo-nent processes, dynamic control, and clinical relevance.Ann. N.Y. Acad. Sci. 1316: 29–52.

7. Pezzulo, G., M.A.A. van der Meer, C.S. Lansink & C.M.A.Pennartz. 2014. Internally generated sequences in learningand executing goal-directed behavior. Trends Cogn. Sci. 18:647–657.

8. Buzsaki, G., A. Peyrache & J. Kubie. 2014. Emergence ofcognition from action. Cold Spring Harb. Symp. Quant.Biol. 79: 41–50.

9. Dragoi, G. & S. Tonegawa. 2011. Preplay of future place cellsequences by hippocampal cellular assemblies. Nature 469:397–401.

10. Wikenheiser, A.M. & A.D. Redish. 2013. The balance offorward and backward hippocampal sequences shifts acrossbehavioral states. Hippocampus 23: 22–29.

11. Wikenheiser, A.M. & A.D. Redish. 2015. Hippocampaltheta sequences reflect current goals. Nat. Neurosci. 18:289–294.

12. Pfeiffer, B.E. & D.J. Foster. 2013. Hippocampal place-cellsequences depict future paths to remembered goals. Nature497: 74–79.

13. Feng, T., D. Silva & D.J. Foster. 2015. Dissociation betweenthe experience-dependent development of hippocampaltheta sequences and single-trial phase precession. J. Neu-rosci. 35: 4890–4902.

14. Jadhav, S.P., C. Kemere, P.W. German & L.M. Frank. 2012.Awake hippocampal sharp-wave ripples support spatialmemory. Science 336: 1454–1458.

15. Suh, J., D.J. Foster, H. Davoudi, et al. 2013. Impaired hip-pocampal ripple-associated replay in a mouse model ofschizophrenia. Neuron 80: 484–493.

16. Olafsdottir, H.F., F. Carpenter & C. Barry. 2016. Coordi-nated grid and place cell replay during rest. Nat. Neurosci.19: 792–794.

17. Gupta, A.S., M.A. van der Meer, D.S. Touretzky & A.D.Redish. 2012. Segmentation of spatial experience by hip-pocampal theta sequences. Nat. Neurosci. 15: 1032–1039.

18. Wu, X. & D.J. Foster. 2014. Hippocampal replay capturesthe unique topological structure of a novel environment. J.Neurosci. 34: 6459–6469.

19. Pfeiffer, B.E. & D.J. Foster. 2015. Autoassociative dynamicsin the generation of sequences of hippocampal place cells.Science 349: 180–183.

20. Wilson, M. & B.L. McNaughton. 1993. Dynamics of thehippocampal ensemble code for space. Science 261: 1055–1058.

21. Zhang, K., I. Ginzburg, B.L. McNaughton & T.J. Sejnowski.1998. Interpreting neuronal population activity by recon-struction: unified framework with application to hip-pocampal place cells. J. Neurophysiol. 79: 1017–1044.

22. Wood, E.R., P.A. Dudchenko & H. Eichenbaum. 1999. Theglobal record of memory in hippocampal neuronal activity.Nature 397: 613–616.

23. Allen, K., J.N.P. Rawlins, D.M. Bannerman & J. Csicsvari.2012. Hippocampal place cells can encode multiple trial-dependent features through rate remapping. J. Neurosci.32: 14752–14766.

24. Pastalkova, E., V. Itskov, A. Amarasingham & G. Buzsaki.2008. Internally generated cell assembly sequences in therat hippocampus. Science 321: 1322–1327.

25. MacDonald, C.J., K.Q. Lepage, U.T. Eden & H. Eichen-baum. 2011. Hippocampal “time cells” bridge the gap inmemory for discontiguous events. Neuron 71: 737–749.

26. Morris, R.G.M. 2006. Elements of a neurobiological theoryof hippocampal function: the role of synaptic plasticity,



synaptic tagging and schemas. Eur. J. Neurosci. 23: 2829–2846.

27. Dragoi, G. & G. Buzsaki. 2006. Temporal encoding of placesequences by hippocampal cell assemblies. Neuron 50: 145–157.

28. Johnson, A., J.C. Jackson & A.D. Redish. 2009. Measur-ing distributed properties of neural representations beyondthe decoding of local variables: implications for cogni-tion. In Information Processing by Neuronal Populations. C.Holscher & M. Munk, Eds.: 95–119. Cambridge UniversityPress.

29. Maurer, A.P., S.N. Burke, P. Lipa, et al. 2012. Greater run-ning speeds result in altered hippocampal phase sequencedynamics. Hippocampus 22: 737–747.

30. Foster, D.J. & M.A. Wilson. 2007. Hippocampal thetasequences. Hippocampus 17: 1093–1099.

31. Zheng, C., K.W. Bieri, Y.-T. Hsiao & L.L. Colgin. 2016.Spatial sequence coding differs during slow and fast gammarhythms in the hippocampus. Neuron 89: 398–408.

32. Skaggs, W.E. & B.L. McNaughton. 1996. Theta phase pre-cession in hippocampal. Hippocampus 6: 149–172.

33. Burgess, N. & J. O’Keefe. 2011. Models of place and gridcell firing and theta rhythmicity. Curr. Opin. Neurobiol. 21:734–744.

34. Chadwick, A., M. van Rossum & M. Nolan. 2015. Mod-elling phase precession in the hippocampus. BMC Neurosci.16(Suppl. 1): O18.

35. Lisman, J. & A.D. Redish. 2009. Prediction, sequences andthe hippocampus. Philos. Trans. R. Soc. B Biol. Sci. 364:1193–1201.

36. Johnson, A. & A.D. Redish. 2007. Neural ensembles in CA3transiently encode paths forward of the animal at a decisionpoint. J. Neurosci. 27: 12176–12189.

37. Robbe, D. & G. Buzsaki. 2009. Alteration of theta timescaledynamics of hippocampal place cells by a cannabinoidis associated with memory impairment. J. Neurosci. 29:12597–12605.

38. Wang, Y., S. Romani, B. Lustig, et al. 2015. Theta sequencesare essential for internally generated hippocampal firingfields. Nat. Neurosci. 18: 282–288.

39. Hasselmo, M.E., C. Bodelon & B.P. Wyble. 2002. A pro-posed function for hippocampal theta rhythm: separatephases of encoding and retrieval enhance reversal of priorlearning. Neural Comput. 14: 793–817.

40. Vertes, R.P. 2005. Hippocampal theta rhythm: a tag forshort-term memory. Hippocampus 15: 923–935.

41. Buzsaki, G. & E.I. Moser. 2013. Memory, navigation andtheta rhythm in the hippocampal–entorhinal system. Nat.Neurosci. 16: 130–138.

42. Buzsaki, G. 2015. Hippocampal sharp wave-ripple: a cog-nitive biomarker for episodic memory and planning. Hip-pocampus 25: 1073–1188.

43. O’Neill, J., T. Senior & J. Csicsvari. 2006. Place-selectivefiring of CA1 pyramidal cells during sharp wave/ripple net-work patterns in exploratory behavior. Neuron 49: 143–155.

44. Kemere, C., M.F. Carr, M.P. Karlsson & L.M. Frank. 2013.Rapid and continuous modulation of hippocampal net-work state during exploration of new places. PLoS One 8:e73114.

45. Leonard, T.K. & K.L. Hoffman. 2017. Sharp-wave ripplesin primates are enhanced near remembered visual objects.Curr. Biol. 27: 257–262.

46. Leonard, T.K. et al. 2015. Sharp wave ripples during visualexploration in the primate hippocampus. J. Neurosci. 35:14771–14782.

47. Davidson, T.J., F. Kloosterman & M.A. Wilson. 2009. Hip-pocampal replay of extended experience. Neuron 63: 497–507.

48. Diba, K. & G. Buzsaki. 2007. Forward and reverse hip-pocampal place-cell sequences during ripples. Nat. Neu-rosci. 10: 1241–1242.

49. Ambrose, R.E., B.E. Pfeiffer & D.J. Foster. 2016. Reversereplay of hippocampal place cells is uniquely modulated bychanging reward. Neuron 91: 1124–1136.

50. Foster, D.J. & M.A. Wilson. 2006. Reverse replay ofbehavioural sequences in hippocampal place cells duringthe awake state. Nature 440: 680–683.

51. Gupta, A.S., M.A.A. van der Meer, D.S. Touretzky & A.D.Redish. 2010. Hippocampal replay is not a simple functionof experience. Neuron 65: 695–705.

52. Karlsson, M.P. & L.M. Frank. 2009. Awake replay of remoteexperiences in the hippocampus. Nat. Neurosci. 12: 913–918.

53. Olafsdottir, H.F., C. Barry, A.B. Saleem, et al. 2015. Hip-pocampal place cells construct reward related sequencesthrough unexplored space. eLife 4: e06063.

54. Grosmark, A.D. & G. Buzsaki. 2016. Diversity in neural fir-ing dynamics supports both rigid and learned hippocampalsequences. Science 351: 1440–1443.

55. Silva, D., T. Feng & D.J. Foster. 2015. Trajectory eventsacross hippocampal place cells require previous experience.Nat. Neurosci. 18: 1772–1779.

56. McClelland, J.L., B.L. McNaughton & R.C. O’Reilly. 1995.Why there are complementary learning systems in the hip-pocampus and neocortex: insights from the successes andfailures of connectionist models of learning and memory.Psychol. Rev. 102: 419–457.

57. Girardeau, G., K. Benchenane, S.I. Wiener, et al. 2009.Selective suppression of hippocampal ripples impairs spa-tial memory. Nat. Neurosci. 12: 1222–1223.

58. Ego-Stengel, V. & M.A. Wilson. 2010. Disruption of ripple-associated hippocampal activity during rest impairs spatiallearning in the rat. Hippocampus 20: 1–10.

59. Sadowski, J.H.L.P., M.W. Jones & J.R. Mellor. 2016. Sharp-wave ripples orchestrate the induction of synaptic plasticityduring reactivation of place cell firing patterns in the hip-pocampus. Cell Rep. 14: 1916–1929.

60. Pennartz, C.M., E. Lee, J. Verheul, et al. 2004. The ven-tral striatum in off-line processing: ensemble reactivationduring sleep and modulation by hippocampal ripples. J.Neurosci. 24: 6446–6456.

61. Ji, D. & M.A. Wilson. 2007. Coordinated memory replayin the visual cortex and hippocampus during sleep. Nat.Neurosci. 10: 100–107.

62. Shin, J.D. & S.P. Jadhav. 2016. Multiple modes ofhippocampal–prefrontal interactions in memory-guidedbehavior. Curr. Opin. Neurobiol. 40: 161–169.



63. Singer, A.C., M.F. Carr, M.P. Karlsson & L.M. Frank. 2013.Hippocampal SWR activity predicts correct decisions dur-ing the initial learning of an alternation task. Neuron 77:1163–1173.

64. van der Meer, M., Z. Kurth-Nelson & A.D. Redish. 2012.Information processing in decision-making systems. Neu-roscientist 18: 342–359.

65. Carr, M.F., S.P. Jadhav & L.M. Frank. 2011. Hippocampalreplay in the awake state: a potential substrate for mem-ory consolidation and retrieval. Nat. Neurosci. 14: 147–153.

66. Friston, K. 2010. The free-energy principle: a unified braintheory? Nat. Rev. Neurosci. 11: 127–138.

67. Doya, K., S. Ishii, A. Pouget & R.P.N. Rao, Eds. 2007.Bayesian Brain: Probabilistic Approaches to Neural Coding.1st ed. The MIT Press.

68. Buckner, R.L. 2010. The role of the hippocampus in pre-diction and imagination. Annu. Rev. Psychol 61: 27–48,C1–C8.

69. Friston, K. 2008. Hierarchical models in the brain. PLoSComput. Biol. 4: e1000211.

70. Lee, T.S. & D. Mumford. 2003. Hierarchical bayesian infer-ence in the visual cortex. J. Opt. Soc. Am. A Opt. Image Sci.Vis. 20: 1434–1448.

71. Rao, R.P. & D.H. Ballard. 1999. Predictive coding in thevisual cortex: a functional interpretation of some extra-classical receptive-field effects. Nat. Neurosci. 2: 79–87.

72. Friston, K. 2003. Learning and inference in the brain. NeuralNetw. 16: 1325–1352.

73. Friston, K., S. Samothrakis & R. Montague. 2012. Activeinference and agency: optimal control without cost func-tions. Biol. Cybern. 106: 523–541.

74. Friston, K., F. Rigoli, D. Ognibene, et al. 2015. Active infer-ence and epistemic value. Cogn. Neurosci. 6: 187–214.

75. Pezzulo, G., E. Cartoni, F. Rigoli, et al. 2016. Active infer-ence, epistemic value, and vicarious trial and error. Learn.Mem. 23: 322–338.

76. Botvinick, M. & M. Toussaint. 2012. Planning as inference.Trends Cogn. Sci. 16: 485–488.

77. Solway, A. & M.M. Botvinick. 2012. Goal-directed decisionmaking as probabilistic inference: a computational frame-work and potential neural correlates. Psychol. Rev. 119:120–154.

78. Penny, W. 2014. Simultaneous localisation and planning. In4th International Workshop on Cognitive Information Pro-cessing (CIP) Copenhagen, Denmark. 1–6.

79. Penny, W.D., P. Zeidman & N. Burgess. 2013. Forward andbackward inference in spatial cognition. PLoS Comput. Biol.9: e1003383.

80. Pezzulo, G., F. Rigoli & F. Chersi. 2013. The mixedinstrumental controller: using value of information to com-bine habitual choice and mental simulation. Front. Cogn.4: 92.

81. Bousquet, O., K. Balakrishnan & V. Honavar. 1998. Is thehippocampus a Kalman filter? Pac. Symp. Biocomput. 657–668.

82. Fuhs, M.C. & D.S. Touretzky. 2007. Context learning in therodent hippocampus. Neural Comput. 19: 3173–3215.

83. Madl, T., S. Franklin, K. Chen, et al. 2014. Bayesian integra-tion of information in hippocampal place cells. PLoS One9: e89762.

84. Sanders, H., C. Renno-Costa, M. Idiart & J. Lisman. 2015.Grid cells and place cells: an integrated view of their nav-igational and memory function. Trends Neurosci. 38: 763–775.

85. Sutton, R.S. & A.G. Barto. 1998. Reinforcement Learning:An Introduction. Cambridge, MA: MIT Press.

86. Packard, M.G. & J.L. McGaugh. 1996. Inactivation of hip-pocampus or caudate nucleus with lidocaine differentiallyaffects expression of place and response learning. Neurobiol.Learn. Mem. 65: 65–72.

87. Rueckert, E., D. Kappel, D. Tanneberg, et al. 2016. Recurrentspiking networks solve planning tasks. Sci. Rep. 6: 21142.

88. Friston, K., T. FitzGerald, F. Rigoli, et al. 2016. Active infer-ence: a process theory. Neural Comput. 29: 1–49.

89. Friston, K., T. FitzGerald, F. Rigoli, et al. 2016. Activeinference and learning. Neurosci. Biobehav. Rev. 68: 862–879.

90. Redish, A.D. 2016. Vicarious trial and error. Nat. Rev.Neurosci. 17: 147–159.

91. Jensen, O. & J.E. Lisman. 1996. Hippocampal CA3 regionpredicts memory sequences: accounting for the phase pre-cession of place cells. Learn. Mem. 3: 279–287.

92. Buzsaki, G. 2010. Neural syntax: cell assemblies, synapsem-bles, and readers. Neuron 68: 362–385.

93. Lisman, J.E. & O. Jensen. 2013. The theta–gamma neuralcode. Neuron 77: 1002–1016.

94. Engel, A.K., P. Fries & W. Singer. 2001. Dynamic predic-tions: oscillations and synchrony in top-down processing.Nat. Rev. Neurosci. 2: 704–716.

95. Arnal, L.H. & A.-L. Giraud. 2012. Cortical oscillations andsensory predictions. Trends Cogn. Sci. 16: 390–398.

96. Zacksenhouse, M. & E. Ahissar. 2006. Temporal decod-ing by phase-locked loops: unique features of circuit-level implementations and their significance for vibrissalinformation processing. Neural Comput. 18: 1611–1636.

97. Ranade, S., B. Hangya & A. Kepecs. 2013. Multiple modes ofphase locking between sniffing and whisking during activeexploration. J. Neurosci. 33: 8250–8256.

98. Grion, N., A. Akrami, Y. Zuo, et al. 2016. Coherencebetween rat sensorimotor system and hippocampus isenhanced during tactile discrimination. PLoS Biol 14:e1002384.

99. Colgin, L.L. 2016. Rhythms of the hippocampal network.Nat. Rev. Neurosci. 17: 239–249.

100. Buzsaki, G. 2006. Rhythms of the Brain. Oxford UniversityPress.

101. Lakatos, P., G. Karmos, A.D. Mehta, et al. 2008. Entrain-ment of neuronal oscillations as a mechanism of attentionalselection. Science 320: 110–113.

102. Morillon, B., T.A. Hackett, Y. Kajikawa & C.E. Schroeder.2015. Predictive motor control of sensory dynamics in audi-tory active sensing. Curr. Opin. Neurobiol. 31: 230–238.

103. van der Meer, M.A.A. & A.D. Redish. 2011. Theta phaseprecession in rat ventral striatum links place and rewardinformation. J. Neurosci. 31: 2843–2854.



104. van der Meer, M.A.A. & A.D. Redish. 2009. Covertexpectation-of-reward in rat ventral striatum at decisionpoints. Front. Integr. Neurosci 3: 1.

105. Lansink, C.S., P.M. Goltstein, J.V. Lankelma, et al. 2009.Hippocampus–leads ventral striatum in replay of place–reward information. PLoS Biol. 7: e1000173.

106. Lansink, C.S., G.T. Meijer, J.V. Lankelma, et al. 2016.Reward expectancy strengthens CA1 theta and beta bandsynchronization and hippocampal–ventral striatal cou-pling. J. Neurosci. 36: 10598–10610.

107. Jones, M.W. & M.A. Wilson. 2005. Theta rhythms coor-dinate hippocampal–prefrontal interactions in a spatialmemory task. PLoS Biol 3: e402.

108. Smallwood, J. & J.W. Schooler. 2006. The restless mind.Psychol. Bull. 132: 946–958.

109. Buesing, L., J. Bill, B. Nessler & W. Maass. 2011.Neural dynamics as sampling: a model for stochasticcomputation in recurrent networks of spiking neurons.PLoS Comput. Biol. 7: e1002211.

110. Berkes, P., G. Orban, M. Lengyel & J. Fiser. 2011. Spon-taneous cortical activity reveals hallmarks of an optimalinternal model of the environment. Science 331: 83–87.

111. Wilson, M.A. & B.L. McNaughton. 1994. Reactivation ofhippocampal ensemble memories during sleep. Science265: 676–679.

112. Louie, K. & M.A. Wilson. 2001. Temporally structuredreplay of awake hippocampal ensemble activity duringrapid eye movement sleep. Neuron 29: 145–156.

113. Lee, A.K. & M.A. Wilson. 2002. Memory of sequentialexperience in the hippocampus during slow wave sleep.Neuron 36: 1183–1194.

114. Eschenko, O., W. Ramadan, M. Molle, et al. 2008. Sustainedincrease in hippocampal sharp-wave ripple activity duringslow-wave sleep after learning. Learn. Mem. 15: 222–228.

115. Atherton, L.A., D. Dupret & J.R. Mellor. 2015. Memorytrace replay: the shaping of memory consolidation byneuromodulation. Trends Neurosci. 38: 560–570.

116. Kumaran, D., D. Hassabis & J.L. McClelland. 2016. Whatlearning systems do intelligent agents need? complemen-tary learning systems theory updated. Trends Cogn. Sci. 20:512–534.

117. Buhry, L., A.H. Azizi & S. Cheng. 2011. Reactivation, replay,and preplay: how it might all fit together. Neural Plast. 2011:203462.

118. Kudrimoti, H.S., C.A. Barnes & B.L. McNaughton. 1999.Reactivation of hippocampal cell assemblies: effects ofbehavioral state, experience, and EEG dynamics. J. Neu-rosci. 19: 4090–4101.

119. Simons, J.S. & H.J. Spiers. 2003. Prefrontal and medialtemporal lobe interactions in long-term memory. Nat. Rev.Neurosci. 4: 637–648.

120. Kali, S. & P. Dayan. 2004. Off-line replay maintains declar-ative memories in a model of hippocampal–neocorticalinteractions. Nat. Neurosci. 7: 286–294.

121. Sutton, R.S. 1990. Integrated architectures for learning,planning, and reacting based on approximating dynamicprogramming. In Proceedings of the 7th International Con-ference on Machine Learning. San Mateo, CA: 216–224.

122. Bishop, C.M. 2006. Pattern Recognition and Machine Learn-ing. Springer.

123. Mehta, M.R. 2007. Cortico-hippocampal interaction dur-ing up-down states and memory consolidation. Nat. Neu-rosci. 10: 13–15.

124. Peyrache, A., M. Khamassi, K. Benchenane, et al. 2009.Replay of rule-learning related neural patterns in the pre-frontal cortex during sleep. Nat. Neurosci. 12: 919–926.

125. Wierzynski, C.M., E.V. Lubenov, M. Gu & A.G. Siapas.2009. State-dependent spike–timing relationships betweenhippocampal and prefrontal circuits during sleep. Neuron61: 587–596.

126. Hok, V., E. Save, P.P. Lenck-Santini & B. Poucet. 2005.Coding for spatial goals in the prelimbic/infralimbic areaof the rat frontal cortex. Proc. Natl. Acad. Sci. U.S.A. 102:4602–4607.

127. Papale, A.E., M.C. Zielinski, L.M. Frank, et al. 2016. Inter-play between hippocampal sharp-wave-ripple events andvicarious trial and error behaviors in decision making. Neu-ron 92: 975–982.

128. Johnson, A., Z. Varberg, J. Benhardus, et al. 2012. The hip-pocampus and exploration: dynamically evolving behaviorand neural representations. Front. Hum. Neurosci. 6: 216.

129. Cheng, S. & L.M. Frank. 2008. New experiences enhancecoordinated neural activity in the hippocampus. Neuron57: 303–313.

130. Tolman, E.C. 1948. Cognitive maps in rats and men. Psy-chol. Rev. 55: 189–208.

131. Hinton, G.E. 2007. Learning multiple layers of representa-tion. Trends Cogn. Sci 11: 428–434.

132. Leutgeb, J.K., S. Leutgeb, M.-B. Moser & E.I. Moser. 2007.Pattern separation in the dentate gyrus and CA3 of thehippocampus. Science 315: 961–966.

133. Tulving, E. 2002. Episodic memory: from mind to brain.Annu. Rev. Psychol. 53: 1–25.

134. Eichenbaum, H. & N.J. Cohen. 2014. Can we reconcilethe declarative memory and spatial navigation views onhippocampal function? Neuron 83: 764–770.

135. Atkinson, R.C. & R.M. Shiffrin. 1968. “Human memory: aproposed system and its control processes.” In The Psychol-ogy of Learning and Motivation: Advances in Research andTheory. 2. K.W. Spence & J.T. Spence, Eds.: 742–775. NewYork: AcademicPress.

136. Miller, G.A. 1956. The magical number seven, plus orminus two: some limits on our capacity for processinginformation. Psychol. Rev. 63: 81–97.

137. Lisman, J.E. & M.A. Idiart. 1995. Storage of 7 plus/minus 2short-term memories in oscillatory subcycles. Science 267:1512.

138. Moscovitch, M., R. Cabeza, G. Winocur & L. Nadel. 2016.Episodic memory and beyond: the hippocampus and neo-cortex in transformation. Annu. Rev. Psychol. 67: 105–134.

139. Dragoi, G. 2013. Internal operations in the hippocam-pus: single cell and ensemble temporal coding. Front. Syst.Neurosci. 7: 46.

140. Malvache, A., S. Reichinnek, V. Villette, et al. 2016.Awake hippocampal reactivations project onto orthogonalneuronal assemblies. Science 353: 1280–1283.



141. Friston, K. & G. Buzsaki. 2016. The functional anatomy oftime: what and when in the brain. Trends Cogn. Sci. 20:500–511.

142. Rabinovich, M., A. Volkovskii, P. Lecanda, et al. 2001.Dynamical encoding by networks of competing neurongroups: winnerless competition. Phys. Rev. Lett. 87: 068102.

143. Buonomano, D.V. & W. Maass. 2009. State-dependentcomputations: spatiotemporal processing in cortical net-works. Nat. Rev. Neurosci. 10: 113–125.

144. Rolls, E.T. & G. Deco. 2015. Networks for memory, percep-tion, and decision-making, and beyond to how the syntaxfor language might be implemented in the brain. Brain Res.1621: 316–334.

145. Grossberg, S. 1978. “A theory of human memory: self-organization and performance of sensory-motor codes,maps, and plans.” In Progress in Theoretical Biology. 5. R.Rosen & F. Snell, Eds.: 233–374. New York: Academic Press.

146. Eacott, M.J. & G. Norman. 2004. Integrated memory forobject, place, and context in rats: a possible model ofepisodic-like memory? J. Neurosci. 24: 1948–1953.

147. Kemp, C. & J.B. Tenenbaum. 2008. The discovery ofstructural form. Proc. Natl. Acad. Sci. U.S.A. 105: 10687–10692.

148. Stickgold, R. & M.P. Walker. 2013. Sleep-dependent mem-ory triage: evolving generalization through selective pro-cessing. Nat. Neurosci. 16: 139–145.

149. Maguire, E.A. & S.L. Mullally. 2013. The hippocampus: amanifesto for change. J. Exp. Psychol. Gen. 142: 1180–1189.

150. Bromberg-Martin, E.S., M. Matsumoto & O. Hikosaka.2010. Dopamine in motivational control: rewarding, aver-sive, and alerting. Neuron 68: 815–834.

151. Ennaceur, A. & J. Delacour. 1988. A new one-trial test forneurobiological studies of memory in rats. 1: behavioraldata. Behav. Brain Res. 31: 47–59.

152. Harvey, C.D., P. Coen & D.W. Tank. 2012. Choice-specificsequences in parietal cortex during a virtual-navigationdecision task. Nature 484: 62–68.

153. Fujisawa, S., A. Amarasingham, M.T. Harrison & G.Buzsaki. 2008. Behavior-dependent short-term assemblydynamics in the medial prefrontal cortex. Nat. Neurosci.11: 823–833.

154. Dejean, C. et al. 2016. Prefrontal neuronal assemblies tem-porally control fear behaviour. Nature 535: 420–424.

155. O’Neill, J., C.N. Boccara, F. Stella, et al. 2017. Superficiallayers of the medial entorhinal cortex replay independentlyof the hippocampus. Science 355: 184–188.

156. Akhlaghpour, H., J. Wiskerke, J.Y. Choi, et al. 2016. Disso-ciated sequential activity and stimulus encoding in the dor-somedial striatum during spatial working memory. eLife 5:e19507.

157. Laurent, G. 1996. Dynamical representation of odors byoscillating and evolving neural assemblies. Trends Neurosci.19: 489–496.

158. Mazor, O. & G. Laurent. 2005. Transient dynamics versusfixed points in odor representations by locust antennal lobeprojection neurons. Neuron 48: 661–673.

159. Daw, N.D. & P. Dayan. 2014. The algorithmic anatomy ofmodel-based evaluation. Philos. Trans. R. Soc. B Biol. Sci.369: 20130478.

160. Gelly, S. & D. Silver. 2011. Monte-Carlo tree search andrapid action value estimation in computer Go. Artif. Intell.175: 1856–1875.

161. Dayan, P., G.E. Hinton, R.M. Neal & R.S. Zemel. 1995. Thehelmholtz machine. Neural Comput. 7: 889–904.

162. Fiser, J., P. Berkes, G. Orban & M. Lengyel. 2010. Statisticallyoptimal perception and learning: from behavior to neuralrepresentations. Trends Cogn. Sci. 14: 119–130.

163. Huang, Y. & R.P.N. Rao. 2014. Neurons as Monte Carlosamplers: Bayesian inference and learning in spiking net-works. In Proceedings of the 27th International Conferenceon Neural Information Processing Systems, Cambridge, MA:1943–1951.

164. Pecevski, D. & W. Maass. 2016. Learning probabilistic infer-ence through spike-timing-dependent plasticity. eNeuro 3:pii: ENEURO.0048-15.2016.

165. Donnarumma, F., D. Maisto & G. Pezzulo. 2016. Problemsolving as probabilistic inference with subgoaling: explain-ing human successes and pitfalls in the tower of hanoi. PLoSComput. Biol. 12: e1004864.

166. Balaguer, J., H. Spiers, D. Hassabis & C. Summerfield. 2016.Neural mechanisms of hierarchical planning in a virtualsubway network. Neuron 90: 893–903.

167. Stoianov, I., A. Genovesio & G. Pezzulo. 2015. Prefrontalgoal codes emerge as latent states in probabilistic valuelearning. J. Cogn. Neurosci. 28: 140–157.

168. Ito, H.T., S.-J. Zhang, M.P. Witter, et al. 2015. A prefrontal–thalamo-hippocampal circuit for goal-directed spatial nav-igation. Nature 522: 50–55.

169. Jezek, K., E.J. Henriksen, A. Treves, et al. 2011. Theta-pacedflickering between place-cell maps in the hippocampus.Nature 478: 246–249.

170. Pezzulo, G. & C. Castelfranchi 2007. The symbol detach-ment problem. Cogn. Process. 8: 115–131.

171. Pezzulo, G. 2017. Tracing the roots of cognition in predic-tive processing. In Philosophy and Predictive Processing, 20.Frankfurt am Main: MIND Group.

172. Schacter, D.L. & D.R. Addis. 2007. The cognitive neuro-science of constructive memory: remembering the past andimagining the future. Philos. Trans. R. Soc. Lond. B Biol. Sci.362: 773–786.

173. Christoff, K., Z.C. Irving, K.C.R. Fox, et al. 2016. Mind-wandering as spontaneous thought: a dynamic framework.Nat. Rev. Neurosci. 17: 718–731.

174. Benedek, M., E. Jauk, R.E. Beaty, et al. 2016. Brain mech-anisms associated with internally directed attention andself-generated thought. Sci. Rep. 6: 22959.

175. Martin, V.C., D.L. Schacter, M.C. Corballis & D.R. Addis.2011. A role for the hippocampus in encoding simulationsof future events. Proc. Natl. Acad. Sci. U.S.A. 108: 13858–13863.

176. Gaesser, B., R.N. Spreng, V.C. McLelland, et al. 2013. Imag-ining the future: evidence for a hippocampal contributionto constructive processing. Hippocampus 23: 1150–1161.

177. Race, E., M.M. Keane & M. Verfaellie. 2013. Losing sightof the future: impaired semantic prospection followingmedial temporal lobe lesions. Hippocampus 23: 268–277.

178. Addis, D.R., T. Cheng, R.P. Roberts & D.L. Schacter. 2011.Hippocampal contributions to the episodic simulation of



specific and general future events. Hippocampus 21: 1045–1052.

179. Brown, T.I. et al. 2016. Prospective representation of nav-igational goals in the human hippocampus. Science 352:1323–1326.

180. Hassabis, D., D. Kumaran, S.D. Vann & E.A. Maguire.2007. Patients with hippocampal amnesia cannot imaginenew experiences. Proc. Natl. Acad. Sci. U.S.A. 104: 1726–1731.

181. Buckner, R.L., J.R. Andrews-Hanna & D.L. Schacter. 2008.The brain’s default network: anatomy, function, and rele-vance to disease. Ann. N.Y. Acad. Sci. 1124: 1–38.

182. Pezzulo, G. 2016. “The mechanisms and benefits of afuture-oriented brain.” In Seeing the Future. K. Michaelian,S.B. Klein & K.K. Szpunar, Eds.: 267–284. Oxford Univer-sity Press.

183. Kurth-Nelson, Z., M. Economides, R.J. Dolan & P. Dayan.2016. Fast sequences of non-spatial state representations inhumans. Neuron 91: 194–204.

184. Kaplan, R., J. King, R. Koster, et al. 2017. The neural rep-resentation of prospective choice during spatial planningand decisions. PLoS Biol. 15: e1002588.

185. Cei, A., G. Girardeau, C. Drieu, et al. 2014. Reversed thetasequences of hippocampal cell assemblies during backwardtravel. Nat. Neurosci. 17: 719–724.

186. Fanselow, M.S. 1990. Factors governing one-trial contex-tual conditioning. Anim. Learn. Behav. 18: 264–270.

187. Robbe, D., S.M. Montgomery, A. Thome, et al. 2006.Cannabinoids reveal importance of spike timing coordina-tion in hippocampal function. Nat. Neurosci. 9: 1526–1533.

188. White, N.M. 1989. Reward or reinforcement: what’s thedifference?. Neurosci. Biobehav. Rev. 13: 181–186.

189. Momennejad, I., E.M. Russek, J.H. Cheong, et al. 2016. Thesuccessor representation in human reinforcement learning.bioRxiv. 083824.

190. Eacott, M.J. & A. Easton. 2007. On familiarity and recall ofevents by rats. Hippocampus 17: 890–897.

191. van der Meer, M., A.A. Carey & Y. Tanaka. 2017. Opti-mizing for generalization in the decoding of internallygenerated activity in the hippocampus. Hippocampus.https://doi.org/10.1002/10.1002/hipo.22714.

192. Wu, C.T., D. Haggerty, C. Kemere & D. Ji. 2017. Hippocam-pal awake replay in fear memory retrieval. Nat. Neurosci.https://doi.org/10.1002/10.1038/nn.4507.

193. Pezzulo, G., F. Rigoli & K. Friston. 2016. Active Inference,homeostatic regulation and adaptive behavioural control.Prog. Neurobiol. 134: 17–35.

194. Donnarumma, F., M. Costantini, E. Ambrosini, K. Friston& G. Pezzulo. 2017. Action perception as hypothesis testing.Cortex. 89: 45–60.


https://doi.org/10.1002/hipo.22714

https://doi.org/10.1002/10.1038/nn.4507

Documents

Internally generated hippocampal sequences as a vantage ...rnel.rice.edu/pubs/assets/Pezzulo2017.pdfPezzulo et al. Hippocampal sequences and future-oriented cognition Figure 1. Three