The Habenulo-Raphe Serotonergic Circuit Encodes an ...nct.brain.riken.jp/publications/201408_Amo_Neuron.pdf · The Habenulo-Raphe Serotonergic Circuit Encodes an Aversive Expectation

Neuron

Article

The Habenulo-Raphe Serotonergic Circuit Encodesan Aversive Expectation Value Essentialfor Adaptive Active Avoidance of DangerRyunosuke Amo,1,2 Felipe Fredes,1,8 Masae Kinoshita,1 Ryo Aoki,1,3 Hidenori Aizawa,1,9 Masakazu Agetsuma,1,10

Tazu Aoki,1 Toshiyuki Shiraki,1 Hisaya Kakinuma,1 Masaru Matsuda,4 Masako Yamazaki,1 Mikako Takahoko,1

Takashi Tsuboi,3 Shin-ichi Higashijima,5 Nobuhiko Miyasaka,6 Tetsuya Koide,6 Yoichi Yabuki,6 Yoshihiro Yoshihara,6

Tomoki Fukai,7 and Hitoshi Okamoto1,2,3,*1Laboratory for Developmental Gene Regulation, RIKEN Brain Science Institute, Saitama 351-0198, Japan2Laboratory for Molecular Brain Science, Department of Life Science and Medical Bioscience, Waseda University, Tokyo 162-8430, Japan3Department of Life Sciences, Graduate School of Arts and Sciences, University of Tokyo, Tokyo 153-8902, Japan4Center for Bioscience Research and Education, Utsunomiya University, Tochigi 321-8505, Japan5National Institutes of Natural Sciences, Okazaki Institute for Integrative Bioscience, National Institute for Physiological Sciences, Aichi

444-8787, Japan6Laboratory for Neurobiology of Synapse, RIKEN Brain Science Institute, Saitama 351-0198, Japan7Laboratory for Neural Circuit Theory, RIKEN Brain Science Institute, Saitama 351-0198, Japan8Present address: Institute of Science and Technology Austria, 3400 Klosterneuburg, Austria9Present address: Tokyo Medical and Dental University, Medical Research Institute, Tokyo 113-8510, Japan10Present address: The Institute of Scientific Research and Industrial Research, Osaka University, Mihogaoka 8-1, Ibaraki, Osaka 567-0047,

Japan*Correspondence: [email protected]

http://dx.doi.org/10.1016/j.neuron.2014.10.035

SUMMARY

Anticipation of danger at first elicits panic in animals,but later it helps them to avoid the real threat adap-tively. In zebrafish, as fish experience more and moredanger, neurons in the ventral habenula (vHb) showedtonic increase in the activity to the presented cue andactivated serotonergic neurons in the median raphe(MR).Thisneuronalactivitycould represent theexpec-tation of a dangerous outcome and be used for com-parison with a real outcome when the fish is learninghow to escape from a dangerous to a safer environ-ment. Indeed, inhibiting synaptic transmission fromvHb to MR impaired adaptive avoidance learning,while panic behavior induced by classical fear con-ditioning remained intact. Furthermore, artificially trig-gering this negative outcome expectation signal byoptogenetic stimulation of vHb neurons evoked placeavoidance behavior. Thus, vHb-MR circuit is essentialfor representing the level of expected danger andbehavioral programming to adaptively avoid potentialhazard.

INTRODUCTION

Learned fear responses are emotional behaviors conserved

among animals. In mammals, panic responses to danger such

as freezing or flight are commonly observed when an aversive

stimulus is expected and are useful as immediate response to

threat. However, learning an adaptive strategy to avoid danger

1034 Neuron 84, 1034–1048, December 3, 2014 ª2014 Elsevier Inc.

by the active avoidance of potentially dangerous environments

is often more effective for animal survival than panic behavior

alone. A candidate site responsible for active avoidance is the

lateral habenula (LHb). In mammals, LHb neurons are phasically

activated to negative or aversive emotional events or by situa-

tions where the outcome does not match the initial expectation,

suggesting a role in transmitting antireward and aversive infor-

mation (Matsumoto and Hikosaka, 2007, 2009). LHb neurons

are connected to GABAergic neurons in the rostromedial teg-

mental nucleus (RMTg) that project to dopamine (DA) neurons

in the ventral tegmental area (VTA) (Jhou et al., 2009; Kaufling

et al., 2009). Via this indirect connection, the phasic activation

of LHb causes a transient repression in the activity of VTA DA

neurons (Matsumoto and Hikosaka, 2007). Recent studies indi-

cated that activation of the LHb-DA neuron pathway is aversive

(Lammel et al., 2012; Shabel et al., 2012; Stamatakis and Stuber,

2012) and hyperactivation of this pathway has been implicated in

the etiology of depression (Li et al., 2011, 2013). Although the

LHb neurons also extensively project directly to the raphe and

regulate serotonergic neuron activity (Bernard and Veh, 2012;

Ferraro et al., 1996; Sego et al., 2014), function of this pathway

is largely unknown.

Negative event-related LHb activity causes a dip in DA activity

that might be utilized to inhibit repetition of the punished

behavior via enhancement of the striatal indirect pathway (Frank

et al., 2004; Hikida et al., 2010) and thus facilitate passive avoid-

ance behavior. However, repression of DA activity alone cannot

explain learning of the behavioral strategy to actively escape

from an environment that warns of the incoming aversive stimuli,

because this involves the reinforcement learning process in

which transition from a risky to safe environment is used as a

positive prediction error. Such an error should be represented

by activation rather than repression of DA neurons (Boureau

mailto:[email protected]


http://crossmark.crossref.org/dialog/?doi=10.1016/j.neuron.2014.10.035&domain=pdf

Figure 1. Schematic Illustration of Model for Active Avoidance

Learning

In the association learning phase (i) of active avoidance learning, the fish

associate cue with electrical shock. Presentation of the conditional cue (red

lamp) with a neutral reward expectation value (V(0) = 0) followed by a negative

reward (unexpected electrical shock, r(1) = �R, R > 0) results in production of

negative prediction error (d(1) = r(1) � V(0) = �R � 0 = �R). This negative

prediction error is used to update the expectation value assigned to the cue (V

(1) = V(0) + ad = 0 + a(�R) =�aR), where a is learning rate. After n-time repeats

of trials, the fish acquire a certain negative expectation value (V(n)z�naR/

�R) associated with the cue. In the avoidance learning phase (ii), the fish

reinforce avoidance behaviors according to the prediction error (d) defined as

the difference of the real reward (neutral, r(n + 1) = 0) from expected negative

reward value (V(n) = �R), calculated as follows: d(n + 1) = r(n + 1) � V(n) = 0 �(�R) = +R. The fish also update the expectation value for the next trial using

prediction error (d), i.e. V(n + 1) = V(n) + ad = �R + a(+R) = �R(1 � a). After

several (m-time) trials with successful avoidance, the expectation value (V(n +

m) =�R(1�ma)) reaches almost zero. In this model, translocation of fish from

the left to the right compartment and that of vice versa are treated equally due

to the symmetry of the experimental system. V, expectation value; �R,

negative reward expectation value; r, actual reward; d, prediction error; a,

learning rate.

Neuron

Habenulo-Raphe Pathway in Active Avoidance

and Dayan, 2011; Fiorillo, 2013). In such neural computation, the

expectation of negative reward has to be continuously repre-

sented in the brain by the time when the real outcome of the

behavior is presented to an animal so that representations of

both the reward expectation value and the real outcome can

be used for further calculation of the prediction error. Which

part of the brain represents this reward expectation value has

not been well known.

The activity of raphe serotonin neurons under LHb control

is one of the candidates that can represent an aversive expecta-

tion value and might be used for active avoidance learning

(Dayan and Huys, 2009; Fiorillo, 2013). In monkeys, serotonergic

neurons in the raphe encode reward expectation values (Naka-

mura et al., 2008; Bromberg-Martin et al., 2010). While DA neu-

rons show phasic changes in activity every time reward value

changes and encode reward prediction error, serotonergic neu-

rons in the raphe show tonic changes in activity after reward

value is updated. Thus, serotonergic neurons are likely to signal

reward expectation value. DA neurons alone may not indicate

what the current value state is, and seronogergic neurons alone

may not indicate how the expected reward value is changing

(Proulx et al., 2014).

In mammals, the LHb consists of multiple subnuclei that are

characterized by different combinations of molecular markers,

and the LHb neurons projecting to different targets such as the

raphe and the VTA are distributed across the boundaries of these

subregions (Aizawa et al., 2012; Andres et al., 1999). Likewise,

the raphe neurons are also functionally heterogeneous with

diverse lineal origins, afferent sources and efferent targets (Jen-

sen et al., 2008; Paul and Lowry, 2013). Such complexity of both

LHb and the raphe has hampered the study of the significance of

this LHb-raphe pathway specifically. In fact, the in vivo electro-

physiological characterization of the neurons with precise subre-

gional or hodological identifications has not been performed yet

in the mammalian LHb.

Zebrafish is amenable to genetic manipulations of neural cir-

cuits, and it has recently been shown that the basic structure

in the telencephalon is similar with that of mammals except

that the mediolateral positioning of the homologous structures

of the dorsal pallium is largely inverted between zebrafish and

mammals (Aoki et al., 2013; Mueller and Wullimann, 2009). We

have previously shown that zebrafish can show both panic re-

sponses to the aversive conditioned stimulus after the classical

fear conditioning and adaptive goal-directed escape behavior

after the active avoidance learning is acquired (Agetsuma

et al., 2010; Aoki et al., 2013) (Movie S1 available online). By

imaging of calcium signals across the entire brain of adult zebra-

fish, a discrete area of the dorsal telencephalon, which was inac-

tive immediately after the active avoidance training, became

active the next day during retrieval of the behavioral program

for an active avoidance response (Aoki et al., 2013). Therefore,

zebrafish can be a suitable model animal to examine how these

different ways of coping with danger are regulated. Beside the

zebrafish homolog of the mammalian LHb, the vHb has much

simpler structure than the mammalian LHb in that the zebrafish

vHb has an exclusive direct projection to the ventro-anterior

corner of the MR, but not to any other parts of the brain (Aizawa

et al., 2011; Amo et al., 2010). This characteristic connection in

N

zebrafish has given us the opportunity to specifically examine

the roles of the vHb and its target serotonergic neurons in adap-

tive active avoidance learning.

RESULTS

Tonic Increase in vHb Activity Represents NegativeReward Expectation ValueAccording to the ‘‘Two-Factor Theory’’ of Mowrer, active avoid-

ance is regarded as a type of reinforcement learning (Dayan,

2012; Mowrer, 1947). In this theory, animals learn active avoid-

ance in two phases, association learning and goal-directed

avoidance learning. Figure 1 elaborates this theory from the

perspective of reinforcement learning theory in which expecta-

tion value (V) and prediction error (d) is recalculated after every

trial (Sutton and Barto, 1998). In the association-learning phase,

euron 84, 1034–1048, December 3, 2014 ª2014 Elsevier Inc. 1035

Figure 2. Learning-Dependent Increase of Firing Frequency Responding to the Aversive Cue in vHb Neurons

(A) Schematic illustration of electrophysiology recording in vivo.

(B) Dorsal view of the Hb in the living Tg(dao:GAL4VP16);Tg(UAS:GFP) fish brain showing the vHb labeled with GFP in the left panel. The right panel is a magnified

image of the boxed area in the left panel illustrating a GFP-positive neuron attached by a glass electrode (arrowhead). TeO, optic tectum; Tel, telencephalon.

(C) Schematic diagram for the classical fear conditioning under the microscope. Pre-Cd, Preconditioning; Cd1, Conditioning 1; Cd2, Conditioning 2; Post-Cd,

Postconditioning.

(legend continued on next page)

Neuron



Neuron


fish stays in the same compartment as the red light cue is pre-

sented as the conditioned stimulus (CS) and receives electrical

shock as a negative reward (�R). During this period, the reward

expectation value (V) gradually changes from 0 to �R ultimately,

and the fish eventually learns to associate the cue with the

looming shock (unconditioned stimulus [US]). In the avoidance-

learning phase, the fish acquire a goal-directed adaptive

response by learning to actively swim fromadangerous compart-

ment with a behavioral state characterized by a negative reward

expectation value to a safe compartment with a neutral actual

reward value (zero risk; Figure 1ii). In this later stage of the active

avoidance learning, reinforcement learning would compute the

difference between the negative reward expectation value (indi-

cated as –R in Figure 1ii) and actual reward value (0) to provide

a positive (or appetitive) prediction error (d = +R) that would rein-

force the associated active avoidance (Sutton and Barto, 1998).

First, to study the response of the vHb neurons during the

early phase of the active avoidance learning (the association-

learning phase, Figure 1i), we mimicked this situation by the

classical fear conditioning. It has already been shown that

both larval zebrafish and adult goldfish can learn the classical

fear conditioning under the situations where the body move-

ments are restricted (Aizenberg and Schuman, 2011; Yoshida

et al., 2004). We examined how the learning affected the

response of the vHb neurons to the presentation of the CS. Adult

zebrafish were kept awake, but immobilized by injection of a

neuromuscular blocker (gallamine) and kept alive under the mi-

croscope by perfusing Ringer’s solution into the mouth (Aoki

et al., 2013) (Figure 2A and see Supplemental Experimental Pro-

cedures for details).

For correctly targeting electrodes to the vHb neurons in living

adult zebrafish, we constructed a transgenic line, Tg(dao:

GAL4VP16), from a bacterial artificial chromosome (BAC) clone

containing a vHb-specific gene, diamineoxidase (dao) (Amo

et al., 2010). We also established the vHb-specific transgenic

line, GT014a, through gene trap screening. To identify and char-

acterize learning signals encoded in the vHb, we recorded vHb

neural activity in vivo from the transgenic fish expressing GFP

in the vHb, Tg(dao:GAL4VP16);Tg(UAS:GFP) orGT014a;Tg(UAS:

(D and E) The examples of the raster plots and the time-lapse histograms for the a

neuron before, during and after conditioning. Pink and red shadows indicate the p

lapse histograms of the first trial in the Cd1 session.

(F and G) The averages of activity increase from the base line activity (4.5 s to 2 s b

Wilcoxon signed-rank test) and phasic activation type neurons (G; n = 8, p = 0.0

(H) The averages of the activity during entire CS presentation period (except for fir

respectively) normalized to the averages of firing frequency during the period fro

activation type neurons (H; n = 9).

(I) The averages of the activity immediately after CS presentation (0.5 s) normalized

presentation (2.5 s) in phasic activation type neurons (I; n = 8).

(J and K) The averages of the activity during US application normalized to the

presentation (2.5 s) in tonic type neurons (J; n = 9) and phasic type neurons (K; n

(L andM) Ratio of the neurons showing different types of fear learning-dependent a

49) (c2 test).

(N) Cue-evoked activity/basal activity ratio recorded from the fish that have exp

association learning phase) and have acquired the active avoidance learning (at th

activity ratio (one-way ANOVA, p = 0.0103; Bonferroni multiple comparison test, c

avoidance learning phase, p > 0.05).

Scale bar represents 50 mm. Error bars represent SEM. *p < 0.05, **p < 0.01. ns,

See also Figure S1.

N

GFP) (Figures 2B, S1A, and S1B) (Aoki et al., 2013). In fish, the Hb

is located on the dorsal surface of the brain, so we could record

neural activity in vivo by loose-patch clamp recording technique

from identified vHb neurons visualized by GFP fluorescence (Fig-

ure 2B). We examined the effect of classical fear conditioning in

52 vHb neurons by conditioning fish to associate a CS (red light)

with an US (electrical shock) and recorded neural activity to the

CS from single vHb neurons (Figure 2C).

Approximately one-third of vHb neurons increased their

activity in response to the CS in postconditioning trials (Post-

Cd in Figure 2C) compared to preconditioning trials (Pre-Cd in

Figure 2C) (increase, 17/52, 32.7%; decrease, 4/52, 7.7%; no

change, 31/52, 59.6%; recorded from 28 fish; unpaired t test;

Figure 2L). These positively responsive neurons were catego-

rized into tonic and phasic activation types depending on their

CS responses (Tonic activation-type, 9/52, 17.3%; Phasic acti-

vation-type, 8/52, 15.4%; Figures 2D–2I and 2L).

Tonic activation-type neurons (9/52, 17.3%, Figure 2L) ex-

hibited no or little phasic response to the CS presentation and

termination in the Pre-Cd and the first trial of Cd1 (Figures 2D

inset, 2F, and 2H). A total of 77.8% (7/9) of tonic neurons showed

a phasic response to the US application at the early stage of the

conditioning session, and this responsewas reduced as the con-

ditioning trials proceeded (Figures 2D and 2J). However, the

response to the CS gradually increased during Cd1, and these

neurons showed a tonic increase in firing frequency throughout

the entire CS presentation period in the Post-Cd session (Figures

2D, 2F, and 2H).

Most (87.5%, 7/8) of the phasic activation-type neurons (8/52,

15.4%, Figure 2L) also showed a phasic response to the US

application at the early stage of the conditioning session, and

this response was reduced as the conditioning trials proceeded

(Figures 2E and 2K). In contrast, neurons of this type gradually

increased phasic responses to the CS onset as the trial pro-

ceeded (Figures 2E, 2G, and 2I).

Notably, an unpaired conditioning control did not induce a sig-

nificant increase in activity during the CS presentation in the ma-

jority of vHb neurons (increase, 1/22, 4.5%; decrease, 10/22,

45.5%; no change, 11/22, 50%; recorded from 11 fish; unpaired

ctivities of a tonic activation type (D) and a phasic activation type (E) single vHb

eriod of CS (0–5.5 s) and US (5–5.5 s), respectively. The insets show the time-

efore CS presentation) of the tonic activation type neurons (F; n = 9, p = 0.0039,

078, Wilcoxon signed-rank test) during Pre-Cd and Post-Cd sessions.

st 0.5 s and last 0.5 s that corresponds to phasic activity period and US period,

m 4.5 s before CS presentation to 2 s before CS presentation (2.5 s) in tonic

to the averages of firing frequency during the period from 4.5 s to 2 s before CS

averages of firing frequency during the period from 4.5 s to 2 s before CS

= 8).

ctivity changes in control vHb-GFP fish (L; n = 52) and vHb-silenced fish (M; n =

erienced adaptation alone (control), received electrical shock 10 times (at the

e end of the avoidance learning phase) was normalized to average of the control

ontrol versus association learning phase, p < 0.01, control versus at the end of

not significant.


Neuron


t test) indicating that neural signals were selective to the learned

association.

Because the phasic response to the US was observed in the

first trial of the conditioning but was reduced in both the tonic-

and phasic-activation types of neurons as the conditioning trials

were repeated, this response can represent the prediction error

as previously pointed out for the LHb activity in monkeys (Matsu-

moto and Hikosaka, 2007, 2009). In contrast, the tonic activity

increase in response to the CS was not observed in the first con-

ditioning trial (Figure 2H). This clearly denied the possibility that

this activity increase also represents the prediction error be-

cause the prediction error in the first trial should be the maxi-

mum. Its gradual increase in a learning-dependent manner and

retention throughout the CS presentation period rather supports

the possibility that this activity increase represents the negative

reward expectation value (aversive expectation value) generated

as a result of the association of the US with the CS (Figure 1).

The Cue-Evoked Tonic Activity of the vHb NeuronsShows Increase and Decrease during the ActiveAvoidance Learning as Predicted by the ReinforcementLearning TheoryAccording to the two-factor theory of the active avoidance

learning (Figure 1), if the tonic activity of the ventral habenula

(vHb) represents the negative reward expectation value, then it

should increase at the earlier stage of learning (the association

learning phase) and decrease at the later stage of learning (the

avoidance learning phase) due to reevaluation of negative

reward expectation value depending on the prediction error

that is also recalculated every time after each trial. To test if

this actually happens, we recorded the multiunit activity (MUA)

from the vHb in the awake fish that were immobilized and

brought under the microscope for the MUAmeasurement imme-

diately after either having experienced the earlier stage of active

avoidance learning (until they received electrical shock ten times

in association with cue presentation: Association Learning

Phase; n = 13 from seven fish) or having completed the active

avoidance learning (at the end of the Avoidance Learning Phase;

n = 12 from six fish). As the control group, we used the fish that

were kept in the experimental chamber for 20min (n = 12 from six

fish) before immobilization and transferring under the micro-

scope for the MUA measurement. To make targeting of the

vHb easier, the GT014a; Tg(UAS:GFP) transgenic fish (the

vHb-GFP fish) were used for recording. We calculated the ratio

of the averaged MUA from the vHb during the cue presentation

period (except for the first 500 ms of this period to remove the

contribution of the phasic activity evoked by the cue) relative

to the averaged MUA before the cue presentation period and

compared the ratios among different phases of the active avoid-

ance learning. In Figure 2N, the average ratio for the control

groupwas normalized to 1. Consistent with the cue-evoked tonic

activity observed in the vHb neurons during and after classical

fear conditioning that was performed for immobilized fish under

themicroscope (Figures 2D and 2F), the vHbMUA during the cue

presentation significantly increased in the fish at the earlier stage

of the active avoidance learning (the association learning phase)

(Figure 2N). Furthermore, the cue-evoked change of the MUA in

the fish at the end of the avoidance learning phase returned to


the level statistically indistinguishable from the ratio in the control

fish (Figure 2N). This increase and decrease of the MUA of the

vHb is exactly consistent with the two-factor theory of the active

avoidance learning (Figure 1) and further supports that the vHb

plays an essential role in the active avoidance learning by encod-

ing the negative reward expectation value.

Glutamatergic vHb Neurons Project to SerotonergicMR NeuronsSimilar to the mammalian LHb, virtually all fish vHb neurons

express vesicular glutamate transporter 2a (vglut2a) mRNA, a

marker for glutamatergic neurons (Figure S1C) (Aizawa et al.,

2012; Brinschwitz et al., 2010). To characterize the projection of

vHb glutamatergic neurons, we crossed the vHb-specific Cre

recombinase line, Tg(dao:Cre-mCherry) with a glutamatergic

neuron-specific line, Tg(vglut2a:loxP-DsRed-loxP-GFP) (Satou

et al., 2012). In this double transgenic fish, dao- and vglut2a-co-

expressing neurons were selectively labeled by GFP. We found

that GFP-labeled glutamatergic neurons in the vHb exclusively

project to the MR where serotonergic neurons are localized and

vHb projection was not observed in any other areas (Figure 3A).

To analyze the functional connectivity between vHb neurons

and serotonergic MR neurons, we produced a transgenic line

harboring the light-activated ion channel Channelrhodopsin-2

fused with red fluorescent protein (ChR2-mCherry), Tg(UAS:

hChR2-mCherry). To achieve specific induction of ChR2 in

the vHb, we produced the vHb-selective line Tg(ppp1r14ab:

GAL4VP16) using the BAC clone of the vHb-specific gene

ppp1r14ab (Figure S1D) identified in a previous study (Amo

et al., 2010). Expression of ChR2-mCherry in Tg(ppp1r14ab:

GAL4VP16);Tg(UAS:hChR2-mCherry) adult fish was restricted

to the vHb and meninges (the vHb-ChR2 fish, Figures S1E

and S1F). We recorded electrophysiological activity from triple

transgenic fish expressing ChR2-mCherry in the vHb and GFP

in the MR serotonergic neurons: Tg(ppp1r14ab:GAL4VP16);

Tg(UAS:hChR2-mCherry);[Tg(�3.2pet1:EGFP) or Tg(ETvmat2:

GFP)]. Nearly all serotonergic neurons in theMRwere genetically

labeled by the Tg(�3.2pet1:EGFP) (Lillesaar et al., 2009) or

Tg(ETvmat2:GFP) (Wen et al., 2008) transgenes (Figures S1G–

S1I). Neural activities from these visually identified serotonergic

neurons were measured by loose-patch clamp recording in

acute brain slices of the triple transgenic fish (Figures 3B–3D).

The activity of serotonergic neurons during optical stimulation

of ChR2-expressing vHb axon termini within the MR was

analyzed (Figure 3D). 8 out of 38 recorded serotonergic neurons

increased their firing frequency during optical stimulation (Fig-

ures 3E and 3F; recorded from seven fish; paired t test) while

others showed no response. The increase in firing was inhibited

by application of the AMPA and NMDA receptor antagonists,

NBQX and AP-5 (n = 4 from four fish; Figures 3G and 3H), respec-

tively. Consistent with the electrophysiological findings, some

vHb axons contacted the soma of serotonergic neurons (Fig-

ure S1J). These data show that vHb neurons activate MR seroto-

nergic neurons via excitatory glutamatergic transmission that

ensures the transfer of the signaling of putative negative expec-

tation value to the serotonergic system.

Recently, it has been shown that the mouse LHb also has the

direct glutamatergic projection to the MR, demonstrating the

Figure 3. Optogenetic Activation of vHb

Axon Termini in the MR Activates Seroto-

nergic Neurons

(A) DsRed (red) and GFP (green) expression pat-

terns in the thick oblique section of the Tg(dao:

cre-mCherry);Tg(vglut2a:loxP-DsRed-loxP-GFP)

double transgenic fish brain. Putative gluta-

matergic neurons in the vHb expressing GFP

(green) send axons bypassing the IPN and termi-

nate in the MR. Nuclei of cells are counterstained

with DAPI (blue). dHb, dorsal Hb; IPN, inter-

peduncular nucleus.

(B and C) A sagittal section of the Tg(�3.2pet1:

GFP);Tg(ppp1r14ab:GAL4VP16);Tg(UAS:hChR2-

mCherry) triple transgenic fish showing expression

of GFP in the serotonergic neurons (green) and

ChR2-mCherry expression in the vHb axons

terminated in the MR (red). (C) Magnified image of

the boxed area in (B). Nuclei of cells are counter-

stained with DAPI (blue).

(D) Schematic illustration for the stimulation and

recording sites in the sagittal slice of the brain. The

right panel shows a GFP-positive neuron in the

raphe attached with the recording electrode

(arrowhead). IL, inferior lobe of hypothalamus; P,

pineal organ

(E) Anexample of the raster plots and the time-lapse

histogram for the activity of the MR GFP-positive

neuron excited by the vHb optical stimulation. Blue

lines indicate the periods with the optical stimula-

tion.

(F) The firing frequency of serotonergic neurons

normalized to the firing frequency during presti-

mulation period (2 s period before stimulation; Pre)

in prestimulation period, optogenetic stimulation

period (3 s stimulation period; Stimulation) and

poststimulation period (2 s period after stimulation;

Post) (n = 8, Friedman test with Dunn’s multiple

comparison test).

(G) An example of the raster plots and the time-

lapse histogram for the same neuron as (E) with

glutamate receptors antagonists. Blue lines indi-

cate the periods with the optical stimulation.

(H) Increase in the firing frequency of the seroto-

nergic neurons induced by the optical stimulation

of the vHb axon termini with or without application

of the glutamatergic receptor antagonists (n = 4,

p = 0.0286; Mann-Whitney test).

(I) Difference between firing activity before and

during optogenetic stimulation of vHb axon was

averaged for all recorded serotonergic neurons (n = 38) and nonserotonergic neuron (n = 27) (p = 0.0006, Mann-Whitney test).

(J) Raster plot and time-lapse histogram for the activity of a raphe GFP-negative neuron excited by the vHb optical stimulation. Blue line indicates the period with

optical stimulation.

(K) Difference between firing activity before and during optogenetic stimulation of vHb axon was averaged for stimulation responsive serotonergic neurons (n = 8)

and stimulation responsive nonserotonergic neuron (n = 1).

Scale bars represent (A) 250 mm; (B) 50 mm; and (C) 25 mm. Error bars represent SEM. *p < 0.05, **p < 0.01, ***p < 0.001, ns, not significant.

See also Figure S1.

Neuron


evolutionary conservation of this pathway (Pollak Dorocic et al.,

2014).

We have also recorded neural activity from 27 GFP-negative

nonserotonergic neurons in the median raphe by loose-patch

clamp recording. As described above, 8 out of 38 recorded

GFP-positive serotonergic neurons in the median raphe showed

response to the vHb axon stimulation. In contrast, 26 out of 27

N

nonserotonergic neurons showed no response to the vHb axon

stimulation except for one neuron. As a consequence, the

average of the evoked activity in response to the vHb axon stim-

ulation has become almost negligible among the 27 measured

nonserotonergic neurons as compared to that among the 38

measured serotonergic neurons (Figure 3I). Even the only one

responsive nonserotonergic neuron showed only a very modest


Figure 4. Entopeduncular Nucleus Neurons Project to the vHb

(A–C) Retrograde labeling from the vHb. Coronal section of the GT014a;Tg(UAS:GFP) brain showing the tracer (neurobiotin) injection site (red in A) in the vHb

expressing GFP (green in A) and the retrogradely labeled neurons in the EP area (red in B and C). The asterisk shows the injection site. (C) Close-up view of the

boxed area in (B). The inset in (C) is a close-up view of the boxed area in (C) showing serotonergic axons (green) in the EP area.

(D) The expression pattern of vglut2a mRNA in the same area shown in (C).

(E) Schematic illustration showing the connections confirmed in this study (arrows) and putative connections (dashed arrows) among the vHb, MR and vEP. Cbll,

Cerebellum.

Scale bars represent (A, C, and D) 50 mm; (B) 200 mm; and (C) inset, 25 mm.

Neuron


increase in activity (Figure 3J). The stimulation-evoked activity

increase in this particular nonserotonergic neuron was less

than one-tenth of the average of the stimulation-evoked activity

increases of the eight positively responding serotonergic neu-

rons (Figure 3K).

To identify telencephalic afferents of the vHb, we labeled neu-

rons retrogradely from the vHb under GFP fluorescence guid-

ance (Figure 4A). Retrogradely labeled neurons are localized in

the ventral entopeduncular nucleus (vEP) (Figures 4B and 4C)

where virtually all neurons are glutamatergic (Figure 4D). This

observation is consistent with prior results in mammal and lam-

prey showing that glutamatergic pallidal neurons project to the

LHb (Shabel et al., 2012; Stephenson-Jones et al., 2012) and it

suggested that aversive expectation value information might

be transmitted, at least partially, from vEP glutamatergic neurons

to MR serotonergic neurons via vHb glutamatergic neurons

(Figure 4E). We also confirmed that these retrogradely labeled

neurons receive serotonergic inputs (Figure 4C, inset). This is

consistent with the prior result that serotonergic neurons in the


raphe project to the EP area in zebrafish (Lillesaar et al., 2009)

and thus suggests that the vHb, MR, and vEP may constitute a

ternary neural circuit (Figure 4E).

The vHb-MR Pathway Is Essential for Active AvoidanceLearningTo confirm whether the vHb-MR pathway encodes the negative

reward expectation value essential for active avoidance learning,

we inhibited this neural circuitry in adult zebrafish. To do this, we

used Tg(UAS:GFP-TeNT) containing a fusion protein of GFP and

tetanus toxin (TeNT) light chain, which is an endopeptidase that

cleaves VAMP2 to inhibit neurotransmission (Agetsuma et al.,

2010; Yamamoto et al., 2003), to create the double transgenic

line Tg(dao:GAL4VP16);Tg(UAS:GFP-TeNT). Anti-GFP immu-

nostaining confirmed vHb-selective expression of GFP-TeNT

(Figures 5A and 5B). Expression of GFP-TeNT was observed in

almost all vHb neurons and their axons could be clearly

observed, bypassing the interpeduncular nucleus (IPN) and ter-

minating exclusively in the ventral corner of the MR (Figures 5C

Neuron


and 5D) (Amo et al., 2010). GFP-TeNT expression was vHb-spe-

cific at both larval and adult stages (Figures 5B and S2A) and

gross morphology of the projection was normal (Figures 5B–5D).

We examined adult zebrafish responses to aversive stimuli with

an active avoidance conditioning paradigm (Aoki et al., 2013). In

this task, fishmust escape from a dangerous compartment within

15 s after the appearance of a red light (CS) to an adjacent safe

compartment to avoid an electrical shock (US) (Figure 5E). An

active avoidance conditioning session continued until fish

achieved a learning criterion of successful avoidance responses

in more than eight of ten trials; if fish did not achieve the criteria

in 60 trials, the session was terminated and fish waited until the

next session. Fish with a successful trial rate in three successive

sessions were regarded as learners (Aoki et al., 2013).

Transgenic fish expressing GFP-TeNT (vHb-silenced fish)

were examined for active avoidance learning and showed a large

reduction in the ratio of fish becoming successful learners

compared to sibling control fish (control, 6/17; vHb-silenced,

1/17; Figure 5F; Movies S1 and S2). Furthermore, the ratio of

avoidance response in the first session was significantly reduced

in the vHb-silenced fish (Figure 5G). Therefore, normal vHb-MR

neurotransmission is essential for active avoidance learning.

Locomotion activity during the acclimation period was similar

between vHb-silenced and control fish (Figure 5H). We also as-

sessed basal anxiety levels with the novel tank-diving test

(Egan et al., 2009) and the open-field test (Peitsaro et al., 2003)

but there were no significant differences observed between

vHb-silenced and control fish (Figures S2B–S2G).

If the vHb transmits the aversive expectation value to seroto-

nergic neurons, disruption of serotonin transmission should

impair active avoidance learning. To examine this hypothesis,

we pharmacologically degraded the termini of serotonergic

axons in the telencephalon while soma of serotonergic neurons

in theMR and their axons into other areas were kept largely intact

by injecting 5,7-dihydroxytryptamine (5,7-DHT) into the telen-

cephalon (Figures 5I, 5J, S2H, and S2I). Consistent with the

defects observed in the vHb-silenced fish, the ratio of correct

avoidance response was significantly reduced in the treated

fish compared to control fish, including 5,7-DHT-injected fish

without obvious loss of serotonergic fibers and vehicle-injected

fish (Figure 5K). As in the vHb-silenced fish, locomotion activity

remained at a level similar to that in control fish (Figure 5L). This

result supports that the serotonergic inputs to the telencephalon,

which contains the aversive expectation value information trans-

mitted from the vHb, is essential for active avoidance learning.

vHb-Silenced Fish Can Learn to Exhibit Panic Responseto the Aversive Cue by Pavlovian Learning in ClassicalFear ConditioningDespite the defect of the vHb-silenced fish in learning active

avoidance task by reinforcement learning, wewonderedwhether

these fish can still acquire panic response to the aversive CS by

Pavlovian learning of classical fear conditioning that also re-

quires the association of CS (red light) with US (electrical shock).

In this test, wild-type fish increases their turning frequency in

response to the CS as a learned response (Experimental Proce-

dures and Figure 5M) (Agetsuma et al., 2010). The vHb-silenced

fish also exhibited a CS-evoked increase in turning frequency as

N

a learned response similar to control fish (Figure 5N). The effect

of the vHb-silencing on turning frequency was not significant

(p = 0.4393). In vHb-silenced fish, the startle response during

the US application was not affected, indicating normal sensory

and motor system function (Figure S2J). These results indicate

that the vHb-MR pathway is dispensable for acquiring panic

response to the aversive CS by Pavlovian learning in classical

fear conditioning.

Optogenetic Tonic Activation, but Not Phasic Activation,of vHb Neurons Induces Avoidance BehaviorIf the vHb-MR pathway actually encodes the negative reward

expectation value, we predicted that artificial tonic activation

of this pathway via optogenetic stimulation of the vHb neurons

should transiently attach a negative reward expectation value

to otherwise neutral learning cues, if presented during optoge-

netic stimulation, and thus induce active avoidance responses

from such cues. To examine this hypothesis, we optogenetically

stimulated the vHb neurons by fixating a tip of thin optical fiber

over the Hb through a tiny hole of the skull in the adult transgenic

zebrafish Tg(ppp1r14ab:GAL4VP16);Tg(UAS:hChR2-mCherry)

where Channelrhodopsin2 (ChR2) is specifically expressed in

the vHb neurons (vHb-ChR2 fish; Figures 6A, 6B, S1E, and

S1F; Movie S3; see also Experimental Procedures), and fish

could swim freely in a rectangular tank subdivided into two

equal-size environments containing either a red or green floor

(Figure 6C). We performed the same procedures on control sib-

ling fish that lacked ChR2 expression. The red and green color of

the floors automatically alternated every 2 min. In one group of

experiments (14 transgenic fish, nine control siblings), optoge-

netic stimulation was given if the fish were in the red area after

the floor colors alternated and turned off if the fish moved from

the red to the green area (see Experimental Procedures and Fig-

ure 6D). In a second group of experiments (eight transgenic fish,

five control siblings), optogenetic stimulation was paired with the

green area after the floors alternated color and turned off if the

fish moved from the green to red area. Fish in both groups of ex-

periments showed similar avoidance tendency, so we combined

the results in the following analyses.

Tonic optogenetic stimulation consistently induced the move-

ment of fish from a stimulation-paired environment to a non-

stimulation area (Figure 6E; Movies S4 and S5). Notably, this

movement does not require learning. Even in the first optoge-

netic stimulation trial, vHb-ChR2 fish sought to escape from

the stimulation-paired area, i.e., spending significantly less

time in the stimulation-paired area than control siblings (p =

0.0041) (Figure 6F), indicating a preference for the nonstimula-

tion area with a neutral (zero) reward and expectation value

rather than the stimulation-paired area with a negative reward

expectation value. However, reinforcement learning facilitated

more efficient escape, because vHb-ChR2 fish continued to

show an improvement in the efficiency of escape over multiple

trials (Figures 6F, 6G, and S3A). In the first stimulation trial, opto-

genetic stimulation induced avoidance in a pseudorandom di-

rection until animals reached the nonstimulation area, although

fish showed some tendency to swim in parallel with the longer

side wall of the tank (recognized as small peaks around 90�

and 270� in Figures 6H and 6I left panel). As the fish underwent


(legend on next page)

Neuron



Neuron


the trials, they preferentially swam to the nonstimulation area

directly and quickly as seen during the last trial of stimulation

session 1 in the vHb-ChR2 fish (recognized as an enhanced

peak around 90� in Figures 6H, 6I, right panel, and 6J). The

learned improvement in avoidance behavior was not observed

in control sibling fish that did not express ChR2 in vHb neurons

(Figures S3B and S3C) and persisted even into the poststimula-

tion session, where ChR2-expressing fish showed a slight but

significant increase in aversion from the stimulation-paired color

even in the absence of optical stimulation (Figure 6K).

Optogenetic stimulation of the vHb induced a mild increase of

locomotion activity in the vHb-ChR2 fish that may reflect an in-

crease in exploration behavior (Figures 6L and 6M). However,

this increase was far milder than the one seen in the startle

response induced by the electrical shock (Figure 6N), excluding

the possibility that optogenetic activation of vHb neurons caused

pain sensation in ChR2-expressing fish that facilitated escape

behavior. Control sibling fish lacking ChR2 showed no active

avoidance to optical stimulation, also eliminating the possibility

that laser illumination alone induced the avoidance action.

As mentioned above, some vHb neurons showed phasic

activity increase at the beginning of CS presentation after fear

conditioning (Figure 2E). This phasic activity of the vHb neurons

may also represent the negative reward expectation value. To

exclude the possibility that this activity alone might be sufficient

to induce the avoidance behavior, we examined the effect of op-

togenetic phasic activation of the vHb in the same behavior para-

digm. In contrast to the tonic activation that induced strong avoid-

ance behavior (Figure 6E), the phasic activation could not induce

significant avoidance behavior (Figure 6O). This result supports

Figure 5. Inhibition of Neural Transmission in the vHb-MR Pathway Imp

Not Impaired

(A) Schematic illustration showing a dorsal oblique view of the axonal projections

interpeduncular nucleus (IPN; red and green) in the adult zebrafish brain. OB, olfac

medial subnucleus of dorsal Hb; dIPN, dorsal IPN; vIPN, ventral IPN.

(B–D) Immunofluorescent staining of the coronal sections of adult Tg(dao:GAL4VP

detection of GFP-TeNT). GFP-TeNT signals appeared in the vHb neurons (B), the

MR (D). Nuclei are counterstained with DAPI (magenta).

(E) Schematic illustration of the active avoidance task. The fish had to cross a hu

swam to the opposite compartment within the CS presentation period. In the ‘‘E

period and then received the US, but still managed to cross the hurdle within per

during either period of the CS presentation or the following US application.

(F) Ratio of fish reaching the success criteria, 80% avoidance in ten sequential tria

c2 test).

(G) Averaged ratio of successful avoidance response during first session of activ

(H) Locomotor activity during the adaptation period (control, n = 13; vHb-silence

(I and J) Serotonergic fibers in the telencephalon (green; serotonin immunohistoc

fish (J and J0). I0 and J0 are magnified images of the boxed areas in (I) and (J), res

(K) Averaged ratio of successful avoidance response during the first session of ac

test).

(L) Locomotor activity during the adaptation period (control, n = 17; serotonergic

(M) Schematic illustration of classical fear conditioning. The fish received the red

two conditioning sessions each of which consisted of five trials.

(N) CS-evoked changes in turning frequency. Both control and the vHb-silence

(control, n = 16; vHb-silenced, n = 12). The values are normalized to the averages o

conditioning session). There was no significant difference of turning response

ANOVA).

Scale bars represent (B–D, I, and J) 200 mm; (I0 and J0) 100 mm. Error bars represen

not significant.

See also Figure S2 and Movies S1 and S2.

N

the idea that the tonic activity in the vHb represents the negative

expectation value that is effectively used for the reinforcement

learning of escape behavior but the phasic activity does not.

Together, these results demonstrate that artificial tonic activa-

tion of vHb neurons by optogenetic stimulation, designed to

mimic the natural tonic activity we recorded in the vHb in

response to a CS, could assign an artificial negative reward

expectation value to the local environment and thus induce

escape toward an area with a neutral (or relatively higher) reward

expectation value.

Interruption of the vHb Output Impairs Representationof Aversive Expectation by the vHbIf the tonic activity increase of the vHb neurons provides the neu-

ral substrate representing a negative reward expectation value,

the gradual increase in the level of the tonic activation of the

vHb neurons that we observed during the cue presentation

period both in the classical fear conditioning under the micro-

scope and at the earlier stage of the active avoidance learning

is thought to be the consequence of the repeated recalculation

of the expectation value that became more and more negative

as trials were repeated, because each trial gave a negative

prediction error (Figures 1i, 2D, and 2H). If this is the case, inter-

ruption of the vHb outputs should impair calculation of such a

negative prediction error and recalculation of the expectation

value, and consequently, inhibit the emergence of the tonic ac-

tivity increase during the conditioned cue presentation periods.

To examine this possibility, we measured vHb neuronal activity

during classical fear conditioning under the microscope in

vHb-silenced fish (Figures 2A–2C and 5B).

airs Active Avoidance Learning while Classical Fear Conditioning Is

from the vHb to the MR (blue) and the subnuclei of the dorsal habenula to the

tory bulb; PP, para-pineal organ; dHbL, lateral subnucleus of dorsal Hb; dHbM,

16);Tg(UAS:GFP-TeNT) fish brain stained with the antibody for GFP (green; for

axon bundles from the vHb bypassing the IPN (C) and the axonal terminal in the

rdle to avoid the US upon CS presentation. In the ‘‘Avoidance’’ behavior, a fish

scape’’ behavior, a fish failed to cross the hurdle during the CS presentation

iod of US application. In the ‘‘Failure’’ behavior, a fish did not cross the hurdle

ls in the active avoidance task (control, n = 17; vHb-silenced, n = 17; p = 0.0339,

e avoidance task (control, n = 17; vHb-silenced, n = 17; Mann-Whitney test).

d, n = 12; p = 0.9350; Mann-Whitney test).

hemistry) of vehicle-injected control fish (I and I0) and serotonergic fiber lesion

pectively. Nuclei are counterstained with DAPI (magenta).

tive avoidance (control, n = 19; serotonergic fiber lesion, n = 10; Mann-Whitney

fiber lesion, n = 9; p = 0.2357; Mann-Whitney test).

light in all sessions, and the electrical shock was paired with the red light in the

d fish show increase of turning frequency in the second conditioning session

f the adaptation sessions. Wilcoxon signed-rank test (comparison with the first

between vHb-silenced and sibling (p = 0.4393, two-way repeated-measure

t (G, K, and N) SEM; (H and L) minimum andmaximum. *p < 0.05, **p < 0.01, ns,


Figure 6. Optogenetic Activation of the vHb-MR Pathway Triggers Avoidance Behavior

(A) Adult zebrafish attached with an optical fiber.

(B) Schematic illustration of the sagittal section of the zebrafish brain showing the position of the optical fiber for optical stimulation.

(C and D) The experimental setup (C) and procedure (D) for the optogenetic vHb activation in a freely swimming zebrafish.

(E) Time spent in the tonic stimulation-paired area in the vHb-ChR2 and control sibling fish (vHb-ChR2, n = 22; control, n = 14; p < 0.0001, Mann-Whitney test).

(F) Time spent in the tonic stimulation-paired area in each trial set of the optogenetic stimulation session 1 and the average time of the stimulation session 2 (vHb-

ChR2, n = 22, control sibling, n = 14). Comparison of time in stimulation-paired area at first trial set of stimulation session between vHb-ChR2 and control sibling;

p = 0.0041,Mann-Whitney test. Comparison of time in stimulation-paired area for trial sets of stimulation session 1within vHb-ChR2 or control sibling; vHb-ChR2,

p = 0.0006; control sibling, p = 0.2416, Friedman test with Dunn’s multiple comparison test (compared to first trial of first conditioning).

(legend continued on next page)

Neuron



Neuron


In vHb-silenced fish, TeNT inhibits neurotransmitter release

from the vHb neurons, but should not by itself affect their own

neural activity. However, we found significantly fewer vHb neu-

rons showing a learning-dependent increase in neural activity

during cue presentation in the vHb-silenced fish (increase, 5/

49, 10.2%; decrease, 6/49, 12.2%; no change, 38/49, 77.6%; re-

corded from 32 fish; unpaired t test; Figure 2M) and particularly

so for the tonic activation-type neurons (2/49, 4.1%). This result

is consistent with the idea that the tonic activity increase of the

vHb neurons provides the neural substrate representing a nega-

tive reward expectation value that is essential for calculation of

the prediction error and recalculation of the expectation value

based on the obtained prediction error.

DISCUSSION

All our results of this study support that vHb tonic neuronal activ-

ity represents a negative reward expectation value (aversive

expectation value), assigned to the CS associated with the aver-

sive stimuli. In fact, this activity showed increase and decrease

during the active avoidance learning exactly as predicted by

the two-factor theory of the active avoidance learning and the

reinforcement learning theory. This activity could act as the neu-

ral substrate underlying the computation of various parameters

for reinforcement learning when fish are learning how to escape

from a dangerous to safe environments by using the transition

from a risky to safe environment as a positive prediction error

(Figure 1). Specific inactivation of the vHb-MR pathway impaired

only the reinforcement learning of active avoidance and abro-

gated the increase in the tonic activity of vHb neurons resulting

from association of the CS with the US at the earlier stage of

the active avoidance learning (Figure 1i). However, this manipu-

lation did not affect the Pavlovian learning of panic response in

classical fear conditioning, implicating segregation of the neural

circuits including the vHb-MR pathway dedicated for adaptive

coping of fear from the circuits for panic response to fear at

some stage of information processing of aversive stimulus.

In monkeys, neurons in the ventral pallidum and raphe show

tonic activity to negative reward from cue onset to reward receipt

(Nakamura et al., 2008; Tachibana and Hikosaka, 2012). In ze-

brafish, the vHb receives glutamatergic projection from the

vEP neurons (Figure 4E). The cluster of the labeled neurons is

(G) Latency tomove from the tonic stimulation-paired area in which the fish receive

trial (vHb-ChR2, n = 21, p = 0.0028, Wilcoxon signed-rank test).

(H) Schematic illustration of the angle of swimming direction. Swimming direction

(I) Example of a trace of escape swimming (upper panel) and the histogram for ave

total swimming distance (lower panel; n = 21) in first and last trial of the stimulat

(J) Averages of distance traveled in 90� ± 10� direction normalized to the total d

n = 21, p = 0.0083, Wilcoxon signed-rank test).

(K) Total time spent in the tonic stimulation-paired area in the adaptation session

p = 0.0494; Wilcoxon signed-rank test).

(L and M) Average of swimming speed during the first optogenetic stimulation tr

signed-rank test) and control sibling (M; n = 13, p = 0.0681, Wilcoxon signed-ran

(N) Change of averaged swimming speed induced by optogenetic activation of the

4; p = 0.0021, Mann-Whitney test).

(O) Time spent in the phasic stimulation-paired area in the vHb-ChR2 and contro

Error bars represent SEM. *p < 0.05, **p < 0.01, ***p < 0.001, ##p < 0.01, ns, not

See also Figure S3 and Movies S3, S4, and S5.

N

likely to be the zebrafish homolog of the habenular-projecting

globus pallidus (GPh) neurons in lamprey and the habenular-pro-

jecting entopeduncular nucleus (EPh) neurons in mouse (Shabel

et al., 2012; Stephenson-Jones et al., 2013). Recently, it has

been shown that the mouse LHb has the direct glutamatergic

projection to the MR as the zebrafish vHb does (Pollak Dorocic

et al., 2014). Thus, the vEP in zebrafish or EPh in mouse (vEP/

EPh), vHb in zebrafish or LHb in mammals (vHb/LHb), and MR

constitutes a neural pathway that is conserved throughout verte-

brate evolution, and the neural activity conducted through this

pathway may act as a neural substrate for representation of

negative reward expectation value in CS-associated environ-

ments. This hypothesis is supported by the preceding result

that the optogenetic activation of the EPh inputs into the LHb

in rats also causes avoidance from the environments associated

with the optogenetic stimulation (Shabel et al., 2012).

The vEP/EPh-vHb/LHb-MR pathway converts glutamatergic

transmission from the vEP/EPh to serotonergic transmission. In

the reinforcement learning, the reward expectation value that

was revised by the previous trial has to be remembered at the

beginning of the trial and to be retained till reward is given at

the end of the trial so that the prediction error will be calculated

and the expectation value will also be revised for the next trial

(Figure 1). Serotonin signal may be a convenient mediator for

such transient memory of expectation value by the cells that

act as the calculator of prediction error by evoking a prolonged

change in the intracellular signaling status.

By using optogenetic methods in mice, both quasi-tonic acti-

vation of the termini of the projection from the LHb to the RMTg

and intermittently repeated phasic activation of the LHb neurons

directly projecting to the VTA induced avoidance from the envi-

ronments where optogenetic stimuli were given (Lammel et al.,

2012; Stamatakis and Stuber, 2012), suggesting the involvement

of these neural circuits in the avoidance learning. As the activities

of some VTA neurons have been reported to represent prediction

errors in mammals (Schultz, 2013), the neural circuits including

the vEP/EPh-vHb/LHb-MR pathway and the DA neurons may

play a critical role for implementation of neural computations in

the brain for aversive reinforcement learning such as calculation

of reward expectation values and prediction errors.

The adult zebrafish brain does not have in its midbrain the

direct homologs of VTA or the substantia nigra pars compacta

d the optogenetic stimulation to the nonstimulation area in the first trial and last

from tonic stimulation-paired area to the nonstimulation area was set as 90�.rages of swimming distances for different directions that were normalized to the

ion session 1 (vHb-ChR2).

istance traveled in the first and last trial of the stimulation session (vHb-ChR2,

and the poststimulation session (control, n = 8, p = 0.3125; vHb-ChR2, n = 14,

ial and first adaptation trial in vHb-ChR2 fish (L; n = 21, p = 0.0101, Wilcoxon

k test).

vHb and electrical shock (optogenetic stimulation, n = 21; electrical shock, n =

l sibling fish (vHb-ChR2, n = 8; control, n = 8; p = 0.5054, Mann-Whitney test).

significant.


Neuron


(SNc) of the mammalian brain, but instead have two DA clusters

in the posterior tuberculum of the diencephalon and another DA

cluster in the posterior tuberal nucleus (Rink and Wullimann,

2002). The former two clusters send ascending projection to

the subpallial region homologous to the mammalian striatum

and may correspond to the mammalian ‘‘meso’’-striatal and/or

‘‘meso’’-limbic systems, while the latter projects to the pallium

and may corresponds to the mammalian ‘‘meso’’-cortical sys-

tem (Rink and Wullimann, 2002). Whether the activities of these

DA neurons can represent the reward prediction error as the

mammalian VTA and SNc neurons do and how these DA neurons

interact with the Hb and the vHb-MR system in zebrafish has to

be investigated in the future.

In the previous experiment performed in monkeys in which air-

puff was given in association with a visual cue, only the neurons

showing the phasic increase in the activity at the beginning of the

presentation period of the air puff-predicting cue were observed

in the LHb, but not the ones showing the tonic increase in the ac-

tivity throughout the cue-presentation period as we observed

here in the zebrafish vHb (Matsumoto and Hikosaka, 2009). In

these experiments, monkeys had already been trained for a

long period before the recording of the neural activity. During

the long training period, monkeys may have developed some

behaviors (or even mindsets) that are effective in counteracting

the discomforts caused by airpuffs. Therefore, by the time of

the experiments, the expectation value for the negative reward

may have returned to zero, causing no tonic activation of the

LHb neurons, even if the monkey LHb has neurons similar to

the zebrafish tonic-activation type vHb neurons.

We previously reported that the lateral subnucleus of the dor-

sal Hb (homologous to the dorsal subregion of the medial Hb

[dMHb] in mammal) is essential for the experience-dependent

conversion of classically conditioned fear responses from

freezing to agitation (Agetsuma et al., 2010). It has also been re-

ported that the dMHb inmouse plays a similar role as the dHbL in

zebrafish in control of fear response (Okamoto and Aizawa,

2013; Yamaguchi et al., 2013). Here, in contrast, we show that

the vHb-MR pathway transfers a negative reward expectation

value signal obtained through the initial association learning to

the reinforcement learning system to enable adaptive active

avoidance. Thus, different habenular subnuclei are engaged in

different forms of fear learning (Okamoto et al., 2012).

The inability of fish with defective vHb-MR signaling to express

adaptive fear behavior in response to dangerous environments

suggests the involvement of the vHb-MR pathway, and by ho-

mology, the mammalian LHb-MR pathway in exaggerated panic

responses to perceived danger that occurs in common anxiety

disorders. Impairment of the serotonin system is involved in

diverse behavioral abnormalities including anxiety, impulsivity,

behavioral inhibition, aggression, feeding, panic and sleep (Cools

et al., 2008; Heisler et al., 2006; Jouvet, 1999; Miyazaki et al.,

2012; Nelson and Trainor, 2007; Soubrie, 1986). This functional

diversity might be linked to diversity in the serotonergic neuron

populations in terms of neural connectivity and cell lineage

(Bang et al., 2012; Kim et al., 2009; Paul and Lowry, 2013).

In fact, the optogenetic activation of the dorsal raphe (DR) Pet-

1-positive neurons reinforced mice to explore the stimulation-

coupled spatial region, shifts sucrose preference, drove optical


self-stimulation, and directed sensory discrimination learning,

suggesting that the DR neurons encode reward rather than pun-

ishment as we suggested for MR (Liu et al., 2014). Relatively sim-

ple neural circuits in zebrafish can also provide a good model to

study such versatile roles of serotonergic neurons other than

those described in this work.

EXPERIMENTAL PROCEDURES

Animals

All protocols were reviewed and approved by the Animal Care and Use Com-

mittees of the RIKEN Brain Science Institute. Transgenic lines generated

in this study are the following; Tg(UAS:GFP-TeNT)rw0146a, Tg(UAS:hChR2-

mCherry)rw0147a, Tg(dao:GAL4VP16)rw0148a, Tg(dao:Cre-mCherry)rw0149a, and

Tg(ppp1r14ab:GAL4VP16)rw0150a. The GT014a line was established through

the gene trap screening using the pT2KSAGFF gene trap vector (Asakawa

et al., 2008; Koide et al., 2009). The other transgenic lines used in this study

are the following (Aizawa et al., 2005; Asakawa et al., 2008; Lillesaar et al.,

2009; Satou et al., 2012; Wen et al., 2008); Tg(brn3a-hsp70:GFP)rw0110b,

Tg(UAS:GFP), Tg(vglut2a:loxP-DsRed-loxP-GFP), Tg(�3.2pet1:EGFP) and

Tg(ETvmat2:GFP).

Loose-Patch Clamp and Multiunit Activity Recording from the vHb

Neuron In Vivo

The Tg(dao:GAL4VP16);Tg(UAS:GFP) and GT014a;Tg(UAS:GFP) adult 6- to

24-month-old zebrafish were used. Loose-patch clamp recording was per-

formed in naive fish. Multiunit activity was recorded immediately after active

avoidance task was terminated at the different phases of learning, i.e. adapta-

tion (control), association phase (receiving 10 shock), and avoidance learning

phase (completed the task). The recoding conditions are described previously

(Aoki et al., 2013).

Active Avoidance Conditioning and Classical Fear Conditioning

The active avoidance and classical fear conditioning tests were performed as

described in our previous reports (Agetsuma et al., 2010; Aoki et al., 2013). The

Tg(dao:GAL4VP16);Tg(UAS:GFP-TeNT) and control sibling adult female 6- to

12-month-old zebrafish were used.

The Optogenetic Activation of the vHb Neurons of Freely

Swimming Fish

The Tg(ppp1r14ab:GAL4VP16);Tg(UAS:hChR2-mCherry) and control sibling

adult 12- to 24-month-old zebrafish were used. We made a small hole on

the skull by the micro drill (World Precision Instruments). Optical fiber was in-

serted and attached on the skull by surgical glue. Optical stimulation and the

bottom color were controlled by MATLAB (MathWorks)-based program.

Details of experimental procedures are available in the Supplemental Exper-

imental Procedures.

SUPPLEMENTAL INFORMATION

Supplemental Information includes Supplemental Experimental Procedures,

three figures, and five movies and can be found with this article online at

http://dx.doi.org/10.1016/j.neuron.2014.10.035.

AUTHOR CONTRIBUTIONS

R.Amo and H.O. designed all experiments and wrote the manuscript. H.O. su-

pervised all experiments. R.Amo performed all experiments and analyses in

collaboration with the other authors. M.K., H.A., M.A., T.S., H.K., M.M., S.H.,

N.M., T.K., Y.Yabuki. and Y.Yoshihara generated transgenic fish lines. M.A.

and T.A. designed behavior tests. M.Y. andM.T. carried out the behavior tests.

M.K. designed electrophysiology experiments. F.F., R.Aoki, and T.T. designed

and performed the in vivo optogenetic experiment. H.A. and M.T. performed

histological experiments. T.F. helped in theoretical interpretation of experi-

mental data and design of experiments.


Neuron


ACKNOWLEDGMENTS

We thank K. Deisseroth for providing the hChR2-mCherry DNA construct; L.

Bally-Cuif for providing Tg(�3.2pet1:GFP) fish; B. Zhang for providing Tg(ETv-

mat2:GFP) fish; K. Kawakami for providing Tg(UAS:GFP) fish, Tol2, iTol2, and

pT2KSAGFF DNA constructs; H. Ohmori and M. Mizutani for advice on micro-

surgery; RIKEN ASI advanced technology support division for support of

experimental chambers production. We also thank S. Ishii, K. Doya, Y. Iso-

mura, T. Ohshima, C. Yokoyama, A.V. Terashima, and Y. Goda for comments

and discussions, and the members of our laboratory for discussions and for

fish care support. This work was supported in parts by Grant in Aid for Scien-

tific Research (23120008) and Strategic Research Program for Brain Science

from MEXT of Japan and CREST from JST, Japan and the Special Postdoc-

toral Researchers Program from RIKEN.

Accepted: October 14, 2014

Published: November 20, 2014

REFERENCES

Agetsuma, M., Aizawa, H., Aoki, T., Nakayama, R., Takahoko, M., Goto, M.,

Sassa, T., Amo, R., Shiraki, T., Kawakami, K., et al. (2010). The habenula is

crucial for experience-dependent modification of fear responses in zebrafish.

Nat. Neurosci. 13, 1354–1356.

Aizawa, H., Bianco, I.H., Hamaoka, T., Miyashita, T., Uemura, O., Concha,

M.L., Russell, C., Wilson, S.W., and Okamoto, H. (2005). Laterotopic represen-

tation of left-right information onto the dorso-ventral axis of a zebrafish

midbrain target nucleus. Curr. Biol. 15, 238–243.

Aizawa, H., Amo, R., and Okamoto, H. (2011). Phylogeny and ontogeny of the

habenular structure. Front. Neurosci 5.

Aizawa, H., Kobayashi, M., Tanaka, S., Fukai, T., and Okamoto, H. (2012).

Molecular characterization of the subnuclei in rat habenula. J. Comp.

Neurol. 520, 4051–4066.

Aizenberg, M., and Schuman, E.M. (2011). Cerebellar-dependent learning in

larval zebrafish. J. Neurosci. 31, 8708–8712.

Amo, R., Aizawa, H., Takahoko,M., Kobayashi, M., Takahashi, R., Aoki, T., and

Okamoto, H. (2010). Identification of the zebrafish ventral habenula as a homo-

log of the mammalian lateral habenula. J. Neurosci. 30, 1566–1574.

Andres, K.H., von During, M., and Veh, R.W. (1999). Subnuclear organization of

the rat habenular complexes. J. Comp. Neurol. 407, 130–150.

Aoki, T., Kinoshita, M., Aoki, R., Agetsuma, M., Aizawa, H., Yamazaki, M.,

Takahoko, M., Amo, R., Arata, A., Higashijima, S., et al. (2013). Imaging of neu-

ral ensemble for the retrieval of a learned behavioral program. Neuron 78,

881–894.

Asakawa, K., Suster, M.L., Mizusawa, K., Nagayoshi, S., Kotani, T., Urasaki,

A., Kishimoto, Y., Hibi, M., and Kawakami, K. (2008). Genetic dissection of

neural circuits by Tol2 transposon-mediated Gal4 gene and enhancer trapping

in zebrafish. Proc. Natl. Acad. Sci. USA 105, 1255–1260.

Bang, S.J., Jensen, P., Dymecki, S.M., and Commons, K.G. (2012).

Projections and interconnections of genetically defined serotonin neurons in

mice. Eur. J. Neurosci. 35, 85–96.

Bernard, R., and Veh, R.W. (2012). Individual neurons in the rat lateral habenu-

lar complex project mostly to the dopaminergic ventral tegmental area or to the

serotonergic raphe nuclei. J. Comp. Neurol. 520, 2545–2558.

Boureau, Y.L., and Dayan, P. (2011). Opponency revisited: competition and

cooperation between dopamine and serotonin. Neuropsychopharmacology

36, 74–97.

Brinschwitz, K., Dittgen, A., Madai, V.I., Lommel, R., Geisler, S., and Veh, R.W.

(2010). Glutamatergic axons from the lateral habenula mainly terminate on

GABAergic neurons of the ventral midbrain. Neuroscience 168, 463–476.

Bromberg-Martin, E.S., Hikosaka, O., and Nakamura, K. (2010). Coding of task

reward value in the dorsal raphe nucleus. J. Neurosci. 30, 6262–6272.

Cools, R., Roberts, A.C., and Robbins, T.W. (2008). Serotoninergic regulation

of emotional and behavioural control processes. Trends Cogn. Sci. 12, 31–40.

N

Dayan, P. (2012). Instrumental vigour in punishment and reward. Eur. J.

Neurosci. 35, 1152–1168.

Dayan, P., and Huys, Q.J.M. (2009). Serotonin in affective control. Annu. Rev.

Neurosci. 32, 95–126.

Egan, R.J., Bergner, C.L., Hart, P.C., Cachat, J.M., Canavello, P.R., Elegante,

M.F., Elkhayat, S.I., Bartels, B.K., Tien, A.K., Tien, D.H., et al. (2009).

Understanding behavioral and physiological phenotypes of stress and anxiety

in zebrafish. Behav. Brain Res. 205, 38–44.

Ferraro, G., Montalbano, M.E., Sardo, P., and La Grutta, V. (1996). Lateral ha-

benular influence on dorsal raphe neurons. Brain Res. Bull. 41, 47–52.

Fiorillo, C.D. (2013). Two dimensions of value: dopamine neurons represent

reward but not aversiveness. Science 341, 546–549.

Frank, M.J., Seeberger, L.C., and O’reilly, R.C. (2004). By carrot or by stick:

cognitive reinforcement learning in parkinsonism. Science 306, 1940–1943.

Heisler, L.K., Jobst, E.E., Sutton, G.M., Zhou, L., Borok, E., Thornton-Jones,

Z., Liu, H.Y., Zigman, J.M., Balthasar, N., Kishi, T., et al. (2006). Serotonin

reciprocally regulates melanocortin neurons to modulate food intake.

Neuron 51, 239–249.

Hikida, T., Kimura, K.,Wada, N., Funabiki, K., andNakanishi, S. (2010). Distinct

roles of synaptic transmission in direct and indirect striatal pathways to reward

and aversive behavior. Neuron 66, 896–907.

Jensen, P., Farago, A.F., Awatramani, R.B., Scott, M.M., Deneris, E.S., and

Dymecki, S.M. (2008). Redefining the serotonergic system by genetic lineage.

Nat. Neurosci. 11, 417–419.

Jhou, T.C., Geisler, S., Marinelli, M., Degarmo, B.A., and Zahm, D.S. (2009).

The mesopontine rostromedial tegmental nucleus: A structure targeted by

the lateral habenula that projects to the ventral tegmental area of Tsai and sub-

stantia nigra compacta. J. Comp. Neurol. 513, 566–596.

Jouvet, M. (1999). Sleep and serotonin: an unfinished story.

Neuropsychopharmacology 21 (2, Suppl), 24S–27S.

Kaufling, J., Veinante, P., Pawlowski, S.A., Freund-Mercier, M.J., and Barrot,

M. (2009). Afferents to the GABAergic tail of the ventral tegmental area in the

rat. J. Comp. Neurol. 513, 597–621.

Kim, J.C., Cook, M.N., Carey, M.R., Shen, C., Regehr, W.G., and Dymecki,

S.M. (2009). Linking genetically defined neurons to behavior through a broadly

applicable silencing allele. Neuron 63, 305–315.

Koide, T., Miyasaka, N.,Morimoto, K., Asakawa, K., Urasaki, A., Kawakami, K.,

and Yoshihara, Y. (2009). Olfactory neural circuitry for attraction to amino acids

revealed by transposon-mediated gene trap approach in zebrafish. Proc. Natl.

Acad. Sci. USA 106, 9884–9889.

Lammel, S., Lim, B.K., Ran, C., Huang, K.W., Betley, M.J., Tye, K.M.,

Deisseroth, K., and Malenka, R.C. (2012). Input-specific control of reward

and aversion in the ventral tegmental area. Nature 491, 212–217.

Li, B., Piriz, J., Mirrione, M., Chung, C., Proulx, C.D., Schulz, D., Henn, F., and

Malinow, R. (2011). Synaptic potentiation onto habenula neurons in the learned

helplessness model of depression. Nature 470, 535–539.

Li, K., Zhou, T., Liao, L., Yang, Z., Wong, C., Henn, F., Malinow, R., Yates, J.R.,

3rd, and Hu, H. (2013). bCaMKII in lateral habenula mediates core symptoms

of depression. Science 341, 1016–1020.

Lillesaar, C., Stigloher, C., Tannhauser, B., Wullimann, M.F., and Bally-Cuif, L.

(2009). Axonal projections originating from raphe serotonergic neurons in the

developing and adult zebrafish, Danio rerio, using transgenics to visualize

raphe-specific pet1 expression. J. Comp. Neurol. 512, 158–182.

Liu, Z., Zhou, J., Li, Y., Hu, F., Lu, Y., Ma, M., Feng, Q., Zhang, J.E., Wang, D.,

Zeng, J., et al. (2014). Dorsal raphe neurons signal reward through 5-HT and

glutamate. Neuron 81, 1360–1374.

Matsumoto, M., and Hikosaka, O. (2007). Lateral habenula as a source of

negative reward signals in dopamine neurons. Nature 447, 1111–1115.

Matsumoto, M., and Hikosaka, O. (2009). Representation of negative motiva-

tional value in the primate lateral habenula. Nat. Neurosci. 12, 77–84.

Miyazaki, K., Miyazaki, K.W., and Doya, K. (2012). The role of serotonin in the

regulation of patience and impulsivity. Mol. Neurobiol. 45, 213–224.


Neuron


Mowrer, O.H. (1947). On the dual nature of learning—a re-interpretation of

‘‘conditioning’’ and ‘‘problem-solving’’. Harv. Educ. Rev. 17, 102–148.

Mueller, T., and Wullimann, M.F. (2009). An evolutionary interpretation of tele-

ostean forebrain anatomy. Brain Behav. Evol. 74, 30–42.

Nakamura, K., Matsumoto, M., and Hikosaka, O. (2008). Reward-dependent

modulation of neuronal activity in the primate dorsal raphe nucleus.

J. Neurosci. 28, 5331–5343.

Nelson, R.J., and Trainor, B.C. (2007). Neural mechanisms of aggression. Nat.

Rev. Neurosci. 8, 536–546.

Okamoto, H., and Aizawa, H. (2013). Fear and anxiety regulation by conserved

affective circuits. Neuron 78, 411–413.

Okamoto, H., Agetsuma, M., and Aizawa, H. (2012). Genetic dissection of the

zebrafish habenula, a possible switching board for selection of behavioral

strategy to cope with fear and anxiety. Dev. Neurobiol. 72, 386–394.

Paul, E.D., and Lowry, C.A. (2013). Functional topography of serotonergic sys-

tems supports the Deakin/Graeff hypothesis of anxiety and affective disorders.

J. Psychopharmacol. (Oxford) 27, 1090–1106.

Peitsaro, N., Kaslin, J., Anichtchik, O.V., and Panula, P. (2003). Modulation of

the histaminergic system and behaviour by a-fluoromethylhistidine in zebra-

fish. J. Neurochem. 86, 432–441.

Pollak Dorocic, I., Furth, D., Xuan, Y., Johansson, Y., Pozzi, L., Silberberg, G.,

Carlen, M., andMeletis, K. (2014). A whole-brain atlas of inputs to serotonergic

neurons of the dorsal and median raphe nuclei. Neuron 83, 663–678.

Proulx, C.D., Hikosaka, O., and Malinow, R. (2014). Reward processing by the

lateral habenula in normal and depressive behaviors. Nat. Neurosci. 17, 1146–

1152.

Rink, E., andWullimann, M.F. (2002). Connections of the ventral telencephalon

and tyrosine hydroxylase distribution in the zebrafish brain (Danio rerio) lead to

identification of an ascending dopaminergic system in a teleost. Brain Res.

Bull. 57, 385–387.

Satou, C., Kimura, Y., and Higashijima, S. (2012). Generation of multiple clas-

ses of V0 neurons in zebrafish spinal cord: progenitor heterogeneity and tem-

poral control of neuronal diversity. J. Neurosci. 32, 1771–1783.

Schultz, W. (2013). Updating dopamine reward signals. Curr. Opin. Neurobiol.

23, 229–238.

Sego, C., Goncalves, L., Lima, L., Furigo, I.C., Donato, J., Jr., and Metzger, M.

(2014). Lateral habenula and the rostromedial tegmental nucleus innervate


neurochemically distinct subdivisions of the dorsal raphe nucleus in the rat.

J. Comp. Neurol. 522, 1454–1484.

Shabel, S.J., Proulx, C.D., Trias, A., Murphy, R.T., and Malinow, R. (2012).

Input to the lateral habenula from the basal ganglia is excitatory, aversive,

and suppressed by serotonin. Neuron 74, 475–481.

Soubrie, P. (1986). Reconciling the role of central serotonin neurons in human

and animal behavior. Behav. Brain Sci. 9, 319–335.

Stamatakis, A.M., and Stuber, G.D. (2012). Activation of lateral habenula in-

puts to the ventral midbrain promotes behavioral avoidance. Nat. Neurosci.

15, 1105–1107.

Stephenson-Jones, M., Floros, O., Robertson, B., and Grillner, S. (2012).

Evolutionary conservation of the habenular nuclei and their circuitry controlling

the dopamine and 5-hydroxytryptophan (5-HT) systems. Proc. Natl. Acad. Sci.

USA 109, E164–E173.

Stephenson-Jones, M., Kardamakis, A.A., Robertson, B., and Grillner, S.

(2013). Independent circuits in the basal ganglia for the evaluation and selec-

tion of actions. Proc. Natl. Acad. Sci. USA 110, E3670–E3679.

Sutton, R.S., and Barto, A.G. (1998). Reinforcement Learning: An Introduction.

(Cambridge, MA: MIT Press).

Tachibana, Y., and Hikosaka, O. (2012). The primate ventral pallidum encodes

expected reward value and regulates motor action. Neuron 76, 826–837.

Wen, L., Wei, W., Gu, W., Huang, P., Ren, X., Zhang, Z., Zhu, Z., Lin, S., and

Zhang, B. (2008). Visualization of monoaminergic neurons and neurotoxicity

of MPTP in live transgenic zebrafish. Dev. Biol. 314, 84–92.

Yamaguchi, T., Danjo, T., Pastan, I., Hikida, T., and Nakanishi, S. (2013).

Distinct roles of segregated transmission of the septo-habenular pathway in

anxiety and fear. Neuron 78, 537–544.

Yamamoto, M.,Wada, N., Kitabatake, Y., Watanabe, D., Anzai, M., Yokoyama,

M., Teranishi, Y., and Nakanishi, S. (2003). Reversible suppression of glutama-

tergic neurotransmission of cerebellar granule cells in vivo by genetically

manipulated expression of tetanus neurotoxin light chain. J. Neurosci. 23,

6759–6767.

Yoshida, M., Okamura, I., and Uematsu, K. (2004). Involvement of the cere-

bellum in classical fear conditioning in goldfish. Behav. Brain Res. 153,

143–148.

Documents

The Habenulo-Raphe Serotonergic Circuit Encodes an ...nct.brain.riken.jp/publications/201408_Amo_Neuron.pdf · The Habenulo-Raphe Serotonergic Circuit Encodes an Aversive Expectation