Thermodynamics of Genuine Nonequilibrium States under Feedback Control

David Abreu and Udo Seifert

II. Institut fur Theoretische Physik, Universitat Stuttgart, 70550 Stuttgart, Germany(Received 13 September 2011; published 19 January 2012)

For genuine nonequilibrium states that even at fixed external control parameter exhibit dissipation, we

extend the Hatano-Sasa equality to processes with feedback control. The resulting bound on the maximal

extractable work is substantially sharper than what would follow from applying the Sagawa-Ueda equality

to transitions involving such states. For repeated measurements at short enough intervals, the power thus

extracted can even exceed the average cost of driving as demonstrated explicitly with a simple, analyti-

cally solvable example.

DOI: 10.1103/PhysRevLett.108.030601 PACS numbers: 05.70.Ln, 05.40.�a

Introduction and main results.—According to the secondlaw of thermodynamics, the work Wout extracted in anyprocess applied to a system coupled to a heat bath oftemperature T cannot exceed the free energy difference(��F) between the initial and final equilibrium states. If,however, additional information I about the state of thesystem during this process becomes available throughmeasurements, more work can be extracted by adjustinga control parameter driving the transition accordingly. Forsuch processes with feedback, the thermodynamic bound

Wout � ��Fþ kBTI ; (1)

with kB Boltzmann’s constant, has been proven for singleand for repeated measurements for both Hamiltonian andstochastic dynamics [1–6]. This relation has been verifiedin an experiment using a colloidal particle trapped by twofeedback-controlled electric fields [7]. Theoretical casestudies of Eq. (1) include Szilard-type engines using two-state systems or a partitioning of phase space in tworegions [8,9] and Langevin systems [10,11]. Forfeedback-reversible processes, the bound in Eq. (1) canbe saturated [12]. For finite-time processes, the maximumextractable work has been investigated for a Brownianparticle in time-dependent harmonic traps [13].Characteristically, all the systems considered so far reachgenuine equilibrium for a constant external control pa-rameter � with a Boltzmann-type distribution and zerocurrents.

On the other hand, there is a large class of systems whichat constant � approach a genuine nonequilibrium steadystate (NESS) with permanent dissipation which under iso-thermal conditions is called the housekeeping heat Qhk ¼T�Shk [14]. Examples include essentially all motor pro-teins, since they typically move with an on average con-stant speed at fixed external conditions like concentrationsof chemicals or applied forces or torques (see, e.g., [15,16],and references therein). Likewise, a colloidal particledriven by a constant force along a periodic potential canserve as another paradigm for such a NESS [17,18].

In this Letter, we study the thermodynamics of feedbackcontrol of such nonequilibrium states. How much (more)work can be extracted if information acquired throughmeasurement is used for optimal driving? Is it possible toextract net work that exceeds even the cost of driving fromtransitions between such states? One could be inclined totry to answer these questions naively by generalizingEq. (1) to this situation in the form

� �Stot � kBI ; (2)

with �Stot being the total entropy change of system andheat bath, thereby getting rid of the no longer well-definedfree energy when dealing with genuine NESSs. Indeed, asshown towards the end of the Letter, Eq. (2) is true for suchsystems even though this fact seems not to have been statedyet explicitly. On the other hand, Eq. (2) is about as uselessas a sensitive bound for practical situations as the ordinarysecond law [Eq. (2) with I ¼ 0] is for non-feedback-driventransitions between NESSs. The permanent dissipationrequired to drive these NESSs masks all finer detailsassociated with the transition between such states.Addressing this issue for transitions without feedback,Hatano and Sasa [14] have derived an equality from whichthey obtained a sensitive bound focusing on the extradissipation.As our first main result, we will generalize their result to

feedback-driven processes and derive the equality

hexp½�ð�stot ��shk þ IÞ�i ¼ 1 (3)

for trajectory-dependent quantities indicated throughout bythe corresponding small letters and setting kB ¼ T ¼ 1from now on such that heat can be replaced by the corre-sponding entropy change. The average is over arbitraryinitial conditions and any number of measurements withsubsequent adaptation of a control parameter implement-ing the feedback. This generalization shares some techni-cal analogies with the Sagawa-Ueda extension [3] of theJarzynski relation [19] to feedback control for transitionsbetween equilibrium states. Our work, however, applies tofundamentally different physical systems, such as artificial

PRL 108, 030601 (2012) P HY S I CA L R EV I EW LE T T E R Sweek ending

20 JANUARY 2012

0031-9007=12=108(3)=030601(5) 030601-1 � 2012 American Physical Society

or biological motors driven by unbalanced chemical reac-tions, or basically any other transport problem, reaching aNESS at constant control parameter.

As the second main result, for repeated measurements atregular intervals, Eq. (3) will be shown to imply

_W out � _W in � _Qhk þ _I (4)

as a bound for the power (which, as any rate, will bedenoted by a dot) that can be extracted under isothermalconditions from such a feedback-driven steady-state ma-chine. Here _W in is the cost spent in driving the system.For a large enough rate of acquiring information, i.e., for_I > _Qhk, a net gain becomes possible, since the extractedpower may then exceed the cost of driving.

Model system.—We first illustrate this inequality for aparadigmatic example for which all relevant quantities canbe calculated analytically. We consider a system which hastwo internal states labeled 1 and 2 that alternate in a spatialdirection. The NESS is generated by a nonvanishing forcef > 0 that drives the system in a cyclic fashion from state 1to 2 to 1 to 2 and so on. An example of such a system isgiven in Fig. 1, which is a simplified model for a rotarymotor protein like the F1-ATPase [15,16]. For each clock-wise transition determined by the rateswþ

1 andwþ2 , driving

the system costs the work f (setting for simplicity thedistance between the states symmetrically to 1), leadingto a nonzero current. Likewise, the same work is recoveredfor any thermally excited step in counterclockwise direc-tion with the rates w�

1 and w�2 . Additionally, we assume

that the states 2 have an energy E higher (or, for �E< 0,lower) than the states 1 with energy 0. E will serve as thecontrol parameter �. The rates

wþ1 ¼ w2

0=w�1 ¼ w0 exp½ðf� EÞ=2� (5)

and

wþ2 ¼ w2

0=w�2 ¼ w0 exp½ðfþ EÞ=2� (6)

are chosen such that they obey local detailed balance [20]and become maximally symmetric to facilitate the

analytical solution, w0 being a characteristic frequencyof the system set to 1 in the following. Maintaining thisNESS without any feedback interference is easily calcu-lated to cost

_W in ¼ _Qhk ¼ 2fsinhðf=2ÞcoshðE=2Þ : (7)

The feedback scheme is implemented as follows; seeFig. 2. Perfect measurements at regular intervals separatedby a time tm are supposed to reveal whether the system is ina state 1 or 2. If the system is in 1, we set the energy of state2 to E> 0 at no cost since the level is not occupied. If atthe next measurement the system is still in 1, we donothing. As soon as a subsequent measurement revealsthat the system is in a state 2, we decrease the energy ofthis state to �E< 0, thus extracting the work Wout ¼2E> 0. The next interval then starts with the system againin the lower energy state (which is now state 2). As soon asa measurement reveals that the system has jumped to 1, weraise the energy of the now empty state 2 toþE again at nocost. In the steady state finally reached in this scheme, thesystem is as likely found in 1 as in 2 on average. Theprobability qð�Þ that it will have switched its state at time �after the last measurement obeys the master equation

_qð�Þ ¼ ðwþ1 þ w�

2 Þ½1� qð�Þ� � ðwþ2 þ w�

1 Þqð�Þ; (8)

the solution of which is

qð�Þ ¼ expð�E=2Þ2 coshðE=2Þ ½1� expð���Þ� (9)

with the relaxation rate � � wþ1 þ w�

1 þ wþ2 þ w�

2 .The average power extracted from this machine in this

feedback-driven NESS is

_W outðtmÞ ¼ EqðtmÞ=tm; (10)

since work can be extracted only after a transition from 1 to2. Likewise, the rate with which information is acquired inthis binary measurement scheme becomes

FIG. 1 (color online). Model rotary motor: The driving forcef > 0 induces 60�-clockwise rotations along which the internalstates 1 and 2 alternate. The transition rates wþ;�

1;2 are given by

(5) and (6).

FIG. 2 (color online). Feedback scheme: The energy of theinternal state 2 is changed from E > 0 to �E if the system hasjumped to this state of higher energy, which happens withprobability qðtmÞ. If state 2 is empty, it can be moved to E> 0.

PRL 108, 030601 (2012) P HY S I CA L R EV I EW LE T T E R Sweek ending

20 JANUARY 2012

030601-2

_IðtmÞ¼�½qlnqþð1�qÞlnð1�qÞ�=tm> _WoutðtmÞ (11)

with q � qðtmÞ.Both rates are shown in Fig. 3 as a function of the time tm

between two measurements and for the energy E�ðtm; fÞwhich at fixed tm and f maximizes _Wout. For tm� � 1, thesystem hardly has had a chance to reach the higher level.Still, the optimal extracted power _Wout ¼ ð4=eÞ coshðf=2Þ,using E�ð0; fÞ ¼ 2, is the largest in this limit, in which therate of acquiring information diverges. For tm� � 1, both_Wout and _I tend to 0 and the optimal extracted power isobtained for E�ð1; fÞ ’ 1:278.

The process with feedback requires less power for driv-ing than the one without feedback as given in (7). Theformer’s power can be determined from the average in-stantaneous cost of driving (counting the time � from thelast measurement), which is

_W inð�Þ ¼ ffqð�Þðwþ2 � w�

1 Þ þ ½1� qð�Þ�ðwþ1 � w�

2 Þg:(12)

The time-averaged rate

_W inðtmÞ �Z tm

0

_W inð�Þd�=tm ¼ _QhkðtmÞ (13)

increases with tm as shown in Fig. 3 and is equal to Eq. (7)for tm ! 1 since the system has then reached its steadystate.

In the limit tm� � 1, the extracted power exceeds thecost of driving for E> f tanhðf=2Þ. Such an overall posi-tive performance can persist up to a time tmðE; f; �Þ but, ofcourse, not for arbitrary long intervals between measure-ments, since the gain per measurement is bounded whereasthe total cost of driving the system scales linearly withtime.

Proof.—The general results, Eqs. (3) and (4), will nowbe proven by assuming a Markovian dynamics on a set ofdiscrete states n. At time �, the system is in a state nð�Þ,jumping at discrete times �j from state n�j to state nþj . Atransition between state m and state n occurs with a ratewmnð�jÞ, which depends on the instantaneous value �j ��ð�jÞ of a control parameter �ð�Þ. The solution of the

master equation

@�pðn; �Þ ¼Xm

½wmnð�Þpðm; �Þ � wnmð�Þpðn; �Þ� (14)

for the probability pðn; �Þ to find the system in state n attime � must be distinguished from the stationary solutionpsnð�Þ corresponding to the instantaneous value of the

control parameter.Integrated along a trajectory 0 � � � t, the total entropy

balance reads [14,21]

�stot ¼ �ssys þ �shk þ �sex: (15)

Here, �ssys � � lnpðnt; tÞ þ lnpðn0; 0Þ is the change insystem entropy with nð0Þ � n0 and nðtÞ � nt. The entropychange associated with the heat bath is the sum of house-keeping entropy �shk and excess entropy

�sex ¼Z t

0d�

Xj

�ð�� �jÞ ln½psnþjð�jÞ=ps

n�jð�jÞ�; (16)

where the sum runs over all jumps in the correspondingtime interval of length t.For processes without feedback, this excess entropy

obeys an integral fluctuation theorem

�gðntÞp0ðn0Þ expð��sexÞ

�¼ 1 (17)

valid for any initial condition p0ðnÞ and any normalizedfunction gðnÞ evaluated at the end point [14,21]. By choos-ing for p0 and g the stationary distribution correspondingto the initial and final value of the control parameter �,respectively, this relation becomes precisely the Hatano-Sasa relation [14]. For the following proof, we will,however, keep p0ðnÞ as an arbitrary initial condition. Wechoose gðnÞ ¼ �n;m, which leads, by using Bayes’ theo-

rem, to the conditioned average

hp�10 ðn0Þ expð��sexÞjnt ¼ mipðm; tÞ ¼ 1 (18)

valid for any m from which we get the equivalentXn0

hexpð��sexÞjn0; nt ¼ mipðm; tÞ ¼ 1 (19)

by conditioning the average also on the initial state andthus canceling the term p�1

0 ðn0Þ.For the process with feedback, we first consider the case

of two measurements at time t1 and t2 with outcomes y1and y2, respectively. The trajectory-dependent informationacquired with the first measurement is [3–6]

FIG. 3 (color online). Extracted power _Wout (solid lines) as afunction of the time tm between two measurements, maximizedwith respect to E (see the inset) for the values f ¼ 1 (black line)and f ¼ 2 (light blue line) of the driving force. The correspond-ing driving power _W in (dash-dotted line) and the rate of acquir-ing information _I (dashed line) are also represented. The twovertical dotted lines show the time tm above which the drivingrequires more power than can be extracted, i.e., _WoutðtmÞ ¼_W inðtmÞ.

PRL 108, 030601 (2012) P HY S I CA L R EV I EW LE T T E R Sweek ending

20 JANUARY 2012

030601-3

I 1 ¼ ln½p1ðn1; t1jy1Þ=pðn1; t1Þ�; (20)

where p1ðn1; t1jy1Þ is the probability to find the system instate n1 � nðt1Þ given the measured value y1. This functionnow serves as the initial condition for the further timeevolution of p1ðn; �jy1Þ, which obeys the master equation(14) with a protocol �ð�Þ that depends on y1. The secondmeasurement yields the information

I 2 ¼ ln½p2ðn2; t2jy2; y1Þ=p1ðn2; t2jy1Þ� (21)

with the analogous definition of p2ðn2; t2jy2; y1Þ andnðt2Þ � n2. In this process with feedback, the total changein system entropy becomes

�ssys ¼ � lnp2ðnt; tjy2; y1Þ þ lnp0ðn0Þ: (22)

The total change of excess entropy can be split into threecontributions from the three intervals i ¼ f0 � � < t1g,ii ¼ ft1 � � < t2g, and iii ¼ ft2 � � � tg as

�sex ¼ �sexi þ�sexii þ �sexiii : (23)

With I ¼ I1 þ I2 and by combining Eqs. (15) and (20)–(23), the average on the left-hand side of Eq. (3) stillconditioned on the result y2, y1 can be written as

�1

p0ðn0Þ e��sex

ipðn1; t1Þ

p1ðn1; t1jy1Þ e��sex

iip1ðn2; t2jy1Þ

p2ðn2; t2jy2; y1Þ e��sex

iiip2ðnt; tjy2; y1Þ�

¼ Xm1;m2

�1

p0ðn0Þ e��sex

i jn1 ¼ m1

�ipðm1; t1Þ

|fflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl{zfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl}�

�e��sex

ii jn1 ¼ m1; n2 ¼ m2

�iip1ðm2; t2jy1Þ|fflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl{zfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl}

�

�e��sex

iiip2ðnt; tjy2; y1Þjn2 ¼ m2

�iii|fflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl{zfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl}

�

¼ 1; (24)

where we have introduced conditioned averages on thethree intervals i, ii, and iii (indicated by subscripts)which eliminate the explicit factors p�1

1 ðn1; t1jy1Þ andp�12 ðn2; t2jy2; y1Þ. The term denoted by � is 1 for any m1

due to Eq. (18). Likewise, the subsequent summation ofoverm1 is 1 due to Eq. (19) as is the remaining sum of overm2. It should be obvious that each further measurementwill lead to another summation of � type, and henceEq. (3) is proven for any number N of measurements.This proof shows that Eq. (3) holds true even as an averageconditioned on a fixed record fy�; 1 � � � Ng. In prac-tice, a further average over all possible results fy�g willtypically be used as we also will for the remainingdiscussion.

Applying Jensen’s inequality to Eq. (3) leads to

��Stot � �Qhk þ I (25)

for the ensemble averages. For isothermal processes, wenow invoke the first law

W in �Wout ¼ �EþQhk þQex ¼ �Eþ Stot � �Ssys:

(26)

The splitting of the total work W into a driving one and anextracted one depends on the specific system and is notfixed by these thermodynamic considerations. If N � 1measurements are repeated at times separated by tm, in-serting (26) in (25) and then dividing by Ntm leads to thebound (4) since both �E and �Ssys are finite boundaryterms which vanish in the limit of a steady-state operation.As an aside, we note that the same reasoning applies if

we start with the valid modification of Eq. (17), where�sex

is replaced by �sex þ�shk [21]. As a result, one obtains aconcise proof of the Sagawa-Ueda equality [3]hexp½�ð�stot þ IÞ�i ¼ 1, which is thus shown to holdtrue even for feedback-driven transitions between thegenuine nonequilibrium states considered here. The result-ing bound (2) is, of course, much weaker than (25) sincethe housekeeping heat is extensive in time.Concluding perspective.—The equality (3) and the re-

sulting inequalities (4) and (25) generalize and sharpenprevious results for feedback-driven systems with time-dependent Hamiltonians or potentials to transitions be-tween genuine nonequilibrium states. Searching for theoptimal protocol following a measurement is an issuethat so far has primarily been asked for the former systems.It should now be investigated within this much wider classencompassing natural and artificial molecular machinesdriven by (or delivering output from) biochemical reac-tions. Our example of a simple driven two-state ‘‘informa-tion machine’’ has shown that its power becomes maximalif the time between measurements is the smallest and that itcan even exceed the cost of driving. We suspect that bothstatements have a broad range of validity, but for exploringits limits more analytical and numerical work will benecessary. Finally, the biggest challenge might be tocome up with a first experimental demonstration of theserelations. Whether the paradigmatic colloidal particledriven along a periodic potential or a molecular motor ismore suitable for such an experiment remains to be seen.

[1] R. Kawai, J. M. R. Parrondo, and C. Van denBroeck, Phys.Rev. Lett. 98, 080602 (2007).

[2] F. J. Cao and M. Feito, Phys. Rev. E 79, 041118 (2009).[3] T. Sagawa and M. Ueda, Phys. Rev. Lett. 104, 090602

(2010).[4] J.M. Horowitz and S. Vaikuntanathan, Phys. Rev. E 82,

061120 (2010).[5] M. Ponmurugan, Phys. Rev. E 82, 031129 (2010).

PRL 108, 030601 (2012) P HY S I CA L R EV I EW LE T T E R Sweek ending

20 JANUARY 2012

030601-4

[6] T. Sagawa and M. Ueda, arXiv:1105.3262.[7] S. Toyabe, T. Sagawa, M. Ueda, E. Muneyuki, and M.

Sano, Nature Phys. 6, 988 (2010).[8] R. Marathe and J.M. R. Parrondo, Phys. Rev. Lett. 104,

245704 (2010).[9] S. Vaikuntanathan and C. Jarzynski, Phys. Rev. E 83,

061120 (2011).[10] K.-H. Kim and H. Qian, Phys. Rev. Lett. 93, 120602

(2004).[11] Y. Fujitani and H. Suzuki, J. Phys. Soc. Jpn. 79, 104003

(2010).[12] J.M. Horowitz and J.M. R. Parrondo, Europhys. Lett. 95,

10 005 (2011).[13] D. Abreu and U. Seifert, Europhys. Lett. 94, 10 001 (2011).

[14] T. Hatano and S. I. Sasa, Phys. Rev. Lett. 86, 3463 (2001).[15] S. Toyabe, T. Okamoto, T. Watanabe-Nakayama, H.

Taketani, S. Kudo, and E. Muneyuki, Phys. Rev. Lett.104, 198103 (2010).

[16] K. Hayashi, H. Ueno, R. Iino, and H. Noji, Phys. Rev. Lett.104, 218103 (2010).

[17] V. Blickle, T. Speck, C. Lutz, U. Seifert, and C. Bechinger,Phys. Rev. Lett. 98, 210601 (2007).

[18] J. R. Gomez-Solano, A. Petrosyan, S. Ciliberto, R.Chetrite, and K. Gawedzki, Phys. Rev. Lett. 103, 040601(2009).

[19] C. Jarzynski, Phys. Rev. Lett. 78, 2690 (1997).[20] U. Seifert, Phys. Rev. Lett. 104, 138101 (2010).[21] U. Seifert, Phys. Rev. Lett. 95, 040602 (2005).

PRL 108, 030601 (2012) P HY S I CA L R EV I EW LE T T E R Sweek ending

20 JANUARY 2012

030601-5