Stochastic Hybrid Systems: Modeling, analysis, and applications to networks and biology João P....

Stochastic Hybrid Systems:Modeling, analysis, and applications to

networks and biology

João P. Hespanha

Center for Control Engineeringand Computation

University of Californiaat Santa Barbara

research supported by NSF

Talk outline

1. A model for stochastic hybrid systems (SHSs)

2. Examples:

• network traffic under TCP

• networked control systems

3. Analysis tools for SHSs

• Lyapunov

• moment dynamics

4. More examples …

Collaborators:Stephan Bohacek (U Del), Katia Obraczka (UCSC), Junsoo Lee (Sookmyung Univ.), Mustafa Khammash (UCSB)

Students:Abhyudai Singh (UCSB), Yonggang Xu (UCSB)

Disclaimer: This is an overview, details in papers referenced…

Deterministic Hybrid Systems

guardconditions

reset-maps

continuousdynamics

q(t) 2 Q={1,2,…} ´ discrete state x(t) 2 Rn ´ continuous state

right-continuousby convention

we assume here a deterministic system so the invariant sets would be the exact complements of the guards

Stochastic Hybrid Systems

reset-maps

continuousdynamics

transition intensities(probability of transition in interval (t, t+dt])

Continuous dynamics:

Reset-maps (one per transition intensity):

Transition intensities:

q(t) 2 Q={1,2,…} ´ discrete state x(t) 2 Rn ´ continuous state

Stochastic Hybrid Systems

reset-maps

continuousdynamics

Special case: When all are constant, transitions are controlled by a continuous-time Markov process

q = 1q = 2

q = 3

specifies q(independently of x)

transition intensities(probability of transition in interval (t, t+dt])

Formal model—Summary

State space: q(t) 2 Q={1,2,…} ´ discrete statex(t) 2 Rn ´ continuous state

Continuous dynamics:

Reset-maps (one per transition intensity):

Transition intensities:

# of transitions

Results:1. [existence] Under appropriate regularity (Lipschitz) assumptions, there exists

a measure “consistent” with the desired SHS behavior2. [simulation] The procedure used to construct the measure is constructive and

allows for efficient generation of Monte Carlo sample paths3. [Markov] The pair ( q(t), x(t) ) 2 Q£ Rn is a (Piecewise-deterministic)

Markov Process (in the sense of M. Davis, 1993)

[HSCC’04]

Stochastic Hybrid Systems with diffusion

reset-maps

stochasticdiff. equation

transition intensities

Continuous dynamics:

Reset-maps (one per transition intensity):

Transition intensities:

w ´ Brownian motion process

Example I: Transmission Control Protocol

server clientnetwork

packets droppedwith probability pdrop

transmitsdata packets

receivesdata packets

congestion control ´ selection of the rate r at which the server transmits packetsfeedback mechanism ´ packets are dropped by the network to indicate congestion

r

Example I: TCP congestion control

server clientnetwork

transmitsdata packets

receivesdata packets

TCP (Reno) congestion control: packet sending rate given bycongestion window (internal state of controller)

round-trip-time (from server to client and back)• initially w is set to 1• until first packet is dropped, w increases exponentially fast (slow-start)• after first packet is dropped, w increases linearly (congestion-avoidance)• each time a drop occurs, w is divided by 2 (multiplicative decrease)

packets droppedwith probability pdrop

congestion control ´ selection of the rate r at which the server transmits packetsfeedback mechanism ´ packets are dropped by the network to indicate congestion

r

Example I: TCP congestion control

per-packetdrop prob.

pckts sentper sec

£ pckts droppedper sec=

# of packetsalready sent

TCP (Reno) congestion control: packet sending rate given bycongestion window (internal state of controller)

round-trip-time (from server to client and back)• initially w is set to 1• until first packet is dropped, w increases exponentially fast (slow-start)• after first packet is dropped, w increases linearly (congestion-avoidance)• each time a drop occurs, w is divided by 2 (multiplicative decrease)

many SHS models for TCP…

long-lived TCP flows(no delays)

on-off TCPflows with delay

long-lived TCP flows with delay

on-off TCP flows(no delays)

[SIGMETRICS’03]

network

Example II: Estimation through network

encoder decoder

white noisedisturbance

xx(t1) x(t2)

process

encoder logic ´ determines when to send measurements to the networkdecoder logic ´ determines how to incorporate received measurements

state-estimator

for simplicity:• full-state available• no measurement noise• no quantization• no transmission delays

network

Stochastic communication logic

encoder decoder

white noisedisturbance

xx(t1) x(t2)

process

encoder logic ´ determines when to send measurements to the network

state-estimator

decoder logic ´ determines how to incorporate received measurements

for simplicity:• full-state available• no measurement noise• no quantization• no transmission delays

network

Stochastic communication logic

encoder decoder

white noisedisturbance

xx(t1) x(t2)

process state-estimator

Error dynamics:

reset error to zero

prob. of sending data in [t,t+dt) depends on current error e

for simplicity:• full-state available• no measurement noise• no quantization• no transmission delays

network

Stochastic communication logic

encoder decoder

white noisedisturbance

xx(t1) x(t2)

with:• measurement noise• quantization• transmission delay

process state-estimator

reset error to nonzero random variables

prob. of sending data in [t,t+dt) depends on current encoder error enec

[ACTRA’04]

error at encoder sidebased on data sent

error at decoder sidebased on data received

Example III: And now for something completely different…

Decaying-dimerizing chemical reactions (DDR):

SHS model population of species S1

population of species S2

reaction rates

S2 0S1 0 2 S1 S2

c1 c2c3

c4

Inspired by Gillespie’s Stochastic Simulation Algorithm for molecular reactions [Gillespie, 76]

Example III: And now for something completely different…

Decaying-dimerizing chemical reactions (DDR):

SHS model population of species S1

population of species S2

reaction rates

S2 0S1 0 2 S1 S2

c1 c2c3

c4

Inspired by Gillespie’s Stochastic Simulation Algorithm for molecular reactions [Gillespie, 76]

Disclaimer: Several other important applications missing. E.g., air traffic control [Lygeros,Prandini,Tomlin] queuing systems [Cassandras] economics [Davis]

Generalizations of the SHS model

1. Stochastic resets can be obtained by considering multiple intensities/reset-maps

One can further generalize this to resets governed by to a continuous distribution [see paper]

Generalizations of the SHS model

2. Deterministic guards can also be emulated by taking limits of SHSs

The solution to the hybrid system with a deterministic guard is obtained as # 0+

# 0+

-1 1

This provides a mechanism to regularize systems with chattering and/or Zeno phenomena…

barrierfunction

g(x)

Example: Bouncing-ball

t

yc 2 (0,1) ´ energy absorbed at impact

Zeno-time

The solution of this deterministic hybrid system is only defined up to the Zeno-time

g

y

g

0 5 10 15-0.2

0

0.2

0.4

0.6

0.8

1

1.2

0 5 10 15-0.2

0

0.2

0.4

0.6

0.8

1

1.2

0 5 10 15-0.2

0

0.2

0.4

0.6

0.8

1

1.2

= 10–2 = 10–3 = 10–4

mean (blue) 95% confidence intervals (red and green)

Example: Bouncing-ball

g

y

g

c 2 (0,1) ´ energy absorbed at impact

Analysis—Lie Derivative

Given scalar-valued function Ã: Rn £ [0,1) ! R

derivativealong solution

to ODE Lf ÃLie derivative of Ã

One can view Lf as an operator

space of scalar functions onRn £ [0,1)

space of scalar functions onRn £ [0,1)

à (x, t) Lf Ã(x, t)

Lf completely defines the system dynamics

Generator of a SHS

continuous dynamicsreset-mapstransition intensities

Given scalar-valued function Ã: Q £ Rn £ [0,1) ! Rgenerator for the SHS

where

Lie derivative

reset term

L completely defines the SHS dynamics

Dynkin’s formula(in differential form)

diffusion term

Disclaimer: see following paper for technical assumptions [HSCC’04]

instantaneous variation

intensity

Given scalar-valued function Ã: Q £ Rn £ [0,1) ! R

where

Generator of the SHS with diffusion

Dynkin’s formula(in differential form)

Attention: These systems may have problems of existence of solution due to jumps!

(stochastic Zeno)

E.g.

no global solutionno local solution

for x(0) > 1

L completely defines the SHS dynamics

“jumping makes jumping more likely”

“probability of multiple jumps in short interval not sufficiently small”

Disclaimer: see following paper for technical assumptions [HSCC’04]

Stochastic communication logics

error dynamicsin remote estimation

Long-lived TCP flows

slow-start congestionavoidance

long-lived TCP flows(with slow start)

Lyapunov-based stability analysis

Expected value of error:

Dynkin’s formula

)

2nd moment of the error:

)

error dynamicsin NCS

Lyapunov-based stability analysis

Expected value of error:

Dynkin’s formula

For constant rate: (e) =

)

2nd moment of the error:

)

assuming (A – /2 I ) Hurwitz

error dynamicsin NCS

Lyapunov-based stability analysis

Dynkin’s formula

One can show…

For constant rate: (e) =

1. E[ e ] ! 0 as long as > <[(A)]2. E[ || e ||m ] bounded as long as > 2 m <[(A)]

For polynomial rates: (e) = (e0 P e)k P > 0, k ¸ 0

1. E[ e ] ! 0 (always)2. E[ || e ||m ] bounded 8 m

both always true 8 ¸ 0 if A Hurwitz(no jumps needed for boundedness)

getting more moments bounded requires

higher jump intensities

Moreover, one can achieve the same E[ || e ||2 ] with a smaller

number of transmissions…

[ACTRA’04, CDC’04]

error dynamicsin NCS

Analysis—Moments for SHS state

z (scalar) random variable with mean and variance 2

Tchebychev inequalityMarkov inequality(8 > 0) Bienaymé inequality

(8 > 0, a2R , n2N)

continuous dynamicsreset-mapstransition intensities

often a few low-order moments suffice to study a SHS…

Polynomial SHSs

A SHS is called a polynomial SHS (pSHS) if its generator maps finite-order polynomial on x into finite-order polynomials on xTypically, when

are all polynomials 8 q, t

continuous dynamicsreset-mapstransition intensities

Given scalar-valued function Ã: Q £ Rn £ [0,1) ! Rgenerator for the SHS

where

Dynkin’s formula(in differential form)

Moment dynamics for pSHS

E.g,

continuous state discrete statex(t) 2 Rn q(t) 2 Q={1,2,…}

Monomial test function: Given

Uncentered moment:

for short x(m)

Moment dynamics for pSHS

continuous state discrete statex(t) 2 Rn q(t) 2 Q={1,2,…}

for short x(m)

Uncentered moment:

)

For polynomial SHS…

) monomial on x polynomial on x linear comb. of

monomial test functions

linear moment dynamics

Monomial test function: Given

Moment dynamics for TCP

long-livedTCP flows

Moment dynamics for TCP

long-livedTCP flows

on-off TCP flows(exp. distr. files)

Truncated moment dynamics

Experimental evidence indicates that the (steady-state)sending rate is well approximated by a Log-Normal distribution

w0 10 20 30 40 50

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

Data from: Bohacek, A stochastic model for TCP and fair video transmission, INFOCOM’03

PDF of the congestion window size w

z Log-Normal

r Log-Normal (on each mode)

)

)

Moment dynamics for TCP

finite-dimensionalnonlinear ODEs

long-livedTCP flows

on-off TCP flows(exp. distr. files)

Long-lived TCP flows

0 1 2 3 4 5 6 7 8 9 10

0.02

0.1

0 1 2 3 4 5 6 7 8 9 100

100

200

sending rate mean

0 1 2 3 4 5 6 7 8 9 100

50

100

sending rate std. deviation

Monte Carlo (with 99% conf. int.) reduced model

moment dynamics in response to abrupt change in drop probability

long-lived TCP flows with delay

(one RTT average)

Truncated moment dynamics (revisited)

For polynomial SHS…

linear moment dynamics

Stacking all moments into an (infinite) vector 1

infinite-dimensional linear ODEIn TCP analysis…

lower ordermoments of interest

moments of interestthat affect dynamics

approximated bynonlinear function of

Truncation by derivative matchinginfinite-dimensional linear ODE

truncated linear ODE(nonautonomous, not nec. stable)

nonlinear approximatemoment dynamics

Assumption: 1) and remain bounded along solutions to

and

2) is (incrementally) asymptotically stable

class KL function

Theorem: 8 > 0 9 N s.t. if

then

Disclaimer: Just a lose statement. The “real” theorem is stated in [HSCC'05]

Truncation by derivative matchinginfinite-dimensional linear ODE

truncated linear ODE(nonautonomous, not nec. stable)

nonlinear approximatemoment dynamics

Proof idea:1) N derivative matches ) & match on compact interval of length T2) stability of A1 ) matching can be extended to [0,1)

Assumption: 1) and remain bounded along solutions to

and

2) is (incrementally) asymptotically stable

class KL function

Theorem: 8 > 0 9 N s.t. if

then

Assumption: 1) and remain bounded along solutions to

and

2) is (incrementally) asymptotically stable

class KL function

Theorem: 8 > 0 9 N s.t. if

then

Truncation by derivative matchinginfinite-dimensional linear ODE

truncated linear ODE(nonautonomous, not nec. stable)

nonlinear approximatemoment dynamics

Proof idea:1) N derivative matches ) & match on compact interval of length T2) stability of A1 ) matching can be extended to [0,1)

Given , finding N is very difficult

In practice, small values of N (e.g., N = 2) already yield good results

Can use

to determine (¢): k = 1 ! boundary condition on

k = 2 ! linear PDE on

Moment dynamics for DDR

Decaying-dimerizing molecular reactions (DDR):

S2 0S1 0 2 S1 S2

c1 c2c3

c4

Truncated DDR model

by matching

for deterministic distributions

[IJRC, 2005]

Decaying-dimerizing molecular reactions (DDR):

S2 0S1 0 2 S1 S2

c1 c2c3

c4

Monte Carlo vs. truncated model

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

x 10-3

0

10

20

30

40populations means

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

x 10-3

0

1

2

3

4populations standard deviations

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

x 10-3

-1.1

-1.05

-1

-0.95

-0.9

populations correlation coefficient

E[x1]

E[x2]

Std[x1]

Std[x2]

x2,x2](lines essentially

undistinguishable at this scale)

Fast time-scale(transient)

Parameters from: Rathinam, Petzold, Cao, Gillespie, Stiffness in stochastic chemically reacting systems: The implicit tau-leaping method. J. of Chemical Physics, 2003

Monte Carlo vs. truncated model

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

10

20

30

40

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

1

2

3

4

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

-1

-0.5

0

0.5

populations means

populations standard deviations

populations correlation coefficient

E[x1]

E[x2] Slow time-scaleevolution

Std[x1]

Std[x2]

x2,x2]

error only noticeable when populations become very small

(a couple of molecules,still adequate to study cellular

reactions)

Parameters from: Rathinam, Petzold, Cao, Gillespie, Stiffness in stochastic chemically reacting systems: The implicit tau-leaping method. J. of Chemical Physics, 2003

Conclusions

1. A simple SHS model (inspired by piecewise deterministic Markov Processes) can go a long way in modeling network traffic

2. The analysis of SHSs is generally difficult but there are tools available(generator, Lyapunov methods, moment dynamics, truncations)

3. This type of SHSs (and tools) finds use in several areas(traffic modeling, networked control systems, molecular biology, population dynamics in ecosystems)

Bibliography

• M. Davis. Markov Models & Optimization. Chapman & Hall/CRC, 1993

• S. Bohacek, J. Hespanha, J. Lee, K. Obraczka. Analysis of a TCP hybrid model. In Proc. of the 39th Annual Allerton Conf. on Comm., Contr., and Computing, Oct. 2001.

• S. Bohacek, J. Hespanha, J. Lee, K. Obraczka. A Hybrid Systems Modeling Framework for Fast and Accurate Simulation of Data Communication Networks. In Proc. of the ACM Int. Conf. on Measurements and Modeling of Computer Systems (SIGMETRICS), June 2003.

• J. Hespanha. Stochastic Hybrid Systems: Applications to Communication Networks. In R. Alur, G. Pappas, Hybrid Systems: Computation and Control, LNCS 1993, pp. 387-401, 2004.

• João Hespanha. Polynomial Stochastic Hybrid Systems. In Manfred Morari, Lothar Thiele, Hybrid Systems: Computation and Control, LNCS 3414, pp. 322-338, Mar. 2005.

• J. Hespanha, A. Singh. Stochastic Models for Chemically Reacting Systems Using Polynomial Stochastic Hybrid Systems. To appear in the Int. J. on Robust Control Special Issue on Control at Small Scales, 2005.

• A. Singh, J. Hespanha. Moment Closure Techniques for Stochastic Models in Population Biology. Oct. 2005. Submitted to publication.

• Y. Xu, J. Hespanha. Communication Logic Design and Analysis for NCSs. ACTRA’04.

• Y. Xu, J. Hespanha. Optimal Communication Logics for NCSs. In Proc. CDC, 2004.

All papers (and some ppt presentations) available athttp://www.ece.ucsb.edu/~hespanha

END

TCP window-based control

1st packet sent

e.g., w = 3

t

2nd packet sent3rd packet sent 1st packet received & ack. sent

2nd packet received & ack. sent3rd packet received & ack. sent1st ack. received )

4th packet can be sent

t

source f destination f

w effectively determines the sending rate r:

round-trip time

t0

t1

t2

t3

0

1

2

w (window size) ´ number of packets that can remain unacknowledged for by the destination

Built-in negative feedback:

r increases congestion

RTT increases (queuing delays)

r decreases

TCP Reno congestion avoidance

1. While there are no drops, increase w by 1 on each RTT (additive increase)

2. When a drop occurs, divide w by 2 (multiplicative decrease)

(congestion controller constantly probes the network for more bandwidth)

additiveincrease

multiplicative decrease

per-packetdrop prob.

pckts sentper sec

£pckts dropped

per sec=

total # of packetsalready sent

Three feedback mechanisms: As rate r increases ) …a) congestion ) RTT increases (queuing delays) ) r decreasesb) congestion ) pdrop increases (active queuing) ) r decreasesc) multiplicative decrease more likely ) r decreases

TCP Reno slow start

3. Until a drop occurs double w on each RTT4. When a drop occurs divide w by 2 and transition to congestion avoidance

(get to full bandwidth utilization fast)

TCP has several other modes that can be modeled by hybrid systems[Bohacek, Hespanha, Lee, Obraczka. A Hybrid Systems Modeling Framework for Fast and Accurate Simulation of Data Communication Networks. In SIGMETRICS’03]

# of packetsalready sent

Long-lived TCP flows

long-livedTCP flows

On-off TCP model

• Between transfers, server remains off for an exponentially distributed time with mean off

• Transfer-size is a random variable with cumulative distribution F(s)

determines duration of on-periods

On-off TCP model

• Between transfers, server remains off for an exponentially distributed time with mean off

• Transfer-size is exponentially distributed with average k (packets)

determines duration of on-periodssimple but not very realistic…

Analysis—Generator of the SHS

continuous dynamicsreset-mapstransition intensities

Given scalar-valued function Ã: Q £ Rn £ [0,1) ! Rgenerator for the SHS

where

Lf ÃLie derivative of Ã

reset instantaneousvariation

intensity

L completely defines the SHS dynamics

Disclaimer: see following paper for technical assumptionsHespanha. Stochastic Hybrid Systems: Applications to Communication Networks. HSCC’04

Dynkin’s formula(in differential form)

Analysis—PDF for SHS state

functional PDEvery difficult to solve

When ( ¢ ; t) is smooth, one can deduce from the generator that

continuous dynamicsreset-mapstransition intensities

Let ( ¢ ; t) be the probability density function (PDF) for the state (x,q) at time t :

inverse of

Long-lived TCP flows

Monte-Carlo & theoreticalsteady-state distribution

(vs. drop probability)

long-lived TCP flows with delay

(one RTT average)

On-off TCP flows

Transfer-sizes exponentially distributed with mean k = 30.6KB (Unix’93 files)Off-time exponentially distributed with mean off = 1 secRound-trip-time RTT = 50msec

SHS for on-off TCP flows

On-off TCP flows

Monte Carlo (solid) reduced model (dashed)

0 0.5 1 1.5 2 2.5 3

0.02

0.1

0 0.5 1 1.5 2 2.5 30

0.05

0.1

0.15

0 0.5 1 1.5 2 2.5 30

10

20

0 0.5 1 1.5 2 2.5 30

20

40

60

sending rate mean

sending rate std. dev.

probabilities

ss

ca

ss

ca

total

ssca

total