TMPA-2017: The Quest for Average Response Time

The Quest for Average Response Time

Tom Henzinger

IST Austria

Joint work with Krishnendu Chatterjee and Jan Otop

Yes/No

Program Analysis

PropertyProgram

Formal Verification

Yes/No

Model Checker

Transition

System (r ) } g)PropertyProgram

Formal Verification

Yes/No

Model Checker

Timed

Automaton (r ) }· 5 g)

Quantitative

PropertyProgram

Formal Verification

Yes/No

Model Checker

Markov

Process

Quantitative

PropertyProgram 8 (r ) Pr(}g) ¸ 0.5)

Formal Verification

Model Checker

Timed

Automaton Program (r ) } g)Property

Quantitative Answer R

Worst or average response time

Quantitative Analysis


From checking correctness

to measuring performance and robustness of software systems:

Quantitative temporal logics

Quantitative automata

Quantitative abstractions

Quantitative synthesis

etc.


From checking correctness

to measuring performance and robustness of software systems:

Quantitative temporal logics

Quantitative automata

Quantitative abstractions

Quantitative synthesis

etc.

None of this has captured average response time.

Observations

r request

g grant

t tick

x neither

§ = {r,g,t,x}

Program Behavior = Observation Sequence

x t t x x r x x t x x x r x t x t x x t g r t t g g x t x t x …

Response Times

x t t x x r x x t x x x r x t x t x x t g r t t g g x t x t x … 4,3,2

Response Times

x t t x x r x x t x x x r x t x t x x t g r t t g g x t x t x … 4,3,2

r t t t t t t t … 1

Response Property

r

g

r,t,xg,t,x

Response Monitor

rr,t,xg,t,x

S

S

g

Response Monitor

rr,t,xg,t,x

S

S

g

Decomposing model checking [Pnueli et al.]

Alternating automata

Run-time verification

Bounded Response

r

gr,x

g,t,x

C := 0

C · 3

tC := C+1

Bounded Response

r

gr,x

g,t,x

C := 0

C · 3

tC := C+1

gC > 3

(Discrete) clocks exponentially succinct,

but not more expressive than finite state.

Bounded Response Monitor

r r,xg,t,x

S

S

g

tC := C+1

C := 0

C · 3

Maximal Response

r

gr,x

g,t,x

C := 0

V := max(V,C)

tC := C+1

V := 0

Maximal Response

r

gr,x

g,t,x

C := 0

V := max(V,C)

tC := C+1

V := 0

Value of an infinite run is liminf of V.

Maximal Response Monitor

r r,xg,t,x

S

S

g

tV := V+1

V := 0V := 0

V := max(V, )

is final value of V.S

Average Response

r

gr,x

g,t,x

C := 0

N := N+1

V := avg(V,C,N)

tC := C+1

V := 0

N := 0

avg(V,C,N) = (V¢(N-1)+C) / N

Average Response Monitor

r r,xg,t,x

S

S

g

tV := V+1

V := 0

V := avg(V, ,N)

N := N+1

V := 0

N := 0

(max,inc) automata:

Master automaton maintains the max of values

returned by slaves (1 max register).

Each slave automaton counts occurrences of t

(1 inc register).

(avg,inc) automata:

Master automaton maintains the avg of values

returned by slaves (1 avg register).

Slaves as above.

Nested Weighted Automata

r r,x

0g,t,x

0S

g

0

t1

S

limavg of weights

sum of weights

Function on weights instead of registers.

Unlike in the qualitative case,

nested weighted automata (“quantitative monitors”) are

more expressive than flat weighted automata:

1. value of flat limavg automaton bounded by largest weight

(cannot specify average response time)

2. flat automata have constant “width” (number of registers)

Deterministic qualitative automaton A: §! ! B

Deterministic quantitative automaton A: §! ! R

Deterministic qualitative automaton A: §! ! B

Deterministic quantitative automaton A: §! ! R

! = x t t x x r x x t x x x r x t x t x x t g r t t g g x t x t x …

Response(!) = 1

BoundedResponse(!) = 0

MaximalResponse(!) = 4

AverageResponse(!) = 3

43 2

tr,g,t,x

Nondeterministic Automaton

t

t t t t t t t t t … values {0, 1}

Nondeterministic qualitative automaton A: §! ! B

A(!) = max{ value(½) | ½ run of A and obs(½) = ! }

Nondeterministic quantitative automaton A: §! ! R

A(!) = sup{ value(½) | ½ run of A and obs(½) = ! }



Emptiness: 9!. A(!) = 1Universality: 8!. A(!) = 1





Emptiness: 9!. A(!) = 1Universality: 8!. A(!) = 1



Emptiness: 9!. A(!) ¸ ¸

Universality: 8!. A(!) ¸ ¸

Transition System = Labeled Graph

r

r

t

x

x

x

t

g

g

g

t

t

t

t

x

x

t

Defines a set of behaviors.

Qualitative Analysis

Given a transition system A and a qualitative property B,

Q1. does some run of A correspond to a run of B ?[emptiness of A £ B ]

Q2. does every run of A correspond to a run of B ?

[as hard as universality of B ]


Given a transition system A and a quantitative property B,

Q1. does some run of A correspond to a run of B with value V ¸ ¸ ?

[emptiness of A £ B ]

Q2. does every run of A correspond to a run of B with V ¸ ¸ ?

[as hard as universality of B ]




Q2. does every run of A correspond to a run of B ?Equivalently: does some run of A correspond to a run of :B ?

[emptiness of A £ :B ]




Q2. does every run of A correspond to a run of B ?Equivalently: does some run of A correspond to a run of :B ?

[emptiness of A £ :B ]

For deterministic B, the complement :B is easy to compute.



Monitor: obs(½1) = obs(½2) ) value(½1) = value(½2)

Deterministic automata are monitors.


Given a transition system A and a quantitative property B,

Q1. does some run of A correspond to a run of B with value V ¸ ¸ ?

[emptiness of A £ B ]

Q2. does every run of A correspond to a run of B with V ¸ ¸ ?

For monitor B, equivalently: does some run of A correspond to a run of B with V < ¸ ?

[emptiness of A £ -B ]

Example

r r

t

t

t

t

t

r

g

g

g

t

t

t

Example

r r

t

t

t

t

t

r

g

g

g

t

t

t

Best maximal response time: 2

Worst maximal response time: 3

Emptiness of (max,inc) automata

Example

r r

t

t

t

t

t

r

g

g

g

t

t

t

Best maximal response time: 2

Worst maximal response time: 3

Emptiness of (max,inc) automata

Best average response time: 1.5

Worst average response time: 3

Emptiness of (avg,inc) automata

Results on (max,inc) Automata

Nondet Monitor

(max,inc) (max,inc)

Emptiness PSPACE PSPACE

Universality · EXPSPACE PSPACE

¸ PSPACE

Qualitative nondeterministic universality PSPACE,

emptiness and deterministic universality PTIME.

Results on (avg,inc) Automata

Nondet Monitor

(avg,inc) (avg,inc)

Emptiness · EXPSPACE · EXPSPACE

¸ PSPACE ¸ PSPACE

Universality undecidable · EXPSPACE

¸ PSPACE


Nondet Monitor Monitor Monitor

(avg,inc) (avg,inc) bounded width constant width

(avg,inc) (avg,inc)

Emptiness · EXPSPACE · EXPSPACE PSPACE PTIME

¸ PSPACE ¸ PSPACE

Universality undecidable · EXPSPACE PSPACE PTIME

¸ PSPACE

How many overlapping requests?

Probabilistic System = Markov Chain

r

r

t

x

x

x

t

g

g

g

t

t

t

t

x

x

t

0.5 0.3

0.2

0.5

0.5

Defines probability for every

finite observation sequence,

and prob density function on

infinite observation sequences.

0.9

0.1

Probabilistic System = Markov Chain

r

r

t

x

x

x

t

g

g

g

t

t

t

t

x

x

t

0.5 0.3

0.2

0.5

0.5

Defines probability for every

finite observation sequence,

and prob density function on

infinite observation sequences.

0.9

0.1

Given prob density function on §!,

monitor specifies random variable V.

Probabilistic Analysis

Given a probabilistic system A and a functional quantitative property B,

Q1. compute the expected value of V on the runs of A £ B

[moment analysis]

Q2. compute the probability of V ¸ ¸ on the runs of A £ B

[distribution analysis]

Probabilistic Example

r r

t

t

t

t

t

r

g

g

g

t

t

t

0.5 0.5


r r

t

t

t

t

t

r

g

g

g

t

t

t

Expected maximal response time: 2.5

Prob of maximal response time at most 2: 0.5

Probabilistic analysis of (max,inc) automata

0.5 0.5


r r

t

t

t

t

t

r

g

g

g

t

t

t

Expected maximal response time: 2.5

Prob of maximal response time at most 2: 0.5

Probabilistic analysis of (max,inc) automata

Expected average response time: 2.25

Prob of average response time at most 2: 0.5

Probabilistic analysis of (avg,inc) automata

0.5 0.5

Results on (max,inc) Automata

Nondet Monitor

(max,inc) (max,inc)


Universality · EXPSPACE PSPACE

¸ PSPACE

Expectation · EXPSPACE

¸ PSPACE

Probability · EXPSPACE

¸ PSPACE


Nondet Monitor

(avg,inc) (avg,inc)

Emptiness · EXPSPACE · EXPSPACE

¸ PSPACE ¸ PSPACE

Universality undecidable · EXPSPACE

¸ PSPACE

Expectation PTIME

Probability PTIME

Markov Decision Process

r

r

t

x

x

x

t

g

g

g

t

t

t

t

x

x

t

0.5

0.5

0.9

0.1

Markov Decision Process

r

r

t

x

x

x

t

g

g

g

t

t

t

t

x

x

t

0.5

0.5

0.9

0.1

Given a policy p:§!!{ $,$,$ },

monitor specifies random

variable V.

Many Open Questions …

E.g., given an MDP A and a monitor B,

compute the policy that maximizes the expected value of B on A

(generalization of mean-payoff game).

r t r t r t t t g t g t g

5,5,5

Matching Requests and Grants

r t r t r t t t g t g t g

5,5,5

7,5,3

Matching Requests and Grants

Quantitative pushdown monitors?

Counter Machine

r

x

g,t,xS

S

g

C = 0

tV := V+1

V := 0

C := 0V := 0

V := max(V, )

rC := C+1

g

C > 0C := C-1

Emptiness for two counters is in general undecidable.

Counter Monitor

t

x

r,g,xS

S

t

V := 0V := 0

V := max(V, )

rV := V+1

gV := V-1

No test on counter values.

Counter Monitor

t

x

r,g,xS

S

t

V := 0V := 0

V := max(V, )

rV := V+1

gV := V-1

No test on counter values. width = 1

Register Automaton

x

V := 0

C := 0

rC := C+1

gC := C-1

Emptiness of (max,inc+dec) decidable [flat, constant width: Alur et al.].

tV := max(V,C)

C := 0

Results on (max,inc+dec) Automata

Nondet Monitor

(max,inc+dec) (max,inc+dec)


Universality undecidable undecidable

Expectation undecidable

Probability undecidable

Results on (avg,inc+dec) Automata

Nondet Monitor

(avg,inc+dec) (avg,inc+dec)

Emptiness open open

Universality undecidable open

Expectation PTIME

Probability PTIME

Quantitative Monitors = Nested Weighted Automata

Unbounded width allows for natural decomposition of specifications

(incl. average response time).

More expressive and more succinct than flat weighted automata.

Emptiness decidable and sufficient for

monitor verification, model measuring, and model repair

(universality can be undecidable, even for constant width).

Probabilistic analysis polynomial for (avg,inc+dec) monitors.

Model Measuring:How much can system A be perturbed without violating qualitative property B ?

Model Repair:How much must system A be changed to satisfy qualitative property B ?

Model Measuring:How much can system A be perturbed without violating qualitative property B ?

For an observation sequence ! we can define

distance d(A,!) (e.g. edit distance) by constructing from A

monitor FA such that FA(!) = d(A,!).

Robustness of A with respect to B:r(A,B) = sup{ e | 8!. d(A,!) · e ) B(!) = 1 }.

Model Repair:How much must system A be changed to satisfy qualitative property B ?

References

Nested Weighted Automata: LICS 2015

Quantitative Automata under Probabilistic Semantics: LICS 2016

Nested Weighted Automata of Bounded Width: MFCS 2016

IST (Institute of Science and Technology) Austria looking for PhD students: phd.ist.ac.at