I think your suggestion is, Can we do two things at once? Well, we’re of the view that we can walk...

I think your suggestion is, Can we do two things at once? Well, we’re of the view that

we can walk and chew gum at the same time.

—Richard Armitage, deputy secretary of state, June 2, 2004

To do two things at once is to do neither.

—Publilius Syrus, Roman slave, first century B.C.

Therefore…

Multitask a few…

..let the rest queue.

Computer systems are not different…

Multi-Programming Level(# concurrent tasks/threads)

Efficiency

(useful work per second)

thrashing

Therefore… ServerActivetasks

Tasks not-yet-started

Q: Max number of tasks allowed to share server?Common Soln: K*

AdmissionControl

K*

Presenting

How to Multitask Smartly

4

A Queueing-theoretic abstraction

server

# jobs at server (n)

Totalspeed(n) K*

(known)

5

A Queueing-theoretic abstraction

PSserver

1

2

K

K = MPL

# jobs at server (n)

Totalspeed(n) K*

FIFO buffer

The G/G/PS-MPL model

• General arrival process for jobs• Job sizes i.i.d. ~ X

• Sizes unknown, distribution of X known

GOAL: Find MPL (i.e. K) to minimize mean response time

response time

n

jobs

each job gets rate

(n)n

:

(known)

Optimal MPL= K* ?

0.25

0.5

0.75

1

1.25

0 5 10 15 20 25

# jobs at server (n)

Totalspeed

(jobs/sec) K*=5

Exponential job sizes (C2 = 1)

Weibull job sizes (C2 = 19)

3

4

5

6

7

4 6 8 10 12 14 16

2.75

3

3.25

4 6 8 10 12 14 16

Mea

n re

spon

se ti

me

MPL

MPL

Mea

n re

spon

se ti

me

K*

K*

OPT MPL

OPT MPL

Example

Poisson(0.8) arrival process

45%improvement

Optimal MPL= K* ?

0.25

0.5

0.75

1

1.25

0 5 10 15 20 25

# jobs at server (n)

Totalspeed

(jobs/sec) K*=5

Exponential job sizes (C2 = 1)

Weibull job sizes (C2 = 19)

3

4

5

6

7

4 6 8 10 12 14 16

2.75

3

3.25

4 6 8 10 12 14 16

Mea

n re

spon

se ti

me

MPL

MPL

Mea

n re

spon

se ti

me

K*

K*

OPT MPL

OPT MPL

Example

Poisson(0.8) arrival process

DEPENDS!

45%improvement

8

Intuition for the effect of MPL

FIFO

server

ProcessorSharing (PS)

9

Intuition for the effect of MPL

FIFO

server

ProcessorSharing (PS)

High job-size variability(C2)

High arrival rate

0.25

0.5

0.75

1

1.25

0 5 10 15 20 25

# jobs at server (n)

Totalspeed(n) K*

MPL

10

Intuition for the effect of MPL

server

FIFO ProcessorSharing (PS)

High job-size variability(C2)

High arrival rate

Optimal MPL ↑C2 ↑

Arrival rate ↑ Optimal MPL → K*

MPL

11

Why is finding the optimal MPL hard?

- No analysis even for the case of Poisson arrivals (M/G/PS-MPL)

Hurdle 1:State dependent

service rates

Hurdle 2:Finite Multi-

Programming-Limit

Hurdle 3:General (high C2) job

size distribution

Mean response time only depends on E[X]

[Kelly 79]

Transform of response time known for Exponential job size distribution

[Rege Sengupta 85]

Heavy traffic diffusion approximation

[Zhang Zwart 09]

Our work

12

Where we are…

GOAL: Optimal MPL for M/G/PS-MPL

SUBGOAL: Approximation for M/G/PS-MPL

What if the traffic rate is unknown?What if the traffic is not Poisson?

GOAL #2: Traffic-oblivious dynamic MPL control policies

?

13

Where we are…

GOAL: Optimal MPL for M/G/PS-MPL

SUBGOAL: Approximation for M/G/PS-MPL

What if the traffic rate is unknown?What if the traffic is not Poisson?

GOAL #2: Traffic-oblivious dynamic MPL control policies

?

14

RESULT: M/G/PS-MPL approximation

PSServer(n)

1

2

KFIFO buffer

Job sizes ~ X

Poisson()

Easy for X Exp (C2=1)

15

RESULT: M/G/PS-MPL approximation

• Involves only the first two moments of job size distribution

• Generalizes-Lee and Longton approx. for M/G/K multi-server system: (n)= n·-Zhang and Zwart approx. for the case (n)=

THEOREM: The above approx. is exact for the following degenerate hyper-exponential job size distribution:

- 2 degrees of freedom- Can match any E[X] and C2 ≥ 1

16

Where we are…

GOAL: Optimal MPL for M/G/PS-MPL

SUBGOAL: Approximation for M/G/PS-MPL

What if the traffic rate is unknown?What if the traffic is not Poisson?

GOAL #2: Traffic-oblivious dynamic MPL control policies

?

17

Where we are…

GOAL: Optimal MPL for M/G/PS-MPL

SUBGOAL: Approximation for M/G/PS-MPL

What if the traffic rate is unknown?What if the traffic is not Poisson?

GOAL #2: Traffic-oblivious dynamic MPL control policies

?

18

Static MPL policy for M/G/PS-MPL

Set MPL = MPL*, where:

19

3

4

5

6

7

8

4 6 8 10 12 14 16 18

7

9

11

13

4 6 8 10 12 14 16

• Approx accurately predicts the behavior of the curve as a function of MPL

• MPL* decreases as arrival rate increases

(Poisson(0.8) arrivals)

(Poisson(0.9) arrivals)

0.25

0.5

0.75

1

1.25

0 5 10 15 20 25

# jobs at server (n)

Totalspeed(n)

MPL

MPL

Mea

n Re

sp. T

ime

Mea

n Re

sp. T

ime

Our approx.

Weibull (C2=19)

Weibull (C2=19)

K*

K*

MPL*

MPL*

K* gives 25% worse performance than MPL*

K* gives 45% worse performance than MPL*

Our approx.

20

Where we are…

GOAL: Optimal MPL for M/G/PS-MPL

SUBGOAL: Approximation for M/G/PS-MPL

What if the traffic rate is unknown?What if the traffic is not Poisson?

GOAL #2: Traffic-oblivious dynamic MPL control policies

?

21

Where we are…

GOAL: Optimal MPL for M/G/PS-MPL

SUBGOAL: Approximation for M/G/PS-MPL

What if the traffic rate is unknown?What if the traffic is not Poisson?

GOAL #2: Traffic-oblivious dynamic MPL control policies

?

22

Traffic-oblivious policies

• Straw man proposal 1: Choose a “robust” static MPL– Must choose MPL=K* : but suboptimal in light/moderate traffic

• Straw man proposal 2: Learn the arrival rate– Can’t adapt to changes on small time scale/correlations

WE DEMONSTRATE: Dynamic MPL control policies which are1. Traffic-oblivious: self-adapt to variations in the arrival process 2. Light-weight: make decisions based only on current queue length, Q(t), and current MPL, K(t)

I don’t know the arrival rate!!

My arrivals are not Poisson!!

23

Traffic-oblivious policies(KEY STEPS)

GOAL: A robust MPL-control policy πOPT

STEP 1: Approximate the orig. job size distribution

X ≈ _____

STEP 2: Construct candidate policies

π1 π2 π3 … πN

STEP 3: Pick the “best” candidate policy

24

Traffic-oblivious policies(KEY STEPS)

GOAL: A robust MPL-control policy πOPT

STEP 1: Approximate the orig. job size distribution

X ≈ _____

STEP 2: Construct candidate policies

π1 π2 π3 … πN

STEP 3: Pick the “best” candidate policy

25

GOAL: Approximate the job size distribution by a tractable distribution

IDEA: The H* distribution:

• Models many small vs. few large jobs• Tractable due to Exponential branch

Traffic-oblivious policies(KEY STEPS)

For age/size independent policies, H* provides a good approximation for mean response time

p and γ chosen to match

E[X] and C2

26

Traffic-oblivious policies(KEY STEPS)

GOAL: A robust MPL-control policy πOPT

STEP 1: Approximate the orig. job size distribution

X ≈ H*

STEP 2: Construct candidate policies

π1 π2 π3 … πN

STEP 3: Pick the “best” candidate policy

27

Traffic-oblivious policies(KEY STEPS)

GOAL: A robust MPL-control policy πOPT

STEP 1: Approximate the orig. job size distribution

X ≈ H*

STEP 2: Construct candidate policies

π1 π2 π3 … πN

STEP 3: Pick the “best” candidate policy

28

GOAL: A family of candidate dynamic policies

IDEA: 1. Assume arrival ~ Poisson( )2. Solve for

3. Obtain the family by varying

Traffic-oblivious policies(KEY STEPS)

Optimal Dynamic MPL policy to minimize mean response time under H* job sizes

Intuitively :What should this look like?

29

Structure of

0

5

10

15

20

25

0 10 20 30 40 50

Current Queue Length

CurrentMPL

MPL ↓

MPL ↑K*=

obtained by combining policy iteration with some new tricks to handle infinite state space

“Stochastic” component “Fluid”

= Target MPL

30

Traffic-oblivious policies(KEY STEPS)

GOAL: A robust MPL-control policy πOPT

STEP 1: Approximate the orig. job size distribution

X ≈ H*

STEP 2: Construct candidate policies

π(λ1) π(λ2) … π(λN)

STEP 3: Pick the “best” candidate policy

31

Traffic-oblivious policies(KEY STEPS)

GOAL: A robust MPL-control policy πOPT

STEP 1: Approximate the orig. job size distribution

X ≈ H*

STEP 2: Construct candidate policies

π(λ1) π(λ2) … π(λN)

STEP 3: Pick the “best” candidate policy

32

GOAL: Robustly choose a member from family

IDEA: Minimize the worst-case relative deviation in an adversarial setting

Traffic-oblivious policies(KEY STEPS)

a

perf. ratio(,) Mean resp. time of policy under Poisson()

Mean resp. time of OPT static MPL under Poisson()

a* = argmin max perf. ratio((a),) - You pick - Adversary sees , picks - You lose perf. ratio(,)- Optimize against worst adversary

33

Traffic-oblivious policies(KEY STEPS)

GOAL: A robust MPL-control policy πOPT

STEP 1: Approximate the orig. job size distribution

X ≈ H*

STEP 2: Construct candidate policies

π(λ1) π(λ2) … π(λN)

STEP 3: Pick the “best” candidate policy min max OPT

34

Back to picking optimal MPL…

Scenario #1Arrival process ~ Poisson(?)

Scenario #2Arrival process ~ ?(?)

(results for Batch Poisson arrivals)

Worst Perf. Loss(vs. OPT Static MPL)

K*Our

Dynamic Policy

K*Our

Dynamic Policy

- Dynamic policies remain “robust” to non-Poisson arrival process- Static policies become worse

45% 20%

14%50%

35

SUMMARY

• Efficiency ≠ Min. response time

• Static MPL policies via the first approximation for M/G/PS-MPL with state dependent service rates

• First traffic-oblivious MPL control policies for resource sharing systems

See paper formore results

MPL*

MPL ↓

MPL ↑