Bargaining Towards Maximized Resource Utilizationin Video Streaming Datacenters

Yuan Feng, Baochun LiDepartment of Electrical and Computer Engineering

University of Toronto

Bo LiDepartment of Computer Science

Hong Kong University of Science and Technology

Abstract—Datacenters can be used to host large-scale videostreaming services with better operational efficiency, as the multi-plexing achieved by virtualization technologies allows differentvideos to share resources at the same physical server. Livemigration of videos from servers that are overloaded to thosethat are under-utilized may be a solution to handle a flash crowdof requests, in the form of virtual machines (VMs). However, suchmigration has to be performed with a well-designed mechanism tofully utilize available resources in all three resource dimensions:storage, bandwidth and CPU cycles. In this paper, we showwhy the challenge of maximizing resource utilization in a videostreaming datacenter is equivalent to maximizing the joint profitin the context of Nash bargaining solutions, by defining utilityfunctions properly. Having servers participating as players tobargain with each other, and VMs as commodities in the game,trades conducted after bargaining govern VM migration decisionsin each server. With extensive simulations driven by real-worldtraces from UUSee Inc., we show that our new VM migrationalgorithm based on such Nash bargaining solutions increases boththe resource utilization ratio and the number of video streamingrequests handled by the datacenter, yet achievable in a lightweightfashion.

I. INTRODUCTION

As an efficient means of providing computing resourcesin a form of utility, cloud computing has recently attracteda substantial amount of attention from both industry andacademia. The paradigm shift to cloud computing is drivenby strong demand, especially from enterprises, to improve theoverall efficiency of using and managing computing resources.Cloud service providers use datacenters to provision a sharedpool of computation, storage and bandwidth resources, to beused by applications when the need arises. Since resources atdatacenters are shared by using virtualization, applications areallowed to statistically multiplex such resources in the form ofvirtual machines.

A large-scale video streaming service requires both com-putation and bandwidth. Due to its highly varied demandfrom users, it is one of the prime examples that migration todatacenters in the cloud becomes more economical. Take thevideo streaming service offered by NetFlix Inc. as an example,with widely varying user demand for bandwidth and the factthat Internet Service Providers (ISPs) bill for 95% of the peak

This research was supported in part by a grant from NSFC/RGC un-der the contract N HKUST610/11, by grants from HKUST under contractsRPC11EG29, SRFI11EG17-C and SBI09/10.EG01-C, by a grant from HuaweiTechnologies Co. Ltd. under the contract HUAW18-15L0181011/PN, and by agrant from Guangdong Bureau of Science and Technology under the contractGDST11EG06.

bandwidth usage, it would be much more economical to usecloud services rather than deploying privately owned mediaservers.

We have also observed from our first-hand real-world ex-periences that demand for video streaming services may peakat different times. Fig. 1 shows the normalized population oftwo videos in three days by analyzing 200 Gigabytes worthof operational traces that we collected at UUSee Inc. [2],one of the leading peer-assisted video streaming providers inChina. As we can see, Video 1 (an on-demand stream) andVideo 2 (a live stream) have peaked on August 12 and August13, 2008, respectively. If privately owned media servers areused by provisioning for peak bandwidth usage, bandwidthwill remain severely under-utilized during off-peak times. Itis an economically sound decision to migrate video streamingservices to datacenters in the cloud.

Aug 13

Aug 1412:00 00:00 12:00 00:00 12:00

Time

0.00.20.40.60.81.0

Norm

alize

d Po

pulat

ion OverloadedVideo 1 Video 2

Fig. 1. The normalized population of two videos from August 12 to 14, 2008.

Once the decision is made to host video streaming serviceswith datacenters in the cloud, the question becomes howdatacenter resources can be better utilized. A datacenter consistsof tens of thousands of physical servers. Since videos mayreach their respective peak demand at different times, thereis an opportunity to increase server utilization ratios by lettingmultiple videos share physical resources on the same server.With virtualization ubiquitously used in Infrastructure-as-a-Service cloud platforms, several VMs, each of which packagingone video, are able to be placed on the same physical server andbenefit from “on-demand” resource provisioning. Specifically,resources at a physical server are shared among all VMs duringoff-peak seasons to achieve a higher utilization ratio; once oneof the videos encounters its peak in demand, VMs packagingthis video can use up all available resources at servers wherethey are placed, in order to satisfy as many requests as possible.

2

One possible challenge we may encounter is that serversmay become overloaded when a video encounters highly burstyrequests, or several videos placed on the same server reach theirpeak period in demand at the same time. Shown in Fig. 1, bothVideo 1 and Video 2 reached their peaks on August 14. Anaive solution will be to move VMs away from overloadedservers to under-utilized ones. Thanks to the support of livemigration with virtual machines [12], migrating from one serverto another with minimal downtime to the streaming service isfeasible in datacenters. With live VM migration, the numberof requests being handled at the same time may be effectivelyincreased by migrating VMs with additional resource neededfrom resource-deficient to resource-rich servers.

Nevertheless, we argue that such migration should be plannedwith care, by fully exploring all possibilities of utilizing cur-rently available resources to handle the increased requests.Since servers in the datacenter provide resources in threemain dimensions of storage, bandwidth and CPU in a tightly-coupled manner, utilization of resources may gradually becomeseverely unbalanced across different dimensions, which impliesunnecessary idling of available resources. Since datacentersare expensive to build and to operate, it is a waste of bothinvestment and energy when resources are under-utilized [7].

In this paper, we seek to design an efficient and practicalalgorithm to maximize resource utilization, with all threedimensions considered, by migrating VMs across servers ina video streaming datacenter. We first formally formulate thechallenge of maximizing system-wide resource utilization as acentralized optimization problem, taking into account practicalconstraints of storage, bandwidth, and CPU cycles. The optimalsolution dictates the destination server to which VMs shouldbe migrated. We find that the optimization problem is inthe form of a Generalized Assignment Problem (GAP) ininteger programming, which is NP-hard and even APX-hard toapproximate. Obtaining a near-optimal solution to this problemrequires to decouple it into several 0-1 knapsack problems anditerate hundreds of times.

Inspired by the power of markets in arbitrating decisions ofboth buyers and sellers in a decentralized fashion, we relate theentire datacenter to a bargaining market. VMs are considered as“commodities” in this market. Every server makes its decisionby participating in this market and bargaining for its desiredcommodities. We model this market as a Nash bargaininggame, and prove that the problem of maximizing resourceutilization in a datacenter is equivalent to that of maximizing thejoint profit in the Nash bargaining solution. The VM migrationdecisions are governed by the individually made bargainingstrategies in each server in a laissez-faire manner, which avoidsadditional CPU consumption at a centralized decision makerand reduces the bandwidth cost of VM migration.

The remainder of this paper is organized as follows. InSec. II, we formulate the challenge of maximizing resourceutilization in virtualized datacenters and motivate to use theNash bargaining solution to this problem. Sec. III shows thatthe formulation is equivalent to the joint profit maximization

problem in a Nash bargaining solution, and describes the VMmigration algorithm. In Sec. IV, we show the effectivenessof our VM migration algorithm by presenting an in-depthanalysis, driven by real-world traces from UUSee Inc. Wediscuss related work and conclude the paper in Sec. V andSec. VI, respectively.

II. MAXIMIZING RESOURCE UTILIZATION IN VIDEOSTREAMING DATACENTERS

We first present an example to show that VM migration mayhelp to fully utilize resources in all three dimensions to handlemore requests, and then formulate the problem of maximizingresource utilization in video streaming datacenters.

A. Benefits of VM Migration

Satisfying one request for streaming a video, at High-Definition (HD) or Standard-Definition (SD) quality levels,requires different amounts of bandwidth. The variety of existingmedia formats requires on-demand transcoding, which resultsin distinct requirements on CPU cycles when processing a videostreaming request. As the number of requests for each videofluctuates over time, the required amount of resources for eachvideo in dimensions of storage, bandwidth and CPU cycles mayincrease in an unbalanced manner. With live migration of virtualmachines (VMs), we may migrate VMs across the boundary ofservers in a datacenter, in order to fully explore possibilities ofutilizing available resources, which leads to more concurrentrequests being handled.

The following conceptual and proof-of-concept example il-lustrates the potential benefits with such migration. Consider avideo streaming datacenter with two servers and three videos.Each of the videos is served by a corresponding VM: VM1,VM2, and VM3 respectively. In this example, Video 1 repre-sents a live video streaming service, e.g., with the standard-definition (SD) quality level, and with network coding adoptedin transmission, which require more CPU but less bandwidthresources. Video 2 represents an on-demand streaming (VoD)service, e.g., in HD quality without using network codingduring transmission, which requires more bandwidth, but withrelatively low CPU demand. Video 3, again, represents a livestreaming service, but no network coding is involved in itstransmission. Resources required to handle one request for eachvideo are summarized in Table I, along with resource capacitiesat servers. We note that there are no additional storage resourcesrequired for handling each request in video streaming services.The size of each video is assumed to be 4 GB is this example.

TABLE IREQUIRED RESOURCES TO ACCOMMODATE ONE REQUEST IN EACH VM, AS

WELL AS THE RESOURCE CAPACITY IN EACH SERVER.

Resources Server 1/2 VM1 VM2 VM3

Storage Space (GB) 8 − − −Bandwidth (Mbps) 9 1 3 1

CPU (MIPS) 10 3 1 2

Suppose initially Video 1 is popular and Video 2 and 3are less popular, as there are five requests for Video 1 and

3

one request for Video 2 and 3, respectively. It is not difficultto find out that the optimal resource utilization strategy is tohave the popular video provided by two servers, each of whichis concurrently multiplexed by one unpopular video. That is,placing (VM1,VM2) in one server and (VM1,VM3) in anotherserver, shown in Fig. 2. In this fit, a total of seven requests canbe handled at the same time.

Server 1 Server 2Request for Video 1

Request for Video 2VM1 VM1VM2 VM3

Request for Video 3

Fig. 2. The initial fit of three videos on two servers.

Server 1 Server 2

VM1 VM1VM2 VM3

(a) Before VM migration

Server 1 Server 2

VM1 VM1 VM2 VM3

(b) After VM migration

Fig. 3. Improved video fit after VM migration.Due to the variation of requests over time, the popularity

of videos changes as time elapses, with, say, three requestsfor Video 1, one request for Video 2 and four requests forVideo 3 at this time. No possible migration plan exists if weuse the naive solution of moving VMs away from overloaded tounder-utilized servers, which means one of the eight requestswill not be satisfied at the same time due to the unbalancedincrease of resource demand in bandwidth and CPU cycles,no matter how the three requests for Video 1 is scheduled.Fig. 3(a) indicates one possible case, where one request forVideo 1 is directed to Server 2. The heavily utilized CPU inServer 2 results in a drop of one request for Video 3, which alsorequires CPU. As a consequence, one of the requests can notbe satisfied immediately, even as resources are still availableon the two servers.

However, if VM migration is conducted in the datacenter,we can swap VM2 in Server 1 and VM1 in Server 2, which isshown in Fig. 3(b). In such a scenario, the VM, which requiresmore resource in one dimension, is fit into the server withanother VM which requires less resource in that dimension ina complementary manner. The eight requests for three videoscan be satisfied by the two servers at the same time, whichleads to a higher level of resource utilization.

From this example, we can see that by migrating VMs acrossthe boundary of servers, a datacenter is able to better utilizeavailable resources and handle more requests at the same time.

B. Maximizing Resource Utilization in a Generalized FormOur primary objective in this paper is to find out how

VM migration strategies should be designed so that resourceutilization in datacenters is maximized. We first present thecontext of our discussion and a model of a datacenter.

Instead of restricting ourselves to video streaming data-centers, we discuss this problem in a generalized form so

that the designed algorithm is also applicable to datacentersholding general application instances. We consider a datacenterconstituting a set of heterogeneous servers, denoted by N . Forevery server i ∈ N , the amount of storage space capacity isCi, in Gigabytes; the amount of bandwidth capacity is Ui, inMbps; and the amount of CPU computing capability is Pi, inMIPS.

Let the set of application instances hosted by the datacenterbe denoted by M. For any k ∈ M, VMk represents thecorresponding virtual machine serving that application. Opera-tional datacenters typically provide customers the flexibility tochoose from a number of different VM instances equipped withdifferent amounts of resources in each dimension. For example,Amazon Elastic Compute Cloud (EC2) provides high-memory,micro and high-CPU instances [1], in order to serve applicationswith different resource demands. To better indicate the amountof dedicated resources used by each VM, let sk be the amountof storage space, bk be the amount of bandwidth, and clk bethe amount of computing capability devoted by VMk to handleone request when serving its corresponding application.

We are aware that it is possible that some applications mayspan over a set of VMs, such as Map-Reduce computationaljobs and multi-tier Web services. These applications may placefurther restrictions on the locality of their own set of VMs,since they may require a large amount of inter-VM bandwidth.To serve these applications well, a set of VMs serving the sameapplication should be located in the same server. In this case,our model considers the set of VMs serving one application asa special VM, in a sense that they are considered as an singleentity. To be exact, for an application k′ ∈M, VMk′ representsthe special VM if k′ spans over a set of VMs, and eachVMk′ requires storage space sk′ , bandwidth bk′ , and computingcapability clk′ to handle one request. Since descriptions ofspecial VMs are in essence the same as regular VMs in ourmodel, we do not distinguish k and k′ in subsequent analysis.

Let Iki (t) be the binary variable that indicates whether VMk

is stored in server i at time t.

Iki (t) =

{1 VMk is stored in server i at time t0 otherwise

Let Dki (t) denote the number of requests for VMk to which

server i handles at time t.Under the assumption that all servers in the datacenter

possess global knowledge of which set of VMs other servershave and how many requests each server handles with respectto each of its VMs, at time t, the centralized resource utilizationmaximization problem in a datacenter can be formulated as abinary integer programming problem:

maxIki(t)

1

|N |∑i∈N

Ri(t) (1)

s.t.∑k∈M

Iki (t)skDki (t) ≤ Ci, ∀i (2)∑

k∈MIki (t)bkD

ki (t) ≤ Ui, ∀i (3)

4

∑k∈M

Iki (t)clkDki (t) ≤ Pi, ∀i (4)∑

i∈NIki (t) = 1, ∀k, (5)

where Ri(t) is the estimated resource utilization ratio at serveri after time t, based on the current number of requests andinformation about resource capacities, and Dk

i (t) is the esti-mated number of requests for VMk to which server i handlesafter time t, which has the following form:

Dki (t) =

∑i∈N

Dki (t)I

ki (t).

Note that ∀i at time t, there is only one Iki (t) = 1, since eachVMk can be possessed by only one server at one time.

To capture three integrated dimensions of resource usage ateach server, we define the estimated resource utilization ratioat each server to be the weighted sum of estimated resourceutilization ratios in dimensions of storage, bandwidth and CPU:

Ri(t) = ωsi (t)r

si (t) + ωb

i (t)rbi (t) + ωc

i (t)rci (t)

= ωsi (t)

∑k I

ki (t)skD

ki (t)

Ci+ ωb

i (t)

∑k I

ki (t)bkD

ki (t)

Ui

+ωci (t)

∑k I

ki (t)clkD

ki (t)

Pi(6)

where ωsi (t), ω

bi (t) and ωc

i (t) are the weights given to resourcesin different dimensions according to server i’s current resourceusage states, constrained by ωs

i (t) + ωbi (t) + ωc

i (t) = 1.In this formulation, Iki (t) is the optimization variable, which

is the binary indicator to denote the placement of each VMafter time t based on the information at present. Inequality (2)stands for the storage space constraint; Inequality (3) representsthe bandwidth capacity constraint; and Inequality (4) denotesthe CPU computing capability constraint. The rationale behindthis is that resources consumed by every server can not exceedthe resource capacity in each dimension. Eq. (5) ensures thateach VM can only be possessed by one server at a time. In acentralized manner, the current VM placement indicator Iki (t),the number of requests handled Dk

i (t), and resource capacitiesof each server Ci, Ui, Pi are supposed to be known at time t.

In our subsequent analysis, we focus on decisions at aspecific time. Therefore, the time indices t in the expressionsare dropped, as we obtain the following optimization problemequivalent to the original one (1).

maxIki

∑i

∑k

(ωsi skD

ki

Ci+ωbi bkD

ki

Ui+ωci clkD

ki

Pi

)Iki (7)

To this end, we have formally described the problem of maxi-mizing resource utilization in a datacenter in a centralized man-ner by optimization problem (7). Based on existing knowledge,we seek to find out what is the best placement strategy of eachVM under the constraint of resource capacities in dimensionsof storage, bandwidth and CPU computing capability, so thatthe resource utilization is maximized.

C. Lagrangian Heuristic vs. Nash Bargaining Solution

The formulation is a comprehensive integer optimizationproblem, which appears to be in the form of a multi-dimensional Generalized Assignment Problem (GAP). TheGAP is NP-hard, and it is even APX-hard to approximation [8].Though we may approach the optimal solution through theLagrangian relaxation heuristic: decoupling it into several 0-1knapsack problems, it incurs high computational complexity.Since our objective is to design a practical VM migrationalgorithm that can be implemented in real-world settings, weseek to design alternative feasible heuristics.

In this paper, we propose to use the Nash bargainingsolution to solve the utilization maximization problem. TheNash bargaining game discusses the situation in which two ormore players reach an agreement regarding how commoditiesare to be distributed among them, so that the social utility gainsare maximized and commodities owned by each player do notexceed its capacity. This is exactly the same as the GAP, whichaims to assign a set of objects to agents so that the total profit ofthe assignment is maximized and all agents do not exceed theirbudget. With the same objective, we believe that the mechanismof the Nash bargaining solution is a suitable alternative.

III. VM MIGRATION ALGORITHM BASED ON NASHBARGAINING SOLUTION

In this section, we prove that the Nash bargaining solutionin the bargaining game can be used to solve the resourceutilization maximization problem in virtualized datacenters.The VM migration algorithm based on the Nash bargainingsolution is presented in detail.

A. The Nash Bargaining Solution

Bargaining problems are known as non-zero-sum gamesthat participating players try to achieve a win-win situation.In the Nash bargaining game, there is always a solution forthe optimal strategy at each player, which guarantees thattheir average payoff is maximized under the assumption thatopposing players also use the optimal strategy.

In Nash bargaining games, each player has a differentanticipation to each commodity, which represents a state ofexpectation that may involve the certainty of some contingen-cies and various probabilities of other contingencies [11]. Forexample, if Bill prefers apple to banana, then he may have ahigher anticipation of apple than that of banana. The utility ofeach player is a function of his anticipations to commodities hehas. The Nash bargaining solution is a Pareto efficient solutionto a Nash bargaining game so that the joint profit, which is theproduct of utility gains of all players, is maximized.

Let Fi be the utility function for player i. Rational playerswill seek to maximize the Nash product

∏Gi, where Gi =

|Fi(x)−Fi(d)|. Fi(d) is the status quo utilities (i.e., the utilityobtained if one decides not to bargain with other players). Nashhas shown that obtaining the maximum of the Nash product willattain the Pareto-optimal solution for the bargaining situation.

5

Theorem 1: The problem of maximizing resource utilizationin a virtualized datacenter is equivalent to the joint profitmaximization problem in the Nash bargaining game.

Proof: Envision a market, in which servers are treated asplayers and VMs are considered as commodities. The Nashbargaining game here is to exchange VMs among the set ofservers so that their joint profit is maximized. Let Ak

i be playeri’s anticipation to commodity k. Define the utility function ofplayer i to be Fi, which is represented as follows:

Fi(x) = Fi(d) + exp(∑k∈M

Aki (I

ki − Iki )).

The definition can be explained as the utility of player i afterbargaining equals its status quo utility plus a function of addedanticipations during bargaining. That is to say, the utility gainof player i can be represented as:

Gi = |Fi(x)−Fi(d)|= exp(

∑k∈M

Aki (I

ki − Iki )). (8)

In practice, the Nash bargaining solution aims to maximizethe Nash Product

∏Gi, which can be interpreted as follows:

max∏i∈N

Gi (9)

s.t.∑k∈M

Iki skDki ≤ Ci, ∀i (10)∑

k∈MIki bkD

ki ≤ Ui, ∀i (11)∑

k∈MIki clkD

ki ≤ Pi, ∀i (12)∑

i∈NIki = 1, ∀k. (13)

Constraints (10), (11) and (12) represent the fact that com-modities each player owns can not exceed its capacity. Eq. (13)confirms that each commodity can only be possessed by oneplayer at a time.

Substitute (8) into (9), the maximization problem in the Nashbargaining game is equivalent to the following ones.

max∏i∈N

Gi ⇐⇒ max exp log∏i∈N

Gi

⇐⇒ max exp∑i∈N

logGi

⇐⇒ max∑i∈N

∑k∈M

Aki I

ki .

If we define player i’s anticipation to commodity k, i.e., Aki

in the following form:

Aki =

ωsi skD

ki

Ci+ωbi bkD

ki

Ui+ωci clkD

ki

Pi,

the optimization problem (9) in the Nash bargaining game canbe rewritten as:

max∑i∈N

∑k∈M

(ωsi skD

ki

Ci+ωbi bkD

ki

Ui+ωci clkD

ki

Pi

)Iki ,

which is equivalent to the resource utilization maximizationproblem (7) in virtualized datacenters. It then follows thatthe joint profit maximization problem in the Nash bargainingsolution is equivalent to the resource utilization maximizationproblem in virtualized datacenters.

Based on the established connection between the two prob-lems, the optimization variable Iki can be viewed as an indicatorof the strategy adopted by each player in the bargaining game.The estimated resource utilization ratio of server i, Ri, can betreated as the profit gains of player i by adopting strategy Iki .This implies that the resource utilization maximization problemcan be solved using the mechanism of Nash bargaining solution.

B. VM Migration in the Bargaining Game

VM-based market organization. We envision the existence ofan online market place, where all servers in a datacenter behaveas players; application VMs are treated as commodities. EveryVM is associated with an anticipation Ak

i from each player,which is the evaluation of VMk from server i’s perspectivebased on its own information. According to the proof ofTheorem 1, Ak

i should be of the following form:

Aki = ωs

i

skDki

Ci+ ωb

i

bkDki

Ui+ ωc

i

clkDki

Pi. (14)

In our bargaining market, the anticipation of player i to eachcommodity is defined as the consumed fraction of resourcesin the corresponding application VM in server i, with respectto dimensions of storage, bandwidth and CPU computingresources. Since our objective is to fully utilize resources inevery server, application VMs requiring more resources willbe more valuable. However, fractions in different dimensionsshould be treated differently, which are weighted by ωs

i , ωbi

and ωci according to Player i’s current resource usage states.

The rationale is that resources with a high utilization ratiowill become the “bottleneck” towards fully utilizing resourcesin other dimensions, hence should be less desirable by thatserver. For simplicity, we define the weights to be inverselyproportional to the current resource utilization ratio in thecorresponding dimension. That is,

ωsi =

1rsi∑1r

, ωbi =

1rbi∑1r

, ωci =

1rci∑1r

,

where∑

1r = 1

rsi+ 1

rbi

+ 1rci

. Recall that rsi , rbi and rci aredefined as the current resource utilization ratio of server iin the dimension of storage, bandwidth and CPU computing,respectively, i.e.,

rsi =

∑k∈M Iki skD

ki

Ci

rbi =

∑k∈M Iki bkD

ki

Ui

rci =

∑k∈M Iki clkD

ki

Pi.

6

The bargaining strategy based on spacial representation.It is proved that the problem of determining whether thereexists a Nash equilibrium in which each player has a specificminimum payoff is NP-complete as a function of the number ofplayers [4], so that the Nash bargaining solution in a bargaininggame is usually approximated by numerical methods suchas the Newton’s method or the bisection method [5], whichrequire numerous iterations. Since our objective is a practicalVM migration algorithm that can be implemented in real-world datacenters and executed in a lightweight fashion, wepropose to adopt a bargaining strategy based on the spacialrepresentation of Nash bargaining games [16].

This bargaining strategy based on the spacial representationfits our design objective in virtualized datacenters, since itreduces the computational overhead significantly by eliminatingthe requirement of generating the utility gain for every singlepossible set of strategy execution. By introducing the utility-distance product φki , a function of the anticipation Ak

i , it isproved that the utility-distance product of a commodity isanalogous to the moment of force by weights based on alever system. By suitably locating a pivot location such thatthe distribution of the utility-distance product is uniformlypositioned about a pivot, equilibrium can be achieved.

From a spacial perspective, outcomes of games have beenassumed to lie in some low-dimensional Euclidean space, suchthat anticipations to the players are defined in terms of distancesfrom them [13]. In such spacial games, anticipation of a playerto a commodity is assumed to be an inverse function of thedistance that commodity lies from the player [4], such thatcommodities of higher anticipation values have a closer spatialproximity (i.e., a shorter spatial distance).

Player ii

k Commodity k1 21 2

d11 d1

2

d21 d2

2

Fig. 4. Spacial representation of a 2-player game.

For example, Fig. 4 shows the 2-dimensional spatial repre-sentation of a 2-player game with player-to-player distancesdefined to be constants. Anticipations of the two players tocommodities 1 and 2 are clearly reflected by spacial distancesd1i and d2i , where i ∈ {1, 2}. In the example, commodities arerepresented as points based on their spatial proximities, whichlie within the boundary enclosed by all participating players.The relative distances of a commodity k to player i is definedas dki , which is in the form of:

dki = f(Aki ) =

1/Aki∑

i (1/Aki )∀i ∈ N , k ∈M. (15)

The distances of each commodity to all players are normalizedso that they add up to a unitary value, i.e.,

∑i d

ki = 1, ∀k. In

our example, player 1 has higher anticipation to commodity 1and player 2 prefers to commodity 2.

In this bargaining strategy, each player possesses commodi-ties sorted by their relative distance dki , such that commoditieswith higher anticipations will be given higher priority. The

!ki = g(Ak

i ) = Aki · f(Ak

i )

dki

Aki = f!1(dk

i )

Player iCommodity k=3 2 14 58 ...

Fig. 5. The utility-distance product of commodities of Player i.

utility-distance product of a player to a commodity is definedas φki = dki ·Ak

i . Fig. 5 shows how commodities are sorted byplayer i and the associated utility-distance product φki of eachcommodity.

From a mechanical perspective, weights on a lever arealigned along the same direction such that weights on theleft hand side generate a collective moment that opposes themoment caused by weights on the right. Similarly, to maintainan equilibrium in a bargaining game, the sum of utility-distanceproducts of all commodities should be equally divided amongall participating players, as shown in Fig. 6. Players 1 and2 are considered to be lying at the end points of the lever,where forces can be applied. The utility-distance products ofcommodities are regarded as weights.

!31 + !4

1 + !21

dk1

!12 + !5

2 + !82

Player 1k=3 2 14 58

2 Playerdk2

Pivot

!k1 = !k

2

Fig. 6. Finding the pivot point in a lever system.

It is then not difficult to find out that the pivot point ofthe lever in our example should be lying between commodities2 and 8, where the collective moment generated by weightsφ31 +φ41 +φ21 on Player 1’s side equals to that of φ12 +φ52 +φ82on Player 2’s side. To be precise, the pivot point in a leversystem is determined by balancing weights between two endpoints, which is:

µ =1

2

∑k∈M

φki .

After determining the pivot point, it is natural that the bargain-ing solution is to assign commodities lying on the left handside of the pivot point to player 1, and commodities on theright hand side to player 2.

This bargaining strategy can be easily generalized to multi-player bargaining games. For the ideal condition whereby allcommodities lie in a space between vertices representing allplayers, the determination of a pivot location can be based onbalancing the utility-distance product with respect to all players,

7

which is given by:

µi =1

|N |∑k∈M

φki , ∀i ∈ N . (16)

Fig. 7 shows an example of a 3-player game. The verticalarrows on each commodity represents the cumulative utility-distance product pki , which is defined as the sum of utility-distance products of commodities whose relative distances arenot greater than that of k, i.e., pki =

∑∀j∈M,dj

i≤dk

iφji . By

finding the pivot point where pk11 = pk2

2 = pk33 , the bargaining

solution to the game is found, which is, in our example,C1 = {1, 3, 5, 7}, C2 = {2, 8, 9, 12}, and C3 = {4, 6, 10, 11}.

Player 1dk1

k=1 5 7 3

12

9

2

11Commodity

46

p11

p51

p71

p113

p43

p63

p52

p92

p122

Pivot

p31 = p8

2 = p103

2

3

dk2

dk3

8 10

Fig. 7. Finding the pivot point in a 3-player game according to the utility-distance products.

After each player determines its own pivot point µi, com-modities inside the market will be assigned to each playeraccordingly. To be precise, each player will be assigned a set ofcommodities Ci, so that the sum of utility-distance products ofCi is maximized but not larger than µi. If one commodity k isassigned to more than one players at one time, the player whohas the smallest relative distance dki will obtain this commodity.To conclude, our VM migration algorithm based on the Nashbargaining solution is summarized in Algorithm 1.

Practicality. Though migrating VMs have potential benefitsto improve the resource utilization ratio, it does not comewithout substantial upfront costs of bandwidth. An exampleorchestration of live VM and storage migration on the testbedthrough HARMONY shows that the transaction throughputis reduced by 12% during VM migration [14]. Since theapplication performance may be negatively affected by live VMmigration, it should be avoided as much as possible.

As a consequence, our VM migration algorithm is triggeredin a laissez-faire manner. Whenever the resources provided byone server can not sustain requests for applications placed onthat server, the migration algorithm is triggered. The idea is,if requests can be satisfied under the current provision, we’llmaintain the same even if the resource utilization is not theoptimal, e.g., when the number of requests for one applicationdecreases dramatically at some time. Note that it might be thecase that even after VM migration, the application fit in thedatacenter still can not handle all requests at the same time,i.e., the total available resources is less than the sum of required

Algorithm 1 The VM Migration Algorithm.1: Each player i computes its anticipation to each commodityk in the market, i.e., Ak

i given by (14). Then, it derivesthe relative distance dki using (15), and the correspondingutility-distance product φki , given by φki = dki ·Ak

i .2: Each player i computes the cumulative utility-distance

product pki for all commodities, according to their relativedistances dki .

3: Each player i determines his pivot point µi according to(16).

4: Each player i finds a subset of commodities Ci, whosecumulative utility-distance products are not larger than µi,i.e., Ci = {k : pki ≤ µi,∀k ∈M}.

5: if Commodity k belongs to more than one Ci,∀i ∈ N then6: The player i′ (k ∈ Ci′ ) who has the smallest dki to

commodity k will obtain this commodity.7: else8: The player i′ (k ∈ Ci′ ) will obtain this commodity.9: end if

10: All players migrate their commodities according to theobtained assignment results.

resources for all requests. In this scenario, the only solution forthe datacenter is to add new servers. To avoid triggering the VMmigration algorithm constantly in this scenario, we restrict theminimum interval between two trigger points to be T , whereT = 20 min in our simulation.

IV. EXPERIMENTAL EVALUATION

In this section, we investigate how the proposed VM mi-gration algorithm performs in practical scenarios. Our real-world trace-driven experimental results validate that our VMmigration algorithm increases resource utilization ratios withfluctuating requests in video streaming datacenters effectively,yet in a lightweight fashion.

Our evaluation of the proposed VM migration algorithm isbased on a C++ implementation in an event-driven simulator,which is driven by real-world streaming requests captured byour traces. We are using 200 Gigabytes worth of operationaltraces, which we have collected throughout the 17-day SummerOlympic Games in August 2008, with UUSee as one of theofficial online broadcasting partner in China. Both VoD and livestreaming videos were involved, with each of them representedby a VM in our simulation. More detailed information of ourtrace used in this simulation is summarized in Fig. 8.

As we can see from the table, videos in the trace vary interms of their bitrates, which results in different bandwidthresource demands in each request. In our simulation, weassume each user asks for 10 seconds of video with everyrequest, i.e., the bandwidth resource required per request rangesfrom 264 KB to 879 KB. Besides, videos also differ fromwhether network coding is applied during transmission, whichresults in deviated CPU cycles required in each request. Forvideos without using network coding, the required CPU cycles

8

Fig. 8. Detailed Information About the Trace

Number of videos(live/VoD)

1625(216/1409)

Number of videos withnetwork coding

1472

Peak number of con-current requests (for allvideos)

24757

Highest video bitrate 879 kbpsLowest video bitrate 264 kbps

0 50 100 150

0.5

1

1.5

2

2.5

Time (hours)

Onl

ine

Use

rs (×

104 )

Population

Fig. 9. The demand pattern in 200hours.

0 50 100 1500

5

10

15

20

25

Time (hours)

Impr

ovem

ent

on O

vera

ll U

tiliz

atio

n

ImprovementCI (95%)

Fig. 10. Average of improvementon resource utilization ratios withthe VM migration algorithm.

0.5 1 1.5 20

5

10

15

20

25

Number of Requests (× 104)

Impr

ovem

ent

on O

vera

ll U

tiliz

atio

n

ImprovementCI (95%)

Fig. 11. Average of improvementon resource utilization ratios vs. thenumber of requests.

per request is assumed to follow a normal distribution ofN(2, 0.25) MIPS; for those with network coding, the requiredCPU cycles per request is assumed to be represented byN(2, 0.25) + bitrate/100 × N(1, 0.25), since it is measuredto be proportional to the normalized bitrate of each video. Wesimulate a system with 25 servers, each of which is assumedto have the same amount of resources: 1000 GB storage space,1000 Mbps bandwidth and 1000 MIPS CPU cycles.

Our objective is to increase resource utilization ratios invideo streaming datacenters, so that they can satisfy as manyrequests as possible with their available resources. We comparethe bargaining-based VM migration algorithm with the naiveVM migration algorithm: only the VM that causes overload ismigrated; and its destination server is greedily selected fromall under-utilized servers, i.e., the one with the most availableresources. Main performance metrics in this simulation are,therefore, the improvement on resource utilization ratios andthe number of requests the datacenter can handle successfully.In addition, we also show the bargaining overhead from animplementation point of view. We run our simulation 20 times,each lasting 100 time intervals.

We first present the improvement on resource utilizationratios by using the bargaining-based VM migration algorithm,as a percentage of the ratios using the naive algorithm. Fig. 9shows the demand variation in 200 hours. Fig. 10 shows im-provements on average resource utilization ratios at all serversover time and their 95% confidence intervals under the demandpattern shown in Fig. 9. As we can observe, the averageimprovement on resource utilization ratios varies with the sametrend as demand patterns, with a maximum improvement ofalmost 20% than that of the naive VM migration algorithm.

The reason that the improvement on resource utilizationis less evident and the number of requests is low is thatmost servers are under-utilized in this scenario. The optimalsolution that maximizes the overall resource utilization by thebargaining-based VM migration algorithm is more likely toconform with the greedy results. However, when the number ofrequests increases, improvements on resource utilization withthe bargaining-based algorithm become more evident. Fig. 11shows the average of improvements on resource utilization ra-tios vs. the number of requests. It is clear that the improvementis significant when the number of requests becomes large.

In addition to resource utilization ratios, another importantperformance metric is the number of requests that the dat-

0 50 100 1500

5

10

15

Time (hours)Impr

ovem

ent o

n th

e N

umbe

rof

Req

uest

s H

andl

ed

ImprovementCI (95%)

Fig. 12. Improvement on the num-ber of requests successfully handledby the datacenter.

0.5 1 1.50

5

10

15

Number of Requests ( 104)

Impr

ovem

ent o

n th

e N

umbe

rof

Req

uest

s H

andl

ed

ImprovementCI (95%)

Fig. 13. Improvement on the num-ber of request successfully handledvs. the total number of requests.

acenter is able to handle, given current available resources.As shown in Fig. 12, it is clear that the number of requeststhe datacenter can respond to has increased accordingly byapplying the bargaining-based VM migration algorithm, with animprovement of up to 9% compared with the naive algorithm.With more than 20000 concurrent requests at the peak time,this implies an increase of more than 1800 requests beinghandled by the datacenter. This result confirms what we ob-served in the improvement on resource utilization ratios, sincethe more efficient available resources are being utilized, themore requests the datacenter can handle. Fig. 13 shows theimprovement on the number of requests successfully handledvs. the total number of requests. We can observe a similar trendas Fig. 11, that the improvement becomes more evident as thenumber of requests increases.

0 200 4000

0.5

1

1.5

Bargaining Samples

Red

uctio

n on

Std

. Dev

.

Reduction

Fig. 14. Reduction in standarddeviations of resource utilization ra-tios.

0 0.5 1 1.50

0.2

0.4

0.6

0.8

1

Reduction on Std. Dev.

CD

F

90th percentile0.45%

Fig. 15. CDF of reductions instandard deviations as a percentage.

To further investigate the effectiveness of our algorithm,in Fig. 14, we show reductions in the standard deviationof resource utilization ratios at each server, as a percentageacross 500 bargaining samples. A reduced standard deviationof resource utilization ratios reflects a more balanced resourceutilization. Fig. 15 shows the CDF of reductions on standard

9

deviations. We can observe that the standard deviation ofresource utilization ratios is successfully reduced, the 90thpercentile of the ratio of such reduction is 0.45%, whichshows that our VM migration algorithm is able to mitigateresource under-utilization due to imbalanced resource usageacross different dimensions.

0 500 10000

5

10

15

20

Bargianing Samples

Bar

gain

ing

Ove

rhea

d

Num. of Migrations

Fig. 16. The number of VM mi-grations after bargaining.

0 5 10 15 200

0.2

0.4

0.6

0.8

1

Number of Migrations

CD

F

90th percentile5 migrations

Fig. 17. CDF of the number of VMmigrations after bargaining.

Finally, it is important to evaluate the VM migration over-head incurred with our bargaining-based solution. We plotthe number of trades conducted after running the bargainingalgorithm to balance the resource utilization across all serversin 1000 bargaining samples in Fig. 16, and the CDF of allsamples in Fig. 17. As we can see, in 90% of the cases, thealgorithm incurs fewer than 5 trades. It reveals that the VMmigration overhead is reasonably low.

V. RELATED WORK

Researchers are starting to find possibilities of moving videostreaming services to virtualized clouds. For example, Aggar-wal et al. investigate the potential of utilizing virtualizationto deliver IPTV services, while their focus is on how thenumber of servers required is efficiently minimized, with onlythe deadline constraint considered [3]. Prior to direct videostreaming, video transcoding is commercially available oncloud environments. For example, HDCloud and Encoding haveprovided flexible but proprietary cloud based video transcodingservices integrated with Amazon EC2, S3, and CloudFrontCDN services [6].

There exist research results show how VMs are to bemigrated to alleviate “hotspots” in datacenters. Wang et al. pro-posed an autonomic provisioning framework, so that resourcescan be dynamically provisioned to different applications ac-cording to their demand on CPU capabilities [15]. Meng etal. presented a placement algorithm that focuses on bandwidthconsumption of each VM [10]. Unlike prior works that onlyconsiders a single resource dimension, this paper addresseschallenges of maximizing resource utilization across multipleresource dimensions, which is much more challenging.

To our knowledge, there exist two papers that discussedresource challenges along multiple dimensions in datacenters.Korupolu et al. propose a placement algorithm by applyingstable matching, taking storage, bandwidth, and CPU resourcesinto account [9]. Our work differs in that the problem theyaddress is a static placement problem, in a sense that onceVMs are placed, they are not migrated over time. Singh etal. [14] describes VectorDot, a load balancing algorithm for

handling multi-dimensional resource constraints in datacenters.However, their objective is to alleviate an overloaded serveras much as possible, while ours is to improve the utilizationratios across the board, as required resources increase in anunbalanced fashion and may negatively affect the utilization ofavailable resources in the datacenter.

VI. CONCLUDING REMARKS

Our focus in this paper is to fully utilize resources indimensions of storage, bandwidth and CPU computing in videostreaming datacenters, by migrating VMs live among serverswhen they are overloaded. We argue that such migration shouldbe conducted with careful planning, in order to fully explorepossibilities of utilizing current available resources, in terms ofstorage, bandwidth and CPU. From time to time, conducted ina laissez-faire manner, we believe that VM migration is helpfulto improve overall resource utilization, and ultimately to deliverbetter performance as more video requests are being handled.As a practical way to govern these VM migration decisions,we have designed an algorithm based on the Nash bargainingsolution. With event-driven simulations based on real-worldvideo streaming traces from UUSee Inc., we show that thebargaining algorithm is able to improve resource utilization overtime, with a small amount of VM migration overhead.

REFERENCES

[1] Amazon EC2 Instance Types. http://aws.amazon.com/ec2/instance-types/.[2] UUSee. http://www.uusee.com.[3] V. Aggarwal, X. Chen, V. Gopalakrishnan, R. Jana, K. Ramakrishnan,

and V. Vaishampayan. Exploiting Virtualization for Delivering Cloud-based IPTV Services. In Proc. IEEE INFOCOM Workshop on CloudComputing, 2011.

[4] R. Baron, J. Durieu, H. Haller, and P. Solal. Finding A Nash Equilibriumin Spatial Games Is An NP-Complete Problem. Economic Theory,23(2):445–454, 2004.

[5] S. Boyd and L. Vandenberghe. Convex Optimization. CambridgeUniversity Press, 2004.

[6] A. Garcia, H. Kalva, and B. Furht. A Study of Transcoding on CloudEnvironments for Video Content Delivery. In Proc. ACM MultimediaWorkshop on Mobile Cloud Media Computing (MCMC), pages 13–18,2010.

[7] A. Greenberg, J. Hamilton, D. A. Maltz, and P. Patel. The Cost of aCloud: Research Problems in Data Center Networks. SIGCOMM Comput.Commun. Rev., 39(1):68–73, 2009.

[8] J. K. Karlof, editor. Integer Programming: Theory and Practice. CRCPress, 2005.

[9] M. Korupolu, A. Singh, and B. Bamba. Coupled Placement in ModernData Centers. In Proc. IEEE International Symposium on Parallel &Distributed Processing (IPDPS), pages 1–12, 2009.

[10] X. Meng, V. Pappas, and L. Zhang. Improving the Scalability of DataCenter Networks with Traffic-aware Virtual Machine Placement. In Proc.IEEE INFOCOM, 2010.

[11] J. Nash. The Bargaining Problem. Econometrica, 18(2):155–162, 1950.[12] M. Nelson, B. H. Lim, and G. Hutchins. Fast Transparent Migration for

Virtual Machines. In Proc. USENIX ATC, 2005.[13] G. Owen. Game Theory. Academic Press Inc, 1995.[14] A. Singh, M. Korupolu, and D. Mohapatra. Server-Storage Virtualization:

Integration and Load Balancing in Data Centers. In Proc. of ACM/IEEEConf. on Supercomputing (SC), pages 1–12, 2008.

[15] X. Wang, D. Lan, G. Wang, X. Fang, M. Ye, Y. Chen, and Q. Wang.Appliance-Based Autonomic Provisioning Framework for VirtualizedOutsourcing Data Center. In Proc. 4th International Conf. on AutonomicComputing (ICAC), page 29, 2007.

[16] K. K. L. Wong. A Geometrical Perspective for the Bargaining Problem.PLoS ONE, 5(4):e10331, 2010.