1816IEICE TRANS. INF. & SYST., VOL.E101–D, NO.7 JULY 2018

PAPER

Energy Efficient Resource Selection and Allocation Strategy forVirtual Machine Consolidation in Cloud Datacenters

Yaohui CHANG†,††a), Student Member, Chunhua GU†b), Fei LUO†, Guisheng FAN†, Nonmembers,and Wenhao FU†, Student Member

SUMMARY Virtual Machine Placement (VMP) plays an importantrole in ensuring efficient resource provisioning of physical machines (PMs)and energy efficiency in Infrastructure as a Service (IaaS) data cen-ters. Efficient server consolidation assisted by virtual machine (VM) mi-gration can promote the utilization level of the servers and switch theidle PMs to sleep mode to save energy. The trade-off between energyand performance is difficult, because consolidation may cause perfor-mance degradation, even service level agreement (SLA) violations. Anovel residual available capacity (RAC) resource model is proposed toresolve the VM selection and allocation problem from the cloud serviceprovider (CSP) perspective. Furthermore, a novel heuristic VM selec-tion policy for server consolidation, named Minimized Square Root avail-able Resource (MISR) is proposed. Meanwhile, an efficient VM al-location policy, named Balanced Selection (BS) based on RAC is pro-posed. The effectiveness validation of the BS-MISR combination isconducted on CloudSim with real workloads from the CoMon project.Evaluation results of experiments show that the proposed combinationBS-MISR can significantly reduce the energy consumption, with an aver-age of 36.35% compared to the Local Regression and Minimum MigrationTime (LR-MMT) combination policy. Moreover, the BS-MISR ensures areasonable level of SLAs compared to the benchmarks.key words: residual available capacity model, server consolidation, vir-tual machine migration, energy consumption, cloud computing

1. Introduction

Cloud computing leverages utility computing, grid comput-ing, and distributed computing to provide services of infras-tructure, platform and software for users [1] and suppliesthe on-demand services via the network. Due to the ever-increasing cloud infrastructure demand, the sharp increaseof data center (DC) in size and numbers has a significant in-crease in power consumption. The energy consumption ofdatacenters increased by 56% worldwide from 2005–2010,which accounts for 1.3% of total electricity use [2]. And aglobal annual datacenter construction size will be $78 bil-lion by 2020 [3]. The energy consumption cost of datacen-ter accounts for 45% of the total operating cost [4], whichwas the largest part of all. The high quality of service (QoS)requires CSPs to make a trade-off between the energy and

Manuscript received September 30, 2017.Manuscript revised February 2, 2018.Manuscript publicized March 30, 2018.†The authors are with School of Information Science and

Engineering, East China University of Science and Technology,Shanghai, 200237, China.††The author is with College of Information Science and Tech-

nology, Shihezi University, Shihezi, Xinjiang, 832003, China.a) E-mail: changyaohui@126.comb) E-mail: chgu@ecust.edu.cn (Corresponding author)

DOI: 10.1587/transinf.2017EDP7321

performance, as aggressive energy saving may lead to per-formance degradation. High energy consumption not onlybrings high operating cost but also results in higher carbonemissions. Therefore, it becomes the major concern to de-sign energy efficient resource management strategies [5].

Low resource utilization [6] is a major factor in the highpower consumption of data centers. It reported that physicalservers’ average CPU utilization is only 10% to 50% at mostof the time in the datacenters [4]. Taking Google as an ex-ample, the utilization of servers of Google’s clusters is lessthan 50% on average [7]. Hence, it is an urgent challengeto design the efficient resources allocation schemes, whichwill not only reduce the energy consumption but also im-prove the resources utilization level under the SLA and QoSconstraints.

Dynamic Voltage and Frequency Scaling (DVFS) andServer Consolidation (SC) are the two effective energy sav-ing techniques widely adopted in virtualized cloud data cen-ters. DVFS technique by adjusting the frequency and volt-age of CPU to save energy, which may result in performancedegradation and prolongs the runtime of the tasks. Virtual-ization technique (VT) promotes the utilization level of theresource by sharing a physical server with several VM in-stances. With VT, SC aggregates VMs into fewer serversthrough migration to reduce the number of active hosts tosave energy.

One major drawback of the current server consolida-tion approaches is that proposed solutions only concentrateon the CPU dimension and ignore other dimensional re-sources such as RAM and bandwidth. VM migration isexpensive, as it not only expands the network bandwidthoverhead but also causes data centers’ network congestion.Furthermore, it can lead to performance degradation andSLA violations. Hence, it is necessary to reduce the numberof VM migrations. Therefore, efficient VMP schemes areneeded to achieve efficient resource utilization and energyconservation.

The main contributions of this paper are: Firstly, resid-ual available capacity (RAC) model is presented to carvethe availability degree of physical servers in the heteroge-neous datacenters. Secondly, an energy efficient VM selec-tion strategy named Minimized Square Root available Re-source (MISR) that used for selecting the proper VM tomigrate is proposed. Moreover, an efficient VM alloca-tion policy based on RAC model named Balanced Selec-tion (BS) that used for finding a new placement for the VM

Copyright c© 2018 The Institute of Electronics, Information and Communication Engineers

CHANG et al.: ENERGY EFFICIENT RESOURCE SELECTION AND ALLOCATION STRATEGY FOR VIRTUAL MACHINE CONSOLIDATION1817

to migrate is presented. Thirdly, the proposed combinedoptimization policy BS-MISR was evaluated on CloudSimwith real workload traces from the PlanetLab. Experimen-tal results show that the BS-MISR significantly reduces theenergy consumption while providing a reasonable level ofSLAs.

2. Related Work

Many studies both in industry and academia have focusedon the energy efficient research of cloud data centers. Seenfrom the CSP perspective, a key requirement is to ensureefficient resource utilization and energy efficient resourceprovisioning [9]. Generally, the VM migration [10] consistsof four divisions: overloaded hosts (hotspot PMs) detec-tion, under-loaded hosts (cold-spot PMs) detection, chooseproper VMs to migrate (VMs selection) and allocate theVMs to under-loaded hosts (VMs placement). The effi-cient VMP is conducive to efficient resource provisioningand power saving.

Known as a VM assignment problem, VMP is criti-cal to the efficient resource provisioning. Resource allo-cation in cloud computing is demonstrated in Fig. 1. TheVMP accomplishes the map function between the VMs andPMs with two steps. At the first step, all cloud tenants’ re-quests are encapsulated into different VMs. The next step,several models and policies which based on different opti-mization objectives are utilized to assign the specific VMto the selected PM to accomplish the placement. VMPcan be divided into two categories: online provisioning andbatch provisioning [11]. The former receives the requestsand places them immediately. The latter collects requests toform a group and places them under several constraints.

Several works formulate VMP as a variable size binpacking problem [12], [29] where PMs are conceived as binsand VMs as items. Therefore, the classical bin-packing al-gorithms should be modified to apply in the VM consoli-dation problem for three main reasons [13]: (a) the multi-dimensional resources (e.g. CPU, Memory etc.); (b) the dif-ferent bin sizes (e.g. the heterogeneous servers); (c) multi-objective optimization functions (e.g. energy, load balanceetc.) with SLA constraints and QoS requirements. In fact,the multi-dimensional bin packing has great difference withmulti-capacity bin packing, which can be seen from the il-lustration in Fig. 2. The details are reported in Sect. 3.1.

Many of heuristic approaches are proposed to solve theVMP such as various greedy algorithms: First Fit (FF), BestFit (BF), First Fit Decreasing (FFD), Best Fit Decreasing(BFD), where they do not provide the global optimum solu-tions. The Ref. [10] proposed several metrics to rank serversby considering an adaptive upper bound based on a statis-tical analysis of historical CPU data. The Median Abso-lute Deviation method (MAD), Interquartile Range method(IQR), Local Regression method (LR) and Robust Local Re-gression (LRR) have been proposed to estimate the over-load thresholds of CPU utilization. And they also proposedthree different virtual machine selection policies: Mini-

Fig. 1 VM allocation in cloud computing

Fig. 2 MDBP, MCBP and VMP

mum Migration Time policy (MMT), Random Selectionpolicy (RS) and the Maximum Correlation policy (MC). Theproposed metrics only use the current CPU utilization asthe main criterion to decide VMs’ migration destinations.Reference [14] proposed a Modified Best Fit Decreasing(MBFD) algorithm by sorting the VMs in the decreasingorder and PMs in the increasing order of their capacity.The limitation of MBFD is only single objective consideredand could not accommodate the scalable situations of datacenters.

The Ref. [23] concentrated on the predictive valuebased on the local regression of historical data, and it di-vided the status of the hosts into three categories: under-utilized, properly-utilized and over-utilized. Based on theLR-MMT introduced in [10], they combined with the re-quested MIPS of the VMs, and (or) the number of VMswhen SLA violation last occurred, the SLA violations weregreatly reduced and achieved better energy conservation.The reduction of the energy consumption is only with a min-imum of 0.15%, and a maximum of 14.12% compared tothe LR-MMT. But the decision method of thresholds is notgiven, which will affect the energy consumption as well asthe SLA metrics.

The Ref. [24] improved the framework proposed in[10], and refined the criteria whether the host is over-loaded.Partly similar to [23], it divided the under-loaded hosts intomore fine-grained states: UH, UM and UL. The new frame-work reduces the number of VM migrations by a quarter,and the energy consumption reduction is 1.15% on averagethan that in [10].

Some other works used bio-inspired and nature-inspired algorithms, such as PSO [12], [42], GA [16],ACO [5], [17], BBO [18], Firefly [19] etc. Sharma [12]focused on the key goals of multi-objective VM alloca-tion based on Particle Swarm Optimization (PSO) and VMmigration to reduce the energy consumption, resource

1818IEICE TRANS. INF. & SYST., VOL.E101–D, NO.7 JULY 2018

wastage and SLA violations. Gao et al. [17] proposed amulti-objective ant colony based VMs allocation at the ho-mogeneous data center which is not realistic in fact. How-ever, the intelligent evolutionary algorithms are sensitive tothe parameters, and need artificial adjustment according tothe system status. Therefore, the cost of parameter opti-mization is high, and it’s a long time to achieve the bettertrade-off between multiple objectives.

Several works paid attention to the predictive frame-works, and variety of host overloading methods [20]–[22],[25] have been proposed. Z. Xiao et al. [21] proposed thedynamic resources allocation using VMs in cloud data cen-ter. The limitation of the Xiao’s work is that if the loadis not predicted appropriately, SLA violation is suffered.Li et al. [22] developed a Bayesian network-based estima-tion model for live VM migration. The Ref. [25] proposedan adaptive fuzzy threshold based manner to detect theover-loaded hosts and under-loaded hosts, which assisted toachieve energy and performance tradeoff.

In this paper, we consider not only the heterogeneityof physical machines, but also the heterogeneity of virtualmachine types. From the perspective of remaining avail-able resources, we design efficient VM selection strategyand placement policy based on the RAC model to saveenergy.

3. Problem Formulation

In this paper, PMs, physical servers and hosts are alterna-tively used with the same meaning in different situations.To facilitate expression, some assumptions have been madeand shown below:

a) The IaaS Cloud environment is assumed and the ten-ants lease slices of the hardware of the datacenter pro-vided by the cloud service provider. Instances of ten-ants are isolated each other under the assistance of vir-tualization.

b) The service requests are received from the cloud ten-ants are encapsulated into the service instances, wherethey perform in the forms of VMs. All VMs need afixed amount of cloud resource (i.e. CPU and memory)for a specified amount of time.

c) The cloud consists of heterogeneous servers, whichconforms to the reality of data centers. A shared cloudstorage resources system is employed in the cloudwhich facilitates data sharing and transport betweenPMs. So network bandwidth and storage requirementsof VMs are out of this paper’s scope.

3.1 VM Placement and Multi-Capacity Bin Packing

Server consolidation in cloud data center is usually treatedas a variable bin packing problem, in which PMs are consid-ered as bins and VMs as items. As the bin packing problemis NP-hard, so it can be solved with heuristic methods suchas Best Fit Decreasing (BFD) algorithm. The solution of the

algorithm uses no more than 11/9·OPT+1 bins, where OPTis the number of bins provided by the optimal solution [27].The multi-capacity bin packing (MCBP) problem [28] isvery similar to the classical multi-dimensional bin pack-ing (MDBP) problem, with the exception that bins are non-homogeneous. The difference is explicated in Fig. 2.

In MDBP, the item can be put into a bin only when geo-metric space is enough whatever how many swaps happenedbetween horizontal and vertical dimensions (see Fig. 2 (a)).But any portion of the horizontal or vertical capacity can beused by only one item in MCBP (see Fig. 2 (b)). In MCBP,once the resource is utilized or occupied by one VM, theresource space cannot be reused by any other VMs at thesame time [29], only if the instance is terminated or migratedout. VMs are dynamically time-varied objects, tightly pack-ing them with traditional bin packing heuristics may lead tothe “stability of placement” [30] concerns. In this research,virtual machine placement problem will be formulated asmulti-capacity vector bin packing (MCBP).

3.2 Formulation of the MCBP for VMP

Supposing there are n items to be packed into at most mbins, and requiring to use as few bins as possible with eachitem’s dimensional request is not exceeded the correspond-ing dimension capacity of that bin. Some related definitionsare given below:Definition 1 (Bins): Let B = {B1,B2, . . . ,Bm} be a set ofm heterogeneous bins (|B| = m) where the sizes of bins areidentical or distinct, the capacity of bin Bi is defined as ad-dimension vector Ci = {Ci,1,Ci,2, . . . ,Ci,k . . . ,Ci,d}, whereCi,k is the k-th dimension resource capacity and Ci,k > 0 forall the k = {1, 2, . . . , d}.Definition 2 (Items): Let X = {X1,X2, . . . ,Xn} be a set of nitems, where the items are required to be packed into as fewbins as possible without exceeding the bin’s capacity. Anitem X j in X can be represented as a d-dimensional vector,X j = (r j,1, r j,2, . . . , r j,k, . . . , r j,d) where r j,k is the k-th dimen-sion requirement of the j-th items and for ∀ k ∈ {1, 2, . . . , d}.Definition 3 (Mapping Function): We define a mappingfunction f : {1, . . . n} → {1, . . . ,m}, such that f ( j) = i,∀ j ∈ {1, 2, . . . , n}, i ∈ {1, 2, . . . ,m}. Formally, a map-ping is feasible if all items have been successfully mappedto bins: ∀ j, f ( j) � φ, and the combined demand of re-sources are within the bin capacities. For ∀k ∈ {1, 2, . . . , d},∑n

j=1 x j,ir j,k < Ci,k where x j,i stands for the item Xj whethermapped to Bi, if yes, x j,i = 1; otherwise 0.Definition 4 (Solution): A solution of the multi-capacitybin packing problem is a feasible mapping that can be rep-resented as S = {S1,S2, . . . ,Si, . . . ,St}, for ∀i ∈ {1, 2, . . . , t},and t ∈ {1, 2, . . . ,m} where Si can be represented Si =

{p3,i, p5,i, p6,i, pq,i} where pq,i represents that the item q canbe packed into Bi. The packing decision is subject to someconstraints as following shows:

n∑j=1

r j,k x j,i < Ci,k (1)

CHANG et al.: ENERGY EFFICIENT RESOURCE SELECTION AND ALLOCATION STRATEGY FOR VIRTUAL MACHINE CONSOLIDATION1819

m∑i=1

x j,i = 1 (2)

x j,i = {0, 1},∀ j ∈ {1, 2, . . . , n},∀i ∈ {1, 2, . . . ,m} (3)

The first constraint means the items requirement ineach dimension should not exceed the corresponding capac-ity of the bin, while the second and third constraints ensurethat each item must be accommodated in a single bin.

4. Energy-Aware Resource Selection and Allocation

The system architecture is presented in Fig. 3. The data cen-ter consists of many server clusters. Moreover, the numberof physical servers in any cluster is bounded because thecluster has limited hardware resources, peak power in termsof maximum power supply that it can consume, and the peaknetwork bandwidth that it can use.

We assume a data center consists of several small com-puting clusters, each of which consists of two or more PMs,and manages the VMs located on it with own service man-ager (we named it as local cluster manager). This scenariois analogous to the cluster schedulers in Google’s. Sim-ilarly to OpenStack, each cloud’s cluster has a cloud or-chestrator. Furthermore, a virtualization hypervisor is em-ployed, where they work together to create VM instanceswith different specifications, allocate them to tenants and en-able instances available for tenants with QoS requirements.Finally, a shared cloud storage system is employed to savedata [31], which can facilitate data sharing between all PMsand easily provide users with a shared cloud storage re-source and enable live migration of VMs rapidly.

4.1 Power Model

With the virtualization technologies, IaaS cloud providersprovide a resource selection interface based on abstractcomputational units (e.g. EC2 compute unit). Hav-ing considered the estimated peak usage of their work-loads, cloud tenants usually rent computational unitsmore than what they really need, which resulted in

Fig. 3 System architecture.

cloud providers have to deal with massive hardwaredeployments.

The main energy consumption in data centers comefrom the computing nodes [32], which are determined byhardware efficiency. Energy consumption of computingnode mainly comes from the components such as CPU,Memory, storage systems and enabled network interfacecards. Most studies [33]–[35] have shown that the powerconsumption by servers can be accurately described by a lin-ear relationship between the power consumption and CPUutilization, even when DVFS is applied. In general, givena CPU utilization u ∈ [0, 1], the power consumed by theserver can be denoted as:

P(u) = Ps + Pd × u (4)

Where u is the percentage of CPU utilization, and Ps refersto static power consumption which is independent of work-load. As Ps reflects the idle server’s high energy consump-tion, it becomes the main motivation for efficient server pro-visioning [36]. Pd refers to the dynamic power consump-tion that mainly depends on a specific usage scenario, clockrates, I/O activity, short-circuiting current and switched ca-pacitance [37]. Let utilization u(t) be the function of time t,and the Eq. (4) can be denoted as:

P(u(t)) = Ps + Pd × u(t) (5)

Definition 5. For each vmj ∈ PMi we define the CPU uti-lization of PMi as the ratio of the CPU resources allocatedto the VMs to the total CPU capacity during the time slot tperiod:

ui(t) =J∑

vmj∈PMi

vmcpuj (t)

Rescpui

(6)

Where vmcpuj (t) stands for the CPU utilization of vmj at t

time slot; Rescpui is the CPU capacity of PMi; and J is the

number of VMs running on PMi, respectively. The totalenergy Ei consumed by PMi at time period t can be definedas:

Ei =

∫tP(ui(t))dt (7)

Given a Cluster with N PMs, the total energy con-sumed E can be expressed as formula (8) as shown below:

E =N∑

i=1

∫t

⎛⎜⎜⎜⎜⎜⎝Ps + Pd ·J∑

vmj∈PMi

vmcpuj (t)

Rescpui

⎞⎟⎟⎟⎟⎟⎠dt (8)

4.2 Residual Available Capacity Model

Power consumption in heterogeneous systems dependsgreatly on the type of processors used in the server farm.Since CPU is the largest energy consumer [35]–[37] of theserver, migrating VMs from one PM to another has a posi-tive impact on reducing the energy consumption, by influ-encing the CPU load of the server. Power efficiency re-flects on how much useful work produced by the server for a

1820IEICE TRANS. INF. & SYST., VOL.E101–D, NO.7 JULY 2018

given power consumption [38]. The higher CPU utilizationof PM, the better power efficiency. The utility function de-scribes the satisfaction for a certain service obtained by thetenant [46], combined with the criteria of the utility func-tion [47], we use the exponential function to carve the PM’senergy efficiency.

U(x) =

⎧⎪⎪⎪⎨⎪⎪⎪⎩0, 0 < x ≤ θ2

e−α(x−θ) − 1, x > θ(9)

Where α and θ affect the sharpness and offset of the function,respectively. Let x be the normalized value which belongsto [0, 1], and we can use the utility function to measure thepower efficiency level (e.g. α = 5, θ = 0). If x be the utiliza-tion of resource (e.g. CPU utilization), the larger PM’s CPUutilization is, the better PM’s power efficiency gets.

Due to the complex and dynamic workload of the cloudsystem, different available resources have varied influenceson the physical servers. In [39], the remaining resourceof physical host Li is defined to represent the performancepower of physical host Li:

Li = aLicpu + bLi

mem (10)

a + b = 1 (11)

Where Li represents the remaining computing power of thephysical host i; Lcpu, Lmem are the remaining CPU resourceand the remaining memory resource, respectively. a, b arethe CPU weight value of Li, and the memory weight valueof Li, respectively.

It’s obvious that physical servers are treated as multi-dimensional resource units in the IaaS cloud scenario. Dif-ferent dimension resource has different importance in dif-ferent scenes. For example, the application requests maybe classified as computing-intensive, data-intensive and I/O-intensive. Therefore, it is necessary to take all the dimen-sional resources into account. We extend the Eq. (10), anduse the residual available capacity (RAC) to measure thephysical servers’ load ability when the utility of servers isidentical. RAC value reflects the utility level of resource uti-lization of the physical server pmi. The definition of RAC isdefined in Eqs. (12)–(13) as follow:

RAC(pmi) =D∑

d=1

λdracdpmi

(12)

D∑d=1

λd = 1, 0 ≤ λd ≤ 1 (13)

Where racdpmi

represents the residual available capacity ofpmi on dimension d that can be utilized and allocated to theVMs, λd is the dimensional weight of pmi on dimension d,and d ∈ {1, 2, . . . , |D|}. Parameter values are obtained by BPNeutral Network (BPNN) with system history data [39].

The general BPNN consists of three layers: the in-put, hidden and output layers. Particularly, BPNN has oneor more hidden layer, thus allowing the networks to model

Fig. 4 Flowchart of server consolidation.

complex functions [40]. Let the number of nodes in the in-put, hidden and output layers be m, s and n, respectively. LetX ∈ Rn×m be the system history data, where n is the numberof weights to be learned. And xp ∈ Rm is an m-dimensionalinput vector, which consists of system workload level, sys-tem performance parameters and system physical resourceutilization in D dimensions. The output vector of the hiddenlayer hp, and the output vector of the output layer yp are[41]:

hp = f (W1 × xp + θh), yp = f (W2 × hp + θo)

Where f (·) is the activation function, W1 ∈ Rs×m is theweight matrix between the input and hidden layers, andθh ∈ Rs is the threshold vector; W2 ∈ Rn×s is the weight ma-trix between the hidden and the output layers, and θo ∈ Rn

is the threshold vector.

4.3 Proposed Algorithms

Server consolidation has several steps (see Sect. 2), wemainly concentrated on two aspects: (a) Select the mostsuitable VM from the over-loaded hosts to migrate, and wenamed this policy as Minimized Square Root VM Selection(MISR). (b) Based on the RAC model, find new placementfrom the under-utilized hosts for the VM to migrate, wenamed it Balanced Selection (BS) policy.

The flowchart of server consolidation is shown inFig. 4. First of all, the threshold based heuristic algorithm isused to select the overloaded hosts. Secondly, the novel pol-icy named MISR for selecting the VMs from over-utilizedhosts is applied. Thirdly, we propose a novel policy for opti-mizing the VMs allocation based on resource-aware capac-ity utility model to form the migration map such as <vm id,pm id> pairs. Finally, VMs allocation module is requestedto complete the server consolidation process.

4.3.1 Host Overloading Detection and Underutilized HostSelection

In the virtualized data centers, different types of applicationsshare the physical resources. On the consideration of com-paring with other adaptive dynamic thresholds, the static uti-lization threshold to detect the over-utilized hosts is utilized.If the utilization rate of PM exceeds a predefined threshold(e.g. 80%), the server is considered to be overloaded.

If the host utilization is lower than the threshold, thehost is regarded as under-loaded. And it may be chosen as

CHANG et al.: ENERGY EFFICIENT RESOURCE SELECTION AND ALLOCATION STRATEGY FOR VIRTUAL MACHINE CONSOLIDATION1821

the destination PM for the VMs to migrate. We try to mi-grate the VMs from current host to another one that remainsunder-loaded after placement. Once all the VMs are mi-grated out from the current host, it is switched off. If not,the host is kept active. The process is literately repeated forall active under-loaded hosts.

4.3.2 Minimized Square Root VM Selection Policy

In this section, we propose a novel policy for selecting theVM from over-utilized hosts. We choose the VM which hasthe minimal impact on load to migrate out, where the impactis defined by the RAC model. The proposed VM selectionstrategy named Minimized Square Root (MISR) is basedon the dimensional weight factors under their resource re-quirements constrains. Furthermore, we use MISR policyto accomplish the VM selection process, under the assump-tion that the network bandwidth is enough stable and has nochanges.

We try to find the minimum value with different di-mensional resource requirements. All the VMs resourcesare normalized first to make sure that all the dimensionalresources are transformed to [0, 1] and remove the differ-ences in different forms of units. The equation used is givenbelow:

zki =

zki

zkmax

(14)

Where zki and zk

max refer to the normalized value and the max-imum value on dimension k. respectively. After that thedimensional difference will be eliminated.

For the consideration of multi-dimensional resource al-location situation, it is necessary to consider the distancedifferences between PMs. As we considered the resourcerequest of the VM and the resource of the PM as multi-dimensional vectors, and need a metric to measure the dis-tance between the resources of the PMs and the originalpoint in multi-dimensional spaces. So we use Euclideandistance as the metric for measuring the abilities of phys-ical servers to serve VMs, which is similar to the Ref. [42]which uses Euclidean distance to determine the ability ofserver’s energy efficiency.Definition 6. For the physical server PMi, the Ed(i) valueis defined as the Euclidean distance between PMi and theorigin point of N-dimensional space as defined below:

Ed(i) =

√∑N

k=1(xk

i − 0)2, ∀k ∈ [1, . . . ,N] (15)

Where xki is the utilization of the physical resource of the

physical server PMi on dimension k. And k denotes the re-source such as CPU, memory, disk and bandwidth.

Having considered the fact of that, different dimensionresources may become bottleneck resource of the system un-der the dynamic workload, and weight method is applied toalter and affect the results. Therefore, the Eq. (15) can berewrite as follow:

Algorithm 1 Minimized Square Root VM Selection (MISR).

Ed(i) =

√∑N

k=1(ωk xk

i )2, ∀k ∈ [1, . . . ,N] (16)

Where ∀k, ωk ≥ 0 and∑N

k=1 ωk = 1.In this paper, we concentrate on designing energy effi-

cient VM selection policy and VM placement policy. Basedon the reality of the cloud system, the parameter weight inEq. (16) can be given according to the experience. With-out loss of generality, we treat each dimensional resourceequally in this paper.

After the data is converted to the normalized data, wetraverse all the VMs listed in the VM Migration List, andtry to find the VM which has the minimum value of Ed(i). Ifnot, no VM is selected to be the most suitable one. The com-plexity of the Algorithm 1 is O(n∗m), where n is the numberof PMs in the OverloadedHostList and m is the number ofVMs in the MigratableVmsList that have to be allocated.

4.3.3 VM Allocation Policy

The RAC based VM allocation named Balanced VM newplacement Selection (BS) is given in Algorithm 2. Foreach VM in the migration list, we first sort the vms inmigrationMap list with a descending order (line 1). Foreach VM in the migration list, we compare the two PM’sutility function value with RAC model, and sort the PMs ina descending order based on the allocated MIPS of the PM(line 3). Then the variable of the allocated host is initializedto be NULL (line 4), for all the host involved in, it checkthe host whether in the excluded hosts list (line 6). Next, wecheck the host whether is suitable for the current VM andmake sure the host will not become the over-utilized hostafter allocation (line 7-9). If all the constraints are satis-fied, the allocated host value will become the current host(line 8). Once the loop process is finished, the current VMand the allocated host are added to the migration map if the

1822IEICE TRANS. INF. & SYST., VOL.E101–D, NO.7 JULY 2018

Algorithm 2 Balanced VM new placement Selection (BS).

value of allocated host is altered (line11-13). Once all theVMs in the migration set have been traversed, the process isfinished and the migration map is returned (line 15).

The complexity of sort VMs in descending order isO(m∗ log m). The complexity of sort PMs in descendingorder is O(n∗ log n). So the complexity of the algorithm 2is O(m∗(log m + n∗ log n + n)), where n is the number ofPMs in the HostList and m is the number of VMs in theMigrationVmsSet that have to be allocated.

5. Experiments and Analysis

5.1 Experiment Setup

Since IaaS is the targeted system, it is natural to evalu-ate the proposed resource allocation algorithms on a large-scale cloud datacenter infrastructure. Due to the difficultyto conduct repeatable large-scale experiments on real in-frastructure, simulations have been chosen as the realisticway to evaluate the performance of the proposed algorithms.The CloudSim [43] is a toolkit for modeling and simulatingcloud computing, which provides essential classes for de-scribing cloud computing such as computational resources,virtual machines, cloud users, and management policies.Therefore, we use CloudSim to evaluate the proposed ap-proaches, where it can ensure the repeatability and repro-ducibility of experiments.

We build a data center with 800 heterogeneous PMsand 2 types of physical servers based on HP ProLiantML110 G4 (Intel Xeon 3040, 2 cores with 1860 MHz, 4GB) and HP ProLiant ML110 G5 (Intel Xeon 3075, 2 coreswith 2660 MHz, 4 GB). In the data center, half of PMs areHP ProLiant ML110 G4 servers and the other half consistsof HP ProLiant ML110 G5 servers. The energy consump-tion data of HP ProLiant ML110 G4 and G5 are provided bySPEC [45].

Four types of VM specifications are used: Micro(500 MIPS, 613 MB RAM), Small (1000 MIPS, 1740 MBRAM), Medium (2000 MIPS, 1740 MB RAM) and Large(2500 MIPS, 870 MB RAM), which are based on Amazon

Table 1 Workload data characteristics (CPU Utilization).

Table 2 Algorithms parameters specifications.

EC2 to simulate the heterogeneous requests.

5.2 Workload

We conduct our experiments on real workload traces,which is publicly available workloads from the CoMonproject [44], a monitoring infrastructure for PlanetLab†. Wehave randomly chosen 10 days’ data as our experimentdataset from the workload traces collected during Marchand April 2011. In the dataset, each VM’s workload traceis stored in a single file while each day’s workload tracesare stored in a directory, which consists of number of VMs’workload traces. The VMs’ workload trace usage data is re-ported every 5 minutes from thousands of VMs which comefrom more than 500 places around the world. The charac-teristics of the VMs’ workload traces in the PlanetLab arepresented in Table 1.

5.3 Benchmark

To evaluate the performance of the BS-MISR, five bench-marks were utilized: (1) Static Threshold and Minimum Mi-gration Time policy (THR-MMT), (2) Interquartile Rangeand Maximum Correlation policy (IQR-MC), (3) Me-dian Absolute Deviation and Maximum Correlation policy(MAD-MC), (4) Robust Local Regression and Random Se-lection policy (LRR-RS) and (5) Local Regression and Min-imum Migration Time policy (LR-MMT). The benchmarkshave better performance in some aspects based on experi-mental verification. For example, the LR-MMT has thebest performance [10], the THR-MMT has the lowest SLAVvalue, while IQR-MC, MAD-MC, and LRR-RS have fewerVM migrations.

For the benchmarks, the utilization threshold is set to0.8 for all the algorithms, and the safety parameter for MADis set to 2.5, for LRR is set to 1.2, and for IQR is set to1.5, respectively. The parameter details of algorithms arepresented in Table 2. For the sake of fairness, all the ex-periments are based on the same parameters for both theBS-MISR and the benchmark algorithms. Each experimentis simulated several times, and the reported results are the

†The PlanetLab project. http://www.planet-lab.org/.

CHANG et al.: ENERGY EFFICIENT RESOURCE SELECTION AND ALLOCATION STRATEGY FOR VIRTUAL MACHINE CONSOLIDATION1823

averaged results.

5.4 Performance Evaluation Metrics

To evaluate the efficiency of the proposed approach, fourevaluation metrics are utilized: energy consumption, SLAviolations (SLAV), energy-SLA violations (ESV), and thenumber of migrations. These metrics are also used in theRefs. [10], [13], [15] and [23]–[25].Energy consumption: The datacenter’s total energy con-sumption is first considered. The energy consumption ofphysical servers mainly depends on the utilization of theCPU, memory, disk and network card. The power data ofthe servers used in the experiments are measured by SPECbenchmark [45].SLA Violations (SLAV): Quality of service (QoS) require-ments are usually given in the form of SLAs [10]. So SLAViolations (SLAV) is a very important indicator of QoS fordata centers. The SLAV metric [10] defined in Eq. (17) isutilized to measure the performance, where SLAVO refers toSLA Violations due to Overutilization and the PDM refersto SLA Performance Degradation due to Migrations as de-fined in Eq. (18).

SLAV = SLAVO · PDM (17)

SLAVO =1M

∑M

i=1

T si

Tai, PDM =

1N

∑N

k=1

Cdk

Crk(18)

Where M is the number of PMs, and T si is the total timethat the PM has experienced the CPU or memory utilizationof 100% leading to an SLA violation. Tai is the total timeof the PM being the active state. N is the number of VMs,and Cdk is the estimate of the performance degradation ofthe VM k caused by migrations; Crk is the total CPU ca-pacity requested by the VM k during its lifetime. And weestimate Cdk as 10% of the CPU utilization in MIPS duringall migrations of the VM k.Energy and SLA Violation (ESV): So as to minimize en-ergy consumption and SLA violations, combined metricESV [10] is employed and shown below:

ESV = Energy · SLAV (19)

The Number of VM Migrations: Live VM migration is acostly operation that involves the amount of CPU process-ing, memory blocks copy and transfer time cost, and band-width cost between the source and destination. And VMmigration consumes non-negligible energy [10]. The morethe number of migrations, the greater the negative impact onperformance.

5.5 Results and Analysis

For the convenience of expression, the workload data aremarked from A to K with chronological order, which is iden-tical to Table 1.

The energy consumption metric is showed in Fig. 5.Seen from the Figure, the lower power consumption it has,

Fig. 5 Energy consumption comparison.

Fig. 6 ESV metric comparison.

the better performance of algorithm it shows. The pro-posed BS-MISR has the lowest energy consumption com-pared with the benchmark solutions. The BS-MISR hasthe least energy consumption while the THR-MMT has thelargest energy consumption.

The energy consumption of BS-MISR reduced 32.56%to 40.13%, with an average of 36.35% reduction, comparedto the LR-MMT. The energy reduction in [23] is 9.38% com-pared with the LR-MMT. The results demonstrate that theBS-MISR has better energy conservation compared with thebenchmarks.

The ESV metric is showed in Fig. 6. The smaller ESVmetric it is, the better result it has. Seen from the figure,the proposed BS-MISR, THR-MMT and the LR-MMT so-lutions always have smaller results. Especially, the ESVof BS-MISR reduces 28.97% to 41.48%, compared withLR-MMT. As ESV = Energy · SLAV , the better result ofBS-MISR in the ESV mainly dues to the excellent energyconservation by comparing with the SLAV metric, which isshown in Fig. 8.

The number of VM migrations is shown in Fig. 7. Themore the number of migrations, the greater negative im-

1824IEICE TRANS. INF. & SYST., VOL.E101–D, NO.7 JULY 2018

Fig. 7 Number of VM migrations comparison.

Fig. 8 SLAV metric comparison.

pacts on the performance metric. The BS-MISR has theleast number of VM migrations in total. The number ofVM migrations of BS-MISR is smaller than the benchmarksin most of the conditions while the THR-MMT has thelargest number of VM migrations on each workload day.Especially, compared with the number of VM migrationsof the LR-MMT, the BS-MISR reduces 10.65% at most,24.41% at least, with an average of 18.56% reduction. Fur-thermore, compared with the benchmarks, the number ofVM migrations of the BS-MISR cuts down 7.06% on aver-age, while it reduces 18.8% (compared to the LR-MMT)at most, and cuts down 2.22% at least (compared to theMAD-MC).

Figure 8 demonstrates the SLAV metric results. Thesmaller SLAV metric value it is, the better excellent result ithas. The proposed solution BS-MISR almost has the samelevel SLA violations compared with LR-MMT which is thebest optimization combination proposed in [10], while theTHR-MMT has the best performance. The SLAVO andPDM metric are shown in the Fig. 9 and Fig. 10, respec-tively. Combined with the Fig. 8, we can make a conclusionthat the PDM metric is the main factor which influences the

Fig. 9 SLAVO metric comparison.

Fig. 10 PDM metric comparison.

SLAV metric.The explanations can be found from the perspective of

BS-MISR itself. First of all, the MISR algorithm pays atten-tion to calculate the Euclidean distance for all dimensionalresources only. Secondly, the BS algorithm employs the sortmechanism to select the VM with maximum CPU requestand place it to the underutilized host with higher CPU uti-lization. After that, the selected host is still keeping under-utilized state after placement. Therefore, BS-MISR does notassure the selected VM has the minimum migration time.

As shown in Eq. (18), the PDM metric has two factors,the total CPU capacity requested by the VM k during its life-time, and the cost of migration which is estimated as 10%of the CPU utilization in MIPS. As discussed above, the BS-MISR selecting the VM k do not assure the minimum migra-tion time, which is also to say, the BS-MISR always try toselect the VM with larger CPU request, so the PDM metricof BS-MISR is much higher than the MMT-style policies.The results shown in Fig. 10 can be found as an evidence.

CHANG et al.: ENERGY EFFICIENT RESOURCE SELECTION AND ALLOCATION STRATEGY FOR VIRTUAL MACHINE CONSOLIDATION1825

Fig. 11 Energy consumption, ESV, SLAV and the number of VM Migra-tions comparison.

Fig. 12 The average number of shutdown hosts, the average time of hostbeing active and the number of active hosts comparison.

5.6 Further Discussion and Analysis

For the sake of clarity, the normalized method is utilized todemonstrate the performance differences for the proposedBS-MISR and the benchmarks.

Figure 11 shows the comparison of the four metrics.The maximum value method (see Eq. (12)) is employed tonormalize all the metrics. For all the evaluation metrics, thesmaller value it is, the better result it has. Seen from the fig-ure, the proposed BS-MISR has the smallest value in energyconsumption, ESV metric and the number of VM migrationscompared with the benchmarks. Only one exception, it’s theSLAV. Next, we will give a glance on the other three metrics,which are the average number of host shut down (termed ashostShutdown), the average time of host active (termed asavgTimeHostActive) and the number of active hosts (termedas activeHostNum) as shown in the Fig. 12.

In particular, with the same number of VM migrationsassumption, the larger number of host shut down does notsignify the better server consolidation effect. Because im-

proper server consolidation results in frequently shut downand power-on the servers. Generally, the higher averageresource utilization, meanwhile, the larger the value ofavgTimeHostActive, the better performance of energy sav-ing. Furthermore, the fewer the number of activeHostNumit has, the better the effect of server consolidation.

As shown in the Fig. 12, the BS-MISR has the leastactiveHostNum, at the same time, it also has the leasthostShutdown while the largest avgTimeHostActive, all to-gether, they give us an explanation why it achieves betterenergy efficiency than the benchmarks.

6. Conclusion

In this paper, the server consolidation with the trade-off be-tween energy and performance in the virtualized data cen-ter is studied and validated. Four important metrics ofthe data center, such as energy consumption, SLA viola-tion, ESV and the number of VM migrations are consid-ered for the VMs consolidation. First of all, the VM place-ment problem is addressed as a multi-capacity bin packingproblem which has been proven is NP-Hard. To achievethe energy-efficiency, a novel resource-aware capacity util-ity model RAC is set up to guide the VM consolidation pro-cess. Moreover, an energy efficient VM selection policynamed MISR based on the RAC model is proposed. Fur-thermore, the VM allocation policy named BS is given tooptimize the consolidation. What’s more, the optimizationcombination BS-MISR is evaluated on the CloudSim withthe real workload traces from PlanetLab. Simulation resultsvalidate that the proposed BS-MISR is reliable and can sig-nificantly reduce the energy consumption compared with thebenchmarks while the BS-MISR remaining the SLA viola-tions at a reasonable level.

Acknowledgments

This research was supported by the National Natural Sci-ence Foundation of China (Grant NO.61472139). Profes-sor Chunhua GU is the corresponding author of this paper.We sincerely appreciate the anonymous reviewers’ valuabletime and comments to assist us to improve this paper.

References

[1] P. Mell and T. Grance, “The NIST definition of cloud computing,”2011.

[2] J. Koomey, “Growth in data center electricity use 2005 to 2010,”A report by Analytical Press, completed at the request of The NewYork Times, 2011, 9.

[3] L. Christian, “Projecting Annual new Datacenter Construction Mar-ket Size,” Technique report, Microsoft Corp., 2010.

[4] L.A. Barroso, J. Clidaras, and U. Holzle, “The datacenter as a com-puter: An introduction to the design of warehouse-scale machines,”Synthesis lectures on computer architecture, vol.8, no.3, pp.1–154,2013.

[5] F. Farahnakian, A. Ashraf, T. Pahikkala, P. Liljeberg, J. Plosila, I.Porres, and H. Tenhunen, “Using ant colony system to consolidateVMs for green cloud computing,” IEEE Trans. Serv. Comput., vol.8,

1826IEICE TRANS. INF. & SYST., VOL.E101–D, NO.7 JULY 2018

no.2, pp.187–198, 2015.[6] S. Homsi, S. Liu, G.A. Chaparro-Baquero, O. Bai, S. Ren, and G.

Quan, “Workload consolidation for cloud data centers with guaran-teed qos using request reneging,” IEEE Trans. Parallel Distrib. Syst.,vol.28, no.7, pp.2103–2116, 2017.

[7] “Pocket gems on google cloud platform,” http://cloud.google.com/customers/pocketgems/.

[8] D. Hagimont, C.M. Kamga, L. Broto, A. Tachna, and N.D. Palma,“Dvfs aware CPU credit enforcement in a virtualized system,”//ACM/IFIP/USENIX International Conference on Distributed Sys-tems Platforms and Open Distributed Processing, pp.123–142, 2013.

[9] B. Song, M.M. Hassan, A. Alamri, A, A. Alelaiwi, Y. Tian, M.Pathan, and A. Almogren, “A two-stage approach for task and re-source management in multimedia cloud environment,” Computing,vol.98, no.1-2, pp.119–145, 2016.

[10] A. Beloglazov and R. Buyya, “Optimal online deterministic algo-rithms and adaptive heuristics for energy and performance efficientdynamic consolidation of virtual machines in cloud data centers,”Concurrency and Computation: Practice and Experience, vol.24,no.13, pp.1397–1420, 2012.

[11] M.M. Rahman and P. Graham, “Responsive and efficient provision-ing for multimedia applications,” Computers & Electrical Engineer-ing, vol.53, pp.458–468, 2016.

[12] N. Sharma and R.M. Guddeti, “Multi-Objective Energy EfficientVirtual Machines Allocation at the Cloud Data Center,” IEEE Trans.Serv. Comput., 2016.

[13] F. Farahnakian, T. Pahikkala, P. Liljeberg, J. Plosila, N.T. Hieu, andH. Tenhunen, “Energy-aware vm consolidation in cloud data cen-ters using utilization prediction model,” IEEE Trans. Cloud Com-put., 2016.

[14] A. Beloglazov and R. Buyya, “Managing overloaded hosts for dy-namic consolidation of virtual machines in cloud data centers underquality of service constraints,” IEEE Trans. Parallel Distrib. Syst.,vol.24, no.7, pp.1366–1379, 2013.

[15] F. Farahnakian, A. Ashraf, T. Pahikkala, P. Liljeberg, J. Plosila, I.Porres, and H. Tenhunen, “Using ant colony system to consolidateVMs for green cloud computing,” IEEE Trans. Serv. Comput., vol.8,no.2, pp.187–198, 2015.

[16] Q. Wu, F. Ishikawa, Q. Zhu, and Y. Xia, “Energy and migrationcost-aware dynamic virtual machine consolidation in heterogeneouscloud datacenters,” IEEE Trans. Serv. Comput., 2016.

[17] Y. Gao, H. Guan, Z. Qi, Y. Hou, and L. Liu, “A multi-objectiveant colony system algorithm for virtual machine placement in cloudcomputing,” Journal of Computer and System Sciences, vol.79,no.8, pp.1230–1242, 2013.

[18] Q. Zheng, R. Li, X. Li, N. Shah, J. Zhang, F. Tian, K.-M. Chao, andJ. Li, “Virtual machine consolidated placement based on multi-ob-jective biogeography-based optimization,” Future Generation Com-puter Systems, vol.54, pp.95–122, 2016.

[19] X.-K. Li, C.-H. Gu, Z.-P. Yang, and Y.-H. Chang, “Virtual ma-chine placement strategy based on discrete firefly algorithm incloud environments,” 2015 12th International Computer Conferenceon Wavelet Active Media Technology and Information Processing(ICCWAMTIP), IEEE, pp.61–66, 2015.

[20] D. Bui, Y.I. Yoon, E.N. Huh, S. Jun, and S. Li, “Energy efficiency forcloud computing system based on predictive optimization,” Journalof Parallel and Distributed Computing, vol.102, pp.103–114, 2017.

[21] Z. Xiao, W. Song, and Q. Chen, “Dynamic resource allocation usingvirtual machines for cloud computing environment,” IEEE Trans.Parallel Distrib. Syst., vol.24, no.6, pp.1107–1117, 2013.

[22] Z. Li, C. Yan, X. Yu, and N. Yu, “Bayesian network-based VirtualMachines consolidation method,” Future Generation Computer Sys-tems, vol.69, pp.75–87, 2017.

[23] E. Oikonomou, D. Panagiotou, and A. Rouskas, “Energy-awareManagement of Virtual Machines in Cloud Data Centers,” Proc.16th International Conference on Engineering Applications of Neu-ral Networks (INNS), 2015.

[24] K. He, Z. Li, D. Deng, and Y. Chen, “Energy-efficient framework forvirtual machine consolidation in cloud data centers,” China Commu-nications, vol.14, no.10, pp.192–201, 2017.

[25] L. Salimian, F.S. Esfahani, and M.-H. Nadimi-Shahraki, “An adap-tive fuzzy threshold-based approach for energy and performance ef-ficient consolidation of virtual machines,” Computing., vol.98, no.6,pp.641–660, 2016.

[26] W. Ding, C. Gu, F. Luo, Y. Chang, U. Rugwiro, X. Li, and G.Wen, “DFA-VMP: An efficient and secure virtual machine place-ment strategy under cloud environment,” Peer-to-Peer Networkingand Applications, pp.1–16, 2016.

[27] M. Yue, “A simple proof of the inequality FFD (L)≤ 11/9 OPT (L)+1, ∀ L for the FFD bin-packing algorithm,” Acta Mathematicae Ap-plicatae Sinica (English Series), vol.7, no.4, pp.321–331, 1991.

[28] I. Hwang and M. Pedram, “Hierarchical, Portfolio Theory-BasedVirtual Machine Consolidation in a Compute Cloud,” IEEE Trans.Serv. Comput., 2016.

[29] M. Mishra and A. Sahoo, “On theory of vm placement: Anomaliesin existing methodologies and their mitigation using a novel vec-tor based approach,” 2011 IEEE International Conference on CloudComputing (CLOUD), IEEE, pp.275–282, 2011.

[30] M. Mishra and U. Bellur, “Whither tightness of packing? the casefor stable vm placement,” IEEE Trans. Cloud Comput., vol.4, no.4,pp.481–494, 2016.

[31] W. Zhu, Y. Zhuang, and L. Zhang, “A three-dimensional virtual re-source scheduling method for energy saving in cloud computing,”Future Generation Computer Systems, vol.69, pp.66–74, 2017.

[32] L. Minas and B. Ellison, Energy Efficiency for Information Tech-nology: How to Reduce Power Consumption in Servers and DataCenters, Intel Press, 2009.

[33] D. Kusic, J.O. Kephart, J.E. Hanson, N. Kandasamy, and G. Jiang,“Power and performance management of virtualized computing en-vironments via lookahead control,” Cluster computing, vol.12, no.1,pp.1–15, 2009.

[34] S. Rivoire, P. Ranganathan, and C. Kozyrakis, “A Comparison ofHigh-Level Full-System Power Models,” HotPower, vol.8, no.2,pp.32–39, 2008.

[35] Y. Tan, Q. Wu, and H. Tang, “Multi-tier energy management strategyfor HPC clusters,” Green Computing and Communications (GreenCom), pp.112–116, 2010.

[36] A. Greenberg, J. Hamilton, D.A. Maltz, and P. Patel, “The costof a cloud: research problems in data center networks,” ACMSIGCOMM computer communication review, vol.39, no.1,pp.68–73, 2008.

[37] Y.C. Lee and A.Y. Zomaya, “Energy conscious scheduling for dis-tributed computing systems under different operating conditions,”IEEE Trans. Parallel Distrib. Syst., vol.22, no.8, pp.1374–1381,2011.

[38] A.M. Sampaio and J.G. Barbosa, “Towards high-available and en-ergy-efficient virtual computing environments in the cloud,” FutureGeneration Computer Systems, vol.40, pp.30–43, 2014.

[39] J. Zhao, K. Yang, X. Wei, Y. Ding, L. Hu, and G. Xu, “A heuristicclustering-based task deployment approach for load balancing usingBayes theorem in cloud environment,” IEEE Trans. Parallel Distrib.Syst., vol.27, no.2, pp.305–316, 2016.

[40] Y. Kuang, R. Singh, S. Singh, and S.P. Singh, “A novel macroeco-nomic forecasting model based on revised multimedia assisted BPneural network model and ant Colony algorithm,” Multimedia Toolsand Applications, pp.1–22, 2017.

[41] X. Dong, Y. Lian, and Y. Liu, “Small and multi-peak nonlinear timeseries forecasting using a hybrid back propagation neural network,”Information Sciences, vol.424, pp.39–54, 2018.

[42] H. Li, G. Zhu, C. Cui, H. Tang, Y. Dou, and C. He, “Energy-efficientmigration and consolidation algorithm of virtual machines in datacenters for cloud computing,” Computing, vol.98, no.3, pp.303–317,2016.

[43] R.N. Calheiros, R. Ranjan, A. Beloglazov, C.A.F. De Rose, and

CHANG et al.: ENERGY EFFICIENT RESOURCE SELECTION AND ALLOCATION STRATEGY FOR VIRTUAL MACHINE CONSOLIDATION1827

R. Buyya, “CloudSim: a toolkit for modeling and simulation ofcloud computing environments and evaluation of resource provision-ing algorithms,” Software: Practice and Experience, vol.41, no.1,pp.23–50, 2011.

[44] K.S. Park and V.S. Pai, “CoMon: A mostly-scalable monitoringsystem for PlanetLab,” ACM SIGOPS Operating Systems Review,vol.40, no.1, pp.74–83, 2006.

[45] SPEC power benchmarks, Standard Performance EvaluationCorporation, http://www.spec.org/benchmarks.html#power

[46] S. Li, D. Yang, Y. Qin, and H. Zhang, “Network cross-layer mappingbased in utility maximization,” Journal of Software, vol.22, no.8,pp.1855–1871, 2011.

[47] X. Shi, and K. XU, “Utility maximization model of virtual machinescheduling in cloud environment,” Chinese Journal of Computers,vol.36, no.2, pp.252–262, 2013.

Yaohui Chang received his M.S. degreein Computer Science from Northeastern Uni-versity (NEU) of China in 2011. Now he isa PhD candidate of Computer Science at EastChina University of Science and Technology(ECUST). His current research interests includecloud computing, scheduling optimization andbig data.

Chunhua Gu received his M.S. and Ph.D.degrees from East China University of Scienceand Technology (ECUST). Professor and PhDsupervisor in the School of Information Scienceand Engineering, ECUST. Senior member ofChina Computer Federation. His research in-terests include cloud computing and internet ofthings.

Fei Luo received his M.S. and Ph.D. degreesfrom Huazhong University of Science and Tech-nology (HUST) in 2004 and 2008, respectively.Associate professor in the School of Informa-tion Science and Engineering, ECUST. His mainresearch interests include cloud computing anddistributed computing.

Guisheng Fan received his M.S. and Ph.D.degrees from East China University of Scienceand Technology (ECUST) in 2006 and 2009,respectively. All in computer science. He ispresently a research assistant of the Departmentof Computer Science and Engineering, ECUST.His research interests include formal methodsfor complex software system, service orientedcomputing.

Wenhao Fu received her B.S. degree fromEast China University of Science and Technol-ogy in 2012 in computer science. She is a Ph.D.student in computer science at East China Uni-versity of Science and Technology. Her currentresearch interests include software engineeringand software fault localization.