How to Measure Energy Consumption in Your Data CenterHow to Measure Energy Consumption in Your Data...

How to Measure Energy Consumption in Your Data Center

Gartner RAS Core Research Note G00205428, Rakesh Kumar, 8 September 2010, R3506 02112011

This research outlines a pragmatic approach that should be used to get an overall measurement of how much energy a data center is consuming. Without a practical measurement methodology, users will waste money and operate the center at suboptimal levels.

Key Findings

• Measuringdatacenterenergyconsumptionatthebuildinglevelcanbeacomplexexercise(dependingontheage,designanduseofthebuilding);itrequiresacarefulbreakdownofthecoolingcomponent.

• Continuouspowerutilizationefficiency(PUE)readingswillbecomethenormformostnewlargedatacenters.

• Rack-levelenergymeasurementisrelativelystraightforward,andallusersshouldbedoingthatnow.

Recommendations

• MeasureenergyacrossIThardware(server,networkingorstoragebox),racksandelectricalfacilitiestogetanaccurateapproximationofthetotalenergyusedinthedatacenter.

• Accuratelymeasurethedata-center-only,building-levelcooling-energyusage,becauseitaccountsformorethan40%ofthedatacenter’senergyconsumption.

• Usetheenergymanagementsoftwaretoolsavailableonthehardwarecomponents(servers,storageboxes,networkingequipment,etc.)tomeasureandmodelITcomponentenergyuse.

• FeedtheenergyusagedatafromITcomponents(servers,storageboxes,networkingequipment,etc.)intobroaderdatacenterinfrastructuremanagement(DCIM)toolstogainanaccurateusagemodelofenergyconsumptionacrosstheentiresite.

• Whereinternalprivatecloudsarebeingdeveloped,startmeasuringvirtualmachine(VM)energyusageasakeycomponenttoallocatingenergyconsumption.

2STRATEGIC PLANNING ASSUMPTIONBy2015,80%ofnewlargedatacenterswillreportcontinuousPUEreadings.

ANALYSISThedatacenterpower,coolingandenergysupply,andcostproblemsarelikelytoworsenduringthenextfewyears,becauseorganizationswillgrowtheirtechnologyinfrastructuresastheycomeoutofthisrecessionaryperiod.Energy-relatedcostsaccountforapproximately12%ofoveralldatacentercostsandarethefastest-risingcost.Moreover,withamorethan5%growthofservershipmentsperyearduringthenexttwoyears,organizationsneedto“forcefully”controltheirenergyconsumptionandcosts.Tomanagetheproblems,datacenteroperatorsneedtomeasureenergy-relateddataacrossthewholesite,includingthebuilding,thefacilitiescomponentsandtheITequipmentportfolio.

GartnerhaswrittenaboutemergingDCIMsoftwaretools,whichwillenableuserstostartdoingthis.“DCIM:GoingBeyondIT”explainsthisconcept,and“TheBenefitsofanIntegratedEnergyManagementSoftwareApproach”explainshowsuchanapproachmaywork.

Despitetheavailabilityofthesetoolsandabodyofexpertsandconsultantsworkingonthistopic,Gartnercontinuestogetcallsfromusersaskingthesimplequestion,“Whereisthebestplacetomeasuretheenergyinmydatacenter?”Theproblemiscompoundedbythefactthatthereisatheoreticaloridealsetofmeasurementsthatwillgiveacomprehensivesetofenergyreadingsacrosstheentiresite;however,formanyusers,thiswillbeinformationoverload.What’sneededisabreakdownoftheidealmodelintoapragmaticapproachthatwillgivesufficientinformationformostoperationalandplanningpurposes.Thisresearchfirstshowsanidealizedapproachbyoutliningthesixareaswhereenergycouldbemeasured.Itthenprioritizestheseareastogiveamore-practicalapproachthatusersshouldadopt.

TheIdealApproachtoDataCenterEnergyManagement

Togetacomprehensive,accurateandreal-timerecordoftheenergyusedindatacenters,usersneedtomeasureacrosssixareas:

1. Building

2. Electricalfacilities

3. Buildingfacilities

4. Racks

5. IThardware(server,networking,storagebox,etc.)

6. VM

ThesesixkeyareasareshowninFigure1.

Bymeasuringacrosstheseareas,datacentersoperatorscanobtainahighlydetailed,comprehensiveand,inmostcases,real-timeusagepictureoftheenergythatisbeingconsumedacrossthewholesite.Thiswillmeanthattheenergythat’scomingintoadatacentercanbeallocatedtomajordatacentercomponents,suchasfacilities,buildings,IT,etc.Thiscanthenbeusedforfutureplanning,allocatingscarcecapacity,suchasspaceorcoolingandappropriatefinancial(chargebackorsimplecostallocation)resources.

However,thisisa“perfectworld”scenario,and,formostorganizations,itwillneitherbenecessarynorpossibletomeasureacrossallsixareas.Thismaybebecauseofcostortechnicallimitationsofthesite;furthermore,dependingonthetoolsused,therewillprobablybesomeoverlapwhenmeasuringacrossallsixareas.Hence,werecommendprioritizingtheseareasanddevelopingapragmaticapproach.

The Pragmatic Approach to Data Center Energy ManagementApracticalimplementationoftheaboveidealapproachisshowninFigure2.Thesixareasareprioritizedanddividedintothreemainsegments.

Energymeasurementacross(1)IThardware(server,networkingorstoragebox),(2)racksand(3)electricalfacilitiesshouldbedoneimmediately.Thiswillbesufficientformostdatacenterneeds.Energymeasurementsacross(4)datacenterbuildingfacilitiesand(5)datacenterbuildingswillbenecessarymainlyforhostingprovidersthatwanttochargecustomersspecificallyforenergyusage.Measurementacross(6)VMswillhappenduringthenextfourormoreyears.TheywillbecomerelevantwhenuserswanttoexaminetheenergyassociatedwithaworkloadandwhenthatworkloadisrunninginaseparateVM.Eachareaisdiscussedbelow.

©2010Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.oritsaffiliates.ThispublicationmaynotbereproducedordistributedinanyformwithoutGartner’spriorwrittenpermission.Theinformationcontainedinthispublicationhasbeenobtainedfromsourcesbelievedtobereliable.Gartnerdisclaimsallwarrantiesastotheaccuracy,completenessoradequacyofsuchinformationandshallhavenoliabilityforerrors,omissionsorinadequaciesinsuchinformation.ThispublicationconsistsoftheopinionsofGartner’sresearchorganizationandshouldnotbeconstruedasstatementsoffact.Theopinionsexpressedhereinaresubjecttochangewithoutnotice.AlthoughGartnerresearchmayincludeadiscussionofrelatedlegalissues,Gartnerdoesnotprovidelegaladviceorservicesanditsresearchshouldnotbeconstruedorusedassuch.Gartnerisapubliccompany,anditsshareholdersmayincludefirmsandfundsthathavefinancialinterestsinentitiescoveredinGartnerresearch.Gartner’sBoardofDirectorsmayincludeseniormanagersofthesefirmsorfunds.Gartnerresearchisproducedindependentlybyitsresearchorganizationwithoutinputorinfluencefromthesefirms,fundsortheirmanagers.ForfurtherinformationontheindependenceandintegrityofGartnerresearch,see“GuidingPrinciplesonIndependenceandObjectivity”onitswebsite,http://www.gartner.com/technology/about/ombudsman/omb_guide2.jsp

3Figure 1. The Ideal Approach to Data Center Energy Management

Source:Gartner(September2010)

EnergyInto Data Center

One Building

TransformerPDUUPS

Cables/Switches

LightingGenerator

Fire SuppressionChiller/Cooling

Two Electrical Facilities

Three Building Facilities

Electromechanical Efficiencies

Processor, System andLoad-Based Inefficiencies

Four Racks

Five ITHardware

Six VirtualMachines

EnergyInto Data Center

One Building

TransformerPDUUPS

Cables/Switches

LightingGenerator

Fire SuppressionChiller/Cooling

Two Electrical Facilities

Three Building Facilities

Electromechanical Efficiencies

Processor, System andLoad-Based Inefficiencies

Four Racks

Five ITHardware

Six VirtualMachines

TransformerPDUUPS

Cables/Switches

LightingGenerator

Fire SuppressionChiller/Cooling

Two Electrical Facilities

Three Building Facilities

Electromechanical Efficiencies

TransformerPDUUPS

Cables/Switches

LightingGenerator

Fire SuppressionChiller/Cooling

Two Electrical Facilities

Three Building Facilities

Electromechanical Efficiencies

Three Building FacilitiesProcessor, System and

Load-Based Inefficiencies

Four Racks

Five ITHardware

Six VirtualMachines

Processor, System andLoad-Based Inefficiencies-

Four Racks

Five ITHardware

Six VirtualMachines

Figure 2. The Pragmatic Approach to Data Center Energy Management

Source:Gartner(September2010)

Needed for real-time PUE and for hosting sites

(4) D/C Building Facilities

(5) D/C Building

Next 48 months for linking energy to workload

(6) Virtual Machines

Within the next 18 monthsWill provide 80% of needs

(1) IT Hardware

(2) Racks

(3) Electrical Facilities

TransformerPDUUPS

Cables/Switches

LightingGenerator

Fire SuppressionChiller/Cooling

Electromechanical Efficiencies

Processor, System andLoad-Based Inefficiencies

Needed for real-time PUE and for hosting sites

(4) D/C Building Facilities

(5) D/C Building

Needed for real-time PUE and for hosting sites

(4) D/C Building Facilities

(5) D/C Building

(4) D/C Building Facilities

(5) D/C Building

Next 48 months for linking energy to workload

(6) Virtual Machines

Next 48 months for linking energy to workload

(6) Virtual Machines

Within the next 18 monthsWill provide 80% of needs

(1) IT Hardware

(2) Racks

(3) Electrical Facilities

Within the next 18 monthsWill provide 80% of needs

(1) IT Hardware

(2) Racks

(3) Electrical Facilities

(1) IT Hardware

(2) Racks

(3) Electrical Facilities

TransformerPDUUPS

Cables/Switches

LightingGenerator

Fire SuppressionChiller/Cooling

Electromechanical Efficiencies

Processor, System andLoad-Based Inefficiencies

TransformerPDUUPS

Cables/Switches

LightingGenerator

Fire SuppressionChiller/Cooling

Electromechanical Efficiencies

TransformerPDUUPS

Cables/Switches

LightingGenerator

Fire SuppressionChiller/Cooling

Electromechanical Efficiencies

Processor, System andLoad-Based InefficienciesProcessor, System and

Load-Based Inefficiencies-

4Readings That Should Be Taken During the Next 18 Months

IT Hardware (Server, Networking or Storage Component)UsersneedtomeasuretheenergyattheindividualITphysicaldeviceorhardwarelevelstogetanaccuratemeasurementoftheenergythatisbeingusedbythatdevice.Thisreferstoservers,networkingdevices,storagesystemsandsubcomponents,suchasrouters,switches,etc.Mostoftheenergyatthisstagegoesintotheservers,andthesecomponentshavethemost-sophisticatedenergymanagementtoolsbuiltin.Theserversandtheworkloadsrunningonthemwillbecomethemostimportantenergymanagementcomponentsduringthenextfewyears.

Hence,it’sreasonableforuserstofocusontheserversonlyatthisstage(theenergyusedbytheotherhardwarecomponentscouldbepickedupattheracklevel).Acommon-senseapproachneedstobedeployedwithreferencetothesubcomponents,becauseitmaynotmakesensetomeasuretheenergyacrossallthehundredsofdevices.Ontheotherhand,manyofthenewerdeviceshavebuilt-inmonitoringtechnology,andthisshouldbeusedwhereappropriate.

Inbladesystems,thehardwarevendorshavetypicallydevelopedsoftwaretoolsthatprovideenergyreadingsattheindividualbladelevel.Forexample,theIBMPowerExecutivesolutionisavailableforIBMBladeCenterandSystemxservers.ItenablesdirectpowermonitoringthroughIBMDirector.HP’sInsightSoftwaresuiteofproductsdoesthesameforHPhardware.Alternatively,vendorssuchasnLyte,Emersonand1Eprovidevendor-agnosticsoftwaretools.

Measuringenergyatthecomponentlevelprovidessomekeybenefits.ItenablesuserstocomparetheenergyefficiencyofdifferentIThardwarecomponents,astheyareactuallyusedintheorganization’sowndatacenters.Thismovesawayfromrelyingpurelyonvendordataforcomponentenergyusageinformationandprovidesanopportunitytocompareausage-basedvendorwithavendorbenchmark(wheresimilardevicesfromdifferentvendorsareused).Measuringwhichdevicesusetheleastamountsofenergyisalsopivotaltoenergy-basedchargeback,and,althoughmostorganizationswillnotoperateinthiswayforanumberofyears,it’susefultostartcollectingthedatanow.OneotheradvantageofmeasuringenergydataattheITcomponentlevelisthatisprovidesanaccuratecost-of-energyreadingforahardwaredeviceoveritsusefullifetime.Thisisvaluablewhenusersupgradehardwareandwanttounderstandthetotalcostofownership(TCO)ofthedevices.

RackAstheenergyinthedatacentercomesintotherack,areadingoftheenergyshouldbetaken.ThisisgenerallyconsideredtobethefirstpointatwhichenergyinadatacentercanbeassociatedspecificallywiththeITequipmenthousedintherack.Mostoftheracksuppliers,suchasRittal,Schneider,Dataracks,Orion,etc.,nowhavesomeformonintelligentpowerstripontherackthatwillprovideanenergyreading.Moreover,supplierssuchasRaritanhaveintelligentpowerdistributionunitsthatprovideintelligent,rack-levelpowermanagementsolutions.Thehardwarevendors,suchasDell,IBMandHP,havesoftwaretoolsbuiltintotheirspecificracksthatprovidearack-levelunderstandingoftheenergybeingused.Also,asusersstarttodeployverticallyintegrated,fabric-basedsolutions,measuringenergyattheracklevelwillgiveusefulenergyreadingsforthewholestack.

Inessence,pickinguptheenergyreadingsattheracklevelisnottoodifficultandisbeingmadeeasierbyvariousbuilt-intools.UsersshouldbeactivelytakingthesereadingsattheinletoftheirITsystems.

Electrical FacilitiesWhenenergycomesintoadatacenter,itisgenerallysplitintotheelectricalfacilitiescomponentsandthebuildingfacilitiescomponents.Inthissection,theenergythatgoesintothecoreelectricalfacilitiescomponentsisexamined.Thesecomponentsincludetransformers,powerdistributionunits(PDUs),uninterruptiblepowersupplies(UPSs),cables,switches,etc.MeasuringtheenergyatthislevelisimportanttogettheoveralldatacenterusagemodelandisrequiredtogetaPUEreading.Manyhavethesecomponentshavebuilt-inmeters,sothereadingscanbeexportedintoacentralrepository.

Although,measuringenergyatthislevelwilltypicallyrequirethehelpandsupportofthefacilitiesteam,ithastwodistinctadvantages.First,theoverallelectricaldistributionatthepointatwhichenergycomesintothedatacenterwilltypicallybedesignedwithadegreeofredundancy—forexample,anN+1ora2Napproach.Thismeansthatreadingswillhavetobetakenateachoftheredundantcomponentlevelstoachieveahighdegreeofaccuracy.Thesecondadvantagedealswiththefactthateachofthecomponentswillhavebuilt-ininefficiencies,whicharetypicallyworkload-based.

Bymeasuringtheenergyconsumedonacontinuousbasisacrossthesedevices,userswillbeabletogetaclearpictureofsomesignificantpiecesofinformation.Forexample,theywillbeabletoseehowthedeviceinefficiencieschangewithworkload.Hence,ifaPDUisrunningatarangeofbetween40%and70%,continuousreadingsofenergymappedontotheworkloadofthedatacenterwillshowwhatlevelofworkloadgeneratesthehighestlevelsofelectricalefficiencyinthePDU.Althoughthiswillprobablynotleadtoanychangesinthewaythedatacenterisrun,itwillprovidevaluableinformationforPDUupgradesordesignsofanewdatacenterforthecompany.

5Readings More Appropriate for Data Center Operators and Hosting Companies

Data Center Building FacilitiesThebuildingfacilitiescomponentsincludelighting,firesuppression,generatorand,ofcourse,thecoolingcomponents.Measuringtheenergyusedherewillprovideusefuldata,becausecoolingissuchalargeconsumerofdatacenterenergy(asmuchas40%ofthetotalenergyused).Inabuildingwherethedatacenterissharedwithofficespace,specificandaccuratemeasurementiskey.Forexample,ifcoolingissharedacrossthewholebuilding,andaproportiongoesintothedatacenter/machineroom,thendifficultiesariseinaccuratemeasurement.

Dependingonthecoolingsystem,itmaybepossibletomeasuretheenergyasthecomputerroomairconditioning(CRAC)level.Ifthisisnotpossible,thensomeapportionmentofenergywillhavetobemade.However,ifthisdone,itwillbedifficulttodeterminehowenergyusageandcostvarywithworkloadandwithusersinasharedenvironment.Ingeneral,wesuggestmeteringtopicklivedata,ratherthanapportionment.Withlightingandfiresuppressionsystems,apportioningusagemaybetheappropriatebecause,basedonclientconversations,theytypicallyformlessthan5%ofthetotalenergyused.Innewdatacenters,allofthesecomponentsshouldbemeteredfromtheoutset,andthedatashouldbefedintoadashboard.

Data Center BuildingForsomedatacenteroperators,itmaybeimportanttomeasurethetotalenergycomingintothedatacenter.Thisisparticularlytrueforlargehostingcompaniesthatwanttoapportionalltheirenergycoststotheirtenants.Thisenergymeasurementmaysoundlikeastraightforwardexercise:simplystickapowermeterattheinletpointandthemeasurementisdone.However,itcanbealittlemorecomplexthanthat.Ifthedatacenterisaseparate,purpose-builtsiteandalltheenergycominginisusedsolelyforthedatacenter,thenasimplemeasurementattheinletpoint(s)isgenerallysufficient.However,ifthedatacenterispartofabuildingthathousesotherpartsofthebusiness(offices,canteens,gyms,etc.),theelectricalnetworkmayhavebeendesignedtosharecomponents,suchaslightingandcooling.

Ageneralfacilitiessectionofthebuildingmayprovidecoolingforthewholebuilding,andaportionofthatcoolingmaybeusedbythedatacenter(machineroom).Inthiscase,anumberofreadingsmayhavetobetakentogetanaccuratefigure.Thepowercomingintothedatacenterstillneedstobemeasuredandrecorded;however,areadingwillalsoneedtobetakenorcalculatedfortheproportionofenergyusedspecificallyforcooling(andpossiblylighting,althoughthismaynotbeamaterialnumber)inthedatacenter.Thiswillbeaddedtotheinletelectricityreading.Together,thesefigureswillgiveapictureofthetotalamountofenergycomingintothedatacenter.

Theproblemisthat,togetreal-timePUEreadings,theenergyforcoolinghastobemeasuredonareal-timebasis,ratherthanaone-offproportionalallocation.Thismayrequireinstallingsomemetersinolderbuildings,andnewbuildingsshouldbedesignedtogetaccurateandcontinuousreadings.VendorssuchasPanduitandSchneider(TAC)havesolutionsthatcanbeused.

Readings More Appropriate to a Workload-Based Approach

VMMostoftheenergyreadingstakenatthefivepointsoutlinedaboverelatetoIThardwareandfacilitiesdevices.Whatwillultimatelybeusefulisassociatingenergyinadatacentertotheworkloadforwhichtheenergyisused.MeasuringenergyattheVMlevelisthefirststeptowardachievingthis.Althoughuserscanestimatethisvalueoruseaproxy,suchasdividingthetotalenergyconsumedbyaserverbythenumberofVMs,actualreadingswillbecomecriticalduringthenextfewyears.Asaresult,toolsarebecomingavailablefromvendorssuchasVMwareand1E.However,mostuserprocessesremaintooimmaturetotakeadvantageofatthistime..

Duringthenextfiveyears,theincreaseduseofvirtualization,theproblemsofVMsprawlandthegrowthofcloudserviceswillbegintomoveuserstothisapproachtodatacenterenergymeasurement.Assoftwareasaservice(SaaS)becomesavailableacrossarangeofhighlyvirtualizedhardwaredevices,measuringacrossVMswillbecomeincreasinglyimportantinunderstandinghowenergyisdistributedacrossdifferentworkloads.

Measuringenergyacrossthedatacenterisacomplex,butincreasinglynecessaryactivity.AnumberofvendorsarecompetinginthisDCIMspace,andusersarebeginningtoadopttheirtechnologies.Here,thesixidealareasformeasuringthatenergyhavebeenprioritizedintoamore-practicalscenario.Weadviseuserstoadoptthispragmaticapproachandstartmeasuringenergyusageacrosstheirsites.