This presentation was given during the Spring, 2012 Data Center World Conference and Expo. Contents contained are owned by AFCOM and Data Center World and can only be reused with the express permission of ACOM. Questions or for permission contact: jater@afcom.com.

Interested in data center power and cooling?

Learn about data center efficiency/power & cooling sessions offered at the upcoming Fall 2012 Data Center World Conference at:

www.datacenterworld.com.

Lowering Operating Costs Through Cooling System Design

Paul Bemis

President

Applied Math Modeling Inc

2

Data Center PUE(Power Utilization Effectiveness)

PUE =

Total Facility Energy

IT EnergyHow effectively is my facility delivering

cooling to my IT equipment?

Total FacilityPower

IT Equipment• Servers• Storage• Telco• etc

3

PUE and DCiE

Power Utilization Effectiveness (PUE)

PUE = Total Facility Energy / Total IT Energy

Data Center Infrastructure Efficiency (DCiE)

DCIE = (Total IT Energy/ Total Facility Energy) x100%

4

• Dominant Parameters are IT and Cooling Energy.

– CRAC/Chiller make up 75% of total “non-IT” load • UPS, PDU, Switchgear are all 2nd order effects

Some Facts About PUE

Source: Uptime Institute5

Data Center PUE(Power Utilization Effectiveness)

• Focusing on Cooling Efficiency provides largest payback• Understanding how cooling systems operate is key

PUE =

Cooling Energy + IT Energy

IT Energy

How effectively is my facility delivering cooling to my IT equipment?

6

• Cooling Units (heat pumps)– Use mechanical shaft power provided by an electric motor

to perform the work necessary to move heat from one location (inside) to another (outside).

– The Coefficient of Performance (COP) of a heat pump is the ratio of the heat pumped (moved) to the supplied work.

– The COP for Data Center Cooling ranges from 2-5*

Cooling System Energy Efficiency

COP = ∆Q∆W

* Patnaik, Marwah, Sharma, Ramakrishnan Department of Computer Science, Virginia Tech, “Sustainable Operation and Management of Data Center Chillers”

Thermal Load In Room

Energy Required to Drive Heat Pump

7

Data Center PUE(Power Utilization Effectiveness)

• The COP of the data center cooling system is the dominant parameter in the PUE calculation.

• The COP can be easily measured in existing data centers.– Ratio of total IT load/ ATS switch load

PUE =

IT Energy/COP + IT Energy

IT Energy

PUE =

1COP + 1 = > Good Approximation

8

How does COP relate to “Set Points”?

• Increasing Cold Air Supply increases COP at a rate of 3.5% for every 1.8F

From ASHRAE 90.1-204, Table 6.8.1 I 9

Schematic and T-s Diagram for Ideal Vapor-Compression Refrigeration Cycle

Reducing compressor “Lift” will reducing energy consumption..10

Techniques for Improving the Efficiency of Data Center Cooling Systems

• Maximize the temperature difference across all heat exchangers– Increase hot air return temperature in data center

• Increase the cold air supply temperatures as high as possible, while maintaining ASHRAE inlet temperature guidelines

11

Maximizing Heat transfer in the Data Center

• Each CRAC/CRAH utilizes a fan/coil system for transferring heat from the data center.

• The governing equation for heat transfer is:

)( SRP TTcmQ −= Heat

Mass Flowrate Specific Heat of air

• It is important to maximize the temperature difference across the coil.

• It is also important to balance the mass flow rates between the IT load and the CRAC/CRAH units

12

Steps to optimize cooling efficiency(and reduce operating costs)

• Determine the total amount of airflow required to satisfy IT thermal load.– Use 156 CFM/KW as a ‘rule of thumb’

• Strive to reduce the supply airflow to match only what is needed by the servers– Maximizes return air temperatures and saves

operation costs on fan energy.

• Begin to increase air supply temperature until worst case rack/server reaches 80.6F– Use containment to unify rack inlet temperatures

13

or

x 100Rack Flow Rate

Air Handler Flow RateReturn Temperature Index (RTI) =

Return Temperature Index (RTI) is a Trademark of ANCIS Incorporated (www.ancis.us). All rights reserved. Used under authorization

Air Handler Delta TRack Delta T

x 100Return Temperature Index (RTI) =

RTI is a measure of net by-pass air or net recirculation air in the data center

14

ASHRAE Specifications:

AllowableRecommended 64.4 – 80.6 F 59 – 89.6 F

Rack Cooling Index (RCIHI) =1 -

Max Allowable Over-Temp

Total Over-Tempx 100

Rack Cooling Index (RCI) is a Registered Trademark of ANCIS Incorporated (www.ancis.us). All rights reserved. Used under authorization

1 - Max Allowable Under-Temp

Total Under-Temp x 100Rack Cooling Index (RCILO) =

RCI is a measure of compliance with the ASHRAE thermal guideline

15

Example Data Center

• Data Center is a raised floor design of 1600 sq ft, 145 kW of IT load, and natural convection return.

• DX based perimeter downflow cooling capacity is 325kW, 255kw nominal (N+1).– Ratio of 1.76x required cooling capacity– Controlled individually, and are constant volume devices

• Interested in reconfiguring data center to improve efficiency– Lower operational costs– Increase “headroom” for future IT load

• Would like to predict the ROI of proposed changes:– Cold/hot aisle containment– VFD’s to improve the balance of supply/return airflow.

16

Analysis Outline

• Perform baseline analysis on existing equipment– Currently uses 4 Lieberts (2-DH412W, 2-DH315W)

• Validate CFD Model• Perform baseline energy calculations

• Study alternatives for optimizing RTI– Investigate shutting down CRAC units or implementing VFD’s– Investigate the need to add containment curtains

• Begin to increase cold air supply temperature– Watch racks inlets until worst case temperature reaches 80F

17

Baseline Model Using Factory Specs

18

Model Validation

)( SRP TTcmQ −=

Crac Number Flowrate (CFM) Measured Predicted Measured Predicted1 12,000 61.2 61 72 692 12,000 60.1 60 73 663 15,200 56.5 57 74 694 15,200 65 65 74 73

54,400 60.7 60.75 73.25 69.25

Supply Temp Return TempFactory Specifications

Crac Number Flowrate (CFM Measured Predicted Measured Predicted1 10,000 61.2 61 72 742 9,000 60.1 60 73 683 10,000 56.5 57 74 714 10,500 65 65 74 76

39,500 60.7 60.75 73.25 72.25

Base Case SpecificationsSupply Temp Return Temp

19

Specifications Room dimensions 28 X 58.5 X 10 feet Room floor area 1625.5 sq. ft. Supply plenum height 1.5 feet Number of downflow CRAC units 4Number of rack rows 20Tota l number of racks 59Number of ti le rows 10Tota l number of ti les 59

Airflow Assessment Supply a ir flow rate from downflow CRACs 39500 CFM Demand a ir flow rate from racks 22559 CFM Tota l demand a i r flow rate 22559 CFM Supply a ir flow ratio(%) 175% (75% excess of the demand a i r flow rate.) Es timated average flow rate through perforated ti les 669 CFM Estimated average pressure drop across the ti les 0.281 lbf/ft2

Thermal Assessment Estimated heat dens i ty 89.02 W/sq.ft. of room area Es timated downflow CRAC cool ing capaci ty 254.94 kW Estimated supply a ir tota l cool ing capaci ty 254.94 kW Estimated tota l rack heat load 144.7 kW Tota l IT cool ing load 144.7 kW

Cool ing capaci ty to heatLoad ratio(%) 176% (76% excess than cool ing demand) Es timated temperature ri se of supply a i r 11.35 F Es timated average temperature of hot a ir 71.35 F (for 60 F supply a ir temperature)

Base Case Colo Facility

Energy Savings Opportunity

20

144 KW Base Case Selected Results

Nearly 2x the amount of airflow required

No racks out of ASHRAE High Temp guidelines

Baseline PUE estimate

More than half out of ASHRAE Low Temp guidelines

Costing as much to cool servers as to power them…

Base Case 144 KW DX

IT Heat Load 144.7 kW5% IT Infrastructure 7 kWTotal IT Heat Load 152 kW

Total Cooling Power 143 kWTotal Supply Air Flow 39,500 CFM

Total Demand Air Flow 22,559 CFMFan Power 67 kW

Average Supply Temp (F) 60.75 FCOP 1.53RTI 57%

RCI hi 100%RCI lo 44%

Total Facility Power 362 kWPUE 2.38

Assumed Cost of Electricity 0.08 $/kW-HrAnnual Cost 253,751 $

21

ASHRAE Conformance Plot

22

Motives for Change• RTI of existing data center is 57%, indicating nearly 100%

bypass airflow.• Current DX units do not support VFD, so reducing supply

airflow is limited to shutting off CRACs– Would require backflow prevention and ICON control

• Moving to new DX CRAC’s will provide the following benefits:– Allows VFD’s plug fans so demand/supply airflow match (RTI = 100%)– Provides increased upside cooling capacity– Increases COP by a factor to at least 3

• Which significantly reduces the energy consumption of cooling system.

Modeling can be used to accurately predict results prior to implementation…23

Step 1: 144KW Contained Case Results Fans @ 80%

24

Step 1: 144KW Contained Case Results Fans @ 80%

25

Step 1: 144KW Contained Case ASHRAE Conformance

26

Step 1: New DX UnitsFans @80% and Contain Cold Aisle

Lowered by 20%

Lowered by 40%

Improved by 23%

RCI dropped, racks too cold

No racks exceed high temps

PUE headed in right direction

Improved by factor of 2x

Can we drive up Air Supply Temp?

Base Case 144 KW DX

Contained Case 144 KW DX

IT Heat Load 144.7 144.7 kW5% IT Infrastructure 7 7 kWTotal IT Heat Load 152 152 kW

Total Cooling Power 143 64 kWTotal Supply Air Flow 39,500 32,000 CFM

Total Demand Air Flow 22,559 22,559 CFMFan Power 67 40 kW

Average Supply Temp (F) 60.75 60.75 FCOP 1.53 3.00RTI 57% 70%

RCI hi 100% 100%RCI lo 44% 34%

Total Facility Power 362 255 kWPUE 2.38 1.68

Assumed Cost of Electricity 0.08 0.08 $/kW-HrAnnual Cost 253,751 179,027 $

Projected Savings 74,724 $% Savings per Year 29%

27

144KW Contained Case Results Fans @ 80% Ts=70F

28

Step 2: Lower Fans 20%, Contain Cold Aisle, and Increase Supply Temp to 70F

No racks exceed low temp

No racks exceed high temps

PUE continues to improve

Base Case 144 KW DX

Contained Case 144 KW DX

Contained Case 144KW 70F

IT Heat Load 144.7 144.7 144.7 kW5% IT Infrastructure 7 7 7.2 kWTotal IT Heat Load 152 152 151.9 kW

Total Cooling Power 143 64 54 kWTotal Supply Air Flow 39,500 32,000 32,000 CFM

Total Demand Air Flow 22,559 22,559 22,559 CFMFan Power 67 40 40 kW

Average Supply Temp (F) 60.75 60.75 70.00 FCOP 1.53 3.00 3.54RTI 57% 70% 70%

RCI hi 100% 100% 100%RCI lo 44% 34% 100%

Total Facility Power 362 255 246 kWPUE 2.38 1.68 1.62

Assumed Cost of Electricity 0.08 0.08 0.08 $/kW-HrAnnual Cost 253,751 179,027 172,204 $

Projected Savings 74,724 81,547 $% Savings per Year 29% 32%

Can we improve RTI a little more??29

Step 3: Lower Fans 37%, Contain Cold Aisle, Supply Temp 70F

30

Step 3: Lower Fans 37%, Contain Cold Aisle, and Increase Supply Temp to 70F

No racks exceed low temp

Some racks exceed high temps

PUE continues to improve

RTI has improved 20%

Lowers overall operating costs by 39%

Base Case 144 KW DX

Contained Case 144 KW DX

Contained Case 144KW 70F

Contained Case 144KW 70F Lower

Fans 37%IT Heat Load 144.7 144.7 144.7 144.7 kW

5% IT Infrastructure 7 7 7.2 7.2 kWTotal IT Heat Load 152 152 151.9 151.9 kW

Total Cooling Power 143 64 54 49 kWTotal Supply Air Flow 39,500 32,000 32,000 25,000 CFM

Total Demand Air Flow 22,559 22,559 22,559 22,559 CFMFan Power 67 40 40 21 kW

Average Supply Temp (F) 60.75 60.75 70.00 70.00 FCOP 1.53 3.00 3.54 3.54RTI 57% 70% 70% 90%

RCI hi 100% 100% 100% 97%RCI lo 44% 34% 100% 100%

Total Facility Power 362 255 246 222 kWPUE 2.38 1.68 1.62 1.46

Assumed Cost of Electricity 0.08 0.08 0.08 0.08 $/kW-HrAnnual Cost 253,751 179,027 172,204 155,788 $

Projected Savings 74,724 81,547 97,963 $% Savings per Year 29% 32% 39%

31

Step 3: Lower Fans 37%, Contain Cold Aisle, Supply Temp 70F

32

Analysis Summary• The combination of VFD’s and Cold Aisle Containment

yield the following benefits– Deploying newer technology DX units and lowering fans speeds

improves RTI, RCI, PUE, and reduces energy costs considerably.– Installing cold (or hot) aisle containment improves air

management and allows cold air supply temperature to be increased, which improves cooling efficiency

• Doing both these things reduces overall energy consumption by an estimated 39%, saving ~$100k per year in operational costs.

• Estimated cost to implement these changes is $300k, providing a ROI of 3 years (excluding energy rebate credits)

33

• Questions Regarding This Presentation and/or Modeling Methods– Paul Bemis

– Paul.Bemis@CoolSimSoftware.com

• For Information, please contact:– Applied Math Modeling Inc,– Jennifer Beliveau

– Jennifer.Beliveau@CoolSimSoftware.com

For more information

34

This presentation was given during the Spring, 2012 Data Center World Conference and Expo. Contents contained are owned by AFCOM and Data Center World and can only be reused with the express permission of ACOM. Questions or for permission contact: jater@afcom.com.

Interested in data center power and cooling?

Learn about the data center efficiency/power & cooling sessions offered at the upcoming Fall 2012 Data Center World Conference at:

www.datacenterworld.com.