1

Liquid Cooling Guidelines

Michael K Patterson, PhD, PE, DCEP

Data Center and Connected Systems Group

2

What’s

Next?

Ice Cooled System

(Circa 1890) General Electric Room Cooler

(Circa 1932) Computer Room Air Conditioner

(Circa 1970)

ASHRAE Overview

1. ASHRAE (American Society of Heating Refrigerating, & Air conditioning Engineering) formed in 1894 is a technical society; specializing in cooling.

2. ASHRAE has over 50,000 members & has members in 135 countries

3. ASHRAE focuses on maintaining an unbiased role within the industry

4. ASHRAE actively writes standards, guidelines, model codes, etc.

3

ASHRAE and Datacenters

• ASHRAE – American Society of Heating, Refrigeration and Air-conditioning

Engineers

•Technical Committee 9.9 – Mission Critical Facilities, Technology Spaces and

Electronic Equipment – Datacom Equipment producers,

(manufacturers of computer hardware, HVAC equipment, software vendors, etc)

– User of Datacom Equipment

(facility owners, operators, managers, etc)

– General Interest

(government agencies, utilities, consultants, academia, testing laboratories, etc.)

• Mission Statement: To be recognized by ALL areas of the datacom

industry as the unbiased engineering leader in HVAC and an effective

provider of technical information for the datacom industry.

4

ASHRAE Datacom Book Series

5

2011 ASHRAE Air-Cooled Thermal Guidelines

A1

A2

A3 A4

ASHRAE – Recommended

*Envelopes Represent Conditions at IT Equipment Inlet

Degrees C 27 32 35 40 45 Class R A1 A2 A3 A4

Why not something similar

for liquid cooling?

6

ASHRAE Datacom Book Series

7

2006 book now supplemented by 2011 WP

8

2011 ASHRAE Liquid Cooling Guidelines

• Drivers for More Detailed Liquid Guidelines – Improve and Increase Liquid

Cooling Use and Reduce Confusion among users and suppliers

• HPC Challenges with more and more power per node and per rack, pushing

to and past air-cooling limits

• Range of cooling architectures not well aligned to IT requirements

• ASHRAE TC 9.9 Response – Enable Opportunity for Innovation

• Defined liquid cooling classes with temperature ranges

• Created guidance to effectively implement liquid cooling through new

White Paper

• ASHRAE TC 9.9 IT Subcommittee: Bull, Cisco, Cray, Dell, EMC, Fujitsu, HP,

IBM, Intel, Juniper, Lucent, Nortel Networks, Oracle/Sun, Seagate, SGI,

Teradata

Cross HPC-IT Team collaboratively authors WP on liquid cooling

9

What is liquid cooling? Which liquid are we talking about?

10

Classes

Typical Infrastructure Design Facility Supply

Water Temp (C) IT Equipment Availability Main Cooling

Equipment

Supplemental Cooling

Equipment W1

Chiller/Cooling Tower Water-side Economizer

Chiller

2 – 17 Now available

W2 2 – 27

W3 Cooling Tower Chiller 2 – 32 Not generally

available, dependent on future demand

W4

Water-side Economizer (with

drycooler or cooling tower)

Nothing 2 – 45

W5 Building Heating

System Cooling Tower

> 45

Specialized systems

Required Cooling Infrastructure: Balance of

Silicon/Datacenter

2011 ASHRAE Liquid-Cooled Thermal Guidelines

11

ASHRAE Classes

12

More from the White Paper

Operational Considerations

• Condensation

• Flow and differential

pressure limits

• Class specific limits

Water flow rates vs capacity

Velocity Limits

• By size / material

System Design

• Materials

• Connections

• Heat Rejection devices

Water Quality

• Corrosion

• Fouling

• Scaling

• Micro

Bibliography

13

As advertised earlier… don’t miss these

Birds of Feather: "'Hot' for Warm Water Cooling"

• Tuesday, 5:30-7:00PM WSCC 613/614

State of the Practice: "'Hot' for Warm Water Cooling"

• Wednesday, 4:00-4:30PM TC 202

14

Wet and Dry Bulb Temperatures

ASHRAE CD, 99.6% of yearly hours, National Laboratory HPC Locations.

60

65

70

75

80

85

Tem

pe

ratu

re (°

F)Wet Bulb Temperature - 99.6% of hours per ASHRAE CD - U.S. National Laboratories

Selected Max. = 79.7 °F (26.5°C)

16

18

21

24

27

29

Tem

pe

ratu

re (°

C)

75

80

85

90

95

100

Tem

pe

ratu

re (°

F)

Dry Bulb Temperature - 99.6% of hours per ASHRAE CD - U.S. National Laboratories

Max. = 99.5 °F (37.5°C)38

35

32

24

27

29

Tem

pe

ratu

re (°

C)

15

Air Cooled Server Dry Cooler or Cooling Tower

Using Dry Cooler Only Water Temp. Supply to Building

109°F (43°C)

Chip Thermal Margin 7°F (4°C)

Using Dry Cooler Only

Chip Thermal Margin 26°F (15°C)

Using Cooling Tower Only

Using Cooling Tower Only Water Temp. Supply to Building

89°F (32°C)

16

EEHPCWG ASHRAE

L1 17 °C Legacy systems w/ chiller 2-17 °C

W1

Cooling tower & Chiller based Rack level cooling

2-27 °C

W2

L2 32 °C

Cooling tower & Chiller based Rack/Component level cooling

2-32 °C

W3

L3 43 °C

Dry cooler based Component level cooling

2-45 °C

W4

Heat re-use opportunity Component level cooling

>45 °C

W5

Infrastructure In

IT Out

Two liquid cooling activities meeting in the middle

17

Thank You! Questions?