1

Energy Efficiency in Data Centers

“What matters most to the computer designers at Google is not speed, but power - low power, because data centers can consume as much electricity as a city” – Eric Schmidt, CEO of Google

Diljot Singh Grewal

2

Some Facts•Data centers consumed 235 billion KWH of

energy 2 worldwide(2010).•Datacenters consumed 1.3% of total

electricity consumption of the world(as on august 2011)

•In 2000 DC used 0.53% , which almost doubled to 0.97% in 2005, by 2010 it rose only to 1.3%

•A rack drawing 20 KWH at 10cents per KWH uses more than 17000$ in electricity.

3

Energy Efficiency

• Run a DC wide workload and measure energy consumed

4

Power Usage Effectiveness (PUE)

• In 2006, 85 % of DC had PUE of greater than 3.0. 5

• Another study estimated it at 2.0 6

• In the state of Art Facility the PUE of 1.1 is achievable.7

5

Reasons:•Staged Deployment•Fragmentation•Following Nameplate Ratings•Variable Load•Excessive/Inefficient Cooling•Excessive/ Inefficient humidity controls…

6

8

7

8

6-12% loss

115kV to 13.2kV Loss

~0.5%

Loss in Wires ~1-3%

Chillers consume 30 – 50% of IT Load.

CRAC units consume 10-30% of IT Load

9

8

10

8

11

Improving Infrastructure•Increasing Temperature to 27 ◦C from 20◦C.•Isolate hot exhaust air from intake•Using High Efficiency UPS and other gear•Google Achieved a PUE of 1.1 by 9

▫Better air flow and Exhaust handling.▫Temperature of Cold Aisle at 27 ◦ C▫Cooling Tower uses Water evaporation▫Per server UPS that has Efficiency of 99.99%

instead of facility wide UPS

12

Google’s PUE over the years

13

Humidity Control•Condensation on Cooling coils can reduce the

humidity•Low (<40% rH) humidity levels can lead to

static buildup (sparks that can damage chips).•Steam Humidifiers are Energy Expensive•Energy Savings??

▫Using evaporative cooling on incoming air .▫Using evaporative cooling to humidify the hot

output air and cool it( which is then used to cool the incoming air)

14

SPUE

•Losses due to power supplies, fans, voltage regulators

Maximum EfficiencyPower supplies 80%Motherboard VRM 70%

𝑇𝑜𝑡𝑎𝑙 𝑃𝑈𝐸=𝑃𝑈𝐸∗𝑆𝑃𝑈𝐸• If both stand at 1.2 then only 70% of the

energy is actually used for computation.

15

18

16

Efficiency of Computing

•Hardest to measure. How Do we Benchmark?

•New benchmarks : Joule-sort and SPEC power

•No benchmarks for Memory or Switches

𝐶𝑜𝑚𝑝𝑢𝑡𝑎𝑡𝑖𝑜𝑛𝑇𝑜𝑡𝑎𝑙 𝐸𝑛𝑒𝑟𝑔𝑦 𝑡𝑜𝐸𝑙𝑒𝑐𝑡𝑟𝑜𝑛𝑖𝑐𝐶𝑜𝑚𝑝𝑜𝑛𝑒𝑛𝑡𝑠

17

Breakdown

18

CPU•Uses up to 50% at peak but drops to 30%

at low activity•Dynamic Ranges

▫CPU 3.5x▫Memory : 2x▫Disks 1.3x▫Switches 1.2x

19

20

10

21

10

22

Energy Proportional Computing.•Low Idle Power and proportional

afterwards•energy spent will be halved by energy

proportionality alone if the system idles at 10%.11

•Might be fine if peak is not that good

23

10

Load level(%) of peak

24

Savings by Energy proportional computing (green line)

11

25

Dynamic Voltage and Frequency Scaling

•The time to wake up from low voltage state depends on voltage differential

•Not useful on Multicore Architectures?

26

The CPU States•ACPI States:

▫Power management component of kernel sends a signal to the Processor Driver to switch to a state

•States:▫C0 Normal Operation▫C1 ,C2: Stops Clocks▫C3 : C2+ reduced Voltage▫C4 : C3 + Turns off memory Cache

27

Mode Name What it doesC0 Operating State CPU fully turned on

C1 Halt Stop main Internal clock via Software, bus and APIC keep running

C1E Enhanced Halt C1 + reduced Voltage

C2 Stop Grant / Stop Clock

Stops clock via Hardware. Bus and APIC Keeps running

C2E Extended S.C. C2 + Reduced Voltage

C3 Sleep Stops clock (Internal or both)

C4 Deeper Sleep Reduces CPU Voltage

C4E/C5 Enhanced Deeper Sleep

Reduces CPU voltage even more and turns off the cache

C6 Deep Power Down Reduces voltage even more(~0V)

28

12

29

12

30

Energy Savings10

31

Results of scaling at Datacenter Level11

32

Results of scaling at Datacenter Level11

33

The Multicore problem•Clock Gating

▫Core level Clock gating•Voltage Gating?

▫Voltage depends on core with high utilization•Lower Wake Up Penalty by using the Cache

▫New architectures have penalties of 60µs down from 250µs.

•Power Gating (Power Control Unit)•Separate Power planes for Core and Un-

core part

34

The Leakage power

35

Software’s Role•Well Tuned Code can reduce the

consumption.•Code that generates excessive interrupts

or snoop requests is not good.•OS Power Manager speculates the future

processing requirements to make a decision according to the settings selected by user.

36

CPU isn’t the only culprit10

37

38

Lets talk Storage•Consumes about 27% power•High Performance Disks to match the µP

Speed•According to IDC report in 2008, total cost

to power and cool a drive is 48 watts. 13

▫ 12 watts for running HDD▫12 watts for storage shelf (HBAs, fans, power

supply)▫ 24 watts to cool the HDDs and storage shelf

39

Power Consumption of a 2.5” drive

40

Electronics & Software•Adaptive Voltage•Frequency Reduction in Low Power

Modes

•Queuing Algorithms to minimize rotational delays

•Algorithm to manage transitions between low and high power modes

41

Mechanical•Lighter Materials•Better motor Design•Using Helium in a sealed case to reduce

air drag▫WD claims energy savings of 23% with

higher capacity(40%)•Load/Unload

42

Tiered System 14

•Manage workloads efficiently among multiple RPMs in a storage system

•Tiered storage▫Tier 0 with solid state drives (5%),▫ Tier 1 with high performance HDDs (15%) ▫Tier 2 with low power HDDs (80%)

43

Tiered Storage

44

Mixed Approach•Mirror HP Disk on Low Power Disk and

use the low power disk under light load.14

•The Low performance disks use significantly low energy than HP Disks.

•Other approaches▫Multispeed Disks: ability to change spin

speed.14

▫Lower Rotational speed but multiple heads

45

Solid State Disks•require up to 90% less power 15

•offer up to a 100 times higher performance 15

•Life span of the SSD depends on the I/O ops and it is not good enough for server yet.

•MLC vs. SLC

46

File system problems?

•Google File system: ▫Distribute data chunks across large

number of systems (entire cluster) for resiliency..

▫But that means all machines run at low activity and do not go idle.

47

Memory•SRAM: Requires constant voltage•DRAM : Since capacitors leak charge, we

need to refresh them every 64 ms (JEDEC)

•Suppose we have 213 rows, then we need to refresh a row every 7.8µs.

48

Alternatives•Low Voltage RAM (LoVo)

▫Runs at 1.25V (DDR2 -1.8V and DDR3 - 1.5V)

▫2-3W per RAM(2GB)•SSD as RAM17

•Future:▫Ferroelectric RAM▫Magnetoresistive RAM (MRAM)

49

Is Performance Per Watt all we need?•If it is, then we should Buy ARM Servers.•Smaller RAM and Laptop HDD’s•20 times lower power but at 5 times lower

performance : High Response times.•Acc. to Google’s Study, The users prefer

10 results in 0.4 sec over 25 in 0.9 sec.

Are few ‘Bulls’ better than a flock of ‘Chickens’?

50

Power Provisioning Costs• Building a Datacenter that can provide

power to servers can be costlier than Electricity costs.

• $10-22 per deployed IT Watt(provisioning cost)

• Cost of 1 Watt of IT Power =• (per year per watt)

• Cost savings from efficiency can save more in provisioning.

5198%,93%,7.5%

100%,90%,11%

92%, 86%, 16%

Peak 85%, slack 17%

52% - 72%,39%

11

52

Safety Mechanism and over subscription

• Since CDF intercepts the top at a flat slope ▫ few intervals when close to full load

• Remove these intervals – even more machines▫De-scheduling tasks▫DVFS (also Power Capping)

53

Virtualization•the energy cost can be minimized by

launching multiple virtual machines.•Virtualized servers have an associated

overhead•Different Types have different behaviors

▫Para virtualized (XEN)▫Full Virtualization(VMware Server)

54

Para Virtualization (XEN)

55

Virtualization Overheads

L : Native Linux X : XEN V: VM-Ware workstation 3.2 U: User mode Linux

16

56

Performance on Java Server Benchmark

Virtualization Performance on SPECjbb 2005Number of VMs

CPUs per VM SUSE SLES 10 Xen 3.0.3

VMWare ESX 3.0.2

1 1 1% 3%1 4 3% 7%4 2 5% 15%4 4 7% 19%

16

57

Power Management in Virtualized Systems

58

Concluding:• Power efficiency in datacenters is constrained

by the performance requirements imposed.•High efficiency gear, Smart design and proper

consolidation can lead to huge gains•Efficiency in server components is an ongoing

research problem.•Data Centers have many components that

affect the overall consumption and synchronization across them is needed to ensure performance and efficiency.

59

References1. Morgan, T. P. (2006, February 28). The server market begins to cool in Q4. The Linux Beacon. 2. EPA Report in 20063. Hiller, A. (2006, January). A quantitative and analytical approach to server consolidation.

CiRBA White Paper, p. 4.4. Personal correspondence with Dale Sartor of LBNL (August 9, 2006).5. M. Kalyanakrishnam, Z. Kalbarczyk, and R. Iyer, “Failure data analysis of a LAN of Windows NT

based computers,” Reliable Distributed Systems, IEEE Symposium on, vol. 0, no. 0, pp. 178, 18th IEEE Symposium on Reliable Distributed Systems, 1999

6. Green Grid, “Seven strategies to improve datacenter cooling efficiency”.7. X. Fan, W. Weber, and L. A. Barroso, “Power provisioning for a warehouse-sized computer,” in

Proceedings of the 34th Annual International Symposium on Computer Architecture, San Diego, CA, June 09–13, 2007. ISCA ’07

8. S. Greenberg, E. Mills, and B. Tschudi, “Best practices for datacenters: lessons learned from benchmarking 22 datacenters,” 2006 ACEEE Summer Study on Energy Efficiency in Buildings.

9. Google Inc., “Efficient Data Center Summit, April 2009”.10.Luiz André Barroso, Urs Holzle, The Data Center as a Computer: An Introduction to the Design

of Warehouse, 2009 , Morgan & Claypool Publishers.11.X. Fan, W. Weber, and L. A. Barroso, “Power provisioning for a warehouse-sized computer,” in

Proceedings of the 34th Annual International Symposium on Computer Architecture, San Diego, CA, June 09–13, 2007. ISCA ’07

60

References12. Technology brief on Power Capping in HP Systems. Available at

http://h20000.www2.hp.com/bc/docs/support/SupportManual/c01549455/c01549455.pdf

13. International Data Corporation, Annual Report, 200814. Enrique V. Carrera, Eduardo Pinheiro, and Ricardo Bianchini, Conserving Disk

Energy in Network Servers, International Conference on Supercomputing,200315. “Solid State Drivers for Enterprise” Data Center Environments Whitepaper HGST16. Paul Barham , Boris Dragovic, Keir Fraser, Steven Hand, Tim Harris, Alex Ho, Rolf

Neugebauer , Ian Pratt, Andrew Wareld, "Xen and the Art of Virtualization", University of Cambridge Computer Laboratory

17. Anirudh Badam and Vivek S. Pai, SSDAlloc: Hybrid SSD/RAM Memory Management Made Easy , 8th USENIX conference on Networked systems design and implementation ,2011, Pg 16

18. M. Ton and B. Fortenbury, “High performance buildings: datacenters—server power supplies,”

Lawrence Berkeley National Laboratories and EPRI, December 2005.

61

ARM Server (Calxeda)•More of a cluster in size of a server•Currently holds 12 Energy Cards (in 1

server)•Each Energy card has 4 Energy cores(1.1

– 1.4 GHz)•Larger L2 Cache•Runs Linux (Ubuntu Server 12.10 or

Fedora 17)•Don’t need to virtualize but give each

application its own node (Quadcore, 4MB L2 4GB RAM)

62

ECX 1000 is ARM Server, others are Intel