Holistic Approaches to Green IT
Dr Hugh Beedie
CTO
ARCCA & INSRV
Cardiff University
Going Green Can Save Money
Introduction What is Green IT ? Drivers & Metrics Data Centres, Power & Cooling Servers, Virtualisation, and Storage PCs, Monitors & Thin clients Printing and Networks Purchasing policies Influencing user behavior & the organisation ‘Thinking outside the box’ – save energy with
Condor
What is Green IT ?
No environmental impact Get real
Use of resources in an efficient way Better
Avoid ‘Greenwash’ Often faulty comparisons
Importance of your Context Your situation, drivers & metrics… Can change over time
Drivers – Why do Green IT?
Increasing demand for CPU & Storage Lack of Space Lack of Power Increasing energy bills (oil prices doubled) Enhancing the reputation of your University &
attracting better students Sustainable IT & Legal Compliance Because we should (for the Planet )
Metrics – Chicken and Egg
You can’t measure everything.So, which numbers do you
measure, when you don’t know which are important ?
Guesstimation
Which use more, PCs or Servers ?10,000 PCs using
150W, 8 hrs/day, 220 days/yr UK electricity £300,000 p.a.
500 servers using 300W + losses (X 2) 24 X 7 UK prices £300,000 p.a.
So where do we start ?
Examples of Metrics
Cost (Watts, TCO, capital investment, etc)Value (CPU Hrs, MFlops, Gbps, Papers)Value/Cost is better ? (MFlops/Watt)Cooling & power losses (PUE)Average kWhr per PC per yearPages printed (or not printed ?)Cost per page (electricity, paper)Space saved
Power Usage Effectiveness
PUE=Total Power/Computer Power
The DetailWhat wastes the power ?
Power Conversion in HV Transformer Before it gets to your room, you lose power.
Transformer Efficiency = 98% or 95% Return On Investment (ROI)?
Capital Cost = £5500 or £5000 (650kVA) New installation (@200kW), ROI = 1 month Replacement, ROI = 1 year Lifetime of investment = 20+ yrs !!!!!
Data Centre Consumption
Powerdelivery40%
80% 100%
Loads
Cooling
Cum
ulat
ive p
ower Bull View
Source: Intel Corp.
7.3%
Voltage Regulators
20W
5.5%
Server fans 15W
7.3%
UPS +PDU 20W
18.2%
PSU 50W
36.4%Load
CPU, Memory,Drives , I/O
100W
25.5%
Room cooling system
70W
Total 275W
Intel View
Cooling Inside the RoomHot Aisle/Cold Aisle Layout
Hot Aisle Containment
Hot Aisle Ceiling Tiles/Cable Trough
Seals in hot air, prevents mixing with room air Chamber
DoorsAccess to hot aisle, locks for security. Prevents hot air mixing
InRow RC unit
InRow air conditioner cools hot exhaust air directly
Rack Air Containment Airflow Diagram
Server exhaust air is prevented from escaping by rear containment
All exhaust air is returned to InRow cooling unit
Fans of InRow cooling unit distribute cool air to front of IT equipment
Top Down View
Front
Rear
InRow Cooling
Unit
InRow Cooling
Unit
NetShelter SX Rack
Servers
Rear Containment
Front Containment
APC Inline RC units • Up to 20kW/rack• Uniform cooling allows raising cold air temp
Cooling Inside the Room
Temperature (deg C)
<24 27.4 30.8 34.3 >37.8
Cooling – Outside the Room
3 Airedale Chillers
Ultima Compact Free Cool 120
Quiet model
Variable speed fans
N+1 arrangement
(We needed that )
And in real life (2008)
Free-cooling
Cooling Load
System operated on 100% Free-Cooling (12% of Year)
System operated on partial Free-Cooling (50% of Year)
System operated on mechanical cooling only(38% of Year)
-7oC3.5oC
12.5oCAmbient Temp.
Mechanical (compressor) coolingFree-cooling
Back Up Power
Full and half load efficiency >92%
Scalable & Modular – could grow as we grew, keeping %max load high
APC 160kW Symmetra UPS
UPS Efficiencies
High Density Servers
Important if you spend on advanced cooling methods
Can have shared power supplies (PSU) Higher efficiency
Blade Servers have shared PSUs Not always good for HPC
Bull supplied half-width, 1U servers. 2, quad core, 80W Harpertown CPUs
Merlin Cluster
Cardiff’s PUE vs Temperature
Average PUE = 1.3
No correlation with temperature ?
Remove 1 120kW Free cooling chillerAdd 1 300kW Evaporative chiller
Expansion for HPC Wales
Key Power Saving Solutions
Low power servers
High Density Servers (sweating the asset)
Modular UPS
(not over provisioned)
Cooling Infrastructure
Chilled Water“Air Handling”
HVTransformer
98% efficiency
Virtualisation and Storage
Virtualisation Many virtual servers in one physical server
Key questions How much can you save ? How much of that is energy ?
Storage Fast disk=450GB, 25W, Slow disk=2TB, 15W Tiering = Energy saving
Business CaseBefore Virtualisation
Equipment FY Unit Total
100 RM servers - replacement cost, inc installation and peripherals
(assumes Dell PowerEdge 1950)
08/09 £2700 £270K
100 ‘Out of warranty’ servers – 2 year extended warranty 08/09 £500 £50K
250 Production Servers - replacement cost, inc installation and peripherals
(assumes Dell PowerEdge 1950)
09 - 11 £2700 £675K
1 Air conditioning upgrades in RWD 1.71 and Main 1.71 Computer Rooms
08/09 £100K £100K
6 12KVA and 16KVA UPS replacements 09/10 £9K £54K
Total hardware replacement costs (1 for 1) £1.15M
450 Cost of electricity for servers (200w per server @12p/KW hour) Annual £94K £283K
450 Cost of electricity for cooling (100w per server @12p/KW hour) * Annual £47K £142K
Total electricity costs £425K
Total hardware and electricity costs £1.24M
* Inefficient raised floor and ceiling mount air conditioning units
Business CaseAfter Virtualisation
Equipment FY Unit Total
5
RM Servers – replacement cost, inc installation and peripherals
(assumes Dell PowerEdge P900 with 32GB RAM, 2TB Disk)
08/09 £5K £25K
5
‘Out of warranty’ servers– replacement cost, inc installation and peripherals (assumes Dell PowerEdge P900 with 32GB RAM, 2TB Disk)
09/10 £5K £25K
13
Production Servers – replacement cost, inc installation and peripherals
(assumes Dell PowerEdge P900 with 32GB RAM, 2TB Disk)
09/11 £5K £65K
2 24TB Central Storage – assumes Sun Fire X4500 or Equivalent 08/09 £15K £30K
46 VMware ESX Licences – one per CPU 08 - 11 £1K £46K
Physical to Virtual (P2V) Software ? 08/09 ? ?
VMware backup software? 08/09 ? ?
Total hardware costs (VMware Infrastructure) £191K
25 Cost of electricity for servers (500w per server @12p/KW hour) Annual £13K £39K
25 Cost of electricity for cooling (250w per server @12p/KW hour)* Annual £6K £18K
Total electricity costs £57K
Total hardware and electricity costs £248K
* Inefficient raised floor and ceiling mount air conditioning units
Virtualisation Business Case3 Year ROI with Virtualisation
Before Total
Total hardware costs £1.15M
Total electricity costs £425K
Total costs £1.57M
After Total
Total hardware costs (VMware Infrastructure)
£191K
Total electricity costs £57K
Total costs £248K
Hardware £959K
Electricity £368K
Total Savings = £1.32M
Planet Filestore
JISC-funded project to measure savings of..Tiered storage – scenario
100TB needed for 30000 (staff and students) High performance for 7000 concurrent users Site mirrored Raid 10 450GB SAS vs Non mirrored Raid 5 2TB SATA ~ factor of 16 in hardware cost Therefore, use HSM (old data on Tier 2)
What about energy ?
Planet Filestore Project
Fast disk (FC/SAS) 450GB, 25W
Slow disk (SATA) 2TB, 15W
Tiered storageTier 1 – DRTier 2 – non DR
100TB Tier 1 (DR)Energy cost
£25,000 p.a.10TB Tier 1 (DR)90TB Tier 2(nonDR)Energy cost
£3,250 p.a.
Energy saving = £21,750
PCs & Monitors
Where to start ?Duty cycle guesstimate
Always on (100%) On during working
hours (20%)
Turn it off at night !!
What next for PCs ?
More stringent power saving policiesAsset Management (reporting)Influencing user behaviour (Joulemeter)
What about Thin Clients ?
Year PC Thin Client
2005 300W 15W
2010 50W 5W
Still some scope, but notwhat there used to be
Printing
Thanks to Liverpool John Moores University (LJMU) for the next 2 slides
Carbon Footprint Savings
• Hundreds of devices removed– Old lasers– Deskjets– Fax machines
• Replaced with fleet of 220 devices
• Automatic power saving after 2 hours
• Use of recycled paper
• FollowMe documents held for 18 hours
Actual Savings at LJMU
Networks
Power used by switches 3.5Gbps/W to 0.5Gbps/W
Cardiff LAN has 50,000 outlets 100kW down to 15kW Potential £85k p.a. by choosing well
PoE (up to 25W down a UTP cable) 5W in the cable, 5W DC-DC conversion 40% loss
Use EnergyStar PCsDiscourage use of the fastest PCsUse energy in your TCO calculationsEnsure suppliers only publicise green PCs
Purchasing Policies
Influencing Behaviour
Context – who pays for electricityTarget your messages
Users• Their PC uses ???W• Pages printed last week ??
Technical types – detail• MFlops, kW, Geeks/Pint
Managers• Bottom line ££££• Reputation.
Finally….
What is Green, but makes your PC use more electricity ?
Answer – CondorSo, what is Condor ?Condor takes spare CPU Cycles for use in
Research calculations Runs at low priority so users don’t notice
So why is it Green ?
Condor - 2.4GHz Core2Duo
0
50
100
Idle (W) 64.5
Max (W) 80.7
Conclusions
Measure, Improve, Measure againLook everywhere for wasteContext changes over time
Make sure you revisit your assumptions
Don’t just do the technical stuffThink out of the box (Condor)
Questions ?
Cardiff’s New Datacentre
£3M grant to purchase a supercomputerNew Datacentre required with appropriate
power, cooling ..Our Context ?
INSRV Sustainability Mission - To minimise CU’s IT environmental impact and to be a leader in delivering sustainable information services.
Limited experience in Datacentre build Server & Data Center Energy Efficiency
(US Congress Report 2007)
What did Cardiff do ?
Ran 2 tenders, many contractors (& subs) HPC & Environment (power, cooling)
Strong project management TCO as key evaluation criterion Value = HPC Benchmark figures Winner = Highest Value/TCO