DYNAMIC INFRASTRUCTURE Helping build a smarter planet

IBM 2009 Copyrighted

Dynamic Infrastructure: Helping build a smarter planet

DYNAMIC INFRASTRUCTURE®

Helping build a smarter planet

Optimization of Virtualized, ConvergedData Center Networks

Renato Recio, DE, IBM Data Center Networking CTO



8 IBM Corporation 1994-2009. All rights reserved.References in this document to IBM products or services do not imply that IBM intends to make them available in every country.

Trademarks of International Business Machines Corporation in the United States, other countries, or both can be found on the World Wide Web at http://www.ibm.com/legal/copytrade.shtml.

Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registeredtrademarks of Intel Corporation or its subsidiaries in the United States and other countries.Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency which is now part of the Office of Government Commerce.ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office.UNIX is a registered trademark of The Open Group in the United States and other countries.Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.Other company, product, or service names may be trademarks or service marks of others.

Information is provided "AS IS" without warranty of any kind.

The customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer.

Information concerning non-IBM products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by IBM. Sources for non-IBM list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-IBM products. Questions on the capability of non-IBM products should be addressed to the supplier of those products.

All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.

Some information addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific levels of performance, function or delivery schedules with respect to any future products. Such commitments are only made in IBM product announcements. The information is presented here to communicate IBM's current investment and development activities as a good faith effort to help with our customers' future planning.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the ratios stated here.

Prices are suggested U.S. list prices and are subject to change without notice. Starting price may not include a hard drive, operating system or other features. Contact your IBM representative or Business Partner for the most current pricing in your geography.

Photographs shown may be engineering prototypes. Changes may be incorporated in production models.

Legal Slide ☺ Trademarks and disclaimers



Today’s Physical Data Center Infrastructure Analogy

Traversing today’s Data Center Infrastructure is analogous to going through a long toll road, using only the “Cash” lanes...

…Each toll plaza requires a stop…

…where a queue forms, …as each car pays the toll…

The end result...– High latency caused by long queues per stop & many stops.

– High staffing cost associated with manning the toll plaza.

We are simplifying DCNs, by working with network partners to:

…automate the process (analogous to an “E-ZPass”),

… flatten the path (analogous to short-cuts).

$$$



Today’s Non-Virtualized DataCenter Network

High OpEx due to many network, storage & server managers

High CapEx & OpEx due to dual fabrics (Ethernet and FC), using lower bandwidth links

High OpEx due to: sprawl of demilitarized Zone appliances

High CapEx & OpEx to support each server, due to many networking layers.

High latency due to many layers, which repeat processing steps per layer.

>2 μsec

>10 μsec

>20 μsec

>30 μsec

High CapEx & OpEx due to lightly utilized servers moving VMs (to increase utilization) is limited by low bandwidth links & manual management issues.



The Data Center is Undergoing Transition

From a physical compute model that has:– VMs held captive by low bandwidth links

& manual network configuration– Few Virtual Machines (VMs) per server– Using lower bandwidth links– Sprawl of manually intensive,

expert based IT management tools– Poor image life-cycle management tools– Physical appliance sprawl

Expense issues with this model:– High capital expenses due to under utilized

servers and multiple fabrics– High operational expenses due to manual

administration of many management tools.

To a virtualized, Dynamic Infrastructure model that:– Has simpler, integrated & virtualized

appliances and systems– Has many highly utilized servers, each with many

VMs; using virtual storage, networks & appliances– Is enabled by high bandwidth links and

migrating to virtual & converged fabrics– Includes new, “faster time to value”

Cloud Building Block (CBB) Multi-Rack acquisitions

The value of this new model is:– Lower capital expenses through

higher server, storage and network utilization. – Lower operational expenses through

automated & integrated managementthat optimizes and automates Data Center infrastructure and delivers self configured services to the business.

1 GbpsEthernet

4 GbpsFibre Channel

DatabaseWeb

Application

8 GbpsFibreChannel

10 Gbps Ethernet or Converged Ethernet Virtual

Storage



It’s time to start thinking differentlyabout Data Center infrastructure.Scale, cloud based server integration,integrate the management,automate virtualization, converge the network, andflatten the infrastructure.



Integrated management One interface for managing all physical and logical resources

Integration Trends

Blade Chassis based integrationIntegrates servers, switch, storageHigher density than modularOver 3x cable reductionImproved RAS (less cables to bump)

Rack based integrationIntegrates servers, switch, manager appliance, storageHigh density rack (100 servers) ships complete, ready to deploy

BladeCenter

iDataPlex

Cloud Building Block (CBB) based Acquisition(Software, server, storage & network)

Switch

Switch

Manager

Blade Servers (with network)

ChassisSwitches

Storage

Manager

CBB based integrationIntegrated, multi-rack server, switch, storage and virtualization solution.Ships with single CBB manager.



Cloud Building Block Solution Example IBM CloudBurst

GTS Quickstart Services:CloudBurst provides rapid implementation of a private cloud environment consisting of the cloud management infrastructure, including the cloud resources to be provisioned

–Initial install, configuration and training services

–Pre-engagement workshop for initial requirements

–Post-engagement workshop for growth

Base Hardware Configuration*:1 42U rack1 3650M2 Mgmt Server, 8 cores, 32GB Ram1 HS22 Blades, 8 cores, 48GB RAMBase configuration:

– 1x BladeCenter chassis– 3 managed H22 blades, 8 cores, 48GB RAM

DS3400 FC attached storage, 5.4TB raw capacity*Base HW configuration can be scaled out per blade server

with up to two chassis and 28 blades.

Cloud Software Configuration:BladeCenter Foundation stack runs on 3650M2

management serverWindows 2003R2 EnterpriseSystems Director 6.1.1 with BOFM, AEM; ToolsCenter 1.0; DS Storage Mgr for DS4000 v10.36; VMware VirtualCenter 2.5 U4; LSI SMI-S provider for DS3400VMware ESXi 3.5 U4 hypervisor on all blades

Thundercloud software stack will ship as VMware images to run on the HS22 management bladeBlueCloud v1.6.1 including appliance wizardsTivoli Provisioning Manager v7.1

–DB2 ESE 9.1; WAS ND 6.1.0.13; TDS 6.1.0.1Tivoli Monitoring v6.2.1

–OS packSUSE Enterprise Linux 10 sp2

A service delivery platform that is pre-integrated at the factory. Prepackaged, pre-configuredservers, storage, networking, software and installation services needed to provide a private cloud

Built-for-purpose based on the architectural requirement of specific workloads

Delivered and supported as a single product






Today’s Non-Virtualized DataCenter NetworkMultiple Platform

Managers• Server• VMs• Ethernet• Fibre Channel

Integrated Platform Manager

• Server• VMs• Ethernet, CEE,

Fibre Channel, FCoE



Data Center Management Components

Service management

Integrated visibility, control & automation across heterogeneous business and technology assets

Align IT operations with the business

Govern and control the business

Optimize the business

Detailed platform managementConsolidated management across systems

Tell me what I have and if it’s working

Let me install, configure and update

Integrated physical and virtual managementServers, storage & network

Automated physical and virtual provisioning



IBM Data Center Management

Automated, Integrated, Flexible, Provisioned by Request

IBM Systems

Director

VMControlNetwork Manager

Tivoli Provisioning Manager

Tivoli Storage Productivity Center

Tivoli Monitoring

Tivoli Self Service Application

Mgt Node

Storage

Storage

Storage

Access Switch

Platform Management

Discovery, Inventory andMonitoring of System

ResourcesActive Energy Controls

Storage & Network ManagementNetwork IntegrationStorage Integration

Automated Relocation

Virtual Systems Management

VM Lifecycle (Create, deploy, relocate & maintain VMs)Image Lifecycle (Capture,

import, create, edit and delete virtual images)

User Request Control Consoles

End user requestsService center

Workflow ControlImage libraries

Config files

Application

Compute Node

Compute Node

Compute Node

Compute Node

Access Switch

Compute Node

Compute Node

Compute Node

Compute Node

Access Switch

Virtual I/O Server (VIOS)

Shared I/O

SMP Hardware System

Hypervisor

Datacenter Fabric






Today’s Virtual Server IntegrationVirtual IO

Server(VIOS) … A

IX

Linu

x

PCIeAdapter

PCIe Port

F

DrvDrvDriverDriver

Hypervisor

HostEthernetAdapter

Port

Drv

AIX

Drv

Virtual Switch

Enet Port Enet Port

VEB

Hypervisor Bypass, eliminates overhead

System virtualization technology offers:– Many OSs / server up to 254 for PowerVM– Integrated virtualization manager– Application & image mobility– High resource utilization– Virtual and dedicated IO– Image quality of service– Dynamic resizing– High availability

Today’s virtual IO technology (e.g. Power)– Native IO Virtualization (IOV)

IO directly shared by adapter hardware.Adds Virtual Ethernet Bridge to adapter.

– IO shared through Virtualization Intermediary(e.g. VIOS on Power).

All IO is performed through VI.Adds overhead to every IO operation.2+ us< 1 us Latency

In External SwitchIn AdapterVirtual Switch

4 8 GB/s 1 GB/sBandwidth



Server Virtual Switching TrendsSystem virtualization, with integrated virtual switches, has been around for decades.

– However, managing virtual switches is too manual today and, as covered earlier, scale-out sprawl magnifies this complexity.

Virtual switching is evolving into two use cases:1. Virtualized systems or clusters that exploit uP & VM trends:

more cores more VMs more VM-VM networking. Attributes:Internal virtual switching in hypervisor or PCIe adapter (with Single-Root IOV).Highest throughput and lowest latency.Virtual switch management integrated with server management.Virtual NIC’s port profile (e.g. VLAN) automatically migrates with VM.Server based access, QoS controls and virtual security appliances.

2. Multi-tier, enterprise DC environments where state of the art networkaccess and traffic controls are more important than VM-VM performance.

External network performs virtual switching, using advanced controls & network security.

Less switches (i.e. no server switches) to manage under common management tool.Emerging automated port profile migration mechanism.

Hypervisor

PCIe

VMs

PCIe

External Switch2.

1.



Automating Port Profile Management for Network based VSWe are working with partners and the industry on a port profile automation mechanism that:– Selects where Virtual Switching is to be done (i.e. server or network).– Automates port profile migration for both cases.

The approach allows automated migration of a VM’s external network profile on activation and during VM migration, by:1. Using management plane protocols to:

A. create/destroy virtual switch port profiles; and B. communicate “port profile to VM associations” to the Hypervisor.

2. Using a control plane protocol to: C. associate a port profile with a specific VM’s MAC Address; and D. de-associate & migrate that profile when the VM migrates.

Hypervisor & PCI SR-IOV adapters implement BPV (Bridge Port Virtualizer) protocols to associate virtual port profiles.– Simplifies VM mobility by mobilizing network profile thru control plane – Routes VM-VM traffic through external switch

Hypervisor

VM

IO

BPV

VM

IOBPV

VM

Port Profile Manager

VM manager

Associated

Hyp. Switch / Port Pr. Mgr

Port Profile IDVM MAC AddressVM MAC Address

Hyp. Switch / Port Pr. Mgr

Port Profile ID

Associate

Switch

B.

A.

C.






Fabric Convergence Options

High performance clusters– IB: 2 4 8 GB/s and

150 <100 ns– Lack native storage & use slow gateways

SMB and Mid-tier Storage– Low $/Gbps (GigE)– Growing and maturing, considering 10 GigE

iSCSI/NAS storage for middle tier servers

SMBstorage

IB

FC Enet

– Clustered File System with Global Namespace

– Centrally managed– Scales horizontally– Integrated Information

Lifecycle Mgt, HA & disaster recovery

Scale-out File System (SoFS)

1u1u1u1u1u1u1u1u1u1u1u1u1u1u1u1u1u1u1u1u

1u1u1u1u1u

1u1u1u1u1u1u1u1u1u1u1u1u1u

TOREnet TORS

FibreChannel

SAN

EthernetLAN

Fibre Channel over Convergence Enhanced Ethernet

Converged switch– Enables 50% CapEx

reduction within the chassis (rack or BC).

– Lower power– Improved RAS– Plugs into existing DC

infrastructure


Dynamic Infrastructure: Helping build a smarter planetFibre Channel over Ethernet

FC over Ethernet– Layers FC frames directly over Ethernet (requires jumbo/mini-jumbo)

– Replacing lowest level of FC with EthernetT11 FC-BB-5 standard (Fibre Channel standards group).

FC over Ethernet requires convergence enhancements to Ethernet:– To carry maximum sized FC packets

Requires Ethernet Jumbo (or mini-jumbo) frames– To provide FC like no-drop behavior in face of congestion

Requires new “Priority-based Flow Control” protocol– To control traffic interferences:

Requires new “Enhanced Transmission Selection” protocol– To detect both ends of the wire support the above

Requires new “Data Center Bridging eXchange” protocolCEE Authors was formed to facilitate/accelerate definition & adoption of above.CEE Authors submitted proposals for above protocols to IEEE 802.1.

EthernetFrame

FCoEEncapsulation

FCPacket



Fabric Convergence Value Proposition vs Barriers

Lower Cost, Lower Power– Less adapters,

cables and switchesImproved RAS– Reduced failure points, time,

misconnections, bumping, …Simpler Management– Single physical fabric to manage.

Customer organizational– Server, Network, Storage silos

Operational management– Quality of service control– Security and access control

Maturity– Standards, technology, management– Resiliency and robustness

Servers Multiple Fabrics One Fabric

BarriersValue Proposition



Converged access switchattaches to FC and EthernetEnables 50% CapEx reduction in networking per row of servers

– Adapters, Cables andAccess Switches.

– Improved RAS (fewer elements)Lower power

– ~36 ~19 KW/hr per rowSimplified management

– Director with integrated network management

Case 2: Dual fabric CBB

Fully converged fabric,with FC attached storageEnables 50% CapEx reduction in networking at data center level

– Eliminates all FC switchesLower power

– ~42 ~21 KW/hr per rowImproved RASSingle DCN Manager

Case 3: Single fabric CBB

Converged access switchattaches to FC and EthernetEnables 50% CapEx reduction within the chassis (Power, modular or BC):

– Adapters, Cables andAccess Switches.

– Improved RAS (fewer elements)Lower power

– ~ 2570 ~1256 W/hr per rackSimplified management

– Director with integrated network management

Case 1: Upgrade

Example DCN Use Case Steps

EthernetLAN

FibreChannel(FC) SAN

FC SAN

EthernetLANBladeRack

EthernetLAN

LargeSMPs

FC SAN

Note: Power estimates are subject to change.

Row of Modular Servers

Row of Modular Servers



IBM FCoCEE Networking Solution Description

Mid-09: FCoCEE use case 1 & 2 on x86 with BladeCenter enablement

– Converged Network Adapters (CNAs) for x86 & BladeCenter

– Converged TOR Switch (Brocade & Cisco)– BladeCenter CEE Pass-thru– IBM System Director

with base network manager

Ethernet Fabric

FibreChannel

Fabric5/09

BrocadeCiscoBrocade

2008

6/09 6/095/09 Base

x86 Systems BladeCenter

TOR Switch

x86Systems

Brocade CNA QLogicCNA



Fabric Convergence Options

High performance clusters– IB: 2 4 8 GB/s and

150 <100 ns– Lack native storage & use slow gateways

SMB and Mid-tier Storage– Low $/Gbps (GigE)– Growing and maturing, considering 10 GigE

iSCSI/NAS storage for middle tier servers

SMBstorage

IB

FC Enet

– Clustered File System with Global Namespace

– Centrally managed– Scales horizontally– Integrated Information

Lifecycle Mgt, HA & disaster recovery

Scale-out File System (SoFS)

1u1u1u1u1u1u1u1u1u1u1u1u1u1u1u1u1u1u1u1u

1u1u1u1u1u

1u1u1u1u1u1u1u1u1u1u1u1u1u

TOREnet TORS

FibreChannel

SAN

EthernetLAN

Fibre Channel over Convergence Enhanced Ethernet

Converged switch– Enables 50% CapEx

reduction within the chassis (rack or BC).

– Lower power– Improved RAS– Plugs into existing DC

infrastructure


Dynamic Infrastructure: Helping build a smarter planetHow does FCoCEE Compare?

+▬=▬Investment (HW, tools, skills) protection

+

++

+++

1996+

=+++

SoFSSoNAS

=+Data Center upgrades

++??

▬++

▬=▬

Storage attachment- Enterprise data tier functions* - Bridges to FC fabrics- Native storage attachment

2009▬

1990s+

2001+

Maturity- Standard hardware available when?- Technology, management

=+SMB and Mid-tier servers

HPC & Analytics

+

++

=

iSCSI / NAS

??+Price/Performance (compared to FC)

+▬Security▬▬Management simplification

+Performance

+Key Market

Segments Targeted

FCoCEEInfiniBand

*For example: synchronous remote-site switch-over; etc…


Dynamic Infrastructure: Helping build a smarter planet Projected FCoCEE Volume Adoption

2010 20112009 2012

Early AdoptersHPC and Analytics

Use Case 1(FCoCEE server upgrade)

Late Majority(FICON)FCoCEE uses FC Layers & Services

Value of capital expense reductionAbility to “plug-into” existing DC

Customer organizational issuesManagement robustness

FCoCEE maturityCurrent economic realities

Factors for faster adoption

Factors for slower adoption

Note: These are early estimates and subject to change.

Use Case 2 (Dual fabric Cloud Cell)

Use Case 3 (Single fabric Cloud Cell)



Flattening Today’s DCN

Database Tier

....

Application TierWeb Tier

Aggregation

Access

Edge

Internet

Core

Storage fabric

x86 servers x86 servers

Routers

DMZAppliances

TORSwitches

Access Convergence- Eliminates FC switches

- Eliminates FC adapters

Layer-2

Layer-3




Database Tier

....


Aggregation

Access

Edge

Internet

Core

Storage fabric


Routers

DMZAppliances

TORSwitches

Access Convergence- Eliminates FC switches

- Eliminates FC adapters

Layer-2

Layer-3



Fabric Convergence Conclusion

No single solution satisfies all requirements.

– IB fits when performance is critical.

– NAS and iSCSI fit well in SMB and middle-tier servers.

– SoFS & SoNAS fit well in scale-out environments.

– As FCoCEE matures it will play well in FC based enterprises seeking convergence.







Database Tier

....


Aggregation

Access

Edge

Internet

Core

Storage fabric


Layer-2

Layer-3

Routers

DMZAppliances

TORSwitches

Core & Edge Integration- Integrates Core switches

with Routers- Consolidate

Appliances



Core & Edge Integration- Integrates Core switches

with Routers- Consolidate

Appliances


Database Tier

....


Aggregation

Access

Edge

Internet

Core

Storage fabric


Layer-2

Layer-3

DMZAppliances

TORSwitches



2009 Ethernet Product Content

Brocade networking:– 4/28 announced two switch families:

NetIron with layer-3 functions (e.g. MPLS/VPLS) that enable Data Center core and router integration

FastIron with Layer-2 functions for small/medium clusters.

Working with Juniper on OEM products– IBM announced an original equipment manufacturing

(OEM) agreement with Juniper Networks.– IBM will rebrand and sell selected

Juniper EX and MX switches and routers. – Availability is expected later this year. – This expands upon IBM and Juniper's long-term

relationship that includes a reseller agreement with IBM Global Technology Services, as well as collaboration on Juniper's Stratus Project, & IBM's 10 worldwide Cloud Labs.

NI MLX, CESFI SX, GS

BI RX

One Manager, One OS

One Manager, One OS, One Release

Deliver a complete DCN switching solution,that begins to flatten the network.




Database Tier

....


Aggregation

Access

Edge

Internet

Core

Storage fabric


DMZAppliances

TORSwitches

Layer-2 Consolidation- Consolidates lower

performing Layer-2Switches & eliminates

Spanning Tree (e.g.Using TRILL and/or

Virtual Switching)

Layer-2

Layer-3




Database Tier

....


Aggregation

Access

Edge

Internet

Core

Storage fabric


DMZAppliances

TORSwitches

Layer-2 Consolidation- Consolidates lower

performing Layer-2Switches & eliminates

Spanning Tree (e.g.Using TRILL and/or

Virtual Switching)

Layer-2

Layer-3




Database Tier

....


Aggregation

Access

Edge

Internet

Core

Storage fabric


DMZAppliances

TORSwitches

SAN Convergence- Converges SAN & LAN

core switches

Layer-2

Layer-3




Database Tier

....


Aggregation

Access

Edge

Internet

Core


DMZAppliances

TORSwitches

SAN Convergence- Converges SAN & LAN

core switches

Layer-2

Layer-3




Database Tier

....


Aggregation

Access

Edge

Internet

Core

DMZAppliances

Virtualize the Network- Simplifies the management

- Further cost reduction

Layer-2

Layer-3

Virtual CoreVirtual

Storage



Data Center Network Flattening Summary

Technology evolution allows for creation of single layer Data Center Network.– Convergence Enhanced Ethernet enables elimination of a dedicated storage fabric – New networking technologies (higher bandwidth links, switch stacking,

switch virtualization and layer-2 multi-pathing) enable flatter networks.– Integrated appliances reduce the number of stand-alone boxes that must be managed.– Convergence drives (IP/Ethernet & FC) network management integration and

virtualization enhancements drive server and network management integration.

N layer 2 layer 1 layer

Aggregation

Access

Storage fabric

Router

CoreVirtual Core

..

Virtual Storage

Virtual Servers



Data Center Networking Simplification Summary

Scalable, cloud based system integration Enables lower CapEx & OpEx– Cloud based, integrated solutions: servers, network, storage, virtualization, software,

platform & service management for: scale-up (large SMPs) & scale-out servers.

Automated virtualization Enables lower CapEx & OpEx– Automated virtual system life cycle management,

includes automated network and storage identity orchestration.

Integrated management Enables lower OpEx– Simplified, integrated management suite that covers servers, network, storage,

virtualization and service management; and thereby enables lower OpEx.

Converged network Enables increasingly lower CapEx & OpEx with each step– Evolutionary network convergence approach.

Flattened infrastructure Enables lower CapEx & OpEx– Leverage advancements in layer-2+ switching and virtualization technologies

to collapse DC networking layers.



IBM Provides Extensive, Optimized, Robust DC Infrastructure

IBM DC Products & Partners in Support of Dynamic Infrastructure

Additionally, Tivoli manages multi-vendor elements.

Extreme virtualization with massive

scalability and advanced security

IBM System z™

IBM System Storage™

IBM Power™Systems

IBM BladeCenter®

IBM System x™

Highly scalable storage virtualization

for SAN and NAS environments

Highly integrated, fully redundant, energy efficient with advanced I/O

Dynamic resource allocation with advanced mobility for AIX, Linux & IBM i OS

More workloads consolidated on fewer servers with exceptional reliability

IBM Systems Director

IBM Data CenterSwitching

High performance links connecting to

virtual and converged networks

Manage your entire datacenter,

physical and virtual, from one place

Documents

DYNAMIC INFRASTRUCTURE Helping build a smarter planet