Upload
others
View
9
Download
1
Embed Size (px)
Citation preview
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 1
Service Oriented Data Center and Storage NetworkingCisco on Cisco Technology Tutorial
Sidney Morgan IT Manager, Cisco IT
Monique LeFors EdmondsenIT Program Manager, Cisco IT, (Host)
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 2
Agenda
Data Center Transformation
SODC Technologies
Cisco IT Overview
SODC Results and Lessons Learned
Services Oriented Data Center (SODC)
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 3
IT Overview
Cisco IT Governance
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 4
Cisco IT Mission
At Cisco, IT is expected to. . .
Support the business strategy
Drive productivity
Facilitate Cisco innovation
Showcase Cisco technology
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 5
Enable “Every” Move with ITCEO sets the Vision, CIO enables the Vision
Productivity
Growth
Experience
OperationalFunctionality
Business Capabilities
Strategies
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 6
Two Views
Those that viewInformation Technologyas Strategic to their business
OR
Those that see IT as a Cost Center
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 7
Production Data Center Transformation
Overview and Issues
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 8
Cisco Data Centers
SJ-12SJ-KLinksys
RTP 5
Amsterdam
Total of 195,000 square feet of
raised Data Center space at Cisco
Scientific Atlanta
Production Data CenterData Centers Development Data Center
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 9
Cisco’s Next-Gen Production Data CenterIT Enables Business Agility and Resiliency
User ExperienceUser ExperienceGrowthGrowth
Business Agility Business Resiliency
Architecture&
Operational Excellence
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 10
Problem Statement
Cisco’s data
centers are nearing capacity, increasing our
Business Risk.
Rather than enabling the business, we are rapidly approaching a situation in which our data centers are inhibiting our agility and resiliency.
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 11
Data Center Operational Impact
1 hr of downtime in Customer Service = 10 hrs of
impact
1 hr of downtime in Manufacturing = $40k-$100k
impact
No Power – No Cooling – No Space = No Innovation
No showcasing = impact to Eng = impact to Sales!
Business requirements increasing quarterly…“No business downtime”
“New business models”
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 12
PDC, Opportunity Enable Every Move with IT
SODC ArchitectureApplication Networking ServicesVFrame Data CenterUnified Network Fabric
Critical Systems Resiliency Tracks EA Policies for new PDCApplication Dependency MappingCMBD
Architecture&
Operational Excellence
Agility and Resiliency
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 13
PDC Site Selection
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 14
PDC Tier
Tier I II III IV
Tier I II III IV
Components Need only (N) N + 1 N + 1 2 (N + 1)
Delivery Paths One only One only One activeOne passive Two active
Single Pointsof Failure Yes Yes Yes No
Concurrently Maintainable No Components
only Yes Yes
Projected Availability /Downtime per Year
99.671%28.8 hours
99.749%22.0 hours
99.982%1.6 hours
99.995%0.4 hours
Tiering system courtesy of The Uptime Institute New PDCExisting
DDCsExisting
PDCs
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 15
PDC Site Selection Strategyfrom 420 Metro Areas Down to 8, Then 1
FiberTornado
Earthquakes Hurricanes
Additional Criteria:Electrical power cost;long-term price stabilityOther costs: real estate, labor, taxes, govt incentivesProximity to existing Cisco IT operationsClose to customersAvailability of technical laborRegulatory environment
Must-haves:U.S. or CanadaNegligible environmental risk (e.g. earthquake, hurricane, tornadoes, etc.)Fiber serviceAt least 2 long distance providers
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 16
The Winner! - RichardsonRichardson, Texas
Runner-upsPhoenixBoulder
Tipping PointsLeverage $21M RDC9 Capital investment
Accelerate Data Center Business value by 12 months
Cisco Community and Campus
Multiple land options at optimal distances at right size
Fiber Infrastructure
Skilled IT resources
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 17
Service OrientedData Center
Overview
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 18
Service Oriented Data Center (SODC)
ServiceOriented
Data Center
Vision
Vision Enablers
SoftwareTechnology
BusinessProcessesPeople Hardware
Technology
SODC Target State:Pooled Virtual Resources, Automated, Standard Services Based, Secure, Intelligent Unified Data Center Network
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 19
SODC Meeting Business Objectives
Enhance business agility– Provisioning resources within 3 days
Improve business continuance– Every component of SODC will be utilized and load balanced at all times.
Reduce Costs– The SODC model will optimally manage the infrastructure to meet user, application, and business needs
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 20
SODC Design Phases
ConsolidateOptimize Data Center Resources
Increase Resource Utilization
VirtualizeVirtual Resource Pools
Increase Availability and Agility
AutomateAdaptive Orchestration
Rapid Delivery of Services
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 21
… Data Center Evolution
20052004 2006 - 2007 2008 - 2010
• Standardization• Virtual Machines
• 4 Tier Silos • Heterogeneous OS• Storage Silos• Low Utilization
• IP Connectivity
• Perimeter Security
• Application Silos• Distributed
• Server Repurposing• VM Mobility• Storage
Virtualization
• Virtualized Network Services
• Virtual Firewalls
• Optimization
• Infrastructure Aligned to Application Services
• Policy Based Management
• Intelligent Data Management
• Tiered Recovery• Usage and SLA-
based Funding Model
LegacyData Center
VirtualData Center
Service OrientedData Center
ConsolidatedData Center
Consolidation PhaseVirtualization Phase
Automation Phase
Compute
Storage
Network
Security
Application
• SANs, VSANs• Tiered Storage
• Consolidate, Centralize
• Consolidated Network Services
• Secure Each Application Tier
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 22
Service OrientedData Center
Technologies
Servers
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 23
SODC Server Landscape
Solaris 3,02325%
Windows3,47529%
Linux 5,276 44%
HPUX1792%11,953 virtual/physical
servers
12,853 Applications
325 Production Databases
Source: Cisco IT, June 2007
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 24
SODC Server Virtualization
Data Center Server ConsolidationImprove Operational AgilityLower Data Center Operating Expense
Increase Utilization of Physical ServersOptimize TCOImprove Data Center Capacity Management
Reduce Service Provisioning TimesRapid deployment of Operational Environments
Increase Operational EfficienciesEase Support of EnvironmentsReduce Planned and Unplanned downtime
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 25
Server Virtualization Considerations
Support ModelSupport Model must drive operational objectives Managed by core SODC Team
Risk vs. Virtualization TargetsISV’s Support?Reduced Risk = Reduced Potential Savings
Keep Clients In MindMinimize Impact of Migrating to Virtual ServersPlatespin, VM Converter Software is Crucial
Communicate VMware Strategy and DirectionSuccess Depends on Leadership Support
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 26
SODC Virtual Data Centers
SJ-12SJ-K RTP
5,7
AmsterdamBoxborough
Bangalore
IsraelAPAC
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 27
SODC VMware Architecture
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 28
SODC Server Virtualization Highlights
New Virtual Machine DeploymentsReducing New Servers Deployed in Data CentersNo Limits on Supported Apps
Support Third-Party and Custom Built Apps in VMware FarmsDrive Clients to VMware by Working with Vendors
Target 2,500 New VMs by August 30, 2007
Server OS and App MigrationsFocusing on Legacy x86 Hardware
Reduce Maintenance CostsLeverage Platespin/VM Converter Software For Migrations
Linux and Windows MigrationsMinimizes Client Impact – No App ReinstallMigrate a Physical Server to a Virtual Server in Less Than 4 Hours
Target 350 Server Decomms/Migrations August 30, 2007
TCO R
OI
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 29
SODC Server Virtualization Metrics
Improved Productivity andIncreased App
Availability2500 Total Virtual Machines Today
95+% Of VMsDelivered Under
3 Days
Over $14 M in Savings And Cost Reduction To Date
14 VMs PerESX Server
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 30
Service OrientedData Center
Technologies
Storage
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 31
Cisco Data Center Storage Landscape
0
500
1000
1500
2000
2500
3000
3500
4000
4500
FY'01 FY'02 FY'03 FY'04 FY'05 FY'06 FY'07
DAS
NAS
SAN
Over 7.8 PB of “raw” storage, Overall Growth Rate: FY’02=69%, FY’03=32%, FY’04=50%, FY’05=58%, FY’06=29%, FY’07=52%
Elimination of all external direct attached storage, focus now on reduction of internal server storage (via remote boot)
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 32
SODC Storage Architecture consolidationCisco IP Network NAS Cisco IP Network
Timeline
2001 2002 2003 2004
“Logical” Cisco Business Functions*Multiple Datacenters (campus/metro)
Gold
Silver
Bronze
Cisco IP NetworkNAS
SAN SAN
SAN
“Physical” Cisco Business Functions
Phase 3:Predicted to take 12–18 months Originally planned to begin in 2004
Phase 1:Originally scheduled for 1 year (2001)
Phase 2:Predicted to take 18–24 months Planned to begin in 2002Start delayed for 12 months
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 33
SODC Storage Architecture consolidation
Multiple Datacenters (Campus/Metro/Global)
Cisco IP LAN
NAS Gateways
IP WAN
FC Fabric FC FabricCisco IP Storage Switching Module
FCIP FCIP
Cisco IP LAN
Timeline
2004 2005
Phase 5:Predicted to take 12—18 months (2005-2006).
Phase 4:Originally scheduled to begin in 2005
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 34
Cisco IT SODC Storage Architecture
IP WANServices
Physical Storage Arrays
Data Center 1 Data Center n
GlobalStorageFabric
Hosts
Remote DC
Platinum
Silver
Gold
Bronze
Metro OpticalNetwork
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 35
PDC SAN Over-View
Two Halls, Hall one holds; 2 storage rows and 8 Zones with two pods per zone
Pods planning on supporting vframe farms of over 200 hosts.
This is not enough storage infrastructure to hold current storage capacity
San design has to be flexible to leverage additional storage arrays anywhere
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 36
Hall one and Hall two interconnected via DWM and Native Fibre
Edge Switches port channeled back to both switches in core for additional reliability
Switches based on 9513 chassis and configured into three different types
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 37
SODC Storage Results
Managed storage per FTE increased from 25 TB to 600 Terabytes over past 5 years
Total Cost of Ownership reduced from .21/MB to .001/MB over past 6 years
Overall utilization increased from 20% to 67% over past 5 years
$69 Million in cost avoidance over past 4 fiscal years ($9M in FY04, $14M in FY05, $27M in FY06, $19M in FY07)
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 38
Service OrientedData Center
Technologies
Network
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 39
Mode of Operation Choices
Active-Standby
Same as todayDoesn’t work well todayOnly critical appsIdle hardwareDifferent configuration
in both DCs
Min infra complexityMed apps complexity
Active-Active-Hybrid
Similar to other customers solutions
Majority of apps Identical configuration
in both DCsNot specific to vendorBest Cisco on Cisco
Med infra complexityMin apps complexity
Active-Active
Used by financial institutions (E.g. metro clusters, multi-master data)
Majority of appsVendor specific
High infra complexityHigh apps complexity
Cen
tral
ized
Cen
tral
ized
Serv
ices
Serv
ices
Dis
trib
uted
Dis
trib
uted
Serv
ices
Serv
ices
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 40
Cisco IT Active/Active-Hybrid Data Center
Web Server(Load Balanced)
Active
App Server(Load Balanced)
Active
Database Server(Local Cluster)
Active
Storage
ReplicationReplicationSyncSync
AsyncAsync
ACE
GSS
Non Cisco DNS Servicewww.cisco.com
DC 1External User
DC 2
Web Server(Load Balanced)
Active
App Server(Load Balanced)
Active
Database Server(Local Cluster)
Standby
ACE
GSS
Storage
Network• Using DNS, users are directed to GSS
• GSS load balances users across both DC #1 & #2
• ACE selects the optimal server to forward request between DC servers
Normal operations• Web and app server are processing requests in both DCs•Transaction logs are applied to remote DB
Note:In the physical implementation active DBs will be distribution across both DCs
Note:Production infrastructure (network, hosts, infra software) configuration is identical in both DCs
HA VFrameData Center
Cluster
GSS
Failure Scenario• Last transaction logs applied• Apps DB connection timeout and reconnect• No runtime apps changes required
Database Server(Local Cluster)
Active
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 41
Continuous Availability with Disaster Recoverywww.cisco.com
Non Cisco DNS Service
External User
ReplicationReplicationAsyncAsync
Web Server(Load Balanced)
Active
App Server(Load Balanced)
Active
Database Server(Local Cluster)
Standby
StorageReplicationReplication
SyncSyncAsyncAsync
ACE
GSS
DC 2 DC 1
Web Server(Load Balanced)
Active
App Server(Load Balanced)
Active
Database Server(Local Cluster)
Active
ACE
GSS
Storage
Remote DC
Web Server(Load Balanced)
Standby orQuick Ship
App Server(Load Balanced)
Standby orQuick Ship
Database Server(Local Cluster)
Warm Standby
ACE
GSS
Storage
HA VFrameData Center
Cluster
Database Server(Local Cluster)
Active
App Server(Load Balanced)
Active
Web Server(Load Balanced)
Active
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 42
SODC OrchestrationSODC POD
Virt
ual
Serv
ice
Tem
plat
e
Network Pool Server Pool Storage Pool
VLANsVirtual Network Services
VSANs
Virtual LUNs
ISM Framework
ISM Fram
ework
ISM Fram
ework
ISM
Fra
mew
ork
Cisco VFRAME Data Center Orchestration
Specific resources selected from PODsNetwork Services ProvisionedVLANs, VSANs configuredSAN is zonedServers get booted with assigned image – Remote ImagesApplication(s) are startedTraffic into logical network turned “on”Failover of Services - Resiliency
OS and A
pp Images
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 43
STORAGE AREA
NETWORK
GbE / 10GbE
DISTRIBUTION LAYER
FCIP10GbE
METRO OPTICAL NETWORK
DWDM
ONS 15454
Catalyst 6509s
Platinum
Silver
Gold
STORAGE POOL
SERVER POOLShared ServiceSwitches
CAMPUS/ CAMPUS/ INTRANET/ INTRANET/ INTERNET/INTERNET/EXTRANETEXTRANET
IP Site Backbone Router (CAT6K) CD
Storage SAN Edge
Storage Core Gateways
(MDS 9500)
NFS/CIFS
MDS 9513s
Console
Fibre Channel
Copper
STORAGE ACCESS LAYER
NAS Gateways
vFrame Managed Pod
vFrame Managed Pod
vFrame M
anaged Pod
vFrame M
anaged Pod
vFrame M
anaged Pod
SODC Architecture
Bronze
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 44
SODC Network POD ArchitectureData Center Distribution GW
Shared Network Services
Data Center DistributionLayer
10 Gig
1RUCAT 4948
CAT 4948
6500Series
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 45
Service OrientedData Center
Technologies
Wide Area Application Services
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 46
ANSBig Picture
SJCAMS
FSO
PDC
Customer
ExtranetPartner
Internet
PDC1
DCDC
Server
Global Site Selector (GSS)
DMZDC
DMZDC
PDC2
DMZPDC1
DMZPDC2
Application Content and Networking System (ACNS)
Application Control Engine (ACE)
1
1
2
2
3
5 7
1
ACE Appliance
4
5
5
4
4 4
6 Wide-Area Application Services (WAAS)
77
6
6 6
4
2
3
7
ApplicationNetworking
Services
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 47
Content SwitchingWhat Is it?
Server Farm (SF)“Group of one or more servers with nearly identical configuration and providing equal functionality”
Server Load Balancing (SLB)“Distribution of data connections across servers in a single serverfarm, such that the overall capacity and availability increase (typically at Layer 3–4)”
Content Switching (CS)“Intelligent distribution of data requests across servers in different serverfarms, such that the appropriate serverfarm is used and the overall capacity, availability and/or functionality increase (up to Layer 7)”
GET *.gif
GET *.asp
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 48
Content SwitchingWhy Do We Need it?
High Availability (HA)Required for mission-critical applicationsAchieved through load-balancing, health monitoring (reactive), server maintenance (proactive)
FunctionalityFlexible back-end infrastructure while keeping user-friendly URL’s through URL load-balancing, manipulation and redirectionExample: retain the www.cisco.com paradigm
Cache controlManual and proactive control of cache-logic on Content Engine
SecurityProvides limited Denial-of-Server (DoS) protectionFacilitates SSL Offload though SSL Service Module or ACE
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 49
The www.cisco.com Paradigm
Separate the user interface (front-end) from the special-purpose server farms (back-end)
Domain Name System (DNS) resolves the vanity name to a Virtual IP address hosted on a content switch
http://www.cisco.com/<foo> Cacheable
/app/bbb.pl
Virtual service
Transparentapplicationmigration
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 50
Symmetric Acceleration (Now)Based on the Wide Area Application Services (WAAS) technologyEnabler for NAS ConsolidationWAN savings
Experienced 40-60% savings (volume) in lab PoCCapacity plan impact: from 68kbps (20 kbps for apps) to 60 kbps (12 kbps for apps)
Acceleration for Emerging MarketsArchitecture
Core (or “Server Edge”) and Edge (or “Client Edge”)Transparent at Layer3-4 (WCCP)Inter-DC traffic not in initial wave
PDCPDC
Tier3 Sites(FSO and WAN hub)
E.g.New York
<etc>
SJC RTP
InternalWAN
Client Edge
Server Edge
WCCP
Edge WAE
Core WAE
WCCP
CentralManager
StorageConsolidation
Core
Edge
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 51
WAAS Bandwidth SavingsSimilar to ACNS, but now for *all* TCP protocolsEnabler for additional consolidation and virtualizationTarget: lower standard from 68 kbps/user to 60 kbps/user
Net Cost Avoidance is $3M - $7M (3 year horizon, WAN + NAS consolidation)
HTTPWAN
volume
HTTP*
HTTP*Non-HTTP Non-HTTP
Non-HTTP
-20%-20%
BeforeACNS After
ACNSBeforeWAAS
AfterWAAS
* Note: ACNS also reduces Video streaming and CIFS for laptop re-imaging
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 52
WAAS Acceleration Metrics
Transaction Time
(Normalized)
Livelink document download
Exchange Attachment Download
** CIFS Download Document
* http applicationtransactions
Before WAASAfter WAAS - InitialAfter WAAS - Subsequent
Note: ANS Test Environment: 256Kbps BW and 150ms latency
* Note: Benefit is variable depending on transaction characteristics** Note: Validated at Moscow and Sao Paulo POC sites
3MB PowerPoint file used in document downloads
2X faster 2.5X faster
15X faster
15X faster
Potential productivity increase of est. $21M (3 year horizon)
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 53
Asymmetric Centralized Acceleration
Web AccelerationModify HTTP context and payload
Reverse Proxy Cache model
Source IP will changeImpact to downstream applicationsProblem already encountered for URL site selectionNeed a common strategy
CustomerSite #1
ACE Appliance
Web
ANM
Device Management
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 54
Service Oriented Data CenterResults and
Lessons Learned
Cisco IT Best Practices
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 55
Cisco IT Data Center Lessons Learned
Build Foundation with Business
Vision
Buy in from Critical
StakeholdersOrganizationand Process
IT“Cost Model”
Executive Support
Evolve to Service Aligned
Organization
ReengineerOperational
Practices
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 56
Q and A
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 57
Further Resources
http://www.cisco.com/go/ciscoit
Cisco on Cisco Website
Order Resourceshttp://cisco.com/en/US/ordering/index.shtml
Call to get Product, Solution and Financing Information1-800-745-8308 ext. 4699
© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSODC EBC 58