Introducing QFabric

7/31/2019 Introducing QFabric

1/137

1


2/137

WilliamOccamwasaphilosopherandmonkwhowasaminimalistinlifeandbecauseof

hisfrequentusageofthesimplicityprinciple,hisnamehasbecomeassociatedto

simplicity.

Otherwaystheexpressioncanbeinterpretedis:keepitsimple

WithQFabric,wehaveusedsimplicitytosolvesomeofthetoughestdatacenter

problemsbykeepingitsimple.

Infact,Juniperhasalwayssolvedveryhardnetworkingproblemswithsimplicityand

innovation.

2


3/137

3


4/137

4


5/137

5


6/137

Recently,inaGartnerDatacenterconference:Customersurveysidentifiedfollowing

twoitemsastheirburningtopics.

Need

for

a

Cloud

strategy

>

Experience PowerandCooling>Operationscosts

Thereisalwaysatradeoffbetweenexperienceandeconomicsinthedatacenter.

Intheverylargeproductiondatacenters liketheGoogles,theFacebooks,delivering

betterexperienceisprioritybecauseitiscoreoftheirbusiness.

HoweverontheotherhandmanyclassicITdatacenters costisanissueandtheymight

bewillingtotradeoffexperienceforeconomics.


7/137

7

Fundamentally,theroleofITinfrastructureistoprovideusersconnectivityto

applicationsandservices.

Mobility,Web2.0andnewertechnologieshaveintroducedmanynewwaysinformation

canbeexchanged:

ClientdevicetoClientdeviceor

ClientdevicetoComputingutilityor

ComputingutilitytoComputingutility.

Inaddition,areliableandhighperformancewideareanetworkisenablingapplication

deliveryovertheWAN.

Thisishelpingcustomerstoconsolidatedatacenterinfrastructureandmanagecoststo

deliversuperior

application

performance.

Inthissession,wewilldiscussdatacenterevolutionandhowQFabricwillhelp

transformdatacenternetworking.


8/137

Theapplicationarchitectureshaveevolvedfromclientservertodistributedapps,causinganunderliningshiftintrafficpattern.

Applicationshave

become

federated

with

architectures

such

as

ServiceOrientedArchitectures,WebServices,andSoftwareasaservice.Thenewerapplicationsareincreasingthecommunicationbetweenservertoservercausingmoreeastwesttrafficwithinthedatacenter.

Servervirtualizationishelpingbusinessesgainefficiencybyconsolidatingmanyphysicalserversintofewerhighperformancevirtualizedservers.

OfallthetraffictypestraversinganEthernetnetwork,storagehasriseninprominence

inrecent

years.

This

traffic

type

has

its

very

own

unique

characteristics

and

demands

aseparatetreatmentthanothertraditionalEthernettraffic.

Tosupportbothvirtualizationandconvergence,thedatacenternetworkneedstooffer: highperformance scale, availabilityawaytosupportstronguserSLAs(aroundthroughput,latency,packetlossandsecurity)

Networkinghasnotprogressedandisnowabarrierslowingthepaceofinnovationintherestofthedatacenter.

8


9/137

Cloudscomeinmanysizesandshapes.Letslookathowcustomersarebuildingclouds.

Customersareinitiallybuildingsmallpoolsofresourceswithcomputeandstorage

resources

this

can

be

a

small

cloud.

Larger

the

pools

better

the

efficiencies

and

the

networkisthefoundationforbuildingthosepoolsleadingtoacloudreadyarchitecture.

9


10/137

Thenumbersofphysicalserversareflatteningoutbutthenumbersofvirtualservers

havebeengrownexponentially.Virtualization providescapitalexpenditurebenefits,but

the

cost

of

managing

virtual

networks

has

increased

the

operational

expenses

drastically.

10


11/137

Thefirstorderofdatacentertransformationisservervirtualizationinsmallpools,

primarilytogainoperationalefficiencyincomputeinfrastructure.Thesecondorderof

transformation

will

be

to

extend

consolidation

beyond

servers

and

create

dynamic

cloudreadyinfrastructuretosupportoperationalflexibilityandbusinessagility.

11


12/137

TheInternethasforeverchangedandiscontinuingtochangethedatacenter.Andwherediditstart?Itstartedwiththeapplication.Yousee,priortotheInternet,allapplicationswereclientserver thatwasstateoftheart.Andinclientserverwhatyou

hadwas

the

monolithic

server

with

amonolithic

application

in

it,

connected

to

astateful

client,andninetyfivepercentofallthetrafficthatwentoutontoaphysicalnetworkwastravelingbetweentheserverandtheclient.[CLICK]Butinsidethatserverwhatwefoundwasabunchofencapsulatedtraffic.Remember,backintheday,wehaddirectattachedstorage;therewasnonetworkforthestorage.So,allthattrafficwasinsidetheserver,andalltheinterprocesscommunicationthatthatapplicationhad thatwascontainedinthememoryoftheserver.So,onlyafractionofthetrafficwasactuallyonaphysicalnetwork.ButalongcametheInternet,andwhatitexposedwasthefundamentalweaknessinclientserver;thosearchitectureswouldneverscale.See,theyweredesignedtosupporthundredsofusers maybeafewthousandsofusers.ButwhattheInternetgaveuswasthe

opportunityto

reach

out

to

our

employees,

to

our

customers,

to

our

partners

and

touchnotthousandsbuttensofthousands maybemillionsofusers.Andyoucouldneverdothatwithclientserver.Sowehadtochangethewaywedesignedtheapplications.ItwasaDarwinianevent.[CLICK]Necessitydroveachangeinthearchitectureandwemovedtowhattodaywecallserviceorientedarchitectures(orSOA),withrichWeb2.0frontends;federatedapplications.Butfundamentallywhatwedidiswetookthatapplicationthatwascontainedwithintheserverandwedisaggregateditacrossthenetwork.Andso,suddenly,whatwehadwasafundamentalshiftinthedatapatternsinthenetwork.Yes,werestillgoingaftertheclientandwehavemoretrafficgoingintotheclientthaneverbefore.Buttodayitonlyaccountsforabout25%ofthetotaltrafficonadatacenternetwork.Therestofitistrafficbetweenservers;trafficbetweenserversandstorage;and

all

that

traffic

that

used

to

be

in

asingle

server

now

is

exposed

across

anetwork


13/137

afundamentalshiftindatapatterns.

12


14/137

With95%ofdatacentertrafficmovingnorthandsouthinthetreebetweenusersand

servers,Traditionalhierarchaltreenetworkarchitectureworkedfineintheearlydays.

However,todaystrafficpatternsfindmostnetworktrafficupto75%iseasttowest,

travelingbetweendevices withinthedatacenter.Asaresult,traversingthedatacenter

requiresasmanyasfivenetworkhops,addingbothlatencyandjitter,whichdirectly

impactsapplicationperformance.

Thenewestroleofthenetworkistoprovideafoundationforthecloudsolution.This

architectureneedsflatnetworkswithanytoanyconnectivitythatreduceslatencyand

jitter,enablingoptimalapplicationbehavior.

13


15/137

Legacydatacenterarchitectureshavetwofundamentalproblems: Theyretoocomplex

andtootopographicallydiverse,whichimpactsperformanceandtheuserexperience.

14


16/137

Intodaysdatacenternetwork,applicationperformanceisoftendependentonthe

physicallocationofserversthataretryingtocommunicatethatis,whereeachofthe

resource

reside

in

the

tree

hierarchy

relative

to

one

another.

Themorehopsrequiredtocompleteatransaction,themorelatencythattransactionis

subjectedto,contributingtounpredictableapplicationperformance.

Inatypicaltreeconfiguration, wehaveaserverandstorageresourcesthataretypically

containedinalocalizedbubbles.Theresourcesareclosetogetheraspossible,which

worksfinewhenwehavefewapps.Butwhenappsgrowscalabilitybecomesanissue

withtreestructures.

Sowhyarebubblesinteresting? Whenprovisioningordynamicallymigratingavirtual

machineinstance,

the

VM

and

its

external

data

sources

would

ideally

reside

within

the

samebubble.

However,iftheVMisinstantiatedoutsidethisbubble,itmightsuddenlyfinditselfthree

networkhopsawayfromdatasourceandtheapplicationwillslowdown.

ThustreesforcetoproactivelymanagethelocationoftheVMinthephysical

environmentinordertomaintainpredictableapplicationbehavior.

15


17/137

Anotherproblemwithtreearchitectureisthat,whenanappliancesuchasafirewallor

loadbalancerisinsertedinlineinthenetwork.

Atypicalsegmentation(VLAN)isdefined,theseattributesaregenerallyrelevanttoa

specificbranchinthenetworkessentiallytheycastashadowdownthatbranchofthe

tree.

WhentheVMisinstantiatedwithintheshadowofitsappliances,itwilloperate

properly. However,ifthatsameVMmigratesoutsidetheshadow,atbestitwillrun

slower.

Atworst,theapplicationmayceasetorunatall.

16


18/137

Inanetwork,therearetwopredominantpointsofmanagement:

theswitchesand

the

interactions

between

the

switches.

Sinceeachswitchisanautonomousdevice,ITmustmanagethemindividually,which

createssignificantoverheadinlargedatacenters.

Theseswitchesalsocommunicatewitheachotherbyusingsharednetworkprotocols

suchasrouting,linkaggregationgroups(LAGs),qualityofservice(QoS)andsecurityall

examplesofprotocolsthatmustbeappliedacrossthenetworkandconfiguredonevery

devicetosupporttheoveralldesign. Thiscausesthenumberofinteractions(often

referredtoascontroltraffic)betweentheswitches,eveniftheyarenotdirectly

connected,togrowgeometrically.


19/137

Thisexplosionofinteractionsfollowsasimpleformulaofn(n1)/2wherenrepresents

thenumberofswitches. Basedonthisformula,10switchescangenerate45such

interactions;

100

switches

can

generate

nearly

5,000

potential

interactions;

and

1,000

switchescancreate5millioninteractions.

Sinceitisnearlyimpossibletodealwiththislevelofincreasingcomplexity,network

operationswilltypicallysegmentthenetworkintosmallersubnetworks.Unfortunately,

thisrunscountertothedesiretoconsolidateallserversandstorageinthedatacenter

intoasinglelarge,moreefficientandmoreelasticpoolofresourcesthevisionbehind

cloudcomputing.

18


20/137

19


21/137

BusinessBenefitsofQFabric

This

slide

presents

the

business

value

points

of

QFabric.

Application

performance

is

an

importantcomponentofmovingtoanarchitecturethatusesQFabric.Byspeedingup

theaccesslayerandallowingforconcepts,suchasvMotion(theabilitytomoveaVM

aroundthedatacenter),notonlytheendusersbutalsotheadministratorscanbenefit.

ThesecondvaluepointisthatQFabricistheidealfoundationforthemoderncloud,with

virtualizedserverandstorageenvironments.Itscaleswell,allowingforover6000ports,

yetmaintainsthesimplicityofmanagingasingledevice,withaflatanytoany

connectivitywhereeveryportisequal,andistheidealfoundationformovingworkloads

aroundthedatacenterdynamically.

Thethird

point

is

simplicity.

Less

hardware

means

there

are

fewer

components,

which

improvesperformance,power,spaceandcooling,aswellasincreasesreliability.QFabric

isalsosimpletomanage,sinceithastheoperationalsimplicityofaswitch.

Lastly,thenatureofthearchitectureallowscustomerstolowertheirOpExandCapEx.


22/137

ThreeProblemstobeSolved

Theproblems

are

related

to

the

basic

flaws

in

atree

structure

in

adata

center.

The

first

probleminvolveshavingtomanageseveralseparatenetworks.Thisiswhyconvergence

isimportant:gettingstorageandallthenetworkstobeinonephysicalnetwork,and

thenusingvirtualizationtocreatethetrafficseparationthatisrequired.

ThesecondissueiscalledMetcalfesRevenge,whichmeansthat,asthecapacityof

thenetworkisexpandedinalinearfashion,thecomplexityincreasesgeometrically:

(N(N1))/2.OneofthefeaturesofQFabricisthatitismanagedasasingledevice:N=1.

Lastly,theres

the

tree

structure

itself.

Traffic

is

trying

to

go

east

and

west

in

an

architectureoptimizedfornorthsouthtraffic.Processinghastoberepeatedateachhop

inthenetworkor,moreimportantlyforthecloud,itmatterswheretheapplicationis

placedrelativetoitsdata.Thiscanleadtoinconsistentapplicationbehaviorwhenthe

applicationismovedaroundinthenetwork.JunipersQFabricprovidesaflat,anytoany

topologytoconnecteverythinginthedatacentertogether.


23/137

ThreeTypesofFabrics

Three

types

of

fabrics

can

be

considered

in

the

marketplace

today:

marketing,

overlay

andswitchfabrics.


24/137

MarketingFabric

The

basic

tree

structure

made

out

of

Ethernet

networks

is

considered

by

some

marketingdepartmentsasafabric.Itrealityitisnotafabricandthushasnoreal

benefitsforthecustomer.


25/137

ProtocolOverlayFabric

The

second

kind

is

called

a

protocol

overlay

fabric.

This

is

a

design

that

uses

a

classic

spineandleaftopologywheretheleavesaretheaccessnodesintothenetworkandthe

spineistheinterconnect;eachnodeinthespineisconnectedtoeveryleafandevery

leaftoeveryspine.Thisisaclassicnonblockingtopologythatcouldbetracedbackto

Mr.CharlesClos,whocameupwithaminimumnumberofdevicesrequiredtobuilda

nonblockingnetwork.

Thishasbeenusedinthewideareanetworkforyears.Whyhasntitbeenusedatthe

datacenter?IthasntbecauseoftheshortcomingsofEthernet.Ethernetisefficientat

thefirsthop(plugandplay),butonceitgetstothepointwheretherearemultiple

paths,

it

has

shortcomings.

In

the

past,

the

Spanning

Tree

Protocol

(STP)

was

used

to

overcometheissue,butSTPallowsonepathtobeusedwhileblockingothers.This

invariablytranslatestoinefficientuseofresources,particularlybandwidth.


26/137

ProtocolOverlayFabric

To

overcome

STPs

disadvantage,

two

new

protocols

have

been

developed.

The

first

one

isTRILL(TransparentInterconnectionofLotsofLinks)andtheotherisSPB(ShortestPath

Bridging).TheyareusedtocreateLayer2tunnels(similartoMPLS,butattheLayer2

level)andtrafficcanthenbespreadacrossdifferenttunnels.Thisleadstoaflatter

topologyandmakeseveryaccessportrelativelyequal,thususingallthebandwidth.

However,therearestilltwotiersofindividualswitches.Moreimportantly,iteliminates

theneedforSTP.Althoughthisleadstoadramaticimprovementtothenetworksofthe

past,Juniperhasdevelopedanevenbettersolution.


27/137

SwitchFabric

This

leads

to

the

third

fabric

type,

known

as

a

Switch

Fabric.

It

represents

building

a

datacenterfabricthatbehavesasasingleswitch,allowingforasinglenetworkasit

enablesconvergence,ismanagedasasingledevice,andhasflat,anytoany

connectivity.

Thisfabrictypehasthebenefitsofthepreviousfabric,butinsteadoftwotiers,ithas

onlyone.Itistheflattestofallthetopologies,whichmakesitthefastest.Also,it

virtualizeslocality likebefore,everyportisequal anditeliminatestheneedtorun

STP.Thereisnoneedtorunanyprotocol(TRILL,SPB)insidethefabricasitisthe

hardwarethatdoesthetransportfrompointAtopointB,justasthefabricinsidea

switch;hence,thenameandbehavior.

Otherbenefitsincludebeingmoreefficient.Fewerdevicesallowforpowersavings

(whicharedirectlyrelatedtocoolingsavings)aswellasspacesavings.Itisalsosimpler

asitismanagedasasingledevice(N=1).Asimplersolutionwithlessdevicesimplies

thatcustomerscanrealizesubstantialcostsavings.


28/137

Giventheproblemswithlegacydatacenternetworkdesigns,anewarchitectural

approachisrequired,onethatistransformationalandinnovative,notincremental.As

Yankee

Group

researchers

put

it:

The

data

center

network

is

now

the

backplane

of

the

virtualdatacenter.

Theidealnextgenerationnetworkarchitectureformoderndatacenterswoulddirectly

connectallprocessingandstorageelementsinaflat,anytoanynetworkfabric.

Optimizedforperformanceandsimplicity,thisnextgenerationarchitecturewould

addressthelatencyrequirementsoftodaysapplications;supportvirtualization,cloud

computing,convergenceandotherdatacentertrends;scaleelegantly;andeliminate

muchoftheoperationalexpenseandcomplexityoftodayshierarchicalarchitecture.

27


29/137

28


30/137


31/137

30


32/137

JuniperNetworkshasdevelopedacomprehensivestrategydesignedtomakethisnew

fabric

based

data

center

network

design

a

reality.

AttheheartofthisstrategyisJunipers321DataCenterNetworkArchitecture,which

eliminateslayersofswitchingtoflattentodaysthreetiertreenetworkstructuretotwo

layersand,withtheQFabricarchitecture,tojustone.

QFabricisthe1in321.

31


33/137

3232

Thesimplest,mostagileandmostefficientnetworkcontainonly2switches(for

availability)Ithasthelowestlatencyandanytoanyconnectivityatfullportcapacity

over

a

non

blocking

fabric.

Unfortunately

real

networks

dont

look

like

this

largely

because

Largenumberofportsthatneedtobeaggregated

Multipleoversubscribedswitchingtiersusedtoaggregatethoselargenumberofports

andstilltryanddoitforlowercostthanafullfledgedanytoanyfullmesh


34/137

3333

WiththeSRX,wecandealwiththephysicalflows.WeacqiredacompanycalledAltor.

vGWisafirewallforvirtualizedserverenvironments,whereitallowsyoutoenforce

policies

between

the

networking

flows,

between

VMs.

The

major

benefit

is

getting

visibilityintowhatthoseflowsare,howtheyoperate,andsettingpolicythatisone

policyacrossthenetwork,regardlessofwhetheritsaphysicalfloworvirtualflow.

Juniperistheonlycompanytodaythatcanletyousetonepolicythatrunsacrossboth

theSRXandvGW.

TheinterdatacenterconnectivitywedoverywelltodaywithMPLSandVPLS.

JunosSpacevirtualcontrol.Oneofthebigproblemsthatwehaveinavirtualized

environmentisthatwehaveasetofvirtualconnectionsinsteadofphysicalconnections.

SoVirtualControlisanapplicationthatwedevelopedwithVMwarethatallowskeeping

virtualand

physical

policies

in

sync.


35/137

34


36/137

3535


37/137

3636


38/137

3737

Nowwehaveasinglepoolorsinglebubbleacrosstheentiredatacenter.

Everythingisequaldistanceandeverythingisonenetworkhopaway. Fromthat

standpoint,

its

the

ideal

seamless

resource

pool

foundation.

TheStratusprojectiswhatcreatedtheQFabric.


39/137

38


40/137


41/137

QFabrictrulyisarevolutionaryarchitecture.Althoughinterestingenough,ithasareally

welldefinedoperationalmodelthatpeopleunderstand thatofasingleswitch.What

we

are

building

is

a

flat,

resilient

fabric

where

everything

is

exactly

one

hop

away,

but

it

scaleswithoutcomplexity.Nequalsone.

40


42/137

Wereallyhadtorethinkthefundamentalwaytoscaleaswitch.Wehadtolookateveryaspectofthedesigntooptimizehowaswitchcanscaleathighperformance.

Dataplane:Theonlywaytoscalethedataplaneistopushtheintelligencetotheedge.The

intelligenceiswhatdealswiththerealworldcomplexityofEthernet,thedifferentprotocolstheswitchhastocommunicatewithexternaldevices.Butwhatsinthemiddleissimplytransport.ItssimplymovingthebitsfrompointAtopointB.Wedonthavetohaveintelligenceinthemiddle.Byhavingtheintelligenceoutthereonedge,Icanscaleatspeed.Moderndatacenterswanttogoasfast,weareat10Gignow.Peoplearealreadyaskingfor40and100Gigsoon.Thatstheonlywaytoscale.

ControlPlane:Ontheotherhand,thecontrolplaneisalittlebitdifferent.Wewanttheintelligencetobeeverywhere.Intelligencehastobelocal,wheretheswitchinghappens,buttheswitchalsohastohaveacentralconsciousnesstoit,sothatthingsarewellcoordinated.Thats

reallywhat

we

talk

about

federated

intelligence.

It

addresses

scale,

but

it

also

retainsresiliency.

ManagementPlane:Finally,wewantamanagementplanewhereN=1.Ithastheoperationalmodelofasingleswitch.WhenwesayN=1,itmeansthatitisonedevicetomanageandtherearenointeractionswithinthefabricitself.Thefabrictakescareofthemanagementdetails.WedontrunspanningtreeorTrilltotrytomanagetheloops,becauseloopsaremanagedthesamewaytheyareintheswitch bythecontrolplaneitself.

Sincewetrulyhaveasinglecontrolplane,viafederated,wecanhaveacompletelydifferentmanagementscheme.SoitislikeasinglepaneofglasstomanageQFabricasifitsasingledevice.

41


43/137

42

Letstakeyouthroughabitofajourneyheretotrytoexplainthearchitecture.Imgoing

tostartwiththeconceptofachassisswitch.Itcouldbean8200fromus.Itcouldbea

Nexus

7000,

Catalyst

6500,

or

an

HP

12000.

They

are

all

basically

very

similar

in

how

theybehave.Wehavelinecards,whichcontaintheports.Wehavesomesortofa

centerplaneorbackplanethatyoupluginto.Thenthereisfabriccircuitry,which

interconnectsalltheportsineachlinecardtoalltheotherportsinthelinecards.

Ethernetpacketscomein,someinitialprocessingattheegressport,andthenthebits

aresprayedacrossthebackplaneinanonblockingfashionandreassembledatthefar

sideandthenEthernetout.WhatyouexperienceasauserisEthernetinandEthernet

out,butyoudontmanagethesebits.Itsaveryefficienttransport.


44/137

43

Onthecontrolplaneside,thereisasingleCPUthatendsupbeingthecentralbrain.We

callitrouteengines.Ciscocallsitasupervisorcard.Itbasicallyisthecentralbrain,but

the

intelligence

is

also

federated,

because

each

of

the

line

cards

has

intelligence.

Where

isthestatemanaged?Theanswerisyes.Itslocal,becausethatiswhatgivesthe

switchitsspeed.Itsalsocentralized,soitcandocontrolplanelearning.Ilearnanew

MACaddress.Everyportknowsaboutit notbecauseIhadtofloodtogetthere,but

becauseitdoescontrolpanelearning.

TheCPUalsoisthepointofconnectionforthemanagementplane.Itprojectsoutthe

CLI,ortheGUI,ortheinterfacetoamoresophisticatedenvironmentlikeJunosSpace,

Tivoli,orsoforth.Italsorunsasetofsoftwarethatautomatesalotofthesoftwarethat

runsthemanagementoftheswitch.Ipluganewlinecardin.Itsautomatically

detected,discovered,configured,andthenbroughtintopartofthesystem.


45/137

Weliketheoperationalmodelofthesingleswitch.Whatwedontlikeisthescaling

model.

Because

when

we

fill

up

all

the

slots,

thats

as

far

as

it

scales.

Now

I

have

to

go

to

a

multitiertypeoftreestructure.

Withmostvendorssolutions,anewswitchneedstobepurchasedtoaddcapacity.

ThisiswhereQFabricinnovationjourneybegan...Wequestionedtraditionalmodeland

evolvedthechassisarchitectureintodistributedchassisarchitecture...

So,wekepttheoperationalmodel,butwechangedthewayitscales.

44


46/137

Westartbymethodicallybreakingthepiecesupphysically.Onthedataplaneside,itreallystartswiththeseparationofthefabricfromthelinecards.Insteadofusingcoppertracestoconnectthings,weusefiberoptics.Thiswaywegofrombeingabletosupport8linecardsto

128

line

cards.

We

get

a

much

better

scale

and

physical

connectivity,

so

we

can

span

it

across

a

largearea.

Whatdothephysicalcomponentsactuallylooklike?Weputthefabricbitsintoachassisandweoptimizethelinecardsforserverconnectivity a1RU,topoftherack,typeofdesign.Inbetween,wehavefour40Gigabitfibersconnectingituptocenterwithstandardfibers,soOM4,Basically,itsEthernetinandEthernetout.Thisworksexactlythesamewayasafabricinsideofachassisswitch.Itssimplytransport.Whenitstransport,itmeansthedevicescanbesimpler.Theycanbefaster,denser,andlessexpensive,becausetheyonlyhavetodotransport.Theydonthavetodealwiththecomplexitiesoftherealworld.Theydonthavetorunmultipleprotocols.Theydontrunprotocols.Theydoonethingandonethingonly.Wewillneverhaveasinglepointoffailure;theminimumconfigurationistwoofthesechassis.Now,itisatthispointintimewhencustomersbegintogetcurious,becausetheywanttotreatthisasanetwork.Butthesearenotswitches.Noticetherearenoconnectionsbetweenthesetwodevicesonthedataplaneitself.Alllinksareactive.Thehardware,usingtheefficientalgorithm,willsay,Thispacketisgoingupthisfiberandcomingbackdownthatfiber.Itsallbuilt

into

the

hardware

itself.

Its

coordinated

from

the

top

to

manage

link

status

and

so

forth.

Forlargescale,wecanhaveuptofouroftheseinterconnectchassis,withonefiberconnectiontoeachone.Italsogivesgreatestresilience.Thisconfigurationwouldsupport128oftheQF/Nodedevices.EachQF/Nodedevicehas4810Gigports.Thereasonwedrawthepictureasacircleisthatsreallythewayitbehaves.EverythingintherealworldconnectstotheQF/Node,andthefabricissimplytheinterconnectbitsherearesimplytransport.SowecallthesetheQF/Nodesandcenterchassisaretheinterconnectchassis.ItsEthernetinanditsEthernetout,withanefficienttransportprotocolinbetween.Mostoftheprocessingisdoneattheingressportandsomeprocessingisdoneattheegressport,justlikeaswitch.Now,herestheamazingthing.ThisfabricisfasterthananyEthernetchassisswitchmadetoday.Itslessthanfivemicrosecondstogofromaportheretothatporttherewithmaximumcablelengths.Theotherthingis,becausethisisanoncongestingtypeofenvironment,becauseitisarelativelysmallnumberofASICs,thejitterinlatencyisextremelylow.

45


47/137


128

line

cards.

We

get

a

much

better

scale

and

physical

connectivity,

so

we

can

span

it

across

a

largearea.


into

the

hardware

itself.

Its

coordinated

from

the

top

to

manage

link

status

and

so

forth.

Forlargescale,wecanhaveuptofouroftheseinterconnectchassis,withonefiberconnectiontoeachone.Italsogivesgreatestresilience.Thisconfigurationwouldsupport128oftheQF/Nodedevices.EachQF/Nodedevicehas4810Gigports.Thereasonwedrawthepictureasacircleisthatsreallythewayitbehaves.EverythingintherealworldconnectstotheQF/Node,andthefabricissimplytheinterconnectbitsherearesimplytransport.SowecallthesetheQF/nodesandcenterchassisaretheinterconnectchassis.ItsEthernetinanditsEthernetout,withanefficienttransportprotocolinbetween.Mostoftheprocessingisdoneattheingressportandsomeprocessingisdoneattheegressport,justlikeaswitch.Now,herestheamazingthing.ThisfabricisfasterthananyEthernetchassisswitchmadetoday.Itslessthanfivemicrosecondstogofromaportheretothatporttherewithmaximumcablelengths.Theotherthingis,becausethisisanoncongestingtypeofenvironment,becauseitisarelativelysmallnumberofASICs,thejitterinlatencyisextremelylow.

46


48/137


128

line

cards.

We

get

a

much

better

scale

and

physical

connectivity,

so

we

can

span

it

across

a

largearea.


into

the

hardware

itself.

Its

coordinated

from

the

top

to

manage

link

status

and

so

forth.

Forlargescale,wecanhaveuptofouroftheseinterconnectchassis,withonefiberconnectiontoeachone.Italsogivesgreatestresilience.Thisconfigurationwouldsupport128oftheQF/Nodedevices.EachQF/Nodedevicehas4810Gigports.

Thereasonwedrawthepictureasacircleisthatsreallythewayitbehaves.EverythingintherealworldconnectstotheQF/Node,andthefabricissimplytheinterconnectbitsherearesimplytransport.SowecallthesetheQF/nodesandcenterchassisaretheinterconnectchassis.ItsEthernetinanditsEthernetout,withanefficienttransportprotocolinbetween.Mostoftheprocessingisdoneattheingressportandsomeprocessingisdoneattheegressport,justlikeaswitch.Now,herestheamazingthing.ThisfabricisfasterthananyEthernetchassisswitchmadetoday.Itslessthanfivemicrosecondstogofromaportheretothatporttherewithmaximumcablelengths.Theotherthingis,becausethisisanoncongestingtypeofenvironment,becauseitisarelativelysmallnumberofASICs,thejitterinlatencyisextremelylow.

47


49/137


50/137

49


51/137

Directorisalsowhatprovidesthesinglemanagementplane.ItisasingleconsciousnessthatwecantalktoanditprojectsoutaCLIorGUIoraDMIinterfaceintomoresophisticatedorchestrationenvironments.Italsodoesalotofautomation.Forexample,

ifIplug

in

anew

QF/Node

device,

its

automatically

detected

and

discovered.

Its

configuredanditslocalinformationissetupandspreadacrossthefabric.

50


52/137

ThedistributedcontrolplaneisimplementedontheinterconnectandQF/Nodedevicesandiscentrallymanagedbycontrol

Clusterofphysicalredundantservers.

AllcontroltrafficiscarriedovertwoseparateGElinkstocontrol.

Controlplaneisolatedthefromthedataplane

51


53/137

NoSinglepointofFailure.QFabrichasresiliencybuiltinatmanystages:Hardware,DataPlane,ControlPlane,ManagementandSoftware.

52


54/137

53


55/137

54


56/137

ThedistributedcontrolplaneisimplementedontheinterconnectandQF/Nodedevicesandiscentrallymanagedbycontrol

Clusterofphysicalredundantservers.

AllcontroltrafficiscarriedovertwoseparateGElinkstocontrol.

Controlplaneisolatedthefromthedataplane

55


57/137


58/137

FCoETransitSwitch:

TheQFX3500offersafullfeaturedDCBimplementationthatprovidesstrong

monitoring

capabilities

on

the

top

of

rack

switch.Inaddition,FCInitiationProtocol(FIP)snoopingprovidesperimeterprotection,

ensuringthatthepresenceofanEthernetlayerdoesnotimpactexistingSAN

securitypolicies.

FCoEFCGateway:

InFCoEFCgatewaymode,theQFX3500eliminatestheneedforFCoE

enablementintheSANbackbone.

Organizationscanaddaconvergedaccesslayerandinteroperatewithexisting

SANswithoutdisruptingthenetwork.

TheQFX3500allowsupto12portstobeconvertedtoFibreChannelwithout

requiringadditional

switch

hardware

modules.

57


59/137

58

Therearethreeuniquepiecesofhardware.Thereistheinterconnectchassis,the

QF/Node,andtheDirector.

58


60/137

Theinterconnectchassisisinahalfrackenvironment sowecanactuallystacktwoon

top.Thefronthastheairintakeandhasthebusinessside.Thesearetheeightfront

facing

fabric

cards.

Each

one

has

16

40

Gigabit

ports.

These

are

QSFP

standard

connectors.Thebacksidehasthedualpowerandredundantfans.Themiddlestageis

fabricsandredundantcontrolboards,whichareaboutmonitoringandmanagingthe

chassisitself.Eachcontrolboardhastwodifferentconnectionstothecontrolplane.We

usestandardEthernetconnectorseverywhere.

Wehaveeighttimes16,or12840Gigconnections.Again,thesearenotswitches,in

thatsense,butifyouadditupusingmarketingmath,whichiscountingbothdirections,

thatis10.2Terabitspersecond.Wecanhaveamaximumoffour.Thats44Terabitsper

second,whichisaninordinateamountofcrosssectionalbandwidth.

59


61/137


62/137

6161


63/137

Directorisa2RUIntelbasedserver.Itsaquadchipwithenoughmemoryandlotsof

networking.Ifyouhavetwodifferentpowergridscomingintoyourdatacenterwe

recommend

connecting

them

on

different

power

grids.

62


64/137

63


65/137

64


66/137

TheQFabrichasbeenindevelopmentfor3yearsandwehave125pendingpatents

applicationsandhavebeengrantedthreesofar.Wehavebeenveryactiveindoingthe

patent

applications,

because

there

is

are

many

unique

innovations

in

QFabric.

65


67/137

66


68/137

67


69/137

68

Thispictureshowsminimumconfigurationthatwillbeshippedinitially.Twochassiswith

twoconnectionseachwillgivethreetooneoversubscriptionattheQF/Node.Thiswill

be

good

up

to

64

devices

or

just

over

3,000

ports.

68


70/137

69

Themaximumconfigurationwouldbeaddingtwomorechassisthatcanbasicallyrewire

itsoQF/Nodecanhaveoneconnectioneach.Thatwouldtaketheconfigurationupto

6,144

ports.

69


71/137

TheQFabricarchitectureprovidescompleteinvestmentprotection,

allowingcustomerstodeploytheQFX3500firstasahighperformancedatacentertop

of

rack

switch

andthentransitionittoaQF/NodeedgedeviceintheQFabricarchitectureattheirown

pace,withoutrequiringawholesalereplacementoftheexistingnetwork.

TheQFabricarchitectureprovidesaclearpathtoasingletierdatacenternetwork,

helpingcustomersimproveapplicationperformanceandattainoperationalflexibility

withminimaladministrativeoverhead.


72/137

Insomeways,wehavenowinterconnectedeverythinginthedatacentertoonegiant

compute/storagefarm.Whataretheapplicationsthatcantakeadvantageofthat?What

will

come

as

a

result

of

it?

..nowthedatacenteristhecomputer..IbelievetherewillbecoolapplicationsthatwillshowupasaresultofQFabric. AndyIngram(VPofMarketing,FSG)

71


73/137

7272


74/137

7373


75/137

7474


76/137

75


77/137

76


78/137

Assumptions:

*Ports:6000ports

*OS:

3:1

ToR

>

Fabric

Ifwebuilda6000portnetwork,wewouldhavethe4chassis;wewouldhave125ofthe

edgenodes.Thiswouldgiveusacompletenonblockingenvironmentthatbehavesasa

singleswitchinLayer2,Layer3.

Ourcompetitionistryingtodothesamething,buttheyhavetakenadifferent

approach.Wetookacleansheetofpaperanddesignedfromthegroundup,which

meanswehaveminimizedthenumberofpiecesofhardware thenumberofdevices

andmadeitassimpleaspossible.

Ourcompetitorsaretryingtobuildanonblockingflatnetworkoutoftheexisting

buildingblocks.

What

that

looks

like

for

Cisco

fabric

path

is

we

have

Nexus

5000s

at

the

edge,havetworowsofNexus7000s,andIhaveanotherinstantiationtogetmyLayer3

functionalityforthedatacenterbecausetheprotocoltheyrunhereisLayer2only.This

isaprettytypicalwaythatyoubuildaspinalleaftypeoftopologyforanonblocking

fabric.Icanttellyouhowlongittooktodrawalltheselines.

Thescarythingisthateachoneoftheselinesrepresents20differentlinksormore.

Whatyoufindisyoucantactuallybuildthis.Whenyoudothis,whatyouhaveisa

wholebunchofloops,soratherthandoingspanningtree,whichwouldturnoffallthis

bandwidth,theyareusingaTrilllikeprotocolcalledfabricpath.ItisnotTrillanditis

notinteroperable

with

anything

else,

so

it

is

still

proprietary.

77


79/137

ButwhatitdoesisitcreatesLayer2tunnels butitisonlyLayer2.Theyactuallyhave

separatehardwaretodoLayer3,whichcreatesitsownsetofchallenges,becausenow

youactuallyhavetotrytodocapacityplanningbetweentwoflowsoftrafficthatyou

neverhadtomeasurebefore.

Asapointofcomparison,wehave1/3thenumberofdevices,require2/3lesspower,

90%less

floor

space,

and

we

are

7to

10

times

faster.

Source:

DataSheets

DesignDocuments

InternalTestingresults

ReferenceLinks:

http://www.cisco.com/en/US/docs/switches/datacenter/hw/nexus7000/installation/guid

e/n7k_sys_specs.html

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/data_sheet_c78

618603.html

http://www.cisco.com/en/US/products/ps9402/products_data_sheets_list.html

*CompetitiveConfiguration:

Spine:

Chassis18slotwith16payloadslots

Linecardsareonlycapableof230Gfabricswitching

10Glinerateperlinecard:23ports

Latency:5usecwithinlinecard

TRILL

like

supported

Edge:

ToRlatency:2usec

10Gdownlink:36ports

10Guplinks:12ports

NoTRILLlikesupported

77


80/137

Somedatacentersbuilddifferentkindsofnetworksfordifferentkindsofappsandwe

believethatonenetworkshouldsolvealltheproblems,ifdoneright.Wepicked6,000

as

a

big

number,

but

that

also

plays

out

if

you

have

500,

1,000,

3,000,

or

6,000.

TheQFabricarchitecturecanscalefromjustafewhundredtothousandsof

server/storageports,helpingcustomersbuildhighlyscalable,highperformance,highly

efficientcloudready(private,publicorhybrid)datacenterinfrastructures.

Source:

DataSheets

DesignDocuments

InternalTestingresults

Reference

Links:

http://www.cisco.com/en/US/docs/switches/datacenter/hw/nexus7000/installation/gui

de/n7k_sys_specs.html

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/data_sheet_c78

618603.html

http://www.cisco.com/en/US/products/ps9402/products_data_sheets_list.html

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/white_paper_c1

1516396.html

*CompetitiveConfiguration:

Spine:

78


81/137

Chassis18slotwith16payloadslots

Linecardsareonlycapableof230Gfabricswitching

10Glinerateperlinecard:23ports

Latency:5usecwithinlinecard

TRILLlikesupported

Edge:

ToRlatency:2usec

10Gdownlink:36ports

10Guplinks:12ports

NoTRILLlikesupported

78


82/137

Assumptions:(Averagenumbers)

Power/KWH $0.11

HVAC

&

Lighting

/

KWH $0.13

RackSpace/Year $9,000

NetworkEngineer/peryr $200,000

79


83/137

Assumptions:(Averagenumbers)

Power/KWH $0.11

HVAC

&

Lighting

/

KWH $0.13

RackSpace/Year $9,000

NetworkEngineer/peryr $200,000

80


84/137

TheQFabricitselfisincustomertrialsaswespeak.Becausethisissolutionandnota

productwegothroughintensiveandextendedcustomertrials.Weareexpectingto

bring

it

to

market

in

Q3

of

this

year.

The

trials

will

probably

last

4

to

6

months,

dependingonthecustomer.

Intermsoffuturedirections,westartedinthemiddleintermsofhowthefabricscales

andthepricepointscales.Wewillhavemegafabricsthatwillultimatelyscaleto10sof

thousandsofphysicalportsandwillhave100softhousandsofvirtualports.Wewill

alsoscaledownthatwillbeveryeconomical.Wewillhave40and100Gigabitaccess

speedsaswell.

81


85/137

QFabric:Theperformanceandoperationalsimplicityofasingleswitchandthe

scalabilityandresiliencythatyouwouldexpectfromasinglenetwork.

Webelievethiswilltransformthedatacenternetwork.QFabricwillmakeitbetterfor

doingvirtualization.QFabricwillmakeitsimplertomanage.QFabricwillmakethedata

centerbetter,whichgivesitabetterlevelofagility.Insteadofhavingthetradeoff,we

believewecandriveuserexperience,andthequalitythereof,andtheeconomicstoa

higherlevel,aswellastakethecosttoalowerlevel.

82


86/137

83

Achangeindatacenternetworkdesignisneededtoensurethatorganizationscantake

fulladvantageoftheirinvestmentsinnewapplications,virtualization,andstorageand

compute

resources.

The

most

efficient

way

for

resources

to

interact

is

for

them

to

be

no

morethanasinglehopawayfromeachother.Itstimetobreakthenetworkbarriers

andbuildanetworkenvironmentthatisoptimizedforperformanceandsimpleto

operate.

QFabricarchitecturewouldaddressthelatencyrequirementsoftodaysapplications,

eliminatethecomplexityoflegacyhierarchicalarchitectures,scaleelegantly,and

supportvirtualization,convergence,andcloudcomputingandotherdemanding

requirementsforthenextgenerationdatacenter.

Insomeways,wehavenowinterconnectedeverythinginthedatacentertoonegiant

compute/storagefarm.

What

are

the

applications

that

can

take

advantage

of

that?

What

willcomeasaresultofit?

..wealwayssaidthenetworkisthecomputer,butnowthedatacenteristhecomputer...IbelievetherewillbecoolapplicationsthatwillshowupasaresultofQFabric. AndyIngram(VPofMarketing,FSG)


87/137

84

HereisrearviewofQFX3500switch

Itishighperf,highden10Gswitchdesignedspecificallyfordatacenterdeployments

It hasmax6310g ports,440gports

Featuring48dualmode10Gports,andfourQSFP+portsina1Uformfactor

QFX3500deliversrichLayer2andLayer3connectivitytodevicessuchasrackservers,

bladeservers,storagesystems.

Itisahighperformanceswitch,offersflexiblelatncymodeswithcutthroughorstore&

forward:bothwithsubmicroseclatency.

Forconvergedserverenvironments,theQFX3500isalsoastandardsbasedFibre

ChanneloverEthernet(FCoE)transitswitchand

FCoEto

Fibre

Channel

(FCoE

FC)

gateway,

enabling

customers

to

protect

their

investmentsinexistingdatacenteraggregationandFibreChannelstorageareanetwork

(SAN)infrastructures

Features

JunosL2andL3

RichDCBimplementation

FCoETransitSwitch/FIPsnooping

FCoEFCGateway(NPIV)

40Greadyhardware

84


88/137

Laye2switching&FeatureRichDCB

ULLcutthroughorstore&forward,LargeMACtableof96K,Servervirtualization

supportinhardware

Layer3switching

ULLcut

through

or

store

&

forward,

Layer

3protocols

in

2H2011

FCoEtransitSwitch/FIPSnoopingandcanprovideFCoEFCGatewayfunctionlaitforup

to12FCPorts

84


89/137

Highfrequencytradinginfinancialservicesuserealtimemarketfeedstoenactmillionsoftradespersecond.Financialinstitutions suchasJPMorganrelyonhighspeednetworkstocarrythesetransactions.Forthem,fasteraccesstotransactions=more

money.

Highperformancecomputing(HPC)usesparallelprocessingforrunningadvancedintensiveoperationsefficientlyandquickly.ThemostcommonusersofHPCsystemsarescientificresearchers,seismographers,militaryandacademicinstitutions.

Infinancial andHPCsector,highperforamnce:lowlatency,LowOversubscription inlowornonblockingconfigurations isakeycriterionforgainingcompetitiveadvantage.IfwelookatexistingplayerssuchasArsita,CiscoorBNT,therehasbeentradeoffbetweenlatency,wirespeedperfandportdensity.

Notanymore withQFX3500

Portdensity

means

fewer

devices,

lesser

number

of

hops

enable

faster

communication

andminimizesoverallapplicationlatencyandshrinkdatacenterfootprint.Scalabilityandportdensitymeansthatthenetworkcanbeexpandedtomeetincreasingdemands.Highthroughputwithabilitytosendlargefilesormanysmallfileswithinacertainamountoftime iscriticalformarketdatadelivery

Financialsrequire

85


90/137

QFX3500combineshighperformancewithultralowlatencywithconsistentperand

highportdens.

Delivering

sub

microsecond

low

latencies

demand

cut

through

and

shared

memory

switchingtechnologies.QFX3500


91/137

Cloudbasedapplicationsprovideuniqueadvantagesforondemandprovisioningand

elasticcomputing

capacity

with

a

pay

as

you

go

pricing

model.Thereqsare:

Scale

HighPortdensity

LargenumberofVLANs

LargeMACaddresstables(VMs):asEastWestVMtrafficinflatL2network

needlargeMACtablesonedgeofnetwork

I/oConvergencetoreducenumberofCNAsandachieveeconomies

WithQFX3500,TargetissmallscalePrivateandPublicCloudsforL2/L310Gserver

access

Cloudcomputingtakestheformofwebbasedtools,applicationsorstoragethatusers

canaccessthroughaweb dynamically.

87


92/137

JuniperssolutionsprovidesthescaleandelasticityneededforCloud

Large

(63)

10GbE

port

densityWithLargest96KMACaddresstables,itSupportlargeL2domain

Ciscocanonlyprovidemax48portwithanexpansionmodule,theydontfullfullI/o

convergenceandhavelimitedmactable

WcaneasilybeatAristaasithasnosacle,withnopathtofabricandlimitedvirtualz

support.

QFX3500providesManagementSimplificationreqiredforcloudsproviderAspartoffabric,multipleQFX3500canbemanagedasasingledevice,which

leadstolesscomplexityandsavingsinopex.

88


93/137

ForAtypicalEnterprisesuchasGM,Mercado,datacneterisacostcenter.

TheEnterprisemarketissegmentedintoSmall,midlevelenterpriseandlarge

:

Agregation

:

10G,

L3SmallmidEnterpriseDC:10GusedforDCAggregation.

Keycharacteristsare highavailability,portdensityandavailabilityof

L2/L3switchingandroutingprotocol.

OntopoftheseLargeEnterprise:need10Ginaccess.

AlsostaorageconvergencewithFcoetransitandgatewayfunhelpsconsolidateresorce

andsavecost.

TheincumbenetweseeherewillbeCiscowith4900MandNx5548.

89


94/137


95/137

QFXstandsoutamongstalltheswitchesinthese3segments

In

HFT/HPC

it

is

the

Fastest

(sub

microsecond

at

all

packet

sizes)

and

with

L3

at

all

ports

InClouditofferstelargetsportdensityandvirtualzationsacle

Inenterprise,itprovidesoptimuncost/portintermsofopexandpower,sapceusageas

wellasamangeabilityaspartoffabric.

91


96/137

CompetitivevendorscansupportonlyasubsetofLAfeatures.

CiscooffersFCoEandFCgatewayfunctionalitybut lacks:

ULLFullL3toeveryport

VEPAor

40Euplinks.

AristaoffersULL,40GEuplinks(maybe),partialL3butlacks

fullL3

BNToffersultralowlatencyandL3butlacks

FCgateway

92


97/137


98/137


99/137

Su:Tomeetthesedifferentneeds,customershavetochoosedifferentswitchestoday.forfinancialapplications wherelowlatencyiscriticalArista7100ispreferredthoughitdoesnotofferportdensityandscale.

ForScaleddatacenters,Cisco5548withits48 10gportsandpathtofabricissuitable.Similarlyforenterprisedatacenters,customersareforcedtocompromiseon

performanceanddeploycatalyst4900tomeetbudget.Introducing PerformancerichQFX3500.It canaddressmultipleDCapplications that

previouslycouldonlybeaddressedwithmultipleswitchesorbysacrificing

functionality.

QFX3500superiorineachsegmentof10GETORwithNocompromisesonfaster,bigger,cheaperandWithoutaddingcomplexity


100/137


101/137


102/137


103/137


104/137

100


105/137


106/137

102


107/137


108/137


109/137


110/137


111/137

Depth

28

Air flow

Front to back

106


112/137


113/137


114/137


115/137


116/137


117/137


118/137


119/137


120/137


121/137


122/137


123/137


124/137


125/137


126/137


127/137


128/137


129/137


130/137


131/137


132/137


133/137


134/137


135/137


136/137


137/137