1 Asr9k Arc Base

7/17/2019 1 Asr9k Arc Base

http://slidepdf.com/reader/full/1-asr9k-arc-base 1/127

© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialViking Hardware Overview 1

ASR9K Overview

ASR9K Overview

ASR9K Chassis

Architecture

RSP

Fabric

Linecard

Life of a Packet

Power Management

Media and File Systems

Online Diagnostics



Cisco ConfidentialViking Hardware Overview 2

ASR 9000 At a Glance

Optimized for Aggregation

of Dense 10GE and 100GE

Designed for Longevity:

Scalable up to 400 Gbps ofBandwidth per Slot

Based on IOS-XR & ANA

for Nonstop Availability and

Manageability

Enables Network

Convergence of Business

and Residential Services




IOS-XR Modular OSAbsolute HW Redundancy

Broad Network ResiliencySchemes

Carrier Class

6.4Tbps Capable SystemEngineered for Nx100GSuperior H-QoS/Scale

Built to Last

Integrated VideoVideo Quality Monitoring

Local Ad-Insertion

Optimal Intelligent Multicast ForwardingVoD Streaming/Cache

Industry Leading Watts/GbpsPay as You Grow Power

Minimal Carbon Footprint

Power Reduction

High Density GE/10GExtensive L2VPN Services

Incorporated SynchEIPoDWDM Ready

Carrier Ethernet Focus

Complete EMS/NMS Support

Consistent ProvisioningModel Comprehensive OAM

Minimize Operational

Expense

Simplicity

ResilientConverged

VideoOptimized

Purpose

Built

“Green”

ASR 9000The (R)evolution Begins!

* Please refer to Road Map

for t im e line of featuresuppor t




Designed with Operators in Mind Software Optimized for Automation, Ease of Use

Simplicity

Manageable via Cisco ANA Toolkit

Cross-platform support of ASR9k, 7600, CRS-1

Standards-based northbound interface to carrierOSS

XR image mgmt, device inventory, fault

correlationCarrier Ethernet end-to-end service provisioning

Embedded ASR9k mgmt with XML over HTTP,SNMP

Complete OAM Coverage

802.3ah for link level fault detection

802.1ag draft 8.1 compliance

MPLS OAM/VCCV support

EtherSLA/Y.1731 Performance Monitoring

(future)




IOS-XR Powers the Edge with ASR 9000A Fully Distributed, Microkernel-Based Architecture

Designed for Scale, HA, and PerformanceFor IP NGN Applications

Next Generation Architecture

Modular Components Applications Architecture

IS-IS

QoS

Distributed Middleware

BGP

Multicast

OAM

VLAN

MAC

Subs

OAM

VLAN

MAC

Subs

OAM

VLAN

MAC

Subs

LC1

Distributed Service Separation

LC2 LCn

MPLS Multicast

RIP BGP

OSPF ISIS

Manageability

Security

Forwarding

Base

Admin

Line Card

HostComposite

RoutingComposite

Microkernel-based design

Highly modular, highly extensible

‘Service-enabled’ blade architecture

Scale through distribution

Unique address tables per linecard

Process-level, stateful subscriber HA

Resilient




Cisco ASR 9000 Architecturally Superior by Design: Chassis & Fabric

Scales to 2x 200G Fabric / Slot

•

Per

-

slot power & cooling

ready for

200G

today

•

Backplane signal integrity

validated for

10G

SerDes

Half Terabit More Dense than Competition

•

ASR9k

: 8 slots *

20x

10GE

=

160x

10GE

•

“Box J”: 11 slots *

10x

10GE

=

110x

10GE

•

“Box A”: 10 slots *

10x

10GE

=

100x

10GE

500G

–

600G

Density Advantage for

ASR9k

Now

200GCard

Maximixed Slot Real Estate

Length x Width x HeightOptimized for 200 Gbps

with Daughtercard

Up to 8x NPUs per Card

Power-Up as You Grow

Modular 1500W Power Bricks

For Green Efficiency

PurposeBuilt


http://slidepdf.com/reader/full/1-asr9k-arc-base 7/127Cisco ConfidentialViking Hardware Overview 7

Dual Fia

80 Gbps

Linecards per Chassis

Bandwidth per Slot

Bandwidth per Chassis

Linecard Density

8 LC + 2 RSP 4 LC + 2 RSP

180 Gbps 180 Gbps

2.8 Terabits 1.4 Terabits

10 slots 6 slots

40 Gbps 40 Gbps

22 Slots

80 Gbps 80 Gbps

6.4 Terabits 3.2 Terabits

400 Gbps 400 Gbps

20 LC + 2 RSP

200 Gbps 200 Gbps

550 Gbps

20 Terabits

400 Gbps

ASR 9000 System ScalabilityOutlasting the Future

0

25,000

50,000

2005 2006 2007 2008 2009 2010 2011 2012

P B / m o MobilityBusiness Internet

Business IP WANConsumer InternetConsumer IPTV/CATV

Source: Cisco Visual Networking Index — Forecast,2007 –2012



Viking History

Viking was initially conceptualized as an L2 aggregation switchwith the Gladiator chassis (with Gracchus and Marcus line cards)in the Metro Ethernet Market with Goryeo taking the routing rolefitting into the SP edge.

When Goryeo was found to be no longer an advantageous

investment, Viking adapted to SP edge market.

This posed changes both in the chassis requirement when RedOctober (with Ryan, Ramius and Mancuso line cards) wasconceptualized and also in the multitude of feature and scalerequirements.


http://slidepdf.com/reader/full/1-asr9k-arc-base 9/127© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialViking Hardware Overview 9

ASR9K Chassis

ASR9K Overview

ASR9K Chassis

Architecture

RSP

Fabric

Linecard

Life of a Packet

Power Management


Platform Fault Management (PFM)Online Diagnostics

Service Console



Chassis Overview ASR-9000 10-slot system

10 slots: 8x linecards + 2x RSP

Half-rack: 17.38”w x 36.75”h x 28”d

Bandwidth (initial)

400Gbps backplane

180Gbps fabric 400G

40G/80G linecards N-100G

Carrier-class hardware redundancy

AC & DC systems

Pay-as-you-grow, modular power

Green emphasis throughout



Hardware Subsystem: 10-slot PowerViking Power Details

750W power & cooling available per LC slot

FCS cards draw significantly less (~500W)

Max system draw 8.5KW, max FCS draw is <= 5KW

AC power supplies are 3kW, DC are 2kW or 1.5kW

Max power configs : 5+1 DC (or) 3+3 AC

FCS power configs: 3+1 DC (or) 2+2 AC



Hardware Subsystem: 10-slot powerPower Distribution (DC N:1 protection)

PS 0 P ow er

Di s t r i b u t i o

nB u s

PS 1

PS 2

PS 3

PS 4

PS 5

Feed A

Feed B

Feed A

Feed B

Feed A

Feed B

Feed A

Feed B

Feed A

Feed B

Feed A

Feed B

LC

LC

RSP

RSP

LC

LC

Fans

Fans

LC

Single power zone, onedistribution bus

All modules load share

2kW and 1.5kW supplies

Each power supply iswired to both ‘A’ and ‘B’feed

Feed failure doubles draw

on remaining feed supply failure increases

draw on remainingsupplies

Shelf 1 (bottom)

Shelf 0 (Top)



Hardware Subsystem: 10-slot powerPower Distribution (AC 1:1 protection)

PS 0 P ow er

Di s t r i b u t i o

nB u s

PS 1

PS 2

PS 3

PS 4

PS 5

LC

LC

RSP

RSP

LC

LC

Fans

Fans

LC



AC power supplies arerates @ 3KW ea.

‘A’ feed wired to top powershelf

‘B’ feed wired to bottom

power shelf

Feed A

Feed A

Feed A

Feed B

Feed B

Feed B

Shelf 1 (bottom)

Shelf 0 (Top)



10-slot chassis: 6kW redundant commons AC “1:1” protection vs. DC “N:1” protection

AC PS3kW

P ow er Di

s t r i b u t i onB

u s

AC PS3kW

AC PS3kW

AC PS

3kW

S YSTEM

LOAD

For 6KW AC, you need 4x 3KW powersupplies (feed failure takes downtwo of them, supply failure kills one)

Feed A

Feed A

Feed B

Feed B

Shelf 1 (bottom)

Shelf 0 (Top)

DC PS2kW

P ow er Di

s t r i b u t i onB

u s

DC PS2kW

DC PS2kW

DC PS

2kW

S YSTEM

LOAD

For 6KW DC, 4x 2KW power supplies(2kw each, feed failure has noimpact, so we protect only against a

supply failure)

Feed A & B

Shelf 1 (bottom)

Shelf 0 (Top)

Feed A & B

Feed A & B

Feed A & B



Hardware Subsystem: 10-slot thermalsViking Cooling Details

Front-to-back airflow

750W cooling per LC slot

FCS cards significantly less

Dual “stacked” fan trays belowLCs

Variable speed fans

at normal ambient temperatures

lower power draw

lower noise profile



Chassis Overview ASR-9000 6-slot system

6 slots: 4x linecards + 2x RSP

¼ rack: 17.38”w x 17.35”h x 28”d

Bandwidth

400Gbps backplane

180Gbps fabric 400G

40G/80G linecards Nx100G

Carrier-class hardware redundancy

AC & DC systems

Pay-as-you-grow, modular power

Green emphasis throughout



Hardware Subsystem: 6-slotViking Power & Cooling Details

side-to-back airflow (no stageheating!)

750W cooling per LC slot

Dual vertical fan trays above cardcage

Tray power draw from 40W-300W

Temperature sensitive greenengineering

FCS draw is significantly less than

750W Variable speed fans

at normal ambient temperatures

lower power draw, lower noiseprofile



Hardware Subsystem: 6-slot powerPower Distribution (AC 1:1 protection)

PS 0

P ow er D

i s t r i b u t i on

B u s

PS 1

PS 2

Feed A

Feed B

Feed C

LC

LC

LC

LC

RSP

RSP

Fans

Single power zone, one

distribution bus, allmodules load share

AC power supplies arerated @ 3KW ea.

Fully populated 40G

system requires <3kW Third source required for

redundancy for >3KWinstallations.

Power Entry Shelf

Fans



Hardware Subsystem: 6-slot powerPower Distribution (DC N:1 protection)

PS 0

P ow er D

i s t r i b u t i on

B u s

PS 1

PS 2

Feed A

Feed B

Feed A

Feed B

Feed A

Feed B

LC

LC

LC

LC

RSP

RSP

Fans



2kW and 1.5kW supplies

Each power supply iswired to both ‘A’ and ‘B’feed

Power Entry Shelf

Fans



ASR9K Power Calculator:

http://wwwin.cisco.com/sptg/crbu/products/asr9000/files/ASR_9000_Power_Model.xls

per-component values computed power values

Qty

Typical

(25C)Max (40C)

Ext. Temp

(55C)

Typical

(25C)

Max

(40C)

Ext. Temp

(55C)

9006 FanTray 0 100 275 375 0 0 0

9010 FanTray 2 200 300 600 400 600 1200

RSP 2 175 205 235 350 410 470

4xNP linecard 4 310 320 350 1240 1280 1400

8xNP linecard 4 565 575 630 2260 2300 2520

TOTALS 4250 4590 5590

To use: select 2 fan trays based on the chassis type. Enter the number of cards and RSPs

configured.

Max power values are tested values with full wire-rate traffic running on all ports, and worst-case

feature sets.

Note that it's the NUMBER OF NPs that matters, NOT the number of ports.

Specific example: the 8-port card with 4 NPs uses far far less power than the 8-port card with

8xNPs

Deployed "real world" systems are likely (but not guaranteed) to draw 10-33% less power than these numbers.


http://slidepdf.com/reader/full/1-asr9k-arc-base 21/127© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialViking Hardware Overview 21

ASR9K System

Architecture

ASR9K Overview

ASR9K Chassis

Architecture

RSP

Fabric

Linecard

Life of a Packet

Power Management



Service Console



Viking Architecture : SummaryCarrier Class Design

High Availability integral to system architecture

Not an afterthought

Modularized System Components – HW and SW

Isolate failure and faults to subsystem/component

HW based signaling for fabric failover

~ Zero packet loss on switchover Redundancy built-in throughout system

RSP, Fabric, Control Plane Ethernet, Chassis ControlBus, Power supplies, Fan Trays

Backplane Point-to-Point connectivity oriented

Limit single point of failuresIsolate faults

Highly Scalable System and dense QoS

Synchronous Ethernet Support

BITS/Network Timing Distribution

MAC

FIC

NPU

NPU

NPU

NPUMAC

MAC

FIC

NPU

NPU

NPU

NPUMAC

MAC

FIC

NPU

NPUMAC

MAC

FIC

NPU

NPU

NPU

NPUMAC

MAC

FIC

NPU

NPU

NPU

NPUMAC

FIC

CPU BITS/DTI

FIC

CPU BITS/DTI

FIC

CPU BITS/DTI

MAC

FIC

NPU

NPU

NPU

NPUMAC

CPUMAC

FIC

NPU

NPU

NPU

NPUMAC

CPU

CPU



ASR9K SW Architecture Highlights

Scalable Forwarding Plane

Ethernet centric, multilayer switch design

Fully distributed packet processing

Two stage forwarding model

IOS XR Based

Sharing same XR release and L3 feature set with CRS-1

Carrier Class infrastructure

Offering S/W modularity, HA, Restartability, Fault Containment, NSF/SSO,SMU, ISSU

Focusing on Carrier Ethernet features

VPLS, EoMPLS, Bridge Domain, EFP, MSTP, Link Bundling, E-OAM, QinQ,IGMP Snooping

High throughput traffic management support

Hierarchical QoS, CBWFQ, WRED, 2R3C Policing

Security, Protection Features

L2,/L3 Security ACLs, CoPP, IP SLA, Mac Limiting, BPDU Filtering



ASR9K Platform basic blocks

Switch Fabric

Route Processor

Switch Fabric

Di s t r i b u t e d

F or w a r d i n g

L i n e

C a r d

Route Processor


F or w a r d i n g

L i n e

C a r d

Line Cards carry forwardingengines, control plane processing

and integral I/O ports

Switch Fabric and Route Processorare combined on one RSP module.

Redundant Route-SwitchProcessors provide central point of

control for the router.

…


F or w a r d i n g

L i n e

C a r d

RSP also provides shared

resources for backplane Ethernet,timing and chassis control




Viking System Block Diagram

OctopusOctopus

CPU

8641D

GE

Switch

SystemTiming Bellagio 2

2Santa

Cruz

Octopus

Punt Path

Punt Path CPU

8641D

GE

Switch


2Santa

Cruz

Octopus

Punt Path

Punt Path

Backplane

Legend

Timing Plane

Data Plane

Arbitration Plane

Control Plane

Punt Path

CPU

8641D

GE

Switch


2Santa

Cruz

Octopus

Punt Path

Punt Path CPU

8641D

GE

Switch


2Santa

Cruz

Octopus

Punt Path

Punt Path

40x1GE

Ryan LC

NP3 NP3

CPU

8641

GE

PHY

NP3 NP3

BRG BRG

10Px

SFP10Px

SFP10Px

SFP

10Px

SFP

OctopusOctopus4x10GE

Mancuso

LC

NP3 NP3

CPU

8641

GE

PHY

NP3 NP3

BRG BRG

10G

PHY

XFP


Ramius

LC

NP3 NP3

CPU

8641

GE

PHY

NP3 NP3

BRG BRG

1 0 G P

H Y

S F P

1 0 G P

H Y

S F P

1 0 G P

H Y

S F P

1 0 G P

H Y

S F P

1 0 G P

H Y

S F P

1 0 G P

H Y

S F P

1 0 G P

H Y

S F P

1 0 G P

H Y

S F P

10G

PHY

XFP

10G

PHY

XFP

10G

PHY

XFP


Thor LCCPU

8641

GE

PHY

10Px

SFP10Px

SFP

CPP

SERPSERP

CPP

PLIM ASIC

RSP1RSP0




Viking Logical Architecture Overview

O c t o p u s O c t o p u s

C P U

8 6 4 1 D

G E

S w i t c h

S y s t e m T i m i n g

B e l l a g i o 2 2 S a n t a

C r u z

O c t o p u s C P U

8 6 4 1 D

G E

S w i t c h



C r u z

O c t o p u s

B a c k p l a n e

Legend

Data Plane

Control Plane

C P U

8 6 4 1 D

G E

S w i t c h



C r u z

O c t o p u s

P u n t P a t h

C P U

8 6 4 1 D

G E

S w i t c h



C r u z

O c t o p u s

40 x 1 GE

Ryan LC

Trident

CPU

8641

GE

PHY

FPGA FPGA

10 x

SFP

10 x

SFP 10 x SFP

10 x SFP


4 x 10 GE

Mancuso

LC CPU

8641

GE

PHY

FPGA FPGA


8 x 10 GE

Ramius

LC CPU

8641

GE

PHY

FPGA FPGA

O c t o p u s O c t o p u s 20 x 1 GE

Thor LC

CPU

8641

GE

PHY

SPA

CPP

FPGA FPGA

CPP

PLIM ASIC

RSP 1 RSP 0

1 0 G E X F P

1 0 G E X F P

1 0 G E X F P

1 0 G E X F P

1 0 G E X F P

1 0 G E X F P

1 0 G E X F P

1 0 G E X F P

1 0 G E X F P

1 0 G E X F P

1 0 G E X F P

1 0 G E X F P

SPA SPA SPA

Trident Trident Trident Trident Trident Trident Trident Trident Trident Trident Trident




ASR9K RSP

(Route Switch Processor)

ASR9K Overview

ASR9K Chassis

Architecture

RSP

Fabric

Linecard

Life of a Packet

Power Management



Service Console

ASR 9K RSP




RSP2 RSP440

Processors 2 x 1.5GHz Freescale8641D CPU

Intel x86 Jasper Forest 4Core 2.27 GHz

RAM (userexpandable)

4GB @133MHz SDR

8GB

6GB (RSP440-TR) and12GB (RSP440-SE)version @1066MHz DDR3

Cache L1: 32KBL2: 1MB

L1: 32KB per CoreL2: 8MB shared

Primary persistentstorage

4GB 16GB - SDD

Secondary persistentstorage (HD/SSD)

30GB - HDD 16GB - SDD

USB 2.0 port No Yes

nV Cluster – EOBCports

No Yes, 2 x 1G/10G SFP+

Switch fabricbandwidth

184G/slot (with dualRSP)

440G/slot (with dual RSP)

ASR 9K RSP (Route/Switch Processors )

Overview

RSP440




Hardware Subsystem: RSPRoute/Switch Processor Details

High performance control plane processor

Redundant RP config

Active/Active fabric model

External interfaces

Console/AUX

2x 10/100/1000 Ethernet

Compact Flash

USB

BITS/DTI




Hardware Subsystem: RSPMain Board Components

CPU: FreeScale 8641D

Memory: 4GB

Hard Drive: 40G HDD

I/O controller: “Long Beach”

Fabric Interface: “Octopus”

Fabric Arbitration: “Bellagio”

Fabric Data path: “Santa Cruz”




CPU Complex Fabric Interface

Timing Domain

EOBC/Control Plane GE

Hardware Subsystem: RSPMain Board Components

CPU8641D

4GB MEM

Octopus

Long BeachI/O FPGA

SantaCruz 0

SantaCruz 1

HDD

Mgt Eth

CF card

Console

PuntFPGA

NVRAM Boot Flash

EtherSwitch

Clock TimeFPGA

BITS

Front Panel

BellagioArbitration

4G CF

Mgt Eth

Aux

Alarm




ASR9K Fabric

ASR9K Overview

ASR9K Chassis

Architecture

RSP

Fabric

Linecard

Life of a Packet

Power Management



Service Console




Hardware Subsystem: FabricFabric Architecture

Fabric is logically separate from LC/RSP

Physically resides on RSP

Similar to 7600, not like CRS/GSR

RSP requests access to fabric just like LCs do

Separate data and arbitration paths

Each LC/RSP has a fabric interface ASIC

1x Octopus 40G usable ethernet connectivity

2x Octopus 80G usable ethernet connectivity

Next-gen fabric interface chips in development




Hardware Subsystem: FabricFabric Architecture (single RSP shown)

Octopus

(LC)

Arbitration

Santa Cruz 1

Santa Cruz 0

Bellagio

Octopus(LC)

Octopus

(LC)

Octopus(RSP)

Octopus

(LC)

Octopus(LC)

Octopus

(LC)

Octopus(RSP)

Data

23G fabric channels

arbitration grant/request signals

?




RSP0

Hardware Subsystem: FabricFabric Architecture

Octopus(LC)

SantaCruz 0

Octopus(LC)

23G fabric channels

fabric data is active/active, arbiter is active/standby

SantaCruz 1

Bellagio

RSP1

SantaCruz 0

SantaCruz 1

Bellagio




RSP0

Hardware Subsystem: FabricScaling Fabric architecture for 80Gbps linecard

Octopus

SantaCruz 0

Octopus

Linecards are forwards/backwards compatible: fabric and backplane designed for 80G today

No changes to existing architecture, arbitration scheme stays the same

FCS fabric has enough switch fabric ports to support 80G/Linecard

Fabric and Backplane designed to support two Fabric Interface Devices seamlessly

SantaCruz 1

Bellagio

RSP1

SantaCruz 0

SantaCruz 1

Bellagio

Octopus

Dual-Octopus80G Linecard

Single-Octopus40G Linecard




(unicast fabric plane)

Hardware Subsystem: Fabric loadsharingUnicast loadsharing

Octopus

SantaCruz 0

Unicast traffic is sent across first available fabric link to destination (maximizes efficiency)

Each frame (or superframe) contains sequencing information

All destination octopus have re-sequencing logic

Additional re-sequencing latency is measured in nanoseconds

SantaCruz 1

Bellagio

(unicast fabric plane)

SantaCruz 0

SantaCruz 1

Bellagio

Octopus 4 3 2 1




(multicast fabric plane)

Hardware Subsystem: Fabric loadsharingMulticast loadsharing

Octopus

SantaCruz 0

Multicast traffic is hashed based on (S,G) info to maintain flow integrity

Very large set of multicast destinations preclude resequencing

Multicast traffic is non arbitrated – sent across a different fabric plane

Ratio of unicast credit to multicast credit is TBD

SantaCruz 1

Bellagio

(multicast fabric plane)

SantaCruz 0

SantaCruz 1

Bellagio

Octopus A1A2B1A3B2C1

Flows exit in-order




Hardware Subsystem: Fabric Active/Active Fabric Details

Each LC connects to and uses BOTH fabrics

One RSP provides enough bandwidth for 80G/slot

HW-based failure detection

very fast RSP failure notification (better MTTR)

Unicast traffic is loadbalanced across active links

Additional RSP provides more speedup, better hashing

No such thing as a single-RSP system (not sold) Always always always test, sell, and compete with both

RSPs installed and running!




Hardware Subsystem: FabricFabric Wiring & Scaling Details

10-slot chassis has 8 LC slots + 2 RSP slots

2x Santa Cruz fabric chips per RSP

Santa Cruz has 20 switch ports @ 20Gbps each

Each LC wired for two ports per Santa Cruz

40G linecards use one port for each Santa Cruz

80G linecards use two Santa Cruz ports each

6-slot chassis has several unwired ports




Hardware Subsystem: FabricFully Populated 40G/slot fabric subsystem: 10x octopus

Octopus(LC) Santa Cruz 0

Octopus(LC)

Santa Cruz 1

Santa Cruz 0

Santa Cruz 1

Octopus(LC)

Octopus(LC)

Octopus(LC)

Octopus(LC)

Octopus

(LC)

Octopus

(LC)

Octopus(RSP)

Octopus(RSP)

(bellagio & arbitration links are not shown)

RSP0

RSP1




Hardware Subsystem: FabricFully Populated 80G/slot fabric subsystem: 18x octopus

Octopus

Santa Cruz 0

Octopus

Santa Cruz 1

Santa Cruz 0

Santa Cruz 1

Octopus Octopus

Octopus

Octopus

Octopus(RSP)

Octopus(RSP)

Octopus

Octopus

Octopus

Octopus

Octopus

Octopus

Octopus

Octopus

Octopus

Octopus

80G

80G

80G

80G

80G

80G

80G

80G

40G40G

(bellagio & arbitration links are not shown)

RSP0

RSP1




Hardware Subsystem: Fabric Arbitration & Superframing Mechanism

20-port 20G crossbar switch w/ external arbitration

Scheduler granularity is per 10G (or 10x1) block

Ingress LC sends request for output LC

Arbiter replies with “credit” for egress LC Egress returns “credit” to LC when packet arrives

Arbitration not strictly required, but ...

Significantly improves switch efficiency

Intelligent “traffic cop”

Maintains state of available credit for each destination port

When no credit we don’t schedule for that port




Hardware Subsystem: Fabric Arbitration & Superframing Mechanism

Superframing significantly improves total throughput

multiple unicast frames from/to same destinations

0 (Empty)Min Sufficient

for Superframe

Max

Superframe

Max

MTU

Min reached (2)

Packet 1 Jumbo (1)

Packet 1 No superframing (1)

Packet 1 Max reached (3)Packet 2Packet 3

Packet 1Packet 2




Hardware Subsystem: Deep Fabric MathCounting the Bits…

each octopus acts and is wired to one “20G fabricchannels” on each santa cruz

raw bandwidth per “20G fabric channel” in the santa cruzfabric chip is:

8 parallel serial lines @ 3.125Mhz each, using 24/26 encoding.

8 * 3.125 * (24 / 26) 23.077 Gbps ~~> 23Gbps

there are two santa cruz chips per RSP:

two santa cruz per RSP: 23 * 2 == 46Gbps per RSP

there are two RSPs per chassis:

46Gbps/RSP * 2 92Gbps per octopus (in active/active mode)




Superframes

With 32 byte DC3 hdr + FPOE + error control + 8-bye alignment overhead, smaller pkts areinefficient in the fabric. So unicast packets are grouped into Superframes < 2K bytes. Since therecould be huge combination of multicast destinations, multicast packets are neither put intoSuperframes nor are arbitrated through the VOQ mechanism.

1st packet DC3 fabric

header inclusive

2nd packet DC3 fabric headerinclusive

3rd packet DC3 fabric headerinclusive

Nth packet

8 bytes

3 FPOE bytes

(interpreted

by fabric)

Pad with zero to align to 8 bytes

4 byte CRC covers

entire frame.

Min. 1068

bytes to

guarantee

line rate

perfmnce

Word0

Word1

Word2

Sequencing

information in

32-byte DC3

header




Load Balancing from Active/Active

Parallel Paths in Redundant, Eight Line Card Chassis

0

1

14

15

0

1

15

0

1

15

0

1

15

RSP0

0

1

15

0

1

15

0

1

15

0

1

15

S C Z

O

c t oF r om-

F a b

First Line Card / Slot 0

Egress Path

14

14

14

14

14

14

O c t oF r om-

F a b

O c t oF r om-

F a b

O c t oF r o

m-

F a b

O c t o

T o-F a b

O c t o

T o-F a b

O c t o

T o-F a b

O c t o

T o-F a b

First Line Card / Slot 0

Ingress Path

Last Line Card / Slot 9

Egress Path

Last Line Card / Slot 9

Ingress Path

RSP1

S C Z

S C Z

S C Z




ASR9K Linecards

ASR9K Overview

ASR9K Chassis

Architecture

RSP

Fabric

Linecard

Life of a Packet

Power Management



Service Console




ASR 9000 Series Line Cards

Integrated L2 / L3 Services Dense Hierarchical QoS

High Density Service Scale

Distributed High Performance Control Plane

Modular Operating System - IOS-XR

Distributed / Flexible Forwarding Plane

Integrated Timing Support

Integrated Media Monitoring

Distributed OA&M Processing

Distributed Netflow

Online Insertion & Removal

40G Line Rate Performance

Pluggable Optical / Electrical Interfaces

Feature support may be software release dependent

A9K-40GE-B 40 Port GE SFP Line Card

A9K-4T-B 4 Port 10GE XFP Line Card

A9K-4T/8-B 8 Port 10GE XFP Line Card




L/B/E Li C d




-

Network Process Unit

STATS MEMORY

FRAME MEMORYLOOKUPMEMORY

TCAM

FIB MAC

NP complex

L/B/E Line CardsWhat’s the Difference?

Each NPU has Four Main memories:

– Lookup/Search Memory (RLDRAM): stores MAC, FIB, and Adjacencies Tables

– TCAM: classification (Vlan Tag (EVCs), QoS and Security ACL

– Stats QDR memory: interface and forwarding statistics, policers data, etc

– Frame memory: buffer memory for Queues

3 LC versions – low, base and extended - differ for size of memories

– TCAM, QDR and Frame memory sizes depend on LC version

Affects number of QoS queues and L2 sub-interfaces supported

– Search Memory is same

System level scale (unicast, multicast, MPLS label) adjacency and MAC

address) not affected by a mix of LCs




Metric Low Queue Medium Queue High Queue

MAC Addresses 512K 512K 512K

IPv4 Routes (total/per VRF) 1M/128K 1M/128K 1M/128K

ARP entries 32K 32K 32K

IPv6 Routes (total/per VRF) 512K/128K 512K/128K 512K/128K

VRFs 4k 4k 4k

L3 Subif/Port 4k 4k 4k

Bridge Domains 8k 8k 8k

MPLS Labels 128k 128k 128k

EFPs (L2 sub-int) 4k 16k 32k

Queues 8/port 64k/32k 256k/128k

Policers 8k 128k 256k

ASR 9000 Ethernet LinecardsCapability Comparison

D i f f e r e n t

C o m

m o n




How to Choose the Right Linecard ?




Typhoon LC

Part NumberTarget

Release

A9K-24x10GE-SE 4.2.0

A9K-24x10GE-TR 4.2.0

A9K-2x100GE-SE 4.2.0

A9K-2x100GE-TR 4.2.0

A9K-1x100GE-SE 4.2.2

A9K-1x100GE-TR 4.2.2

A9K-36x10GE-SE 4.2.2

A9K-36x10GE-TR 4.2.2

Part Number

Target

Release

A9K-RSP440-SE 4.2.0

A9K-RSP440-TR 4.2.0

A9K-MOD80-SE 4.2.0

A9K-MOD80-TR 4.2.0

A9K-MOD160-SE 4.2.1

A9K-MOD160-TR 4.2.1

A9K-MPA-2x10GE 4.2.1









Netflow Support




Licensable Features

L3 VPN Service Line Card LicenseEnables Provisioning of L3VPN Service Interfaces on the LC

Infrastructure VRF Line Card License

Enables Provisioning of L3 Interfaces in up to 8 VRFs on the LC Media Monitoring System License

Enabled Provisioning of Media Monitoring Points on any LC

Feature support may be software release dependent




Do I need a License ?







Linecards

Codename I/O BW Engine Comments

Ryan 40 x GE 40G 4 x Trident NPU

Ramius 8 x 10GE 40G 4 x Trident NPU 2:1 oversubscribed

Mancuso 4 x 10GE 40G 4 x Trident NPU

Thor 4 x SPA 20G 2 x CPP10 (Popeye)

Son-of-Thor 4 x SPA 40G 2 x CPP10 (Olive)

Odin 8 x 10GE 80G 8 x Trident NPU

Valhalla 16x10GE 80G 8 x Trident NPU 2:1 oversubscribed

FCS switch fabric provides ~80Gbps per LC slot

Roadmap to 200 Gbps with future switching silicon

RE D

O C T OBE R

H d S b t R d O t b Li d




Hardware Subsystem: Red October LinecardLinecard Details

40Gbps linecard Scalable architecture

-4xTen, 40xGE, “2 plus 20”

-80G linecard in 2H CY09

Carrier Ethernet/Transport focus

Flexible, microcoded architecture

Base & expanded memory options

additional NP memory -> more scale

Advanced IP software licence

better L3 routing scale, adv. features

Hard are S bs stem Red October Linecard




Hardware Subsystem: Red October LinecardRed October Board Components

Physical interfaces:8xTenGE, 4xTenGE, 40xGE, “2+20”

Forwarding/Queueing

“Trident” Network Processor

Header translation and interconnect

“Bridge”

Fabric Interface

“Octopus”

CPU: 8641D w/ 2GB memory

Internal GE for chip programming

Hardware Subsystem: Red October LC



Cisco ConfidentialViking Hardware Overview 63via backplane

Hardware Subsystem: Red October LC8x10GE 2:1 oversubscribed Linecard : “Ramius”

10GE PHY

RSP 0

10GE PHY

10GE PHY

10GE PHY

10GE PHY

10GE PHY

10GE PHY

10GE PHY

Trident 0

Trident 1

Trident 2

Trident 3

BridgeFPGA 0

Trident Memory

BridgeFPGA 1

Trident Memory

Trident Memory

Trident Memory

Octopus

CPU8641D

2GB memory2GB flash

gigEcontrolnetwork

RSP 1

daughtercard

NetworkClocking

XFP3

XFP7

XFP2

XFP6

XFP

1

XFP5

XFP0

XFP4





Hardware Subsystem: Red October LC4x10GE linerate Linecard: “Mancuso”

RSP 0

Trident 0

Trident 1

Trident 2

Trident 3

BridgeFPGA 0

Trident Memory

BridgeFPGA 1

Trident Memory

Trident Memory

Trident Memory

Octopus

CPU8641D

2GB memory2GB flash

gigEcontrolnetwork

RSP 1

daughtercard

10GE PHY

10GE PHY

10GE PHY

10GE PHY

XFP

XFP

XFP

XFPNetworkClocking





Hardware Subsystem: Red October LC40xGE linerate Linecard: “Ryan”

RSP 0

Trident 0

Trident 1

Trident 2

Trident 3

BridgeFPGA 0

Trident Memory

BridgeFPGA 1

Trident Memory

Trident Memory

Trident Memory

Octopus

CPU8641D

2GB memory2GB flash

gigEcontrolnetwork

RSP 1

daughtercard

10xGE SFP

10xGE SFP

10xGE SFP

10xGE SFP

NetworkClocking

Hardware Subsystem




via backplane

Hardware Subsystem8x10GE line rate Linecard : “Odin”

10GE PHY

RSP 0

10GE PHY

10GE PHY

10GE PHY

Trident 0

Trident 1

RavenFPGA 0

Trident Memory

Trident Memory

Octopus 0

CPU8641D

CPU memory

2GB flash

gigEcontrolnetwork

RSP 1

NetworkClocking

XFP

XFP

XFP

XFP

Trident 2

Trident 3

Trident Memory

Trident Memory

10GE PHY

10GE PHY

10GE PHY

10GE PHY

Trident 4

Trident 5Raven

FPGA 1

Trident Memory

Trident Memory

XFP

XFP

XFP

XFP

Trident 6

Trident 7

Trident Memory

Trident Memory

Octopus 1

Hardware Subsystem




via backplane

Hardware Subsystem16x10GE 2:1 oversubscribed Linecard : “Valhalla”

10GE PHY

RSP 0

Trident 0

Trident 1

RavenFPGA 0

Trident Memory

Trident Memory

Octopus 0

CPU8641D

CPU memory

2GB flash

gigEcontrolnetwork

RSP 1

NetworkClocking

XFP

Trident 2

Trident 3

Trident Memory

Trident Memory

Trident 4

Trident 5Raven

FPGA 1

Trident Memory

Trident Memory

Trident 6

Trident 7

Trident Memory

Trident Memory

Octopus 1

10GE PHYXFP

10GE PHYXFP

10GE PHYXFP

10GE PHYXFP

10GE PHYXFP

10GE PHYXFP

10GE PHYXFP

10GE PHYXFP

10GE PHYXFP

10GE PHYXFP

10GE PHYXFP

10GE PHYXFP

10GE PHYXFP

10GE PHYXFP

10GE PHYXFP

Hardware Subsystem




Hardware Subsystem2x10GE + 20xGE linerate : “JetFire”

RSP 0

Trident 0

Trident 1

Trident 2

Trident 3

BridgeFPGA 0

Trident Memory

BridgeFPGA 1

Trident Memory

Trident Memory

Trident Memory

Octopus

CPU8641D

CPU memory2GB flash

gigEcontrolnetwork

RSP 1

daughtercard

10GE PHY

10GE PHY

XFP

XFP

NetworkClocking

10xGE SFP

10xGE SFP

Hardware Subsystem: Red October Linecard




Hardware Subsystem: Red October LinecardSynchronous Ethernet support on existing HW

RSP has BITS input, DTI andcentralized Clock Distribution hardware

Full support for L1 Sync-E on linecards(XR 3.9)

Flexible time sourcing: Line cardscapable of recovering clock andsending to RSP and receiving Transmitclock from RSP

Frequency synchronization support viaSPAs also (CEoPS, Metronome ..)

Future Hardware capable ofIEEE1588-v2

via backplane

10GE PHY

RSP 0

10GE PHY

10GE PHY

10GE PHY

10GE PHY

10GE PHY

10GE PHY

10GE PHY

Trident 0

Trident 1

Trident 2

Trident 3

BridgeFPGA 0

Trident Memory

BridgeFPGA 1

Trident Memory

Trident Memory

Trident Memory

Octopus

CPU8641D

2GB memory2GBflash

gigEcontrolnetwork

RSP 1

daughtercard

NetworkClocking

XFP

XFP

XFP

XFP

XFP

XFP

XFP

XFP

Hardware Subsystem: Trident Fwd Engine




Hardware Subsystem: Trident Fwd EngineTrident: 10Gbps forwarding/feature engine

10Gbps feature rich ethernet forwarding ASIC

Utilizes multiple cores for packet processing

(Similar to CPP/QuantumFlow Processor)

Multi-stage microcoded architecture

Integrated Traffic managers, Ethernet MACs

Multiple external memories, amount determines scale

Classification TCAMs, MAC/IP lookups,

Frame Buffers, Counters/stats




Trident NPU Block Diagram

Micro-EnginePipeline

1 5 G b i t / sX A UI +

G

i gE S GMI I MA C s ( or )

X A UI 1 0 G MA C s

Trident NPU Memory Interfaces

Ethernet Side Fabric Side

Host PCI-Express

HostSGMII

(“punt path”)

TMb

TMc

1 0 x G

i gE - or -2 x1 0 Gi gE

E t h

er n e t I n t er f a c e s

TMa

FrameMemory

TM ControlMemory

LookupTables

CountersStatisticsLookups

OptionalTCAM72

DRAM333MHz

DDR2/RLDRAM2

DRAM333MHz

DDR2/RLDRAM2

DRAM333MHz

DDR2/RLDRAM2

SRAM(36x1)

300MHzQDR

(72x1)300MHz

F a b r i cI n t er f a c e

S i x-l a

n eX A UI / 1 5 G b i t / s e c




Trident NPU Architecture Highlights

TM

FrameMemory

FromLink/SF/

Host

External Search Memory( RLDII, TCAM)

ControlMemory

Embedded searchMemory

keys searchresults

searchresults

ToLink/SF/

Host

TOP

parseTOP

Search ITOP

modifyTOP

Search IITOP

resolve

StatisticsMemory

TM TM

High-learn I/f

Loopback interface

256KB

256KB

FrameMemory

Trident NPU Single Chip, Programmable, up to 12Gbps w/ 64B packets, full duplex, wire speed networkprocessor (up to 12 Gbps ingress plus 12 Gbps egress per chip)

Traffic Manager Support WRED, total 32K queues (8 queues per subscriber x 4K subscribers) per TM

Up to 5 levels of traffic shaping hierarchy

ProgrammingModel

Pipeline programming model w/o parallel programming or multi-threading. Parallelization doneby HW

ProgrammingLanguage

CISC-like assembly instruction set with powerful bit manipulation instructions.

Macro and Assembly programming.

HW Assistance QoS (Traffic Manager)

Classification / Searching

Counter Updates for statistics maintenance

Hardware: Trident Network Processor




Multicore/Multistage processing complex

10Gbps+ full duplex forwarding/feature engine

External Memory Interfaces

EgressBQS

engine

64k

queues

IngressBQS

engine

32kqueues

PPE

PPE

PPE

PPE

PPE

PPE

PPE

PPE

PPE

PPE

PPE

PPE

PPE

PPE

PPE

PPE

PPE

PPE

PPE

PPE

High Speed on-chip memory

InterfaceMAC/PHY

switchfabric

interface

BQS : “buffering / queueing / scheduling”

fabric

RSP

RSP

Hardware Subsystem: Trident Fwd Engine




Hardware Subsystem: Trident Fwd EngineTrident: TCAM partitioning

256k entry TCAM on each NP, divided into tables:

96k entries @ 144b key size (QoS, ACL, iFIB)

16k entries @ 576b key size (QoS, ACL, iFIB)

96k for EFP classification

specific application determines which key size used

applications share space WITHIN a key-size table

i.e. no hard-partitioning between QoS/ACL/iFIBetc.

ingress & egress features share same TCAM space




show controller np all summary

RP/0/RSP1/CPU0:ASR9010-A#sh controller np summ all

Mon Apr 6 10:31:21.480 UTC

Node: 0/0/CPU0:

----------------------------------------------------------------

[total 4 NP] Driver - Version 10.26a Build 1010 ( Jan 28 2009, 12:00:00 )

NP 0 : Hardware Revision: v2 A1

: Ucode - Version: 255.255 Build Date: ( Mar 19 2009, 21:56:00 )



NP 2 : Hardware Revision: v2 A1: Ucode - Version: 255.255 Build Date: ( Mar 19 2009, 21:56:00 )



Node: 0/5/CPU0:

----------------------------------------------------------------

[total 4 NP] Driver - Version 10.26a Build 1010 ( Jan 28 2009, 12:00:00 )












show controller np counter np0 loc <>

RP/0/RSP1/CPU0:ASR9010-A#sh controller np counters np0 loc 0/0/cpu0

Mon Apr 6 10:34:11.161 UTC

Node: 0/0/CPU0:

----------------------------------------------------------------

Show global stats counters for NP0, revision v3

Read 7 non-zero NP counters:

Offset Counter FrameValue-------------------------------------------------------------------------------

25 PARSE_FABRIC_RECEIVE_CNT 19601

53 XAUI_TRAINING_PKT_DISCARD 1093

57 RESOLVE_INGRESS_L2_PUNT_CNT 29294

85 DIAGS 9786

197 PUNT_STATISTICS 447536

199 PUNT_DIAGS_RSP_ACT 9789

201 PUNT_DIAGS_RSP_STBY 9719

For detail explanation on what these counters mean, please look at: asr9k_ucode-cntrs.doc ( EDCS-778662)




show controller np ports all loc <>

RP/0/RSP1/CPU0:ASR9010-A#sh controller np ports all loc 0/0/cpu0

Mon Apr 6 10:44:41.289 UTC

Node: 0/0/CPU0:

----------------------------------------------------------------

NP Bridge Fia Ports

-- ------ --- ---------------------------------------------------

0 1 0 TenGigE0/0/0/3

1 1 0 TenGigE0/0/0/2

2 0 0 TenGigE0/0/0/1

3 0 0 TenGigE0/0/0/0




sh controller fabric fia link-status loc <>

RP/0/RSP1/CPU0:ASR9010-A#sh controller fabric fia link-status loc 0/0/cpu0

Mon Apr 6 09:40:18.677 UTC

Fia to Arbiter sync status

ARB 0 SYNCED

ARB 1 SYNCED

Fia to Crossbar sync status

Fabric Link 0 SYNCED







sh controller fabric fia stats loc <>RP/0/RSP1/CPU0:ASR9010-A#sh controller fabric fia stats loc 0/0/cpu0

FIA DDR Packet counters:

==========================

From Bridge#[0] 228791To Bridge #[0] 91984

From Bridge#[1] 955781

To Bridge #[1] 942233

FIA SuperFrame counters:

==========================

To Unicast Xbar[0] 39221




To MultiCast Xbar[0] 0




From Unicast Xbar[0] 39225




From MultiCast Xbar[0] 0

From MultiCast Xbar[1] 86872From MultiCast Xbar[2] 0

From MultiCast Xbar[3] 4456

FIA Total Drop counters:

====================

Ingress drop: 0

Egress drop: 0

Total drop: 0




sh controllers fabric fia bridge stats loc <>UC - Unicast , MC - Multicast

LP - LowPriority , HP - HighPriority

Cast/ Packet Packet Error ThresholdPrio Direction Count Drops Drops

--------------------------------------------------------------------------------

Unicast Egress Stats

********************

UC HP Fabric to NP-0 0 0 0

UC LP Fabric to NP-0 25497 0 0







----------------------------------------------------------------

UC Total Egress 1150508 0 0

Multicast Egress Stats

*********************

MC HP Fabric to NP-0 25424 0 0

MC LP Fabric to NP-0 0 0 0

MC HP Fabric to NP-1 25424 0 0MC LP Fabric to NP-1 0 0 0





---------------------------------------------------------------

MC Total Egress 101696 0 0

Cast/ Packet Packet

Prio Direction Count

--------------------------------------------------

sh controllers fabric fia bridge ddr-status




glocation <>

RP/0/RSP1/CPU0:ASR9010-A#sh controllers fabric fia bridge ddr-status loc 0/rsp0/cpu0

Fri Apr 17 08:37:38.032 UTC

Fia DDR# Status

-------- ------

0 SYNCED

1 SYNCED

RP/0/RSP1/CPU0:ASR9010-A#sh controllers fabric fia bridge ddr-status loc 0/0/cpu0

Fri Apr 17 08:38:45.050 UTC

Fia DDR# Status

-------- ------

0 SYNCED

1 SYNCED




sh controllers fabric fia bridge sync-status loc <>RP/0/RSP1/CPU0:ASR9010-A#sh controllers fabric fia bridge sync-status loc 0/0/cpu0

Fri Apr 17 08:33:15.429 UTC

Bridge# NP# Status

------- --- ------

0 0 SYNCED

0 1 SYNCED

1 2 SYNCED

1 3 SYNCED

RP/0/RSP1/CPU0:ASR9010-A#sh controllers fabric fia bridge sync-status loc 0/rsp0/cpu0

Fri Apr 17 08:42:49.929 UTC

Invalid Node. NP link status can be checked only on LCs

sh controllers fabric fia bridge flow-control loc




g0/0/cpu0

RP/0/RSP1/CPU0:ASR9010-A#sh controllers fabric fia bridge flow-control loc 0/0/cpu0

Fri Apr 17 08:41:53.502 UTC

UC - Unicast , MC - Multicast

LP - LowPriority , HP - HighPriority

Cast/ BackPressure Backpressure

Prio Direction Count

---------------------------------------------------

Ingress BackPressure Counters

******************************

FIA to bridge-0 0

FIA to bridge-1 0

All Bridge to NP-0 0




Egress BackPressure Counters

****************************

Bridge-0 to FIA 0

Bridge-1 to FIA 0

UC LP NP-0 to Bridge0 0UC LP NP-1 to Bridge0 0

UC LP NP-2 to Bridge1 0

UC LP NP-3 to Bridge1 0

UC HP NP-0 to Bridge0 0




MC LP NP-0 to Bridge0 0






Extended (-E) Line Cards

Address the highest service scale requirements Address the highest buffering capacity requirements

Feature / Performance are virtually identical –B Line Cards

A9K-40GE-E 40 Port GE SFP Extended Line Card

A9K-4T-E 4 Port 10GE XCP Extended Line Card

A9K-4T/8-E 8 Port 10GE XCP Extended Line Card




Major Differences from –B Line Cards

Relative to -B, the Extended (-E) Line Cards provide~2x increase in Interface Density

~2x increase in Bridge Domain Density

~4x increase in Queue Density

~3x Packet Buffer Capacity

Relative to –B, the Extended Line Cards haveadditional High-Speed Memories / TCAM Resources

Otherwise the –E Line Cards are virtually identical to -B




Examples of Non-Differences From –B LCs

Layer 3 Route Scale Layer 2 Route Scale

Support of BGP

Support of L3VPNs

Simultaneous Support of L2 / L3

Support of Dense Hierarchical QoS

Forwarding Performance

Li C d K M i




Line Card Key Metrics

Metric -B Line Cards -E Line CardsMAC Addresses 1M 1M

IPv4 Routes 1M 1M

IPv6 Routes 256k 256k

VRFs 4k 4kEFPs 16k 32k

Bridge Domains 8k 8k

Queues 96k 384k

Policers 128k 256kDHCP Clients 128k 128k

Packet Buffer 50ms 150ms

Scale support may be software release dependent

Hardware Subsystem: SIP 20G (Thor)




Next Gen SPA interface Processor

(SIP) for Viking : Dynamic Subscriber Management

Flexibility to include additional applicationslike Mobile Backhaul

20G SIP based on Quantum FlowProcessor (QFP)

Two SPA variants (at FCS) : 10 x 1GE,1 x 10GE

Target future support for ChOC / ATM /TDM SPAs ..

8641D Dual Core CPU and memory

Hardware Subsystem: SIP 20G (Thor)

Hardware Subsystem: Thor




QFP 0

Fabric

InterfaceSerpent

FPGAHyTop

FPGA

SPA

I/F

DRAM

Drakar

FPGA

CPU

SPA

I/F

HH SPA

PEX8509

HH SPA

HH SPA

HH SPA

yBlock Diagram w/ 4x Half-height SPAs

Ingress QFP complex

CPU daughtercard

SPA daughtercard

QFP Memory

Egress QFP complex

QFP 1

QFP Memory

Hardware Subsystem: QFP




• 40 Packet processors – 4 contexts (threads) each

1.2GHz processors with associated caches + DRAM packet memory

Single TCAM4 I/F (can cascade 1-4 devices)

C-language (supports stack)

HW assist for locks, PLU, hash LU, stats, WRED, policers, range lookup,

crypto, CRC

• Buffer/queue subsystem (CPP-BQS)

supports up to 128k HW queues per direction

256MB max packet buffer

• HW hierarchical scheduler

Fully configurable # of layers based on HQF

yQuantumFlow Processor (formerly CPP)

Viki C d




Viking Codenames

Ryan – 40x1GE Linerate LC for Red October/Samurai Ramius – 8x10GE Over-subscribed LC for Red October/Samurai

Mancuso - 4 x10GE Linerate LC for Red October/Samurai

Jetfire – 2x10GE + 20 GE Linerate LC for Red October/Samurai

Odin – 8x10GE Linerate LC for Red October/Samurai

Valhalla – 16x10GE Over-subscribed LC for Red October/Samurai

Maximus – RSP for Red October/Samurai

Red October – 10 Slot Chassis

Samurai – 6 Slot Chassis




ASR9K Life of aPacket

ASR9K Overview

ASR9K Chassis

Architecture

RSP

Fabric

Linecard

Life of a Packet

Power Management


Platform Fault Management (PFM)

Online Diagnostics

Service Console

P k t P th S h t




Packet Path Snapshot

SFP/XFP

OpticsNP3c

Bridge

FPGAOctopus

B a c k pl an e

Santa

Cruz

Santa

Cruz

SFP/XFP

OpticsNP3c

Bridge

FPGAOctopus

Moto 8641D

OctopusPunt

FPGA

Moto 8641D

Normal Path

Punt Path

32G DDR

per bridge

15G XAUI

per NP3

10G MAC

per PHY

2 x 23G

Chico

per LC

Punted to RSP if

FPOE says so

Life of a Packet




via backplane

Ingress Packet Path

RSP 0

Trident 0

Trident 1

Trident 2

Trident 3

BridgeFPGA 0

Trident Memory

BridgeFPGA 1

Trident Memory

Trident Memory

Trident Memory

Octopus

CPU8641D

2GB memory2GB flash

gigEcontrolnetwork

RSP 1

daughtercard

1

4

3

2

NetworkClocking

10GE PHY

10GE PHY

10GE PHY

10GE PHY

XFP

XFP

XFP

XFP

B id FPGA




Bridge FPGA

The Bridge FPGA is acting as a traffic adaptorbetween DDR interface of Octopus ASIC andXAUI-like interface of G8000 Networkprocessor

There are two data paths from the NPBridge3 FPGA:

Manipulates the header and does interface conversion, to theOctopus

This path handles all main data and also central punt data thatare routed

Local CPU through a switched GE link

used to process punt data locally or the packets may be sent tothe RSP card via the backplane GE or fabric link

Internal Packet Headers




Internal Packet Headers

Life of a Packet




RSP0

Fabric Packet Path

Octopus(LC)

Santa

Cruz 0

Octopus (LC)

SantaCruz 1

Bellagio

RSP1

SantaCruz 0

SantaCruz 1

Bellagio

1: Fabric Request

3: Fabric Grant

2: Arbitration

4: load-balanced

transmission across fabric links

5: credit return

Life of a PacketE P k P h




via backplane

Egress Packet Path

RSP 0

Trident 0

Trident 1

Trident 2

Trident 3

BridgeFPGA 0

Trident Memory

BridgeFPGA 1

Trident Memory

Trident Memory

Trident Memory

Octopus

CPU8641D

2GB memory2GB flash

gigEcontrolnetwork

RSP 1

daughtercard

4

1

2

35

10GE PHY

10GE PHY

10GE PHY

10GE PHY

XFP

XFP

XFP

XFP

NetworkClocking

Linecard backplane GE links




Linecard backplane GE links

The backplane GE links, one to each RSP card, areused primarily for control plane functions such asapplication image download, system configuration datafrom IOS-XR, statistics gathering and line card power-up and reset control.

sh controller backplane ethernet links loc <>




sh controller backplane ethernet links loc <>

RP/0/RSP1/CPU0:ASR9010-A#sh controller backplane ethernet links loc 0/rsp0/cpu0

Fri Apr 17 12:43:02.946 UTC

Node | Link State

----------------------------------

0/RSP0/CPU0 | Up (Local CPU)

0/RSP1/CPU0 | Up (Peer CPU)

0/0/CPU0 | Up

0/1/CPU0 | Down

0/2/CPU0 | Down

0/3/CPU0 | Down

0/4/CPU0 | Down

0/5/CPU0 | Up

0/6/CPU0 | Down

0/7/CPU0 | Down

RP/0/RSP1/CPU0:ASR9010-A#sh controller backplane ethernet links loc 0/rsp1/cpu0 Fri Apr 17 12:43:41.535 UTC

Node | Link State

----------------------------------

0/RSP1/CPU0 | Up (Local CPU)

0/RSP0/CPU0 | Up (Peer CPU)

0/0/CPU0 | Up

0/1/CPU0 | Down

0/2/CPU0 | Down

0/3/CPU0 | Down

0/4/CPU0 | Down

0/5/CPU0 | Up

0/6/CPU0 | Down

0/7/CPU0 | Down

Life of a PacketE M lti t P k t P th




via backplane

Egress Multicast Packet Path

RSP 0

Trident 0

Trident 1

Trident 2

Trident 3

BridgeFPGA 0

Trident Memory

BridgeFPGA 1

Trident Memory

Trident Memory

Trident Memory

Octopus

CPU8641D

2GB memory

gigEcontrolnetwork

RSP 1Non-subscriber

Non-subscriber

Non-subscriber

Non-subscriber

Non-subscriber

Non-subscriber

Non-subscriber

Non-subscriber

Non-subscriber

Non-subscriber

Non-subscriber

Non-subscriber

Non-subscriber

Non-subscriber

Non-subscriber

Non-subscriber

mcast





via backplane


RSP 0

Trident 0

Trident 1

Trident 2

Trident 3

BridgeFPGA 0

Trident Memory

BridgeFPGA 1

Trident Memory

Trident Memory

Trident Memory

Octopus

CPU8641D

2GB memory

RSP 1Non-subscriber

Non-subscriber

Non-subscriber

Non-subscriber

Non-subscriber

Non-subscriber

Non-subscriber

Non-subscriber

Non-subscriber

Non-subscriber

Non-subscriber

Non-subscriber

Non-subscriber

Non-subscriber

Subscriber

Non-subscriber

mcast

NewSub





Subscriber

via backplane


RSP 0

Trident 0

Trident 1

Trident 2

Trident 3

BridgeFPGA 0

Trident Memory

BridgeFPGA 1

Trident Memory

Trident Memory

Trident Memory

Octopus

RSP 1Non-subscriber

Non-subscriber

Non-subscriber

Non-subscriber

Non-subscriber

Non-subscriber

Non-subscriber

Non-subscriber

Non-subscriber

Non-subscriber

Non-subscriber

Subscriber

Non-subscriber

mcast

Subscriber

Subscriber

Add replication only as new subscribersare added, prune when tree is empty...





via backplane


RSP 0

Trident 0

Trident 1

Trident 2

Trident 3

BridgeFPGA 0

Trident Memory

BridgeFPGA 1

Trident Memory

Trident Memory

Trident Memory

Octopus

RSP 1

Non-subscriber

Non-subscriber

Subscriber

Subscriber

Subscriber

Subscriber

Non-subscriber

Non-subscriber

Non-subscriber

Subscriber

Non-subscriber

mcast

Subscriber

Subscriber

SubscriberSubscriber

Replicate intelligently and using need-based tree

saves bandwidth and processing cycles, while allowing for per-replicaservices delivery

Non-subscriber

(Pruned)(Pruned)

(leave)

Viking MulticastViking Multicast Forwarding Plane




LC2

Switch Fabric

LC1LC3

T0 T1 T3

Fabric Interface

B1B0

IGMP joinsMulticastSource

[1] IngressTrident RPF

check.

FGID : Fabric Group ID MGID : Multicast Group ID

[2] Switch Fabric Replicatesthe packet to the appropriate

Fabric Interface chips on theLine Cards using the FGID.

(Ex: 1110). Does not gothrough VOQ

[5] T Replicates the packet tothe local Interfaces using the

OIL

T1 T2T0 T3

R

R

RP

1 1 1

1 0

Fabric Interface Fabric Interface

B1B0 B1B0

[4] Bridge Replicates thepacket to the appropriate Tsusing the MGID. Has 32K

MGID entries w/ 2 bits for Tidentification

R

R

[3] Fabric Interface Replicatesthe packet to the appropriate

Bridges using the MGID.150KB Buffer for Multicast and

32K MGID entries.

[0] IGMP joins for

subscriberregistration.

Replication EnginesR

T0 T1 T2 T3T2

Viking Multicast Forwarding Plane

ASR9K Overview




ASR9K PowerManagement

ASR9K Overview

ASR9K Chassis

Architecture

RSP

Fabric

Linecard

Life of a Packet

Power Management



Online Diagnostics

Service Console

Power Management




Power Management

Power Management is a customer requirement that the featureavailable start from FCI (3.7.2) release.

Power Management enabled ASR9K chassis have the capability toprevent an FRU from being powered on, in the event that there isnot enough system power available to accommodate the FRU.

Power Management uses Power Monitor software feature whichallows user to monitor the power used and how much is available.

Power Management feature also alarms the user if there is notsufficient power or any other related events.

Power management feature is not available in 3.7.1 and onGladiator chassis.

System Overview




System Overview

PwrMon processIdentifies and communicates with power supply modules via I2C

Shelfmgr process

Allows or Denies LC power-up based on (1) estimated powerrequirements & (2) available power modules

Publishes Power Supply OIR events to internal consumers.

EnvMon process

Publishes power budget status to CLI

InvMgr processPublishes detailed power supply description to inventory

Feature Description




Feature Description

Inventory Description for Power Supplies

Indicate how many power module in system

Indicate power module source is AC or DC

RP/0/RSP0/CPU0:Samurai(admin)#show inventory… …

NAME: "power-module 0/PM0/SP", DESCR: "1.5kW DC Power Module"PID: A9K-1.5KW-DC , VID: V00, SN: TLD123950YS NAME: "power-module 0/PM1/SP", DESCR: "2kW DC Power Module"PID: A9K-2KW-DC , VID: V01, SN: TLD1214001S

==================

RP/0/RSP0/CPU0:RO(admin)#show inventory

… … NAME: "power-module 0/PM0/SP", DESCR: "3kW AC Power Module"PID: A9K-3KW-AC , VID: V00, SN: TLD123850NH NAME: "power-module 0/PM2/SP", DESCR: "3kW AC Power Module"PID: A9K-3KW-AC , VID: VOO, SN: TLD123650RA

Feature Description




Feature Description

Power Management Command Summary

Power Management Command Summary

Command Purpose

RP/0/RP0/CPU0:Router(admin)# show platform Displays information and status for eachnode in the system includes power supply

RP/0/RP0/CPU0:Router(admin)# show inventory To retrieve and display information aboutall the Cisco products that are installed inthe router , includes power supply

RP/0/RP0/CPU0:Router(admin)# show environmentpower-supply

Displays power supply capacity andcurrent information

RP/0/RP0/CPU0:Router (admin-config)#hw-modulepower [disable] location { location}

To power on a specified line card or

disable the node power-on feature

RP/0/RP0/CPU0:Router (admin-config)#hw-modulepower override location { location}

To allows a card to be forced to power up,regardless of an unprogrammed EEPROMvalue

Feature Description




Feature DescriptionChecking available power capacity

RP/0/RSP0/CPU0:VKG-3(admin)#show environment power-supply Mon Oct 12 12:25:05.399 UTC

R/S/I Modules Sensor Watts Status

0/PM0/*host PM 3000 Ok



Power Shelves Type: AC

Total Power Capacity: 9000WUsable Power Capacity: 9000WSupply Failure Protected Capacity: 6000WFeed Failure Protected Capacity: 3000W Worst Case Power Used: 2615W

Slot Max Watts---- ---------0/0/CPU0 375

0/1/CPU0 3750/RSP0/CPU0 2500/RSP1/CPU0 2500/7/CPU0 3750/FT0/SP 495 (default)0/FT1/SP 495 (default)

Worst Case Power Available: 6385WSupply Protected Capacity Available: 3385WFeed Protected Capacity Available: 385W

ASR9K Overview




Media and FileSystem

ASR9K Overview

ASR9K Chassis

ArchitectureRSP

Fabric

Linecard

Life of a Packet

Power Management



Online Diagnostics

Service Console

Devices On RSP




Devices On RSP

Device Type Quantity Size FS Type

Harddisk SAS 1 70GB QNX4

eUSB NAND 2 2GB/ea QNX4

C. Flash

(removable)

Compact

Flash

1 1GB DOS/QNX4

Flash NOR 2 64MB QNX FFSv3

NVRAM 1 512KB Cisco

Devices On LC




Devices On LC

Device Type Quantity Size FS Type

eUSB NAND 1 2GB QNX4

Flash NOR 1 128MB QNX FFSv3

Partitions On RSP




Partitions On RSP

Device Partition Size Purpose

Harddisk /harddisk:

/harddiska:

/harddiskb:

Raw

70GB(total)

dumps/ etc.

Primary for kernel dumps

eUSB /disk0:

/disk0a:

/disk1:

/disk1a:

1.6GB

0.4GB

1.6GB

0.4GB

IOS-XR Packages

Golden Disk or Mirror Disk

C. Flash

(removable)

/compactflash: 1GB General Purpose Storage

Partitions On RSP (cont’d)




Partitions On RSP (cont d)


Flash 1 Raw

/bootflash:

Raw

43MB

MONLIB

MBI images

ROMMON/Firmware

Flash 2 Raw

/configflash:

Raw

28MB

MONLIB

OBFL data

Secondary for kernel dumps

NVRAM /nvram:

/nvram-raw:

Variables, logs

Reboot history; crashinfo,PCDS

Partitions On LC




Partitions On LC


eUSB /eusb0:

/eusb0a:

1.6GB

0.4GB

Local Kernel Dumps

Flash Raw/bootflash:

Raw

56MBMONLIB

ROMMON/Firmware/Kerneldumps

DOS-QNX4 File System Utility Comparison




DOS QNX4 File System Utility Comparison

DOS QNX4

Types 6, 4, 11, 12 77, 78, 79, 80

FileOrganization

Block-based Extent-based

Patition fdisk fdisk

Initialize mkdosfs dinit

fs check chkdosfs chkfsys

Verify media dcheck dcheck

Standard QNX utilities. Refer to QSSL web site.

CLI: show filesystem




CLI: show filesystem

RP/0/RSP0/CPU0:ios#show filesystem

File Systems:

Size(b) Free(b) Type Flags Prefixes

- - network rw qsm/dev/fs/tftp:

- - network rw qsm/dev/fs/rcp:

- - network rw qsm/dev/fs/ftp:

…

1644150784 1404023296 flash-disk rw disk1:

1644150784 1404023296 flash-disk rw disk0:

35645292544 35628865024 harddisk rw harddisk:

1022427136 1022418944 flash-disk rw compactflash:

8075067392 8073070080 harddisk rw harddiska:

7840202752 7838258688 harddisk rw harddiskb:

411041792 410914816 flash-disk rw disk1a:

411041792 410903552 flash-disk rw disk0a:

224256 165888 nvram rw nvram:

290816 290816 nvram rw nvram-raw:

44695552 23889200 flash rw bootflash:

28966912 28862044 rw configflash:

show media




show media

RP/0/RSP0/CPU0:VKG-3(admin)#sh mediaFri Apr 9 15:05:34.364 EST

Media Information for 0/RSP0/CPU0.Image Current Part

Mountpoint FsType FsType Size State DrvrPid Mirror Flags================================================================================/disk0: QNX4 QNX4 1.5G Mounted 0069666 Enabled/disk0a: QNX4 QNX4 0.4G Mounted 0069666/disk1: QNX4 QNX4 1.5G Mounted 0069676/disk1a: QNX4 QNX4 0.4G Mounted 0069676/compactflash: FAT16 FAT16 1.0G Mounted 0081977

/harddiska: QNX4 QNX4 7.5G Mounted 0049184/harddiskb: QNX4 QNX4 7.3G Mounted 0049184/harddisk: QNX4 QNX4 33.2G Mounted 0049184/lcdisk0: QNX4 (?) Not Present/lcdisk0a: QNX4 (?) Not Present

RP/0/RSP0/CPU0:VKG-3#sh media loc 0/0/cpu0Fri Apr 9 15:13:51.680 EST

Media Information for 0/0/CPU0.Image Current Part

Mountpoint FsType FsType Size State DrvrPid Mirror Flags================================================================================/lcdisk0: QNX4 QNX4 1.5G Mounted 0049185/lcdisk0a: QNX4 QNX4 0.4G Mounted 0049185

ASR9K Overview




Online Diagnostics

ASR9K Overview

ASR9K Chassis

ArchitectureRSP

Fabric

Linecard

Life of a Packet

Power Management



Online Diagnostics

Service Console

What is online diagnostics




g

It defines a suite of tests for each card in system, although sometests test inter-card links

Tests run in background, faults reported to Platform Fault Manager

Tests can be enabled and disabled individually, test frequency canbe changed through configuration commands

Components Covered by FCI:

- Control path and data path links

- access to scratch registers for major ASICs and FPGAs/CPLDs

- integrity of FPD images

EOBC paths




p

EOBC heartbeat tests




Bi-directional;

Runs on all Line cards and Standby RSP

every 5 seconds, by default

Interval can be changed by changing configuration

User Interface




Generic user interface : GOLD-XR, same as on CRS Only admin users have privilege to access online diagnostic commands:

commands are in admin mode

Most tests running in the background by default

Show commands to see what tests are available, running frequency ofeach test, etc.:

“show diagnostic content location <node>”

Show commands to see tests result:“show diagnostic result location <node> [test <test-id> detail ]”

Change test running frequency or enable/disable a test:use admin configuration commands:“diagnostic monitor interval location <node> test <test-id> <day>

<hh:mm:ss.mille-sec>” “diagnostic monitor location <node> test <test-id> disable”

Sample screen shots




Sample screen shots

RP/0/RSP0/CPU0:ios(admin)#show diagnostic result location 0/rsp0/cpu0

Current bootup diagnostic level for RP 0/RSP0/CPU0: minimal

RP 0/RSP0/CPU0:

Overall diagnostic result: PASS

Diagnostic level at card bootup: minimal

Test results: (. = Pass, F = Fail, U = Untested)

1 ) PuntFPGAScratchRegister ---------> .

2 ) XbarInterfaceScratchRegister ----> .

3 ) ClkCtrlScratchRegister ----------> .

4 ) IntCtrlScratchRegister ----------> .

5 ) CPUCtrlScratchRegister ----------> .

6 ) XbarSwitchIdRegister ------------> .

7 ) EccSbeTest ----------------------> U

8 ) SrspStandbyEobcHeartbeat --------> U

9 ) SrspActiveEobcHeartbeat ---------> U

10 ) FabricLoopback ------------------> .

11 ) PuntFabricDataPath --------------> U

12 ) BootflashVerify -----------------> U

Referencias




Referencias

ASR9000/XR Understanding andtroubleshooting Fabric issues in the A9K https://supportforums.cisco.com/document/12135016/asr9000xr-understanding-and-troubleshooting-fabric-issues-a9k

ASR9000/XR: Troubleshooting packetdrops and understanding NP dropcounters

https://supportforums.cisco.com/docs/DOC-15552

https://supportforums.cisco.com/document/12135016/asr9000xr-understanding-and-troubleshooting-fabric-issues-a9k

































Documents

1 Asr9k Arc Base