National LambdaRail Layer 2 and 3 Networks 3 May 2005 Internet2 Spring Members Meeting Layers 2 & 3 Users BOF

National LambdaRail

Layer 2 and 3 Networks

3 May 2005

Internet2 Spring Members Meeting

Layers 2 & 3 Users BOF

Joint Techs - Feb. 13, 2005 2

NLR L2 Services SummaryGoals

Provide circuit-like options for users who can’t use, or don’t need, a 10G Layer1 wave.

Experiment on future national L2 network capabilities.– Create distributed nationwide broadcast

domain.– Create tools and procedures for automated

and user-controlled provisioning of L2 resources.– Create network management tools for time-of-

day based resource reservation.


NLR Layer 2 locations(Sites in blue are already connected)

Jacksonville: Level3, 814 Phillips HwyAtlanta: Level3, 345 Courtyard, Suite 9Raleigh: Level3, 5301 Departure DriveWashDC: Level3, 1755/1757 Old Meadow Road Suite: 111, McLean VA.NYC: MANLAN, 32 Avenue of the Americas, 24th FloorPittsburgh: Level3, 143 South 25thCleveland: Level3, 4000 Chester Avenue.Chicago: Level3, 111 N. Canal, Suite 200Kansas City: Level3, 1100 Walnut Street, MO.Denver: Level3, 1850 Pearl St, Suite 4Seattle: PacWave, 1000 Denny Way (Westin)Sunnyvale: Level3, 1360 Kifer Road Suite: 251Los Angeles: Equinix, 818 W. 7th Street, 6th Floor El Paso: Wiltel, 501 W. OverlandHouston: Wiltel, 1124 Hardy St.Tulsa: Wiltel, 18 W. ArcherBaton Rouge: Wiltel, No Address Yet Albuquerque: Level3, 104 Gold St.


HOU

TUL

ELP

KAN

Layer 2 Initial Logical Topology

BAT

LAXALB

PIT

WDC

CLE

ATL

RAL

CHINYC

DENSVL

SEA

JAC

10GE wave

Cisco 6509 switch


NLR L2 Hardware

Cisco Catalyst 6509-NEBs-A

Chassis Height: 36.65 in. 93.09cm

Chassis Width: 17.20 in. 43.68cm

Chassis Depth: 20.32 in. 51.61cm

Weight:

• 270 lb fully loaded

DC Power: 2 -2500 WDC power suppliesWe will need 1 60A feed for each


Generic NLR L1, L2 and L3 PoP Layout

CRS-1

Colo

EastWest

NLR demarc

15808

6509

15454

DWDM

1G wave, link or port10G wave, link or port


NLR L2 Services SummaryInitial Services

Dedicated Point to Point Ethernet – VLAN between 2 members with dedicated bandwidth from sub 1G to multiple 1G.

Best Effort Point to Multipoint – Multipoint VLAN with no dedicated bandwidth.

National Peering Fabric – Create a national distributed exchange point, with a single broadcast domain for all members. Initial user ports will be 1GE.


NLR L2 Services SummaryExpected Near Term Services

Dedicated Point to Multipoint : Dedicated bandwidth for multipoint connections

Scavenger: Support less-than-best-effort forwarding. This would be useable for all connections.

Connections: Support 10GE user ports


NLR L2 Services Summary

Possible Long-range Services

User-controlled Web-based provisioning and configuration Allow users to automatically create new services, or reconfigure existing services on the network using a web-based tool.

Time-sensitive provisioning – Allow users to have L2 connections with bandwidth dedicated only at certain times of day, or certain days.


Layer 3


NLR L3 Services Summary Base Features, Day 1

BGP to each member, with their 2 connections configured any way they’d like. Options include but aren’t limited to: primary/backup, route-based load balanced, or 1 for traffic/1 for measurement.

ISIS core IGP

IPv4 Multicast: PIM, MSDP, MBGP by default.

IPv6: BGP peering by default

Security: Each member will be able to black-hole traffic to one of their hosts using special BGP community


NLR L3 Services SummaryFeatures Document

Day One Connection Each member gets a 10GE connection and a VLAN backhauled over the L2 network to a second node.

General operations of the network, including base features (configuration with no experiment running), connections, and communication of experiments.

Future PossibilitiesPeering with other R&E networks.

Commodity Internet Connections or peering.


NLR Layer 3 locatonsAtlanta: Level3, 345 Courtyard, Suite 9

WashDC: Level3, 1755/7 Old Meadow Rd Suite: 111,McLean VA.

NYC: MANLAN, 32 Avenue of the Americas, 24th floor

Chicago: Level3, 111 N. Canal, Suite 200

Denver: Level3, 1850 Pearl St, Suite 4

Seattle: PacWave, 1000 Denny Way

Los Angeles: Equinix, 818 W. 7th Street, 6th Floor

Houston: Wiltel, 1124 Hardy St.


UNM

LA

Duke/NC

FLR

OK

PSC

HOU

Layer 3 Initial Logical Topologyshowing planned member connections

LAX

WDC

ATL

CHI

NYC

DEN

SEA

Cisco CRS-1 router

10GE wave

CIC ITN

ITNITN

PNWGP

UCAR/FRGP

LEARN

Cornell

CENIC

GATech

MATP

NLR L2

NLR L2

NLR L2

NLR L2

NLR L2t

NLR L2

NLR L2

NLR L2


NLR L3 Hardware Cisco CRS-1 Half-rack

Chassis Height: 38.5 in. 99.06cm, (with base cosmetics)

Chassis Width: 17.5 in. 44.45 cm

Chassis Depth: 36.6 in 93.0 cm; (40.5 in. 100.844 cm, including full cosmetics)

Weight: 330.8 lb (138 kg) chassis with fans, PDU, and blanks (as shipped)

650 lb (294.8 kg) chassis as shipped, including power shelves, all line cards, and route processors

DC Power: 7900 WDCNeeds 3 65A feeds per power supply

Clearance: 3 feet front and back required.

Four-post mounting is strongly preferred.


CRS1 base configurationCRS-1 (half-rack) 2 route processors (RPs)4 switch fabric cards 2 Power Entry Modules2 control plane software bundle licenses (IOS-XR) with crypto2 memory modules for each RP (required) – 2GB each1 8x10GE line card1 multi-service card (MSC)1 8x10GE PLIM1 line card software license1 extra MSC1 extra line card software license8 XENPAK 10G-LR opticsfan trays and air filters


CRS1 base configuration

Sites that had at least 7 of their 8 10GE interfaces assigned at initial installation receive a second 8x10GE, including the MSC and software license.

Chicago Denver Houston

These location have a total of 12 XENPAK 10G-LR optics modules.We call the first configuration “A”, and the configuration

with the additional 8x10GE type “B”. The NLR layer 3 network will be comprised initially of 5 type “A” routers and 3 type “B” routers.


Layer 3 Engineering sub- committee recommendations

• Add logical router capability to select sitesLogical routers are a collection of line cards and Route Processors (RP) which together form a distinct router within the chassis. In order to provide additional logical routers for hardware-level separation of services, additional line cards and RP resources must be installed. Logical routers may allow multiple potentially destabilizing experiments to be run simultaneously on the NLR network with the least

• Add 1GE availability to select sitesThis provides a lower speed and less expensive class of interface to which connectors can attach allows for temporary connections, direct monitoring and measurement connections, and direct attachment of reasonably priced experimental tools such as traffic generators and protocol testers.

• Add additional 10GE to select sites

There are three main reasons for recommending this addition. First, with the base configuration, all but one site is already oversubscribing the slot from day 1. Also, some sites are already completely full and have no room for expansion, even with oversubscription. Although oversubscription may be fine for many experimental scenarios, it would be better to have the option of oversubscription or not, depending on the project at hand. Because the costs are so heavily loaded into the line cards, it is probably unrealistic to expect the “next” connector to absorb the costs of moving to a second 8x10GE card, and it will be a while before 10GE is supported in SPA format. Third, similar to the 1GE line cards recommended above, it provides a more interesting architecture with greater fanout and provides another line card which can be used in the case of logical routers



Private Test-lab Network Connections.

Route Advertiser Connections: Get active commodity routing table for experiments, but no actual commodity bandwidth drainage.

Pre-emptible Connections: Allow other types of connections to use unused ports on a temporary basis, such as for a conference or measurement project.


NLR L3 Services SummaryFuture Options

MPLS (Traffic Engineering, L2/L3 VPNs): This can be done on an experimental basis Day 1. May be rolled into Base Feature list if there is general interest.

IPv6 Multicast: Will be implemented as support and standards mature.

User-Controllable Resource Allocation: Will be supported as experiments, and rolled into the base feature list if there is general use and interest.


NLR L3 Services SummaryService Expectations Document (Exhibit F, Attachment C)

Set user expectations for service on L3 network

Make clear the experiment support model

Service Expectations,

SLA isn’t good measure since the network may appear “down” because of experiments.

Network may not have same uptime as production network, but will have same level of service and support as production network



Experiment Support

Each experiment will have a representative from the L2/L3 Support Center and a representative from the ESC.

If necessary, the prospective experiment will be sent to the NNRC for review.

L2/L3 Services staff will be responsible for scheduling network assets for experiments and will see the experiment through to completion.

In general, experiments will be scheduled on a first-come-first-served basis.


NLR L3 Services Summary Network Conditions

Way of communicating the current state of the network to users.

Users may choose to have their interfaces automatically shutdown under any Network Conditions they desire.

Users will receive notification of changes to Network Condition, with focused communication to those who will be turned on or off because of it.

Tools will be available for users to monitor and track Network Conditions.


Network ConditionsNetCon 7- No Experiment Currently ActiveNetCon 6- Experiment Active, No Instability

ExpectedNetCon 5- Possible Feature Instability/No

General Instability ExpectedNetCon 4- Possible Network InstabilityNetCon 3- Congestion ExpectedNetCon 2- Probable Network Instability/Possible

Impact on Connecting NetworksNetCon 1- Network Completely Dedicated


NLR Layer 2/3 Service Center projects

Out of band monitoring/access network

Database development

General support applications

Specific support applications

Experimentation support

Monitoring and measurement support






NLR Layer 2/3 - discussion What do users want/need?Tools?User groups?Monitoring and measurement ability?Full routes? Control over route propagation?Control over protocols and timers?Direct access to login and configure routers?A router “ghost” service?Instruction/workshops?Commodity access or ISP collaboration?Collaboration with projects like PlanetLab and WAIL?


NLR User Resources

• http://noc.nlr.net

• [email protected]



http://noc.nationallambdarail.org/

mailto:[email protected]



Documents

National LambdaRail Layer 2 and 3 Networks 3 May 2005 Internet2 Spring Members Meeting Layers 2 & 3 Users BOF