NLR Layer2/Layer3 Users BOF NLR status update Albuquerque Internet2 Joint Techs 8 February 2006 Brent Sweeny, Indiana University Jon-Paul Herron, Indiana

NLR Layer2/Layer3 Users NLR Layer2/Layer3 Users BOFBOF

NLR status updateNLR status update

Albuquerque Internet2 Joint Albuquerque Internet2 Joint TechsTechs

8 February 20068 February 2006Brent Sweeny, Indiana UniversityBrent Sweeny, Indiana University

Jon-Paul Herron, Indiana UniversityJon-Paul Herron, Indiana UniversityJohn Moore, North Carolina State John Moore, North Carolina State

UniversityUniversity

NLR Layer2/Layer3 users BOFNLR Layer2/Layer3 users BOF 22

AgendaAgenda

1.1. Progress Report Progress Report

2.2. Questions (non how-do-I-connect)Questions (non how-do-I-connect)

3.3. What’s the Layer2/3 connection What’s the Layer2/3 connection process?process?

4.4. Experiments case studiesExperiments case studies

5.5. ““Ask the experts”—questions, special Ask the experts”—questions, special situations, discussion, “how-do-I”, “I situations, discussion, “how-do-I”, “I need…”, etc.need…”, etc.


NLR Engineering/Support NLR Engineering/Support OrganizationOrganization

A very A very distributed, coordinateddistributed, coordinated organization: organization: Service desk at IndianaService desk at Indiana Layer1 NOC and engineering at CENICLayer1 NOC and engineering at CENIC Layer2/3 NOC and engineering at IndianaLayer2/3 NOC and engineering at Indiana

Also: Measurement, monitoringAlso: Measurement, monitoring Tech mailing lists for Layer2 and Layer3 usersTech mailing lists for Layer2 and Layer3 users Technical documentationTechnical documentation

Experiments support center at North CarolinaExperiments support center at North Carolina


The Big News about NLR The Big News about NLR Layers 2 and 3:Layers 2 and 3:

It’s working!It’s working! People are connected, and using it!People are connected, and using it!


Who’s Who—Who’s Who—aboutabout 150 150 institutionsinstitutionsNLR Members NLR Members (for participants see(for participants see http://http://

www.nlr.net/members_participants.htmlwww.nlr.net/members_participants.html))

Corporation for Education Network Initiatives in California (CENIC)

Pacific Northwest GigaPop (PNWGP)Pittsburgh Supercomputing Center and the University of

PittsburghDuke University, representing a coalition of North Carolina

universitiesMid-Atlantic Terascale PartnershipInternet2® (no participants currently) Florida LambdaRail, LLCSouthern Light Rail, Inc.Committee on Institutional Cooperation (CIC)Cornell UniversityLouisiana Board of RegentsOklahoma State Board of RegentsLonestar Education and Research Network (LEARN)University of New Mexico, on behalf of the state of New MexicoUniversity Corporation for Atmospheric Research (UCAR),

representing a coalition of universities and government agencies from Colorado, Wyoming, and Utah

http://www.nlr.net/members_participants.html

http://www.nlr.net/members_participants.html


NLR PacketNet/FrameNet Current Status


Review:Review:NLR architecture and NLR architecture and

servicesservices


National LambdaRail designNational LambdaRail design

NLR WaveNet PoP NLR WaveNet & FrameNet PoP NLR WaveNet, FrameNet & PacketNet PoP

NLR owned fiberBAT

SAN

STA

CHISLC

HOU

DAL

SYR

TUL

PENELP

KAN

PHOLAX

ALB

PIT

WDC

OGD

BOI

CLE

ATL

POR

RAL

NYC

SAA

DENSVL

SEA

JAC


Generic NLR L1, L2 and L3 PoP Layout

CRS-1

Colo

EastWest

NLR demarc

15808

6509

15454

DWDM

1G wave, link or port10G wave, link or port


DAL

SYR

TUL

PENELP

PHO

BAT

ALB

HOU

WDC

OGDCLE

NYC

SAA

4

4

JAC4

4

4

44

4

4

4

4

4

4

4

4

4

4

4

44

44

4

44

4 4

SLC4

4

Level3 fiberWilTel fiber

4

4

RAT4

Cisco 15808 terminalCisco 15808 OADMCisco 15454 terminalCisco 15454 OADM

8

4

4

8

STA

CHIKAN

PIT

BOI

CLE

ATL

POR

RAL

DENSVL

SEA

8

8

8

8

8 8

8

8

8

8

88

8

8

88

88

8

8

8

8

8

4

4

LAX 8

8

NLR Layer 1 “WaveNet”NLR Layer 1 “WaveNet”


Layer 1 Phase 2 Layer 1 Phase 2 DeploymentDeployment

DAL

SYR

TUL

PENELP

KAN

PHO

BAT

ALB

HOU

WDC

OGDCLE

NYC

SAA

DEN

4

4

JAC4

4

4

44

4

4

4

4

4

4

4

4

4

4

4

44

44

4

44

4 4

SLC4

4

Level3 fiberWilTel fiber

LAX4

4

RAT4

Cisco 15808 terminalCisco 15808 OADMCisco 15454 terminalCisco 15454 OADM

8

4

4

8


Layer 1 baselineLayer 1 baseline

Opportunity to connect into lambda fabricOpportunity to connect into lambda fabric Point to pointPoint to point Other endpoint could be anywhereOther endpoint could be anywhere

Early examples:Early examples: HOPIHOPI UltralightUltralight iGRIDiGRID SC05 (Supercomputing)SC05 (Supercomputing)


Layer 2 Network Design Layer 2 Network Design “FrameNet”“FrameNet”

HOU

TUL

ELP

KAN

BAT

LAXALB

PIT

WDC

CLE

ATL

RAL

CHINYC

DENSVL

SEA

JAC

10GE wave10GE managed waveYellow sites are done

Cisco 6509 switch

SVL


Layer 2 installation statusLayer 2 installation statusAll Switch installations completed:

Sunnyvale Denver Kansas City ChicagoCleveland Pittsburg Raleigh Washington DCJacksonville Atlanta Los Angeles TulsaEl Paso Houston New York City Baton RougeAlbuquerque

Layer2 backbone interconnections status:

All layer2 backbone interconnects are done except:•Tulsa-Kans•Tulsa-Hous (that is, both directions out of Tulsa)•New York City-Wash•New York City-Clev (e.g. both directions out of NYC)•Los Angeles-Sunnyvale


Layer 2 service baselineLayer 2 service baseline(see also FrameNet Technical Guide)(see also FrameNet Technical Guide)

1GE Connection into local 65001GE Connection into local 6500 1GE Access to “national exchange fabric”1GE Access to “national exchange fabric” Additional Additional OptionsOptions::

Dedicated point to point Etherrnet, Nx1GEDedicated point to point Etherrnet, Nx1GE Best-effort point to multipoint (no dedicated Best-effort point to multipoint (no dedicated

bw)bw) Soon:Soon:

10GE ports10GE ports Dedicated point-to-multipointDedicated point-to-multipoint


GoalGoal

Provide circuit-like options for users Provide circuit-like options for users who can’t use, can’t afford, or don’t who can’t use, can’t afford, or don’t need, a 10G Layer1 wave via point-need, a 10G Layer1 wave via point-to-point layer2 VLANs.to-point layer2 VLANs.

Experiment with large-scale layer2 Experiment with large-scale layer2 capabilities.capabilities.


National Exchange FabricNational Exchange Fabric Multipoint, public, best-effort, resilient, permanent, stableMultipoint, public, best-effort, resilient, permanent, stable

Non-dedicated bandwidth Non-dedicated bandwidth NLR allocated addresses, peer with any other member across the layer 2 exchange policy-free.NLR allocated addresses, peer with any other member across the layer 2 exchange policy-free. Possible to have more than one (say by MTU).Possible to have more than one (say by MTU). Ready to go today.Ready to go today. Current participants: Duke, MATP, SLR—several more coming…Current participants: Duke, MATP, SLR—several more coming…

Backup connections to networks such as Abilene, commodity Backup connections to networks such as Abilene, commodity providers, etc.providers, etc.

Point-to-point, private, permanent, stablePoint-to-point, private, permanent, stable Could be best-effort or guaranteed.Could be best-effort or guaranteed. Could be nailed-up or resilient.Could be nailed-up or resilient. Load-balance or leave idle until needed.Load-balance or leave idle until needed. Ready to go today, though we only have pricing for the guaranteed nailed-up case.Ready to go today, though we only have pricing for the guaranteed nailed-up case. Example: NRENExample: NREN


To enable a flexible topology for NLR To enable a flexible topology for NLR layer3layer3

Best-effort, private, temporary, experimentalBest-effort, private, temporary, experimental We have 8 layer3 nodes, but the topology between We have 8 layer3 nodes, but the topology between

them can be made much more interesting by creating them can be made much more interesting by creating various connections over the layer2 network.various connections over the layer2 network.

Enables layer3 experimentation.Enables layer3 experimentation.

To provide members with a second path To provide members with a second path into the NLR layer3 networkinto the NLR layer3 network

Point-to-point, private, permanent, experimentalPoint-to-point, private, permanent, experimental Connect to a second node on the layer3 backbone.Connect to a second node on the layer3 backbone. Load-balance or leave idle until needed.Load-balance or leave idle until needed. Included in membership.Included in membership.


Temporary connections for special Temporary connections for special projectsprojects

Guaranteed, private, temporary, stableGuaranteed, private, temporary, stable For remote instrumentation, where the member For remote instrumentation, where the member

only has the remote resource reserved for a only has the remote resource reserved for a limited window.limited window.

For conferences, demos, and other special For conferences, demos, and other special events.events.

Provides a low latency/jitter path if needed.Provides a low latency/jitter path if needed. Nailed-up if latency is critical, probably resilient Nailed-up if latency is critical, probably resilient

if not.if not. Could be point-to-point or multipoint.Could be point-to-point or multipoint. Technically, this is possible today, but we have Technically, this is possible today, but we have

no pricing model for it.no pricing model for it.


Bootstrapping circuit-like researchBootstrapping circuit-like research Point-to-point, private, guaranteed, Point-to-point, private, guaranteed,

temporary, nailed-up, experimentaltemporary, nailed-up, experimental To enable a researcher to get started while waiting To enable a researcher to get started while waiting

for funding or provisioning of a layer1 circuit.for funding or provisioning of a layer1 circuit. Similar to a special event, but more experimental, Similar to a special event, but more experimental,

a probably a stronger need for it to be nailed-up.a probably a stronger need for it to be nailed-up. Technically, this is possible today, but we have no Technically, this is possible today, but we have no

pricing model for it.pricing model for it.

Provide control plane network for optical Provide control plane network for optical experimentsexperiments

Permanent, resilient, experimentalPermanent, resilient, experimental A topology could be created for the oob A topology could be created for the oob

management network needed for some dynamic management network needed for some dynamic optical networking experiments (GMPLS, etc.)optical networking experiments (GMPLS, etc.)


Cluster/Grid LANCluster/Grid LAN Multipoint, private, guaranteed, experimentalMultipoint, private, guaranteed, experimental

To enable remote clusters to appear on the same LAN.To enable remote clusters to appear on the same LAN. It is not known if spanning tree would be wanted.It is not known if spanning tree would be wanted. It could evolve into a more production-like It could evolve into a more production-like

requirement.requirement. Technically, this is possible today, but we have no Technically, this is possible today, but we have no

pricing model for it.pricing model for it.

Experiment directly with Layer 2Experiment directly with Layer 2 Could be of any type (experimental, obviously)Could be of any type (experimental, obviously)

Web-based provisioning, direct user requests, etc.Web-based provisioning, direct user requests, etc. Concern about interaction with more production-like Concern about interaction with more production-like

requirements.requirements.


ALBbackhaul

BATbackhaul

RALbackhaul

JACbackhaul

TULbackhaul

PITbackhaul

HOU

LAX

WDC

ATL

CHINYC

DEN

SEA

Cisco CRS-1 router

10GE waveYellow sites are installed

Layer 3 Network Layer 3 Network “PacketNet”“PacketNet”

LAX


Layer 3 installation statusLayer 3 installation status

All layer3 router installations are complete.

All interconnections between layer3 backbone routers are complete now except:

•New York City (both directions)


Layer3 connections are up Layer3 connections are up to:to:

NLR member sites: MATP UCAR Duke/NC PNW CENIC PSC

Peer networks:Peer networks: CAnetCAnet USGSUSGS TranspacTranspac

Coming peers:ESnetDRENStarLightNREN

Exchange points:StarLight (now)Pacific Wave (signed)(others soon)


PacketNet Peering PrinciplesPacketNet Peering Principles

Very simple: “AUP-free”Very simple: “AUP-free”


Layer3 Peering DetailsLayer3 Peering DetailsMember ConnectionsMember Connections

• Prefix Lists are used, with no approval process for updatesPrefix Lists are used, with no approval process for updates• Only routes NLR will normally prevent are:Only routes NLR will normally prevent are:

• Bogons and private addresses Bogons and private addresses • Transit of other upstream providers Transit of other upstream providers


Layer3 Peering DetailsLayer3 Peering DetailsNon-Member R&E PeersNon-Member R&E Peers

• ASN Lists are usedASN Lists are used• Only routes NLR will normally prevent are:Only routes NLR will normally prevent are:

• Bogons and private addresses Bogons and private addresses • Other than that, NLR will cater its peering to meet the expectations and needs for Other than that, NLR will cater its peering to meet the expectations and needs for

each peereach peer


Layer3 service baselineLayer3 service baseline(see also PacketNet Technical Guide)(see also PacketNet Technical Guide)

Each member gets two routed connectionsEach member gets two routed connections ““local” 10GE local” 10GE VLAN backhauled to 2VLAN backhauled to 2ndnd node node

BGP peering with NLR L3 networkBGP peering with NLR L3 network IPv4 unicastIPv4 unicast IPv4 multicast (MBGP/PIM/MSDP)IPv4 multicast (MBGP/PIM/MSDP) IPv6 unicast (multicast later)IPv6 unicast (multicast later) An ‘experimental’ (changeable, changing) An ‘experimental’ (changeable, changing)

L3 networkL3 network


Layer 3 coming servicesLayer 3 coming services

Likely eventual logical routersLikely eventual logical routers More 1GE optionsMore 1GE options More 10G optionsMore 10G options Pre-emptable connectionsPre-emptable connections MPLSMPLS More user control—scheduling, testing, More user control—scheduling, testing,

etcetc User access to measurement dataUser access to measurement data


NOC webpagesNOC webpagesfor Layer2/Layer3for Layer2/Layer3

(noc.nlr.net)(noc.nlr.net) ToolsTools

ProxyProxy Weathermap (layer2, layer3)Weathermap (layer2, layer3) UtilizationUtilization

Documents, notably:Documents, notably: FrameNet Technical GuideFrameNet Technical Guide PacketNet Technical GuidePacketNet Technical Guide PacketNet BGP CommunitiesPacketNet BGP Communities


NLR User ResourcesNLR User Resources http://www.nlr.nethttp://www.nlr.net http://noc.nlr.nethttp://noc.nlr.net [email protected]@nlr.net [email protected]@nlr.net [email protected]@nlr.net

http://noc.nationallambdarail.org/

mailto:[email protected]

