Dynamic Multi -Layer Mesh Networks: A Provider’s Perspective

Dynamic Multi-Layer Mesh

© 2009 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.

Dynamic Multi-Layer Mesh Networks:

A Provider’s Perspective

Peter MagillExecutive Director, Optical Systems ResearchAT&T Labs

RECONFIGURABLITY Eliminating unnecessary

transponders leads to

Tingye Li, AT&T Bell Labs

March 1994

RECONFIGURABLITYfurther enhances the efficiency of optical add/drops.

transponders leads to

dramatic savings.

TX

The Multiwavelength Optical NETworking VisionMONET: DARPA, AT&T, Lucent, Bellcore, Bell Atlantic , ...

EL

EC

TR

ON

ICS

WIT

CH

ING

LA

YE

RS

AP

PL

ICA

TIO

NS

LA

YE

R X-C WDM X-Connect

WDM Routing Star

WDM Add/Drop Mux

WDM NETWORK ELEMENTS

EL

EC

TR

ON

ICA

PP

LIC

AT

ION

S

OPTICALACCESS

LAN

DATA

VIDEO

MULTIMEDIA

VOICE

SONETATM

IMAGE

ATM

LEGACYSWITCH

SONET

LOCAL-EXCHANGE NETWORK LONG-DISTANCE NETWORK

EL

EC

TR

ON

ICS

WIT

CH

ING

LA

YE

RS

CO

NF

IGU

RA

BL

EO

PT

ICA

L, W

DM

(MONET

) L

AY

ER

S

PRIVATENETWORK

WITH DIRECTOPTICAL ACCESS

OP

TIC

AL

EL

EC

TR

ON

IC

X-C

Ring

X-C

X-C X-C

X-C

X-C

X-C

LEGACYSWITCH

Saleh, OFC ’96; Wagner, et al, JLT June ’96;Alferness, et al, OFC ’97; Garrett, et al, JSAC Sept ‘98

A.A.M.Saleh, 1998

The ever-increasing demand for increased capacity and level

of service at a lower cost are key drivers fueling the evolution

of core optical networks from statically provisioned optical

links interconnected with electronic switching and

regeneration to more complex and flexible, optically switched

mesh topologies with dynamic provisioning.

Let’s Dissect Abstract of Symposium

mesh topologies with dynamic provisioning.

Page 4

What does this mean?

ECOC 2009 - Symposium 6.7

point-to-point

fixed OADM

2-degree ROADM (static)

multi-degree ROADM (static)

multi-degree ROADM

dynamic (few minutes; via planners)

multi-degree ROADM

dynamic (few sec; via

automated

multi-degree ROADM dynamic

Photonic Networking Timeline

multi-degree dynamic R-ADM (few sec; via automated

process)

multi-degree R-ADM dynamic

( <100 ms )“Optical” Networking

(static) (static) minutes; via planners)

automated process)

dynamic ( <100 ms )

GMPLS

Page 5 ECOC 2009 - Symposium 6.7

mesh

GMPLS?

Feasible

?

NOT feasible with today’s transmission systems

Why Rapidly-Dynamic ROADM Networks

Not Feasible?

A long-haul (L > ~600 km) transport system contains:

• Many optical amplifiers:

• Most with adjustments needed for multiple pump laser powers

• Some with adjustable gain flattening filters (GFF)

• All with control circuits for optical transient control

• Multiple ROADMs:

• Usually with per-channel variable optical attenuator adjustments • Usually with per-channel variable optical attenuator adjustments

• Multiple transponders

• Each with tunable laser

• Many with tunable dispersion compensators (TDC)

• Each with variable optical attenuator (VOA)

• Together, there DOZENS of things which need to be adjusted (“tuned”) and collection of control loops makes some of them get VERY slow [frequently minutes to converge]


DWDM Transport - Today

• Longer provisioning cycle than desired– Many manual steps

– Across the country (both ends + maybe middle)

– Mux/demux inflexible

• Static mapping: circuit �� wavelength

Transponders ROADM LayerTransponders cannot be pre-deployed withoutcommitting wavelengths.

ROADM Layer

ROADM

T/R

ROADM

T/R

ROADMROADM

ROADMROADM

ROADM

ROADM

ROADM

Page 7ECOC 2009 - Symposium 6.7

DWDM Transport Evolution – Next Step

• Colorless add/drops

– Currently: transponders are tunable but demux is not.

– Tunable demux will enable transponders to be pre-deployed, and circuits to be turned up rapidly. Transponder at B can be connected to A or C simply by tuning pair of transponders to the same wavelength, and setting demux and ROADMs properly. ROADM Layer ROADMproperly.

– Wavelengthdoes not needto be chosenin advance.

ROADM Layer

ROADM A

T/R

ROADMB

ROADMC

T/R

T/R


When Discussing Alternate Network

Architectures …

• What some call “IP/WDM”

– Proposal to have ‘colored’ (long-haul wavelength) interfaces directly on IP routers

– Or integrate long-haul optics in IP routers

• Others call this an “alien wavelength” architecture

• In AT&T’s inter-city network:


• In AT&T’s inter-city network:

– AT&T will require 100G (OTU4) ‘gray’ (non-WDM) optics on routers

– AT&T will have NO colored optics in routers

– AT&T will consider multiple vendors for routers and for transport

– While IP routers are a big source of traffic, they are not the only one

– We also have other network layers and private line traffic which use long-haul transport – not through core routers

Project GRIPhoN

Globally Reconfigurable Intelligent Photonic Network

An AT&T Labs Research project to study:

• feasibility of a dynamic wavelength layer

• cross-layer control and timing issues

• control and management software requirements

Collaborators:

Martin Birk, Angela Chiu, Bob Doverspike, Mark Feuer, Pete Magill, Emmanuil Mavrogiorgis, Jorge Pastor, Sheri Woodward, Jennifer Yates, Joy Zhang


Fiber Cross-Connect (FXC)

A-ports and B-ports: each N in number

Connections established between any pair of an A-port and a B-port

Connections don’t care about direction or wavelength of light

A B

demux mux

Recent developments:• Low optical loss (< 3dB)• Very low cost

FXC

A B

Subsequent slides don’t showA & B separately, and connectionlines represent fiber PAIRS.

T/R

T/R

T/Rtransponders


Dynamic Wavelengths in Transport

• Purpose of a transport system:

• convey a bit stream from one client to another

• over a long distance (> 2 km)

• with an error rate < ~10-14

• “Client” could be:

• IP Router, OTN switch, Ethernet switch, SONET/SDH ADM

• Other electronic switches or cross connects• Other electronic switches or cross connects


λ1

λ2

λ3

λN

R

R

R

R

λ1

λ2

λ3

λN

C

C

C

C

C

C

C

C

Proposed ROADM Architecture: Overall

• ROADM = Reconfigurable Optical Add/Drop Multiplexer

West East

South

ROADM

Wavelength-Selective

Cross-Connect

(WSXC)

Line-side


T/R T/R T/R T/R

Line-sideFiber Cross-Connect

CLIENT

Client-sideFiber Cross-Connect

CLIENTCLIENT

L-FXC

C-FXC

T/R

Any client can use any transponder to access any unused optical channel, routed in any direction.

ROADM Architecture:

Colorless Add/Drop• Today the “R” in ROADM is misleading

• Reconfiguration of commercially available ROADMs includes manual steps of cabling fiber pair from the line side of each transponder to wavelength MUX/DMUX

• CANNOT be done remotely

• Instead, East

ROADM

Page 14

• Instead, insert low-loss FXC between transponderand mux/ dmux

East

T/R


Cross-Connect

(WSXC)


L-FXC

T/R

ECOC 2009 - Symposium 6.7

ROADM Architecture:

Steerable Transponders• For multi-degree ROADM (e.g., with fiber routes to North, South, East and West)

• Transponders should be grouped into “banks” – One (or two for redundancy)

• But then any transponder from any bank should be able to be used for any fiber direction (N, E, S or W)

– Gives carrier flexibility since changes in traffic are notoriously hard to predict

ROADM


West East

South

T/R T/R T/R T/R

ROADM


Cross-Connect

(WSXC)


L-FXC

T/R

ROADM Architecture:

Client-Side FXC

• For full flexibility, each transponder should not be permanently “wed” to a given client

• Connections between clients and transport system should also be remotely reconfigurable

• Any transponder should be able to connect to any client


T/R T/R T/R T/R

CLIENT


CLIENTCLIENT

C-FXC

T/R

Reprise: Full Node Architecture

- One Option

West East

South

ROADM


Cross-Connect

(WSXC)


T/R T/R T/R T/R


CLIENT


CLIENTCLIENT

L-FXC

C-FXC

T/R

• Now any client can use any transponder to access anyunused optical channel, connected in any direction.


Alternative ROADM Design: Serve Banks of

Transponders Directly From ROADM Core

West East

South


Cross-Connect

(WSXC)

transponder

banktransponderWSS

WSS

• Transponder bank served by another fiber-degree of WSXC• Enables transponder steering without line-side FXC • Scales gracefully over entire lifetime

CLIENT CLIENTCLIENT

T/RT/RT/R

transponder

bankT/R

C-FXC

T/RT/R


Another Possible Implementation

Node A’s WSXC

• This ROADM also supports growth of degree (in service)

fixedjumpers


Desirable Node Attributes

• Full flexibility

– Transponders also not dedicated to a particular color or direction

– Called “colorless” and “steerable”

• Scaleable architecture

– Low first cost, yet able to grow as traffic grows

• Graceful growth

– Ability to add new routes and more transponders while node remains in service

direction-less

while node remains in service

• Hitless operation

• Directional separability

– Single failure should not affect all add-drops or all routes

• Minimize loss/OSNR degradation to maximize reach

• Explicitly test/verify correct lightpath routing

• Low cost


Possible Applications for GRIPhoN

Network efficiency

• Wavelength Re-grooming

Faster, Easier Provisioning

• Dynamic Wavelength Service

Maintenance and Restoration

• At Layer 1: for Planned Cable Intrusions

• At Layer 1: for un-planned cable intrusions

• At Layer 3: Backbone router maintenance


Application

Wavelength Re-grooming

Today, circuit-to-wavelength mapping: static

Over years, choice of wavelength and route for many circuits become sub-optimal (because of blocking, optical “reach” constraints)

Ability to change wavelength for:

• given circuit over • given circuit over

• given route

is quite valuable

Imperative to have no service disruption


Present Mode of Operation (Manual)change wavelength : λλλλ1�� λλλλ4

Node AROADM

core

Node BROADM

coreCDE

λ1 λ4 λ1 λ4 λ1 λ4 λ1 λ4

0. 1A, 1B active at λ1, connected to λ1 ports

1. Take service down!2. MANUALLY insert 2A, 2B1A

2A

1B

2B

Client Client

•Need technicians at all sites (2; more if regenerated)•Traffic affecting – perceptible outage

2. MANUALLY insert 2A, 2B3. MANUALLY connect to λ4 ports 4. MANUALLY tune 2A, 2B to λ4 and

activate 5. MANUALLY move client fibers at

A, B6. Restore service7. Release 1A, 1B for other uses

1A 1B


Step 0

Bridge-and-Roll

OT OT

Step-by-step operation to change wavelength

Utilizes:

• optical splitter on each client transmitter and

• additional pair of optical transponders (OT)

Step 1Client a Client bOT OT

Step 2

Step 3

GRIPhoN goal:• < 50-100 millisecond interruption to circuit• Will not trigger higher-layer interactions

original transponder pair now available for other operations


Application

Rapid Provisioning: Dynamic Wavelength

Service

For photonic layer to be dynamic• need spare resources (transponders, lightpaths, etc.)

• deployed all over the network

Enterprise customers could use Dynamic Wavelength Service (and the reserve of potential capacity) to turn up a class of wavelength circuits capacity) to turn up a class of wavelength circuits in minutes rather than months [provided the customer has adequate connectivity to core]

Customers could turn capacity up (or down) with seasonal or even time-of-day load shifts

To enable this carrier will need new processes to monitor transport utilization and provide sufficient resources


Customer

Dynamic Wavelength Service

- Customer Need

Customer

Site 3

Dynamic – these demands may come and go

Customer

Site 2

OC-48

Customer

Site 1

Router

EthernetSwitch

EthernetSwitch

Customer

Site 4

40 Gb/s

40 Gb/s


Dynamic Wavelength Service –

ROADM Layer

100Gb/s

FX

C

100

Gb/s

OTN

Customer

Site

100Gb/s

MetroNetwork

Inter-city

Network

ROADM Layer

FX

CF

XC

RO

AD

M

OTN

POP


OTN Layer


Network Layers

100Gb/s

FX

C

100

Gb/s

OTN

Customer

Site

100Gb/s

MetroNetwork

Inter-city

Network

ROADM Layer

FX

CF

XC

RO

AD

M

OTN

POP


OTN Layer

GRIPhoN

Controller

DWS-Controller

Dynamic Wavelength Service -

Management Plane

100Gb/s

FX

C

100

Gb/s

OTN

Customer

Site

100Gb/s

MetroNetwork

Inter-city

Network

ROADM Layer

Controller

FX

CF

XC

RO

AD

M

OTN

POP



Data Plane

0.6- 2.5Gb/s

circuits

Customer

OTN Layer

Customer

Site 3

100Gb/s

FX

C

100

Gb/s

OTN

Channelized 10 Gb/s

Customer

Site 1

Router

EthernetSwitch

100Gb/s

EthernetSwitch

MetroNetwork

Inter-city

Network

40 Gb/s

40 Gb/s

circuits

Customer

Site 2

Customer

Site 4

ROADM Layer

FX

CF

XC

RO

AD

M

OC-48

OTN

POP


ApplicationRapid Provisioning: Private Line (PL)

Private Line business could potentially use a Dynamic Wavelength Service internally

Using the same reserve capacity, provision PL circuits via Dynamic Wavelength Service Controller


Application

Layer 1 Maintenance:

Planned Cable Intrusions

Geographically large network of fiber routes – all buried

Frequent public works, roadway changes, etc. which require fiber re-routes

Potentially disrupting traffic during few-hour period Potentially disrupting traffic during few-hour period (overnight) for multiple PL customers

Higher layer services (like IP) restore themselves


Planned Maintenance Events - 2009

original cable

new route

working path

B

A

EXAMPLE: Dept of Transportation informs carrier that a cable must be moved to make way for a new road project

1) Local alternate route found, and new cable installed

roadproject

hut

hut


2) Once the new cable is ready, a network event is scheduled – verify that there are no conflicting events

– notify customers

3) Before the event, conduct a walk-through

4) Network Event:a. Verify that the alternate fiber path is ready

b. Move traffic to alternate fiber path

c. Splice in new cable and check each fiber

d. Move traffic back to working path (over new route & cable)


EXAMPLE: Dept of Transportation informs carrier that a cable must be moved to make way for a new road project


Possible Future Planned Maintenance

Events – with GRIPhoNoriginal cable

new route

working path

B

Aroadproject


2) Once the new cable is ready, a network event is scheduled

– verify that there are no conflicting events,

– notify customers

3) Before the event, conduct a walk-through

4) Network Event:

a. Verify that the alternate fiber path is ready

b. Move traffic to alternate fiber path

c. Splice in new cable and check each fiber

d. Move traffic back to working path (over new route & cable)Page 34 ECOC 2009 - Symposium 6.7

Un-Planned Events (Cable Cuts) –

with GRIPhoN

(formerly) working

B

A

EXAMPLE: Joe the Backhoe Operator takes an underground swipe “a little too deep,” severing a buried fiber cable

0) Protection path pre-calculated

Oops!

0) Protection path pre-calculated1) Systems alarm:

– Routers will react, triggering OSPF and/or BGP re-convergence– Transport reacts, triggering GRIPhoN restoration sequence

2) Photonic layer restoration:– GRIPhoN switches move traffic to restoration paths, restoring L1 capacity– For Private Line – out of service for ~1 minute instead of ~6 hours– For Layer 3 – results in more reliable Layer 1 (double failures, etc.)

3) Later (next few days):a. Repair cableb. Use GRIPhoN to move traffic back to working path – no customer impact


ApplicationLayer 3 Maintenance:

Backbone Router Upgrades, etc.

IP routers tend to have a lower availability than transmission equipment

IP routers must be brought down regularly for software upgrades

To be very reliable, IP network architecture must To be very reliable, IP network architecture must allow any one router to be down at any time

Use GRIPhoN to provide dynamic transport to off-load routers and divert traffic for L3 maintenance

Solution:


Major Challenges

Operations Support Systems (OSSs)

Planning/provisioning process & organizational structure

What should be the ultimate holder of the optical-layer data?• The network (some combination of ROADMs and vendor EMS)?

• Centralized control systems or OSS?

• Some combination of both?

For photonic layer to be dynamic• Need spare resources (transponders, lightpaths, etc.)

• How much? Where? – Strong influence on business case


Conclusions

• There are multiple applications for a dynamic wavelength layer

• AT&T Labs Research is studying the feasibility question with a prototype lab implementation

• Results to be published soon


BACKUP


• Metro (metropolitan area):

– within and around a city

– ~200 km reach

– carry lots of local traffic

– cascade many ROADMs (~16)

• Regional:

– a few states of USA, each region; nationwide overall

– ~1000 km reach, 2-3 ROADMs

AT&T’s Network Covers Many Scales


– ~1000 km reach, 2-3 ROADMs

– feeder for ultra-long haul

– recent & older fiber

• Core or Ultra-Long Haul:

– nationwide

– ~1500 km reach

– ~6 ROADMs

– good quality fiber

coherent receiver canhelp with PMD

Documents

Dynamic Multi -Layer Mesh Networks: A Provider’s Perspective