Ciena Core Modernization WP

8/6/2019 Ciena Core Modernization WP

1/14

White Paper

Modernizing the Core With OTNSwitching & 40/100G Transport

Prepared by

Sterling Perrin

Senior Analyst, Heavy Reading

www.heavyreading.com

On behalf of

www.ciena.com

December 2010
http://www.heavyreading.com/http://www.heavyreading.com/http://www.ciena.com/http://www.ciena.com/http://www.ciena.com/http://www.heavyreading.com/


2/14

TABLE OF CONTENTS

I. INTRODUCTION ................................................................................................... 3II. MARKET TRENDS IN THE CORE ....................................................................... 4III. PAIN POINTS REQUIRING THE CORE TO "MODERNIZE" .............................. 5IV. OPERATOR REQUIREMENTS FOR THE NEXT-GENERATION CORE ............... 7V. ARCHITECTURAL APPROACHES BEING EVALUATED .................................. 85.1 Switched Sonet/SDH ............................................................................................. 85.2 IP over DWDM (IPoDWDM) .................................................................................. 85.3 Switched OTN ....................................................................................................... 95.4 Transport Rate Options ......................................................................................... 9

Continue to Add 10G Capacity .............................................................................. 9Upgrade Infrastructure to 40/100G ........................................................................ 9

VI. OTN & 40/100G TRANSPORT: THE RIGHT APPROACH ................................ 106.1 40G & 100G Core Transport ............................................................................... 106.2 OTN Switching .................................................................................................... 11VII. CONCLUSION .................................................................................................... 14

LIST OF FIGURES

Figure 1: Global Mobile Broadband Connections by Technology ..................................... 4

Figure 2: Power Consumption by Element Type ............................................................... 5Figure 3: Drivers for 100G Deployments ......................................................................... 10Figure 4: Expected Client Interface Mix for 100G Core Networks in 2013 ...................... 11Figure 5: The OTN Universal Transport Network ............................................................ 12Figure 6: Core Modernization Network Diagram ............................................................. 13

HEAVY READING | DECEMBER 2010 | WHITE PAPER | OTN SWITCHING & 40/100G TRANSPORT 2


3/14

I. IntroductionFaced with continued high growth in IP traffic, large installed bases of TDM services and ele-ments that are not going away anytime soon, and slow revenue growth overall, operators arewrestling with how best to upgrade their networks to handle their network traffic and service de-mands, while remaining viable and competitive business entities. The core network, in particular,

is becoming a key focus area for many operators worldwide, following several years of invest-ments in their access and metro networks.

This white paper assesses the key pain points operators face in the core and provides an over-view of the main architectural options available to them. Specifically, we discuss switched Sonet/SDH networks (the PMO), integrated optics on core routers (IP over DWDM), and switched OTNnetworking. Heavy Reading makes the case for switched OTN networking, combined with 40-Gbit/s and 100-Gbit/s transport, as the best architectural approach for operator core networkmodernization.



4/14

II. Market Trends in the CoreNetwork operators in all regions of the world face a fundamental dilemma. On the one hand,global IP traffic continues to grow rapidly driven by both consumer and business applications. Onthe other hand, operator revenue is growing at single-digit rates at best and is not keeping pacewith network traffic growth.

On the consumer side, demand continues for bandwidth-intensive third-party content, such asover-the-top (OTT) video, peer-to-peer applications, gaming, and HD video. On the enterpriseside, customers continue to migrate from TDM private lines, Frame Relay, and ATM to higher-capacity Ethernet services (both private lines and VPNs), wavelength services, and IP VPNs.

Tier 1 ISP NTT Communications reports that IP traffic on its global network has increased at a 75percent CAGR from 2002 through 2009, with no signs of abating. Other operators that report traf-fic growth statistics, such as AT&T, consistently report their IP traffic growth in the 50+ percentrange annually. Meanwhile, these historical statistics do not fully reflect the future growth in ma-chine-to-machine traffic, driven by distributed data centers processing cloud-based applications.Nor do these historical growth rates adequately capture the mobile broadband surge that is justbeginning. Today, mobile data's contribution to IP traffic volume is small, but the rapid rise of mo-bile broadband subscribers coupled with the migration from low-speed technologies to advanced3G (such as HSPA+) and 4G technologies promises to change this picture dramatically.

Pyramid Research forecasts that the number of mobile broadband connections worldwide willreach 1 billion during 2012, with the growth driven by advanced 3G and 4G (see Figure 1). As aresult, mobile broadband traffic is set to boom over the next five years, growing three times fasterthan wireline traffic.

Figure 1: Global Mobile Broadband Connections by Technology

Source: Pyramid Research, part of the Light Reading Communications Group



5/14

III. Pain Points Requiring the Core to "Modernize"Operators are faced with several pain points in the core. Growth in IP traffic has led to prolifera-tion of core routers and core router ports processing IP traffic. Many operators have found that alarge percentage of traffic moving through core routers is transit traffic that does not need to beprocessed by the core IP/MPLS router. This traffic is bound for destinations further along in the

network, and the transit routers are simply performing a regeneration or grooming role at thoseintermediate locations.

IP/MPLS router processing of transit traffic is significant for three reasons:

Operators are reporting that as much as 60 percent of traffic passing through core routersis transit traffic bound for other destinations, so the amount of transit traffic in routed net-works is far from trivial.

Core router ports are tremendously expensive, particularly when compared to lower-layerports, such as Ethernet or Layer 1 OTN.

Aside from capex issues, core routers are tremendous opex consumers in terms of phys-ical rack space, power, and management complexity. Some large operators have re-

ported to Heavy Readingthat they are simply running out of physical space to house theirrouters. On power consumption, a core router consumes more power, by far, than anylower-layer network element, giving operators a strong incentive to transport their bits atthe lowest layer possible. Figure 2, from Verizon, illustrates this point.

Figure 2: Power Consumption by Element Type

Source: Verizon, 2009

IP traffic continues to grow at high rates, most likely around 50 percent per year, basedon several industry research sources and anecdotal figures gathered in Heavy Readingnetwork operator interviews. This means that operators are hard pressed to figure outbetter ways to scale their IP networks more efficiently.

New Metro Ethernet Forum (MEF) Ethernet services are growing in popularity globally.These Ethernet services are currently supported over IP/MPLS infrastructure across thecore but it isn't cost-effective to use sophisticated IP/MPLS routers to handle these Layer2/2.5 services.

Operators are seeing increasing demand for wavelength services and high-capacity pri-vate line services supporting applications, such as video transport, data center connectiv-ity, and low-latency transport. Routed networks cannot effectively support these services.



6/14

At the same time, the installed base of Sonet/SDH transport equipment is not up to the task ofscaling the packet network. Sonet/SDH was sufficient when network traffic was primarily low-speed TDM with some Ethernet/IP, but that network model has now been reversed. Networks oftoday and certainly tomorrow are primarily packets with some TDM. TDM may still be growingin incumbent operator networks (particularly high-speed TDM), but the growth in packet traffic isfar greater.

Forcing packets into Sonet/SDH framing is an inefficient means of transport; yet, until recently,operators have had no other choice but to transport packets over Sonet/SDH to use its manage-ment, resiliency, and reliability functions. Early implementations of carrier Ethernet, MPLS, andpure DWDM transport lacked these abilities.

In addition, there is tremendous operator momentum around 100G transport as the next-gen coretransport rate. Yet, the Sonet/SDH standards have been capped at 40G rates (OC768/STM256).Operators migrating to 100G transport must use Ethernet and OTN.



7/14

IV. Operator Requirements for the Next-Generation CoreOperators have different approaches to managing their networks. Regardless of whether operatorskeep packet and transport operations teams separate or converge them into a single group, HeavyReadingbelieves that operators require the following for their next-generation core networks.

Significant increase in packet capacity (on the order of tenfold improvements every fourto five years)

Efficient switching and grooming for both TDM and packet traffic

Greater automation in service and network provisioning and management

Resiliency and reliability to match (or exceed) existing Sonet/SDH network norms

Efficient support of new high-capacity service types

Ability to retain existing Sonet/SDH and core router networks as long as needed

Greater packet intelligence to handle traffic that is predominantly packet



8/14

V. Architectural Approaches Being EvaluatedIn order to address the pain points discussed above and to meet their future requirements, opera-tors are evaluating several different architectures for core transport and switching. Often, coretransport and switching are being evaluated simultaneously. This section provides an overview ofthe primary architecture options for the core.

5.1 Switched Sonet/SDH

Standalone optical crossconnects deployed widely in operator networks today are based onSonet/SDH switching fabrics. These optical crossconnects are effective in networks that are pri-marily Sonet/SDH with some Ethernet, but were not designed for networks that are primarilybased on Ethernet and IP. Limited Ethernet functionality has been added to optical crossconnectsover time, allowing for Ethernet over Sonet/SDH or a small amount of Ethernet switching, but theyfundamentally remain Sonet/SDH network devices. Faced with capacity constraints, operatorscan upgrade these early-generation Sonet/SDH-based optical crossconnects to newer, high-capacity versions, but Heavy Readingviews this more as an interim solution to the packet trafficproblem for networks dominated by packet traffic and packet traffic growth.

5.2 IP over DWDM (IPoDWDM)

The integration of DWDM transponders onto routers has been touted by major IP router suppliersas a means of saving capex spend in the core by eliminating separate DWDM equipment sittingnext to core routers in central offices. The integration of DWDM long reach optics on routers istypically called IP over DWDM (IPoDWDM). While there have been some IPoDWDM deploy-ments over the last few years, Heavy Readingsees this type of architecture as falling short insolving the operator pain points discussed in Section III.

Although IPoDWDM architecture does reduce transponder capex by eliminating the short-reachoptics connection from the router to the DWDM system, disadvantages on capex, opex, and or-ganization levels often outweigh the short-reach optics savings. The major disadvantages ofIPoDWDM architectures are discussed below.

As noted earlier, router ports are expensive when compared to switch or transmissionports. Historically, the decline in cost per bit for router ports has lagged in comparisonwith other networking technologies, notably Ethernet switching and optical transport. Thishigh cost makes the router an excessive cost component for forwarding tandem (transit)traffic across intermediate locations, for example. Thus, for many operators, IPoDWDMsolves the wrong capex problem.

In addition, the IPoDWDM architecture ties the operator to the router vendor for both thecore router and DWDM transport network. Yet DWDM interfaces on routers have not keptpace with overall DWDM transmission advances over the past couple of years. Technicalprogress related to increased data rates and new modulation formats have resulted inexpensive 10G or 40G router interfaces becoming outdated before their investment hasdepreciated to an acceptable level. Future wavelength capacity growth beyond 100G will

only exacerbate this problem. Several operators have told Heavy Readingthat they needto keep their DWDM layer and the core router layers separate specifically for this reason.

Further, the use of alien wavelengths over third-party optical transmission equipment ispossible but it is not recommended for high-performance operation. Alien wavelengthstypically perform at reduced performance levels because transmitter/receiver and linesystem configurations are not optimized. Network locations, such as metros, where longdistance or high capacities are not required, may be suitable for alien wavelengths; but itmust be pointed out that such solutions are not well integrated into a host system's faultmanagement system, and consequently will be more complicated to operate.



9/14

Finally, IPoDWDM poses organizational challenges that are not trivial. Historically, opera-tor organizations have been siloed between transport and data. While there is a cleartrend toward melding these organizations together, we have seen greater acceptancetoward uniting Ethernet and optical transport groups but still keeping IP routing separate.Thus, IPoDWDM forces an organization change that most operators are not yet willing orable to make. In the absence of a unified transport and IP organization, IPoDWDM quick-

ly causes internal conflicts. Who is managing the IPoDWDM devices the transportgroup or the routing group?

5.3 Switched OTN

OTN is an optical transport standard developed by the ITU-T and first standardized in 1998. It isalso known as ITU G.709 and "digital wrapper." In OTN, the ITU defined a payload encapsulation,OAM overhead, forward error correction (FEC), and multiplexing hierarchy. The result is a trans-port standard that includes the benefits of Sonet/SDH (such as resiliency and manageability) butwith improvements for transporting data payloads. OTN was specifically designed to be multiser-vice transport container for any type of service from TDM to packet, and this is one of the tech-nology's strongest value propositions.

OTN has been widely deployed for transport within long-haul networks, particularly because theinherent FEC features enable optical transmission over longer distances. A newer trend we areseeing globally is high operator interest in adding large-scale OTN switching in addition to exist-ing OTN transport to their next-generation networks. In the next section, we explore the benefitsof OTN-switched networks in detail.

5.4 Transport Rate Options

Coupled with the transport and switching technology decisions discussed above, operators mustalso make key decisions about their future transport rates when building out their next-generationcore networks. The options are to continue to base their core networks on 10G or to add 40Gand/or 100G to the network. We discuss these transport rate options briefly below.

Continue to Add 10G Capacity

Core networks have standardized on 10G transport. We believe that 10G will continue to play asignificant role in networks for many years to come, in no small part because they have comedown in price and will continue to decline for many years. However, 10G has been the networkstandard for more than a decade. Many networks that were well-served by 10G capacities at thebeginning of the decade are now straining from years of 50+ percent annual traffic growth. Whilethese operators can continue to add 10G waves to these routes, it becomes operationally com-plex and inefficient to keep doing so (if higher-rate options are available). In addition, growth in10GE enterprise connectivity drives operators that serve these enterprises to look beyond 10Gfor transport of these services.

Upgrade Infrastructure to 40/100G

Operators looking to upgrade their core network capacities now have both 40G and 100G as via-ble options. During the last couple of years, 40G transport has moved from the early-adoptionphase to mass-deployment phase. Heavy Reading counts more than 110 40G deploymentsworldwide as of mid 2010, with operators including AT&T, Verizon, Deutsche Telekom, RelianceGlobalcom, China Telecom, and many other Tier 1 and 2 operators. In July 2010, the IEEE rati-fied the 40G/100G Ethernet standard for client interfaces. Coupled with the ITU-T's recommenda-tion for 100G transport completed at the end of 2009, this means that 40GE and 100GE transportover 40G and 100G DWDM networks is fully standardized. In fact, initial commercial 100G dep-loyments have begun.



10/14

VI. OTN & 40/100G Transport: The Right ApproachAmong the options for core network modernization described above, Heavy Readingsees grow-ing operator momentum globally around the combination of 40G and/or 100G transport coupledwith a large-scale OTN-switched core. Furthermore, we see operators combining these 40/100Gtransport and OTN-switching functions within a single core network element. Transport is no

longer simple point-to-point static capacity augmentation. It is now a switched networking layerwith the flexibility to provide capacity from anywhere to anywhere.

6.1 40G & 100G Core Transport

Although 10G is now a low-cost technology, driven by its maturity and high volumes, for the corein particular many operators around the world have seen an immediate need to move beyond10G. Figure 3 ranks the primary drivers for deploying 100G, based on a Heavy Readingoperatorsurvey of operators around the world. Note that network capacity exhaust ranked second on thelist a key indicator that operators are running out of network capacity.

Figure 3: Drivers for 100G Deployments

RANK DRIVER SCORE

1 Connect 100GE interfaces on core routers 4.02

2 Network capacity exhaust 3.91

3 Better economics compared with 10G 3.90

4 Better economics compared with 40G 3.79

5 Wholesale customer services 3.32

6 Enterprise customer services 3.15

7 OTU4/ODU4 transport services 3.12

N=94Source: The Road to 100 Gbit/s Transport Networks: A Heavy Reading Multi-Client Study, June 2009

Significantly, the operator migration is not just about 40G but also about 100G transport, along-side (and sometimes instead of) 40G transport. Verizon, which became the first operator to com-mercially deploy 100G systems in 2009, is just one example of an operator that is pursuing bothtransport rates.

Given the timing of 40G and 100G and the restraints on carrier spending, a rip-and-replace ap-proach to 40G and/or 100G is not an option. Not surprisingly, in the same Heavy Readingopera-tor survey referenced above, operators listed the need to interoperate with existing DWDM linesystems as the top operational requirement in choosing a 100G system. 100G (and 40G) must beable to work on the existing fiber plant and the existing 10G line system.

Modulation techniques based on dual-polarization quadrature phase-shift keying (DPQPSK),combined with digital manipulation of bits through coherent receivers and digital signal proces-sors (DSPs), are enabling 40G and 100G transmission on existing networks built for 10G trans-mission with performance that closely matches that of 10G. For example, at the 2010 OFC confe-rence, Ciena demonstrated 40G ultra-long-haul transmission at 3,100 km and 100G transmissionat 1,500 km, all using commercially available products. In addition, DSPs combined with electron-ic dispersion compensation eliminates the need for dispersion compensating fibers, which yieldtremendous cost savings, particularly in ultra-long-haul and undersea networks. The technologyalso reduces the number of amplifiers in the network and allows 40G/100G to work with existingamplifiers deployed. Eliminating dispersion compensating fibers is also important for reducinglatency, an increasingly important consideration for enterprises, particularly in finance.



11/14

A final key point on higher-speed transport: Line-side migration (i.e., the DWDM equipment) doesnot mandate a client-side migration (i.e., the router equipment). In other words, operators will mi-grate their core DWDM networks to 40G and 100G but will continue to offer 10G services to cus-tomers and interface to 10G ports on routers for a long time to come. Here, switched OTN, withits ability to groom traffic to efficiently pack pipes, will be a critical requirement. Muxponders (suchas 4x10G) will also be used for point-to-point connections, though such non-switched connectionsare far less efficient at using available bandwidth when compared to mesh-switched OTN.

Figure 4 shows operator expectations for client interfaces in the long-haul core for 2013, basedon Heavy Reading's operator survey.

Figure 4: Expected Client Interface Mix for 100G Core Networks in 2013

N=94Source: The Road to 100 Gbit/s Transport Networks: A Heavy Reading Multi-Client Study, June 2009

6.2 OTN Switching

OTN in the core makes sense for several reasons:

Operators envision OTN as a universal transport protocol for all service client types

OTN is designed to transmit 10GE, 40GE, and 100GE transparently

Newer ODU0 and ODUflex standards extend OTN to further sub-wavelength granularityfor Gigabit Ethernet and to non-traditional data rates, such as arbitrary-rate Ethernetpayloads and Fibre Channel variants

OTN inherently incorporates FEC technology, which is key for transmitting at long-hauland ultra-long-haul distances, as needed in core networks

OTN provides the required OAM functionality that many operators require for transportingGE and 10GE traffic over long distances.

(Note that several of these arguments also apply to metro networks, and Heavy Reading is see-ing growing interest among operators to add OTN to the metro. However, we limit the scope ofthis paper to core network modernization only.)

Traditionally, operators have relied on Sonet/SDH to provide electrical layer functions of sub-wavelength level grooming, performance monitoring, management, maintenance, protection, andrestoration. All of these functions are key requirements for delivering true carrier-class services.



12/14

DWDM does not have these capabilities, as these functions can only be done electrically or, atleast, are best done electrically. To date, operators have relied primarily on Sonet/SDH to performthese functions, even when the traffic being carried is packet. Thus, operators will transport pack-et over Sonet/SDH over a wavelength, with the Sonet/SDH providing the grooming (assumingoptical crossconnects are used), OAM, protection, and restoration. For one, OTN is asynchron-ous, and so does not require the costly and complex timing that Sonet/SDH requires. Yet, OTN isa transparent protocol that can carry asynchronous traffic, such as GE and synchronous traffic,such as Sonet/SDH, and even a mixture of synchronous and asynchronous, without interferingwith the OAM of the client traffic itself.

Finally, compared to Sonet/SDH, OTN is better matched to WDM rates, as well as high-speedEthernet rates, from GE through 100GE. OTN's current defined rates are:

OTU1: 2.7 Gbit/s, for OC48/STM16 transport

OTU2: 10.7 Gbit/s, for OC192/STM64 and 10GE transport

OTU3: 43 Gbit/s, for OC768/STM256 and 40GE transport

OTU4: 112 Gbit/s, for 100GE transport

ODU0: 1.25 Gbit/s, designed to combining multiple GE streams within the OTU-N con-tainers listed above.

Furthermore, newer OTN standard functionality called ODUflex matches OTN containers to anypayloads in 1.25-Gbit/s increments (i.e., n x 1.25 Gbit/s), including fitting unusual rates such as3G HD-SDI, 4G Fibre Channel, 5G InfiniBand, variable rate Ethernet, and 16G Fibre Channel (inthe future).

Thus, with OTN operators are finally able to replace Sonet/SDH with a new universal transportprotocol that is better tuned to packet data and DWDM. Figure 5 illustrates the concept of OTNas the new universal transport protocol.

Figure 5: The OTN Universal Transport Network

Source: Ciena and Heavy Reading, 2009

Offloading routers of transit traffic by using WDM and OTN enables operators to reduce capexwhile freeing up router capacity to handle future growth in IP traffic. It also reduces overall net-



13/14

work delay, which is added every time traffic has to pass through an intermediate router. Theadded delay can be a significant problem for operators, particularly in delay-sensitive video andgaming networks. Finally, router offload brings benefits in reducing packet loss and increasednetwork availability. All router networks struggle to match the five nines (sometimes six nines)availability of an optical transport network.

Figure 6 illustrates in simple terms the new architecture and interconnectivity of a convergedLayer 0-2 (40/100G) infrastructure providing connectivity to Layer 3 network (10G evolving to100GE), and including non-routed services.

Figure 6: Core Modernization Network Diagram



14/14


VII. ConclusionThe core network has become a focal point for network modernization as continued growth in IPtraffic has led to proliferation of core routers and core router ports processing IP traffic routersthat are expensive in terms of both capex and opex.

The primary core modernization options available to operators are: to continue to invest in theirswitched Sonet/SDH networks by adding more capacity; to build out IPoDWDM networks usingintegrated optics on routers; or to deploy switched packet-optical/OTN networks. Of these op-tions, Heavy Readingsees switched OTN as the best network upgrade approach, particularly fornetwork operators that have large installed bases of Sonet/SDH equipment and services that willneed to be accommodated for many years to come, even as they invest in Ethernet and IP.

Switched OTN uniquely combines the "carrier-class" transport qualities of Sonet/SDH with theflexibility to efficiently handle Ethernet transport from GE through 100GE when combined with40G and/or 100G transport. With switched OTN and 40G/100G transport in the core, operatorswill finally be able to replace Sonet/SDH with a new transport protocol that is tuned to Ethernetand high-speed DWDM, while simultaneously supporting the legacy Sonet/SDH network and ser-vices for as long as needed.

Documents

Ciena Core Modernization WP