Upload
chika-yoshimura
View
298
Download
0
Embed Size (px)
Citation preview
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
“ OCN Experience to Handle the Traffic Growth and the Future “
Takeshi Tomochika <takeshi.tomochika(at)ntt.com> Chika Yoshimura <chika.yoshimura(at)ntt.com>
NTT Communications, OCN
23th February 2011
1
Copyright © 2011 NTT Communications Corporation. All Rights Reserved. 2 2
Background
• Internet traffic is growing more and more • One of the most important missions of ISPs
- to carry the traffic with stability & without any congestion
• Making the backbone robust
• We are talking about: - current traffic situation in Japan - issues at OCN when designing the backbone network - future visions
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
Agenda
1. Current situation of Internet traffic in Japan
2. What is OCN? 3. Current issues we are facing 4. Future visions 5. Wrap up
3 3
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
879.6
1234
689.5
860.8
319.7
425.4 469.1
523.6 636.6 721.7
812.9
988.4
1453.7
1362.9
567.5 512.5
468.7 400.5
354.3 320.2 278.8
872.4
631.5
943.4
0
200
400
600
800
1000
1200
1400
1600
Tota
l A m
ount
of Tra
ffic (
Gpb
s)
Internet Traffic Trend in Japan
2004 2005 2006 2007 2008 2009 2010
Download
Upload
17.8%
7.5%
4 * Average traffic
source: Internet Traffic Trends in Japan ( MIC* ) MIC: Ministry of Internal Affairs and Communications
• Total amount of Broadband Traffic is 1.46Tbps (Download) - 17.8% growth compared to last year
• Upload traffic decreased over the last half year (872Gbps)
Copyright © 2011 NTT Communications Corporation. All Rights Reserved. 5 5
• Traffic volume per subscriber growing - 16kbps (in 2004) -> 45kbps (in 2010)
Internet Traffic Trend in Japan (cont.)
source: Internet Traffic Trends in Japan ( MIC* ) MIC: Ministry of Internal Affairs and Communications
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
0
5,000,000
10,000,000
15,000,000
20,000,000
25,000,000
30,000,000
35,000,000
subscribers
Total 19,557,100 23,301,100 25,744,769 28,303,003 30,115,989 31,709,116 32,040,792
D SL 13,675,800 14,517,800 14,235,925 13,133,113 11,601,734 10,134,491 9,735,140
C ATV 2,959,710 3,309,480 3,565,427 3,827,417 4,083,072 4,300,594 4,352,878
FTTH 2,896,930 5,457,690 7,931,837 11,329,886 14,418,215 17,195,696 17,788,535
2004 2005 2006 2007 2008 2009 2010
source: Ministry of informa1on and Communica1ons Sta1s1c Database
FTTH
DSL
Broadband subscribers in Japan
6 6
• Growing Broadband subscribers • Shifting from DSL (metal) to FTTH (optical fiber)
Internet Traffic Trend in Japan (cont.)
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
Overview
• Internet traffic in Japan has been growing consistently
• Traffic will keep rising in the future
- ISPs have to … • design a robust backbone network to deal with the situation
• How backbone we have been making? • How bandwidth we have?
7
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
Agenda
1. Current situation of Internet traffic in Japan
2. What is OCN? 3. Current issues we are facing 4. Future visions 5. Wrap up
8 8
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
67% 54% 100% 100% 100%
NTT EAST NTT WEST NTT DATA
NTT Holding
: Domes(c Open IP Service Brand
NTT Communications
-‐ Domes&c Long Distance -‐ Interna&onal -‐ Internet
-‐ Domes&c Local (Eastern Japan)
-‐ Domes&c Local (Western Japan)
-‐ Systems Integra&on
-‐ Mobile
: Interna(onal Open IP Service Brand
Outline of NTT Group
h9p://www.hk.n9.com/en/index.html 100%
NTT docomo
100% h9p://www.hknet.com/en/home/index.htm
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
Houston
Australia
Taiwan Korea
Indonesia
Dusseldorf
Madrid
US Backbone
EUROPE Backbone
Equinix San Jose
NASA
PAIX
Equinix Chicago
Equinix Ashburn
Equinix Dallas
DE-CIX
AMSIX PARIX
ESPANIX
LINX
Philippines
Malaysia
Vietnam
India
Thailand
Geneva
Paris Frankfurt
Singapore
Equinix New York
London
HK IX
Atlanta Amsterdam
OCN愛知 OCN Aichi
OCN Kyoto
OCN Osaka
OCN Hyogo
OCN Hiroshima
OCN Fukuoka
OCN Hokkaido
OCN Kanagawa
OCN Miyagi
OCN Chiba
OCN Tokyo
OCN Tokyo west
160 Gbps
20 Gbps
20 Gbps
400 Gbps
20 Gbps
60 Gbps
120 Gbps
60 Gbps
1Gbps NSPIXP-3
40Gbps JPNAP Osaka
130Gbps JPNAP
10Gbps dix-ie
Osaka GW
ICP
20 Gbps
80 Gbps
440 Gbps
20 Gbps
China
Hong Kong
San Jose
Seattle
Chicago
New York
Washington D.C.
Dallas
Miami
Los Angeles
Osaka
Osaka Core
Aichi Core
Gunma Core
Tokyo Core
Tokyo GW
ASIA Backbone
Tokyo
国内接続 Domestic External 1Gbps
Internet Multifeed
NTT Communications’ IP Backbone Network
November 2010
340 Gbps
200 Gbps
200 Gbps
200 Gbps
200 Gbps
380 Gbps
170 Gbps
220 Gbps
OCN Backbone
ntt.net
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
0
100
200
300
400
500
600
700
800Oct-97
Oct-98
Jul-99
Jan-00
Jul-00
Jan-01
Jul-01
Jan-02
Jul-02
Jan-03
Jul-03
Jan-04
Jul-04
Jan-05
Jul-05
Jan-06
Jul-06
Dec-06
Aug-07
Dec-07
Jun-08
Nov-08
May-09
Nov-09
May-10
Nov-10
Gbps
between east and west
ntt.net
IX
Bandwidth history of OCN
11
Jul 2001 Started OCN IPv6
Tunnel Commercial Service
Mar 2003 installed 10G-IF in the backbone
Feb 2010 300G between
Japan to the U.S. (ntt.net)
Jan 2011 400G between
Japan to the U.S. (ntt.net)
Feb 2007 100G between
Japan to the U.S. (ntt.net)
Nov 2001 Started
FTTH service
Jan 2001 Started OCN ADSL
Service
Dec 1999 Started OCN IPv6
Tunnel Trial Service
Dec 1996 Started OCN
×2400
×5800
×14000
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
• Make our network larger and larger as Internet traffic grows
• Issues we have been facing
• Efforts we have been making
12
What we are doing
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
Agenda
1. Current situation of Internet traffic in Japan
2. What is OCN? 3. Current issues we are facing 4. Future visions 5. Wrap up
13 13
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
1. Scalability of Router Forwarding Tables
2. Link Aggregation
14
The issues we are facing
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
FIB table of OCN • FIB(Forwarding Information Base) table has been growing
• Causes of growing FIB 1. BGP full routes (more than 340,000 in February 2011) 2. Prefixes with no-export 3. ECMP, {i, e} bgp-multipath
source: http://bgp.potaroo.net/
Growth of the BGP Table - 1994 to Present
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
Scalability of Router Forwarding Tables
• When a rerouting event occurs, potentially thousands of routes must be updated
• It took a lot of time to converge the routes
FIB of router-A prefix output interface(s)
10.1.0.0/16 IF-1 IF-2
LAG-3(IF-4, 5)
10.2.0.0/16 IF-1 IF-2
LAG-3(IF-4, 5)
IF-1
10.1/16 10.2/16…
LAG-3 (IF-4,5)
IF-2
router A
router B router C router D
×
×
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
• We were facing a problem: - OSPF neighbor went down due to FIB table convergence
• Between router A and B - Link Aggregation (LAG) had been enabled (minimum-links = 1) - OSPF neighbor had been connected through the LAG interface
• When all member-links but one had been to make disabled
- We had expected the OSPF neighbor to remain up
17
router A
router B
member-link 1
Link Aggregation
member-link 2(down)
member-link 3(down)
member-link 4 (down)
member-link 5 (down)
member-link 6 (down)
Scalability of Router Forwarding Tables
OSPF neighbor
went down
Copyright © 2011 NTT Communications Corporation. All Rights Reserved. 18
• What happened?
router A
router B
member-link 1
Link Aggregation Interface
member-link 2
member-link 3
member-link 4
member-link 5
member-link 6 × disable
(1) Router A detected the interfaces were down
(2) Router A started updating FIB
(3) Router A finished updating FIB
(4) Router A chose another interface to send OSPF hello
OSPF hello more than OSPF dead-timer
Router-A could not send any OSPF hello packets during (1) – (3), then the neighbor went down
Scalability of Router Forwarding Tables
OSPF hello
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
• Hierarchical FIB - Cisco: BGP Prefix Independent Convergence(PIC) - Juniper: indirect-nexthop For more information: BGP Convergence in much less than a second http://www.nanog.org/meetings/nanog40/presentations/ClarenceFilsfils-BGP.pdf
• Fewer routes to be updated
• Improving the route convergence time
Scalability of Router Forwarding Tables
19
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
Link Aggregation Issues
20
• A lot of Link Aggregation 10GE Interfaces in the backbone
OCN愛知 OCN Aichi
OCN Kyoto
OCN Osaka
OCN Hyogo
OCN Hiroshima
OCN Fukuoka
OCN Hokkaido
OCN Kanagawa
OCN Miyagi
OCN Chiba
OCN Tokyo
OCN Tokyo west
160 Gbps
20 Gbps
20 Gbps
400 Gbps
20 Gbps
60 Gbps
120 Gbps
60 Gbps
1Gbps NSPIXP-3
40Gbps JPNAP Osaka
130Gbps JPNAP
10Gbps dix-ie
Osaka GW
ICP
20 Gbps
80 Gbps
440 Gbps
20 Gbps
Osaka Core
Aichi Core
Gunma Core
Tokyo Core
Tokyo GW OCN Backbone
国内接続 Domestic External 1Gbps
Internet Multifeed
340 Gbps
200 Gbps
200 Gbps
200 Gbps
200 Gbps
380 Gbps
• Traffic balance issues (Traffic Polarization) • Operation issues • Other issues
20
Copyright © 2011 NTT Communications Corporation. All Rights Reserved. 21
LAG Issues ~traffic balance issues (1/3)~
• Traffic balance in the LAG(1) – Can’t use per-packet round-robin
• Simple round-robin bring about packet reordering in a flow – Hashing algorithm: calculate the hash value based on the packet information (IP
address, MAC address, and etc.) to decide Output I/F
• Traffic are distributed per flow using the hash values
Ø Issue 1: traffic-unbalance by variation of flow
4 links
Packet
IP MAC Payload
to LAG-A calcurate the hash
hash=1 → Out I/F=e1 hash=2 → Out I/F=e2 ・・・・・・
LAG-A
Flow
Packet
LAG
Copyright © 2011 NTT Communications Corporation. All Rights Reserved. 22
• Traffic balance in the LAG(2) – Issue 2: The less # of hash elements, the worse traffic-balanced
Ø as a result, less effective use of bandwidth
5 links LAG 4 links LAG 3 links LAG
IF#1 H1、H6 IF#2 H2、H7 IF#3 H3、H8 IF#4 H4 IF#5 H5
IF#1 H1、H5 IF#2 H2、H6 IF#3 H3、H7 IF#4 H4、H8
IF#1 H1、H4、H7 IF#2 H2、H5、H8 IF#3 H3、H6
2:2:2:1:1
10+10+10+10*1/2+10*1/2=40
2:2:2:2
10+10+10+10=40 3:3:2
10+10+10*2/3=26.7
Traffic cannot be evenly distributed due to the hash mechanism
e.g.: Traffic balance in a LAG when # of hash elements is 8
Only use 40G / 50G
←Traffic balance ratio ←Effective bandwidth in the LAG
Only use 27G / 30G
LAG Issues ~traffic balance issues (2/3)~
22
Copyright © 2011 NTT Communications Corporation. All Rights Reserved. 23
cf. Difference in traffic balance by # of hash elements
5 links LAG 4 links LAG 3 links LAG
IF#1 H1,H6 IF#2 H2,H7 IF#3 H3,H8 IF#4 H4 IF#5 H5
IF#1 H1,H5 IF#2 H2,H6 IF#3 H3,H7 IF#4 H4,H8
IF#1 H1,H4,H7 IF#2 H2,H5,H8 IF#3 H3,H6
40 40 26.7
5 links LAG 4 links LAG 3 links LAG
IF#1 H1,H6,・・・H26,H31 IF#2 H2,H7,・・・H27,H32 IF#3 H3,H8,・・・H28 IF#4 H4,H9,・・・H29 IF#5 H5,H10,・・・H30
IF#1 H1,H5,・・・H29 IF#2 H2,H6,・・・H30 IF#3 H3,H7,・・・H31 IF#4 H4,H8,・・・H32
IF#1 H1,H4,・・・H28,H31 IF#2 H2,H5,・・・H29,H32 IF#3 H3,H6,・・・H30
7:7:6:6:6
10+10+10*6/7+10*6/7+
10*6/7=45.7
8:8:8:8
10+10+10+10=40
11:11:10
10+10+10*10/11=29.1
A large number of hash elements is better
LAG Issues ~traffic balance issues (2/3)~
←Traffic balance ratio ←Effective bandwidth in the LAG
e.g.1: Traffic balance in a LAG when # of hash elements is 8
e.g.2: Traffic balance in a LAG when # of hash elements is 32
23
Copyright © 2011 NTT Communications Corporation. All Rights Reserved. 24
• Traffic balance by ECMP (Equal Cost Multi Path) and LAG: Case1
– If calculation logic of LAG is the same as ECMP’s, it will bring about unbalanced traffic in physical links
LAG1
LAG2
calc. for ECMP①
calc. for LAG①
unbalanced (in LAG)
evenly balance (ECMP)
same logic
Some routers have the same calculation logics for ECMP and LAG as a default
LAG Issues ~traffic balance issues (3/3 - 1)~
flow 1 flow 2
flow 3 flow4
flow 1 flow 2 flow 3 flow 4
24
no traffic
no traffic
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
• Traffic balance by ECMP and LAG : Case2 – If calculation logic of ECMP is the same as that of previous ECMP, it will
bring about unbalanced traffic
25
LAG
LAG
LAG Issues ~traffic balance issues (3/3 - 2)~
calc. for ECMP①
calc. for ECMP①
calc. for LAG②
*change*
same logic
unbalanced (ECMP)
flow 1
flow 2
25
no traffic
no traffic
flow 1 flow 2 flow 3 flow 4
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
• Traffic balance by ECMP and LAG : Case3 – If calculation logic of LAG is the same as that of ECMP at the previous node, it
will bring about unbalanced traffic
26
LAG
LAG
calc. for ECMP①
calc. for ECMP③
same logic
calc. for LAG②
unbalanced (in LAG)
calc. for LAG① *change*
Need to consider balance logics, network topology, configurations
LAG Issues ~traffic balance issues (3/3 - 3)~
flow 1
flow 2
26
unbalanced (in LAG)
flow 5
no traffic
no traffic
flow 1
flow 2 flow 3 flow 4
flow 5
* Some latest routers can include a router-ID in the seed of hash to avoid case 2,3
Copyright © 2011 NTT Communications Corporation. All Rights Reserved. 27
• LAG operation (1)
– In the case of silent-failure, traffic through the fault link will drop
– LACP (Link Aggregation Control Protocol) ・ Sending and receiving control frames in physical links
・Attention to Interoperability
– BFD Per Member Link (Bidirectional Forwarding Detection)
transmission device
LAG Issues ~operation issues (1/3)~
Router Router LAG-I/F
27
Copyright © 2011 NTT Communications Corporation. All Rights Reserved. 28
• LAG operation (2)
– Switching policy of LAG-I/F • minimum-link (trunk-threshold) • threshold whether LAG-I/F is up or down
• This switching policy is important for effective use of LAG
• consider the entire network topology
e.g.: minimum-link when the policy is 70% in LAG
(1)Normally, packets are forwarded to all the link-up I/Fs
(3) LAG I/F goes down, and traffic move
minimum-link = 3
LAG Issues ~operation issues (2/3)~
28
# of links in LAG 3 4 5 6 7 8 9 10
minimum-link 3 3 4 5 5 6 7 7
LAG LAG
(2) still LAG is up, as # of up-links is not less than 3
Copyright © 2011 NTT Communications Corporation. All Rights Reserved. 29
• LAG operation (3)
– Ping for test • Packet goes through only one physical
interface • Need to test each interface with letting the
rest go down • expect Ethernet OAM
LAG Issues ~operation issues (3/3)~
29
Copyright © 2011 NTT Communications Corporation. All Rights Reserved. 30
• Limitations on # of links in a LAG • Issues of physical wiring
– Increased # of physical links -> Complicated maintenance
• Need a well-thought-out plan for LAG – How to assign physical links to Line Cards
• based on redundant policy • MTBF for each part
• Cost
– e.g. Policy 1: keep LAG-I/F up as much as possible • assign each physical link to each LC, minimum-link = 1
– e.g. Policy 2: Switching traffic to the other LAG immediately • assign all physical links to one LC, minimum-link = # of links
– e.g. Policy 3: Between policy 1 and policy 2
• LAG is troublesome
NOTE: this is NOT NTT Communications’ equipment
LAG Issues ~other issues~
30
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
Agenda
1. Current situation of Internet traffic in Japan
2. What is OCN? 3. Current issues we are facing 4. Future visions 5. Wrap up
31 31
Copyright © 2011 NTT Communications Corporation. All Rights Reserved. 32
How to handle traffic growth
32
(1) Change Network • new routers, new switches • new router-interface (100GE)
(2) Control Traffic • Cache Servers • CDN
Copyright © 2011 NTT Communications Corporation. All Rights Reserved. 33
Expectation for 100GE
• Need 100GE I/F – Bandwidth over 1Tbps – LAG is troublesome
• Request – Lower price
• CFP is expensive • LR10
– Support long-distance transmission (ER4) – Higher Capacity
• Capacity per chassis will be decreased when migrating from 10GEs to 100GEs in some current routers
– LAG of 10GE and 100GE simultaneously
– Interoperability, 100GE LAG, Ether OAM – Next step: 400GE, 1T Ether
33
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
The Best award of Interop Tokyo for 2 years in a row
2009 100GE-SR10 demonstrated with transmission equipment and traffic generator 2010 100GE transmission network (100GE-LR4) was provided for practical operation
2010/6/8 News Release
NTT Com, Infinera and Ixia to Provide World’s First Practical 100 Gbps Ethernet Interconnection at Interop Tokyo 2010
Our preparation for 100GE
34
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
transit peer
users
(1) transit/peer sides
(2) backbone
(3) edge sides
Cache servers • Legal changes in January 2010 in
Japan - became legal to install cache servers
without permission(s)
• Demerits not to install cache servers - Streaming delays because of carrying the
traffic from peer/transit network - more bandwidth - costs for transit network
• Merits to install cache servers - Transit cost saving, - Bandwidth saving - Fixing delay
• Issues 1. Equipment performance (cache hit ratio,
lack of bandwidth… ) 2. Where to place 3. When equipment failure
35
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
Agenda
1. Current situation of Internet traffic in Japan
2. What is OCN? 3. Current issues we are facing 4. Future visions 5. Wrap up
36 36
Copyright © 2011 NTT Communications Corporation. All Rights Reserved.
Wrap up
• The total traffic in Japan has been consistently increasing. - The traffic will keep growing in the future.
• We are continuing to design a strong backbone network. - But we have some designing/operational issues
• We are going to need 100GE in the near future to deal
with the situation. • How is your network? Do you have any ideas or
suggestions to cope with the expected growth of traffic in the future?
37