Analysys Mason International IP Interconnection 23 Feb 2011

8/13/2019 Analysys Mason International IP Interconnection 23 Feb 2011

1/62

Overview of recent changes in the IPinterconnection ecosystem

Michael Kende, Partner

23 January 2011

Ref: 17038-075

.


2/62

Overview of recent changes in the IP interconnection ecosystem

Ref: 17038-075 .

Contents

1 Executive summary 11.1 General international trends 11.2 Globalization of the use of Internet Exchange Points 11.3 Other significant trends 31.4 Conclusion 4

2 Introduction 5

3 Overview of the IP interconnection ecosystem 83.1 The traditional Internet ecosystem 8

3.2 Introduction to Internet Exchange Points 15

4 General international trends 194.1 Increase in access 194.2 Increase in content 214.3 Conclusion 23

5 The globalization of the use of IXPs 255.1 Evolution of the topology of the Internet 255.2 Evolution of the role of IXPs 36

5.3 Conclusion 39

6 Other significant trends 416.1 The value of Internet content and applications has dramatically increased 426.2 Bandwidth-intensive types of Internet traffic have become predominant 436.3 Cloud-computing services have become widely utilized 476.4 Internet connection has become a key feature of many new devices and services 486.5 Conclusion 50

7 Conclusion 52

Annex A: Glossary of terms

Annex B: About Analysys Mason

Annex C: Authors and acknowledgements


3/62

Overview of recent changes in the IP interconnection ecosystem

Ref: 17038-075 .

Copyright 2011. The information contained herein is the property of Analysys MasonLimited and is provided on condition that it will not be reproduced, copied, lent ordisclosed, directly or indirectly, nor used for any purpose other than that for which it wasspecifically furnished.

Analysys Mason Limited818 Connecticut Avenue NWSuite 300Washington DC 20006USATel: (202) 331 3080Fax (202) 331 [email protected]

www.analysysmason.com

Registered in England: Analysys Mason LimitedBush House, North West Wing, AldwychLondon WC2B 4PJ, UKReg. No. 5177472


4/62

Overview of recent changes in the IP interconnection ecosystem | 1

Ref: 17038-075 .

1 Executive summary

2010 saw the fifteenth anniversary of the commercialization of the Internet backbone as we knowit today, when the US National Science Foundation Network (NSFNET) Backbone Service (overwhich networks exchanged traffic) was decommissioned in favor of the current commercialInternet backbone. When it was closed, four Network Access Points (NAPs) across the US weredesignated as the points at which traffic could be exchanged by the new commercial backbones,and these became the hub for Internet interconnection in the early years of the commercialInternet, when the Internet itself was very US-centric.

But this situation has evolved significantly over the past 15 years: there have been major changesin the Internet ecosystem, having corresponding impacts on interconnection arrangements. This

paper focuses on two of the main evolutions in the Internet ecosystem, namely the globalization ofthe Internet as usage and content spread around the world, and the enormous increase in overall

Internet traffic resulting from ever richer content and applications. This paper also describes indetail how interconnection arrangements evolved in response to these trends, and explains in

particular how Internet Exchange Points (IXPs) have helped to promote efficient interconnection between all of the players in the Internet ecosystem.

1.1 General international trends

Two main underlying trends have marked the development of the Internet over the past 15 years:

Globalization of Internet traffic

In the early years, Internet infrastructure was primarily located in the US,and therefore usage and traffic exchange were also focused on the NorthAmerican region. But Internet development has relatively quickly reachedother markets, first in developed countries of Western Europe, then in fast-growing countries such as those in South-East Asia, and more recently inthe developing countries in most parts of the world. Thus Internet usage hasgrown exponentially while it has globalized.

Increase in quantityand quality of Internet content

This growth of Internet users has fueled an increase in the content availableon the Internet, as the increase in users globally is indivisible from theincrease in locally-generated content. Also, Internet content has

progressively increased in value and volume, thanks to an improved qualityof service, substantial investments made in Internet backbone and accesscapacity, and the development of premium content.

1.2 Globalization of the use of Internet Exchange Points

Historically, most Internet traffic transited through US-based NAPs in which backbonesinterconnected their networks and exchanged traffic. As all ISPs had to connect to the US for


5/62


Ref: 17038-075 .

international transit, they also used these links to exchange domestic and regional traffic, an effectreferred to as tromboning . Tromboning involves the carriage of traffic on unnecessarily long andtechnically sub-optimal routes, and this adversely impacts both transit costs and network

performance. As a result and because of the globalization trends described above tromboning

became a significant issue for ISPs.

To remedy the situation, Internet market players began to create more interconnection facilitiesacross a range of countries and regions, in the form of IXPs. These were designed for Internet

players (including ISPs and content providers as well as backbones) to peer and/or to sell or purchase transit over direct cross-connections and deliver connectivity closer to the end users,while also accommodating the increases in the volume of content.

Following the initial successes of several IXPs in Europe, the expansion of IXPs has acceleratedand globalized. To summarize, the evolution of the Internet interconnection infrastructure can be

split into three geographical phases:

1. In the first phase, exchange of international Internet traffic was concentrated in the US (US-centric).

2. In the second phase, traffic exchange migrated to developed countries in Europe and Asia thatform the core of the OECD (OECD-centric). Between 2001 and 2003, the number of IXPs inEurope doubled to reach more than 80, representing more than half of the total number ofIXPs worldwide. 1

3. The third phase of evolution taking place now focuses on the rest of the world (ROW-centric),and is moving towards a global Internet in which regional Internet Exchange Points are usedfor local traffic.

Two main reasons explain the rationale for the fast development of regional IXPs that has beenobserved in the past 15 years. First, economic factors : peering at a national IXP allows for theefficient exchange of local traffic, and the reduced use of tromboning translates into a reduction in

payments to upstream transit providers in other countries. As the price of regional connectivity felldramatically following liberalization in OECD countries in particular, intra-regional trafficexchange became even more affordable. The second factor relates to network performance :

by exchanging traffic at a nearby IXP, ISPs were able to reduce the traffic route distances and thusthe latency issues experienced by end users, compared to a situation where traffic has to passthrough one or more backbones for intra-regional traffic or locally hosted content.The multiplication of interconnection points also improved the resiliency of the interconnectednetworks.

Some of the new IXPs experienced a tremendous success, becoming hubs for international contentand connectivity for entire regions. Such successes are not only beneficial to national and regional

1 Source: https://www.euro-ix.net/news/meetevent/napla_2008.pdf
https://www.euro-ix.net/news/meetevent/napla_2008.pdfhttps://www.euro-ix.net/news/meetevent/napla_2008.pdf


6/62


Ref: 17038-075 .

Internet connectivity, bringing enhanced Internet performance and increased usage, but also benefit the hosting countries as a whole through additional foreign investments, the developmentof local content, and other indirect economic benefits.

The reasons for these successes are multiple, including open access policies at internationalgateways, and facilitated conditions for foreign carriers and content providers to connect into theIXP and sell their services. More generally, it is useful if reforms occur along the lines of the WTOReference Paper on Basic Telecommunications relating to establishing an independent regulatorand providing for transparent interconnection, in particular in facilitating internationalconnectivity. In addition, lowering the barriers to entry, both with respect to foreign directinvestment and fewer regulatory barriers relating to requirements such as licensing, will increasethe competitiveness of the sector. 2

1.3 Other significant trends

In developed countries, the evolution of the Internet ecosystem continues due to several trends thatare all interrelated, but that may be listed as follows:

The value of Internet content hasdramaticallyincreased

Content has gained in value thanks to an improved quality of access and thecorresponding availability of premium content. As a result, content

providers are more and more able to leverage the value of their content inorder to reduce the cost of delivering their traffic, and money flows are nolonger directed mostly to ISPs and backbone providers for Internet accessand traffic.

Bandwidth-intensive types oftraffic have become

predominant

Bandwidth-intensive types of traffic such as video are experiencing amassive growth, with a major impact on bandwidth requirements. There isalso a significant impact on the direction of traffic flows: an increasingamount of traffic is being downloaded from centralized sites such asYouTube, while at the same time peer-to-peer services result in significanttraffic between end users.

Cloud computing

services arebecoming widelyutilized

Cloud computing services, ranging from online email to business

applications, have been embraced as barriers to their adoption have been progressively lifted, thanks to the improvement of Internet access, thedevelopment of security safeguards, and the resulting increase in awarenessand confidence in cloud computing. This ever-increasing usage of cloudcomputing services make consumers more sensitive to latency and otherquality measures that impact access to their cloud applications.

2 Similar propositions have been developed by the International Chamber of Commerce and can be found in more detail in Telecoms

liberalization, an international business guide for policy makers, October 2007

(http://www.iccwbo.org/uploadedFiles/TELECOMS%20LIBER-edition%20Final.pdf)
http://www.iccwbo.org/uploadedFiles/TELECOMS%20LIBER-edition%20Final.pdfhttp://www.iccwbo.org/uploadedFiles/TELECOMS%20LIBER-edition%20Final.pdf


7/62


Ref: 17038-075 .

Internet connectionhas become amandatory feature

for many new

consumer devices

Many new electronic devices have emerged that incorporate Internet access:examples include mobile devices with features adapted to data traffic;netbooks and tablets designed to enable Internet access to cloud computingapplications; and new video-streaming devices that rely on broadband. As a

result, many traditional communications services have migrated to theInternet, as exemplified by services such as TV over IP and Voice over IP.

These trends highlight that the need to establish local or regional traffic and content hubs will notabate in emerging markets, and indeed this need is likely to be exacerbated by advances in theavailability of mobile and fixed broadband. In particular, these trends will magnify the need forincreased bandwidth and decreased latency both factors that will require regions to establishIXPs and caching techniques that keep more traffic within the region in order to control cost andheighten performance.

1.4 Conclusion

The globalization of users and content, the increase of high-bandwidth access and multimediacontent, and other interrelated trends that are observed in developed countries (but are bound toexpand to the rest of the world over time) are fueling the growing needs for the development andglobalization of IXPs. This evolution seems essential, in order to control the correspondingincrease in costs and concerns over latency.

This highlights the need for emerging countries to support the development of such Internationalconnectivity solutions which are key to sustaining and expanding a healthy Internet ecosystem,and to build an environment favorable to investment in infrastructure and services.


8/62


Ref: 17038-075 .

2 Introduction

This paper marks the 15th anniversary of the commercialization of the Internet backbone as weknow it today. During that time, there have been significant changes in the Internet ecosystem, andwe focus in this paper on two of these changes. First came the expanded globalization of traffic

beyond North America, beginning with users and then content. Second came an enormousincreases in traffic , resulting not just from the increase in the number of users, but also from theincreased access bandwidth as users migrated to broadband, as well as the correspondingexplosion of multimedia content that became available. These changes were supported by ongoingchanges in the location of Internet Exchange Points (IXPs) that have helped to promote efficientinterconnection between all of the players in the Internet ecosystem.

On April 30, 1995, the US National Science Foundation Network (NSFNET) Backbone Servicewas decommissioned in favor of the current commercial Internet backbone. The NSFNEToperated a national backbone ne twork allowing regional networks to connect at supercomputingsites. 3 At this time, the objective was to create an open network linking academic researchers andallowing them access to distant supercomputers over the network at no cost. When the NSFNETnetwork went online in 1986 and allowed commercial traffic to be carried (and not just NSFNETtraffic) it consisted of six sites interconnected with leased 56kbit/s Digital Data System (DDS)links. This was upgraded to 45Mbit/s links in 1991.

At the time that the NSFNET was decommissioned, in 1995, Internet access was predominantlydial-up, and in terms of content, email was simple text, and the Web was not yet multi-media.Although NSFNET was a global backbone, linking more than 28,000 domestic US networks andmore than 22,000 networks outside the US, 4 the Internet was mainly dedicated to North Americanusers (almost 70% of the total number of Internet users in 1995) who relied on low-speed

broadband access (typically 128kbit/s). Since then, traffic has exploded, as shown in Figure 2.1 below.

3 From 1987 to 1995, the NSFNET Backbone was designed, managed, and operated on behalf of the NSF by Merit Network, Inc., a

non-profit corporation governed by public universities, partnering with IBM, MCI, and the State of Michigan.

4 Source: California State University, NSFNET -- The First National, then Global Backbone Network

(http://som.csudh.edu/cis/lpress/471/hout/nsfnet.htm)
http://som.csudh.edu/cis/lpress/471/hout/nsfnet.htmhttp://som.csudh.edu/cis/lpress/471/hout/nsfnet.htm


9/62


Ref: 17038-075 .

Figure 2.1: Global IP

traffic per month [Source:

Cisco]

NSFNET was the backbone over which networks exchanged traffic. When it was closed in favorof commercial backbones, there was no longer a single backbone that could be used for trafficexchange. Instead, four Network Access Points (NAPs) across the US were designated as the

points at which traffic could be exchanged by the new commercial backbones, and these becamethe hub for Internet interconnection in the early years of the commercial Internet. The presence of

these four NAPs in the US explains the historical aggregation of Internet traffic in the US duringthis period. As this paper will show, since then the global distribution of traffic has evolvedconsiderably.

Internet interconnection has not been regulated, enabling commercial arrangements known as peering and transit to be negotiated based on evolving business requirements between the backbones and Internet service providers (ISPs). The introduction of commercial backbones notonly modified the structure of the Internet infrastructure ecosystem, but also resulted in changes tothe type of Internet traffic carried, as this greatly increased the amount of commercial trafficcarried and exchanged in addition to the university and research traffic previously carried.

This paper describes the changes that have occurred in the Internet ecosystem since thecommercialization of the Internet backbone 15 years ago. In particular, we analyze the evolution inthe nexus of interconnection, from the historical base of the US, to increased interconnection inother developed countries and developing countries, and how this evolution facilitated the growthin the number of users and their usage of the Internet. Finally, we show how the type of Internettraffic is changing, and how there is an accompanying evolution of the different participants in theInternet ecosystem and their inter-relationships that will further promote, and reward, the spread ofInternet interconnection around the world.

0

10 000

20 000

30 000

40 000

50 000

60 000

1 9 9 6

1 9 9 7

1 9 9 8

1 9 9 9

2 0 0 0

2 0 0 1

2 0 0 2

2 0 0 3

2 0 0 4

2 0 0 5

2 0 0 6

2 0 0 7

2 0 0 8

2 0 0 9

2 0 1 0

2 0 1 1

2 0 1 2

2 0 1 3

P B / m o n

t h


10/62


Ref: 17038-075 .

The remainder of this paper is laid out as follows:

Section 3 introduces the main concepts required to understand the relationships between thedifferent participants of the Internet Interconnection ecosystem.

Section 4 details the main international trends in the Internet interconnection ecosystem overtime, primarily the globalization of Internet users and the increase in content.

Section 5 analyses the globalization of IXPs that has resulted from these international trends.

Section 6 provides an outlook of the main Internet trends that have emerged over the past 15years and that are shaping the globalizing Internet infrastructure.

Section 7 summarizes the primary conclusions reached.


11/62


Ref: 17038-075 .

3 Overview of the IP interconnection ecosystem

In this section, we introduce the different players of the Internet ecosystem, as well as thetraditional relationships between them, including the original forms of peering and transitrelationships. We start with the traditional stakeholders involved in delivering traffic end-to-end,and then discuss IXPs, which were originally introduced to increase the efficiency of the exchangeof traffic, and have grown to have a critical role in the globalization of Internet traffic. As we willshow in Sections 4 and 5, in recent years as the traffic has become more global, the location of theIXPs has changed as well.

3.1 The traditional Internet ecosystem

This section introduces the primary participants in the Internet ecosystem, and describes at a highlevel the relationships that have developed between them for operational and financial purposes.

3.1.1 The various types of players within the Internet ecosystem

The Internet is a network of networks interconnected with one another, and operated by differentInternet providers. Traditionally, we can distinguish four different roles in the value chain ofcontent delivery over the Internet (involving data transfers between content providers and endusers over an Internet connection):

content providers and aggregators ISPs Internet backbone providers end users.

This classification can still be used to describe the Internet of today, and thus will be used in the rest ofthis paper, although new types of players have emerged in the past 15 years and others cover more thanone role. In particular, we will introduce the important role of the IXP in Section 3.2. Figure 3.1 provides a simplified illustration of the value chain of content delivery over the Internet.

Figure 3.1: Value chain of content delivery over the Internet, in the early Internet [Source: Analysys Mason]


12/62


Ref: 17038-075 .

As elements in this value chain, the four types of market players identified above could be definedas follows:

A content provider (and aggregator ) creates content for the Internet and/or aggregates thiscontent, to make it available to its Internet users or customers. A content provider will getInternet access via an ISP which provides the data transmission service to the rest of theInternet. The roles of content creation and content aggregation may be played by differententities, and for instance content aggregators may gather content from multiple externalsources. However, for clarity purposes, we will consider that both roles are played by a singleentity, for example The Financial Times online.

An ISP offers its customers access to the Internet via a data transmission technology such asdial-up, DSL, cable modem, wireless, or a dedicated high-speed connection. Its customers areInternet end users and content providers. In this paper, we will consider that ISPs have either a

regional or national footprint, for example America Online in the US, or Free in France

An Internet backbone provider delivers traffic to and from third-party networks through itsinfrastructure of national or international high-speed fiber-optic networks. An Internet

backbone provider s network is typically connected to the networks of several ISPs and toother backbone providers networks . The entity may also act as an ISP in selling servicesdirectly to large enterprise users; examples include Level 3 and France Telecom.

An end user accesses the Internet via a fixed or mobile device connected to the Internet by anISP. The end user may be an individual consumer, an enterprise or organization.

3.1.2 Relationships between the various players

We describe in this section the relationships between the different stakeholders of the Internetecosystem introduced above. The operational and financial relationships detailed in Figure 3.2 aredescribed at a high level in the rest of this section.

Figure 3.2: Money flows between the players in the Internet ecosystem [Source: Analysys Mason]


13/62


Ref: 17038-075 .

Relationship between content provider and ISP: Internet access

Content providers traditionally did not possess their own networkinfrastructure, and therefore bought Internet access from an ISP:the ISP allows the content provider to make its content availableto any Internet user, by ensuring the routing of traffic between thecontent providers servers and the end users.

When the content provider is a sizable organization, it connects to the ISP with high-capacity bandwidth, dedicated routers and specific servers. When it is a smaller organization or anindividual (e.g. a blog owner), the ISP generally hosts the content on its own servers and will notuse dedicated infrastructure. It should be noted that, depending on the number of users that accessthe providers content, the quantity of traffic generated, and the network infrastructure it owns andoperates, the content provider can either connect to an ISP or connect directly to a backbone

provider thereby effectively itself acting as a large ISP.

In terms of money flows, the content provider typically pays the ISP for its services, based on thevolume of the interconnection between them.

Relationship between ISP and Internet backbone: Internet transit

An ISPs footprint may be limited to a certain geographical area, andits network consists of connections between its points of presence(PoPs) and to its own users. Therefore, in order to offer its users aglobal reach to the Internet, an ISP needs to enter into transit relationships with at least one Internet backbone provider, which provides it with access to the rest of the backbone providers transitcustomers as well the customers of other backbones with which it peers.

In the simplest case, an ISP establishes a single connection to a backbone provider at a PoP. Often,however, an ISP has more than one PoP at a national level, and thus several separate connectionsto a backbone provider at those PoPs. An ISP may also be a customer of multiple backbone

providers often referred to as multi-homing connecting to each of them at one or more PoPs.

Relationship between Internet backbones: Internet peering

As a backbone providers network consists of connections between its PoPs and to its owncustomers, in order to sell access to the entireInternet it must connect to other backbone

providers. Providers of a similar size willtypically enter in a relationship called settlement-

free peering (or more commonly just peering ), whereby each backbone exchanges traffic betweenits own customers and those customers of the other backbone. This is typically settlement-free,


14/62


Ref: 17038-075 .

meaning that neither backbone provider pays the other for the exchanged traffic. Instead, peeringenables backbones to sell access to the entire Internet to its own transit customers, such as ISPs.Therefore, backbone providers will only enter into a peering relationship with another backbonewhen it is mutually beneficial. For this reason, backbone providers will enter into peering

agreements only with peers that meet a list of objective criteria.

These criteria stem in large part from the decisions that peering partners take regarding how toroute traffic between them, in response to two features of Internet interconnection. First, a centralfeature of the Internet protocol (IP) is that it is connectionless , meaning that the transmission pathis determined by network availability and not by a pre-determined circuit in particular, the pathtaken by a web request need not be the same as the return path taken by the response. Second,Internet backbones will typically interconnect in multiple locations for efficiency, in order to keeplocal traffic in the same region, and for network redundancy or resilience to ensure a continualconnection if one link fails.

As a result, two backbones that are connected in a number of places must determine where trafficis exchanged between them, in order to fairly distribute the carriage of exchanged traffic. Thecommon solution is what is known as hot-potato routing , in which, as the name suggests, theoriginating backbone hands over the traffic as quickly as possible namely, as close as possible toits point of origination. Similarly, the peering partner will do likewise with the return traffic. This

process is illustrated in Figure 3.3 and described in more detail below.

Figure 3.3: Hot-potato routing [Source: Analysys Mason]


15/62


Ref: 17038-075 .

In this example, an end user connected via an ISP to Backbone 1 is requesting content from acontent provider connected via another ISP to Backbone 2. The following steps occur:

1. Backbone 1 exchanges the request with Backbone 2 at the nearest IXP, which is IXP 1.2. This traffic is then carried by Backbone 2 to the content provider.3. The return traffic is then transferred back to Backbone 1 through the nearest IXP (IXP 2).4. Finally, Backbone 1 carries the return traffic to the end user.

Many backbones publish peering policies detailing the criteria that another backbone must meet inorder to qualify for peering. Generally speaking, peering policies have two relevant sections,aimed in large part at managing the traffic flow between the networks:

Network requirements specifying the geographical scope of the network and its raw capacity.These requirements are designed to ensure that peering partners have made similar

investments and that the backbone requesting peering will not be able to free ride oninvestments made by the backbone receiving the request. For instance, a national backbonewill not want to peer with a regional backbone, because in this case all traffic exchange wouldhave to take place within the regional backbones reach, and the national backbone wouldcarry traffic outside the regional backbones network in both directions.

Peering requirements additional geographical and traffic requirements specific to the peering connections between the two backbones. Again, this is to prevent free riding andensure that both backbones receive equal benefits from the connections. For instance, thegeographical requirements ensure that connections take place in multiple locations across a

country in order to distribute traffic exchange. 5 Further, some backbones require that the trafficthat they receive from the backbone requesting peering is roughly balanced with the traffic thatthey send to that backbone, so that with hot-potato routing they do not use more capacity forthe peering connection than their partner. This requirement is often referred to as a balancedtraffic ratio .

As an example, the list below summarizes the main objective requirements that must be met for acarrier to have settlement-free peering with AT&T. 6

5 Pee ring traffic is typically exchanged on a national or regional level, and not between continents. See, for instance, AT&Ts

requirements in Figure 3.4.

6 Specifically, to peer with AT&T WorldNet network.


16/62


Ref: 17038-075 .

Type of requirement Details of requirement

Capacity A peer must operate a US-wide IP backbone whose links have a capacity of10Gbit/s or greater

National presence A peer must meet AT&T at a minimum of three mutually agreeable geographicallydiverse points in the US. The US interconnection points must include at least onecity on the East Coast, one in the central region, and one on the West Coast, andmust currently be chosen from AT&T peering points in a list of selected metropolitanareas (New York City, Washington DC, etc.)

International presence

A peer must interconnect in two mutual non-US peering locations on distinctcontinents where the peer has a non-trivial backbone network. These non-USpeerings will be with AT&Ts regional AS only

Traffic A p eers traffic to/from the interconnected AT&T US network must be on -net only,and must amount to an average of at least 7Gbit/s in the dominant direction to/from

AT&T in the US during the busiest hour of the month

Bandwidth Interconnection bandwidth for private peers must be at least 1Gbit/s at each US

interconnection point

Exclusivity A network (ASN) that is a customer of an AT&T US network for any dedicated IPservices may not simultaneously be a settlement-free peer of that same network

Quality of Service A peer must have a professionally managed 24 7 network operation center. It mustagree to repair or otherwise remedy any problems within a reasonable timeframe. Itmust also agree to actively cooperate to resolve security incidents, denial of serviceattacks, and other operational problems

Traffic ratio A peer must maintain a balanced traffic ratio between its network and AT&T, andmust have:

no more than a 2:1 in-out ratio of traffic (traffic into AT&T - traffic out of AT&T

network), on average each month a reasonably low peak-to-average ratio

Existing peers whose in-out ratio rises above 2:1 will be expected to work with AT&Tto implement best-exit routing or to take other suitable actions to balance transportcosts

Routing policy A peer must abide by the following routing policy:

it must use the same peering AS at each US interconnection point and mustspecify a consistent set of routes at each point, unless otherwise mutually agreed

no transit or third-party routes are to be announced; all routes exchanged mustbe the peer's or those of its customers

the peer must filter route announcements from its customers by prefix

neither party shall abuse the peering relationship by engaging in activities suchas: pointing a default route at the other, or otherwise forwarding traffic fordestinations not explicitly advertised, resetting next-hop, selling or giving next-hop to others

Financial stability A peer must be financially stable

Figure 3.4: AT&T peering requirements (as of December 2010) [Source: AT&T 7 ]

7 http://www.corp.att.com/peering/
http://www.corp.att.com/peering/http://www.corp.att.com/peering/


17/62


Ref: 17038-075 .

In contrast to a transit connection, which may only require a connection at one place and givesaccess to the entire Internet, settlement-free peering requires multiple connections and gives accessonly to the peers customers. Therefore, backbone providers must connect to multiple other

backbones in order to achieve a global reach. Another difference from transit is that traffic

delivered under peering is subject only to best efforts , and there is no formal quality-of-serviceguarantee. It may, therefore, involve a lower quality than a transit connection (particularly asregards latency), for which there are Service Level Agreements (SLAs).

While peering is traditionally settlement-free, another form does exist, known as paid peering .This is a mix of peering and transit: like peering, it only pr ovides access to a peers directcustomer base, and not the entire Internet, but like transit one provider pays the other for thisaccess. Paid peering is used by ISPs and content providers to lower their transit costs and improvethe performance (by using more direct routing of traffic compared to transit). For instance, ifISP A establishes a paid peering agreement with ISP B, then traffic between customers of ISP A

and ISP B will be exchanged directly by the two ISPs, rather than going through a third-partytransit provider at a higher cost.

Relationship between ISP and the end user: Internet access

The ISP provides the end user with Internet connectivityfor a certain fee; this is typically a flat fee per month, butmay also depend on the volume of traffic used. The enduser will pay separately for any content consumed,

directly to a content provider or aggregator.

Evolution of interconnection arrangements

It is also important to note that backbone providers follow a fairly clear evolutionary path as theygrow in size. Smaller backbone providers cannot meet the settlement-free peering requirements oflarger providers, and therefore need to purchase transit from them. But as they grow, theirrelationships with other providers begin to change, as they are progressively able to meet therequirements to peer with larger backbones, and eventually grow to the point where they can beginto sell transit themselves. Figure 3.5 illustrates this evolution of backbone providers as theyaccumulate more traffic and their traffic exchange ratios change. This progression is veryimportant for players that hope to evolve over time from paid transit arrangements to settlement-free peering arrangements, and for policy makers to understand the market evolution withoutregulatory intervention.


18/62


Ref: 17038-075 .

Figure 3.5: Usual backbone provider relationships evolution

3.2 Introduction to Internet Exchange Points

As discussed previously, the Internet ecosystem 15 years ago had a rather simple structurecharacterized by relatively clear and hierarchical relationships, with end users and content

providers buying Internet access from ISPs, ISPs buying Internet transit from a limited number of backbone providers to get access to the whole Internet, and backbone providers peering with eachother to ensure the full extent of their Internet connectivity. An array of IXPs arose to offer pointswhere providers can connect, and these IXPs have undergone significant globalization and changein role, as we will see in later sections. Here we introduce these IXPs, beginning with the early

Network Access Points.

3.2.1 Network Access Points

When the NSF decided to commercialize the Internet backbone, it designated four dispersed NAPs: in San Francisco (operated by PacBell), Chicago (BellCore and Ameritech), New York(SprintLink) and Washington DC (MFS). These points provided what is known as public peering

between backbone providers, in which all participants exchange traffic through a common switch.Figure 3.6 below illustrates the architecture of the early Internet, in which NAPs served the

backbones at the top of the hierarchy.

1. The backbone is too small to peer with any otherbackbones, so buys transit for almost all the Internetreach

2. The backbone gets large enough that others willentertain peering; typically peers with large number ofsmall backbones

3. The backbone gains peering agreement with one (ortwo) large backbone, substantially cutting transitcosts

4. The backbone gains peering agreements withmultiple large backbones; the backbone starts cuttingback on peering with smaller backbones

6. The backbone provider increasingly begins to selltransit instead of purchasing transit

Number ofpeers

Transitcosts

Transitrevenues

5. The backbone does not need transit relationshipsanymore, and has no more transit costs

1

3

4

5

6

2


19/62


Ref: 17038-075 .

Figure 3.6: Simplified architecture of the early Internet [Source: Analysys Mason]

There are two noteworthy aspects to these early years of the commercial Internet. First, much ofthe hierarchy of the Internet was based in the US for a number of reasons examined in Section 4: in particular, the original core infrastructure was in the US, much of the Internet content was

hosted in the US, and international routes to and from the US were relatively cheap because of theamount of capacity on trans-Atlantic and trans-Pacific subsea cables.

Second, since the online market was still young, and most users were accessing the Internet viadial-up, content providers only had a limited size. In this context, we should note that it is only 15years ago that the first of todays major Internet companies started to emerge: Amazon wasfounded in 1994, Yahoo in 1995, eBay in 1995, and Google in 1998. In the following, we examinehow this situation has changed, and the implications of those changes.

These initial NAPs provided an efficient point where the backbones could meet and exchange

traffic while avoiding the cost of directly interconnecting with all other backbones for peering. Asthe Internet became more international, and content became a greater source of traffic, the roles ofthe exchange points was heightened significantly, as we show below.

3.2.2 Internet Exchange Points

The nature of Internet exchange has evolved over the past 15 years. NAPs were used for public peering, whereby traffic was exchanged using a common switch, which soon became congesteddue to the relatively large volume of traffic that resulted from the initial Internet boom of the late

1990s. At the same time, the operators of the NAPs had the potential to favor their own services:


20/62


Ref: 17038-075 .

for instance, when the MAE-East NAP was owned by MCI WorldCom, it required the use of MCIcircuits to access MAE-East services. 8

As a result, a migration began towards peering at neutral IXPs. Companies such as Equinix beganto open large data centers where operators could host their servers and/or connect to one anotherusing direct cross-connections to effectively create a neutral IXP. In some countries, non-profitIXPs run by a consortium of members were established. Indeed, today several of the largest IXPsuse this model, including the London Internet Exchange (LINX) and the Amsterdam InternetExchange (AMS-IX). These IXPs place shared switches in a number of data centers, includingthose of Equinix, and connect them together with high-capacity fiber to create what is known as avirtual IXP . Members are able to choose which data center to use for hosting, and can use thevirtual IXP to connect to members in any of the other data centers.

The move to such IXPs enabled what is known as private peering , whereby service providers can

directly connect and thereby control congestion over each bilateral link with other service providers. The service providers could also sell or purchase transit over these direct cross-connections at the IXPs. As a result, these IXPs became magnets for the entire Internet ecosystem,where the entire range of stakeholders including ISPs and content providers could buy and selltheir services to one another at relatively low cost, in a highly competitive and resilientenvironment.

There are clearly strong network effects that support the growth of an IXP. The more service providers there are in an IXP, and the more traffic they exchange, the more attractive it is for otherservice providers to locate at the same IXP, in order to arrange connectivity with the most service

providers. However, the growth in traffic described above has provided a strong countervailing pressure for IXPs to proliferate across populated areas of countries and regions. Having adispersion of IXPs has benefits for backbone providers in a peering relationship, in terms ofreducing latency, ensuring quality of service (via redundancy), and controlling costs.

First, where the two parties are in the same region, having a nearby IXP enables the traffic toremain local, rather than tromboning to a more distant location. This enhances the quality ofservice thanks to reduced latency. The existence of multiple IXPs in different locations within thesame region also provides redundancy, and thus protection in case of natural disaster, for instance.

Second, where the two parties are in different regions, having dispersed IXPs enables backbone providers to effectively share traffic loads: one will carry the originating traffic and the other willcarry the return traffic, using hot-potato routing.

In this way, IXPs began to develop in a direction significantly different from the intention of theoriginal NAPs, thereby enlarging their role in the ecosystem from basic Internet networkinterconnection points to important aggregators of Internet players and traffic. Instead of justfacilitating exchange between Internet backbones, IXPs became magnets for the other key players

8 Source: William B. Norton, Internet Service Providers and Peering


21/62


Ref: 17038-075 .

in the ecosystem, namely content providers and ISPs, and enabled them to directly interconnectwith one another, reducing their need to purchase transit from backbones. The result was to lowercosts while also improving the performance of the Internet by facilitating direct connections

between ISPs and content providers.

Over time, IXPs have become magnets within the entire Internet ecosystem, thanks to theirability to reduce costs and improve performance for Internet stakeholders. In particular,IXPs have played a major role in reducing tromboning in regions that have established anIXP hub.


22/62


Ref: 17038-075 .

4 General international trends

Due to the primarily US location of the early Internet infrastructure, usage and traffic exchangewas at first focused on the North American region. However, Internet development has relativelyquickly moved beyond these areas and reached other markets, first in developed countries ofWestern Europe, then in fast-growing countries in South-East Asia, and more recently indeveloping countries in most parts of the world. In all regions this evolution has bothaccommodated and promoted growth in local Internet content creation and consumption.

The increase in Internet usage has arisen from a virtuous circle between Internet access andInternet content , in two overlapping stages. First, as more users accessed the Internet outside theUS, more content was developed to meet the se users needs for local and regional information,

services, and applications. Then, as users began to migrate to broadband, multimedia content became available that was accessible through broadband, which in turn made broadband accessmore attractive for new users. The result has been a tremendous explosion in traffic across theglobe, which has in turn required new means of efficiently exchanging that traffic. An overview ofthese global trends is provided in this section.

4.1 Increase in access

Internet access increased enormously as it spread across the globe, based initially on dial-up accessand in many countries on shared access at cyber cafs or Internet cafs. Then broadband, bothfixed and mobile, increased the bandwidth of access as well as the mobility. Together, these trendshave drastically increased demand for Internet content. Geographically, the explosion in usage hasspread from North America to other developed regions, and then developing countries. Indeed,today a significant portion of Internet users are to be found in large markets like China and India.China now boasts more than 400 million Internet users, and other emerging Asian nations are alsofuelling the growth of traffic in this region. As shown in Figure 4.1 below, in 1995 NorthAmerican users represented around 70% of global Internet users, a figure that declined to only15% by the end of 2009. As can be seen, the distribution of usage across the wo rlds regions isconverging on the distribution of population across these regions (shown in the column at the far

right of the chart).


23/62


Ref: 17038-075 .

Figure 4.1: Evolution of

the distribution of Internet

users by region,

compared to the

population split as of2009 [Source: ITU]

The increase in usage has come particularly from Internet broadband subscriptions that allow endusers to access the new high-bandwidth services such as video streaming. At the end of 2009, therewere more than 450 million fixed broadband subscriptions globally, a 10-fold increase since 2001.This dramatic uptake of fixed broadband has been accompanied by a (more recent) surge in mobile

broadband usage, as shown in Figure 4.2 below. By 2009, there were 110 million mobile

broadband subscriptions in 2009, and according to CTIA, revenues from wireless data in the USalone amounted to US$41.5 billion, up from just US$211 million in 2000. 9 Similar growth hascharacterized markets in Europe and much of Asia as well, resulting in a demand for Internetservices that is distributed much more evenly across the world than 15 years ago, as shown in thenext section.

9 http://www.ctia.org/media/industry_info/index.cfm/AID/10323

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 9 9 5

1 9 9 6

1 9 9 7

1 9 9 8

1 9 9 9

2 0 0 0

2 0 0 1

2 0 0 2

2 0 0 3

2 0 0 4

2 0 0 5

2 0 0 6

2 0 0 7

2 0 0 8

2 0 0 9

2 0 0 9

( p o p . )

US & Canada Europe Asia Latin America Africa

P o p u

l a t i o n s p

l i t
http://www.ctia.org/media/industry_info/index.cfm/AID/10323http://www.ctia.org/media/industry_info/index.cfm/AID/10323


24/62


Ref: 17038-075 .

Figure 4.2: Broadband

subscriptions worldwide

[Source: Analysys Mason

Research]

Not only has the number of Internet access users experienced a significant growth, but also thetraffic generated by each individual user has increased enormously. This can be explained in part

by the increase in availability of even higher capacity Internet links, and the growing consumptionof bandwidth-intensive applications such as video downloads (see Section 6.2) , the developmentof cloud computing services (see Section 6.3) , and the take-up of a large variety of Internet-

connected devices (see Section 6.4) .

4.2 Increase in content

The globalization and the exponential growth of Internet users have fueled an increase in theamount of content made available online. Indeed, the increase in users globally is indivisible fromthe increase in content: users are bound to consume more local content for language, cultural and

proximity reasons and therefore the global growth in users has led to content that is produced(and consumed) outside the initial geographical focus of the Internet. Further, as the means ofaccess shifted to broadband, the nature of the content shifted accordingly to high-bandwidthmultimedia content.

As an illustration of the exponential growth experienced over the past 10 years, Google softwareengineers Jesse Alpert and Nissan Hajaj announced on July 25, 2008 that Google Search haddiscovered one trillion unique URLs. This represents a dramatic growth compared to the situation10 years earlier, when Google had indexed only around 26 million pages. 10 Other metrics present asimilar picture: for example, the following two graphs present the evolution of the number of

10 http://googleblog.blogspot.com/2008/07/we-knew-web-was-big.html

0

100

200

300

400

500

600

2 0 0 1

2 0 0 2

2 0 0 3

2 0 0 4

2 0 0 5

2 0 0 6

2 0 0 7

2 0 0 8

2 0 0 9

S u b s c r i p

t i o n s

( m i l l i o n s

)

Fixed broadband Mobile broadband
http://en.wikipedia.org/wiki/Google_Searchhttp://googleblog.blogspot.com/2008/07/we-knew-web-was-big.htmlhttp://googleblog.blogspot.com/2008/07/we-knew-web-was-big.htmlhttp://en.wikipedia.org/wiki/Google_Search


25/62


Ref: 17038-075 .

Internet hosts (basically the number of computers connected to the Internet), as estimated by theInternet Systems Consortium, 11 and the number of Internet websites, as estimated by Netcraft.


host count between 1995

and 2009 [Source:

Internet Systems

Consortium]


the total number of

websites between 2000

and 2009 [Source:

Netcraft]

The take-up of Internet blogs could be used as another illustration of this trend. Blog usage only began at the very end of the 1990s, being popularized by the arrival of the first hosted blog tools(Blogger.com was created in 1999, and later bought by Google in 2003). Yet by December 2010

11 Source: Internet Domain Survey 2009, Internet Systems Consortium (http://www.isc.org/solutions/survey)

0

100

200

300

400

500

600

700

800

J a n -

9 5

J a n -

9 6

J a n -

9 7

J a n -

9 8

J a n -

9 9

J a n -

0 0

J a n -

0 1

J a n -

0 2

J a n -

0 3

J a n -

0 4

J a n -

0 5

J a n -

0 6

J a n -

0 7

J a n -

0 8

J a n -

0 9

H o s

t c o u n t s

( m i l l i o n s

)

0

50

100

150

200

250

M a r - 0

0

M a r - 0

1

M a r - 0

2

M a r - 0

3

M a r - 0

4

M a r - 0

5

M a r - 0

6

M a r - 0

7

M a r - 0

8

M a r - 0

9

W e b s i

t e s

( m i l l i o n )
http://www.isc.org/solutions/surveyhttp://www.isc.org/solutions/survey


26/62


Ref: 17038-075 .

there were more than 150 million blogs available on the Internet, according to Blogpulse. 12 Thisconfirms the significance of the so-called Web 2.0 trend, i.e. user-generated content specificallydesigned for interactions with other Internet users.

As well as there being more websites, the volume of available content has also grown, as a resultof a virtuous circle of more powerful computers, the availability and adoption of broadbandaccess, the increase in bandwidth available for carrying Internet traffic, and the introduction ofmultimedia content, in particular video, along with new ways to share that video such as peer-to-

peer. Figure 4.5 shows how new categories of Internet video have arisen in the last five years, andare projected to become the dominant category of content over the next few years.

Figure 4.5: Consumer

Internet traffic forecasts

[Source: Cisco Visual

Networking Index]

Note: File sharing includes traffic from peer -to-peer applications such as BitTorrent and eDonkey, as well as web-based file sharing

As described later (see Section 6.1) , Internet content has progressively gained in value, thanks toan improved quality of service, substantial investments made in Internet backbone and access

capacity, and the development of premium content. This increase in value has sustained the growthof Internet users, and vice versa, in what could be seen as a positive spiral.

4.3 Conclusion

The combination of the trends described above has resulted in an exponential growth of Internettraffic in general, and in the traffic carried by Internet backbone providers in particular. Trafficgrowth has in turn naturally generated significant changes in the Internet ecosystem, as market

12 Source: http://www.blogpulse.com/

0

2,000

4,000

6,000

8,000

10,000

12,000

2006 2007 2008 2009 2010 2011 2012 2013 2014

P B p e r m o n

t h

Web and Email File SharingInternet Gaming Internet VoiceInternet Video com. Internet Video to PCInternet Video to TV
http://www.blogpulse.com/http://www.blogpulse.com/


27/62


Ref: 17038-075 .

players adapted by modifying their network architectures in order to carry the increased trafficmore efficiently. In particular, it has created demand for IXPs across countries and regions in orderto deliver connectivity closer to end users while also accommodating the increases in the volumeof content.


28/62


Ref: 17038-075 .

5 The globalization of the use of IXPs

We have seen how the increase in the number of Internet users outside the US led to acorresponding increase in Internet traffic in other regions of the world, with online content beingcreated, hosted and accessed from outside the US. In the present section, we discuss how theseinternational trends have caused a globalization of IXPs. We first discuss how the networktopology of the Internet evolved so as to enable a more efficient carriage of traffic between andwithin various regions. We then describe how this evolution has in turn modified the role of IXPs,and how IXPs have accelerated the regionalization of the exchange of traffic.

5.1 Evolution of the topology of the Internet

Broadly speaking, Internet exchange has gone through three phases

1. US-centric an initial phase centered on the US2. OECD-centric a second phase involving a shift in focus to include developed countries in

Europe and Asia forming the core of the OECD 13 3. ROW-centric the current phase focusing on the rest of the world.

As shown in Figure 5.1, the number of IXPs in operation has kept growing at a significant pace,although so far the majority of them are still in OECD countries.

Figure 5.1: Number ofoperated IXP in the world

[Source: Packet Clearing

House (www.pch.net) ,

Analysys Mason

estimates]

Note: Data prior to 2005 has been reconstructed based on estimates from Packet Clearing House

13 Organization for Economic Cooperation and Development

0

50

100

150

200

250

300

350

400

1 9 9 9

2 0 0 0

2 0 0 1

2 0 0 2

2 0 0 3

2 0 0 4

2 0 0 5

2 0 0 6

2 0 0 7

2 0 0 8

2 0 0 9

2 0 1 0

T o t a l n u m

b e r o

f I X P s

North America Europe Asia Latin America Africa
http://www.pch.net/http://www.pch.net/


29/62


Ref: 17038-075 .

5.1.1 Phase 1: US-centric

As discussed above, when the NSF decided to commercialize the Internet backbone, it designatedfour NAPs across the US. A significant amount of international traffic passed through the US atthe time, either to connect with US providers to get access to US content and users, or for transit,sometimes back to the same country of origination. Even though little data are available toaccurately quantify the reliance on the US for traffic, an OECD paper dated 1998 14 mentioned that,at that time, more than half of intra-European and intra-Asian IP traffic was transported via theUS. This traffic can be grouped into three categories:

Tromboning this occurs when traffic from within one country or region flows throughanother country to be exchanged and delivered back to the original country or region. In theearly years of the Internet, most tromboning took place via the US.

Importing of content when content is accessed in other countries such as the US, including insome cases domestic content that is hosted in the US for economic reasons.

Natural traffic flows involving email and data access between users in different countries,that effectively makes up the flow of traffic that characterizes the global web of users that isthe Internet.

Tromboning involves the carriage of traffic on unnecessarily long and technically sub-optimalroutes. One of the main reasons for tromboning was the high transport costs in Europe prior toliberalization. As an example, in 1998, the monthly price of a 2Mbit/s connection between London

and Paris was US$38,000, while a connection of the same capacity between London and Virginiawas US$30,000 even though Virginia is almost 25 times further away than Paris. 15 Tromboningarose because all ISPs at the time had to connect to the US for international transit, and theysometimes also used these links to exchange domestic and regional traffic in order to avoid thecost of directly connecting to other ISPs, given the high costs of those connections.

In addition, Europe naturally imported a significant amount of content from the US, due to the predominantly US-originated nature of content at that time. For example, at the end of 1997,Swisscom s transatla ntic Internet link carried on average six times as much data from the US as itcarried to it.

The US was the de facto hub of the Internet for many of the early years. Several factors canexplain this situation:

Historical factors The original core infrastructure was in the US, and the majority of onlinecontent was hosted there: in 1997, more than half of all public Internet hosts (i.e. connected

14 Source: OECD, Internet Traffic Exchange: developments and policy, 1998

15 Source: Emanuele Giovannetti and Cristiano Andrea Ristuccia, Estimating market power in the Internet backbone. Using the IP

transit bad- X database, March 2003. (http://ec.europa.eu/information_society/activities/internationalrel/docs/itu/giovtransitpr.pdf)
http://ec.europa.eu/information_society/activities/internationalrel/docs/itu/giovtransitpr.pdfhttp://ec.europa.eu/information_society/activities/internationalrel/docs/itu/giovtransitpr.pdfhttp://ec.europa.eu/information_society/activities/internationalrel/docs/itu/giovtransitpr.pdf


30/62


Ref: 17038-075 .

users and content providers) were based in the US. 16 As a result, users outside the US wereaccessing primarily US-based content instead of local content, and European ISPs wereheavily relying on the existing US backbone infrastructure.

Economic factors Before the liberalization process began in Europe, and due to the lack ofinfrastructure competition, domestic connections were penalized by the prohibitive price ofintra-regional capacity. In contrast, US markets had been liberalized and open to competitionfor several years, and bandwidth was more abundant and available at lower prices.

Network performance factors At that time, 10 out of the 13 root-level servers 17 in the worldwere located in the US. Since these root servers process the information required to routeInternet traffic, the proximity of US-based NAPs to them ensured a faster delivery of thistraffic.

The use of US-based NAPs to exchange foreign traffic was inefficient, and has led to two sets ofresponses over time. The effective approach was a market-driven response to create more localexchange points in places outside the US, reducing tromboning and making traffic exchange moreefficient (see Section 5.1.2) . The other approach was a policy response, with a call for regulatoryintervention that went by the rubric of International Charging Arrangements for Internet Services(ICAIS). In this debate, which has taken place at the Asia-Pacific Economic Cooperation (APEC)forum and the International Telecommunication Union (ITU), countries underserved by Internetinfrastructure have sought to address bearing the costs of connecting to US-based IXPs for trafficexchange. While there is still ongoing debate about some form of policy response, the market-driven response oriented around competitive market investment, establishment of local IXPs,and growth of local content has effectively globalized traffic exchange, and we review thosechanges now.

The lack of IXPs outside the US created inefficiencies in the early Internet networkinfrastructure, best illustrated by the tromboning that characterized the traffic flow betweenEurope and the US at the time.

5.1.2 Phase 2: OECD-centric

To address the inefficiencies mentioned above, national and regional networks started tointerconnect to each other and exchange their traffic at regional IXPs outside the US. This beganwith the developed countries at the core of the OECD, notably in Western Europe and parts ofAsia. These IXPs were typically started by a group of ISPs joining together to choose a location

16 Source: Network Wizards July 1997 Survey, cited by OECD.

17 Internet addresses are based on the Domain Name System (DNS) in which there are a relatively small number of top-level domains

such as .com or .uk. Root-level servers process the DNS information that maps Internet addresses (such as www.google.com) onto

the IP addresses of the physical servers of the hosts (such as 173.194.37.104). Accordingly, in the early Internet the major part of

routing traffic (routing information between an ISPs DNS and the global server) was exchanged with the US.


31/62


Ref: 17038-075 .

and share resources in developing a switch. For instance, the London Internet Exchange (LINX) began in 1994 with five members (including the incumbent telecom operator, British Telecom) asa means to localize traffic exchange, and from there grew into one of the largest IXPs in the world.Similarly, the 1990s saw the establishment of the AMS-IX in Amsterdam, NetNod in Stockholm,

and DE-CIX in Frankfurt, while in Asia several significant exchanges opened in the late 90s andearly 00s, like the Hong Kong Internet Exchange (1995) and the Japan Internet Exchange (1997).These IXPs, which are qualified as Internet hubs, are today among the largest in the world. Thetable below provides information on a selection of the most important IXPs.

IXP name Location Date

established

Max throughput

(Gbit/s)

Deutscher Commercial InternetExchange

Frankfurt am Main, Germany 1995 3078

Amsterdam Internet Exchange Amsterdam, Netherlands 1997 1239

London Internet Exchange London, UK 1994 823Equinix Exchange 11 countries in US, Europe, Asia 1998 750

Moscow Internet Exchange Moscow, Russia 1995 450

Japan Network Access Point Tokyo and Osaka, Japan 2001 266

NetNod Internet Exchange(Sweden)

Five locations in Sweden:Stockholm, Malm, Sundsvall,Gothenburg, Lule

1997 204

Ukrainian Internet ExchangeNetwork

Kiev, Ukraine 2000 180

New York International InternetExchange

New York, US 1996 145

Japan Internet Exchange Tokyo, Japan 1997 145

Note: This list only includes IXPs for which traffic data is publicly available. Other large IXPs (e.g. FreeIX and NAP) do not appear as they do

not publish statistics on their throughput.

Figure 5.2: Significant IXPs around the world [Source: IXP websites, Wikipedia]

The number of IXPs in Europe doubled between 2001 and 2003 to reach more than 80,representing more than half of the total number of IXPs (and more than 70% of OECD IXPs). 18 There were several reasons for this successful emergence of IXPs outside the US, linked to theincrease in the portion of traffic that is local or regional:

Economic factors Peering at a national IXP reduced tromboning, which translated into areduction in payments to upstream transit providers in the US. Also, the price of regionalconnectivity fell dramatically following liberalization in OECD countries, making intra-regional traffic even more affordable.

18 Source: https://www.euro-ix.net/news/meetevent/napla_2008.pdf
https://www.euro-ix.net/news/meetevent/napla_2008.pdfhttps://www.euro-ix.net/news/meetevent/napla_2008.pdf


32/62


Ref: 17038-075 .

Network performance factors By exchanging traffic at a non-US-based IXP for intra-regional traffic or locally hosted content, ISPs were able to reduce the length of traffic routes,compared to a situation in which they used backbones in the US. This positive impact onnetwork performance has been confirmed by measurement studies. One such study concludes

that overall the main technical goals behind setting up the IXPs (efcient packet transfer,lower end-to-end latency and so on) are actually met .19 Other examples confirms the positiveimpact of an IXP on latency. 20

The establishment of IXPs in OECD countries outside the US has significantly improvedthe performance of the Internet in these countries, particularly by reducing latency, and hasdrastically reduced Internet connectivity costs.

These trends are illustrated by the increase in the proportion of intra-regional traffic over time inEurope, and later in Asia. The following charts provide evidence of this evolution: each chartshows the percentage of international Internet bandwidth provisioned from countries in eachregion of the world to other countries, by region. The first chart (Figure 5.3) shows that since 1999the bulk of European bandwidth connected European countries, with the proportion going to theUS21 falling from 30% in 1999 to under 20% in 2010, displaced to some degree by Asian

bandwidth. It should be noted that by 1999, most tromboning traffic had already been eradicatedthanks to the introduction of major European IXPs.

19 Source: Mohammad Zubair Ahmad and Ratan Guha, A measurement study determining the effect of Internet eXchange Points on

popular webservers (http://www.mzubair.net/web/Files/Wise-1569325693.pdf )

20 Note: the ITU indicates that for African ISPs, tromboning adds 200 to 900 milliseconds to each transmission, whereas with a local

IXP in place, adjacent ISPs can route traffic t o each others networks in 520 milliseconds (Source: Eric Lie, International Internet

interconnection - next generation networks and development, CSR 2007 Discussion Paper, 2007)

21 The figures in the following charts show the combined traffic to both the US and Canada. Since the vast majority of this traffic is to

the US, for simplicity we refer in the text simply to the US.
http://www.mzubair.net/web/Files/Wise-1569325693.pdfhttp://www.mzubair.net/web/Files/Wise-1569325693.pdfhttp://www.mzubair.net/web/Files/Wise-1569325693.pdfhttp://www.mzubair.net/web/Files/Wise-1569325693.pdf


33/62


Ref: 17038-075 .

Figure 5.3: International

Internet bandwidth from

European countries, by

region [Source:

Telegeography, GlobalInternet Bandwidth

(www.telegeography.com)]

As shown in Figure 5.4, Internet bandwidth from Asian countries, which in 1999 was still heavilyskewed towards the US, has largely been displaced by bandwidth between Asian countries, alongwith more capacity to Europe as well. As a result, bandwidth from the US has fallen from over90% in 1999 to under 50% in 2010.

Figure 5.4: InternationalInternet Bandwidth from

Asian countries [Source:

Telegeography, Global

Internet Bandwidth

(www.telegeography.com)]

Over time, as more multimedia content was consumed and volumes grew, the largest IXPs also

became magnets for content providers seeking to establish themselves closer to their end users,

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 9 9 9

2 0 0 0

2 0 0 1

2 0 0 2

2 0 0 3

2 0 0 4

2 0 0 5

2 0 0 6

2 0 0 7

2 0 0 8

2 0 0 9

2 0 1 0

U.S. & Canada Europe Asia Latin America Africa

0%

10%20%

30%

40%

50%

60%

70%

80%

90%

100%

1 9 9 9

2 0 0 0

2 0 0 1

2 0 0 2

2 0 0 3

2 0 0 4

2 0 0 5

2 0 0 6

2 0 0 7

2 0 0 8

2 0 0 9

2 0 1 0



34/62


Ref: 17038-075 .

and to save in the cost of transit to deliver their content. As a result, several of the IXPs havegrown into significant hubs for content and traffic.

As shown in Figure 5.5 below, the Netherlands, followed by the UK, has a distinct edge in theamount of international Internet bandwidth per population compared with a wide variety ofcountries, due in no small part to the significant role of their IXPs in attracting international

backbones and content providers. Similar activity is taking place in Asia and to a lesser degree inother parts of the world as IXPs act to help to localize traffic and attract a surrounding ecosystemof backbones and content providers.

Figure 5.5: International Internet bandwidth per head of population [Source: Telegeography, Global

Internet Bandwidth (www.telegeography.com), ITU, Euromonitor, company data]

The use of an IXP to avoid tromboning, and to develop a location into a major regional orinternational hub involves a number of factors, not all of which are within the control of the IXP

and its members. First, several of the largest hubs, including LINX and AMS-IX, benefitted fromthe early membership of the national incumbent telecom operator in the IXP: the presence of sucha player can maximize the exchange of domestic traffic via either peering or transit. Second, usageis also promoted by good management of the IXP and flexible policies for locating at the IXP thataccommodate all stakeholders. Finally, a supportive regulatory environment and strong rule of lawcan promote foreign investment by content providers and backbone providers, especially if liberal

policies that minimize regulatory burdens are in place, such as open access at internationalgateways, and low prices for backhaul to connect from the international gateways to the IXPs.

00

20

40

60

80

100

120

140

160

2 0 0 2

2 0 0 3

2 0 0 4

2 0 0 5

2 0 0 6

2 0 0 7

2 0 0 8

k b i t / s

Australia

USA

Canada

Germany

France

Singapore

UK

Netherlands


35/62


Ref: 17038-075 .

The ability of an IXP to reduce tromboning depends on several factors, including: the earlymembership of the national incumbent, which maximizes the exchange of traffic; flexible

policies as regards to membership; and a liberal regulatory environment favoring foreigninvestment and open international access.

Case study the Amsterdam IXP

The Amsterdam Internet Exchange (AMS-IX) is a good example of a successful IXP. AMS-IXstarted in 1994 and is today the second-largest IXP in the world when measured by peakthroughput, or the largest when measured by average throughput. It operates on a non-profit basis(members fees only cover costs), neutral (offering the same treatment for all members) andindependent (acting as an association). It connects more than 390 parties, including ISPs,

backbones, content providers, etc. 22

The reasons for its success at becoming an Internet hub include its early launch compared to itsEuropean peers, the membership of the incumbent KPN, its location close to an Internet rootserver (which improves performance see Section 5.1.1 above), and a welcoming regulatory andinvestment regime.

5.1.3 Phase 3: ROW-centric

In its most recent phase, the focus of development for the Internet has been moving from OECDcountries to the rest of the world (ROW). In Africa, for instance, IXPs started to be implemented

in 2002 and 2003. Before 2002, IXPs existed only in two out of the 53 countries in the continent(South Africa and Zimbabwe), but by 2003 the number of countries with IXPs had risen to 10, and

by December 2010 there were 20. 23

The relatively recent and slow take-up in Africa can be explained by the obstacles that stand in theway of the establishment of IXPs in emerging economies. The liberalization process is usually farfrom being completed, and regulators are not always familiar with the advantages and possibilitiesoffered by IXPs. In addition, the providers of international cable and satellite links are often alsoincumbent telecom operators that have strong incentives to maintain the status quo, and to deter or

delay competitive market entry. These operators are usually inclined to slow down the process ofintroducing IXPs, as they fear the erosion of the revenues they gain from internationalconnectivity. Finally, members may refuse to join the IXP, or even fight its establishment.

22 Source: http://www.ams-ix.net/connected

23 Source : http://www.onlineafrica.net/business/updated-list-of-african-ixps/
http://www.ams-ix.net/connectedhttp://www.onlineafrica.net/business/updated-list-of-african-ixps/http://www.onlineafrica.net/business/updated-list-of-african-ixps/http://www.ams-ix.net/connected


36/62


Ref: 17038-075 .

The development of IXPs in emerging countries typically faces an incomplete liberalization process, incumbents with incentives to maintain the status quo regarding internationalconnectivity, and sometimes the regulators lack of knowledge of the full extent of the

benefits of an IXP.

On a regional level, while traffic flows have changed significantly, both Africa and South Americastill rely heavily on connections to Europe and the US respectively. As shown in Figure 5.6 below,international bandwidth from African countries used to be mostly to the US (over 70% of

bandwidth from Africa in 2000). However, in recent years the US has been supplanted by Europe,which now has a greater percentage of capacity than the US did. The remainder of bandwidthlargely goes to Asia, with less than 5% now going to the US.



African countries [Source:


Internet Bandwidth

(www.telegeography.com),

Analysys Mason]

The bandwidth from Latin America presents largely the same picture, with some minor differences(see Figure 5.7 below). Between 1999 and 2010, the percentage of bandwidth going to the US fell

from just under 90% to just over 80%. The bandwidth going to other regions is minimal.

0%10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 9 9 9

2 0 0 0

2 0 0 1

2 0 0 2

2 0 0 3

2 0 0 4

2 0 0 5

2 0 0 6

2 0 0 7

2 0 0 8

2 0 0 9

2 0 1 0

U.S. & Canada Europe Asia Africa


37/62


Ref: 17038-075 .



Latin American countries

[Source: Telegeography,

Global Internet Bandwidth(www.telegeography.com),

Analysys Mason]

The main similarities between Africa and Latin America are that over 80% of their Internet bandwidth is connected to another region (Europe and the US respectively). At the same time,little bandwidth goes between countries within the region 17% in the case of Latin America, and

just 1% in Africa. This highlights the relative lack of liberalization of regional connectivity, whichin turn results in tromboning via major IXPs in Europe or the US, along with an apparent

reluctance of content providers and other stakeholders to invest in regional hubs.

Notwithstanding the present market facts, in the long run the Internet topology seems to beoriented toward a ROW-centric model, as today 89 countries possess their own IXPs 24 to exchangenational and regional Internet traffic, and reduce the cost associated with international Internetaccess. It is important to note that this evolution has not required specific regulatory intervention(e.g. the aforementioned ICAIS model has never been adopted): rather, the increased globalizationhas been the result of commercial solutions adopted by national and regional networks.

This diversification and multiplication of Internet traffic hubs outside the US, and thus the

evolution from a US-centric to a ROW-centric model, is confirmed by the changing pattern ofinternational Internet bandwidth from the US and Canada, as illustrated in Figure 5.8.

24 Source: Packet Clearing House Report on Internet Exchange Point Locations (https://prefix.pch.net/applications/ixpdir/summary/)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 9 9 9

2 0 0 0

2 0 0 1

2 0 0 2

2 0 0 3

2 0 0 4

2 0 0 5

2 0 0 6

2 0 0 7

2 0 0 8

2 0 0 9

2 0 1 0

https://prefix.pch.net/applications/ixpdir/summary/https://prefix.pch.net/applications/ixpdir/summary/


38/62


Ref: 17038-075 .


Internet bandwidth from US

and Canada [Source:


Internet Bandwidth(www.telegeography.com),

Analysys Mason]

Despite an exploding growth of Internet users in Asia, the share of Internet bandwidth from the USto Asia has not increased, as a large part of Asian Internet bandwidth remains either domestic ortransits via Europe. Similarly, European bandwidth has remained relatively stable, as a large partof intra-European bandwidth is handled by the numerous major Internet hubs in this region.African bandwidth goes almost fully to Europe, and the share of Internet bandwidth from the US

to this region is close to zero. Only Latin America still relies heavily on the US traffic routes.

The relative lack of liberalization of regional connectivity in Africa and Latin Americaresults in tromboning via major IXPs in Europe or the US, and there is an apparentreluctance of content providers and other stakeholders to invest in regional hubs.

Nevertheless, in the long run the Internet topology seems to be oriented toward a ROW-centric model, as today 89 countries possess their own IXPs. This evolution has not requiredspecific regulatory intervention (e.g. the ICAIS model has never been adopted), but is theresult of commercial solutions adopted by national and regional networks.

Case study the Kenya IXP

The case study of Kenya is interesting as it illustrates the barriers that obstruct the development ofIXPs in emerging countries, but nonetheless represents an early success story that can be emulated

by other emerging countries. In 2000, the Telecommunications Service Providers Association ofKenya (TESPOK) undertook to organize a neutral and non-profit IXP for its members. Afteralmost one year of preparatory work, the Kenya Internet Exchange Point (KIXP) was launched in

November 2000. Then the incumbent, Telkom Kenya, filed a complaint with the regulator arguingthat the IXP violated its exclusive monopoly on the carriage of international traffic: ISP servicesare open to competition, but they rely on Telkom Kenya for the underlying infrastructure, the latter

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 9 9 9

2 0 0 0

2 0 0 1

2 0 0 2

2 0 0 3

2 0 0 4

2 0 0 5

2 0 0 6

2 0 0 7

2 0 0 8

2 0 0 9

2 0 1 0



39/62


Ref: 17038-075 .

having the exclusive right to operate a national backbone for the purpose of carrying internationaltraffic. This action

Documents

Analysys Mason International IP Interconnection 23 Feb 2011