Internet Exchange Points (IXPs)

Philip Smith

E2 Workshop, AfNOG 2005

Objectives To be able to explain what is an

exchange point To be able to explain why ISPs

participate in IXPs To understand why IXPs are important To review some current IXP designs

used today To think about how to set up an

exchange point in your environment

Introduction to Internet Exchange Points

A bit of history What are they? Why use them?

A Bit of History…

End of NSFnet – one major backbone move towards commercial Internet

private companies selling their bandwidth need for coordination of routing

exchange between providers Traffic from ISP A needs to get to ISP B

Routing Arbiter project created to facilitate this

What is an Exchange Point Network Access Points (NAPs)

established at end of NSFnet original “exchange points”

Major providers connect their networks and exchange traffic

High-speed network or ethernet switch Simple concept – any place where

providers come together to exchange traffic

Internet Exchange Points

ISPs connect at Exchange Points or Network Access Points to exchange traffic

XP 1 XP 2

ISP A

ISP B

Conceptual Diagram of an IXP

ISP Router

ISP Router

ISP Router

Exchange Point Medium

Why IXPs?

Multiple service providers Each with Internet connectivity

InternetInternet

AA BB

Why IXPs?

Is not cost effective Backhaul issue causes cost to

both partiesInternetInternet

AA BB

Why IXPs?

Domestic Interconnection

InternetInternet

AA BB

Exchange Structures layer 2 models (the NAP or IXP)

tailored bilateral policies

Why use an IXP?

PEERING Shared medium vs. point-to-point Shared

can exchange traffic with multiple peers at one location via one interface

Point-to-Point for high volumes of traffic

Why use an IXP?

KEEP LOCAL TRAFFIC LOCAL!!! ISPs within a region peer with each

other at local exchange No need to have traffic go overseas

only to come back Much reduced latency and increased

performance

Why use an IXP?

SAVES MONEY!!! Traffic going overseas means transit

charges paid to your upstream ISP Money stays in local economy

Used to provide better local infrastructure and services for customers

Customers pay less for Internet access

Therefore more customers sign up ISP has more customers, better business

Why use an IXP?

VASTLY IMPROVES PERFORMANCE!!! Network RTTs between organisations

in the local economy is measured in milliseconds, not seconds

Packet loss becomes virtually non-existent

Customers use the Internet for more products, services, and activities

Why use an IXP? Countries or regions with a successful

IXP have a successful Internet economy Local traffic stays local Money spent on local ‘net infrastructure Service Quality not an issue

All this attracts businesses, customers, and content providers

The Need for Exchanges

AS 2

AS 1

US

AFRICA

Local traffic travels via the US

The Need for Exchanges

AS 2

AS 1

US

AFRICA

Keep Local Traffic Local!

Exchange Point Design

Ethernet switch Has superseded all other types of

network devices for an IXP From the cheapest and smallest 12 or

24 port 10/100 switch To the largest 32 port 10GigEthernet

switch

What can go wrong? Too many exchange points in one

region competing exchanges defeats the

purpose Becomes expensive for ISPs to

connect to all of them

An IXP is not a competition, it’s not a business

What can go wrong? IXPs try to compete with their

membership IXPs run as a closed privileged club For example:

Interconnecting IXP locations with their own transit service

Providing access to end users rather than just Service Providers

Restrictive membership criteria (closed shop) Interfering with ISP business decisions e.g.

Mandatory Multi-Lateral Peering

Exchange Point policies/politics AUPs

Acceptable Use Policy Minimal rules for connection

Fees? Some IXPs charge no fee Other IXPs charge cost recovery A few IXPs are commercial

Nobody is obliged to peer Agreements left to ISPs, not mandated by

IXP

Exchange Point etiquette

Don’t point default route at another IXP participant

Be aware of third-party next-hop Only announce your aggregate

routes Filter! Filter! Filter!

And do reverse path check

Exchange Point examples

LINX in London, UK Ethernet switches

AMS-IX in Amsterdam, NL Ethernet switches

JPNAP in Tokyo, Japan Ethernet switches

Exchange Points in Africa CR-IX – Cairo, Egypt iBiX – Ibadan, Nigeria JINX – Johannesburg, South Africa KINIX – Kinshasa, Dem Rep of Congo KIXP – Nairobi, Kenya MOZIX – Maputo, Mozambique RINEX – Kigali, Rwanda SZIXP – Mbabane, Swaziland TIX – Dar es Salaam, Tanzania UiXP – Kampala, Uganda

Source: http://www.nsrc.org/AFRICA/afr_ix.html

Features of IXPs Redundancy

multiple switches Support

NOC to provide 24x7 support for problems at the exchange

DNS, Route Collector, Content & NTP servers CCTLD servers Content redistribution systems such as Akamai Route Collector – Routing Table view

Features of IXPs Location

neutral co-location facilities Address space

Peering LAN AS

If using Route Server (Route servers) Statistics

Traffic data – for membership

More info about IXPs

http://www.ep.net/ep-main.html Excellent resource for ip address

allocation for exchanges, locations of XPs in the world, AUPs and other policies

http://www.pch.net/documents Another excellent resource of IXP

locations, papers, IXP statistics, etc

Things to think about...

Do you need to be at an Exchange Point?

Would you want to start an Exchange Point?

Would keeping local traffic local benefit your ISP?

Would your environment (politically, etc.) support an Exchange Point?

Discussion

How would you build an exchange point in your environment?

Who would connect? What services would you provide? What policies would you enforce? What does your environment look

like? Is it feasible to set up an IXP?

Important to Remember...

Exchange Points can be as simple as an ethernet HUB!!!!

Keeping local traffic local improves performance cheaper often simple to do!

Exercise

Building an IXP

AS 1

BGP to provider

AS 3

AS 5

AS 7

AS 9

AS 2

AS 4

AS 6

AS 8

AS 10

AS100 AS200

AS 1

AS 3

AS 5

AS 7

AS 9

AS 2

AS 4

AS 6

AS 8

AS 10

196.200.220.224/28

SWITCH

Ethernet to IXP

Introduction to Route Collectors

Route Collector Background

What is a Route Collector? Features of a Route Collector Purpose of a Route Collector IXP Design with a Route Collector

What is a Route Collector?

Usually a router or Unix box running BGP

Gathers routing information from service provider routers at an IXP

Does not forward packets

Purpose of a Route Collector

To provide a public view of the Routing Information available at the IXP Useful existing members to check

functionality of BGP filters Useful for prospective members to check

value of joining the IXP Useful for the Internet Operations

community for troubleshooting purposes E.g. www.traceroute.org

Route Collector at an IXP

IXP R3R2

R1

ROUTE Collector

Route Collector Requirements Router or Unix system running BGP Peers eBGP with every IXP member

Accepts everything; Gives nothing Uses a private ASN Connects to IXP Transit LAN

“Back end” connection Second Ethernet globally routed Connection to IXP Website for public

access

Route Collector Implementation

Most IXPs now implement some form of Route Collector

Benefits already mentioned Great public relations tool Unsophisticated requirements

Just runs BGP

Introduction to Route Servers

Route Collector plus more

Route Server Background

What is a Route Server? Features of a Route Server Advantages of using a Route

Server Exchange Point Design with a

Route Server

What is a Route Server?

All the features of a Route Collector But also:

Announces routes to participating IXP members according to their routing policy definitions

Implemented using the same specification as for a Route Collector

Features of a Route Server Helps scale routing Simplifies Routing Processes on ISP

Routers Insertion of RS Autonomous System

Number in the Routing Path Handling of Multi-Exit Discriminator Route Flap Damping Uses Policy registered in IRR

Diagram of N-squared Mesh

With the Route Servers

RS based Exchange Point Routing Flow

TRAFFIC FLOW ROUTING INFORMATION FLOW

Advantages of Using a Route Server

Helps scale Routing Separation of Routing and

Forwarding Simplify Routing Configuration

Management on ISPs routers Enforce Good Routing Engineering Helps prevent the spread of bogus

routing information!

Disadvantages of Using a Route Server

ISPs lose direct policy control Dependent on 3rd party for

configuration and troubleshooting Insertion of RS Autonomous

System Number in the Routing Path

Peering with the Route Servers

Any ISP attached to an IXP can peer with the Route Servers

ISP must register their policy in the Internet Routing Registry Most IXPs who provide the RS facility

also provide a local IRR for policy registration

Must use BGP

Things to think about...

Would using a route server benefit you? Helpful when BGP knowledge is

limited Avoids having to maintain a large

number of eBGP peers

Introduction to the IRR

The Internet Routing Registry

What is the Routing Registry Contact names, email addresses and

telephone numbers for an AS Routing policy for an AS (what other

ASes does it connect to, which routes do they exchange)

Information about routes (most important is which AS originates the route)

Several other types of information

What is the Routing Registry?

Distributed database collectively known as Internet Routing Registry (IRR) APNIC, RIPE, ARIN, RADB, etc http://www.irr.net/docs/list.html

Providers register routing policy Used for planning, debugging and

generating backbone router configs

What is the Routing Registry?

Can be used by anyone worldwide debugging configuring engineering routing addressing

What happens if I don’t use the IRR

Routing Horror Stories AS7007 announcing bogus routes

Inconsistent policy at network borders Peers and upstreams need physical

notification of policy changes Mistakes easily made

So, I need to use the database because…..

Filters generated off the IRR protect against inaccurate routing information

Makes troubleshooting and debugging easier

Keep track of policy Security Filter! Filter! Filter!!

Why Bother using the IRR?

View of global routing policy in a single cooperatively maintained database

to improve integrity of Internet’s routing

generate router configs protect against inaccurate routing info

distribution verification of Internet routing

Why Bother using the IRR?

Many providers require that you register your policy (or they won’t peer with you)

Describing Policy

Use the policy languages to describe your relationship with other Peers routes importing routes exporting specific policies

interfaces, MEDs, communities

register routes with origin AS

Querying the Database

whois -h whois.ripe.net AS702 whois -h whois.ripe.net AS1849-

MAINT whois -h whois.ripe.net 158.43.0.0

How to Register your IRR policy

Register one or more maintainers Register AS and policy information Register Routes Describes your import and export

policy At the very least, provides contact

information

Router Configuration

Currently configs by hand - slow and inaccurate

Configuring routers using the IRR lots of tools available!!! IRRToolSet maintained by ISC

route and Aspath filters. Import and export

Filtering is a good thing...

Router Configuration

IRR

Configuration Machine

IRR Database

Server

Router

How do I use the IRR to generate configurations

Tools available to generate config files for most BGP implementations

IRRToolSet http://www.isc.org/sw/IRRToolSet/ Started off as RAToolSet as a project of

ISI Moved to RIPE NCC custodianship and

became IRRToolSet Enhanced to support RPSL (RFC2622)

Now maintained by ISC

How do I participate?

Set up your own registry Private for your ISP? Community for the region? Download the software (from ISC)

Use one of the many public IRR systems Ask AfriNIC to set one up?

Things to think about...

How would you register your policy? Try to describe it in an aut-num object

How would registering your policy benefit you? The community?