SIX and some best practices for running an IXP

Matjaž Straus Istenič, SIX, [email protected]

SIX and some best practices for running an IXPAll that stuff around the switch

1sreda, 04. december 13

mailto:[email protected]


Matjaž Straus Istenič, 8.9.2011

About me

2



About me

2



About me

2

Triglav, 2864 m



Agenda• Slovenian Internet Exchange - SIX• all that stuff around the switch• practical examples

– addressing– best practices– configuration examples for the IXP– configuration examples, guidelines and hints for members

2/417


Matjaž Straus Istenič, SEE2, Macedonia, 4/2013

SIX - operated by ARNESsince 1994

photo: http://www.pivo-lasko.si/

4

Slovenia has tradition - not only in breweries


http://www.pivo-lasko.si

http://www.pivo-lasko.si


SIX - the history

5

• started in february 1994- two members

• 1995: two more members• 1996: two more...

– and the big Telecom• 1997 ... 2002: more alternative providers• 2000: second location, interconnect with the first at 2001• 2003: third location (LIX, decommissioned 3/2012)• 2006: first IPv6 at SIX• 2009: new location at Ljubljana Technology Park



SIX - the forbidden graphs

6

0

3

6

9

12

15

18

21

24

27

30

1994 1996 1998 2000 2002 2004 2006 2008 2010 2012

SIX members

• 26 members• > 50% with 10 G• most with IPv6



SIX - the IXP technology

7



SIX - today• L2• two locations• *national• Cisco 4500X• bird route server• IXP manager and portal

8*note: one cross-border link



Stuff around the switch• proper location with many fibre providers

– a building with one single provider is a bad idea• different fibre paths inside of the building• power supplies and grounding• cooling system• physical security• staff, support, remote hands• good and accurate documentation

9



Stuff around the switch (cont.)

• monitoring and alarming• ticketing system• mailing lists• web portal• best current practices and knowledge base• contracts, SLAs, billing, ...

• planning for a collocation/datacenter

10



The power

11



The power• allocate up to 20 kW per rack• actual usage 5 kW - 10 kW per rack• dual separate circuit breaker for each rack• power supply redundancy

– dual feed from electrical distribution company– separate dual UPS system N+1 and PDU– diesel generator (redundant)

• cooling equipment is independently dual powered, including chillers

• how much power does datacenter use– monitoring on UPS, on PDU– monitoring total on main branch circuit

• typicaly the load will double in 5 years

12



Cooling• full redundancy of cooling system

– two different power grids– separate piping– chiller redundancy– room units redundancy

• hot/cold isle– reduce air mixing– cold aisle with barriers made of metal, plastic or fiberglass– use blanking panels on the cabinets without servers

• no need for double floor– run network cabling over the top of the cabinets– "in row" cooling

• recommended temperature in cold isle is between 23 - 25 °C• cooling system rating must be 1.3 x IT load rating• make sure that the space will allow for future growth

– for more cooling capacity and redundancy if required• Power usage effectiveness (PUE = Total Facility Power/IT Equipment Power)

– typical PUE is 2.0 or higher

13


Matjaž Straus Istenič, 8.9.2011 14




Fire protection• sensing the smoke/fire

16

type ✔ ✗

aspiration sensor

• very sensitive• early warning• single point of electrical

instalation• targeted sensing is possible

• more expensive• air ducting under the ceiling

must be installed

optical sensor• cheaper• can be used as confirmation

for fast aspiration sensors

• less sensitive• each sensor needs its own

cable



Fire protection• extinguishing fire

17

Gaseous fire extinguishing systemAll are considered safe for breathing after release, although, products of burning plastics are always dangerous!Gaseous fire extinguishing systemAll are considered safe for breathing after release, although, products of burning plastics are always dangerous!Gaseous fire extinguishing systemAll are considered safe for breathing after release, although, products of burning plastics are always dangerous!Gaseous fire extinguishing systemAll are considered safe for breathing after release, although, products of burning plastics are always dangerous!

type active substance ✔ ✗

displacement of air

Inergen- mixture of gases, displaces air with “air” with less oxygen

• totally natural• environmentaly

neutral

• big storage requirements• high pressure (200 or 300 bar)• computer room needs big exhaust vents• big rush of gas at release causes dust and

objects to lift

chemical action

Novec 1230- chemical bonding, cooling

• small storage area• stored as fluid• very small greenhouse

gas footprint

• has some effect on environment• expensive• stored under pressure (40/50 bar)chemical

actionFM200 (phasing out)- chemical bonding

• small storage area• small greenhouse gas

footprint

• being phased out• has some ozone depletion impact• stored under pressure (40/50 bar)

cooling water mist• totally natural• environmentaly

neutral

• water in computer room is not a good idea ;-)• possible condensation on cold surfaces



Fire protection• extinguishing fire

17

Gaseous fire extinguishing systemAll are considered safe for breathing after release, although, products of burning plastics are always dangerous!Gaseous fire extinguishing systemAll are considered safe for breathing after release, although, products of burning plastics are always dangerous!Gaseous fire extinguishing systemAll are considered safe for breathing after release, although, products of burning plastics are always dangerous!Gaseous fire extinguishing systemAll are considered safe for breathing after release, although, products of burning plastics are always dangerous!

type active substance ✔ ✗

displacement of air

Inergen- mixture of gases, displaces air with “air” with less oxygen

• totally natural• environmentaly

neutral

• big storage requirements• high pressure (200 or 300 bar)• computer room needs big exhaust vents• big rush of gas at release causes dust and

objects to lift

chemical action

Novec 1230- chemical bonding, cooling

• small storage area• stored as fluid• very small greenhouse

gas footprint

• has some effect on environment• expensive• stored under pressure (40/50 bar)chemical

actionFM200 (phasing out)- chemical bonding

• small storage area• small greenhouse gas

footprint

• being phased out• has some ozone depletion impact• stored under pressure (40/50 bar)

cooling water mist• totally natural• environmentaly

neutral

• water in computer room is not a good idea ;-)• possible condensation on cold surfaces



Examples and guidelines• addressing• IXP port configuration• guidelines for members• goodies

18



Examples: addressing• a single subnet taken from independent address space

– member address is assigned per location• address schema at SIX

19

91.220.194.n/24n = n1 = 2..99 at location 1n = n1 + 100 = 102..199 at location 2n = 1, 101 for route-reflectors

2001:7f8:46:0:L:N::<AS>/64L = 0 at location 1L = 1 at location 2N = 0 for a single router, otherwise N = 1, 2, ...AS = member AS in decimalAS = 51988 for route-server- diverse lower 24 bits which

form solicited-node mcast address



Examples: IXP port configuration• access port on Cisco 4500X

20

interface TenGigabitEthernet1/6 description -- member (AS...) -- switchport access vlan <x> switchport mode access switchport nonegotiate switchport port-security load-interval 30 datalink flow monitor FlowMonitor-L2 input storm-control broadcast level 1.00 storm-control action shutdown no cdp enable spanning-tree portfast spanning-tree bpduguard enable service-policy input INPUT-200M-EF!policy-map LIMIT-QUEUE-200 class class-default queue-limit 200!

flow record StandardFlow-L2 match datalink mac source address input match datalink mac destination address input collect interface input collect interface output collect counter bytes long collect counter packets collect timestamp sys-uptime first collect timestamp sys-uptime last!flow exporter FlowExporter destination <x.y.z.w> vrf mgmtVrf source FastEthernet1 transport udp <port> template data timeout 60!flow monitor FlowMonitor-L2 record StandardFlow-L2 exporter FlowExporter cache timeout active 60!



Examples: IXP port configuration• interconnecting ports

– aggregated to EtherChannel with LACP– maximal MTU

21

interface TenGigabitEthernet1/1 switchport access vlan <N> switchport mode access switchport nonegotiate mtu 9198 load-interval 30 datalink flow monitor FlowMonitor-L2 input channel-protocol lacp channel-group 48 mode active!interface TenGigabitEthernet1/2 switchport access vlan <N> switchport mode access switchport nonegotiate mtu 9198 load-interval 30 datalink flow monitor FlowMonitor-L2 input channel-protocol lacp channel-group 48 mode active!

interface Port-channel48 description -- IX-trunk -- switchport switchport access vlan <N> switchport mode access switchport nonegotiate mtu 9198 bandwidth 10000000 load-interval 30 datalink flow monitor FlowMonitor-L2 input flowcontrol receive on!port-channel load-balance src-dst-ip



Guidelines for members• the bads (proxy ARP, redirects)• access port configuration• BGP

– routing considerations (next-hop, localization)– safety (MD5 authentication)– policy (filtering announcements)

– control received prefixes– control advertised prefixes

– RPKI

22


Presenter Name, Date

Proxy ARP in action• incident at AMS-IX, DE-CIX, ...

23

- proxy ARP enabled

- router has no IP address from the peering LAN- router has a default route

or

- router has a more specific route for the peering LAN

reference: Maksym Tulyuk, Wolfgang Tremmel, reported at RIPE63http://ripe63.ripe.net/presentations/


http://ripe63.ripe.net/presentations/

http://ripe63.ripe.net/presentations/


ARP hijacking• no RS, full BGP mesh between R2, R3 in R4• normal situation

24source: ARP Hijacking Mitigation, 19th Euro-IX Forum, 10/2011, Steven Bakker @ams-ix.net

R1MAC A

R4MAC D

R3MAC C

R2MAC B

R1 is at AR3 is at CR4 is at D

R1 is at AR2 is at BR4 is at D

R1 is at AR2 is at BR3 is at C

R2 is at BR3 is at CR4 is at D

BGP

BGP

BGP



ARP hijacking (2/8)

• R1 send bogus ARP replies


R1MAC A

R4MAC D

R3MAC C

R2MAC B

R1 is at AR3 is at CR4 is at D

R1 is at AR2 is at BR4 is at D


R2 is at AR3 is at AR4 is at A

R2 is at BR3 is at CR4 is at D

BGP

BGP

BGP



ARP hijacking (3 /8)

• ARP cache poisioned• BGP down• traffic stops


R1MAC A

R4MAC D

R3MAC C

R2MAC B






ARP hijacking (4/8)

• hijacker R1 isolated• ARP caches recover with BGP packets• BGP up• traffic normalizes after a few

minutes


R4MAC D

R3MAC C

R2MAC B

R3 is at CR4 is at D

R2 is at BR4 is at D

R2 is at BR3 is at C

BGP

BGP

BGP



What if route-server is being used?


• RS, partial BGP between RS and R2, R3• normal situation

RSMAC D

R3MAC C

R2MAC B

R1MAC A

R1 is at AR3 is at CRS is at D

R1 is at AR2 is at BRS is at D


BGP

BGP

R2 is at BR3 is at CRS is at D


Presenter Name, Date 29source: ARP Hijacking Mitigation, 19th Euro-IX Forum, 10/2011, Steven Bakker @ams-ix.net

• R1 send bogus ARP replies

RSMAC D

R3MAC C

R2MAC B

R1MAC A

R1 is at AR3 is at CRS is at D

R1 is at AR2 is at BRS is at D


BGP

BGP

R2 is at AR3 is at ARS is at A

R2 is at BR3 is at CRS is at D

ARP hijacking (6/8)



ARP hijacking (7/8)


• ARP cache poisioned• BGP with RS down• traffic stops

RSMAC D

R2MAC B

R3MAC C

R1MAC A






ARP hijacking (8/8)

• hijacker R1 isolated• ARP caches partially recover with BGP packets• BGP up• traffic is being

blackholed– R2 and R3 still have bogus entries

for each other

– outage can last for hours


RSMAC D

R3MAC C

R2MAC B

R1 je na AR3 je na ARS je na D

R1 je na AR2 je na ARS je na D

R2 je na BR3 je na C

BGP

BGP

R1MAC A



ARP Sponge• update ARP caches• mitigates unknown unicast

32

from to message

sponge B reply: R3 is at C

sponge B request: where is R3? - tell R3 at C

sponge B request: where is R2? - tell R3 at C

B C reply: R2 is at B

source: ARP Hijacking Mitigation, 19th Euro-IX Forum, 10/2011, Steven Bakker @ams-ix.net

• spoofed unsolicited ARP reply• spoofed gratuitous ARP query• spoofed ARP request

(dve muve jednim udarcem ;-))

• 3 ARP updatemethods: R3

MAC C

R2MAC B

R3 is at C

R2 is at B

R2 is at BR3 is at Cin case of unknown unicast: - the unknown is here



eBGP

eBGP

R1

R2

R3

A

B

2001:db8::/48

next-

hop(2

001:d

b8::/4

8) = B

next-hop(2001:db8::/48) = A or B?

Tuđe nećemo, ...

33

• R2 peers with R1 but not with R3

• R2 doesn’t want to receive any traffic from R3



eBGP

eBGP

R1

R2

R3

A

B

2001:db8::/48

next-

hop(2

001:d

b8::/4

8) = B

next-hop(2001:db8::/48) = A or B?

Tuđe nećemo, ...

33





eBGP

eBGP

R1

R2

R3

A

B

2001:db8::/48

next-

hop(2

001:d

b8::/4

8) = B

next-hop(2001:db8::/48) = A or B?

Tuđe nećemo, ...

33

✔ preffered path✗ path to avoid





eBGP

eBGP

R1

R2

R3

A

B

2001:db8::/48

next-

hop(2

001:d

b8::/4

8) = B

next-hop(2001:db8::/48) = A or B?

Tuđe nećemo, ...

33

✔ preffered path✗ path to avoidnext-hop self in eBGP✔



with next-hop self at R1 next-hop for 2001:db8::/48 at R3 is A, not B



eBGP

eBGP

R1

R2

R3

A

B

2001:db8::/48

next-

hop(2

001:d

b8::/4

8) = B

next-hop(2001:db8::/48) = A or B?

... svoje ne damo!

34

✔ preffered path✗ path to avoid

• R1 receives traffic and sends it back via the same port

• R1 doesn’t want to redirect any traffic to R2



eBGP

eBGP

R1

R2

R3

A

B

2001:db8::/48

next-

hop(2

001:d

b8::/4

8) = B

next-hop(2001:db8::/48) = A or B?

... svoje ne damo!

34

no ip redirects✔✔ preffered path✗ path to avoid

• R1 receives traffic and sends it back via the same port

• R1 doesn’t want to redirect any traffic to R2

ICMP redirect messages should not be sent



Unreachables and PMTU discovery

35

9000

1500

ICMP 3/4Packet too big, fragmentation required and DF flag set


http://en.wikipedia.org/wiki/IPv4#Packet_structure



Unreachables and PMTU discovery

35

9000

1500

ICMP 3/4Packet too big, fragmentation required and DF flag set

ICMP unreachables are always sent for IPv4

note:In IPv6, ICMP Packet-too-big message is not an “Unreachable”





Example: member port configuration• turn off anything but IP and ARP

36

! example for Cisco IOS!interface TenGigabitEthernet3/3 ip address x.y.z.w 255.255.255.0 ip access-group IxIncoming in ip access-group IxOutgoing out no ip redirects no ip proxy-arp ipv6 address 2001:.../64 ipv6 enable ipv6 traffic-filter IxIncoming6 in ipv6 traffic-filter IxOutgoing6 out ipv6 nd reachable-time 300000 ipv6 nd ra suppress no ipv6 redirects storm-control broadcast level 1.00 no cdp enable!

– no proxy ARP– no redirects– no vendor proprietaryprotocols like CDP

– no broadcasts– no IPv6 RA– ICMP unreachables areused in PMTU discovery in IPv4



Multiple locations• routing considerations

– localize traffic– minimize traffic between locations

37

IXP LANinterconnect

A A

B B

C D

IXP @location 1 IXP @location 2





37

IXP LANinterconnect

A A

B B

C D






37

IXP LANinterconnect

A A

B B

C D




Localization

38

membe

r at o

ne

locati

on

membe

r at tw

o loc

ation

siB

GP

eBGP

eBGPR1

R2

R3

A

B

2001:db8::/48

next-hop(2001:db8::/48) = A

next-hop(2001:db8::/48) = B

next-hop(2001:db8::/48) = R2



Localization

38

membe

r at o

ne

locati

on

membe

r at tw

o loc

ation

siB

GP

eBGP

eBGPR1

R2

R3

A

B

2001:db8::/48

next-hop(2001:db8::/48) = A

next-hop(2001:db8::/48) = B

next-hop(2001:db8::/48) = R2



Localization

38

membe

r at o

ne

locati

on

membe

r at tw

o loc

ation

siB

GP

eBGP

eBGPR1

R2

R3

A

B

2001:db8::/48

next-hop(2001:db8::/48) = A

next-hop(2001:db8::/48) = B

next-hop(2001:db8::/48) = R2

I’m marking my prefixes with community for blue location

I’m marking my prefixes with community for red location

I preffer prefixes with red community

• use BGP communities



Examples: localization• Cisco IOS

39

! router at location 1ip community-list 61 permit 65432:1!route-map AnnounceToIX permit 10 set community 65432:1!route-map AcceptFromIX permit 10 ! this location match community 61route-map AcceptFromIX permit 20 ! other location - worse metric set metric +1!router bgp <member-AS> template peer-policy IX route-map AcceptFromIX in route-map AnnounceToIX out next-hop-self send-community! address-family ipv4|6 neighbor <R1> inherit peer-policy IX neighbor <R2> inherit peer-policy IX!

! router at location 2ip community-list 62 permit 65432:2!route-map AnnounceToIX permit 10 set community 65432:2!route-map AcceptFromIX permit 10 ! this location match community 62route-map AcceptFromIX permit 20 ! other location - worse metric set metric +1!router bgp <member-AS> template peer-policy IX route-map AcceptFromIX in route-map AnnounceToIX out next-hop-self send-community! address-family ipv4|6 neighbor <R1> inherit peer-policy IX neighbor <R2> inherit peer-policy IX!



/* router at location 1 */protocols { bgp { local-as <member-AS>; group Ix { type external; import [ LocalizeTraffic AcceptFromIx ]; export AnnounceToIx; } }}policy-options { policy-statement AcceptFromIx { <member policy at receive> } policy-statement AnnounceToIx { term Localize { then { community set IxLocation1; next term; } } <member policy for announcements> } policy-statement LocalizeTraffic { term LocalTraffic { from community IxLocation1; then next policy; } term OtherTraffic { then { metric add 1; } } } community IxLocation1 members 65432:1;}

Examples: localization• Juniper JUNOS

40

/* router at location 2 */protocols { bgp { local-as <member-AS>; group Ix { type external; import [ LocalizeTraffic AcceptFromIx ]; export AnnounceToIx; } }}policy-options { policy-statement AcceptFromIx { <member policy at receive> } policy-statement AnnounceToIx { term Localize { then { community set IxLocation2; next term; } } <member policy for announcements> } policy-statement LocalizeTraffic { term LocalTraffic { from community IxLocation2; then next policy; } term OtherTraffic { then { metric add 1; } } } community IxLocation2 members 65432:2;}



BGP• authenticate (secure) BGP session• filter announcements• sanity checks• a must-read

– BGP Operations and Securityhttp://tools.ietf.org/id/draft-jdurand-bgp-security-02.txt

– Internet Exchange Route Server Operationhttp://tools.ietf.org/html/draft-ietf-grow-ix-bgp-route-server-operations-01

41


http://tools.ietf.org/id/draft-jdurand-bgp-security-02.txt

http://tools.ietf.org/id/draft-jdurand-bgp-security-02.txt

http://tools.ietf.org/html/draft-ietf-grow-ix-bgp-route-server-operations-01

http://tools.ietf.org/html/draft-ietf-grow-ix-bgp-route-server-operations-01


Example: BGP filters

42

router bgp 65432 template peer-policy Ix6 route-map AcceptFromIx in route-map AnnounceToIx out filter-list 200 out prefix-list FromIx6 in prefix-list ToIx6 out next-hop-self remove-private-as maximum-prefix 1000 send-community! template peer-session Ix6 password <default_key> ttl-security hops 1 update-source Vlan50! neighbor <...> remote-as 65000 address-family ipv6 neighbor <...> inherit peer-policy Ix6 neighbor <...> inherit peer-session Ix6 neighbor <...> password <another_key> neighbor <...> filter-list 100 in...!

• remove your own communities <your-as>:<community>• accept only communities that are meaningful for you• respect “no-export”• do not remove other communities for no reason

• properly mark your prefixes

• limit the number of accepted prefixes(beware of the full routing table!)

• authenticate with MD5• TTL security (optional)



Example: prefix filters

43

router bgp 65432 template peer-policy Ix6 filter-list 200 out prefix-list FromIx6 in prefix-list ToIx6 out ... exit-peer-policy ! neighbor <...> remote-as 65000 address-family ipv6 neighbor <...> inherit peer-policy Ix6 neighbor <...> filter-list 100 in!ipv6 prefix-list FromIx6 seq 5 deny ::/0ipv6 prefix-list FromIx6 seq 10 deny <our-prefix>/32ipv6 prefix-list FromIx6 seq 15 deny <our-prefix>/32 ge 33ipv6 prefix-list FromIx6 seq 15 deny ::/0 ge 48ipv6 prefix-list FromIx6 seq 20 deny 2002::/16 ge 17ipv6 prefix-list FromIx6 seq 99 permit 2000::/3 ge 4!ipv6 prefix-list ToIx6 seq 5 permit <our-prefix>/32ipv6 prefix-list ToIx6 seq 10 permit <customer1>/32ipv6 prefix-list ToIx6 seq 15 permit <customer2>/48...!ip as-path access-list 100 permit ^(65000_)+$ip as-path access-list 100 permit ^(65000_)+.*(65001_)+$ip as-path access-list 100 permit ^(65000_)+.*(65002_)+$!ip as-path access-list 200 permit ^$ip as-path access-list 200 permit ^(<our-customer1-AS>_)+$ip as-path access-list 200 permit ^(<our-customer2-AS>_)+$

• beware of the default route!• desi se i najboljima ;-)

• do not accept your own prefixes - they should stay at home• do not accept too specific prefixes• SIX policy: /8 .. /25 for IPv4, /16 .. /48 for IPv6

• block martians• announce your own and nothing else



Register your route objects

44

$ whois -h whois.ripe.net -- '-i or AS2107' | grep ^routeroute: 109.127.192.0/18route: 141.255.192.0/18route: 149.62.64.0/18route: 153.5.0.0/16route: 164.8.0.0/16route: 164.8.0.0/17route: 164.8.128.0/17route: 164.8.128.0/20route: 178.172.0.0/17route: 185.13.52.0/22route: 193.138.1.0/24route: 193.138.2.0/24route: 193.2.0.0/16route: 194.249.0.0/16route: 212.235.128.0/17route: 88.200.0.0/17route: 92.244.64.0/19route: 95.87.128.0/18route6: 2001:1470::/29route6: 2001:1470::/32



Example: What is registered at RIPE?

45

$ peval -h whois.ripe.net -protocol ripe -no-as AS-ARNES((AS28933 AS2121 AS51988 AS42909 AS12785 AS50195 AS2107 ))

$ peval -h whois.ripe.net -protocol ripe 'afi ipv6 AS-ARNES'({2A00:1600::/32, 2A00:1368::/32, 2001:1470::/32, 2001:7F8:46::/48, 2001:67C:64::/48, 2001:678:4::/48, 2001:678:5::/48})

• peval– list of ASNs at the end of the AS-PATH– list of prefixes



Example: What is registered at RIPE?

46

• public whois servers

$ whois -h filtergen.level3.net -- "-v6 RIPE::RS-ARNES-HOSTED"Prefix list for policy RIPE::RS-ARNES-HOSTED = RIPE::RS-ARNES-HOSTED

2001:503:c27::/482001:503:231d::/482001:658:4::/482001:658:5::/482001:67c:44::/482001:7f8:46::/48



Register your peering information

47



Register your peering information

47https://www.peeringdb.com/


https://www.peeringdb.com

https://www.peeringdb.com


RPKI-based BGP route origin validation

48



RPKI-based BGP route origin validation

48

https://certification.ripe.net/


https://certification.ripe.net

https://certification.ripe.net


Goodies• route server (reflector)• IXP manager• looking-glass router• graphs

– public– or members only– or private

• meetings :-)

49



Route server• use route server from day 1

• SIX uses bird - http://bird.network.cz• how it works• goodies– enforced policy– community based prefix filtering– “automatic” localization

50


http://bird.network.cz

http://bird.network.cz


Bird at SIX

51

member 1

R1

R101

member 2

R2

R102

MASTERRIBRS

R101 RTBGP

R2 RT

R102 RT

BGP

BGP

R1 RTBGP

R101 POLICY

R102 POLICY

R2 POLICYR1 POLICY



Bird at SIX

51

member 1

R1

R101

member 2

R2

R102

MASTERRIBRS

R101 RTBGP

R2 RT

R102 RT

BGP

BGP

R1 RTBGP

R101 POLICY

R102 POLICY

R2 POLICYR1 POLICY

All the magic happens here!



Bird at SIX

51

member 1

R1

R101

member 2

R2

R102

MASTERRIBRS

R101 RTBGP

R2 RT

R102 RT

BGP

BGP

R1 RTBGP

R101 POLICY

R102 POLICY

R2 POLICYR1 POLICY

All the magic happens here! All valid routes are here.



Bird at SIX

51

member 1

R1

R101

member 2

R2

R102

MASTERRIBRS

R101 RTBGP

R2 RT

R102 RT

BGP

BGP

R1 RTBGP

R101 POLICY

R102 POLICY

R2 POLICYR1 POLICY

All the magic happens here!

Policy (pipe) filters received routes, marks them, adjusts preference according to location, filters advertised routes.

All valid routes are here.



Route server• improved (enforced) security– filtering based on routing registry–matching on prefix and origin AS

– blocks martians– blocks default– blocks more specifics

5227



Route server• improved (enforced) security– filtering based on routing registry–matching on prefix and origin AS

– blocks martians– blocks default– blocks more specifics

5227

It is you who is

responsible for the

security of your

network !



Example: route server - custom filtering• based on BGP communities

53

description community extended community

Prevent announcement of a prefix to a peer 0:peer-as soo:0:peer-as

Announce a route to a certain peer 51988:peer-as soo:51988:peer-as

Prevent announcement of a prefix to all peers 0:51988 soo:0:51988



Example: route server - localization• we adjust the route preference according to AS_PATH length

54

member 1

R1

R101

member 2

R2

R102

MASTERRIBRS

R101 RTBGP

R2 RT

R102 RT

BGP

BGP

R1 RTBGP

R101 POLICY

R102 POLICY

R2 POLICYR1 POLICY



Example: route server - localization• we adjust the route preference according to AS_PATH length

54

member 1

R1

R101

member 2

R2

R102

MASTERRIBRS

R101 RTBGP

R2 RT

R102 RT

BGP

BGP

R1 RTBGP

R101 POLICY

R102 POLICY

R2 POLICYR1 POLICY

import from member RT/BGP to master RIB:preference = 100;if bgp_path.len > 50 then preference = 0; else preference = 100 - ( 2 * bgp_path.len );export from master RIB to member RT/BGP:if same_location() then preference = preference + 1;



IXP Manager• portal and RS manager

55



IXP Manager• portal and RS manager

55

https://github.com/inex/IXP-Manager/wiki





Looking glass

56screenshot from NLNOG RING

http://lg.ring.nlnog.net/


http://lg.ring.nlnog.net



Looking glass

56

https://github.com/sileht/bird-lg/

screenshot from NLNOG RINGhttp://lg.ring.nlnog.net/







Looking glass• ...or, at least, route-collector

56

https://github.com/sileht/bird-lg/

[email protected]:rc> show route aspath-regex 2107.* active-path terse table inet6.0

inet6.0: 132 destinations, 330 routes (132 active, 0 holddown, 0 hidden)Restart Complete+ = Active Route, - = Last Active, * = Both

A Destination P Prf Metric 1 Metric 2 Next hop AS path* 2001:678:4::/48 B 170 1 0 >2001:7f8:46:0:1::2107 2107 42909 I* 2001:678:5::/48 B 170 1 0 >2001:7f8:46:0:1::2107 2107 42909 I* 2001:1470::/29 B 170 1 0 >2001:7f8:46:0:1::2107 2107 I* 2001:1470::/32 B 170 1 0 >2001:7f8:46:0:1::2107 2107 I* 2a00:1600::/32 B 170 1 0 >2001:7f8:46:0:1::2107 2107 50195 I* 2a00:d440::/29 B 170 1 0 >2001:7f8:46:0:1::2107 2107 58046 I

screenshot from NLNOG RINGhttp://lg.ring.nlnog.net/









Graphs

57



Graphs

57



Graphs

57



Graphs

57



Graphs

57



Graphs

57

Collect data

and graph as

much as you can



Meet the community

58CC EssjayNZ/flickr


Thank you!
