Bandwidth management and optimization

1

Bandwidth management and optimization

BCrouter

14-16 March 2006Dirk Janssens

ICTS – K.U.Leuven

2

Introduction into introduction

BCrouter is an ongoing network project Not all features are already implemented or

ready for 3th party deployment

Constructive feedback What do you expect from a good solution

Try to fulfill as many expectations as possible

3

Overview1. Introduction

Problem Expectations BCrouter solution

2. BCrouter solution Components Example network setups Integration Security considerations

3. BCrouter Introduction Components Commands and logging Routing and Netfilter setup Quota/Bandwidth exceptions

4. BCpolicer Introduction Design principles Policing alternatives Complete design

5. Case study: KotNet6. Development

Current status Future Wish list

4

Introduction: problem

Bandwidth usage rises rapidly Increasing Internet population ‘Richer’ content (HTML,Flash,…) P2P download applications Video/music streaming

Bandwidth availability is limited Expensive uplink

• No alternatives Expensive hardware

5

Introduction: problem

Majority of bandwidth used by minority of users

Minority of users cause network congestion cause problems for other users

Example: K.U.Leuven KotNet Student network across region of Leuven 20 000 active students 5% of users caused 50% of used bandwidth

6

Introduction: problem Users are anonymous

Only known by IP address• Very easy to change IP address to be anonymous

Everyone can (ab)use the network What to do if external complaints come in?

User awareness is needed Let the user take responsibility of his own network usage

• Give the user a ‘personal credit’ he can use (network quota)• Notify/block the user if his/her PC acts ‘strange’ and give instructions

Answer: User authentication Makes it possible to map every action on the network to an

individual person Prevents unauthorized access Makes it possible to use ‘personal’ network settings and actions

7

Introduction: expectations Login system

Each user must authenticate him/herself before using the network No extra software or configuration needed on client hosts

Bandwidth regulation Works for all protocols and traffic Prevent that a minority of users take away all the bandwidth for

the majority of users Allow exceptions to certain (educational) sites

• E.g. OS security updates, e-learning site… Maximize responsiveness for interactive traffic

• E.g. Slow down bulk traffic, but don’t touch SSH unless really needed

Every user and/or IP can have its own personal bandwidth settings• E.g. Different settings for a lab computer and personal PC

Distribute the individual bandwidth over the individual active network connections

8

Introduction: expectations Volume quota

Every user and/or IP is only allowed to use a certain fixed amount of traffic

• Learns the user how to manage his Internet behavior Slow down traffic when a user and/or IP generates too much

traffic Every user and/or group and/or IP can have its own personal quota

settings• E.g. personal vs. lab PC, limited guest accounts...

A user and/or IP is never blocked from the network (real-time small band)

If a user and/or IP who is on 'small band' stops downloading for a few minutes, the user immediately can use a limited amount of traffic again at normal speed.

9

Introduction: BCrouter solution

Why? Didn’t find another solution that fulfills all the

expectations• No open source projects

• Commercial black boxes not really an option

• It’s interesting, fun and challenging High performance needed

• Old quota/login system was maxed out

• Network usage still increases

10

Introduction: BCrouter solution Features

User login system• ‘Unlimited’ number of users• Users can login multiple times at different location

Group based routing• ‘Unlimited’ number of user groups possible• Every group has its own independent routing and policy

Bandwidth regulation and volume quota• Individual user/group and IP address based settings with no

performance impact• Prevent network congestion by dynamically regulating maximum

bandwidth• Powerful quota and bandwidth exception possibilities

User friendly• No user side configuration needed• Nice user webpage with information and history information• Automatically redirect to login site for login

11


Quota/bandwidth limiting to both user and IP Example 1:

• Assign user:– Quota of 1 Gigabyte– Refill the quota at rate of 1 Gigabyte/month– Maximum speed: unlimited

• Assign IP:– Quota of 10 Mbyte– Refill the quota at rate of 5 Kilobytes/second– Maximum speed: 20 Kilobytes/sec

• Result:– User settings to determine the maximum volume a user can

download each month– IP settings to limit the ‘real-time’ bandwidth usage

12


Quota/bandwidth limiting to both user and IP Example 2:

• Assign user:– Unlimited quota

– Maximum speed: 50 Kilobytes/second

• Assign IP:– Quota of 10 Mbyte

– Refill the quota at rate of 5 Kilobytes/second

– Maximum speed: 20 Kilobytes/sec

• Result:– If a user logs in multiple times, the sum of all logins cannot

exceed the maximum user speed. The speed is divided across the hosts that are logged in.

13


14


15


16


17


18


19


20

Solution: components

Frontend Login server Redirect server

Backend User database server Log/History server “BCrouter” router

21


Login server Serves secure web pages to the users

• Login page• Statistics page• Technical information page• …

Contacts the user database server for validating user accounts

Contacts the history server to gather historical information about logins and/or quota

Contacts BCrouter to check current quota and/or login status and performs login/logout

22


Redirect server Redirects HTTP requests to the login page on the Login

server

Gets all the traffic that requires a login from non-logged-in hosts

Redirect done by a webpage (not TCP level) Separate dedicated host because can get DoS Real time network anomaly detection

• Detect virus/worm before login… even for 1st time users• Coupled to automatic user blocking system

23


User database Contains all known users Contacted by the login server Can be any type of server

• LDAP

• Radius

• Custom type of authentication

• …

24


Log/history server Receives logs from BCrouter Parses received log files Store processed information in a database

• Historical login information

• Historical account information

Database contacted by the login server Possibility to use data mining techniques to

detect suspicious user behavior

25


BCrouter Implements the core functionality Linux based solution Sends detailed quota reports and issued

commands to the log server Contacted by the login server

• Get quota information about user and/or IP

• Get login status of user and/or IP

• Perform login and logout operations

26

Solution: internet router setup

Assumptions A few 1000’s of users

• Limit by log/history server

Manage the internet connection Auto redirect to login website Minimize the used Internet bandwidth

27

Solution: internet router setup

Login server

User

database

InternetInternet

Log/History

server

Internal management

network

Web cacheNAT

Firewall

BCrouter

Internal backbone network

Internal backbone network

Redirect

server

28

Solution: main router setup

Assumptions A few 1000’s of users

• Limit by log/history server

Manage the entire network Auto redirect to login website Central DHCP server is used to distribute IP

addresses Minimize the used Internet bandwidth

29

Solution: main router setup

Login server

User

database

InternetInternet

Log/History

server

Internal management

networkWeb cache

NAT

Firewall

BCrouter

Internal netInternal net




DHCP

DNS

Redirect

server

30

Solution: setup remarks

Webcache and NAT are between BCrouter and Internet BCrouter needs to ‘see’ the user IP address Otherwise not possible to make user and IP distinction

Advantage:• Transparent web caching is possible

Disadvantage:• Cached contents are also accounted and speed limited

31

Solution: integration

Suitable for each network?• Ethernet based networks

• BCrouter does not support any routing protocols (RIP,EIGRP…)

• BCrouter can also play a Cisco Netflow probe

• High performance – Gigabit speeds with dual CPU system

• Redundancy (still in development)– Possible to have backup BCrouter in hot standby

32


Scalability BCrouter server

• Supports virtual unlimited users– Tested up to 50 000 users (1 Gigabyte RAM)– Handles up to 60 000 login/logout operations per second

• Supports virtual unlimited IP addresses– Tested up to 200 000 IP’s (1 Gigabyte RAM)

• Supports up to 300 000 packets/sec (1.5 Gigabit)– Dual Xeon 3.6Ghz

Clustering (Not yet implemented)• Possible to use multiple BCrouter servers

– Each server handles a part of the given network segments– Inter-BCrouter communication to exchange quota changes

33


Quota/bandwidth exceptions? Yes… very powerful exception capabilities Exception flags

• IP speed limit• User speed limit• IP accounting• User accounting• No login required

Exceptions can be made for hosts or even entire networks (both local and/or internet)

34

Solution: integration Quota/bandwidth exceptions examples:

Default: • Login required• Accounting to both user and local IP• Obey both user and local IP speed limits

Local host A does not have to login to access the Internet, but still uses IP quota and speed settings

• E.g. Embedded device that can’t login and needs network access Traffic from Internet host B is always possible from any local host

and is never accounted, but local host IP speed limits are obeyed• E.g. Website with security patches

Any combination of exception flags is possible in either direction for any host/network

35

Solution: security considerations

Account abuse Example

• User A powers off his PC without logging out• Malicious user X takes IP of user A• X continues to work with credentials of user A

Solution: Auto logout• Possibility 1: BCrouter performs logout after X minutes of inactivity• Possibility 2: Ping probes• Possibility 3: DHCP server

– Login server checks if IP that wants to login has been issued by the DHCP server. Refuse login with static IP

– Use very short DHCP lease times (e.g. 15 minutes)– Run script every few minutes that logs out inactive DHCP leases

DHCP based auto-logout is preferred

36

BCrouter: introduction

Let’s take a look at the core element: BCrouter Components of BCrouter Commands and logging Routing and Netfilter setup Quota/Bandwidth exceptions

37

BCrouter: introduction

Login server

User

database

InternetInternet

Log/History

server

Internal management

networkWeb cache

NAT

Firewall

BCrouter





DHCP

DNS

Redirect

server

38

BCrouter: components

‘open’ black boxLinux operating system

User space• DHCP forwarder

• Syslog daemon

• BCrouter daemon

• Network configuration script

Kernel space• BCpolicer module

39


BCrouter

daemon

DHCP

forwarder

Syslog

daemon

Kernel space

User space

Kernel logging

Netfilter

framework

BCpolicerInput

interfaces

Output

interfaces

Management

interface

40


DHCP forwarder Forward broadcast DHCP DISCOVER to a central

DHCP server

Dhcp-fwd • http://www.nongnu.org/dhcp-fwd/

Very simple application User space application running in chroot jail Listens in ‘promiscuous mode’ on specified interfaces

41


Syslog daemon Send logs to a remote log server for remote

processing

Syslog-ng • http://freshmeat.net/redir/syslog-ng/10178/url_homepage/syslog-ng

Very powerful options (filtering, multi logserver…)

Logs both user space as kernel logs

42


BCrouter daemon Provides a network-based console to the

BCpolicer kernel module

Simple Perl script (Forking TCP server) Allows simultaneous management access Listens on a network socket (telnet port 23) Communicates with the kernel module

43


Network configuration script Provides entire interface, routing and Netfilter

configuration and setup

Shell script Executed at boot time

44


BCpolicer kernel module Receives login/logout commands and performs

accounting and routing decisions

Core element of BCrouter (ipt_bcpolicer) Works entirely in kernel space Loadable module which implements an iptables

target

45

BCrouter: commands & logging

Commands Login/logout

• login [username] ip [x.x.x.x] reason [text]• logout [username] reason [text]• logout [ip] reason [text]

Query information• show user ip [x.x.x.x]• show ip user [username]• show quota ip [x.x.x.x]• show quota user [username]

Configuration• conf ip …• conf user …• export all

Miscellaneous• show uptime

46


Commands examplebcrouter1# export allbcrouter1# login user kuleuven/u0022948 ip 10.91.91.1 reason login demo200 OK: login - 1142031358930300 - kuleuven/u0022948 (1) on 10.91.91.1 (login demo)bcrouter1# show ip user kuleuven/u0022948204 OK: show ip user - 10.91.91.1bcrouter1# show user ip 10.91.91.1203 OK: show user ip - kuleuven/u0022948bcrouter1# login user kuleuven/u0022948 ip 10.91.91.2 reason 2nd login200 OK: login - 1142031429848045 - kuleuven/u0022948 (1) on 10.91.91.2 (2nd login)bcrouter1# show ip user kuleuven/u0022948204 OK: show ip user - 10.91.91.2,10.91.91.1bcrouter1# export allconf user kuleuven/u0022948 …. conf ip 10.91.91.1 …login user kuleuven/u0022948 ip 10.91.91.1 reason recovering statefull infoconf ip 10.91.91.2 …

47


network segment Time of log Log sequence number Name of segment Traffic counters

• Bytes and packets• Download and upload• Accounted and not

accounted• Dropped and accepted

Number of active IP’s

host Time of log Log sequence number IP address Username (–none- if no

login) Traffic counters

• Bytes and packets• Download and upload• Accounted and not

accounted• Dropped and accepted

Logging Log commands and responses Log network/host statistics

48

BCrouter: routing & Netfilter

Routing with BCrouter is done by a BCPOLICER target in the PREROUTING mangle table that alters the fwmark value of the packet and uses this value as selector for policy based routing.

49


Use Linux networking capabilities IEEE 802.1Q support (VLAN technology)

• Used to limit the number of physical interfaces

Policy based routing (Routing rules)• Used for implementing user groups

Netfilter/Iptables framework• Used for host exception lists

50

BCrouter: routing & Netfilter IEEE 802.1Q support

Virtual LAN technology (VLAN) Operates on the data link layer (OSI layer 2)

• Adds 4 extra bytes to existing ethernet header Allows multiple LAN’s over 1 physical wire (trunking)

Each VLAN id has its own interface device• E.g. eth0.5 indicates VLAN id 5 on physical interface eth0

‘vconfig’ tool

VLAN

enabled

Device

VLAN

enabled

Devicedot1Q ‘trunk’ containing VLAN 1,2,3

VLAN 1

VLAN 2

VLAN 3

51


Policy based routing (Routing rules) Multiple routing tables

• Routing policy database (RPDB)

• Each entry in the database:– contains a routing table

– has a priority number

– has a selector (src addr, dst addr, incoming iface, tos, fwmark)

• RPDB entries are scanned according priority

• If the selector of RPDB entry applies, that routing table is used

‘ip rule’ tool

52

BCrouter: routing & Netfilter Netfilter/Iptables framework

Extensive IP packet filtering rules There are 3 ‘tables’

• Each table has its own purpose Each table contains several ‘chains’

• Each chain operates on a different location in the network packet flow

• Chains can also be custom-defined Each chain can contain several ‘rules’

• Each rule contains a selector and a target• Rules are examined in the order they are listed in the chain• If a selector matches, the action is determined by the target

‘iptables’ tool

53

BCrouter: routing & Netfilter Filter table: Firewall type rules

INPUT: for packets destined for the box itself FORWARD: for packets being routed through the box OUTPUT: for locally generated packets

NAT: Native Address Translation type rules PREROUTING: for altering packets as soon as they come in OUTPUT: for altering locally generated packets before routing POSTROUTING: for altering packets as they are about to go out

Mangle: Packet alteration type rules PREROUTING: for altering packets before routing OUTPUT: for altering locally generated packets before routing INPUT: for altering packets destined for the box itself FORWARD: for altering packets being routed through the box POSTROUTING: for altering packets as they are about to go out

54

OutputRouting

InputRouting


PREROUTING

Netfilter framework

FORWARD

INPUT OUTPUT

POSTROUTING

User spaceUser space

I

F

A

C

E

I

F

A

C

E

55


Packet filtering and routing are normally 2 different things

However: There is a packet ‘property’ that can be changed by an

iptables target and can be used as a routing rule selector: fwmark

Fwmark Netfilter MARK value Integer value Associated within the kernel with the packet Does not alter the packet itself

56

BCrouter: routing & Netfilter Routing with BCrouter is done by a

BCPOLICER target in the PREROUTING mangle table that alters the fwmark value of the packet and uses this value as selector for policy based routing.

Fwmark value after BCPOLICER selects routing table: 1 … 97 : routing table belonging to the user group of

the identified user 98: routing table for non-identified users 99: routing table for redirecting users to the login site 100: routing table that drops all packets

57


I

F

A

C

E

I

F

A

C

E

PREROUTING mangle table

Selector Target .

Exception1 MARK ex.flag1

Exception2 MARK ex.flag2

…

Everything BCPOLICER

Input routing

Priority Selector Routing table

1 fwmark=1 Group1

2 fwmark=2 Group2

3 fwmark=98 NoGroup

4 fwmark=99 Redirect

5 fwmark=100 Drop

Quota/bandwidth exception list changes fwmark before entering

BCPOLICER

Fwmark changed by BCPOLICER to select correct routing table

58

BCrouter: exceptions

Exceptions passed to BCPOLICER via fwmark value Possible to use almost all Iptables selection features List before BCPOLICER target in PREROUTING Fwmark value

• IP speed limit (src: 0x0001 dst: 0x0002)• User speed limit (src: 0x0004 dst: 0x0008)• IP accounting (src: 0x0010 dst: 0x0020)• User accounting (src: 0x0040 dst: 0x0080)• No login required (src: 0x0100 dst:0x0200)

Final value: logical OR of the wanted flags Example

Traffic from Internet host B is always possible from any local host and is never accounted, but local host IP speed limits are obeyed

Src: localnet Dst: hostB mark: 0x0001||0x0100 = 0x0101 Src: hostB Dst: localnet mark: 0x0002||0x0200 = 0x0202

59

BCpolicer: introduction

Everything up until now was on macroscopic level

Let’s dig (a little bit) into the real core: ipt_bcpolicer

The next slides are greatly simplified Would take couple of hours to completely

explain the entire design

60


TCP/IP TCP: reliable protocol

• Every received packet must be acknowledged to Sender by Receiver

• Sender retransmits when no acknowledgement received within certain time

• Exponential delay between retransmits

• When relatively few packets are dropped, transmission speed goes down significantly

61


Policing vs. Shaping

PolicingAccept/drop packetsDon’t delay packetsNo queuesNo scheduling

ShapingDon’t drop packetsDelay packetsQueuingScheduling

62

BCpolicer: design principles Design started by determining the specifications of an

‘ideal’ bandwidth/quota system (with no user login) Different settings for each individual network segment For each network segment:

• Independent up/download settings• Distribute available bandwidth across active hosts, favoring low-

traffic volume hosts a little• Dynamically regulate maximum bandwidth per host to allow

maximum bandwidth usage without overloading the segment or uplink

For each host:• Distribute host bandwidth equally over active connections• Tendency to favor interactive traffic above bulk traffic• Enforce traffic quota by dynamic bandwidth speed regulation

63

BCpolicer: design principles

Result General principle: leaky bucket system Lots of modifications and tweaks required

POLICER

MeanFillRate

TokenBucketMaxSize

CurrentRate(0…BurstRate)

TokenBucketSize

TokenBucket

64

BCpolicer: design principles Example:

Download quota: 4GB/month Maximum download speed: 50KByte/sec

DownloadTokenBucketMaxSize• = 4GTokens

DownloadBurstRate• = 50KTokens/sec

DownloadMeanFillRate• Must fill up an empty TokenBucket in 30 days• =4GTokens/(30days*24hours*60minutes*60seconds)• =1657 Tokens/sec

65


Policer summary Multi exponential stream policer

• ‘smooth’ policing

• Distribute individual host bandwidth equally over its active connections

• Tendency to favor interactive traffic above bulk traffic

66


Specifications list For each host:

• Distribute host bandwidth equally over active connections

– OK : Multi logarithmic stream policer algorithm

• Tendency to favor interactive traffic above bulk traffic

– OK : Multi logarithmic stream policer algorithm

• Enforce traffic quota by dynamic bandwidth speed regulation

– OK : Accurate quota control because there are no Tokens created or destroyed in the bucket system

67


Specifications list Different settings for each individual network segment

• OK : Each IP has its own buckets with its own settings For each segment:

• Independent up/download settings– OK : Use separate buckets

• Distribute available bandwidth across active hosts, favoring low-traffic volume hosts a little

– To be done

• Dynamically regulate maximum bandwidth per host to allow maximum bandwidth usage without overloading the local segment or uplink

– To be done

68


Prevent network congestion: Implement a ‘valve’ between TokenBucket and the

Policer• If the ‘valve’ is fully open:

– Maximum host speed is BurstRate

• If the ‘valve’ closes:– Slow down maximum host speed

• ‘valve’ control– If no network segment congestion: fully open– Network segment congestion: close valves for the local IP’s that

cause the congestion until no congestion anymore

• ‘valve’ is named ‘RateFactor Correction’• The state of the valve is called RateFactor

69


BCrouter

InternetInternet

BCpolicerFeedback

algorithm

Host

70


POLICER

MeanFillRate

TokenBucketMaxSize

CurrentRate(0…BurstRate)

TokenBucketSize

TokenBucket

RateFactor CorrectionRateFactor

(e.g. <1)

71

BCpolicer: policing alternatives

Advantage of policing Simple to implement

• No queuing• No scheduling

Very fast Drawbacks of policing

Policing overhead problem• Relative problem:

– Suppose unlimited stream would be 100Kbyte/sec– Limit to 10Kbyte/sec– 10 to 15% overhead -> receiving from the Internet 12KByte/sec

• More overhead on low latency, high bandwidth connections– E.g. Limit to 10KByte/sec; receiving 20KByte/sec

72


‘TCP Window Size’ alteration TCP Window Size

• Field in TCP packet header• Message from Receiver to Sender• Indicates to Sender how much data can be sent to Receiver

Should decrease retransmission overhead• Avoid fast retransmission on fast connections

Alters packets passing BCrouter• Possibly bad and unforeseen consequences?

No need for queuing Integration in the BCpolicer scheme

• Use the DelayTime value to decrease the window size

73


Shaping Alter the Round Trip Time of a connection

• ‘simulate’ slow connection• TCP will try to adjust by slowing down

Extra load and complexity due to delay queue management

Integration in the BCpolicer scheme• BCpolicer already works with delay times• Instead of accepting or dropping, put packet into a

delay queue for the delaytime

74

BCpolicer: complete design

Up until now, we only focused on the bucket design

Complete system Addition of user buckets

• Same principle as IP buckets

Addition of exception handling• Makes program flow complex

75

BCpolicer: main flowStart Src IP kotnet?

Yes

No

Return IPT_CONTINUE

srcnlr=1

Mark translation

Src userbucketVerdict (UV1)

Src ipbucketVerdict (IV1)

Src logge

din?

REDIRECT

srciac!=0or

srcisl!=0

Dst IP kotnet?

Yes

No

dstnlr=1

Dst userbucketVerdict (UV2)

Dst ipbucketVerdict (IV2)

Dst logge

din?

DROP

Yes (nlr)

No (lr)

Yes

No

Yes

No

Yes

No

Yes

No

Yes (nlr)

No (lr)

Final Verdict&

Accounting &

Logging

srcnua=1

andsrcnus=

1

dstnua=1

anddstnus=

1

Yes

No No

Yes

Src IP Bucket calculation

dstiac!=0or

dstisl!=0

Clear src streamupdate flag

Clear dst streamupdate flag

76

BCpolicer: bucket verdictBucket

calculations

No

RespectSpeedlimit?

TBS-PktSize>0

Verdict=DROP Verdict=ACCEPT

No

Yes

PBS>PTT+PktSize

Verdict=ACCEPT

Yes

No

Calculate StreamII and get the correct SAPS and SPLAT for this

packet

QSAPS=32- ((32*SAPS) / MPS)QDelayF=64- ((64*PBS) / PTT)

PDT=DelayMatrix(QSAPS,QDelayF)

SPLAT+PDT<PacketTSNo

Yes

Verdict=DROP Verdict=ACCEPT

Entrypoint

Return Verdict

Yes

PBS>PktSize

Yes

No

Store StreamII and new SAPS (SAPSnew)Set streamupdate flag for the current ip

(used for accounting)

77

BCpolicer: bucket calculation

Yes

No

No

Yes

Entrypoint

Endpoint

TB_ADD_INTERVALpassed since the

last call?

Calculate new TBSTBS=max( TBS+TBMFR*TBLAT,TBMS)

Calc max CurrentREffectiveR=max CurrentR

PBS+EffectiveR<PBMS

Calc max EffectiveRCurrentR = max EffectiveR

(CurrentR = = EffectiveR)TBS=TBS-CurrentR

FactorR=0

PBS=PBS+EffectiveR

78

BCpolicer: f.verdict&acc&log

Entrypoint

Src IP kotnet?

No

Yes SOURCE IPAccounting Calculation

DESTINATION IPAccounting Calculation

FinalVerdict=DROP

Return FinalVerdict

FinalVerdict=ACCEPT

If PktTime>SITLL+LOGSI

Print Src IP logline &

clear printed counters

IfPktTime>SSTLL+LOGSS

Print Src Segment logline &


REDIR verdict in

UV1 || IV1 || UV2 || IV2?

No

Yes

FinalVerdict=REDIR

DROP verdict in

UV1 || IV1 || UV2 || IV2?

No

Yes

FinalVerdict=

REDIR?

No

Yes

Dst IP kotnet?

No

Yes

If PktTime>SITLL+LOGSI

Print Dst IP logline &


IfPktTime>SSTLL+LOGSS

Print Dst Segment logline &


79

BCpolicer: accounting

FinalVerdict=

DROP

No

Yes

Entrypoint

Return

UsrTBS += PktSize

UsrTBS - = PktSize UsrPBS - = PktSize

nua=1Yes (no uac)

No (uac)

nus=1Yes (no usl) No (usl)

iac=1

IpTBS += PktSize

isl=1

IpTBS - = PktSize IpPBS - = PktSize

No (no iac)

Yes (iac)

No (no isl) Yes (isl)

nua=1Yes (no uac) No (uac)

SIFDB + = PktSizeSIFDP++

SINFDB + = PktSizeSINFDP++

SINFAB + = PktSizeSINFAP++

SIFAB + = PktSizeSIFAP++

SSTDB + = PktSizeSSTDP++

SSTDI++ if neededSSTAI++ if needed

SSTAB + = PktSizeSSTAP++

SSTAI++ if neededstreamupdate=1

Yes

No SAPS at SII=SAPSnew

80

Case study: KotNet

Goals Connect K.U.Leuven association students and

personnel to the campus network and Internet from their homes

Enhance possibility of study and research in an academic environment

Low entrance fee and costs

KotNet is an enabler for the E-university

81

Case study: KotNet

History 1994: Started by students in their student home 1996-97: pilot & test projects 1997-98 en 1998-99

• 1°, 2° operational phase K.U.Leuven residences• access via cable TV network

– test project in 1997-98– operational since Sept. 1998 (2 cable TV operators)

2002 : Wireless KotNet at public places• Test project in student restaurants• Gradually extended to libraries, meeting rooms, classrooms

2004 : Agreements with associated partners

82

Case study: KotNet

Providers K.U.Leuven – direct fiber to university

residences K.U.Leuven – wifi hotspots in student

restaurants, auditoria, ... UPC Belgium – via cable modem Telenet/Iverlek – via cable modem

83

Case study: KotNet

Technology Layer 2

• VLAN's are used to implement city-wide networks– connect remote buildings to centralized KotNet routers

• spanning-tree for automatic redundancy Layer 3

• use of private address space (10.*.*.*)• split data streams

– traffic to K.U.Leuven networks is routed through the central firewall

– HTTP traffic is redirected to a web cache cluster– other traffic is handled by a Network Address Translator

cluster

84

Case study: KotNet Policies

Users will consume mega-amounts of bandwidth if you let them. Not all of their traffic is of an educational nature...

• implementation of policies and enforcement of these policies is absolutely essential

p2p traffic is not allowed (Gnutella, Kazaa, eDonkey, ...)• enforced by iptables filters on the NAT servers

individual servers are not allowed• enforced by a 200 Mbyte/day upload limit• exceptions are possible for registered servers

Internet bandwidth is not free• enforced by a 4 Gbyte/month/USER download limit

85

Case study: KotNet Residences

Approx 50 buildings Approx 4000 connections Spread geographically across the city of Leuven

• K.U.Leuven owned fiber network Each building is a different VLAN Ethernet network

• Old : – 10Mbit HUB– 10Mbit Half-duplex fiber uplink to KotNet backbone

• New : – 100Mbit Switch (Cisco 2950)– 100Mbit Full-duplex fiber uplink to KotNet backbone

86

10Mbit HUB

Case study: KotNet (residences)

10Mbit FOT

10Mbit FOTKotNet

BackboneNetwork

KotNet BackboneNetwork

100Mbit Switch

Old Network New Network

Gigabit 802.1Q VLAN trunk

Aggregation Switch

87

Case study: KotNet

Cable provider Telenet Coverage across the region of Leuven Approx 1500 cable modems Approx 5000 users 3 Cable segments

• 13 Mbit downstream capacity each

• 3 Mbit upstream capacity each

• Single VLAN (flat network)

88

Case study: KotNet (Telenet)

KotNet Backbon

e

KotNet Backbon

e

Cable segment

Cable Router (L3)

100Mbit FOT 100Mbit FOT

Telenet location

100Mbit Flat network

89

Case study: KotNet

Cable provider UPC Coverage across the region of Leuven and

Brussels Approx 5100 cable modems Approx 15000 users 52 Cable segments

• 50 Mbit downstream capacity each

• 3 * 10 Mbit upstream capacity each

• Each cable segment is a different VLAN

90

Case study: KotNet (UPC)

KotNet Backbon

e

KotNet Backbon

e

Cable Head end (L2)

UPC locationGigabit 802.1Q VLAN trunk

52 Cable segments

91

Case study: KotNet

KotNet Backbone Gigabit Ethernet based Redundant where possible

• spanning-tree protocol

Separated from K.U.Leuven backbone Spread geographically across the city of

Leuven

92

Case study: KotNet

KotNet Backbone-Residences-Telenet-UPC

KotNet Backbone-Residences-Telenet-UPC

Gigabit 802.1Q VLAN trunk containing 105 KotNet VLAN’s

NAT

K.U.LeuvenWeb cache

KUL-KotNet

Firewall

InternetInternetKULeuvennetwork

KULeuvennetwork

Monitor switch Flow logger

BCrouter

Redirect

NAT

Association

Gigabit mirror port

KUL-INET

Firewall

Cisco WCCP

93

Case study: KotNet

Pre-BCrouter 2 High-end Cisco 7500 routers

• 30.000 EUR each• Couldn’t handle large network segments

– Long user access lists requiring lots of CPU power

• Policy based routing not really flexible• Logout operation very expensive (ACL reload)• Inflexible shaping

– Not possible per ip/user/stream/usergroup and certainly not everything together

Need for more scalable solution!!

94

Case study: KotNet

BCrouter Replaces both Cisco routers Hardware:

• Dell Poweredge 2650 • Dual Intel(R) Xeon(TM) CPU 3.20GHz with Hyperthreading • ServerWorks GC-LE chipset, 400MHz front side bus, 2:1

memory interleaving, 5 PCI buses (three of which are are PCI-X capable)

• 1 Gig ECC ram • Two Intel E1000 PCI-X 133Mhz Network interfaces• 3000 EUR

Extremely powerful and flexible configuration

95

Case study: KotNet BCrouter settings

Only 1 simultaneous login allowed by login system Normally: login required, obey user/IP quota/speed User buckets

• Download quota of 4Gbyte/month– Max speed (BurstRate): none

• Upload quota of 200Mbyte/day– Max speed (BurstRate): none

IP buckets• Download quota of 8Gbyte/month

– Max speed (BurstRate): 250Kbyte/sec• Upload quota of 8Gbyte/month

– Max speed (BurstRate): 250Kbyte/sec

96

Case study: KotNet BCrouter exceptions

DHCP, DNS server• No login required, no user acc/bw, ip acc/bw

Login website reachable• No login required, no user acc/bw, ip acc/bw

Local antivirus download website reachable• No login required, no user acc/bw, ip acc/bw

Local FTP server reachable• No login required, no user acc/bw, ip acc/bw

Unblock website reachable• No login required, no user acc/bw, ip acc/bw

E-learning website reachable• Login required, no user acc, user bw, ip acc/bw

97

Case study: KotNet

User blocking system (admin view) Receives information from number of probes

• Email system• Flowlogger• Local honeypots

Gets an IP address• Looks up the corresponding user• Logout user• Put user in a blacklist to prevent user login• Send email to the user

98

Case study: KotNet

User blocking system (user view) User is logged out Receives email containing instructions

• Webmail is always reachable

• Local antivirus download site also reachable

If the user tries to login:• Message his account is blocked

• More detailed instructions about the blocking reason

• Link to unblock website

User unblocks himself• Only allowed 1 time each day

99

Case study: KotNet Flow Logger

Emits Cisco Netflow data to logging system• Very detailed traffic information (for each stream):

– Start and stop time– Source and destination IP– Source and destination port– Number of bytes transmitted/received– Number of packets transmitted/received

High performance Netflow probe:• nProbe (http://www.ntop.org)

Used to detect network anomalies• Host scans and port scans• Coupled to automatic user blocking system

100

Case study: KotNet

Email No direct SMTP connections to the internet All emails must pass the Central Anti Virus

cluster• Scans for viruses (Mcafee Antivirus)

• Performs spam classification (Spamassassin)

• If a virus is detected, user is blocked– User blocking system looks at sending IP

101

Case study: KotNet

Transparent caching No user settings required to use caching Reduce the traffic on the Internet link Cisco Content Engine

• Squid based

• WCCP– ‘transparent’ redundancy

• Needed a quick solution

102

Case study: KotNet

Firewalling Done on BCrouter

• Beginning of PREROUTING mangle table

TCP port 135,139,445 are blocked for anyone• Prevent virus outbreaks

103

Case study: KotNet

P2P blocking Done on NAT cluster Do a pattern search in every packet

• Look for certain P2P strings– Kazaa, Bittorrent, Edonkey

• Iptables string module

• Rather CPU expensive

• Planning to make a better version someday

104

Case study: KotNet Transition to BCrouter

Phased transition• Created 2 testing KotNet segments on BCrouter• Test connectivity and performance • Modify login scripts and backend for BCrouter• Convert 2 real KotNet segments to BCrouter• Convert all residences to BCrouter• Convert all UPC to BCrouter• Convert all Telenet to BCrouter• Parallel volume accounting on user webpage

– Real-time counters from BCrouter– History information by previous KotNet accounting system

No real problems encountered

105

Case study: KotNet Traffic impact (early testing)

User mode test version of BCpolicer No HTTP traffic Most popular P2P applications blocked Only used IP addresses for accounting

• PolicerBucketMaxSize: 150 Mbyte• MeanFillRate: 50 Kbyte/sec• BurstRate: 250 Kbyte/sec

Results:• 50% reduction in traffic• Only 5% of users ‘hit’ BCpolicer

Users noted improvement in network quality

106

Case study: KotNet

Some Numbers Current daily numbers

• 400 Mbit/sec throughput total (18u – 23u)– 150 Mbit Non-http

– 30 Mbit K.U.Leuven

• 80 Kpackets/sec

• Up to 20 000 simultaneous users

BCrouter load• 33% CPU at peak

107

Case study: KotNet

Management utilities The next few slides show the web-based

information interface

108

Case study: KotNet

109

Case study: KotNet

110

Case study: KotNet

111

Case study: KotNet

112

Case study: KotNet

113

Development: current status

BCrouter itself is functioning Design is done by specifications Implementation of ipt_bcpolicer is not yet

100% ready• Dynamic bandwidth regulation by congestion is not

yet implemented (Fixed RateFactor of 1)

Login system is not self-contained Currently uses the previous login system from

KotNet

114

Development: future Redundancy

Timeframe: June 2006 Hot-standby backup system which takes over if primary fails Further development needed

Build a stand-alone login/log/history system Timeframe: September 2006 Easy custom layout Minimal dependencies Already have some proof of concept code Can be done independently of further development Login system without history information can be done very

quickly Planning: hire a job student for this during the summer

115

Development: future Black Box solution

Timeframe: Jan 2007? Depends on stand-alone login system Packaging and generalizing scripts

• Packaging of BCrouter partly done TCP window size policing

Timeframe: unknown Still requires research Planning: make a thesis out of this

FactorRate Correction Timeframe: unknown

• Not an issue anymore on KotNet Implementation and testing

116

Development: wish list

Transform the policer into a shaper Probably more work than expected Planning: make a thesis out of this someday

Add the automatic blocking features Currently uses the previous KotNet solution

117

Conclusion

BCrouter concept offers great possibilities User authentication Advanced and flexible quota and bandwidth

management High performance and scalable solution

BCrouter still requires a lot of implementation Concepts are already tested in the field

118

THE END

Contact information:

Dirk JanssensICTS K.U.Leuven

[email protected]

Documents

Bandwidth management and optimization