Embed Size (px)
Citation preview
It’s Their Bandwidth …
Let Them Decide!
Richard SammisManager
IUB Network Operations
Melody ChildsManager
Residential IT ServicesOffice of the Vice President for
Information TechnologyUniversity Information Technology
Services
INDIANA UNIVERSITY
Brian D. VossAssociate Vice
PresidentTelecommunication
s
Copyright Statement
• Copyright Melody Childs, Richard Sammis and Brian D. Voss , 2002. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.
What are we doin’ today?
We want to take on a journey through the past three years – OUR past three years
Through a number of events and changes
And weave a story of success in dealing with the ‘War on Bandwidth’ that resulted from the arrival of Peer-to-Peer applications
What we’re not touching on …
• DMCA Implications
• Ethics of copyright ‘conversion’
That presentation will be called:
It’s Their Jail Time, Let Them Serve It!
First … a little about us
Indiana University Bloomington, Indiana
Public University
8 Campuses
IU Bloomington
35,000 students
11,000 Campus Housing Residents on the Bloomington campus.
Indiana University – THE Leader in Information Technology in Higher Education
IU Pioneers Enterprise Licensing Agreement with MicrosoftFirst University In Nation To Do So
IU Selected to Host
Internet2 Abilene Network
IU A Most-Wired Campus – Again!
IU Knowledge Base – Netscape’s Choice as Best On the WebIUPUI Breaks Ground
on New Building for Tech Research
IT @ IU
• Organization• Leadership
• Strategic Vision
VP IT
OVPIT/UITS Organization Structure
VP IT
Look back at May of 1999 …• Residential IT Services was a division of Residential
Programs, and NOT the IT organization
• Web browsing was the most predominant application that consumed bandwidth
• IUB campus had a total of 15Mbps of commodity internet (CI) drainage
• Students/Residents shared music by swapping CDs
Our World was about to change …
Dramatically!
The Perfect Storm
Events were converging …
• UITS takes over Residence IT Services
• Optical Fiber Infrastructure (I-Light) to be built
• Peer-to-peer (P2P) was about to become something we all heard a LOT about
Converging eventsSupplying IT Services to Residents
• OVPIT and IUB campus, under President’s direction, negotiated to move IT services in Residence Halls to UITS– Complaints about service quality on a ‘most wired’
campus– Focus of Residential Programs was “room and board”
… IT had the same importance as in-room cable TV
• In April 2000, transition was announced
• State of network in Residence Halls was poor and outdated (equipment hadn’t be upgraded since 1994)– Needed to bring up to campus standard of 10Mbps
switched ethernet, connected to campus gigabit backbone
– Resident IT services were not up to par with those available on campus; services needed to be integrated so that all could benefit from a unified service architecture
Converging eventsSupplying IT Services to Residents
Converging eventsOptical Fiber Infrastructure
• IU led the building of I-Light– I-Light is a multi-fiber, multi-conduit privately
owned and operated (by the Universities) optical fiber infrastructure
– Key elements• Gigabit links between all campuses – essentially puts
everyone on the same campus backbone• Co-location space in large carrier hotel, where fiber
‘jumper’ links are available to all major internet service carriers (including Internet2)
• Near-unlimited ability to expand – just add equipment!
Converging EventsPeer-to-Peer Emerges
Not unlike another Ivy-League drop-out from ~20 years before, 19-year-old Shawn Fanning left college to focus on developing a program that he had devised during his freshman year that would allow students to share their favorite music via MP3 format
His product – and company – Napster, was about to change everything about the way in which Universities looked at network services
Roll Tape … Fall 1999
• IUB enters Fall of ‘99 with a fractional DS3 connection to the CI; 20Mbps ready, expandable to full 45Mbps – As Fall progresses, this is quickly ‘dialed-up’ to full 45Mbps as
performance degrades (and complaints increase)– Napster hits campus – though UITS doesn’t know it
• I-Light is still an idea – money appropriated but it has yet to hit the drawing board – Partner negotiations commence; much legal wrangling about how
to own and manage it– Technical designs are explored
• Residential IT Services still a part of Halls administrative structure, and complaints start on the first day of move-in
And In January 2000, it hit!
• Our CI connection of 45Mbps (DS3) pegged
• Upon investigation, we discovered that ~70% was Napster
• Students (and faculty and staff) complained about poor internet performance
Initial (knee-jerk) Reaction:
Predictable (knee-jerk) Response
Escalating the situation …
Indiana University sophomore Chad Paulsen …
BestPath: Technical Masterpiece!
• If you can’t beat ‘em … work with ‘em!• IU Developed ‘BestPath’ Algorithm
– Developed by IU in conjunction with Napster – Programmed by Napster and incorporated into their
server software– It orders the list of possible servers available to serve
a particular item, using the local net first, followed by Internet 2, and the local autonomous system
– Was used successfully, and published as a draft RFC see http://bestpath.iu.edu/internetdraft.txt
Heroes at last!!!!!
Wa-oooops … spoke too soon!
Tactical Withdrawal …
Repercussions run deep …
AVP Voss Suffers Heart Attack!!
Talk about your Y2K!
• Limped along – Napster hung around (though ultimately, later, it disappeared)
• Gnutella and other apps were there, but not as popular as Napster
• Looked at the Summer as a chance to re-trench and re-group
Okay … NOW what?
• Technically – Separate Resident traffic from Campus traffic– Find ways to rate-limit traffic – like applying a
faucet to the ‘offending’ flow of traffic– Add More Bandwidth if possible to afford it
• Procedurally– Get UITS Administration ‘off point’!!!– Involve Residents in Decision Making
Procedural Solutions
• UITS had just taken over IT support in the Residence Halls– Already had an IT Advisory Committee formed
from that process
• Took the issue to Residence (student) Government: This is your bandwidth … how do you want to use it?
Let’s take a brief look at Residential IT Services and the
formation of ResNET
Melody Childs
Residential IT Services
• VPIT and IUB Chancellor transfer responsibility for IT services to UITS in April 2000.
• Residential IT Services is a part of Teaching and Learning IT Division of UITS – the support arm of the organization– Tightly integrated with Support Center (Helpdesk),
campus-based student technology services, education programs, and other front-line IT support teams
• Adequately funded, supplying IT equipment, support services, and a rep at the “IT Table”
Funding for IU’s ResNet
• Residential IT Services & Network Operations are Units of IU’s Central Computing Organization
• Mandatory Student Technology Fee supports on-campus tech centers, software, & free training
• Funding for Residential Tech Centers and In-Room Services from housing fees
• Network service (ResNet) combination housing fees and central computing budget
Challenges at the outset
• Modernize Halls Network Infrastructure
• Modernize and life-cycle fund IT infrastructure (computers, network, etc.)
• Develop quality consultation and support services
• Seemlessly integrate on-campus and residential IT services and technology
• Build productive rapport with Residents
Technical Steps• From May 2000 through May 2001, UITS busy
modernizing ResNet
• Building infrastructure– All Ethernet repeaters replaced with switches– 10/100 Ethernet provided to each pillow– Every jack (15000) activated– Fiber risers installed between closets
• Campus Backbone– Building service upgraded from 10Mps to 100Mbps (with
Gigabit capacity available)
Technical Steps
• Would plan to Add second CI connection for Fall 2000; however, would not be available until mid-semester– Paid for it out of the Residence Network Budget
• Experimented with filtering (on our routers) selected ports ‘frequented’ by file sharers
Residential I.T. Services23 Technology Centers
260 Workstations
50 E-mail stations
10,000+ in-room connections
15,000 Active Data Jacks11,000 Residents
10,500 Connected Computers
Connectivity SupportE-mail & Academic Applications
Anti-Virus SoftwareSecurity
ResNet IT Committee
Student Government included in management of new ResNet
Formed during transition planning (Spring 2000); now given new purpose
Residence Halls Association President Chairs IT Committee
Members include: student government reps, Housing Administrators, and UITS (central IT support provider)
Fall 2000 – Timing
• We weren’t sure how we’d stay off Page-1 of the Indiana Daily Student once Fall classes began
• Knew that our first meetings with students to discuss this issue wouldn’t be until after classes started
• Knew that our new bandwidth wouldn’t be in until mid-Semester (October/November)
But we had a little help ….
That seemed to go on for weeks
And kept us out of the news!
“Since its inception, the entire premise of the Internet centered on file sharing. Many people don’t realize that fact…” (www.gnutella.com)
“KaZaA Media Desktop (KMD) is the No.1 peer-to-peer (p2p) application for finding, downloading, playing and sharing files with millions of other users.” (www.kazaa.com)
“Talk about the Donkey. Please don't post anything refering (sic) to piracy or copyrighted material.” (www.edonkey2000.com forum)
New Applications – Same Results
ResNET or RecNET?
Working Out Solutions
Brainstorming Session with Student Government Offered Many Creative Solutions to a Mutual Problem:
* Policies & Sanction
* User Education
* Technical Solutions
User EducationCreating a more informed student body
Discussion Items:* Mailbox flyers* Student Newspaper ads* Individual Dorm Government meetings* IU’s on-line Knowledge Base:
http://kb.indiana.edu* Informational web site:
http://resnet.indiana.edu
http://resnet.indiana.edu/
Policies & Sanctions
Implement Facilitative Use Policy• Policy extrapolated from standard ‘Users Rights,
Privileges, and Responsibilities• http://www.itpo.iu.edu/IT11.html Basically states: Persons whose non-Indiana University-
mission-related activities are consuming an inordinate amount of Indiana University technical resource will be contacted by the appropriate responsible service manager/administrator and asked to cease that activity.
• Escalating sanctions from e-mail warnings to Dean of Students to loss of network privileges
Watchdog Notification System
• Developed in house• Sends email to internet ‘abusers’• Daily abuse level is definable• Requires expertise to answer emails and
help users.• Works best on the outbound bandwidth
problem• Quickly shows diminishing returns as very
heavy users are eliminated
WatchDog Output
WatchDog Output
Technical Solutions
Richard Sammis
Technical Solutions
Discussion Items:
* University game & media servers
* Add more bandwidth
* The parallel universe suggestion:
one ResNet & one RecNet
* Restrict all multimedia
* Rate Limiting
Router access lists
• Can identify applications by tcp port• This is a crude method especially as
applications get more sophisticated• Requires frequent monitoring and research
and high level of expertise• Affords several methods of limiting the
identified class of traffic• QBSS• Rate limiting
NetworkVantage(Ecoscope) by Compuware
• Purchased for promise of objective measure of internet performance (response time by application) and for traffic analysis– Replaces anecdotal information– Gives trending information– Establishes baseline performance– Uses multiple probes (PCs) and a
management station
NetworkVantagetraffic analysis report
Packeteer Packet Shaper
• Gives detailed traffic analysis dynamically and by hour, day, week, etc.
• Recognizes applications by going deeper into the packet (beyond tcp port) and gets regular pattern recognition updates
• Uses tcp windowing to meter flow rather than dropping packets
• Allows very for easy implementation of policies• Problems/limitations:
– Throughput limitations– Can discover so many classes that it bogs down
NetFlow data analysis
• Asta tools
• Mark Fulmer/Dave Plonka --flow tools
The “NEW” Campus Network(s)
ResNet CampusCI CI
Move-In Week 2001the new ResNet is put to the test
EVEN before the first day of classes, P2P Applications Audiogalaxy, KaZaA and Gnutella were consuming 88% of IU’s ResNet
June 2001 to June 2002Halls Internet traffic graph
Status: May 2002
• The year had gone along pretty well – however, we were ‘out of gas’ on both our CI links by the middle of Spring 2002 Semester
• We were seeing spread of ‘Recreational Use’ beyond Halls connection (to Campus connection) – student employees and regular faculty/staff were making use of KaZaAa!
• Knew we would need more bandwidth to survive in the Fall
Summer 2002
Brian D. Voss
Summer 2002
• Leveraged I-Light and the Indiana GigaPoP to get better pricing on CI Bandwidth
• Got rates of ~$100 per Mbps (had been paying ~$350 per Mbps)
• Purchased over 3X the bandwidth for the same amount of money
• Started Fall 2002 with 155Mbps for campus and 155Mbps for Halls
A representation of the actual growth of internet drainage for the IU network (IUB/IUPUI-Regionals) capacity in the Fall of 2002
Once and for all solution?
Not Likely!
Traffic dumped onto GigaPoP from IUB
Traffic from GigaPoP to IUB
Fall 2002 Update
• Implemented 30Mbps limit on outbound ‘recreational’ ports using ROUTER ACCESS LISTS
• Can identify applications by tcp port– This is a crude method especially as applications get
more sophisticated– Requires frequent monitoring and research and high level
of expertise– Affords several methods of limiting the identified class of
traffic• QBSS• Rate limiting
Bandwidth utilization
Some Interesting graphics
IUB TrafficOutbound CI
Inbound CI
Inbound I2
Outbound I2
Commodity Internet UsePurdue IU IHETS/ITN
Check out what happens with Filters
Outbound to the CI
Inbound from the CI
Move-in Week Starts
Outbound limits imposed
Look at what happens when you turn-off the filter
Max Capacity
Internet2 is also impacted
Traffic dumped to I2 (by Purdue, IU, IHETS)
Traffic Received from I2 by GigaPoP
Further steps …
• Just recently removed constraint from all but one outbound port– You can notice the rise in outbound traffic that resulted
How we looked today (10/03) Net Exporters (again)
IUB Outbound (peak ~400M)
IUB Inbound (peak ~300M
Lessons Learned
• Relax and enjoy the horror! – Fighting it only makes it harder and more painful!
• Including students (via their government) is a good idea
• Splitting Campus Network CI from Resident Network CI is a good idea
• Adding Bandwidth (as much as you can afford) is a good idea
Lessons Learned
• Problem is not just about students downloading – more about CI users (and I2 users) using student machines as servers
Campuses become net-exporters of dataTypical use curve shows outbound greatly exceeding
inbound; which slows requests, which in-turn slows inbound
Halls act like ‘big MP3/DVD store houses
• Can’t over-throttle outbound, because it will impact inbound performance– Must experiment to find right level of limiting
Conclusions
• Technology Alone won’t resolve your bandwidth crisis• More Bandwidth Alone won’t resolve your bandwidth
crisis• Policy Alone won’t resolve your bandwidth crisis• Education and Student Involvement Alone won’t
resolve your bandwidth crisis
• It takes a combination of all four and an investment in time (and infrastructure)
In the end …
“The present generation of students uses computers and the Internet in a different way than does the generation of their parents or grandparents, who are of the same generation as the university administrators who make decisions about network use. To today's students, the computer is a composite tool both for their academic work and for recreation.”
Michael A. McRobbie, V.P for Information Technology
Questions & Answers
INDIANA UNIVERSITY
Richard SammisManager
IUB Network Operations
Melody ChildsManager
Residential IT Services
Brian D. VossAssociate Vice
PresidentTelecommunication
s
http://php.indiana.edu/~bvoss/bandwidth.html
