Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
Experimentation on Networking Testbeds
Fraida FundNYU Polytechnic School of Engineering
Why do we experiment?
● Validation○ Test performance of new and existing applications,
protocols, and systems● Exploration
○ Open new areas of investigation.
Simulation vs. “Real” Experimentation
What’s wrong with simulations?
Simulations are used very widely in networking research, but these suffer from● Simplifying assumptions● Misapplication of models and parameters● Not all simulators are created equal
Simplifying Assumptions
Common assumptions in radio propagation models used by wireless simulations [3]:
Simplifying Assumptions: Example
Default propagation model in most simulators assumes the world is “flat”
Real radio signals are affected by the terrain
Misapplication of parametersDefault parameters are not “sane” in all scenarios!
● wireless simulators are complex with many “knobs”○ ns-2 version 2.27 had 674 default variables [1]
● “out-of-the-box” configuration may not be what you think it is
Results are only valid if appropriate parameters are used
Parameters vary widely and are often chosen without any good reason.
Example: Simulation parameters reported across 61 Mobile Ad-Hoc Network (MANET) papers [1]
Misapplication of parameters
Differences between simulatorsExample: one Mobile Ad-Hoc Network (MANET) flooding algorithm evaluated in three widely used* network simulators, using the same parameters [2]
● OPNET (6.3%)● ns-2 (43.8%)● GloMoSim (10.0%)
* The number in parentheses tells what percent of papers in ACM’s MobiHoc conference (2000-2005) used each simulator [1]
Differences between simulators
Metric 1: Effective transmission range differs in absolute value and also the general trend between simulators
Differences between simulators
Metric 2: Effect of mobility on protocol reliability. Reliability was more than 2x smaller in ns-2 than in the others
Differences between simulators
Metric 3: Effect of mobility on protocol overhead. OPNET receives three times more duplicate packets
Differences between simulators
Metric 4: Average time to flood the network is much higher in OPNET
Simulations are a great tool, but ● they are only as good as the simulator, the
settings, and the assumptions used by the model
Why experiment on testbeds?
● Why can’t I run my experiment on my own phone, home WiFi network, laptop computer?
Why experiment on testbeds?Example: Wireless experiments conducted in a hallway between two buildings on a college campus [4]
Results depend on what direction you move along the hallway
Why experiment on testbeds?
● Why can’t I run my wireless experiment on my own phone, home WiFi network, laptop computer?
Answer: to do so, you must assume● your RF environment is “typical”● you understand your environment ● your environment is the same each time you run the experiment● if you can’t detect wireless interference, it isn’t there
Networking testbeds
Testbeds are designed for reproducibility and lend themselves to a methodical approach
Networking testbeds
reproducibility: the ability of an entire experiment or study to be reproduced, either by the researcher or by someone else working independently
Networking testbeds
reproducible research: the idea that scientific claims are published together with their data and software code so that others may verify the findings and build upon them.
Networking Testbeds
Fide
lity
to “
real
” In
tern
et
Programmability of network
Simulation (ns-2/ns-3,OPNET)
Emulation(Emulab, DETER)
Testbed with Programmable Network
(ORBIT, CORNET)
Collection of Heterogeneous Testbeds
(GENI, similar projects)
Internet as a Testbed(Seattle)
Distributed Testbed
(PlanetLab)
In-house computer lab(typical environment)
GENI
GENI infrastructure
GENI rack
A "rack" is a cluster of high-powered servers ("compute nodes") connected to a management switch and an OpenFlow switch.
Management switch
OpenFlow switch
Compute Node
Compute Node
Compute Node
Compute Node
Internet
GENI backbone network
Management switch
OpenFlow switch
Bob’s Reserved Compute Node
Bob’s Reserved Compute Node
Alice’s Reserved Compute Node
Alice’s Reserved Compute Node
Internet
GENI backbone network
Management switch
OpenFlow switch
Bob’s Reserved Compute Node
Bob’s Reserved Compute Node
Alice’s Reserved Compute Node
Alice’s Reserved Compute Node
Internet
GENI backbone network
Testbed concepts
● Aggregate, resource○ Aggregate types: InstaGENI, ExoGENI, are rack
● Project, Slice, Sliver● RSpec
EL 6383 Lab/Project
Overview
The lab and project component for this course includes:● lab lectures on:
○ networking testbeds (right now!)○ experiment design,○ communicating results, statistics, ○ methodical approach to experimentation,
reproducible results
Overview (continued)
● lab exercises (tentative plan):○ Hello GENI○ Flow and congestion control ○ Routing○ Software defined networks
(applying concepts from course material, together with lab lectures on network performance evaluation.)
Overview (continued)
● project○ work in groups of 3 or 4○ can choose to extend a lab exercise or do something
new○ must apply concepts from lab lectures on network
performance evaluation○ must use methodical, reproducible approach
Overview (continued)
● reproducibility○ every student will be assigned a project from another
group○ you will reproduce their results (following their
instructions) and evaluate and review the quality of their work
Timeline
● 4 lab exercises before midterm● Project proposals due after midterm● Project submission due 3 weeks before end
of course● Reviews due at last class session
Details on requirements, submission instructions, etc. will be announced on lab website and by email
Logistics
● Lab assignments and project requirements will be posted on lab website:http://witestlab.poly.edu/~ffund/el6383/
● Lab 1 (Hello GENI) will be posted tomorrow○ due 2/11
Lab 1: Hello GENIUse your GENI account for the first time and do some one time setup
● Login to the GENI Experimenter Portal for the first time● Join a project (EL6383-NYU-S2015)● Specify SSH keys for use logging into resources
Learn how to reserve, login to, and release resources in GENI● Create and renew a slice● Generate and reserve your own topology of GENI resources using Flack● Learn how to login to compute resources using ssh● Learn how to delete resources in GENI
Understand the difference between the control and data plane interfaces on each node● Use these resources to do a very simple first experiment● Use ping to test connectivity between the reserved resources● Use ifconfig to identify your data and control plane interfaces
Getting help
● Office hours on lab website○ later in the semester, there will be more frequent
office hours● Asking for help on the Internet
○ GENI Users Group: https://groups.google.com/forum/#!forum/geni-users
○ If you ask a question, cite it in your report
References1. Stuart Kurkowski, Tracy Camp, and Michael Colagrosso. 2005. MANET simulation studies: the
incredibles. SIGMOBILE Mob. Comput. Commun. Rev. 9, 4 (October 2005), 50-61. 2. David Cavin, Yoav Sasson, and Andre Schiper. 2002. On the accuracy of MANET simulators. In
Proceedings of the second ACM international workshop on Principles of mobile computing (POMC '02). ACM, New York, NY, USA, 38-43.
3. David Kotz, Calvin Newport, Robert S. Gray, Jason Liu, Yougu Yuan, and Chip Elliott. 2004. Experimental evaluation of wireless simulation assumptions. In Proceedings of the 7th ACM international symposium on Modeling, analysis and simulation of wireless and mobile systems(MSWiM '04). ACM, New York, NY, USA, 78-82.
4. Ryan Burchfield, Ehsan Nourbakhsh, Jeff Dix, Kunal Sahu, S. Venkatesan, and Ravi Prakash. 2009. "RF in the Jungle": effect of environment assumptions on wireless experiment repeatability. In Proceedings of the 2009 IEEE international conference on Communications(ICC'09). IEEE Press, Piscataway, NJ, USA, 4993-4998.
5. Shivendra Panwar. “Testbeds and the evolving research paradigm” (Slides). 2012. Panel on Network Testbeds at the 26th IEEE Computer Communications Workshop (CCW 2012).