39
Experimentation on Networking Testbeds Fraida Fund NYU Polytechnic School of Engineering [email protected]

Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

  • Upload
    others

  • View
    4

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

Experimentation on Networking Testbeds

Fraida FundNYU Polytechnic School of Engineering

[email protected]

Page 2: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

Why do we experiment?

● Validation○ Test performance of new and existing applications,

protocols, and systems● Exploration

○ Open new areas of investigation.

Page 3: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

Simulation vs. “Real” Experimentation

Page 4: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

What’s wrong with simulations?

Simulations are used very widely in networking research, but these suffer from● Simplifying assumptions● Misapplication of models and parameters● Not all simulators are created equal

Page 5: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

Simplifying Assumptions

Common assumptions in radio propagation models used by wireless simulations [3]:

Page 6: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

Simplifying Assumptions: Example

Default propagation model in most simulators assumes the world is “flat”

Real radio signals are affected by the terrain

Page 7: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

Misapplication of parametersDefault parameters are not “sane” in all scenarios!

● wireless simulators are complex with many “knobs”○ ns-2 version 2.27 had 674 default variables [1]

● “out-of-the-box” configuration may not be what you think it is

Results are only valid if appropriate parameters are used

Page 8: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

Parameters vary widely and are often chosen without any good reason.

Example: Simulation parameters reported across 61 Mobile Ad-Hoc Network (MANET) papers [1]

Misapplication of parameters

Page 9: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

Differences between simulatorsExample: one Mobile Ad-Hoc Network (MANET) flooding algorithm evaluated in three widely used* network simulators, using the same parameters [2]

● OPNET (6.3%)● ns-2 (43.8%)● GloMoSim (10.0%)

* The number in parentheses tells what percent of papers in ACM’s MobiHoc conference (2000-2005) used each simulator [1]

Page 10: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

Differences between simulators

Metric 1: Effective transmission range differs in absolute value and also the general trend between simulators

Page 11: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

Differences between simulators

Metric 2: Effect of mobility on protocol reliability. Reliability was more than 2x smaller in ns-2 than in the others

Page 12: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

Differences between simulators

Metric 3: Effect of mobility on protocol overhead. OPNET receives three times more duplicate packets

Page 13: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

Differences between simulators

Metric 4: Average time to flood the network is much higher in OPNET

Page 14: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

Simulations are a great tool, but ● they are only as good as the simulator, the

settings, and the assumptions used by the model

Page 15: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

Why experiment on testbeds?

● Why can’t I run my experiment on my own phone, home WiFi network, laptop computer?

Page 16: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

Why experiment on testbeds?Example: Wireless experiments conducted in a hallway between two buildings on a college campus [4]

Results depend on what direction you move along the hallway

Page 17: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

Why experiment on testbeds?

● Why can’t I run my wireless experiment on my own phone, home WiFi network, laptop computer?

Answer: to do so, you must assume● your RF environment is “typical”● you understand your environment ● your environment is the same each time you run the experiment● if you can’t detect wireless interference, it isn’t there

Page 18: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

Networking testbeds

Testbeds are designed for reproducibility and lend themselves to a methodical approach

Page 19: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

Networking testbeds

reproducibility: the ability of an entire experiment or study to be reproduced, either by the researcher or by someone else working independently

Page 20: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

Networking testbeds

reproducible research: the idea that scientific claims are published together with their data and software code so that others may verify the findings and build upon them.

Page 21: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

Networking Testbeds

Page 22: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

Fide

lity

to “

real

” In

tern

et

Programmability of network

Simulation (ns-2/ns-3,OPNET)

Emulation(Emulab, DETER)

Testbed with Programmable Network

(ORBIT, CORNET)

Collection of Heterogeneous Testbeds

(GENI, similar projects)

Internet as a Testbed(Seattle)

Distributed Testbed

(PlanetLab)

In-house computer lab(typical environment)

Page 23: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

GENI

Page 24: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

GENI infrastructure

Page 25: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

GENI rack

A "rack" is a cluster of high-powered servers ("compute nodes") connected to a management switch and an OpenFlow switch.

Page 26: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

Management switch

OpenFlow switch

Compute Node

Compute Node

Compute Node

Compute Node

Internet

GENI backbone network

Page 27: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

Management switch

OpenFlow switch

Bob’s Reserved Compute Node

Bob’s Reserved Compute Node

Alice’s Reserved Compute Node

Alice’s Reserved Compute Node

Internet

GENI backbone network

Page 28: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

Management switch

OpenFlow switch

Bob’s Reserved Compute Node

Bob’s Reserved Compute Node

Alice’s Reserved Compute Node

Alice’s Reserved Compute Node

Internet

GENI backbone network

Page 29: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

Testbed concepts

● Aggregate, resource○ Aggregate types: InstaGENI, ExoGENI, are rack

● Project, Slice, Sliver● RSpec

Page 30: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

EL 6383 Lab/Project

Page 31: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

Overview

The lab and project component for this course includes:● lab lectures on:

○ networking testbeds (right now!)○ experiment design,○ communicating results, statistics, ○ methodical approach to experimentation,

reproducible results

Page 32: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

Overview (continued)

● lab exercises (tentative plan):○ Hello GENI○ Flow and congestion control ○ Routing○ Software defined networks

(applying concepts from course material, together with lab lectures on network performance evaluation.)

Page 33: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

Overview (continued)

● project○ work in groups of 3 or 4○ can choose to extend a lab exercise or do something

new○ must apply concepts from lab lectures on network

performance evaluation○ must use methodical, reproducible approach

Page 34: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

Overview (continued)

● reproducibility○ every student will be assigned a project from another

group○ you will reproduce their results (following their

instructions) and evaluate and review the quality of their work

Page 35: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

Timeline

● 4 lab exercises before midterm● Project proposals due after midterm● Project submission due 3 weeks before end

of course● Reviews due at last class session

Details on requirements, submission instructions, etc. will be announced on lab website and by email

Page 36: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

Logistics

● Lab assignments and project requirements will be posted on lab website:http://witestlab.poly.edu/~ffund/el6383/

● Lab 1 (Hello GENI) will be posted tomorrow○ due 2/11

Page 37: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

Lab 1: Hello GENIUse your GENI account for the first time and do some one time setup

● Login to the GENI Experimenter Portal for the first time● Join a project (EL6383-NYU-S2015)● Specify SSH keys for use logging into resources

Learn how to reserve, login to, and release resources in GENI● Create and renew a slice● Generate and reserve your own topology of GENI resources using Flack● Learn how to login to compute resources using ssh● Learn how to delete resources in GENI

Understand the difference between the control and data plane interfaces on each node● Use these resources to do a very simple first experiment● Use ping to test connectivity between the reserved resources● Use ifconfig to identify your data and control plane interfaces

Page 38: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

Getting help

● Office hours on lab website○ later in the semester, there will be more frequent

office hours● Asking for help on the Internet

○ GENI Users Group: https://groups.google.com/forum/#!forum/geni-users

○ If you ask a question, cite it in your report

Page 39: Networking Testbeds Experimentation on - NYU - Homewitestlab.poly.edu/~ffund/el6383/files/Experimentation+on+Testbeds.pdf · Experimentation on Networking Testbeds Fraida Fund NYU

References1. Stuart Kurkowski, Tracy Camp, and Michael Colagrosso. 2005. MANET simulation studies: the

incredibles. SIGMOBILE Mob. Comput. Commun. Rev. 9, 4 (October 2005), 50-61. 2. David Cavin, Yoav Sasson, and Andre Schiper. 2002. On the accuracy of MANET simulators. In

Proceedings of the second ACM international workshop on Principles of mobile computing (POMC '02). ACM, New York, NY, USA, 38-43.

3. David Kotz, Calvin Newport, Robert S. Gray, Jason Liu, Yougu Yuan, and Chip Elliott. 2004. Experimental evaluation of wireless simulation assumptions. In Proceedings of the 7th ACM international symposium on Modeling, analysis and simulation of wireless and mobile systems(MSWiM '04). ACM, New York, NY, USA, 78-82.

4. Ryan Burchfield, Ehsan Nourbakhsh, Jeff Dix, Kunal Sahu, S. Venkatesan, and Ravi Prakash. 2009. "RF in the Jungle": effect of environment assumptions on wireless experiment repeatability. In Proceedings of the 2009 IEEE international conference on Communications(ICC'09). IEEE Press, Piscataway, NJ, USA, 4993-4998.

5. Shivendra Panwar. “Testbeds and the evolving research paradigm” (Slides). 2012. Panel on Network Testbeds at the 26th IEEE Computer Communications Workshop (CCW 2012).