Embed Size (px)
DESCRIPTION
These are the slides for a plenary talk I gave at GEC 11 in Denver in July 2011. The primary message is that we, the networking community need to fundamentally improve the way we do experimental research.
Citation preview
from imagination to impact © 2011 NICTA. All Rights Reserved.
Maximilian Ott With lots of help from Thierry Rakotoarivelo & Guillaume Jourjon
Will RSpec help us to do better Science?
2 © 2011 NICTA
3 © 2011 NICTA
Why are we doing what we are doing?
4 © 2011 NICTA
Curious Bunch - Want to figure out new things
5 © 2011 NICTA
The advancement of science depends in general on the interplay between experimental studies and theory
Wikipedia
6 © 2011 NICTA
Many interesting Things only happen at Scale
Young Hyun, CAIDA
7 © 2011 NICTA
Need for LARGE Experimental Facilities
8 © 2011 NICTA
9 © 2011 NICTA
10 © 2011 NICTA
Hypothesis
Experiment
Observation
Analysis
11 © 2011 NICTA
Hypothesis
Experiment
Observation
Analysis
12 © 2011 NICTA
How “well” do we do experiments today?
Missing Model
Validation
Missing Experiment Description
Missing Replication Information
Missing Precision
Information
OK
30% (10)
“Evaluation” section of SIGCOMM 2010 papers
58% (19)
21% (7) 9%
(3) 12% (4)
13 © 2011 NICTA
How can we improve this?
• The Art & Science of Experimentation – The art of designing experiments – Scientifically sound methodology & analysis
• Dissemination – Overcoming the 6 page paper information barrier
• Learning – Reflection on the methodology – Developing Best Practices
• Tools – To streamline processes – To record processes and outcomes
14 © 2011 NICTA
Need for Methodologies & Best Practices
Observations
Model 1 Model 2 …
Ha , H0 Ha , H0
Experiment Experiment
New Obs + Analysis
15 © 2011 NICTA
Need to teach Experimenting at Scale
Internet
High Speed
Dedicated
Network
Control
Network
Remote
Experiment
Server
Student
Student
Experiment
Queue
Tested application and protocol
Measurement points
Network
In the Box
ireel.npc.nicta.com.au
16 © 2011 NICTA
Need new ways to disseminate Findings
Authors are invited to submit Full Papers of up to 10 pages and Testbed Practices Papers of up to 6 pages, including references, figures and tables, formatted according to the IEEE conference format. ?
17 © 2011 NICTA
LabWiki: Publishing Your Lab Book
Studied Phenomenon
Model 1
AnalysesFalsify and/or
refine the models
HA , H0
ExperimentExperimentExperiment
Observations
Model i
HA , H0
ExperimentExperimentExperiment
...
Portal
OEDL Script
Measure-ments
database
R Script
0
20
40
60
80
100
120
140
160
180
200
0 50 100 150 200 250 300 0
50000
100000
150000
200000
250000
300000
350000
Dis
tan
ce
(m
)
Ba
nd
wid
th
Time (s)
DistanceBandwidth
18 © 2011 NICTA
Observations
Model 1 Model 2 …
Ha , H0 Ha , H0
Experiment Experiment
New Obs + Analysis
Notes
Code
Experiment Description
Analysis
Runs
Measurements
19 © 2011 NICTA
Observations
Model 1 Model 2 …
Ha , H0 Ha , H0
Experiment Experiment
New Obs + Analysis
Notes
Code
Experiment Description
Analysis
Runs
Measurements
20 © 2011 NICTA
Observations
Model 1 Model 2 …
Ha , H0 Ha , H0
Experiment Experiment
New Obs + Analysis
Notes
Code
Experiment Description
Analysis
Runs
Measurements
Summary
21 © 2011 NICTA
Summary
Researcher
Summary
Reader
Comment
Reviewer
Analysis
Review
22 © 2011 NICTA
We are working on it
• OMF – OEDL to describe experiments – Experiment Controller to run them
• OML – Capture measurements & meta-data
• Portal – Wiki: Keep notes with references – Schedule & monitor runs – Analysis with integrated R server – Repository for all artifacts
23 © 2011 NICTA
App 1
App 2
e1
e2
c1
Measure
ment Lib
rary
(O
ML)
ControlNetwork
. . .
. . .
Experiment Node
OML Server
SQL Database
Control Node
Proxy Server
ON/OFF
Sensingdaemon
System Monitor
sensor device
Y/Z
Coo
rdin
tate
X Coordinate
Link Quality:Very GoodGoodPoorVery Poor
Wiki
OMF Portal
OML
R OEDL Visualization
24 © 2011 NICTA
Call to Action
• The Art & Science of Experimentation – The art of designing experiments – Scientifically sound methodology & analysis
• Dissemination – There need to be better ways than 10 page papers
• Learning – Reflection on the methodology – Developing Best Practices – Teaching experimentation at scale
• Tools – To streamline processes – To record processes and outcomes
