Will RSpec help us to do better science

from imagination to impact © 2011 NICTA. All Rights Reserved.

Maximilian Ott With lots of help from Thierry Rakotoarivelo & Guillaume Jourjon

Will RSpec help us to do better Science?

2 © 2011 NICTA

3 © 2011 NICTA

Why are we doing what we are doing?

4 © 2011 NICTA

Curious Bunch - Want to figure out new things

5 © 2011 NICTA

The advancement of science depends in general on the interplay between experimental studies and theory

Wikipedia

6 © 2011 NICTA

Many interesting Things only happen at Scale

Young Hyun, CAIDA

7 © 2011 NICTA

Need for LARGE Experimental Facilities

8 © 2011 NICTA

9 © 2011 NICTA

10 © 2011 NICTA

Hypothesis

Experiment

Observation

Analysis

11 © 2011 NICTA

Hypothesis

Experiment

Observation

Analysis

12 © 2011 NICTA

How “well” do we do experiments today?

Missing Model

Validation

Missing Experiment Description

Missing Replication Information

Missing Precision

Information

OK

30% (10)

“Evaluation” section of SIGCOMM 2010 papers

58% (19)

21% (7) 9%

(3) 12% (4)

13 © 2011 NICTA

How can we improve this?

•  The Art & Science of Experimentation –  The art of designing experiments –  Scientifically sound methodology & analysis

•  Dissemination –  Overcoming the 6 page paper information barrier

•  Learning –  Reflection on the methodology –  Developing Best Practices

•  Tools –  To streamline processes –  To record processes and outcomes

14 © 2011 NICTA

Need for Methodologies & Best Practices

Observations

Model 1 Model 2 …

Ha , H0 Ha , H0

Experiment Experiment

New Obs + Analysis

15 © 2011 NICTA

Need to teach Experimenting at Scale

Internet

High Speed

Dedicated

Network

Control

Network

Remote

Experiment

Server

Student

Student

Experiment

Queue

Tested application and protocol

Measurement points

Network

In the Box

ireel.npc.nicta.com.au

16 © 2011 NICTA

Need new ways to disseminate Findings

Authors are invited to submit Full Papers of up to 10 pages and Testbed Practices Papers of up to 6 pages, including references, figures and tables, formatted according to the IEEE conference format. ?

17 © 2011 NICTA

LabWiki: Publishing Your Lab Book

Studied Phenomenon

Model 1

AnalysesFalsify and/or

refine the models

HA , H0

ExperimentExperimentExperiment

Observations

Model i

HA , H0

ExperimentExperimentExperiment

...

Portal

OEDL Script

Measure-ments

database

R Script

0

20

40

60

80

100

120

140

160

180

200

0 50 100 150 200 250 300 0

50000

100000

150000

200000

250000

300000

350000

Dis

tan

ce

(m

)

Ba

nd

wid

th

Time (s)

DistanceBandwidth

18 © 2011 NICTA

Observations

Model 1 Model 2 …

Ha , H0 Ha , H0

Experiment Experiment

New Obs + Analysis

Notes

Code

Experiment Description

Analysis

Runs

Measurements

19 © 2011 NICTA

Observations

Model 1 Model 2 …

Ha , H0 Ha , H0

Experiment Experiment

New Obs + Analysis

Notes

Code

Experiment Description

Analysis

Runs

Measurements

20 © 2011 NICTA

Observations

Model 1 Model 2 …

Ha , H0 Ha , H0

Experiment Experiment

New Obs + Analysis

Notes

Code

Experiment Description

Analysis

Runs

Measurements

Summary

21 © 2011 NICTA

Summary

Researcher

Summary

Reader

Comment

Reviewer

Analysis

Review

22 © 2011 NICTA

We are working on it

•  OMF –  OEDL to describe experiments –  Experiment Controller to run them

•  OML –  Capture measurements & meta-data

•  Portal –  Wiki: Keep notes with references –  Schedule & monitor runs –  Analysis with integrated R server –  Repository for all artifacts

23 © 2011 NICTA

App 1

App 2

e1

e2

c1

Measure

ment Lib

rary

(O

ML)

ControlNetwork

. . .

. . .

Experiment Node

OML Server

SQL Database

Control Node

Proxy Server

ON/OFF

Sensingdaemon

System Monitor

sensor device

Y/Z

Coo

rdin

tate

X Coordinate

Link Quality:Very GoodGoodPoorVery Poor

Wiki

OMF Portal

OML

R OEDL Visualization

24 © 2011 NICTA

Call to Action

•  The Art & Science of Experimentation –  The art of designing experiments –  Scientifically sound methodology & analysis

•  Dissemination –  There need to be better ways than 10 page papers

•  Learning –  Reflection on the methodology –  Developing Best Practices –  Teaching experimentation at scale

•  Tools –  To streamline processes –  To record processes and outcomes

25 © 2011 NICTA

omf.mytestbed.net wiki.mytestbed.net max.ott@nicta.com.au