Upload
job-blake
View
217
Download
0
Tags:
Embed Size (px)
Citation preview
Hands on Experiment Design
Foutse Khomh, Queen’s University, CanadaMassimiliano Di Penta, University of Sannio,
Italy
Use of acceptance test during change tasks
Usage of natural language to specify (change) requirements: current state of the practice highly inaccurate error prone
“Agile movement” advocates [Melnik et alt. 2004]: Acceptance test cases constitute an expressive
form of documentation Acceptance test cases are “talking”
representation of the requirements We focus on evaluating whether Fit
acceptance tests are useful in maintenance/evolution tasks
Framework for Integrated Test
The most well-known open source implementation of the table-based acceptance testing approach. A table represents a set of test cases.
Fit lets: customers/analysts write “executable”
acceptance tests using simple HTML tables (Fit tables).
developers write “Fixtures” to link the test cases with the actual system itself.
to compare (Test Runner) these test cases with actual values, returned by the system, and highlights the results with colors.
Red for failures Green for passed tests.
“The complete picture”
Text Reqs.
Fit Table
Fixture
Analyst(and customer) (i, o)
System
io’
Developer
o ≠ o’
Test Runner
Output Fit TableO = expected output
O’ = actual output
failure
Examples of Fit Tables
Core Fit tables:Column Fit tables for testing
calculationsAction Fit tables for testing
user interfaces or workflows;
Row Fit tables to check the result of a query
(some) other Fit tables:Summary Fit tablesHtml Fit tablesCommand line Fit tables…
fit_tests.DiscountStructure
small bags
beverage
discount
total price()
2 Coffee 0 1.24
4 Tea 0 2.48
5 Coffee 1 2.1
5 Tea 0 3.1
7 Coffee 1 3.34
7 Tea 0 4.34
fit.ActionFixture
start fit_tests.VerifySupply
enter type of product Coffee
check number of small bug remained 10
enter number of box 5
press buying boxes
check number of small bug remained 310
Column Fit table
Action Fit table 1 box = 60 small bags
small bag price = 0.62
FitNesse
A collaborative testing and documentation tool.
It supports Java (eclipse plug-in), .Net, C++
It Combines Fit with a Wiki Web for writing “natural language requirements” + Fit tables.
It provides a simple way to run tests (Fit tables) and suits.
It supports sub Wikis for managing multiple projects.
http://fitnesse.org
Text only vs. Text + Fit
Text only Text + Fit
Change Requirement: Change price of Boxes
The vendor of boxes of beverages changed his selling policy. Each five bought boxes (of the same type) one is added as a gift.
Change Requirement: Change price of Boxes
The vendor of boxes of beverages changed his selling policy. Each five bought boxes (of the same type) one is added as a gift.
fit.ActionFixturestart fit_tests.VerifySupplyenter type of product Coffeecheck
number of small bug remained 310
enter number of box 11
press buying boxescheck
number of small bug remained 960
check cash account 504
What to compare
Where do we start?
We have an idea/conjecture about a cause effect relation
We have a theory Thus we can formulate a hypothesis And to test it.. we run an experiment!
The planning steps follows...
Planning: steps
Contextselection
Definition
Hypothesisformulation
Variableselection
Selection ofsubjects
Experimentdesign
InstrumentationValidity
evaluationExperiment
design
Study planning
Context Selection
Select the type of study to perform: survey, case study, and experiment . In this training we focus on experiment (more details @page 10 of the lecture slides)
Identify the objects and subjects of the study (more details @ page 40-49 of the lecture slides)
Hypothesis formulation
Formulate the null hypothesis.
(more details @ page 50-54 of the lecture slides)
Variables Selection
Dependant variables Used to measure the effect of treatments Derived from hypotheses
(more details @ page 56-68 of the lecture slides)
Variables Selection
Independant variables Variables we can control and modify The choice depends on the domain knowledge One independent variable is the main factor of our
experiment Other independent variables are the co-factors
(more details @ page 70-72 of the lecture slides)
Experiment Design Is the set of treatment tests
Combinations of treatments, subjects and objects
Defines how tests are organized and executed
Based on the formulated hypotheses Experimental design is based on three
principles1. Randomization2. Blocking3. Balancing(more details @ page 74-98 of the
lecture slides)
Validity evaluation Answer the following crucial questions :
To what extent are our results valid? They should be at least valid for the population of interest Then, if we could generalize…
Mitigates the following threats to validity1. Conclusion validity (C)2. Internal validity (I)3. Construct validity (S)4. External validity (E)
Prioritize threats to validity: Many threats are conflictual
(more details @ page 101- 121 of the lecture slides)
SummaryWhat to produce: Experiment definition Context selection Hypotheses formulation Identification of dependent and
independent variables Experiment design Experiment procedure and identification
of the required material
Operation After having designed an experiment we
need to execute it We are in touch with subjects for the first
time Besides the pre-experiment briefing and
training
Although the design and plan are perfect, everything depends on the operation If something goes wrong in a couple of hours
we could waste months of work…
Experiment operation: steps
Preparation
Experimentdesign
Execution
Datavalidation
Experimentdata
Experiment operation
Preparation Obtain consent from participants Do not disseminate sensitive data Reward in some way participation Before the experiment, it is advisable to show
a short presentation explaining the experiment and its objectives
Introduce the objects and the instrumentation Describe in detail the steps of the experiment
(more details @ page 125- 130 of the lecture slides)
Execution Different ways to execute an experiment
Online You can actually monitor the experiment
Offline Distribute the task via email and wait for results
Run a post-experiment questionnaire to understand whether anything went wrong with
clarity of objectives material, time available, tasks and to collect some qualitative information.
(more details @ page 131- 136 of the lecture slides)
Data validation Once the experiment has been
completed, we need to do a consistency check on the collected data Were treatments correctly applied? Did subjects understand the provided forms? Did subjects correctly fill the forms? Remove subjects that
Did not participate to the experiment Exhibited a weird behavior (e.g. did not pay
attention to the task) Try to have a quick look at data as soon
as possible At least using descriptive statistics
Replication Replication is essential in experimentation
You should document your experiments so that others can
Repeat the experiment with different subjects Replicate your statistical analysis
Look at the non-replicable case of the cold fusion!
(more details @ page 138-139 of the lecture slides)