Experimental Evaluation of Experimental Evaluation of Pair ProgrammingPair Programming
Experimental Evaluation of Experimental Evaluation of Pair ProgrammingPair Programming
Copyright, 2001 © Jerzy R. Nawrocki
European Software Control & MetricsEuropean Software Control & Metrics
ESCOM’01ESCOM’01
European Software Control & MetricsEuropean Software Control & Metrics
ESCOM’01ESCOM’01
Jerzy Nawrocki, Adam WojciechowskiPoznan University of TechnologyPoznan University of Technology
Poznan, PolandPoznan, Poland
J. Nawrocki, Experimental Evaluation ..
Plan of the lecturePlan of the lecturePlan of the lecturePlan of the lecture
IntroductionIntroductionPersonal Software ProcessPersonal Software ProcesseXtremme ProgrammingeXtremme ProgrammingDescription of the experimentDescription of the experimentResultsResultsConclusionsConclusions
J. Nawrocki, Experimental Evaluation ..
IntroductionIntroductionIntroductionIntroduction
Pair programming
CreatorCreator Quality Quality assurerassurer
if (x=y)if (x=y) z=0;z=0;
Must beMust bex==yx==y
J. Nawrocki, Experimental Evaluation ..
IntroductionIntroductionIntroductionIntroduction
After some time..
CreatorCreatorQuality Quality assurerassurer
x-=y;x-=y;elseelse
How toHow totest it?test it?
J. Nawrocki, Experimental Evaluation ..
IntroductionIntroductionIntroductionIntroduction
Nosek’s experiment
Write a script that Write a script that performs a database performs a database consistency check.consistency check.
J. Nawrocki, Experimental Evaluation ..
IntroductionIntroductionIntroductionIntroduction
Nosek’s experiment
30 minutes
42 minutes
Completion time (aver.)Completion time (aver.)
J. Nawrocki, Experimental Evaluation ..
IntroductionIntroductionIntroductionIntroduction
Nosek’s experiment
71 %
100 %
Completion Completion timetime
EffortEffort
143 %
100 %
J. Nawrocki, Experimental Evaluation ..
IntroductionIntroductionIntroductionIntroduction
Main weakness
‘‘If several tasks each take an If several tasks each take an hour, combine them to form a hour, combine them to form a larger tasklarger task.’.’
Kent BeckKent Beck
Extreme Programming ExplainedExtreme Programming Explained
‘‘If several tasks each take an If several tasks each take an hour, combine them to form a hour, combine them to form a larger tasklarger task.’.’
Kent BeckKent Beck
Extreme Programming ExplainedExtreme Programming Explained
Only one short assignment (45’).Only one short assignment (45’).Only one short assignment (45’).Only one short assignment (45’).
J. Nawrocki, Experimental Evaluation ..
IntroductionIntroductionIntroductionIntroduction
The Utah experiment
1414
11
22
33
. . .. . .
1414
11
22
33
. . .. . .
4 programming assignments4 programming assignments
6 weeks6 weeks
4 programming assignments4 programming assignments
6 weeks6 weeks
50% - 60%
100%
Completion timeCompletion time
J. Nawrocki, Experimental Evaluation ..
IntroductionIntroductionIntroductionIntroduction
Weak points
1414
11
22
33
. . .. . .
1414
11
22
33
. . .. . .
What were the assignments What were the assignments about?about?
What was the process?What was the process?
What were the sizes?What were the sizes?
What was the time in hours?What was the time in hours?
What was the deviation in What was the deviation in time and size?time and size?
What were the assignments What were the assignments about?about?
What was the process?What was the process?
What were the sizes?What were the sizes?
What was the time in hours?What was the time in hours?
What was the deviation in What was the deviation in time and size?time and size?
J. Nawrocki, Experimental Evaluation ..
IntroductionIntroductionIntroductionIntroduction
About our experiment
Aim: experimental evaluation of pair programming.Aim: experimental evaluation of pair programming.
XP-like processXP-like process
PSP-like processPSP-like process
Fall semester 1999/2000;Poznan University of Technology, Poznan, PolandFall semester 1999/2000;Poznan University of Technology, Poznan, Poland
J. Nawrocki, Experimental Evaluation ..
Plan of the lecturePlan of the lecturePlan of the lecturePlan of the lecture
IntroductionIntroduction
Personal Software ProcessPersonal Software ProcesseXtremme ProgrammingeXtremme ProgrammingDescription of the experimentDescription of the experimentResultsResultsConclusionsConclusions
J. Nawrocki, Experimental Evaluation ..
Personal Software ProcessPersonal Software ProcessPersonal Software ProcessPersonal Software Process
Time and defects are recorded.Defect type standard.
00
Coding standard. Size measurement.Process Improvement Proposal.
0.10.1
Software size estimation.Test reports.
11
Task planning.Schedule planning.
1.11.1
22Design templates.Code & design reviews.
2.12.1
Incremental approach 33
J. Nawrocki, Experimental Evaluation ..
Personal Software ProcessPersonal Software ProcessPersonal Software ProcessPersonal Software Process
PlanningPlanning
DesigningDesigning
CodingCoding
CompilingCompiling
TestingTesting
PostmortemPostmortem
RequirementsRequirements
Product + Product + datadata
J. Nawrocki, Experimental Evaluation ..
Plan of the lecturePlan of the lecturePlan of the lecturePlan of the lecture
IntroductionIntroductionPersonal Software ProcessPersonal Software Process
eXtremme ProgrammingeXtremme ProgrammingDescription of the experimentDescription of the experimentResultsResultsConclusionsConclusions
J. Nawrocki, Experimental Evaluation ..
eXtreme Programming (XP)eXtreme Programming (XP)eXtreme Programming (XP)eXtreme Programming (XP)
Pair programming.
Test-centred quality assurance.
Simple solution.
Spike solutions.
Keep moving.
Pair programming.
Test-centred quality assurance.
Simple solution.
Spike solutions.
Keep moving.
We applied:We applied:
User stories.
An on-site customer representative.
Planning game.
CRC cards.
Continuous integration
User stories.
An on-site customer representative.
Planning game.
CRC cards.
Continuous integration
We didn’t apply:We didn’t apply:
J. Nawrocki, Experimental Evaluation ..
Plan of the lecturePlan of the lecturePlan of the lecturePlan of the lecture
IntroductionIntroductionPersonal Software ProcessPersonal Software ProcesseXtremme ProgrammingeXtremme Programming
Description of the experimentDescription of the experimentResultsResultsConclusionsConclusions
J. Nawrocki, Experimental Evaluation ..
Description of the experimentDescription of the experimentDescription of the experimentDescription of the experiment
XP-like pair XP-like pair programmingprogramming
XP2XP2
PSP 0.1 (time, PSP 0.1 (time, defect & size defect & size measurement)measurement)
PSPPSP
Test-centred Test-centred QA, Spike QA, Spike solutionssolutions
XP1XP1
J. Nawrocki, Experimental Evaluation ..
Description of the experimentDescription of the experimentDescription of the experimentDescription of the experiment
Programming assignments
1. Estimate the mean and standard deviation of a sample of n real numbers.1. Estimate the mean and standard deviation of a sample of n real numbers.
2. Calculate the linear regression parameters.2. Calculate the linear regression parameters.
3. Count the logical lines in a program, omitting comments and blank lines.3. Count the logical lines in a program, omitting comments and blank lines.
4. Count the total program LOC, the total LOC in each object the program contains, and the number of methods in each object.
4. Count the total program LOC, the total LOC in each object the program contains, and the number of methods in each object.
C/C++C/C++
J. Nawrocki, Experimental Evaluation ..
Plan of the lecturePlan of the lecturePlan of the lecturePlan of the lecture
IntroductionIntroductionPersonal Software ProcessPersonal Software ProcesseXtremme ProgrammingeXtremme ProgrammingDescription of the experimentDescription of the experiment
ResultsResultsConclusionsConclusions
J. Nawrocki, Experimental Evaluation ..
ResultsResultsResultsResults
Completion Time (Average)
0:00
1:12
2:24
3:36
4:48
Zad 1 Zad 2 Zad 3 Zad 4
PSP
XP1
XP2
There is almost no difference between XP2 and XP1.There is almost no difference between XP2 and XP1.
Prog 1 Prog 2 Prog 3 Prog 4
J. Nawrocki, Experimental Evaluation ..
ResultsResultsResultsResults
Completion Time (Std. Deviation)
0:00
0:28
0:57
1:26
1:55
2:24
Zad 1 Zad 2 Zad 3 Zad 4
PSP
XP1
XP2
Prog 1 Prog 2 Prog 3 Prog 4
Pair programming is more predictable than Pair programming is more predictable than individual one.individual one.
J. Nawrocki, Experimental Evaluation ..
ResultsResultsResultsResults
Average Program Size (LOC)
0
100
200
300
400
500
Zad 1 Zad 2 Zad 3 Zad 4
PSP
XP1
XP2
Prog 1 Prog 2 Prog 3 Prog 4
J. Nawrocki, Experimental Evaluation ..
ResultsResultsResultsResults
Standard Deviation in Code Size
0
50
100
150
200
Zad 1 Zad 2 Zad 3 Zad 4
PSP
XP1
XP2
Prog 1 Prog 2 Prog 3 Prog 4
Pair programming leads to more stable solutions.Pair programming leads to more stable solutions.
J. Nawrocki, Experimental Evaluation ..
ConclusionsConclusionsConclusionsConclusions
• XP-like pair programming appears XP-like pair programming appears less efficient than it is reported by less efficient than it is reported by J.T. Nosek and L. Williams et al.J.T. Nosek and L. Williams et al.
• Pair programming is more Pair programming is more predictable one than individual one predictable one than individual one both in completion time and both in completion time and program size.program size.
• The experiment was restricted to The experiment was restricted to relatively small programs (150 - 400 relatively small programs (150 - 400 LOC).LOC).