Christian Bird Empirical Software Engineering Group Microsoft Research, Redmond Empirical Software...
If you can't read please download the document
Christian Bird Empirical Software Engineering Group Microsoft Research, Redmond Empirical Software Engineering at Microsoft: Transitioning Research into
Christian Bird Empirical Software Engineering Group Microsoft
Research, Redmond Empirical Software Engineering at Microsoft:
Transitioning Research into Practice 1
Slide 2
A little about me... Computer Science Undergrad Worked for
large tech company writing software Went back to school because
There has to be a more principled way Decisions & Policies
based on intuition and anecdotal evidence complex code has more
severe bugs! Just add more people Correlation => Causation XML
will solve all problems No bugs if 100% statement coverage 2
Slide 3
Many projects run over budget Most ship late And yet more money
is spent on maintenance after release than before (80%) I dont
think its because developers are inept Developing Software is
Expensive and Time Consuming "The F-35 mission systems software
development and test is tending towards familiar historical
patterns of extended development and deferrals to later
increments." 3
Cholera Outbreak of 1854 Outbreak began in Soho, London on
August 31, 1854 Many false ideas Miasma Divine Intervention It just
happens Government concluded they could do nothing 5
Slide 6
Dr. Snows Cholera Investigation Had hypotheses about spread of
Cholera Interviewed families Collected geographic data Observed the
community 6
Slide 7
The beginning of a scientific field Broad Street Pump
Considered the beginning of epidemiology Cholera outbreak stopped
7
Slide 8
How do we make projects work better more often? Empirical
Method Gather Data Examine Relationships Make Changes & Build
Tools 8
Slide 9
Does anyone care? Improve processes Target resources Improve
quality and productivity They do if we ask the right questions!
$300 Billion Market 9
Slide 10
Social Dynamics in Programming Design and programming are human
activities; forget that and all is lost. - Bjarne Stroustrop. The
C++ Programming Language 10
Slide 11
Results in this talk 1. 1. How does ownership and expertise
affect software quality? 2. 2.How do we determine who should be
coordinating work? 11
Slide 12
Distributed development is difficult, but possible to do with
little effect on post-release failures. 12
Slide 13
Christian Bird, Nachiappan Nagappan, Brendan Murphy, Harald
Gall, Premkumar Devanbu Dont Touch My Code! Examining the Effects
of Ownership on Software Qualit y 13
Slide 14
Dealing with Large Systems Divide system into modules and
interfaces Assign modules to teams/developers This leads to strong
ownership practices in commercial contexts 14
Slide 15
Is this a good thing? thin spread of application domain
knowledge is a big problem (Curtis et al, [CACM 1988]) Expertise is
related to code contributions (Mockus [ICSE 2002], McDonald [CSCW
2000]) Authorship improves code understanding (Fritz et al [ICSE
2010]) Repeated exposure has positives outcomes in other
disciplines (Darr [Management Science, 1995]) 15
Slide 16
Ownership & Expertise Can we quantify the effect of
component ownership on defects? What is the effect of many
contributions from people with low expertise? 16
Slide 17
Ownership Terms Major Contributor a developer that has made at
least 5% of the total commits. Minor Contributor a developer that
has made less than 5% of the total commits. Ownership the
proportion of commits made by the highest contributing developer.
On a per component basis 17
Slide 18
Windows VistaWindows 7
CategoryMetricPre-releasePost-releasePre-releasePost-release
Ownership Metrics Total0.840.700.920.24 Minor0.860.700.930.25
Major0.260.29-0.40-0.14 Ownership-0.49 -0.29-0.02 "Classical"
Metrics Size0.750.690.700.26 Churn0.720.690.710.26
Complexity0.700.530.560.37 Failure Correlation Analysis Minor
Contributors has higher correlation than any other measure 18
Slide 19
Correlations can be deceiving Guess which has more failures Now
guess which is larger, more complex, and had more changes 19
Slide 20
Regression Analysis Allows us to control for component
characteristics such as size, complexity, and churn Skewed
distribution Use variance explained (R 2 ) to evaluate model
improvement 20
Slide 21
Regression Analysis Results Addition of all measures were
statistically significant. Total had less of an affect and improved
model less than Minor. Model Windows VistaWindows 7
Pre-releasePost-releasePre-releasePost-release Base (code
metrics)26%29%24%18% Base + Total40% (+14%)35% (+6%)68% (+35%)21%
(+3%) Base + Minor46% (+20%)41% (+12%)70% (+46%)21% (+3%) Base +
Minor + Major48% (+2%)43% (+2%)71% (+1%)22% (+1%) Base + Minor +
Major + Ownership50% (+2%)44% (+1%)72% (+1%)22% (+0%) 21
Slide 22
Relationship to Failures Metric Effects of Failures Size,
Complexity, ChurnMedium Positive Total ContributorsLarge Positive
Minor ContributorsLargest Positive Major ContributorsSmall Positive
OwnershipSmall Negative Only statistically significant in 3 of 4
cases 22
Slide 23
But WHY do some components have so many minor contributors?
23
Slide 24
The Major-Minor-Dependency relationship Dependency Major
Contributor Minor Contributor Foo.ex e I need to fix Foo I need to
change Bar which is used by Foo Bar.dll 24 Monte Carlo simulation
showed that MMD happens twice as often as would be expected by
chance
Slide 25
Use Monte Carlo simulation to compare observed phenomenon with
random graphs exhibiting the same major and minor degree
distributions Is this anything? 25
Slide 26
Graph Rewiring Do this n 2 times Is this anything? Foo.ex e
Bar.dll Baz.sy s kernel.dl l sol.exe ie.exe Monte Carlo showed that
MMD happens twice as often as would be expected in a random graph
26
Slide 27
Replicating Defect Prediction Pinzger, Nagappan, Murphy [FSE
08] Whole Network Precision75% Recall 82% Without Minors
Precision44% Recall 58% Without Majors Precision84% Recall 88%
Foo.ex e Bar.dll Baz.sy s kernel.dl l sol.exe ie.exe Minor
Contributors are vital to predictive power 27
Slide 28
Recommendations 1. 1. Changes made by minor contributors should
be reviewed with more scrutiny. 2. 2. Potential minor contributors
should communicate desired changes rather than making them. 3. 3.
Components with low ownership should be given priority by QA
resources. 28
Slide 29
Takeaways Components that have changes from low expertise
developers have more failures (even when controlling for the usual
suspects). Dependency relationships are the driving factor behind
many minor contributors. Ownership relationships affect defect
prediction techniques. But! Practices and Policies can improve
ownership! 29
Slide 30
More people More code Coordination overhead can dominate the
project and breakdowns, leading to: Decreased Productivity Lower
Quality More coordination The Problem of Large Software 30
Slide 31
The Solution? Divide it up! Parnas introduced the notion of a
module, a responsibility assignment. Business Logic Web Service
User Interface Database Layer Reporting Difficult in practice Some
tasks cross many components Some components are affected by many
tasks Tasks and components have dependencies on each other 31
Slide 32
Branches to the rescue Create a separate workspace for
development of a feature, fix, or maintenance task. Initial stable
state Development work Completio n Deliver changes 32
Slide 33
The Cost of Isolation Branches allow a temporary reprieve from
requirements of awareness. Conflicting changes to the system will
eventually manifest. 33
Does STC apply to branching? Branch 1 Branch 2 Coordination
Requirement work related work 35
Slide 36
Our theory of branches 1. 1. Branch is created for some goal
(add a feature, fix bugs on a subsystem, etc.) 2. 2. Developers
making modifications define a virtual team. 3. 3. Different teams
working on branches with similar goals introduce coordination
requirements may go unmet. 36
Slide 37
How do we identify similar branches? A Branch is characterized
in two ways: The changes required to accomplish the goal of the
branch The contributors making those changes 37
Slide 38
Operationalizing Branch Profiles 38
Slide 39
Branch Profile Vectors 39
Slide 40
Measuring Similarity On any given branch Small proportion of
contributors are an order of magnitude more active than the others
Subset of files account for the vast majority of the changes.
40
Slide 41
Branch Similarity Example Goal Similarity: 0.81 Team
Similarity: 0.03 41
Slide 42
How often is our theory followed? Compare all pairs of branches
by file similarity and developer similarity. Dark areas mean many
branch pairs in that area. Some of these are branches with few
files, but there are some high volume branches with disparate
teams. Most pairs of branches are not similar Same devs working on
different things is OK Same files should mean same people Same
files, but different team means possible problems 42
Slide 43
Which Branches Need Coordination? Compare all pairs of branches
by file similarity and developer similarity. Dark areas mean many
branch pairs in that area. Same files, but different team means
potential problems Different Files Same Files Different Teams Same
Teams 43
Slide 44
Quantitative Analysis Similarity Correlation Team Similarity
> Goal Similarity 44
Slide 45
Next Steps Identify outcomes and their relationship to
adherence to or violation of our theory. Provide real-time tools to
project managers to alert them to coordination needs. Branch Health
45
Slide 46
Code Flow for a Single File Blue nodes are edits to the file
Orange nodes are move operations 46
Slide 47
Assessing a Branch Simulate alternate branch structure to
assess cost and benefit of individual branches Cost: Average Delay
Increase per Edit How much delay does a branch introduce into
development? Cost: Integrations per Edit on a Branch What is the
integration/edit ratio within a branch? Benefit: Provided Isolation
per Edit How many conflicts does a branch prevent per edit? 47
Slide 48
Removing a Single Branch 48
Slide 49
49
Slide 50
50
Slide 51
51
Slide 52
52
Slide 53
53
Slide 54
54
Slide 55
Which Branch Should I Remove? Delay (cost) Provided Isolation
(benefit) 55
Slide 56
Low ownership leads to poor quality. We can identify
coordination needs. Empirical Software Engineering 56