Embed Size (px)
DESCRIPTION
To detect integration errors as quickly as possible, organizations use automated build systems. Such systems ensure that (1) the developers are able to integrate their parts into an executable whole; (2) the testers are able to test the built system; (3) and the release engineers are able to leverage the generated build to produce the upcoming release. The flipside of automated builds is that any incorrect change can break the build, and hence testing and releasing, and (even worse) block other developers from continuing their work, delaying the project even further. To measure the impact of such build breakage, this empirical study analyzes 3,214 builds produced in a large software company over a period of 6 months. We found a high ratio of build breakage (17.9%), and also quantified the cost of such build breakage ranging from 904.64 to 2034.92 man-hours. Interviews with 28 software engineers from the company helped to understand the circumstances under which builds are broken and the effects of build breakages on the collaboration and coordination of teams. We quantitatively investigated the main factors impacting build breakage and found that build failures correlate with the number of simultaneous contributors on branches, the type of work items performed on a branch, and the roles played by the stakeholders of the builds (for example developers vs. integrators). http://mcis.polymtl.ca/publications/2014/icsm14_noureddine.pdf
Citation preview
Why do Automated Builds Break?
An Empirical Study MC IS
http://imgs.steps.dragoart.com/how-to-draw-a-flying-dragon-dragon-in-flight-step-8_1_000000102347_5.gifhttp://www.printed-editions.com/upload/standard/Harold_Edgerton_Pigeon_Release_69.jpg
NoureddineKerzazi
FoutseKhomh
BramAdams
Imagine 2,000 developers working in the same branch, and one breaks the build, then you end
up with 1,999 unhappy people!
[T. Zimmermann @ SCAM 2014]
Trunk- vs. Branch-based Development
3
What is "breaking the build"?
Trunk- vs. Branch-based Development
3
What is "breaking the build"?
trunk ...
branch 1
Trunk- vs. Branch-based Development
3
What is "breaking the build"?
trunk ...
branch 1
branch 2
Trunk- vs. Branch-based Development
3
What is "breaking the build"?
trunk ...
branch 1
branch 2
Trunk- vs. Branch-based Development
3
What is "breaking the build"?
trunk ...
branch 1
branch 2
Trunk- vs. Branch-based Development
3
What is "breaking the build"?
trunk
integrate
buildtest
...
branch 1
branch 2
Trunk- vs. Branch-based Development
3
What is "breaking the build"?
trunk
integrate
buildtest
...
branch 1
branch 2
Trunk- vs. Branch-based Development
3
What is "breaking the build"?
trunk
integrate
buildtest
...
branch 1
branch 2
Trunk- vs. Branch-based Development
3
What is "breaking the build"?
trunk
integrate
buildtest
...
branch 1
branch 2
Trunk- vs. Branch-based Development
3
What is "breaking the build"?
trunk
integrate
buildtest
down-time
integrate
buildtest
...
branch 1
branch 2
Trunk- vs. Branch-based Development
3
What is "breaking the build"?
trunk
down-time
team-level breakage
...
branch 1
branch 2
Trunk- vs. Branch-based Development
3
What is "breaking the build"?
trunk
integrate
buildtest
down-time
team-level breakage
...
branch 1
branch 2
Trunk- vs. Branch-based Development
3
What is "breaking the build"?
trunk
merge
down-time
team-level breakage
...
branch 1
branch 2
Trunk- vs. Branch-based Development
3
What is "breaking the build"?
trunk
merge
down-time
team-level breakage
integrate
buildtest
...
branch 1
branch 2
Trunk- vs. Branch-based Development
3
What is "breaking the build"?
trunk
merge
down-time
down-time
team-level breakage
integrate
buildtest
...
branch 1
branch 2
Trunk- vs. Branch-based Development
3
What is "breaking the build"?
trunk
merge
sync
down-time
down-time
team-level breakage
X
integrate
buildtest
...
branch 1
branch 2
Trunk- vs. Branch-based Development
3
What is "breaking the build"?
trunk
merge
sync
down-time
down-time
team-level breakage
X
integrate
buildtest
integrate
buildtest
...
branch 1
branch 2
Trunk- vs. Branch-based Development
3
What is "breaking the build"?
trunk
merge
sync
down-time
down-time
down-time
team-level breakage
X
integrate
buildtest
integrate
buildtest
...
branch 1
branch 2
Trunk- vs. Branch-based Development
3
What is "breaking the build"?
trunk
merge
sync
down-time
down-time
down-time
team-level breakage
X
integrate
buildtest
integrate
buildtest
integrate
buildtest
...
branch 1
branch 2
Trunk- vs. Branch-based Development
3
What is "breaking the build"?
trunk
merge
sync merge
down-time
down-time
down-time
team-level breakage
X
integrate
buildtest
...
branch 1
branch 2
Trunk- vs. Branch-based Development
3
What is "breaking the build"?
trunk
merge
sync merge
down-time
down-time
down-time
team-level breakage
X
integrate
buildtest
integrate
buildtest
...
branch 1
branch 2
Trunk- vs. Branch-based Development
3
What is "breaking the build"?
trunk
merge
sync merge
down-time
down-time
down-time
team-level breakage
project-level breakage
X
...
4
Build Breakage should be Avoided!
you"are"stuck
your"team"members"are"stuck
testers"are"stuck
(a0er"merge)other+teams"are"stuckdeadlines"slip
morale"drops
5
RQ1. How does Build Breakage
Impact a Project?
RQ2. Why does the Build Break?
RQ3. Which Factors Impact Build Breakage?
Web-Based, Financial System
6
200 employees
>1.5 MLOC .NET (8,524 files)
25 branches6 inter-dependent projects
2 geographical sites
agile development
across different
branches6 months of data
model 7
interviewmeasure
triangula9on
8
RQ1. How does Build Breakage
Impact a Project?
RQ2. Why does the Build Break?
RQ3. Which Factors Impact Build Breakage?
Median of 8.3% Broken Builds per Day (avg. of 22.3%)
9
%broken builds
0"
10"
20"
30"
40"
50"
60"
70"
80"
90"
100"
18,07,13"
25,07,13"
01,08,13"
08,08,13"
15,08,13"
22,08,13"
29,08,13"
05,09,13"
12,09,13"
19,09,13"
26,09,13"
03,10,13"
10,10,13"
17,10,13"
24,10,13"
31,10,13"
07,11,13"
14,11,13"
21,11,13"
28,11,13"
05,12,13"
12,12,13"
19,12,13"
26,12,13"
02,01,14"
09,01,14"
16,01,14"
23,01,14"
30,01,14"0
10
20
3040
50
60
70
80
90100
0"
100"
200"
300"
400"
500"
600"
700"
800"
900"
1000"
Architecture)
Branch-2)
Branch-3)
Branch-4)
Branch-5)
Branch-6)
Branch-7)
Branch-8)
Branch-9)
Branch-10)
Branch-11)
Branch-12)
Branch-13)
Branch-14)
Branch-15)
Branch-16)
QuickFix)
Branch-18)
PreRelease)
Branch-20)
Branch-21)
Trunk)
Branch-23)
Branch-24)
Branch-25)
DownD
me)by)branch)(m
in))
Average Median Downtime of 1 Hour per Branch
10
#min.
re-architecting can cause very long
downtime
100
200
300
400
500
600
700
800
900
1000
Estimating Cost of Team-Level Build Breakage for Branch
11
X
integrate
buildtest
integrate
buildtest
time
integrate
buildtest
integrate
buildtest
time
+
...+
$$$(branch)in man-hours=
Estimating Cost of Team-Level Build Breakage for Project
12
$$$(project)in man-hours =
...
$$$(branch 1)in man-hours
$$$(branch 2)in man-hours
+
+
Total of 1.25 to2.83Man-Months Wasted (in 6 months)
13
0100
200
300
400
500
Branch−18
Branch−2
Branch−5
QuickFix
Branch−12
Branch−21
Branch−10
Branch−25
Branch−3
Branch−11
Branch−23
Branch−15
Branch−7
Branch−14
Branch−24
PreR
elease
Branch−4
Branch−6
Branch−20
Branch−13
Branch−16
Branch−8
Arch.
Branch−9
Trunk
re-architecting
fixing trunk
upper bound lower bound
M/H
14
1 hour of downtime (avg.) per branch
>=1.25 man-months lost
14
RQ2. Why does the Build Break?
1 hour of downtime (avg.) per branch
>=1.25 man-months lost
Semi-Structured Interviews
15
architects
integrator
technical leads
testers
front-end developers
back-end developers
>3 years of
experience
20th percentile most successful/broken builds
50 minutes
Analysis of Interviews
16
awareness (Q1-Q4)
causes (Q5-Q6)
prevention (Q7-Q8)
X
Q1.
Q2.
...card sort
25% of Interviewees Track the Build Results
17
I switched off the Build notifications to avoid any distraction. I check the results of my builds only in predefined time windows within the day.
I’m always aware about the build notification, especially the red ones. When a build notification for a branch shows up, I examine not only that build, but also the history of build failure on that branch.
Awareness of a Breakage Takes on Average 3 Hours
18
When we see in the work items system tracker that the developer is done and we do not see the build result, then we have to communicate with the concerned developer to verify where the latest code to test is.
Breakage Takes on Average 1 Hour to Resolve
19
>concerned about project's overall
consistency
concerned about own feature/bug fix
3 Major Causes of Build Breakage
20
forgetting to commit certain files
(libraries, templates, configuration files, ...)
committing from the wrong
workspace (branch)
build scripts do not specify transitive
library dependencies
feature
libA 1.0
libB 2.1
feature
libA 1.0
libB 1.5
Developers Active in 4 to 15 Branches in Parallel
21
I was assigned bugs on a specific project within the overall solution. I unload all other projects from the solution to focus only on one project.
I am a nomad moving from a branch to another. I map one workspace by branch to avoid check-in work in progress related to other branch.
Only 12% has Routine for Avoiding Breakages
22
Before I check- in, I always compare diligently my pending changes with the latest version from the server. In this way I am not wasting team members’ time with broken builds.
New developers joining a development project should be required to learn the basic practices of code committing, as part of the process known as onboarding
23
1 hour of downtime (avg.) per branch
>=1.25 man-months lost
25% track CI results
detection takes 3 h
caused by missing file
(dependencie)s and
accidental commits
active in 4-15 branches in parallel
23
RQ3. Which Factors Impact Build Breakage?1 hour of downtime
(avg.) per branch
>=1.25 man-months lost
25% track CI results
detection takes 3 h
caused by missing file
(dependencie)s and
accidental commits
active in 4-15 branches in parallel
Statistical Model of Impact
24
vs. vs. vs. vs. vs.
code/file churn
commit vs. merge
feature vs. bug fix vs. integration
P1 P2 P3 P4
local vs. remote
day of week/time of day
breakage?
Geog_Distance
Working_Hour
Working_Day
Churn..Files.
Churn..Code.
Timeline.P1_P4.
Build_Type
Work_Items
NumberofContributors
Role
●
●
●
●
●
●
●
●
●
●
10 20 30 40 50
mtcars.rf
%IncMSE
Factors having Higher Breakage
25
larger team
%incMSEAUC: 0.78
Geog_Distance
Working_Hour
Working_Day
Churn..Files.
Churn..Code.
Timeline.P1_P4.
Build_Type
Work_Items
NumberofContributors
Role
●
●
●
●
●
●
●
●
●
●
10 20 30 40 50
mtcars.rf
%IncMSE
Factors having Higher Breakage
25
P1 P2 P3 P4
larger team
P1
%incMSEAUC: 0.78
Geog_Distance
Working_Hour
Working_Day
Churn..Files.
Churn..Code.
Timeline.P1_P4.
Build_Type
Work_Items
NumberofContributors
Role
●
●
●
●
●
●
●
●
●
●
10 20 30 40 50
mtcars.rf
%IncMSE
Factors having Higher Breakage
25
P1 P2 P3 P4
time of day
larger team
P1
late commits, longer downtime
%incMSEAUC: 0.78
Geog_Distance
Working_Hour
Working_Day
Churn..Files.
Churn..Code.
Timeline.P1_P4.
Build_Type
Work_Items
NumberofContributors
Role
●
●
●
●
●
●
●
●
●
●
10 20 30 40 50
mtcars.rf
%IncMSE
Factors having Higher Breakage
25
vs. vs. vs. vs. vs.
P1 P2 P3 P4
time of day
larger team
P1
late commits, longer downtime
%incMSE
architects
AUC: 0.78
Geog_Distance
Working_Hour
Working_Day
Churn..Files.
Churn..Code.
Timeline.P1_P4.
Build_Type
Work_Items
NumberofContributors
Role
●
●
●
●
●
●
●
●
●
●
10 20 30 40 50
mtcars.rf
%IncMSE
Factors having Higher Breakage
25
vs. vs. vs. vs. vs.
feature vs. bug fix vs. integration
P1 P2 P3 P4
time of day
integration > feature > fix
larger team
P1
late commits, longer downtime
%incMSE
architects
AUC: 0.78
Geog_Distance
Working_Hour
Working_Day
Churn..Files.
Churn..Code.
Timeline.P1_P4.
Build_Type
Work_Items
NumberofContributors
Role
●
●
●
●
●
●
●
●
●
●
10 20 30 40 50
mtcars.rf
%IncMSE
Factors having Higher Breakage
25
vs. vs. vs. vs. vs.
feature vs. bug fix vs. integrationcommit vs. merge
P1 P2 P3 P4
time of day
integration > feature > fix
merge > commit
larger team
P1
late commits, longer downtime
%incMSE
architects
AUC: 0.78
Geog_Distance
Working_Hour
Working_Day
Churn..Files.
Churn..Code.
Timeline.P1_P4.
Build_Type
Work_Items
NumberofContributors
Role
●
●
●
●
●
●
●
●
●
●
10 20 30 40 50
mtcars.rf
%IncMSE
Factors having Higher Breakage
25
vs. vs. vs. vs. vs.
feature vs. bug fix vs. integrationcommit vs. merge
P1 P2 P3 P4
code churnfile churn
time of day
integration > feature > fix
merge > commit
more churn
larger team
P1
late commits, longer downtime
%incMSE
architects
AUC: 0.78
Geog_Distance
Working_Hour
Working_Day
Churn..Files.
Churn..Code.
Timeline.P1_P4.
Build_Type
Work_Items
NumberofContributors
Role
●
●
●
●
●
●
●
●
●
●
10 20 30 40 50
mtcars.rf
%IncMSE
Factors having Higher Breakage
25
vs. vs. vs. vs. vs.
feature vs. bug fix vs. integrationcommit vs. merge
P1 P2 P3 P4
code churnfile churn
local vs. remote
time of day
integration > feature > fix
merge > commit
more churn
local teams
larger team
P1
late commits, longer downtime
%incMSE
architects
AUC: 0.78
Geog_Distance
Working_Hour
Working_Day
Churn..Files.
Churn..Code.
Timeline.P1_P4.
Build_Type
Work_Items
NumberofContributors
Role
●
●
●
●
●
●
●
●
●
●
10 20 30 40 50
mtcars.rf
%IncMSE
Factors having Higher Breakage
25
vs. vs. vs. vs. vs.
feature vs. bug fix vs. integrationcommit vs. merge
P1 P2 P3 P4
code churnfile churn
local vs. remote
day of week
time of day
integration > feature > fix
merge > commit
more churn
local teams
larger team
P1
late commits, longer downtime
%incMSE
architects
AUC: 0.78
Geog_Distance
Working_Hour
Working_Day
Churn..Files.
Churn..Code.
Timeline.P1_P4.
Build_Type
Work_Items
NumberofContributors
Role
●
●
●
●
●
●
●
●
●
●
10 20 30 40 50
mtcars.rf
%IncMSE
Factors having Higher Breakage
25
vs. vs. vs. vs. vs.
feature vs. bug fix vs. integrationcommit vs. merge
P1 P2 P3 P4
code churnfile churn
local vs. remote
day of week
time of day
integration > feature > fix
merge > commit
more churn
local teams
larger team
P1
late commits, longer downtime
%incMSE
architects
AUC: 0.78
26
1 hour of downtime (avg.) per branch
>=1.25 man-months lost
25% track CI results
detection takes 3 h
caused by missing file
(dependencie)s and
accidental commits
active in 4-15 branches in parallel
role
team size in branch
type of work item/
build
timeline
branch 1
branch 2
Trunk- vs. Branch-based Development
3
What is "breaking the build"?
trunk
merge
sync merge
down-time
down-time
down-time
team-level breakage
project-level breakage
X
...
model 7
interviewmeasure
triangula9on
Total of 1.25 to2.83Man-Months Wasted (in 6 months)
13
0100
200
300
400
500
Branch−18
Branch−2
Branch−5
QuickFix
Branch−12
Branch−21
Branch−10
Branch−25
Branch−3
Branch−11
Branch−23
Branch−15
Branch−7
Branch−14
Branch−24
PreR
elease
Branch−4
Branch−6
Branch−20
Branch−13
Branch−16
Branch−8
Arch.
Branch−9
Trunk
upper bound lower bound
26
1 hour of downtime (avg.) per branch
>=1.25 man-months lost
25% track CI results
detection takes 3 h
caused by missing file
(dependencie)s and
accidental commits
active in 4-15 branches in parallel
role
team size in branch
type of work item/
build
timeline
![Peeking Behind the NAT: An Empirical Study of Home Networksconferences.sigcomm.org/imc/2013/papers/imc061-groverA.pdf · Our work builds on our own previous work [32], which uses](https://img.pdfslide.net/doc/110x75/5f45318534e45704ba33dbf9/peeking-behind-the-nat-an-empirical-study-of-home-our-work-builds-on-our-own-previous.jpg)
