AN AUTOMATED APPROACH TO ASSIGN SOFTWARE CHANGE REQUESTS (Ph.D. Presentation)

An Automated Approach to Assign SoftwareChange Requests

Ph.D. Thesis

Yguarata Cerqueira Cavalcanti

Centro de Informatica – UFPE

March 20, 2014

Agenda

1 Introduction

2 Literature Review

3 Survey on CR Assignment

4 Proposal

5 Experiment

6 Conclusions

1 Introduction

2 Literature Review


4 Proposal

5 Experiment

6 Conclusions

Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions

Change Management

Every software project changes (1st Lehman’s law)

user needsdefectsnew functionalities

Changes are made during software development or after release(software maintenance and evolution)

Changes need to be managed, instead you lose control

component versionssoftware versions (different clients)

1/39 Yguarata Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests


Change Requests (CRs)

CR describes a defect to be fixed, an adaptive orperfective change, or a new functionality.

CRs are stored and managed through CR Repositories.



CR Assignment



Why CR Assignment Matters?

Select developers considering the low fixing time yet keepingsatisfactory quality

Needs good knowledge on the project

However, dozens to hundreds CRs daily

Labor-intensive and time consuming

Susceptible to mistakes

37%-44% of CRs did not reach the right developer

Reassignments (rework!)

































Research Objective

To propose an automated approach for CR assignment

Information Retrieval (IR) models

Rule-based expert systems

Context-aware information



Research Methodology


1 Introduction

2 Literature Review


4 Proposal

5 Experiment

6 Conclusions


Systematic Mapping Study

The process

1 Research questions

2 Searches in the literature (protocol)

3 Selection of papers, tools, and services

4 Classification (two schemes)

5 Analysis and synthesis of the results



Research Questions

Defined two questions for the mapping study

Question 1 – What are the current challenges andopportunities regarding CR repositories and how do theyimpact software development?

Question 02 – Do the tools and online services for CRmanagement address any of the challenges pointed out asa result of the answers to Question 01? If so, how do theyaddress such challenges?



Research Questions






Research Questions






Criteria

Inclusion:Theory, practice, and approaches

CR artifacts written in natural language

Unique studies

Exclusion:summaries of tutorial or workshop

posters

keynotes

studies with no scientific analysis

studies published in unknown sources



Searches

SEARCH ENGINES

Automated: Google, ACM, IEEE, Citeseer, Elsevier, Scirus,ScienceDirect, Scopus, ISI, SpringerLink, and Wiley

Manual: DBLP

KEYWORDS

Bug report, change request, modification request, defect track,software issue, bug tracking

STUDIES SELECTION

1150→ superficial reading → 321→ full reading → 142



Tools Selection and Analysis

Tools Online Services

Bugzilla http://www.bugzilla.org SourceForge http://www.sourceforge.net

MantisBT http://www.mantisbt.org Launchpad http://www.launchpad.net

Trac http://trac.edgewall.org Code Plex http://www.codeplex.com

Redmine http://www.redmine.org Google Code http://code.google.com

Jira http://www.atlassian.com GitHub http://www.github.com

Do they address any of the challenges?

How?


http://www.bugzilla.org

http://www.sourceforge.net

http://www.mantisbt.org

http://www.launchpad.net

http://trac.edgewall.org

http://www.codeplex.com

http://www.redmine.org

http://code.google.com

http://www.atlassian.com

http://www.github.com


Classification Schemes

Classification Scheme 1: created a taxonomy for Researchareas and topics

Classification Scheme 2: used a taxonomy for InformationRetrieval models



Classification Scheme 1

Taxonomy for Challenges and Opportunities



Classification Scheme 2

Information Retrieval (IR) Taxonomy

Representation Reasoning RepositoryQuery Document

With logic

Withuncer-tainty

With learning

CR

s(e

.g.

Bu

gzill

a)

Com

mit

Log

(e.g

.C

VS

,S

VN

)

Sou

rce

Co

de

Key

wor

d-b

ased

Pat

tern

-bas

ed

Str

uct

ura

l

Str

eam

ofC

har

acte

rs

Vec

tor

Sp

ace

Str

uct

ura

l

Log

ic

Alg

ebra

Gra

ph

Th

eori

es

Pro

bab

ility

Th

eori

es

Fu

zzy

Set

Th

eori

es

Neu

ral

Net

wor

k

Sym

bol

icL

earn

ing

Su

pp

ort

Vec

tor

Mac

hin

es

Dec

isio

nT

rees

/Tab

le

Laz

yL

earn

ing

Bay

esia

nS

tati

stic

s

Gen

etic

Alg

orit

hm

s

Reg

ress

ion

An

alys

is

Lea

rnto

Ran

k

Table: Taxonomy for the classification of the IR models and techniquesused in each approach. This is an extension of the taxonomy created byCanfora and Cerulo [1].



Concluding Remarks from the Review

Automated and Semi-Automated approaches for CRchallenges

Combinations of software repositories

Possibility of mixing up the approaches

Lack of contextual information in the approachesI.e.: CR assignment needs workload, developer knowledge,priority, and politics issues

Difficulty in assessing the approaches

State-of-the-art still far from the state-of-the-practice


1 Introduction

2 Literature Review


4 Proposal

5 Experiment

6 Conclusions


Survey’s Research Questions

RQ1. How much time does the CR Assignment activitiestake? (amount of CRs, individual time, and reassignments)

RQ2. What are the strategies used to assign CRs to theappropriate developers?

RQ3. What is the complexity involved in assigning CRs todevelopers?



Questionnaire

38 questions

8 open-ended

30 closed-ended (mostLikert-scaled)

Three steps validation



Questionnaire

38 questions

8 open-ended

30 closed-ended (mostLikert-scaled)

Three steps validation



Population Sample

Around 400 software developers from Brazilian FederalOrganization for Data Processing (SERPRO)

From three main sites in the south of Brazil

Porto Alegre, Florianopolis, and Curitiba



Responses

Periodically remainder emails

38 responses out of 400 (9%)

Is it enough? Yes!

In SERPRO, project leaders and managers are likely to havethe desired profile



Data Analysis I

RQ1. How much time does a CR assignment take?

It is common to assign almost 20 CRs per day

Each CR takes around 5 to 10 minutes to be assigned

Reassigning CRs is not so frequent in the SERPROorganization

20 CRs ∗ 10 min = 3.3 hours (per developer/day)

Plus reassignments (±10 minutes)

For bigger projects and open source it gets worse



Data Analysis II

RQ2. What are the strategies used to assign CRs?

1 Consider workload

2 Severity and criticality

3 Talk to developers before assignment

4 Select developers with more familiarity on the problem

5 Select developers who have solved similar CRs

6 Developers with better knowledge on the project

7 Developers who master the tools

8 Affinity



Data Analysis III

RQ3. What is the complexity involved in assigning CRs?

According to the strategies, CR assignments require:

Good knowledge on the project(s)

The ability of communicating to other people

The ability of information seeking in different repositories

The capability to retain the knowledge that is acquired duringthis cognitive process

Assign CRs to different teams

Assign CRs to different projects



Survey Replication

Application of the same survey design

Dataprev

Instituto Reconcavo de Tecnologia (IRT)

Confirmation of initial results


1 Introduction

2 Literature Review


4 Proposal

5 Experiment

6 Conclusions


The Solution

An Automated Approach to AssignSoftware Change Requests



Requirements

CRs must be assigned according to their

severity and criticality

workload of developers

developers experience

interpersonal relationships

rely on contextual information (software repositories)



Strategy to Automated CR Assignment



Rule-Based Expert System (RBES)

rule "Critical CRs, or CRs for module C"

when

$cr: ChangeRequest(severity == CRITICAL || module =="C")

then

$cr.assignTo(developer("[email protected]"))

end

rule "Change Requests for modules A and B"

when

$cr: ChangeRequest(module =="A" || module =="B")

then

$cr.assignTo(availableDeveloper(Workload.WEIGHTED ))

end



Information Retrieval Model With Learning

Support Vector Machine (SVM)

Training (Black arrows)

Recommendation (Gray arrows)

10-fold cross-validation


1 Introduction

2 Literature Review


4 Proposal

5 Experiment

6 Conclusions


Questions

Q1: What is the accuracy of the proposed approach forautomated CR assignment?

Q2: What is the necessary effort to setup the approach in asoftware development project?

Q3: Does the achieved accuracy pay the necessary effortneeded in the setup?



Experiment Design

Proposed approach versus pure SVM

Proposed approach: SVM, expert system and context



Experiment Design

Proposed approach versus pure SVM

Proposed approach: SVM, expert system and context



Hypotheses

Null Hypothesis

H0: µ(accuracy with our approach) <= µ(accuracy with SVM)

µ(payoff with our approach) <= µ(payoff with SVM)

Alternative Hypothesis

H1: µ(accuracy with our approach) > µ(accuracy with SVM)

µ(payoff with our approach) > µ(payoff with SVM)



Hypotheses

Null Hypothesis

H0: µ(accuracy with our approach) <= µ(accuracy with SVM)

µ(payoff with our approach) <= µ(payoff with SVM)

Alternative Hypothesis

H1: µ(accuracy with our approach) > µ(accuracy with SVM)

µ(payoff with our approach) > µ(payoff with SVM)



Testing dataset

CRs from two modules of Novo SIAFI project (SERPRO)

Module A = 781 CRs

Module B = 1031 CRs



Configuration of the Proposed Approach

Rules extraction

Interviews with 4 workers and analysis of CR samples

Total of 14 rules

Context information

developers vacation

developers project allocation


Assignment strategy configuration

1 execute simple rules

2 execute complex rules

3 SVM (instead of manual assignment)




Rules extraction


Total of 14 rules

Context information

developers vacation










Rules extraction


Total of 14 rules

Context information

developers vacation










Rules extraction


Total of 14 rules

Context information

developers vacation









Results I

Q1. What is the accuracy of the proposed approach forautomated CR assignment?

New approach: Module A = 45% and Module B = 34%

SVM: Module A = 38% and Module B = 23%

An improvement of 18% on Module A and 48% on B

Null hypothesis refuted



Results II

Q2. What is the necessary effort to setup the approach in asoftware development project?

38 hours (rule extraction, context information, strategy)

Q3. Does the achieved accuracy pay the necessary effortneeded in the setup?

10 minutes for each CR assigned

SVM saved 89 hours

New approach saved 117 hours

Economy of 28 hours vs. 38 hours for setup

Null hypothesis not refuted (for this context!)



Threats to the Validity

Generalization of the results (only CRs from one project)

Variety of metrics (Precision, Recall, and F-measure)

SVM learning process (quality of text data)

Difficult to assess the configuration time (trial and error forrules extraction)

Implementation of the approach (bug-free?)


1 Introduction

2 Literature Review


4 Proposal

5 Experiment

6 Conclusions


Conclusions I

Research Contribution

Mapping study on CR repositories investigation

Questionnaire-based survey with practitioners

An approach for automated CR assignment

Validation of the approach

Tools

Prototype and plugins

Test bed for new research



Conclusions II

Academic Contributions

Cavalcanti, Y. C., Neto, P. A. D. M. S., Machado, I. D. C., de Almeida, E. S.,

and de Lemos Meira, S. R. (2013b). Towards Understanding Software ChangeRequest Assignment: A survey with practitioners.In Proceedings of the 17th International Conference on Evaluation andAssessment in Software Engineering (EASE’2013), pages 195–206

Cavalcanti, Y. C., da Mota Silveira Neto, P. A., do Carmo Machado, I., Vale,

T. F., de Almeida, E. S., and de Lemos Meira, S. R. (2013a). Challenges andOpportunities for Software Change Request Repositories: a systematic mappingstudy.Journal of Software: Evolution and Process.Online first

More publications are under work: CBSoft’2014 tool session,ICSME’2014, JSEP journal



Conclusions III

Future work

Investigate new algorithms for workload balancing

Investigate methods and techniques for automatic extraction ofassignment rules

Perform new experimental studies

Address other issues of CR management


An Automated Approach to Assign SoftwareChange Requests

Ph.D. Thesis

Yguarata Cerqueira Cavalcanti

Centro de Informatica – UFPE

March 20, 2014

References

Lotka’s Law

Few developers fix the most of CRs


References

The Problem’s Characteristics

There are complex factors which influence CR assignment

Factors vary from one organization to another

Such as developers’ workload, CRs attributes, interpersonalrelationships, and developers know-how

Consider different rules for the assignments

Thus, automated approaches should be context-aware


References

Component Diagram of the Solution


References

Prototype Tool Architecture


References

References I

[1] Canfora, G. and Cerulo, L. (2004). A taxonomy of informationretrieval models and tools. Computing and InformationTechnology , 12(3), 175–194.

[2] Cavalcanti, Y. C., da Mota Silveira Neto, P. A.,do Carmo Machado, I., Vale, T. F., de Almeida, E. S., andde Lemos Meira, S. R. (2013a). Challenges and Opportunities forSoftware Change Request Repositories: a systematic mappingstudy. Journal of Software: Evolution and Process. Online first.

[3] Cavalcanti, Y. C., Neto, P. A. D. M. S., Machado, I. D. C.,de Almeida, E. S., and de Lemos Meira, S. R. (2013b). TowardsUnderstanding Software Change Request Assignment: A surveywith practitioners. In Proceedings of the 17th InternationalConference on Evaluation and Assessment in SoftwareEngineering (EASE’2013), pages 195–206.
