Automated topic naming to support cross-project analysisof software maintenance activities

Abram HindleDept. of Computer Science

University of California, DavisDavis, CA, USA

abram@softwareprocess.es

Neil A. ErnstDept. of Computer Science

University of TorontoToronto, Ontario, CANADAnernst@cs.toronto.edu

Michael W. GodfreyDavid Cheriton School of

Computer ScienceUniversity of Waterloo

Waterloo, Ontario, CANADAmigod@uwaterloo.ca

John MylopoulosDept. Information Eng. and

Computer ScienceUniversity of Trento

Trento, ITALYjm@disi.unitn.it

ABSTRACT

Researchers have employed a variety of techniques to ex-tract underlying topics that relate to software developmentartifacts. Typically, these techniques use semi-unsupervisedmachine-learning algorithms to suggest candidate word-lists.However, word-lists are difficult to interpret in the absenceof meaningful summary labels. Current topic modeling tech-niques assume manual labelling and do not use domain-specific knowledge to improve, contextualize, or describe re-sults for the developers. We propose a solution: automatedlabelled topic extraction. Topics are extracted using LatentDirichlet Allocation (LDA) from commit-log comments re-covered from source control systems such as CVS and Bit-Keeper. These topics are given labels from a generalizablecross-project taxonomy, consisting of non-functional require-ments. Our approach was evaluated with experiments andcase studies on two large-scale RDBMS projects: MySQLand MaxDB. The case studies show that labelled topic ex-traction can produce appropriate, context-sensitive labelsrelevant to these projects, which provides fresh insight intotheir evolving software development activities.

Categories and Subject Descriptors

D.2.9 [Software Engineering]: Management—Lifecycle;D.2.1 [Software Engineering]: Requirements/Specifications—Tools

General Terms

Human Factors, Management, Measurement

Keywords

Topic analysis, LDA, non-functional requirements

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.MSR ’11, May 21-22, 2011, Waikiki, Honolulu, HI, USACopyright 2011 ACM 978-1-4503-0574-7/11/05 ...$10.00.

1. INTRODUCTIONA key problem for practising software maintainers is gain-

ing an understanding of why a system has evolved the wayit has. This is different than how a system has evolved,because the behaviour of change itself is the how. Lookingback on streams of artifacts scattered across different repos-itories, inferring what activities were performed, when, andfor what reasons, is hard to do without expert advice fromthe developers involved. In this work we seek to providea method of automatically labelling development topics ex-tracted from commit logs.

Topic modeling is a machine learning technique which cre-ates multinomial distributions of words extracted from a textcorpus. This technique infers the hidden structure of a cor-pus using posterior inference: the probability of the hiddenstructure given the data. Topic models are useful in softwaremaintenance because they summarize the key concepts in acorpus, such as source code, commit comments, or mailing-list messages, by identifying statistically co-occurring words.Among other uses, it can quickly give developers an overviewof where significant activity has occurred, and gives man-agers or maintainers a sense of project history.

While machine learning techniques can automatically iden-tify clumps of commonly recurring terms, devising an ap-propriate summary label for each clump/topic is harder. Agiven topic extracted from a set of commit logs might consistof the following terms: “listener change remove add fire”.This topic might reasonably be labelled as “event handling”by a developer who understands the domain well, despitethe fact that this label does not appear in the word listitself. Current approaches to topic labelling rely on man-ual intervention by human experts, and also are limited toproject-specific topic labels. In this paper, we introduce la-belled topic extraction, an approach to topic labelling thatcreates labels automatically that are project independent.

In general, the fruits of mining software artifacts are of-ten project specific and hard to generalize. However, in ourprevious work we investigated topic trends — that is, topicsthat recur over time — we observed that topic trends oftencorresponded to non-functional requirements (NFRs) [11],which is further emphasized in this paper due to the largenumbers of NFR labelled topics. This is encouraging, as

NFRs have the property of being cross-domain and widelyapplicable. In this sense, they are useful abstractions fordeveloper conversations about different software projects.Furthermore, there is a series of standards on NFRs, suchas ISO9126 [12], that are specifically intended to apply toprojects of varying types; this suggests that our goal oftrying to extract NFR-related development topics, such asthose related to software quality models, holds promise.Concrete applications of topics and labelled topic extrac-

tion range from project dashboards to annotating softwareartifacts such as revisions and bug reports with NFR-relatedtags. Project dashboard [9] are typically employed by man-agers and are used to provide quick summaries of the effortput into a software project. In this case, labelled topic ex-traction would allow managers to track effort related to NFRtopics, such as portability. These techniques also allow forthe annotation of commit comments and other software arti-facts with NFRs. This would enable querying of bug reportsand artifacts by relevant NFRs. For instance a manager canconfirm if their developers were focused on usability by look-ing for usability-relevant revisions and bug reports.In this paper, we describe automated labelled topic extrac-

tion. It addresses two gaps in the topic mining literature:

1. Topic mining of software has been limited to one projectat a time. This is because traditional topic miningtechniques are specific to a particular data-set. Auto-mated labelled topic extraction allows for comparisonsbetween projects.

2. Topic modeling creates word lists that require inter-pretation by the user to assign meaning. Like (1), thismeans that it is difficult to discuss results indepen-dent of the project context. Our technique automati-cally, or with some initial training, assigns labels acrossprojects.

This paper makes the following contributions:

• introduces the concept of labelled topic extraction, us-ing a non-functional requirements (NFR) taxonomy forour labels;

• evaluates three kinds of automatic topic labelling meth-ods: semi-unsupervised labelling of topics (word-lists),supervised labelling of topics with a single NFR (ma-chine learning), and supervised labelling of topics withmultiple NFRs (multi-label machine learning);

• provides a method of cross-project analysis via topiclabelling; and applies these techniques to visualize NFRsover time, and to analyze maintenance activities.

We begin by discussing related work in Section 2. Next,we describe how we generated our data (Section 3.1). Forsemi-unsupervised classification (Section 3.2), we begin bycreating word-lists to signify when a topic matches an NFRlabel. We then apply our classifier and analyze the results.In Section 3.3, we manually annotate the topics, and usethose annotations as training data for supervised classifica-tion. To demonstrate an application of labelled topic extrac-tion, we use an exploratory case study of two open sourcedatabase systems to show how named topics can be com-pared between projects (Section 4). The paper concludeswith a discussion of limitations (Section 5), and future work.

2. PREVIOUS WORKThe idea of extracting higher-level concerns and topics,

also known as concepts, aspects or requirements, has beenapproached from documentation-based and repository-basedperspectives.

Cleland-Huang and her colleagues have investigated min-ing requirements documents for non-functional requirements(NFR) (quality requirements) [5]. Their approach is similarto ours, as they mined keywords from NFR catalogues. Theydiffer because they mine requirements documents where aswe mine revisions. They demonstrated a recall of 80% withprecision of 57% for the security NFR, but could not find areliable source of keywords for other NFRs. Instead, theydeveloped a supervised classifier by using human experts toidentify an NFR training set. Our research differs becausewe use a more comprehensive set of terms based on the tax-onomy we chose. Another difference is that we make cross-project comparisons instead of focusing on a single project.

Similarly, Mockus and Votta [18] studied a large-scale in-dustrial change-tracking system. Mockus and Votta lever-aged WordNet [8], an English-language “lexical database”that contains semantic relations between words, includingcommon related forms (similar to word stemming), meronymyand synonymy. They used WordNet for word roots as theyfelt the synonyms would be non-specific and cause errors.Mockus et al. validated their labels with system developers.Since we study multiple projects, instead of a single project,these kind of interviews are somewhat infeasible (particu-larly in the distributed world of open-source software).

Another approach is to extract concerns from softwarerepositories. Marcus et al. [15] use Latent Semantic Index-ing (LSI) to identify commonly occurring concerns for soft-ware maintenance. The concerns are given by the user, andLSI is used to retrieve them from a corpus. Topic mod-elling generates topics that are independent of a user query,and relate only to word frequencies in the corpus. WithConcernLines, Treude et al. [19] show tag occurrence usingcolour and intensity. They mine developer created tags inorder to analyze the evolution of a single product. The pres-ence of a well-maintained set of tags is obviously essentialto the success of this technique.

In Baldi et al. [2], topics are named manually: human ex-perts read the highest-frequency members of a topic and as-sign a label accordingly. As discussed earlier, given the topic“listener change remove add fire”, Baldi et al. would assignthe label event-handling. The labels are reasonable enough,but still require an expert in the field to determine them.Furthermore, these labels are project-specific, because theyare generated from the data of that project. E.g., we mighthave a label called ‘Oracle’ in the MySQL case, since Oracleowns MySQL. Our approach differs: first of all, we automatethe process of naming the topics; secondly, we label topicswith project-independent terms, in order to permit cross-project comparison.

Mei et al. [17] use context information to automaticallyname topics. They describe probabilistic labelling, using thefrequency distribution of words in a topic to create a mean-ingful phrase. They do not use external domain-specific in-formation as we do, but we do not generate phrases fromthe topics.

Finally, in Ernst et al. [6], we describe our earlier project,similar to this, that identifies changes in quality require-ments in GNOME software projects; this approach was more

Figure 1: Research methodology process view.

exploratory, had less validation, uses different word-lists,solely uses text-matching, and does not leverage machinelearning strategies. Hindle et al. [11] propose a windowedmethod of topic analysis that we extend with labelled top-ics, NFRs and new visualizations.

3. STUDY DESIGN AND EXECUTIONFigure 1 gives an outline of our methodology. We began by

gathering source data and creating topic models. For semi-unsupervised labelling, we generated three sets of word-listsas signifiers for NFRs. In supervised learning, we trainedour data with manual annotations in order to match topicswith NFRs. Finally, these topics were used to analyze therole of NFRs in software maintenance.

3.1 Generating the dataTo evaluate our approach, we sought candidate systems

that were mature projects and had openly accessible sourcecontrol repositories. We selected systems from the sameapplication domain, to control for differences in functional,rather than non-functional, requirements. We selected MySQL3.23 and MaxDB 7.500 as they were open-source, partially-commercial database systems. MaxDB started in the late1970s as a research project, and was later acquired by SAP.As of version 7.500, released April 2007, the project has over940, 000 lines of C source code1. The MySQL project startedin 1994 and MySQL 3.23 was released in early 2001. MySQLcontains 320, 000 lines of C and C++ source code. We ex-plicitly chose older versions of mature projects from a stable

1generated using David A. Wheeler’s SLOCCount,http://dwheeler.com/sloccount.

problem domain to increase the likelihood that we wouldencounter primarily maintenance activities in our studies.

For each project, we used source control commit com-ments, the messages that programmers write when theycommit revisions to a source control repository. Most com-mits we observed had commit comments. Commit com-ments are often studied by researchers, as they are the mostreadily accessible source of project interactions, and devel-opers are often required to create them by the repositorymechanism (e.g., Git). Additionally, relying only on commitcomments makes our approach more generalizable, as we donot assume the presence of other artifact corpora. An exam-ple of a typical commit message, from MySQL, is: “historyannotate diffs bug fixed (if mysql real connect() failed therewere two pointers to malloc’ed strings, with memory corrup-tion on free(), of course)”. We extracted these messages andindexed them by creation time. We summarized each mes-sage as a word distribution minus stop-words such as “the”and “at”.

For the commit message data-sets of each project, we cre-ated an XML file that separated commits into 30 day peri-ods. We chose a period size of 30 days as it is smaller thanthe time between minor releases but large enough for thereto be sufficient commits to analyze [11]. For each 30 dayperiod of each project, we input the messages of that periodinto Latent Dirichlet Allocation (LDA), a topic analysis al-gorithm [3], and recorded the topics the algorithm extracted.

A topic analysis tool such as LDA will try to find Nindependent word distributions within the word distribu-tions of all input messages. Linear combinations of these Nword distributions are meant to represent and recreate theword distributions of any of the original messages. TheseN word distributions effectively form topics: cross-cuttingcollections of words relevant to one or more of our commitmessages. LDA extracts topics in an unsupervised manner;the algorithm relies solely on the source data and word dis-tributions of messages, with no human intervention.

In topic analysis a single document, such as a commitmessage, can be related to multiple topics. Representingdocuments as a mixture of topics maps well to source coderepository commits, which often have more than one pur-pose [11]. For this paper, a topic represents a word distribu-tion generated from a group of commit log comments whichare related by their content.

We applied Blei’s LDA implementation [3] against theword distributions of these commits, and generated lists oftopics per period. We set the number of topics to generateto 20, because past experimentation showed that fewer top-ics might aggregate multiple unique topics while any moretopics dilutes the results and creates indistinct topics [11].

3.1.1 The high-level labelsTo facilitate cross-project comparison, we used a taxon-

omy of NFRs. This taxonomy is based on the ISO qual-ity model, ISO9126 [12]. ISO9126 describes six high-levelNFRs: maintainability, functionality, portability, efficiency,usability, and reliability 2. We claim that these NFRs aremaintenance concerns (to varying degrees) in all softwareprojects, and are therefore well-suited for comparisons be-tween projects.

2While there may be lingering debate in some circles aboutthese terms, an ISO standard seems like a reasonable start-ing point for our work.

3.1.2 Creating a validation corpusTo evaluate both semi-unsupervised and supervised clas-

sification, we created a validation set of manually labelledtopics. For MySQL 3.23 and MaxDB 7.500, the annota-tors (the first two authors) annotated each extracted topicin each period with the six NFR labels listed above. Anno-tators did not annotate each other’s annotations, but somebrief inspection of annotations was used to confirm that theannotators were acting similarly. We looked at each pe-riod’s topics, and assessed what the data — consisting ofthe frequency-weighted word lists and messages — suggestedwas the label for that topic. We were able to pinpoint theappropriate label using auxiliary information as well, suchas the actual revisions and files that were related to the topicbeing annotated. For example, for the MaxDB topic con-sisting of a message “exit() only used in non NPTL LINUXVersions”, we tagged that topic portability. Given the top-level annotations of none, portability, efficiency, reliability,functionality, usability, and maintainability, the annotatorsannotated each topic with the relevant label. Sometimesthey used finer-grained annotations that would be aggre-gated up to one of these higher-level labels.We validate classification performance using the Receiver

Operating Characteristic area-under-curve value [7], abbre-viated ROC, and the F-measure, which is the harmonicmean of precision and recall, i.e., 2 ∗ (P ∗R)/(P +R).ROC values provide a score reflecting how well a partic-

ular learner performed for the given data. ROC maps tothe more familiar concepts of precision/sensitivity and re-call/specificity: it plots the true positive rate (sensitivity)versus the false positive rate (1 - specificity). A perfectlearner has a ROC value of 1.0, reflecting perfect recall andprecision. A ROC result of 0.5 would be equivalent to arandom learner (that is, issuing as many false positives astrue positives). The ROC of a classifier is equivalent to theprobability that the classifier will rank a randomly chosenpositive instance higher than a randomly chosen negativeinstance. We consider our labelling classifiers acceptable ifthey outperform a random classifier (0.5).

3.2 Semi-unsupervised labellingIn this section we describe how to label topics based on

dictionaries mined from sources external to the projects.

3.2.1 Generating word listsIn order to automatically label each topic with one of the

six high-level NFRs, we associate each NFR with a list ofkeywords (word-lists, in our parlance). These word-lists weredetermined a priori and were not extracted from the projectsthemselves. We intersected the words of the topics and thewords of our word-lists. We “labelled” a topic if any of itswords matched any of the word-list’s words. A topic couldmatch more than one NFR. We used several different sets ofword-lists for comparison, which we refer to as exp1, exp2,and exp3 in the text which follows.Our first word-list set, exp1, was generated using the on-

tology described in Kayed et al. [13] . That paper constructsan ontology for software quality measurement using eightysource documents, including research papers and interna-tional standards. The labels we used:

integrity, security, interoperability, testability, maintain-ability, traceability, accuracy, modifiability, understand-ability, availability, modularity, usability, correctness, per-formance, verifiability, efficiency, portability, flexibility,reliability.

Our second word-list, exp2, uses the ISO9126 taxonomydescribed above (Section 3.1) to seed the word-lists. Theterms from ISO9126 may not capture all words occurringin the topics that are nonetheless associated with one ofthe NFRs. For example, the term “redundancy” is one weconsidered to be relevant to discussion of reliability, but isnot in the standard. We therefore took the NFRs from theISO9126 and added to them.

To construct these expanded word-lists, we used Word-Net [8]. We then added Boehm’s software quality model [4],and classified his eleven ‘ilities’ into their respective ISO9126NFRs. We did the same for the quality model produced byMcCall et al. [16]. We then did a simple random analysisof mailing list messages from an open source project, KDE.If we judged a given message to contain terms that wererelated to one of the NFRs in ISO9126, we added it to ourword-list. This allowed us to expand our word-lists withmore software-specific terms.

For the third set of word-lists, exp3, we extended the word-lists from exp2 using unfiltered WordNet similarity matches.Similarity in WordNet means siblings in a hypernym tree.We do not include these words here for space considerations(but see the Appendix for our data repository). It is notclear the words associated with our labels in exp3 are specificenough. For example, the label maintainability is associatedwith words ease and ownership. In general, as we proceedfrom word-list in exp1 to that in exp3, our lists become moregeneric.

3.2.2 Automatic Labelled Topic ExtractionUsing our three word-lists (exp1, exp2, exp3), we labelled

our topics with an NFR where there was a match betweena word in the list and the same word somewhere in the dis-tribution of words that constitute the topic. A named topicis a topic with a matching NFR label. Unnamed topics oc-cur where there is no such match, which may indicate eithera lack of precision, or simply that this topic is not asso-ciated with non-functional requirements. All experimentswere run on MaxDB 7.500 and MySQL 3.23 data. LDA ex-tracted 20 topics per period for each project. This labellingis semi-unsupervised because the corpus is not derived fromthe project being analyzed, and we did not label the project’stopics ourselves for a training set. The motivation behindthis technique is that because most software often addressessimilar issues, thus we can use the domain knowledge ofsoftware to label relevant topics.

Table 2 shows how many topics were labelled for MaxDBand MySQL.

For exp1 the labels with the most topics were correct-ness (182 topics) and testability (121). We did not seemany results for usability or accuracy, which were associ-ated with fewer than ten topics. We also looked for corre-lations between our labels: excluding double matches (self-correlation), our highest co-occurring terms were verifiabilitywith traceability, and testability with correctness (76 and 62matches, respectively).

Label Related terms

Maintainability testability changeability analyzability stability maintain maintainable modu-larity modifiability understandability interdependent dependency encapsula-tion decentralized modular

Functionality security compliance accuracy interoperability suitability functional practical-ity functionality compliant exploit certificate secured “buffer overflow” policymalicious trustworthy vulnerable vulnerability accurate secure vulnerabilitycorrectness accuracy

Portability conformance adaptability replaceability installability portable movablenessmovability portability specification migration standardized l10n localizationi18n internationalization documentation interoperability transferability

Efficiency “resource behaviour”“time behaviour” efficient efficiency performance profiledoptimize sluggish factor penalty slower faster slow fast optimization

Usability operability understandability learnability useable usable serviceable usefulnessutility useableness usableness serviceableness serviceability usability gui acces-sibility menu configure convention standard feature focus ui mouse icons uglydialog guidelines click default human convention friendly user screen interfaceflexibility

Reliability “fault tolerance” recoverability maturity reliable dependable responsiblenessresponsibility reliableness reliability dependableness dependability resilienceintegrity stability stable crash bug fails redundancy error failure

Table 1: NFRs and associated word-lists – exp2

Project Measure exp1 exp2 exp3

MaxDB 7.500 Named Topics 281 125 328Unnamed Topics 219 375 172Total Topics 500 500 500

MySQL 3.23 Named Topics 524 273 773Unnamed Topics 476 727 227Total Topics 1000 1000 1000

Table 2: Automatic topic labelling for MaxDB andMySQL

For exp2, there are more unnamed topics than exp1. Onlyreliability produces a lot of matches, mostly with the word“error”. Co-occurrence results were poor. This suggests ourword lists were overly restrictive.For exp3, we generally labelled more topics. As we men-

tioned, the word-lists are broad, so there are likely to befalse-positives (discussed below). The most frequent label,265 topics, was usability, and the least frequent label, 44topics, was maintainability. Common co-occurrences werereliability with usability, efficiency with reliability, and effi-ciency with usability (200, 190, and 150 topics in common,respectively).

3.2.3 Analysis of the semi-unsupervised labellingFor each quality we assessed whether semi-unsupervised

labels matched the manual annotations. As described inSection 3.1 we used both ROC and F-1 measures to eval-uate the performance of the classification. Figure 2 showsour ROC results for MaxDB and MySQL. We describe F-1results in the text below.Because our ground truth annotations were relevant only

to ISO9126, we estimate that exp1 had poor performancevia the overlap between ISO9126 and the Kayed ontology.For exp1 the F-measures for MaxDB were from 0 to 0.18

with an average of 0.03, and for MySQL were from 0 to 0.16with an average of 0.05.

For exp2, the average F-measure (macro-F1) for MaxDBwas 0.24 with a range 0.091 to 0.37, and 0.16 for MySQLwith a range of 0 to 0.41. MaxDB had an average precisionand recall of 0.25 and 0.22 while MySQL had 0.41 and 0.10respectively.

For exp3, the average F-measure (macro-F1) for MaxDBwas 0.26 with a range 0.11 to 0.47, and 0.36 for MySQL witha range of 0.10 to 0.65. MaxDB had an average precisionand recall of 0.16 and 0.67 while MySQL had 0.3 and 0.48respectively.

Thus we find that reliability and usability worked well forMaxDB in exp2 and better in exp3. exp1 performed poorly.MySQL had reasonable results within exp2 for reliability andefficiency. MySQL’s results for efficiency did not improve inexp3 but other qualities such as functionality did improve.Many ROC scores were 0.6 or less, but our classifier stillperformed substantially better than random.

3.3 Supervised labellingSupervised labelling requires expert analysis of the correct

class/label to assign a label to a topic. In our approach, weuse the top-level NFRs in the ISO9126 standard [12] for ourclasses, but other taxonomies are also applicable.

We used a suite of supervised classifiers, WEKA [10], thatincludes machine learning tools such as support vector ma-chines and Bayes-nets. We also used the multi-labellingadd-on for WEKA, Mulan [20]. Traditional classifiers labeltopics with a single class, whereas Mulan allows for a mix-ture of classes per topic, which is what we observed whilemanually labelling topics. The features we used are wordcounts/occurrence per topic, if a word occurs frequentlyenough in a topic we consider it a feature of the topic.

To assess the performance of the supervised learners, wedid a 10-fold cross-validation [14], a common technique forevaluating machine learners. The original data is partitionedrandomly into ten sub-samples. Each sample is used to test

portability efficiency reliability functionality maintainability usability total

0.8

0.4

0.45

0.5

0.55

0.6

0.65

0.7

0.75

Figure 2: Performance, ROC values (range: 0–1), of semi-unsupervised topic labelling for each NFR and perword-list. The dashed line indicates the performance of a random classifier. This graph shows how well thesemi-unsupervised topic labelling matched our manual annotations.

Portability Eciency Reliability Functionality Maintain. Usability

1

0.4

0.45

0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

Figure 3: ROC value for the best learner per la-bel for MaxDB and MySQL. Values range from 0–1.Dashed line indicates the performance of a randomclassifier.

against a training set composed of the nine other samples.We have reported these results below.

3.3.1 Analysis of the supervised labellingBecause our data-set was of word counts we expected

Bayesian techniques, often used in spam filtering, to performwell. We tried other learners that WEKA [10] provides: rulelearners, decision tree learners, vector space learners, andsupport vector machines. Figure 3 shows the performanceof the best performing learner per label: the learner that hadthe highest ROC value for that label. The best learner isimportant because one uses a single learner per label. When

applying our technique, for each NFR one should select thebest learner possible.

Figure 3 shows that MaxDB and MySQL have quite dif-ferent results, as the ROC values for reliability and function-ality seem swapped between projects.

For both projects Bayesian techniques did the best out ofa wide variety of machine learners tested. Our best learn-ers, Discriminative Multinomial Naive Bayes, Naive Bayesand Multinomial Naive Bayes are all based on Bayes’s the-orem and all assume, naively, that the features supplied areindependent. One beneficial aspect of this result is that itsuggests we can have very fast training and classifying sinceNaive Bayes can be calculated in O(N) for N features.

The range of F-measures for MySQL was 0.21 to 0.77 withan average of 0.48, while MaxDB had a range of 0.17 to 0.61with an average of 0.39.

The less-frequently occurring a label, the harder it is toget accurate results, due to the high noise level. Never-theless, these results are better than our previous word-listresults of exp2 and exp3, because the ROC values are suffi-ciently higher in most cases (other than MaxDB reliabilityand MySQL efficiency). The limitation of the approach wetook here is that we assume labels are independent; how-ever, labels could be correlated with each other. The nextsection (3.4) addresses the issue of a lack of independenceand correlation between labels using multi-label learners.

3.4 Applying multiple labels to topicsAs noted in Section 3.1, each topic in our data-set can be

composed of zero, one, or more NFRs. For example, a com-mit message might address reliability in the context of effi-ciency, or make a maintainability improvement in the sourcecode that related to usability. However, traditional machinelearning techniques, such as Naive Bayes, can only map top-ics to a single class. The Mulan [20] library encapsulatesseveral different multi-label machine learners which can label

CLR HOMER

1

0.4

0.5

0.6

0.7

0.8

0.9

BR

(a) MySQL

CLR HOMER

1

0.4

0.5

0.6

0.7

0.8

0.9

BR

(b) MaxDB

Figure 4: MySQL and MaxDB macro (grey) and micro (black) ROC results per multi-label learner. Possiblevalues range from 0–1. Dashed line indicates the performance of a random classifier.

elements with multiple labels. Mulan also includes methodsfor determining the performance of such techniques.Two perspectives to evaluate multi-label learners are with

micro or macro measurements (used in Figure 4). Macromeasurements are aggregated at a class or label level (perclass) while micro measurements are at the element level(per element). A macro-ROC measurement is the averageROC over the ROC values for all labels, where a micro-ROCis the average ROC over all examples that were classified.For MaxDB, the macro-ROC values are undefined becauseof poor performance of one of the labels.Figure 4 presents the results of Mulan’s best multi-label

learners for our data. Calibrated Label Ranking (CLR)is a learner that builds two layers. The first layer deter-mines if an entity should be labelled, while the second layerdetermines what labels should be assigned. The Hierar-chy Of Multi-label classifiERs (HOMER) and Binary Rel-evance (BR) act as a hierarchy of learners: BR is flat, whileHOMER tries to build a deeper hierarchy for a more ac-curate learner [20]. These classifiers performed better thanother multi-label classifiers as they have the best micro andmacro ROC scores. The multi-label and single-label learnershad similar performance: for MySQL, BR and Naive Bayeshad similar macro-ROC scores of 0.74.

4. UNDERSTANDING SOFTWARE MAIN-TENANCE ACTIVITIES

As we mentioned in the introduction, a key issue in soft-ware maintenance is understanding why a system has evolvedthe way it has [1]. In this section we demonstrate the valueof labelled topic extraction in addressing this issue. Labelledtopics address why because they show reasons why changesoccurred rather than how. The how of a change is the changeitself, the purpose or why is what we are after. We investi-gate the history of the two large-scale database systems thatwe studied. We use our technique to show the topic of de-velopment efforts over time in each project. We motivatedour investigation with two research questions:

1. Do NFR frequencies change over time? If a particularNFR was of more interest at one point in the life-cycle

than another, this suggests that development activityshifted focus. For example, if a developer expectedto see a recent focus on reliability, but instead usabil-ity dominated, they might re-prioritize upcoming workitems.

2. Do projects differ in their relative interest in NFRs? Aproject manager, especially a systems-manager, wouldbe interested in knowing whether a particular NFR,such as reliability, was more important for one projectthan another. This could be to confirm the initial de-sign goals, or to track the progress on that quarter’sobjectives. The difference in NFR proportion is inter-esting because it implies a difference in focus betweentwo projects.

Figures 5a and 5b show the temporal patterns of NFRfrequencies. There are two measures represented. One, therelative frequency, shown in the grey histogram boxes, rep-resents the number of topics with that NFR in that period,relative to the maximum number of topics assigned to thatNFR. For example, in Figure 5a we see a spike in portabilityand functionality frequency in September 2002. The sec-ond, absolute frequency, is shown using cell intensity, andcompares the number of topics labelled with that NFR perperiod relative to the maximum number of labelled topicsoverall. For instance, Figure 5a shows that the NFRs func-tionality, portability and maintainability contain more la-belled topics, since these NFRs have been more intenselyshaded; one interesting stream is efficiency which shows pe-riodic activity, does this suggest that once maybe efficiencyrelated changes have longer lasting effects? The topmostrow in each diagram lists historical events for that project(such as a release).

We analyzed each project’s developer mailing list for ex-ternal validation. We use labelled topic extraction to pick outthe underlying NFR activity behind these events. For exam-ple, both projects show a high number of NFRs recognized atthe first period of analysis. This is due to our window choice:we deliberately targeted our analysis to when both MySQL3.23 and MaxDB 7.500 were first announced. For MaxDB,version 7.5.00 was released in December of 2003. We know

(a) MySQL 3.23

(b) MaxDB 7.500

Figure 5: NFR label per period. Each cell represents a 30-day period. Grid cell intensity (saturation) ismapped to label frequency relative to the largest label count of all NFRs. Grey histogram bars indicatelabel frequency relative to that particular NFR’s largest label count. Dashed vertical lines relate a projectmilestone (*Key events* ) to our topic windows.

that release 7.5.00.23 saw the development of PHP inter-faces, possibly accounting for the simultaneous increase inthe portability NFR at the same time. The gap in MaxDB(Figure 5b) is due to a shift in development focus (fromFebruary 2005 to June 2005) to MaxDB 7.6, which is re-leased in June 2005.The release of MySQL we study (Figure 5a) was the first

to be licenced under the GPL. Version 3.23.31 (January,2001) was the production release (non-beta), and we see aflurry of topics labelled with functionality and maintain-ability. After this point, this version enters the maintenancephase of its life-cycle. In May 2001, there is an increase inthe number of topics labelled with portability. This might berelated to release 3.23.38, which focused on Windows com-patibility. Similarly, in August, 2002, both functionality andportability are frequent, and mailing list data suggests thisis related to the release of version 3.23.52, a general bug fixwith a focus on security (a component of the functionality

NFR in the ISO9126 model). After this point, efforts shiftto the newer releases (4.0, 4.1, 5.0). We now address ourresearch questions:

4.1 Do NFR frequencies change over time?In both projects the frequencies generally decreased with

age. However, there are variations within our NFR labels.In MySQL, usability and efficiency do not appear very of-ten in topics. A proportionately smaller number of commitsaddressed these NFRs. Certain peaks in topic numbers coin-cide with a particular emphasis from the development teamon issues such as new releases or bug fixes. This suggeststhat maintenance activity is not necessarily strictly decreas-ing with time, but rather episodic and responsive to outsidestimuli. In MaxDB, we can observe that Maintainabilitytopics became more prevalent as MaxDB matures. Thisis likely due to our analysis time-frame for MaxDB beingshorter than the time-frame for the MySQL product.

4.2 Do projects differ in their relative topic in-terest?

Yes. MySQL 3.23 had proportionally more topics labelledfunctionality, while MaxDB had proportionally more effi-ciency related topics. MaxDB was a very mature release“donated” to the open-source community, whereas MySQLwas in its relative infancy, and security problems were morecommon (security is a component of functionality in theISO9126 model). In both cases portability was a constantmaintenance concern and was prevalent throughout the life-time of the projects. It may surprise developers how oftenportability arises as a concern.

5. DISCUSSION

5.1 Annotation observationsWe found many topics that were not actually non-functional

requirements (NFRs) but were often related to them. Forinstance, concurrency was mentioned often in the commitlogs and was related to correctness and reliability, possiblybecause concurrent code is prone to bugs such as race condi-tions. Configuration management and source control relatedchanges appeared often; these kinds of changes are slightlyrelated to maintainability. A non-functional change thatwas not quality-related was licensing and copyright; manychanges were simply to do with updating copyrights or en-suring copyright or license headers were applied to files. Inthese cases we assigned the None label to the topic.We noticed that occasionally the names of modules would

conflict with words related to other non-functional require-ments. For instance, optimizers are very common modules indatabase systems: both MySQL and MaxDB have optimizermodules. In MySQL the optimizer is mentioned but oftenthe change addresses correctness or another quality. Despitethis difference, the name of the module could fool our learn-ers into believing the change was always about efficiency.In these cases the advantages of tailoring topic names tospecific project terminologies are more clear. Project spe-cific word-lists would avoid automated mistakes due to thenames of entities and modules of a software project.

5.2 Summary of techniquesWhile an unsupervised technique such as LDA is appeal-

ing in its lack of human intervention, and thus lower effort,supervised learners have the advantage of domain knowl-edge, which typically means improved results. Creating an-notated topics (i.e., manual labels) for training is painstak-ing, but with a suitably representative set of topics, the effortis acceptable. To annotate all topics took us approximately20 hours per project, but we estimate only 10% of the topicsneed annotation to produce useful results.Very rarely did exp2 and exp3 (semi-unsupervised word

matching) ever perform as well as the supervised machinelearners. For MaxDB, reliability was slightly better detectedusing the static word list of exp2. In general, the machinelearners and exp3 did better than exp2 for both MaxDBand MySQL. For both MySQL and MaxDB usability wasbetter served by exp2. Usability was a very infrequent label,however, which made it difficult to detect in any case.The semi-unsupervised labelling had difficulty distinguish-

ing between common labels and infrequent labels. The learn-ers would occasionally mislabel a topic deserving of an infre-

quent label with a more common label. The word-lists forcorrectness tended to be too lengthy, non-specific and broad,especially if WordNet words were used, since the NFRs aretypically loosely defined concepts in common parlance.

We found that the multi-label learners of BR, CLR andHOMER only did as well or worse for Macro-ROC as thesingle-label Naive Bayes and other naive Bayes-derived learn-ers. This suggests that by combining together multiple NaiveBayes learners we could probably label sets of topics effec-tively, but it would require a separate Naive Bayes learnerper label.

With ROC values ranging from 0.6 to 0.8 we can see thereis promise in these methods. exp2 and exp3 both indicatethat static information can be used to help label topics with-out any training whatsoever. MySQL and MaxDB’s ma-chine learners made some decisions based off a few sharedwords: bug, code, compiler, database, HP UX, delete, memory,missing, problems, removed, add, added, changed, problem,and test. Adding these words to the word-lists of exp2 andexp3 could improve performance while ensuring they wereonly domain specific.

If the techniques used in exp2 and exp3 were combinedwith the supervised techniques, we could reduce the trainingeffort by boosting training sets with topics classified with thesemi-unsupervised techniques. Both Naive Bayesian learn-ers and the word-list approaches were computationally ef-ficient. These results are promising because they indicatethat these techniques are accurate enough to be useful whilestill maintaining acceptable run-time performance.

Since this work focuses on labelling natural language com-mit log comments we feel it can be adapted to other naturallanguage artifacts such as mailing-list discussions and bugreports, even though that was not evaluated in this paper.Bug reports might not exhibit the same behaviour as com-mits in terms of dominant topics.

5.3 Threats to validityConstruct validity – we used only commit messages rather

than mail or bug tracker messages. To extend further wewould need matching repositories for each project. Possiblythey would have influenced our results, but there would bea degree of correlation between the corpora. Our taxonomyfor software NFRs is subject to dispute, but seems generallyaccepted. Finally, there are exogenous sources, such as in-person discussions, which we did not access, an omission asnoted in Aranda et al. [1]

Internal validity – We improved internal validity by try-ing to correlate and explain the behaviours observed in theanalysis with the historical records of the projects. We didnot attempt to match our results to any particular model.We were unable to assess inter-rater reliability.

External validity – Our data originated from OSS data-base projects and thus might not be applicable to commer-cially developed software or other domains. Furthermore,our analysis techniques rely on a project’s use of meaning-ful commit messages, although we feel this is the most fre-quently occurring form of developer comments.

Reliability – each annotator, the first two authors, followedthe same protocol and used the same annotations. However,only two annotators were used; their annotations could bebiased as we did not analyze for inter-rater reliability be-cause they did not rate the same documents.

5.4 Future workThere are several avenues of further investigation. More

external validation would be useful. Although we validatedour comparisons using a mailing list for each project, inter-views with developers would provide more detail. We alsothink multi-label learning techniques, although in their in-fancy, are crucial in understanding cross-cutting concernssuch as NFRs. We want to leverage different kinds of ar-tifacts to discover threads of NFR-related discussions thatoccur between multiple kinds of artifacts. Finally, we wouldlike to extend this analysis to other domains, to see whatpatterns might occur in, for example, a consumer-facing soft-ware product.

6. CONCLUSIONSThis paper presented a cross-project data mining tech-

nique, labelled topic extraction. Previous topic analysis re-search produced project-specific topics that needed to bemanually labelled. To improve on this, we leveraged soft-ware engineering standards, specifically the ISO9126 qualitytaxonomy, to produce a method of partially-automated (su-pervised) and fully-automated (semi-unsupervised) topic la-belling. Since the word-list technique is not project-specific,we used it to compare two distinct projects, where we showedour technique produced interesting insight into maintenanceactivity.We validated our topic labelling techniques using multi-

ple experiments. We first conducted semi-unsupervised la-belling using word-lists. Our next approach was supervised,using single-label and multi-label learners. Both kinds oflearners performed well with average ROC values between0.6 and 0.8. These results, along with the efficient perfor-mance of our learners, demonstrate that labelled topic ex-traction is a promising approach for understanding the oc-currence of non-functional requirements in software projects.

APPENDIX

Our data and scripts are available at:http://softwareprocess.es/nomen/

A. REFERENCES[1] J. Aranda and G. Venolia. The secret life of bugs:

Going past the errors and omissions in softwarerepositories. In International Conference on SoftwareEngineering, pages 298–308. IEEE, Sep 2009.

[2] P. F. Baldi, C. V. Lopes, E. J. Linstead, and S. K.Bajracharya. A theory of aspects as latent topics. InConference on Object Oriented Programming SystemsLanguages and Applications, pages 543–562, Nashville,2008.

[3] D. M. Blei, A. Y. Ng, and M. I. Jordan. LatentDirichlet Allocation. Journal of Machine LearningResearch, 3(4-5):993–1022, May 2003.

[4] B. Boehm, J. R. Brown, and M. Lipow. QuantitativeEvaluation of Software Quality. In InternationalConference on Software Engineering, pages 592–605,1976.

[5] J. Cleland-Huang, R. Settimi, X. Zou, and P. Solc.The Detection and Classification of Non-Functional

Requirements with Application to Early Aspects. InInternational Requirements Engineering Conference,pages 39–48, Minneapolis, Minnesota, 2006.

[6] N. A. Ernst and J. Mylopoulos. On the perception ofsoftware quality requirements during the projectlifecycle. In International Working Conference onRequirements Engineering: Foundation for SoftwareQuality, Essen, Germany, June 2010.

[7] T. Fawcett. An introduction to ROC analysis. PatternRecognition Letters, 27(8):861 – 874, 2006.

[8] C. Fellbaum, editor. WordNet: An Electronic LexicalDatabase. MIT Press, 1998.

[9] S. Few. Information Dashboard Design: The EffectiveVisual Communication of Data. O’Reilly Media, 1edition, Jan. 2006.

[10] M. Hall, E. Frank, G. Holmes, B. Pfahringer,P. Reutemann, and I. H. Witten. The WEKA DataMining Software: An Update. SIGKDD Explorations,11(1):10–18, 2009.

[11] A. Hindle, M. W. Godfrey, and R. C. Holt. What’shot and what’s not: Windowed developer topicanalysis. In International Conference on SoftwareMaintenance, pages 339–348, Edmonton, Alberta,Canada, September 2009.

[12] Software engineering – Product quality – Part 1:Quality model. Technical report, InternationalStandards Organization - JTC 1/SC 7, 2001.

[13] A. Kayed, N. Hirzalla, A. Samhan, and M. Alfayoumi.Towards an ontology for software product qualityattributes. In International Conference on Internetand Web Applications and Services, pages 200–204,May 2009.

[14] R. Kohavi. A study of cross-validation and bootstrapfor accuracy estimation and model selection. InInternational Joint Conference On ArtificialIntelligence, pages 1137–1143, Toronto, 1995.

[15] A. Marcus, A. Sergeyev, V. Rajlich, and J. Maletic.An information retrieval approach to concept locationin source code. In 11th Working Conference onReverse Engineering, pages 214–223, November 2004.

[16] J. McCall. Factors in Software Quality: PreliminaryHandbook on Software Quality for an AcquisitonManager, volume 1-3. General Electric, November1977.

[17] Q. Mei, X. Shen, and C. Zhai. Automatic labeling ofmultinomial topic models. In International Conferenceon Knowledge Discovery and Data Mining, pages490–499, San Jose, California, 2007.

[18] A. Mockus and L. Votta. Identifying reasons forsoftware changes using historic databases. InInternational Conference on Software Maintenance,pages 120–130, San Jose, CA, 2000.

[19] C. Treude and M.-A. Storey. ConcernLines: Atimeline view of co-occurring concerns. InInternational Conference on Software Engineering,pages 575–578, Vancouver, May 2009.

[20] G. Tsoumakas, I. Katakis, and I. Vlahavas. Miningmulti-label data. In O. Maimon and L. Rokach,editors, Data Mining and Knowledge DiscoveryHandbook. Spring, 2nd edition, 2010.