Improving Software Developers’ Fluency byRecommending Development Environment Commands

Emerson Murphy-HillDepartment of Computer Science

North Carolina State UniversityRaleigh, North Carolinaemerson@csc.ncsu.edu

Rahul Jiresal and Gail C. MurphyDepartment of Computer Science

University of British ColumbiaVancouver, Canada

jiresal,murphy@cs.ubc.ca

ABSTRACTSoftware developers interact with the development environ-ments they use by issuing commands that execute variousprogramming tools, from source code formatters to buildtools. However, developers often only use a small subsetof the commands offered by modern development environ-ments, reducing their overall development fluency. In thispaper, we use several existing command recommender al-gorithms to suggest new commands to developers based ontheir existing command usage history, and also introduceseveral new algorithms. By running these algorithms ondata submitted by several thousand Eclipse users, we de-scribe two studies that explore the feasibility of automati-cally recommending commands to software developers. Theresults suggest that, while recommendation is more diffi-cult in development environments than in other domains,it is still feasible to automatically recommend commandsto developers based on their usage history, and that usingpatterns of past discovery is a useful way to do so.

Categories and Subject DescriptorsD.2.6 [Software Engineering]: Coding Tools and Tech-niques

Keywordsdiscovery, software developers, commands, IDEs

1. INTRODUCTIONSoftware tools can help developers meet users’ increasing

demands for better software. For instance, research suggeststhat documentation tools can help developers use unfamiliarapplication programming interfaces to create code severaltimes faster [19], code smell tools can help developers makemore informed judgments about design [15], and debuggingtools can help developers fix bugs more successfully [7].

To use these tools, developers issue commands to theirdevelopment environment, whether that environment is an

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.SIGSOFT’12/FSE-20, November 11–16, 2012, Cary, North Carolina, USA.Copyright 2012 ACM 978-1-4503-1614-9/12/11 ...$10.00.

editor like emacs [29] or an integrated development environ-ment like Eclipse [28]. Commands may range from simpletext editing functions, such as copying text, to sophisticatedprogramming actions, such as generating code or synchro-nizing with a repository. In 2009, logs collected from over 1million users interacting with the Eclipse development envi-ronment indicate the use of over 1100 distinct commands.

Although the community of all software developers usea wide range of commands, any given developer tends touse only a small number. After reaching their own personalfixpoint, developers tend to face difficulties discovering andbecoming aware of other helpful commands. Consider theopen resource command in Eclipse that allows a developer tosimply type the first few characters of a file, press return andthe desired file is opened almost instantly. Several bloggershave praised open resource, including “10 Eclipse Naviga-tion Shortcuts Every Java Programmer Should Know” [23].Other bloggers have noted it is the “biggest time-saver I’vestumbled upon” [24], and a “command I use religiously” [25].However, despite the acclaim given to this command, basedon Eclipse Usage Data Collector [26] data for May 2009, ofthe more than 120,000 people who used Eclipse, more than88% do not use open resource.

This command discovery problem can be addressed inmany ways. One way is through searchable documentation,yet the developer must know of a command’s existence inorder to search for it. Another way is to improve the visi-bility of a command, such as through a special toolbar, yetthis comes at the expense of the discoverability of other com-mands that are not on the toolbar. Another way is throughtip-of-the-day messages, which recommend a random com-mand at regular intervals, yet the suggested command mayhave no bearing on what a developer actually needs.

In this paper, we address the command discovery prob-lem by making custom command recommendations to a de-veloper by analyzing the developer’s past history of com-mand usage and comparing it to a larger community’s useof development commands. This approach builds on a sim-ilar approach taken by Matejka and colleagues [9, 12] torecommend commands useful in a computer-aided drawingapplication, AutoCAD. Since Eclipse includes about 1100distinct commands while AutoCAD contains about 750 [9],recommending commands in development environments ap-pears to be more challenging; indeed, our results in Sec-tion 5 suggest that existing algorithms do not perform aswell with Eclipse data as they do with AutoCAD data. Weextend Matejka and colleagues’ work by introducing fournovel algorithms for recommending relevant commands to

software users. We evaluated six algorithms in two evalu-ations: one automated evaluation and the second with livesoftware developers. In the automated evaluation, we foundthat our best novel algorithm made recommendations withabout 30% accuracy, which is about 20% higher than recom-mendations produced by the best previous algorithm. Al-though 30% accuracy may seem low, we believe adopting asmall number of commands at a continual rate is an appro-priate way to learn new commands and that delivery mech-anisms are available to make it possible for developers tolearn those recommendations that are relevant. We discussthese points in further detail in Section 8.

We first review previous work on feature awareness andcommand recommendations (Section 2). We then presenta proof-of-concept study that explores discovery trends inEclipse to help explain the motivation behind our novel al-gorithms (Section 3). We then describe the existing algo-rithms we applied and the novel algorithms we developed(Section 4) before presenting the results of an automated ex-ploration of the effectiveness of the algorithms (Section 5).We also performed a study with developers to gain humaninsight into the value of the recommendations (Section 6).We discuss various aspects of our approach (Section 7) andfuture work (Section 8) before summarizing the paper (Sec-tion 9).

2. RELATED WORKSeveral pieces of previous research have attempted to in-

vestigate the problem of command awareness and ways torecommend commands to users of complex software. We di-vide the latter work into three categories: human-to-human,inefficiency-based, and history-based recommendation.

2.1 Software Feature AwarenessThe challenge of making software learnable has been a

topic of considerable study among the human-computer in-teraction community. One aspect of learnability is aware-ness of what functionality is available in a user interface.When Grossman and colleagues conducted a study of learn-ability issues for users of AutoCAD, a computer-aided draft-ing application, they found that a “typical problem was thatusers were not aware of a specific tool or operation whichwas available for use” [6]. Although less well-documented,the awareness problem has been described in integrated soft-ware development environments as well, such as Campbelland Miller’s discussion of discoverability as a barrier to refac-toring tool use [2].

2.2 Human-to-Human RecommendationPerhaps one of the most natural, most overlooked, and

most effective ways to learn is from a peer. In the general ed-ucational literature, situated learning, where a person learnssomething new from another person during routine activi-ties, has been suggested as a mechanism by which knowledgemay spread “exceedingly rapidly and effectively” [8, p. 93].

Learning software from peers appears to be effective aswell. Suppose Hilaria needs to submit a bug report, butshe does not know the command to use to attach a patchto the bug report. Hilaria asks a colleague, who shows Hi-laria the menu item and hotkey for attaching a patch. Thisscenario is known as Over the Shoulder Learning, or OTSLfor short, where a software user learns from another user atthe same computer [20]. In the context of software develop-

ment, Cockburn and Williams as well as Muller and Tichyhave suggested that software developers learn effectively inthis way during pair-programming [3, 13]. For example, Hi-laria might be programming with a peer, notice that thepeer is refactoring without using the refactoring commands,and may then recommend to the peer that they try a refac-toring command. Our research has suggested that althoughdiscovering tools from peers in such a way is effective, it isrelatively rare [16]. Moreover, a software developer cannotdiscover a new command using OTSL nor pair programmingwhen she is working without other developers or she is themost fluent command-user in her peer group. The approachthat we propose in this paper, using a recommender system,does not depend on a developer’s peer group.

2.3 Inefficiency-based RecommendationOne approach that does not rely on peers is a recom-

mender system that makes command recommendations basedon inefficient behavior. The Spyglass system, for example,works in a manner quite similar to the way that Hilariarecommended a refactoring command during pair program-ming [21, 22]. For example, when Hilaria is using Eclipse,Spyglass might notice that she often uses the find referencescommand repeatedly on methods. Spyglass would then rec-ommend that Hilaria instead use the show call hierarchy com-mand, reasoning that the task that Hilaria is performing iswalking up a call tree, a task that can be accomplished infewer steps using the recommended command.

A problem with this approach is that the recommendersystem must be pre-programmed to recognize inefficient be-havior as a precondition for each and every command thatit want to recommends. For example, how such a systemsidentifies inefficient behavior prior to recommending the findreferences command is entirely different from how it recog-nizes inefficient behavior prior to recommending the opentype hierarchy command. On the other hand, a history-based recommender system like the one in this paper, canrecommend one command as easily as the next, without anyspecial command-specific programming.

2.4 History-Based RecommendationOther researchers have used history-based recommenders,

the approach taken in this paper, for software systems inother domains. We briefly outline previous research here,and more extensively discuss the advantages and disadvan-tages of the algorithms in Section 4.

The Toolbox system created by Maltzahn and Vollmarcreates a database of commands invoked by Unix users inorder to identify commands that users have discovered re-cently [11]. Using the information in the database, Toolboxsends an email interview to a user who discovers a new com-mand. In the email, Toolbox solicits information about howthe new command works. The response is then distributedvia an online newsletter, so that other members of the com-munity can discover the command as well. This system relieson users completing the interview, whereas the algorithmsdescribed in this paper do not require user interaction for arecommendation to be made.

Linton and colleagues created OWL, a system that makescommand recommendations to Microsoft Word users [10].OWL observes what commands the community is using, andmakes recommendations to individuals to change their com-mand usage behavior. For example, if OWL observes that

commit

update

compareWithRemote

showHistory

compareWithTag

replace compareWithRevision

merge

sync

Figure 1: A discovery graph for Eclipse CVS com-mands.

the community of Word users use the find command morethan any other command, yet you do not use it at all, OWLwould recommend that you use find as well.

More recently, Matejka and colleagues proposed using ahistory-based recommender called CommunityCommands torecommend commands to users of AutoCAD, a computer-aided drafting system [12]. Matejka and colleagues used twocollaborative-filtering algorithms to compare a user’s com-mand history with the history of many other users of Au-toCAD, and used the comparison to make a recommenda-tion. The researchers showed that CommunityCommands’best algorithm could predict the next command that anAutoCAD user would discover next about 27% more accu-rately than Linton’s algorithm. The researchers also showedthat their algorithms could make command recommenda-tions that AutoCAD users preferred over the recommenda-tions made by Linton.

In this paper, we build on this work in two ways. First,we extend their recommender systems into a new domain,software development environments. Second, we explicitlybuild upon each of their algorithms to create several newalgorithms, which we discuss in Section 4.

3. DISCOVERY TRENDS IN ECLIPSESeveral of the algorithms we investigate in this paper model

how users discover commands as a means to predict new,potentially relevant commands. Why might this be a fruit-ful approach? To answer this question, in this section wepresent a small proof-of-concept study of how Eclipse usersdiscover two sets of commands.

To perform this study, we use data from the Eclipse UsageData Collector (UDC). This data set contains informationsubmitted voluntarily by users of Eclipse, information in-cluding events that capture which Eclipse plugins are loaded,which user interface views and perspectives are shown, andwhich commands users invoke. Events in this data set aretagged by timestamps and anonymous user identifiers. We

examined data from 2009, which includes a total of about1.3 billion events from more than a million users.

We filtered this UDC data in two ways. First, we onlyincluded command events, such as invocations of the openresource command. Second, we included events only fromusers who reported data every month in 2009 because wewanted to improve our chances of exploring real, long-termdiscovery trends. This reduced the data set to about 22million command uses from about 4300 Eclipse users.

To examine discovery trends, we chose to focus our studyon two sets of commands in Eclipse and make two specifichypotheses:

• CVS Commands. CVS commands enable develop-ers to interact with their CVS source revision reposi-tories. Commands include update and commit, whichenable developers to get the latest version from andcommit to their repositories, respectively. We choseto study CVS commands because we view one com-mand in particular — synchronize — as an advancedcommand that users would be more likely to discoverafter discovering other CVS commands first. We hy-pothesized this discovery ordering because synchronizeis not strictly a necessary command to use CVS, butit may improve productivity.

• Java Editing Commands. Java editing commandsenable software developers to more quickly edit theircode. Examples of such commands include tools totoggle comments and perform refactorings. We choseto study editing commands because our previous ex-perience with refactoring tools [17] suggests that onecommand in particular — rename — invokes a very ba-sic refactoring tool, and thus we hypothesize that it islikely discovered before other refactoring commands.

We tested these hypotheses by building discovery graphsfor both of these sets of commands. Figures 1 and 2 showthese graphs, as produced by GraphViz [5]. In each graph,each node represents a command that users can discover. Wedefine a command discovery as occurring only when two con-ditions are met. First, a command discovery can only occurafter the user’s base command usage window, which repre-sents a period of time where we assume that all commandsused in this period are tools that the developers know, andany commands used after this period are commands thatthe developer has discovered. In this paper, we fix the basecommand usage window to the first 40 sessions for whichwe have data. We divide a user’s command history into ses-sions by grouping two commands into the same session ifthey were invoked within 60 minutes of one another. Themotivation for using sessions is to remove spurious discoverypreconditions – for instance, if two commands are discov-ered around the same time, we assume that learning aboutone command was not a precondition for learning about theother. Based on a manual exploration of the UDC data,we judged that the 60 minute tolerance and the 40 sessionwindows were reasonable values to group working periodstogether, although we did not perform full sensitivity anal-ysis. Second, a command discovery occurs the first time auser uses a new command only if she uses that command asecond time at some point later in the data set. We add thissecond condition to attempt to rule out false-positive dis-coveries where the user inadvertently used a command butthen never used it again.

create_getter_setter

rename_element

toggle_comment

move_element

open_type_hierarchy

extract_method

open_hierarchy

formatcorrection_assist_proposalssearch_references_in_project open_editor override_methods

extract_constant

search_references_in_workspace organize_imports

Figure 2: A discovery graph for Eclipse Java editing commands.

In Figures 1 and 2, an edge from node a to node b rep-resents users who discovered a command a first and thendiscovered b in a later session. The thickness of a line rep-resents the number of users who discovered a and then b.Thus, thicker lines means that more people discovered a thenb. The lightness of the line indicates the number of userswho first discovered b, then a. Thus, perfectly black linesindicate that everyone who discovered a and b discovereda first, whereas very light lines indicate that just as manypeople discovered a first as discovered b first. To reduce thevisual complexity of the graphs, we eliminated edges whichrepresent the discovery of only a few people; for Figure 1,we excluded edges with fewer than 7 people (per direction),and in Figure 2, fewer than 14.

For example, in Figure 2, notice a fairly dark line fromrename element to extract method. The line thickness in-dicates that 18 people first discovered the rename elementcommand, then discovered the extract method command.The line lightness indicates that 6 people first discovered ex-tract method, then discovered rename element. As anotherexample, Figure 1 shows a light, thin line from showHistoryto compareWithRemote. For this line, 8 people discoveredthe showHistory command first, and 6 people discovered thecompareWithRemote command first.

In the whole data set, a large number of people use CVS(n=41,734) and Java editing commands (n=351,657), but inthe filtered data set, most trends in the graphs are supportedby typically fewer than 20 people. This is partly becausemany people only reported data for a short period of time,and partly because some people used the same tools bothbefore and after the base command usage window.

By inspecting the two discovery graphs, we can evalu-ate our hypotheses with regard to the rename refactoringand CVS synchronize commands. Looking at Figure 1, syncdoes not actually seem to be discovered after other CVScommands, but is actually a predecessor of update and com-pareWithRemote. Looking at Figure 2, we can see that re-name element appears as a predecessor to the three otherrefactoring commands shown (move element, extract method,and extract constant).

Overall, the graphs reveal that there are discovery trendscommon between difference Eclipse users, although we didnot consistently predict those trends. Certain commandsdo tend to be discovered earlier than other commands, sug-gesting that usage data may be a valuable predictor of theorder of command recommendations, and that developerswith different levels of command maturity may benefit frombeing recommended different commands. For example, a de-veloper who is unfamiliar with refactoring may find it moreuseful to first be recommended the rename command beforethe extract method command.

4. DESCRIPTION OF THE ALGORITHMSIn this section, we describe two existing algorithms (as

indicated by the sub-section headings) and four new algo-rithms for recommending commands to users. Through-out this section, we will use a simple hypothetical exam-ple, shown in Table 1. In the table, each row represents adeveloper, Hilaria, Ignacio, and Chuy, each using the samedevelopment environment on different machines. Each userhappens to use only two commands per day on Monday,Tuesday, and Wednesday; these commands are called Copy,Rename, Cut, Commit, Paste, Format, and Update. For exam-ple, on Monday, Hilaria uses Copy once, then Rename once;on Tuesday she uses Copy once, then Paste once; and onWednesday she uses Copy once, and then Format once.

Using different techniques, each algorithm provides a sug-gestion for which command we should recommend to a useron Thursday. For example, based on Table 1, we can seethat Chuy does not use command Paste nor command For-mat, so we could reasonably recommend either command tohim. Which command is more useful?

Monday Tuesday WednesdayHilaria Copy Rename Copy Paste Copy FormatIgnacio Copy Cut Copy Cut Copy PasteChuy Cut Commit Cut Commit Cut Update

Table 1: Developers’ command use over three days.

4.1 Existing: Most PopularLinton and colleagues’ [10] algorithm recommends the com-

mands that are the most-used by the community. This al-gorithm is executed by counting the number of times eachcommand has been used in the entire user community. Thiscount represents the algorithm’s confidence level (δ) thatthis recommendation will be useful. To make a recommen-dation to a specific user, the algorithm recommends com-mands with the highest confidence values, excluding com-mands that the user already uses. Suppose that we wantto recommend a command to Hilaria (Table 1). This algo-rithm would recommend command Cut, since Cut has themost uses (and thus highest confidence level) of any com-mand that she does not already use.

The intuition behind this algorithm is that the commandsthat are used the most are also the most useful. A main ad-vantage of this algorithm is that it can make recommenda-tions to any user, including users whose history is unknown.One disadvantage is that its recommendations are not tai-lored; all users are assumed to be equal. This assumptionmay largely hold in some kinds of applications, but it isunlikely to hold in more complex software where differentusers perform different tasks using different workflows indifferent roles. As we alluded to in the introduction, devel-opment environments are likely to be one of these complexpieces of software due to differing languages (for example,python versus ruby), developer roles (programmer versusmanager), and differing tasks (fixing bugs versus adding fea-tures). These differences may make recommendations madeby this algorithm less relevant.

4.2 Existing: Collaborative FilteringCollaborative filtering algorithms can be divided into two

categories. The first is user-based collaborative filtering,where the command history of the user that we would liketo make a recommendation for is compared to the historiesof other users, and recommendations are made based solelyon the commands used by users with similar histories. Thesecond category is item-based collaborative filtering, wheresimilarities between commands are calculated based on thecommunity’s usage history, and a recommendation for a sin-gle user can be made by looking for commands similar to theones he already uses. The confidence level δ for collaborativefiltering algorithms is based on similarities between users orsimilarities of commands. We will not explain these algo-rithms in any more depth here; instead, we refer to readerto Matejka and colleagues for a technical explanation oncommand recommendation using collaborative filtering [12].

Looking at our example (Table 1), suppose we use user-based collaborative filtering to recommend a command toHilaria. The algorithm would find similarities in the histo-ries of Hilaria and Ignacio because they both use commandsCopy and Paste, and thus would recommend the commandCut. If we used item-based collaborative filtering to rec-ommend a command to Hilaria, the algorithm would noticethat, for example, someone who uses Copy also uses Cut, andsince Hilaria uses Copy, it would thus recommend Cut.

The intuition behind this family of algorithms is that usersand commands with similar histories are good sources of rel-evant recommendations. The main advantage of this familyof algorithms is that recommendations are tailored to spe-cific users. One disadvantage is that the algorithms ignoresequential and temporal aspects of command usage. For in-

stance, suppose that Table 1 displays only the first weekon the job for Hilaria and Ignacio, but Chuy been workingwith this software for 10 years. Also suppose that whenChuy started, his history looked very much like Hilaria’s,but changed into a more “expert” pattern of command us-age within his first few weeks on the job. If we ran a user-based collaborative filtering algorithm, very little similaritywould be found between Hilaria and Chuy, compared to Hi-laria and Ignacio. This is a problem because Chuy’s historycould be an excellent indicator of what command Hilariashould discover next, because Chuy was once a novice likeHilaria and successfully transitioned into an expert. In otherwords, traditional collaborative filtering ignores how usagepatterns change over time.

4.3 Novel: Collaborative Filtering with Dis-covery

In the next family of algorithms, we model how usagepatterns change over time with what we call “collaborativefiltering with discovery,” which are based in part on sequen-tial pattern mining [18]. With these algorithms, we modelhow users discover new commands, then run standard col-laborative filtering on their discovery patterns.

The algorithm operates as follows. For each user in acommunity, given each user’s command history, determinetheir base command usage, B, that is, all of the commandsthat they know up to a certain point in time t (a tuningparameter). Then, initialize a ruleset D= {}. Then startingat time t+1:

1. For each used command x where x 6∈B, D=D∪{y →x : y ∈B}.

2. Update B to reflect the newly discovered commands:B=B∪{x}

Continue these two steps until the end of the user’s com-mand history is reached. Next, we apply either user-basedcollaborative filtering or item-based collaborative filtering tothe ruleset D for each user in the community to obtain a rec-ommendation for the desired user. The result will be a listof command recommendations, each in the form x → y atconfidence level δ, where δ propagates from the underlyingcollaborative filtering algorithm. Ideally, the desired userwill already know command x, and not know command y.In that case, we add y to the set of recommendations atconfidence level δ. If the user does not know x or y, then werecommend only x at confidence level δ. Finally, after thelist of recommended commands has been produced, the com-mands recommended at the highest confidence levels will berecommended first.

For example, let us apply this algorithm to our examplefrom Table 1. Suppose we want to make a command rec-ommendation for Ignacio. First, let base command usage bedetermined by a single day (for any real data set, using a sin-gle day may yield inaccurate results; it is unlikely that a userwill use all the commands that she knows over the course ofjust one day). Thus, B for Hilaria is {Copy,Rename }, B forIgnacio = {Copy,Cut }, and B for Chuy is {Cut,Commit }.Looking at Tuesday and Wednesday, we see that: Hilariafirst discovers Paste, then Format; Ignacio discovers Paste,and Chuy discovers Update. This leads to the rule sets dis-played in Table 2.

Using the rulesets in the table, we can then apply collabo-rative filtering; we will use user-based collaborative filtering

Hilaria {Copy → Paste,Rename → Paste,Copy → Format,Rename → Format,Paste → Format}

Ignacio {Copy → Paste,Cut → Paste}Chuy {Cut → Update,Commit → Update}

Table 2: Rulesets for developers in Table 1.

in this example. Since we want to make a recommendationfor Ignacio, user-based collaborative filtering will notice asimilarity between the discovery rulesets for Hilaria and Ig-nacio; they both share Copy → Paste. Because of this simi-larity, the algorithm will suggest discovery rules that may beapplicable to Ignacio {Rename → Paste, Copy → Format,Rename → Format, Paste → Format}, resulting in recom-mendations for commands Rename and Format. Notice thatthis algorithm did not recommend Update, because there isno evidence that Ignacio and Chuy discover in the same way.

The intuition behind this family of algorithms is that itmakes command recommendations based on discovery styles.One major limitation of this approach is that it requires along enough usage history to capture discovery patterns. Forexample, if we would like to make a recommendation to auser who only has a week’s worth of history, we are facedwith two difficulties when picking the t parameter. If wepick a t that is early (say, two days into the week), any com-mands that we observe him “discovering” later in the weekmay not be discoveries at all; they may in fact be commandsthat he only needed later on in the week. However, if we picka t that is late (say, just before the last day of the week), werisk not observing any new commands being used, and thusthe algorithm could make no recommendations at all.

4.4 Novel: Most Widely UsedA slight extension of the Most Popular algorithm is what

we call the “most widely used” algorithm. This algorithmignores repeated uses of a command from a user, and insteadeach command’s confidence level δ is based on the numberof people who use that command. In our example (Table 1),suppose that we want to recommend a command to Hilariausing this algorithm. Because commands Copy , Cut , andPaste are the most widely used commands, each used bytwo users in the community, and Hilaria does not use Cut,the algorithm would recommend Cut to Hilaria.

The intuition behind this algorithm is that a tool usedby many people is a useful tool. The advantage is that itdoes not give undue weight to commands that are used alot, but by few people. One disadvantage is that commandswhich few people use, even if the users find them exceedinglyuseful, are unlikely to be recommended using this algorithm.

4.5 Novel: Advanced DiscoveryAnother algorithm we call “advanced discovery” is a sim-

pler version of the collaborative filtering with discovery algo-rithms above. The algorithm starts by producing a rulesetfor each user in the community. Then, given a user who wewould like to make recommendations for and the set of com-mands that she has used B, we create a set of all discoveryrules D from the community in the form x →y such that x∈B and y 6∈B. The algorithm then recommends the com-mands with the highest confidence level δ, where δ is definedas the number of occurrences of x→y in the set D such thatthe user under consideration has not used y.

Let us return to our example in Table 1, which gener-ates the rulesets shown in Table 2. Suppose we want tomake a recommendation for Ignacio. Ignacio’s set B is{Copy,Cut,Paste} and set D is

{Copy → Format,Paste → Format,Cut → Update}

Counting commands from D that Ignacio does not know,we have δ is 2 for Format and δ is 1 for Update. Thus, thisalgorithm would recommend Format to Ignacio.

The intuition behind this algorithm is that it makes rec-ommendations based on what a person already knows bylooking for commands that other people have discovered whoknew the same commands. The advantage of this algorithmis that it recommends commands for which the user has the“prerequisites” that other users had when they learned thosecommands.

4.6 Novel: Most Popular DiscoveryThis algorithm counts the number of times a a command is

discovered in the community. A command is recommendedwhen it has not already been used by the developer and hasthe highest δ, defined as the number of users who have usedthe command after the base command usage window.

For example, take the users in Table 1. The commandPaste was discovered by 2 users, Format by 1 user, and Up-date by 1 user. With this algorithm, if we wanted to makea recommendation to Chuy, we would recommend Paste.

The intuition behind this algorithm is that it recommendscommands that many other people have discovered in thepast. The main disadvantage is the same as with the “MostPopular” algorithm; it may recommend commands that arenot relevant to the tasks or style of the user.

5. AUTOMATED EVALUATIONWe conducted an automated comparison of algorithms to

investigate three research questions:

1. How feasible is automated recommendation of devel-opment environment commands?

2. How does automated recommendation of developmentenvironment commands compare to automated recom-mendation of commands in other software?

3. How do our novel algorithms compare to existing al-gorithms?

5.1 ImplementationWe implemented each algorithm in Java. Our implemen-

tation uses data structures from the Sequential Pattern Min-ing Framework [30], an open-source data mining framework.Our implementation also uses collaborative filtering algo-rithms from Apache Mahout [27], an open source machinelearning library. However, we modified Mahout’s machinelearning algorithms to align with the algorithms describedby Matejka and colleagues [12].

5.2 MethodologyTo investigate our research questions, we conducted a k-

tail evaluation as described by Matejka and colleagues [12].A k-tail evaluation first divides each user’s command his-tory into a training set and a testing set. The testing setcontains the last k tools that the developer discovered, and

the training set contains the entire usage history up untilthe first command in the testing set was discovered. Fi-nally, the evaluation runs each algorithm on the trainingset, and compares the recommendations made by each al-gorithm to the commands in the testing set. To illustratethe evaluation, consider again Table 1, which shows howthree imaginary users use commands. First, setting k=1,we create the training set for each user (for example, Hilaria= {Copy,Rename,Copy,Paste,Copy}) and the correspondingtesting sets (for example, Hilaria = {Format}). Finally, wetrain each algorithm on the data in the training sets, anduse the output of the algorithm to predict the command(s)in the testing set. In our example, a successful outcome foran algorithm is that, given the command usage for Hilaria,the algorithm recommends Format. In simple terms, a moresuccessful algorithm will be one that can more often predictthe last commands discovered by a set of users.

This type of evaluation has several advantages but alsolimitations, when compared to a live evaluation on real users(see Section 6). One advantage is that it allows the algo-rithm to make predictions for a very large number of users,reducing any biasing effect a single user might have. It alsoallows us to evaluate the algorithm with minimal distur-bance to users, allowing us to avoid possible Hawthorneeffects [4]. On the other hand, the evaluation only ap-proximates the quality of a recommendation because it mayfalsely classify a use of a command as a discovery becausethe user may actually be using the command again after us-ing it before data collection began, rather than discoveringthe command for the first time. Moreover, this evaluationonly measures whether the algorithms correctly predict whatcommand that users naturally discovered, not what com-mand would have been most useful for them to discover.

We created two variants of the k-tail evaluation becausethe standard k-tail described by Matejka and colleagues makesa key assumption, namely that the last command used bya user is actually a useful discovery of a command by thatuser. However, an analysis of the UDC data suggests thatsome Eclipse users only used certain commands once; ratherthan being useful discoveries, these may instead have beeninstances where the user tried a command and did not find ituseful. To mitigate this risk, we created two variants: k-tailmulti-use and k-tail multi-session. With k-tail multi-use, weonly include commands in the test set which have been usedmultiple times. With k-tail multi-session, we only includecommands that have been used in multiple sessions. Theoriginal k-tail evaluation, where a single use constitutes auseful discovery, we will refer to as k-tail standard.

In order to attain comparable results, we attempted touse the same algorithm and evaluation parameters used byMatejka and colleagues [12]. In the Appendix to this paper,we list those parameters.

5.3 DatasetWe used the dataset with 4309 Eclipse users, described

in Section 3. To fairly compare the algorithms, we onlyinclude users for which every algorithm can make a recom-mendation. For example, this excluded some users, such asthree who only appeared to use one command repeatedly in12 months; we suspect that these were automated Eclipsetest systems that did little more than start and stop anEclipse instance at regular intervals. As another example,some users appear to not discover commands after a rea-

sonable base command usage window, so algorithms such asAdvanced Discovery are unable to make recommendationsfor those users. Again, this illustrates the main limitation ofour discovery algorithms. When reporting results, we reportthe total number of users for which every algorithm was ableto make a recommendation.

5.4 ResultsFigure 3 displays the total number of correct recommen-

dations for each algorithm under each k-tail variant. Thenumber of Eclipse users for which all recommender algo-rithms produced results is shown in each column after “n=”.Algorithms introduced in previous work are shown with greybars while our novel algorithms are shown in black.

With regard to our first research question, the results inFigure 3 suggest that it is feasible to recommend develop-ment environment commands to software developers basedon previous usage history. Overall, the algorithms correctlypredicted between 1 in 10 to 3 in 10 commands, similar toprevious results that correctly predicted around 2 in 10 [12].One might ask if such a low percentage of useful recommen-dations is worthwhile. We argue that providing a developerwith one useful recommendation may make it worthwhile towade through nine not so useful ones, as long as it is easyto determine which recommendations are useful.

With regard to our second research question, the resultssuggest that recommendations for Eclipse commands aremore difficult to make correctly than for AutoCAD. For com-parison, on the graph we mark the three results obtainedby Matejka and colleagues with an asterisk (*) [12]. Forthe three comparable algorithms, Most Popular, User-BasedCollaborative Filtering, and Item-Based Collaborative Fil-tering, results for Eclipse were about 24% lower than therecommendations for AutoCAD. In Section 7, we discusswhy such a difference may exist.

With regard to our third research question, overall ournovel algorithms faired favorably compared to the existingalgorithms. Linton’s original Most Popular algorithm [10]had the lowest percentage of correct recommendations. User-Based Collaborative Filtering using Discovery, a novel al-gorithm we introduced, faired the best in two out of threeevaluations, being beaten in the Standard evaluation by Ad-vanced Discovery. We were surprised how often Most WidelyUsed was correct, considering that it is such a straightfor-ward extension of Linton’s original algorithm. Overall, thisevaluation suggests that the User-Based Collaborative Fil-tering using Discovery algorithm provides the most usefulrecommendations of the algorithms studied.

The algorithms’ performance varied significantly. Themost efficient, Most Popular, took a few milliseconds tomake each recommendation, whereas User-Based Collabora-tive Filtering with Discovery took up to 10 minutes to makerecommendations for a user. We view even 10 minutes as ac-ceptable because, in a practical implementation, each user’srecommendations would be computed on her own computer,in the background. Such distributed recommender systemalgorithms have been explored by Ali and van Stam [1].

6. LIVE EVALUATIONWe performed a second study to evaluate the quality of

our recommendations. In this study, we again borrow fromMatejka and colleagues [12] and try to make recommenda-tions to real developers, and then ask for their feedback

 

11.6%

22.4%

17.2%

11.6%

23.6%

23.2%

22.3%

23.3%

0% 10% 20% 0% 10% 20% 0% 10% 20% 30%

Most Popular

Most Widely Used

Item‐Based

User‐Based

Advanced

Popular

Item‐Based 

User‐Based 

    Collaborative Filtering

Discovery

Collaborative Filtering withDiscovery

15.4%

27.5%

24.0%

16.8%

22.8%

22.6%

28.6%

29.8%

15.2%

26.9%

22.2%

15.3%

22.8%

22.8%

27.5%

28.2%

Multi-Session (n=2471)Multi-Use (n=2855)Standard (n=3165)

*

*

*

Figure 3: % recommendations correct in each k-tail evaluation for each algorithm.

about the quality of recommendations. This study comple-ments the automated study in that it significantly alleviatesthe automated study’s main limitation, that is, that it wasunclear whether the recommendations were truly useful.

6.1 Participants and DatasetWhile the Matejka and colleagues study was able to con-

tact and recruit AutoCAD users because their own companydeveloped AutoCAD, we did not have any direct access toEclipse users through the Eclipse Foundation, due to pri-vacy concerns. Therefore, we took two small conveniencesamples of software developers; we treat each sample of de-velopers separately in the remainder of this paper. We feltthat taking a relatively small sample was appropriate at thisstage in the research, before building a tool appropriate forfield deployment and evaluation. We also felt that a conve-nience sample was acceptable, given that our methodologywas double-blind, as we explain in Section 6.2. We againused the 2009 dataset to train the algorithms.

The first set of developers we call the “experts”: these 4people had extensive software development experience and5 to 8 years of Eclipse experience. All were located inVancouver, Canada. The second set of developers we call“novices”: these participants were a mix of software devel-opers and students, all with some Eclipse development ex-perience. Each used Eclipse for Java development, and hadbetween 1 month and 6 years of experience with Eclipse,with a median of 2 years of Eclipse experience.

6.2 MethodologyMatejka and colleagues used Linton’s Most Popular, User-

Based Collaborative Filtering, and Item-Based Collabora-tive Filtering to generate 8 recommendations per algorithmfor a total of 17 users of AutoCAD. Without knowing whichalgorithm recommended which command, users respondedto a survey containing questions about novelty and useful-

ness of the command recommendations. For novelty, userswere asked to use a 5-point Likert scale to rate their agree-ment with the statement“I was familiar with the command,”from “strongly agree” to “strongly disagree”. For usefulness,users rated on a 5-point Likert scale their agreement withthe statement “I will use this command.”

We delivered the recommendations slightly differently tothe two different participant sets. Whereas Matejka and col-leagues sent their questions via survey, we asked the expertsto rate the commands during a face-to-face or phone-basedinterview. This allowed us to infer why the commands wereor were not novel and useful. For the novices, we evolved ourmethodology slightly; the second author showed the partici-pant each command using screencasts or live screen sharing,told them what keyboard shortcut (if any) invoked the com-mand, then asked participants to rate the novelty and use-fulness of the command over the phone. This difference be-tween the delivery methods for novices and experts presentsa threat to the validity of our study, specifically, that novicesmay have rated novelty and usefuless differently than ex-perts due in part to differences in delivery methods.

To reduce bias for both novices and experts, the first au-thor of this paper produced the recommendations from eachalgorithm, combined and randomized the commands, thengave the list to the second or third author to use to interviewthe participants. In this way, the study was double-blind;neither the interviewer nor the participant knew which al-gorithm produced which command.

6.3 ResultsSome algorithms were not able to recommend commands

for every user; we did not have long enough command us-age histories for several participants to make recommenda-tions with the Collaborative Filtering with Discovery algo-rithms. This was the case for 1 of the 4 experts and 7 ofthe 9 novices. Participants were recommended a mean of 31

commands each because there was overlap in the commandrecommendation set produced by each algorithm.

Overall, how many recommendations were rated novel byparticipants depended on the user group. For experts, par-ticipants reported that only about 26% of commands werenovel. For the novices, participants reported that about 80%of recommended commands were novel. In both cases, it wassurprising that many commands were not considered novelby participants. In discussing why those commands were notnovel, participants felt that either (1) they use an alterna-tive command to the recommended one that they feel suitsthem better or (2) that they already use the command. Wesuspect that when developers reported that they already usethe recommended command, the reason is that the EclipseUsage Data Collector does not always record a commandusage in the data or records the command in a different waywhether invoked via a key binding or a tool menu. These re-sults suggests that further work on the data collection side ofUDC to capture all command invocation events will improvethe usefulness of all algorithms’ recommendations.

For the commands that participants rated as novel, Fig-ure 4 displays how many commands each algorithm pro-duced that participants rated as either useful or not useful.The left column lists the name of the algorithm. Asterisksindicate that, as we mentioned, these algorithms could notproduce recommendations for some users. The middle ma-jor column indicates the number of useful and not usefulrecommendations for novices, while the right major columnindicates the same for experts.

For experts, most algorithms produced fewer useful rec-ommendations than not useful ones. Two exceptions thatproduced just as many useful as not useful recommendationswere Item-Based Collaborative Filtering and User-Based Col-laborative Filtering using Discovery. Item-Based Collabo-rative Filtering using Discovery was the only algorithm toproduce more useful recommendations than not useful ones.This result is consistent with our automated evaluation thatCollaborative Filtering with Discovery algorithms do well,as compared to the other algorithms.

For novices, all algorithms produced more useful recom-mendations than non-useful ones. The best performers wereLinton’s Most Popular and the Item-Based CollaborativeFiltering with Discovery algorithms, both of which producedzero not-useful recommendations. The Item-Based Collabo-rative Filtering and User-Based Collaborative Filtering withDiscovery algorithms also performed well, having only onenot-useful recommendation each. User-Based CollaborativeFiltering produced the most useful recommendations, butalso produced the most not-useful recommendations. Over-all, the results where Collaborative Filtering with Discoveryperforming relatively well are consistent with our automatedevaluation, although the results are more difficult to drawconclusions about because the Collaborative Filtering withDiscovery algorithms were not able to produce recommen-dations for many novice users. The surprise from this eval-uation is that Linton’s Most Popular did so well, when itperformed so poorly on the automated evaluation. We dis-cuss why this might be in the next section.

Despite these promising results, they likely do not reflectactual acceptance rates of recommendations if those recom-mendations were delivered by a fully automated system.Specifically, these results may overestimate the likelihoodof a successful recommendation because a person delivered

the recommendation. However, we are currently in the pro-cess of building a recommender system that accompaniestool recommendations with social endorsements from a de-veloper’s peers [14]. Such endorsements may improve thelikelihood that a recommendation is accepted. Nonetheless,a recommender system algorithm, such as the ones evaluatedhere, remain crucial to the success of this proposed system.

7. DISCUSSIONOverall, the results suggest that our Collaborative Filter-

ing using Discovery algorithms provide relatively good com-mand recommendations. However, our Collaborative Filter-ing using Discovery algorithms also require a longer-termusage history to make accurate recommendations than dosimpler algorithms. Perhaps a reasonable compromise is ahybrid-based approach, where simpler algorithms are usedto recommend commands to developers with little or no his-tory, and more advanced discovery algorithms are used fordevelopers with a more extensive history.

Our results suggest that good recommendations can beobtained by modeling short discovery patterns, such as thatthe rename command is usual discovered before the moveelement command. Our technique is based on sequentialpattern mining [18], which in the past has been used to de-termine customer buying habits over time. Our techniquecould be expanded to more fully implement sequential pat-tern mining by modeling longer and more sophisticated pat-terns of discovery. Other data mining algorithms may alsoimprove the quality of command recommendations further.

We were surprised that command recommendations forsoftware developers were significantly (about 24%) less ac-curate than recommendations for AutoCAD users in k-tailevaluations. Although we are uncertain of the exact causeof this difference, we believe that it may be due to howcommands are invoked in Eclipse versus in AutoCAD. InEclipse, actions (such as switching editors) may or maynot be invoked through commands, depending on how theyare implemented and how well the implementors adhere tocommand-invocation conventions. Strict, uniform adhesionis especially unlikely because Eclipse is open-source and de-veloped by many individual contributors, in contrast to Au-toCAD, which is closed-source. Thus, the command pre-dictability differences may not be as much related to dif-ferences in domain as to differences in command frameworkimplementation. We hope to test this hypothesis by runningour recommendations over data produced by closed-sourcedevelopment environments, such as Visual Studio.

In the live evaluation, novices were more readily accept-ing of recommendations than experts. There may be severalreasons for this difference. First, novices simply know fewercommands than experts. Second, with novices we includedthe keyboard shortcut for each recommendation, whereaswith experts we did not. Perhaps the fact that the recom-mendations were more concrete made them more appealing.Third, novices may be more flexible and open to more possi-bilities in the development environments they use. In otherwords, as developers gain an experience with an IDE, theymay become more prone to “tool inertia” [16]. If that is thecase, it may be beneficial for IDE command recommendersystems to make recommendations early in a developer’s ca-reer, so that the system can build a track record of successby the time the developer becomes more experienced and itbecomes harder to make useful recommendations.

0 2 4 6 8 10 12 14 16 18 20 0 2 4 6 8

Most Widely Used

Item‐Based

User‐Based

Advanced

Popular

Item‐Based* 

User‐Based* 

    Collaborative Filtering

Discovery

Collaborative Filtering withDiscovery

ExpertsNovices

1

1

6

6

7

2

4

6

1

4

3

2

2

2

3

1

5

5

6

1

2

2

3

7

9

18

15

7

Most Popular 17

2

Figure 4: The number of commands rated as useful or not useful by participants. Useful recommendationsare solid bars; not useful recommendations are white bars.

We were surprised that Linton’s Most Popular algorithmperformed so well for novices, yet did so poorly in the live ex-pert evaluation, the automated evaluation, and in the evalu-ations performed by Matejka and colleagues [12]. This sug-gests that which recommendation algorithm will be the mostuseful may depend in part on how mature the software useris to whom the recommendation is made.

Both evaluations were limited in that we were unable tomeasure recall, that is, the fraction of total relevant com-mands that an algorithm recommended. Calculating recallwould require, for each developer, that we determine whichof the hundreds of commands that they did not use wererelevant. This would have been impossible in the k-tail eval-uation and impractical in the live evaluation.

8. FUTURE WORKBuilding on the results presented here, we feel that there

are several promising areas for future work.One area for future work is improved detection of mean-

ingful discovery and adoption events. Our live participantswere surprised at the occasional poor quality of the rec-ommendations, such as a Subversion user who was recom-mended three CVS commands by the User-Based Collab-orative Filtering Algorithm. The problem could be thatour current discovery algorithms assume that any time anew command is used (or used repeatedly), that it is anindicator of a useful command. Alternatively, future algo-rithms could recognize discovery as some other data pat-tern. For instance, an algorithm could detect replacementbehavior, where a developer frequently uses a command fora period of time, then switches to using some other com-mand frequently. For example, a developer may frequentlyuse Eclipse’s find references command, but may later switchto using the open call hierarchy command, because the latteris a more efficient way to complete her tasks.

While the evaluations that we performed roughly estimatehow well our algorithms make useful command recommen-dations, future studies could evaluate usefulness in moremeaningful ways. Rather than making predictions aboutwhat command users would do naturally or ask developerswhether they think that they would use the recommended

commands, we could instead perform a longer term studywhere we evaluate whether the recommended commandsare actually adopted. A clever study might be able to de-termine more objectively whether the recommendation of acommand actually results in improved productivity, highersoftware quality, or novel software artifacts.

Finally, future command recommender systems may bene-fit from including richer information than simply which com-mand is recommended. Specifically, one participant in ourlive evaluation wanted to know how much time a commandwould save him compared to the commands he already uses.Moreover, that developer not only wanted to know whetherother developers find commands useful, but who those devel-opers were, so he could assess his trust in those commands.In previous work, we speculated that leveraging trust in thisway is essential for recommending commands that the de-veloper will fully embrace [16].

9. CONCLUSIONIn this paper, we introduced the notion of improving the

fluency of software developers by automatically recommend-ing development environment commands. Building on pre-vious work, we introduced several novel recommendation al-gorithms, including three that model patterns of commanddiscovery. In a proof of concept analysis and two studiesdrawing on data from several thousand Eclipse users, wedemonstrated the utility of discovery patterns, comparedseveral algorithms’ ability to predict command usage, andevaluated the novelty and usefulness of the algorithms’ rec-ommendations with real developers. Our results suggestthat it is feasible to recommend integrated development en-vironment commands to software developers, but researchremains to determine whether the recommended commandsresult in an improvement to the development process.

AcknowledgmentsWe thank our study participants and Giuseppe Carenini fortheir help. Thanks to Wayne Beaton at the Eclipse Foun-dation for access to data from the Usage Data Collector.

10. REFERENCES

[1] K. Ali and W. van Stam. TiVo: making showrecommendations using a distributed collaborativefiltering architecture. In Proceedings of theInternational Conference on Knowledge Discovery andData Mining, KDD ’04, pages 394–401. ACM, 2004.

[2] D. Campbell and M. Miller. Designing refactoringtools for developers. In Proceedings of the 2008Workshop on Refactoring Tools, pages 1–2, 2008.

[3] A. Cockburn and L. Williams. The costs and benefitsof pair programming. In In eXtreme Programming andFlexible Processes in Software Engineering, pages223–247, 2000.

[4] G. Diaper. The hawthorne effect - a fresh examination.Educational Studies, 16(3):261–267, 1990.

[5] J. Ellson, E. Gansner, L. Koutsofios, S. North, andG. Woodhull. Graphviz - open source graph drawingtools. In Graph Drawing, volume 2265 of LectureNotes in Computer Science, pages 594–597. SpringerBerlin / Heidelberg, 2002.

[6] T. Grossman, G. Fitzmaurice, and R. Attar. A surveyof software learnability: metrics, methodologies andguidelines. In Proceedings of the 27th InternationalConference on Human Factors in Computing Systems,CHI ’09, pages 649–658, New York, NY, USA, 2009.ACM.

[7] A. J. Ko and B. A. Myers. Finding causes of programoutput with the java whyline. In Proceedings of the27th International Conference on Human Factors inComputing Systems, CHI ’09, pages 1569–1578, NewYork, NY, USA, 2009. ACM.

[8] J. Lave and E. Wenger. Situated Learning: LegitimatePeripheral Participation. Cambridge University Press,1st edition, September 1991.

[9] W. Li, J. Matejka, T. Grossman, J. A. Konstan, andG. Fitzmaurice. Design and evaluation of a commandrecommendation system for software applications.ACM Trans. Comput.-Hum. Interact., 18(2):6:1–6:35,July 2011.

[10] F. Linton, D. Joy, H. Schaefer, and A. Charron. OWL:A recommender system for organization-wide learning.Educational Technology & Society, 3(1):62–76, 2000.

[11] C. Maltzahn. Community help: discovering tools andlocating experts in a dynamic environment. InCHI’07: Proceedings of the SIGCHI Conference onHuman Factors in Computing Systems, pages 260–261,New York, NY, USA, 1995. ACM.

[12] J. Matejka, W. Li, T. Grossman, and G. Fitzmaurice.CommunityCommands: Command recommendationsfor software applications. In Proceedings of the 22ndAnnual ACM Symposium on User Interface Softwareand Technology, UIST ’09, pages 193–202, 2009.

[13] M. M. Muller and W. F. Tichy. Case study: Extremeprogramming in a university environment. InProceedings of the 23rd International Conference onSoftware Engineering, ICSE ’01, pages 537–544,Washington, DC, USA, 2001. IEEE Computer Society.

[14] E. Murphy-Hill. Continuous social screencasting tofacilitate software tool discovery. In The Proceedingsof the International Conference on SoftwareEngineering, pages 1317–1320. IEEE, 2012.

[15] E. Murphy-Hill and A. P. Black. An interactiveambient visualization for code smells. In Proceedingsof the 5th International Symposium on SoftwareVisualization, SOFTVIS ’10, pages 5–14, 2010.

[16] E. Murphy-Hill and G. C. Murphy. Peer interactioneffectively, yet infrequently, enables programmers todiscover new tools. In Proceedings of the ACM 2011Conference on Computer Supported Cooperative Work,CSCW ’11, pages 405–414, 2011.

[17] E. Murphy-Hill, C. Parnin, and A. P. Black. How werefactor, and how we know it. In Proceedings of the31st International Conference on SoftwareEngineering, ICSE ’09, pages 287–297, Washington,DC, USA, 2009. IEEE Computer Society.

[18] R. Srikant and R. Agrawal. Mining sequentialpatterns: Generalizations and performanceimprovements. In P. Apers, M. Bouzeghoub, andG. Gardarin, editors, Advances in DatabaseTechnology U EDBT ’96, volume 1057 of LectureNotes in Computer Science, pages 1–17. SpringerBerlin / Heidelberg, 1996. 10.1007/BFb0014140.

[19] J. Stylos, A. Faulring, Z. Yang, and B. Myers.Improving api documentation using api usageinformation. In Visual Languages and Human-CentricComputing, 2009. VL/HCC 2009. IEEE Symposiumon, pages 119 –126, sept. 2009.

[20] M. B. Twidale. Over the shoulder learning:Supporting brief informal learning. Proceedings of theACM Conference on Computer Supported CooperativeWork, 14(6):505–547, 2005.

[21] P. Viriyakattiyaporn and G. C. Murphy. Challenges inthe user interface design of an IDE tool recommender.In Proceedings of the 2009 ICSE Workshop onCooperative and Human Aspects on SoftwareEngineering, CHASE ’09, pages 104–107, 2009.

[22] P. Viriyakattiyaporn and G. C. Murphy. Improvingprogram navigation with an active help system. InProceedings of the 2010 Conference of the Center forAdvanced Studies on Collaborative Research, CASCON’10, pages 27–41, New York, NY, USA, 2010. ACM.

[23] 10 Eclipse navigation shortcuts every Javaprogrammer should know. http://bit.ly/PVgPZl,May 2007.

[24] Eclipse - open resource shortcut - ctrl+shift+r.http://greatwebguy.com/programming/eclipse/

eclipse-open-resource-shortcut-ctrlshiftr/,Mar. 2007.

[25] Keyboard shortcuts - Eclipse.http://parenthetical-thoughts.blogspot.com/

2008/08/keyboard-shortcuts-eclipse.html, Aug.2008.

[26] Usage Data Collector Results, 2009. Website,http://www.eclipse.org/org/usagedata/reports/

data/commands.csv.

[27] Apache mahout. http://mahout.apache.org/, Mar.2012.

[28] Eclipse. http://www.eclipse.org, Mar. 2012.

[29] emacs. http://www.gnu.org/software/emacs/, Mar.2012.

[30] SPMF - a sequential pattern mining framework.http://www.philippe-fournier-viger.com/spmf/

index.php, Mar. 2012.

Appendix

Parameter Value Description RationaleSimilarity Function Cosine

SimilarityUsed to calculate the similarity betweentwo users or two commands.

Same as used by Matejka and col-leagues [12].

User NeighborhoodSize

32 The number of similar users from whichto draw commands to recommend foruser-based collaborative filtering algo-rithms.

Left unspecified by Matejka and col-leagues [12]; in initial experiments withuser-based collaborative filtering, we foundthat a value of 32 provided good results.

Base Command Us-age Window

40 sessions See Section 4.3 Based on observations about UDC data;appears short enough to enable recommen-dations but long enough to include mostcommands the user knows.

k 1 See Section 5.2 Same as used by Matejka and col-leagues [12].

Number of Recom-mendations

10 Maximum number of recommendationsthat each algorithm is allowed to pro-duce.

Same as used by Matejka and col-leagues [12].

α 1 Tuning parameter that serves as a scal-ing factor for confidence level (δ).

Left unspecified by Matejka and col-leagues [12]; any positive value appears toproduce the same results as any other pos-itive value.

Table 3: Parameters used in algorithm comparison.