Upload
masud-rahman
View
152
Download
3
Embed Size (px)
Citation preview
TOWARDS A CONTEXT-AWARE META SEARCH ENGINE FOR IDE-BASED RECOMMENDATION ABOUT PROGRAMMING ERRORS & EXCEPTIONSMohammad Masudur Rahman, Shamima Yeasmin, and Chanchal K. RoyDepartment of Computer ScienceUniversity of Saskatchewan
CSMR-18/WCRE-21 Software Evolution Week (SEW 2014), Antwerp, Belgium
2
Software Research Lab, U of SSOFTWARE MAINTENANCE, BUGS & EXCEPTIONS
A common experience!!
3
Software Research Lab, U of S
EXCEPTION HANDLING: IDE SUPPORT
1
2
4
Software Research Lab, U of S
EXCEPTION SEARCH QUERY
Class can not access a member of class java.util.HashMap$HashIterator with modifiers "public final”
5
Software Research Lab, U of S
EXCEPTION HANDLING: WEB SEARCH
Traditional web search•No ties between IDE and web browsers•Does not consider problem-context•Environment-switching is distracting & Time consuming•Often not much productive (trial & error approach)
6
Software Research Lab, U of S
IDE-BASED WEB SEARCH About 80% effort on Software Maintenance,
(Ponzanelli et al, ICSE 2013) Bug fixation– error and exception handling Developers spend about 19% of time in web
search, (Brandt et al, SIGCHI, 2009)o IDE-Based context-aware web search is
the right choice
7
Software Research Lab, U of S
EXISTING RELATED WORKS Rahman et al. (WCRE 2013)
ERA version of this paper Outlines basic idea, limited experiments
Cordeiro et al. (RSSE 2012) Based on StackOverflow data dump Subject to the availability of the dump, not easily
updatable Uses limited context, only stack trace Very limited experiments
Ponzanelli et al. (ICSE 2013) Based on StackOverflow data dump Uses limited context, only context-code Not specialized for exception handling
8
Software Research Lab, U of S
EXISTING RELATED WORKS Poshyvanyk et al. (IWICSS 2007)
Integrates Google Desktop in the IDE Not context-aware
Brandt et al. (SIGCHI 2010) Integrates Google web search into IDE Not context-aware Focused on usability analysis
9
Software Research Lab, U of S
MOTIVATION EXPERIMENTS
Search Query
Common for All
Google Unique
Yahoo Unique
Bing Unique
Content Only
32 09 16 18
Content and Context
47 09 11 10
75 Exceptions (details later) Individual engine can provide solutions for 58
exceptions at most. Maximizing total solutions
10
Software Research Lab, U of STHE KEY IDEA !! META SEARCH ENGINE
11
Software Research Lab, U of S
PROPOSED IDE-BASED META SEARCH MODEL
Start search
Results
Web page
12
Software Research Lab, U of SPROPOSED IDE-BASED META SEARCH MODEL Distinguished Features (5)
IDE-Based solution Web search, search result and web browsing all from IDE No context-switching needed
Meta search engine Captures data from multiple search engines Also applies custom ranking techniques
Context-Aware search Uses stack traces information Uses context-code (surroundings of exception locations)
Software As A Service (SAAS) Search is provided as a web service, and can be leveraged
by an IDE. http://srlabg53-2.usask.ca/wssurfclipse/
PROPOSED IDE-BASED META SEARCH MODEL Two Working Modes Proactive Mode
Auto-detects the occurrence of an exception Initiates search for exception by client itself Aligned with Cordeiro et al. (RSSE’ 2012) & Ponzanelli et
al. (ICSE 2013) Interactive Mode
Developer starts search using context menu Also facilitates keyword-based search Aligned with traditional web search within the IDE
14
Software Research Lab, U of S
SEARCH QUERY GENERATION Search Query required to collect results from
the Search Engine APIs and to develop the corpus.
Query generation Uses stack trace and Context code Collects 5 tokens of top-most degree of
interests from stack trace. Collects 5 most frequently invoked methods
in the context-code. Combined both token list to form the
recommended keywords for the context.
15
Software Research Lab, U of S
RESULT RANKING ASPECTS (4) Content-Relevance
Considers page title, body content against search query Context-Relevance
Considers stack traces from webpage against target stack trace
Considers code snippets against context-code extracted from IDE
Link Popularity Considers the Alexa & Compete site rank Estimates a normalized score from those ranks
Search Engine Confidence Heuristic measure of confidence for the result Considers the frequency of occurrence Considers the weight of each search engine
16
Software Research Lab, U of S
PROPOSED METRICS & SCORES Content Matching Score (Scms)
Cosine similarity based measurement Stack trace Matching Score (Sstm)
Structural and lexical similarity measurement of stack traces
Code context Matching Score (Sccx) Code snippet similarity (code clones)
StackOverflow Vote Score (Sso) Total votes for all posts in the SO result link
17
Software Research Lab, U of S
PROPOSED METRICS & SCORES
Site Traffic Rank Score (Sstr)-- Alexa and Compete Rank of each link
Search Engine weight (Ssew)---Relative reliability or importance of each search engine. Experiments with 75 programming queries against the search engines.
Heuristic weights of the metrics are determined through controlled experiments.
18
Software Research Lab, U of S
EXPERIMENT OVERVIEW 75 Exceptions collected from Eclipse IDE
workspaces of grad-students of SR Lab, U of S, and different online sources (StackOverflow, pastebin)
Related to Eclipse plug-in framework and Java Application Development
Solutions chosen from exhaustive web search with cross validations by peers
Recommended results manually validated. Results compared against existing
approaches and search engines.
19
Software Research Lab, U of S
PERFORMANCE METRICS Mean Precision (MP) Recall (R) Mean First False Positive Position (MFFP) Mean Reciprocal Rank (MRR)
20
Software Research Lab, U of S
RESULTS FOR SCORE COMPONENTSScore Components
Metrics Proactive Mode (Top 30)
Interactive Mode (Top 30)
Content MPTEFR
0.037156 (75)74.66%
0.048165 (75)86.66%
Content +Context
MPTEFR
0.037655 (75)73.33%
0.051466 (75)88.00%
Content + Context + Popularity
MPTEFR
0.038156 (75)74.66%
0.051966 (75)88.00%
Content +Context + Popularity +Confidence
MPTEFR
0.038056 (75)74.66%
0.053868 (75)90.66%
[ MP = Mean Precision, R = Recall, TEF= Total Exceptions Fixed]
21
Software Research Lab, U of S
RESULTS OF EXISTING APPROACHESRecommender Metrics Top 10 Top 20 Top 30Cordeiro et al. (only stack traces)
MPTEFR
0.020215 (75)20.00%
0.012818 (75)24.00%
0.008518 (75)24.00%
Proposed Method (Proactive Mode)
MPTEFR
0.088651 (75)68.00%
0.052955 (75)73.33%
0.038056 (75)74.66%
Ponzanelli et al. (only context-code)
MPTEFR
0.02437 (37)18.92%
0.01357 (37)18.92%
0.00997 (37)18.92%
Proposed Method (Proactive Mode)
MPTEFR
0.100030 (37)81.08%
0.062132 (37)86.48%
0.045032 (37)86.48%
[ MP = Mean Precision, R = Recall, TEF= Total Exceptions Fixed]
22
Software Research Lab, U of S
RESULTS OF SEARCH ENGINESSearch Engine Metrics Top 10 Top 20 Top 30Google MP
TEFR
0.157157 (75)76.00%
0.086457 (75)76.00%
0.058057 (75)76.00%
Bing MPTEFR
0.101355 (75)73.33%
0.053358 (75)77.33%
0.036458 (75)77.33%
Yahoo MPTEFR
0.098654 (75)72.00%
0.053957 (75)76.00%
0.036957 (75)76.00%
StackOverflow Search
MPTEFR
0.022614 (75)18.66%
0.014017 (75)22.66%
0.009717 (75)22.66%
Proposed Method (Interactive mode)
MPTEFR
0.122959 (75)78.66%
0.073664 (75)85.33%
0.053868 (75)90.66%
23
Software Research Lab, U of S
THREATS TO VALIDITY Search not real time yet, generally takes
about 20-25 seconds per search. Multithreading used, extensive parallel processing needed.
Search engines constantly evolving, same results may not be produced at later time.
Experimented with common exceptions, which are widely discussed and available in the web.
24
Software Research Lab, U of S
LATEST UPDATES More extensive experiments with 150
exceptions. Achieved 92% accuracy. Eclipse plugin release (
https://marketplace.eclipse.org/content/surfclipse) Context-aware Keyword search with
automatic query completion feature. Visual Studio 2012 Plugin under
development. Extensive User Study ongoing.
25
Software Research Lab, U of S
SURFCLIPSE TOOL DEMONSTRATION Tool Demo video:
https://www.youtube.com/watch?v=hGbyF4YveaI
26
Software Research Lab, U of S
THANK YOU !!!
27
Software Research Lab, U of S
REFERENCES[1] M.M. Rahman, S.Y. Mukta, and C.K. Roy. An IDE-Based Context-
Aware Meta Search Engine. In Proc. WCRE, pages 467–471, 2013.[2] J. Cordeiro, B. Antunes, and P. Gomes. Context-based
Recommendation to Support Problem Solving in Software Development. In Proc. RSSE, pages 85 –89, June 2012.
[3] L. Ponzanelli, A. Bacchelli, and M. Lanza. Seahawk: StackOverflow in the IDE. In Proc. ICSE, pages 1295–1298, 2013.
[4] D. Poshyvanyk, M. Petrenko, and A. Marcus. Integrating COTS Search Engines into Eclipse: Google Desktop Case Study. In Proc. IWICSS, pages 6–, 2007.
[5] J. Brandt, P. J. Guo, J. Lewenstein, M. Dontcheva, and S. R. Klemmer. Two Studies of Opportunistic Programming: Interleaving Web Foraging, Learning, and Writing Code. In Proc. SIGCHI, pages 1589–1598, 2009.
28
Software Research Lab, U of S
SAMPLE STACK TRACEjava.net.ConnectException: Connection refused: connectat java.net.DualStackPlainSocketImpl.connect0(Native Method)at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source)at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)at java.net.AbstractPlainSocketImpl.connect(Unknown Source)at java.net.PlainSocketImpl.connect(Unknown Source)at java.net.SocksSocketImpl.connect(Unknown Source)at java.net.Socket.connect(Unknown Source)at java.net.Socket.connect(Unknown Source)at java.net.Socket.<init>(Unknown Source)at java.net.Socket.<init>(Unknown Source)at test.SockTest.main(SockTest.java:13)
29
Software Research Lab, U of S
SAMPLE CONTEXT CODEtry {Socket client = new Socket("localhost", 4321);ObjectOutputStream out = new ObjectOutputStream(client.getOutputStream());out.flush();ObjectInputStream in = new ObjectInputStream(client.getInputStream());System.out.println("Buffer size: " + client.getSendBufferSize());for (int i = 0; i < 10; i++) {
if (i == 3) {Thread.currentThread().interrupt();System.out.println("Interrupted.");}out.writeObject("From Client: Hellow." + i);out.flush();System.out.println(in.readObject());}} catch (Exception e) {e.printStackTrace();}
30
Software Research Lab, U of SSEARCH QUERY FOR CORPUS DEVELOPMENT
java.net.ConnectException Connection refused connect currentThread
31
Software Research Lab, U of SITEMS USED FOR RELEVANCE CHECKING
java.net.ConnectException Connection refused connect currentThread+Sample Stack Trace+Sample Context Code
32
Software Research Lab, U of S
SAMPLE STACK TRACE (2)java.lang.ClassNotFoundException: org.sqlite.JDBCat java.net.URLClassLoader$1.run(Unknown Source)at java.net.URLClassLoader$1.run(Unknown Source)at java.security.AccessController.doPrivileged(Native Method)at java.net.URLClassLoader.findClass(Unknown Source)at java.lang.ClassLoader.loadClass(Unknown Source)at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)at java.lang.ClassLoader.loadClass(Unknown Source)at java.lang.Class.forName0(Native Method)at java.lang.Class.forName(Unknown Source)at core.ANotherTest.main(ANotherTest.java:18)
33
Software Research Lab, U of S
CONTEXT CODE (2)try{//code for making connection with a sqlite databaseClass.forName("org.sqlite.JDBC");Connection connection=null;connection=DriverManager.getConnection("jdbc:sqlite:"+"/"+"test.db");Statement statement=connection.createStatement();String create_query="create table History ( LinkID INTEGER primary key, Title TEXT not null, LinkURL TEXT not null);";boolean created=statement.execute(create_query);System.out.println("Succeeded");}catch(Exception exc){exc.printStackTrace();}
34
Software Research Lab, U of SSEARCH QUERY FOR CORPUS DEVELOPMENTjava.lang.ClassNotFoundException org.sqlite.JDBC db ClassLoader execute
35
Software Research Lab, U of SITEMS USED FOR RELEVANCE CHECKING
java.lang.ClassNotFoundException org.sqlite.JDBC db ClassLoader execute +Sample Stack Trace+Sample Context Code