CSMR10a.ppt

A Heuristic-based Approach to Identify Concepts in Execution Traces

CSMR 2010 - Madrid (Spain) 1

Fatemeh Asadi*

Massimiliano Di Penta**

Giuliano Antoniol*

Yann-Gaël Guéhéneuc**

* Ecole Polytechnique de Montréal, Canada

** Dept. Of Engineering – Univ. of Sannio, Italy

Motivations

• Software systems lack adequate documentation

• Developers try to understand systems through

– Static analyses, visualizations built upon static data

– Dynamic analyses, requiring the execution of the system

• (Dynamic) concept identification

Identify sets of method calls in execution traces responsible


– Identify sets of method calls in execution traces responsible for the implementation of domain concepts or user-observable features

– Existing approaches based on static analysis [Anquetil and Lethbridge (1998)], dynamic analysis [Wilde and Scully (1995) Tonella and Ceccato (2004)], IR techniques [Poshyvanyk et al. (2007)], or hybrid ones [Eaddy et al. (2008)]

Proposed approach

� A novel approach that analyzes execution traces and groups together method calls that:

� (i) sequentially invoked together/in sequence

� (ii) cohesive and decoupled from a conceptual point of view

� Assumptions


� Let us consider a feature is being executed in a scenario

– e.g., “Open a Web page from a browser”or “Save an image in a paint application”

� The set of methods related to the feature is likely to be:

– (i) conceptually cohesive

– (ii) decoupled from those of other features

– (iii) sequentially invoked

Proposed approach

� Step I – System instrumentation

� Step II – Execution trace collection

� Step III – Trace pruning and compression

� Step IV – Textual analysis of methods’


� Step IV – Textual analysis of methods’ source code

� Step V – Search-based concept identification

Step I and Step II – Getting Traces

� Step I - System instrumentation

� System instrumented using the MoDeC instrumentor

– MoDeC tool to extract and model sequence diagrams for Java systems

� Java bytecode instrumentation tool


– Inserts appropriate and dedicated method invocations in the system to method/constructor entry/exit, points

– Allows for trace tagging

� Step II - Execution trace collection

� We exercise a system following operation sequences taken from user manuals or use case descriptions

Step III – Trace Pruning and Compression

� Removing methods not very useful for feature identification

� Methods occurring in many scenarios

– Are often utility methods

– We use the same idea of tf-idf in Information Retrieval

� Too frequent methods

– Could be for example related to crosscutting concerns

– We remove methods having a frequency


– We remove methods having a frequencyQ3 + 2 ×××× IQR (75% percentile + 2 × the interquartile range)

� Trace compression

� Aim: collapse repetitions in execution traces

� Purpose: reduce the search space for Step V

� Examples:

– m1(); m1(); m1();

– m1(); m2(); m1(); m2();

� Performed using the Run Length Encoding (RLE)

� Applied for sub-sequences having an arbitrary length

m1();m1; m2();

Step IV

� Conceptual cohesion and coupling determined according to [Marcus et al., 2008] and [Poshyvanyk et al., 2006]

� Index identifiers, comments contained in methods

� Extraction of identifiers and comment words

� Camel-case splitting of composed identifiers

� Stop word removal (English + Java keywords)


� Stop word removal (English + Java keywords)

� Stemming using the Porter stemmer

� Indexing using tf-idf

� Reduce the term-document space into a (smaller) concept-document space using Latent Semantic Indexing (LSI)

– Helps to cope with synonymy and homonymy

– Concept space=50

Step V� We use a search-based optimization technique based on Genetic

Algorithms (GA) to split traces into segments

� Representation: a bit-vector where 1 indicates the end of a segment

� Mutation: randomly flips a bit (i.e., splits or merge segments)

m1 m2 m1 m3 m4 m1 m4 m6 m1

0 1 0 0 1 0 0 0 1

Trace splitting

Representation


� Mutation: randomly flips a bit (i.e., splits or merge segments)

� Crossover: two-points

� Selection: Roulette Wheel

0 1 0 0 1 0 0 0 1

0 1 0 0 1 0 0 0 1

0 0 1 0 0 1 0 0 1

1 0 1 0 0 1 0 0 0 110

0 1 0 0 0 1 0 0 1

0 0 1 0 1 0 0 0 1

Step V – Quality of the Solution� Fitness Function:

� Segment Cohesion is the average (textual) similarity between any pair of methods in a segment

� Segment Coupling is the average (textual) similarity


� Segment Coupling is the average (textual) similarity between a segment and all other segments in the trace

� Other GA parameters

� 200 individuals

� 2,000 generations for JHotDraw and 3,000 for ArgoUML

� 5% mutation probability, 70% crossover probability

� Distributed GA implementation (across 4 servers)

Empirical Study

• Goal: analyze the novel concept location approach based• Purpose: of evaluating its capability of identifying

meaningful concepts• Quality focus: accuracy and completeness of the

identified concepts• Context: an implementation of our approach and

execution traces extracted from two open source


execution traces extracted from two open source systems, JHotDraw and ArgoUML

Research Questions

� RQ1: How stable is the GA, through multiple runs, when identifying concepts into execution traces?

� RQ2: To what extent the identified concepts match the ones in the oracle?


concepts match the ones in the oracle?

� RQ3: How accurate is the identification of concepts in execution traces?

RQ1: GA stability� We compute the overlap between segmentations obtained in multiple runs using the Jaccard overlap Score

� Two segments overlaps when they contain calls in the same position of the trace

� Because a segment of trace T1 overlaps with more segments of T2,


2/4 3/42/3

� Because a segment of trace T1 overlaps with more segments of T2, the highest similarity is chosen

m1 m2 m1 m3 m4 m1 m4 m6 m1

m1 m2 m1 m3 m4 m1 m4 m6 m1

Run 1

Run 2

RQ1: Results


� Average overlap between 72% and 84%

� Slightly higher convergence for ArgoUML

� Ability of the algorithm to converge, despite the relatively large search space

RQ2: Matching with the Oracle

� We manually tag start-end of features while executing the system� Using the MoDeC instrumentation tool

� While executing the instrumented system, the user triggers the introduction of <Start> and <Stop> tags in the trace


� Matching between identified traces and oracle computed as in RQ1

2/4 3/42/3

m1 m2 m1 m3 m4 m1 m4 m6 m1

m1 m2 m1 m3 m4 m1 m4 m6 m1

Run 1

Oracle

RQ2: Results


� High overlap for some features

� e.g., Draw rectangle or Draw circle

� Lower for features obtained adapting other ones

� e.g., Add text obtained adapting Draw rectangle

� In other cases, low overlap is due to large segments split into more smaller and cohesive ones

RQ3: Accuracy in trace identification

� Computed similarly to RQ2, however we use Precision instead of Jaccard overlap Score


2/3 3/42/2

m1 m2 m1 m3 m4 m1 m4 m6 m1

m1 m2 m1 m3 m4 m1 m4 m6 m1

Run 1

Oracle

RQ3: Results


� Precision often very high

� In most cases above 85% and often equal to 100%

� Low precision (mean 32%) for Add text

� Relatively low (mean 69%) for Draw rectangle

� These two features are difficult to be distinguished

Inspection of the obtained segments� Add class (ArgoUML)

� The approach split this long feature of 199 methods sequence into 5 segments related to sub-features (creation of objects, adding the project class, handling namespace, setting object properties, handling persistence of the diagram)

� Create note (ArgoUML)

� Only the first part (50 methods) of the trace composed of 88 calls was identified

� Problems related to multi-threading

� Problems related to collapsing (during compression) loops containing variants


� Cut rectangle (JHotDraw)

� Only the last 39 out of 172 calls were included in the segment

� Methods related to adding to the clipboard and showing the rectangle as “cut”

� First methods related to GUI events and split in many small segments

� Spawn window (JHotDraw)

� 72 out of 197 methods included

� The remaining ones were related to setting up menu command properties

Threats to Validity

� Construct validity (relation btw. theory and observation)� Multi-threading can change the ordering of calls in multiple executions of the same scenario

� A better assessment of the actual content of the obtained segments is needed

� Internal validity (presence of confounding factors)


� Internal validity (presence of confounding factors)� Trace tagging may be imprecise, again due to multi-threading

� Noise due to utility methods

� GA intrinsic randomness

� External validity (generalization of findings)� We analyzed two different systems, multiple traces

� As usual, further empirical evaluation is needed

Conclusions� We proposed a search-based approach to automatically locate

concepts in execution traces

� By splitting traces into conceptually cohesive and decoupled segments

� Empirical study on traces from JHotDraw and ArgoUML shows that

� The approach is stable

� Identified segments highly precise

� Finer-splitting wrt. high-level features


� Finer-splitting wrt. high-level features

� Limitations due to: multi-threading, GUI events, feature adaptation..

� Work-in-progress:

� Improve performance

� Use enhanced compression techniques

� Automatically label identified concepts

� Perform an extensive empirical validation

Thank You!


Questions?

Technology

CSMR10a.ppt