Research issues in object oriented software testing

Research Issues in Object Oriented Software Testing

Submitted to: Jitender kumar Chhabra

Submitted By: Anshul Rani

Object Oriented Testing:- Basics Software testing is a process of executing a program or application with

the intent of finding the software bugs.

It can also be stated as the process of validating and verifying that a software program or application or product

Meets the business and technical requirements that guided it’s design and development

Works as expected

“The testing of software is an important means of assessing the software to determine its quality.”[1]

[1] G.Suganya, S.Neduncheliyan , “A Study of Object Oriented Testing Techniques: Survey and Challenges”, IEEE international Conference on Innovative Computing Technologies (ICICT), pp. 1-5, 29 March, 2010.

Object Oriented Testing:- BasicsNeed of Object oriented Testing: In object oriented softwares Classes provide an excellent structuring mechanism. They allow a system to be divided into well-defined

units. which may then be implemented separately. Classes support information hiding. A class can export a purely procedural interface and the internal

structure of data may be hidden. . object-orientation encourages and supports software reuse. This may be achieved either through the simple

reuse of a class in a library, or via inheritance, whereby a new class may be created as an extension of an existing one [2].

“These Extra facilities provided by OO design may lead to some types of faults that are difficult to detect using traditional testing techniques.”[1]

Studies suggests that traditional testing techniques, such as functional testing, statement testing and branch testing, are not viable for detecting OO faults. To overcome these deficiencies, it is necessary to adopt an object-oriented testing technique that takes these features into account. [3]


[2] Kolling, Michael, and John Rosenberg. "Support for object-oriented testing." Technology of Object-Oriented Languages,IEEE TOOLS 28th. Proceedings, pp.204-215 ,1998.

[3] Tang, M. H., M. H. Chen, and M. H. Kao,"Investigating test effectiveness on object-oriented software-a case study." Proceedings of the 12th Annual International Software Quality Week (1999).

Testing

Black-box

White-box




Black-box testing also known as Functional testing, is focused on program’s functionality without knowing the program’s internal structure. [1]

White-box testing also known as Structural testing , is designed to examine the internal structure of the program [2,3].

Object Oriented Testing (Categorization)

Object Oriented Testing (Types)

Traditional testing techniques

(adopted for OO)

Method Testing

Class testing

System Testing

Regression Testing

Integration Testing


ensures Statement Coverage to ensure that all statements have been traversed at least once, Decision Coverage to ensure all conditional executions and Path Coverage to ensure the execution the true and false part of the loop.

Class testing is performed on the smallest testable unit in the encapsulated class. Each operation as part of a class hierarchy has to be tested because its class hierarchy defines its context of use.

OO does not have a hierarchical control structure so conventional top-down and bottom- up integration tests have little meaning. Integration testing can be applied in three different strategies: Thread-based testing, Use-based testing, Cluster testing. Various types of System Testing include: Recovery testing, Security testing, Stress testing, Performance testing

Regression testing is performed similar to traditional systems to make sure previous functionality still works after new functionality is added.

Object Oriented Testing (Types)

Specialized Techniques for

oo

Fault – Based Testing

Scenario – Based Testing

Deep (architectural)

structure Testing

Surface Structure Testing


[4] Roger S.Pressman “Software Engineering – A Practitioner’s Approach” McGraw Hill International Edition., 2005 [5] ]http://www.exampler.com/testing-com/ ritings/2-scen.htm

Fault – Based testing is the method used to design tests that have a high probability finding probable errors of the software [4]. This type of testing can be based on the specification (user's manuals, etc.) or the code.

This new type of testing concentrates on what the customer does, not what the product does. It means capturing the tasks (use cases, if you will) the customer has to perform, then using them and their variants as tests. Of course, this design work is best done before you've implemented the product. [5]

A basic principle of testing is that we must trick our self into seeing the product in a new way. If the product has a direct manipulation interface, we'll test it better if we pretend functions are independent of objects [5].

Various constructs that can be tested using Deep testing can be found in [1].

Object Oriented Testing(When to Test)

“Test Early, Test Often, Test enough” [6]

[6] McGregor, John D., and David A.Sykes, “A practical guide to testing object-oriented software.” Addison-Wesley Professional, 2001.

Postponing testing towards the end of the project should be avoided. Start testing at reasonable points during the analysis and design phases of a project. This will help to uncover problems early in the development process.

Since testing often consumes 40~50% of development efforts, and consumes more effort for systems

Testing should be performed on every iteration of the development process. Products should be tested after the end of each iteration of the analysis, design and implementation phases. Complete testing of every aspect of software is infeasible.

Possible Directions

Object Oriented Testing

Improvement in Existing Testing

Techniques

Introducing New Techniques

(saving time and cost)

Prioritization of test cases (to save testing

time )

Automatic Test Case Generation

Issues Identified1. Introducing new techniques which are best suited for the OO Design than

existing techniques : Though object oriented programs are helpful in programming large systems, testing of such systems requires much more effort and time.

2. Efficient Test case generation approaches for all testing specially for integration testing: —Testing is way of insuring quality of the product. Object oriented testing is challenging task. It becomes more intense in integration level when different units interact for proper functioning of system.

3. Automatic test case generation 4. Prioritization of test cases: Software testing is expensive and time consuming,

especially for complex software. To reduce time test cases are generated automatically and a drawback is a huge generated test suite. Test case prioritization (TCP) involves the explicit prior planning of the execution order of test cases to increase the effectiveness of software testing activities by improving the rate of fault detection earlier in the software process.

New Testing Techniques

Different MethodologiesFor

OO Softwares

Transitions from Traditional Testing to OO Testing(An improvement in efficiency of software Testing)[7]

Traditional Approach

Metamorphic Approach

Metamorphic Approach for OO

(GFT algorithm)

Diff. Approaches Associated limitations and Approach Description

Oracle Problem:[8 ]

In some situations, it is impossible or practically too difficult to decide whether the program outputs on the test cases are correct. So it is difficult to determine whether the program execution results are identical to the expected results.

Problems:1..Lacking necessary metamorphic relations constructing criteria. [9,10]2. Generate a large number of test cases that have similar function.3. Testing is inadequate because of low functional coverage.

When we test object-oriented programs all these methods mentioned above are not possible. So it is necessary to propose a object-oriented metamorphic relation construction method. So authors introduced GFT algorithm GFT algorithm[11] to generate a limited number of base pairs as a test case based on algebraic specifications.

[7] Zhang, Xinglong, Lei Yu, and Xuemei Hou. "A method of metamorphic relations constructing for object-oriented software testing.“, 17th IEEE/ACIS International Conference on. Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD), pp. 399-406, IEEE, 2016.

[8] Chen, Tsong Y., Shing C. Cheung, and Shiu Ming Yiu. "Metamorphic testing: a new approach for generating next test cases." Department of Computer Science, Hong Kong University of Science and Technology, Tech. Rep. HKUST-CS98-01 (1998).

[9] ZHAN W, SONG H, "Achievements and Challenges of Metamorphic Testing," in proc. WCSE 2013: IEEE Fourth World Congress on Software Engineering, pp.73-77, 2013.

[11] Chen H Y, CHEN T.Y, "In Black and White: An Integrated Approach to Class-Level Testing of Object-Oriented Programs," ACM Transactions on Software Engineering and Methodology, vol. 10, no. 3, pp. 250-295, Dec. 1998.

Model Based Distributed Testing of Object Oriented Programs Issue addressed:New Technique identifiedSummary:

Object oriented programs are helpful in programming large systems, testing of such systems requires much more effort and time.

In study, the program is analyzed to create a model based on System Dependence Graph(SDG)

SDG is then used to find locations within the program where the state of the program can be froze and reused while executing other test cases.

Details of Proposed Methodology: A novel approach to distributed testing of object oriented programs based on state where

intermediate state of the program is saved for reuse. The system brings about efficiency to testing by reusing saved states of the program during

execution. The state of a program that is executing can be saved by using snapshot-ting techniques. In Java, continuation object can be used to capture everything in the java stack of a client (host)

node. The continuation objects is then serialized and sent to server which reinstates the state on a client

node with the help of a client supervisor running on the client node. Authors proposed a hosted model for this system where we have the client supervisor running

above the operating system.

[12] Kumar, KS Vipin, and Sheena Mathew. “Model Based Distributed Testing of Object Oriented Programs." In Procedia Computer Science, Vol No 46, pp. 859-866, 2015.

Conformity Testing by Optimal Constraints for Object Oriented Programs [44]

In this a new constraint model for testing the conformity of overriding methods during inheritance operation for an object oriented (OO) system is introduced.

This model is based on formal specification techniques and can be used to generate test data in derived classes.

Idea is to use an optimal constraint and a partitioning technique based on a mathematical analysis to automate the conformity testing of overriding methods.

[44] Louzaoui, Khadija, Khalid Benlhachmi, and Jihane Alami Chentoufi. "Conformity testing by optimal constraints for object oriented programs." I4th IEEE International Colloquium on Information Science and Technology (CiSt), , pp. 188-193, 2016.

[45] Bashir, Muhammad Bilal, and Aamer Nadeem. "A fitness function for evolutionary mutation testing of object-oriented programs." In IEEE 9th International Conference on Emerging Technologies (ICET, pp. 1-6, 2013.

A Fitness Function for Evolutionary Mutation Testing of Object-Oriented Programs [45]

Mutation testing is a fault based testing technique that helps generating effective test cases for software under test.

Mutation testing is computationally expensive by nature. Using evolutionary testing in conjunction with mutation testing can reduce computational cost and the

whole process can be automated to facilitate testers. Fitness function for the evolutionary mutation testing that supports all object oriented features, guides

the search by considering object's state separately, helps determining infection in the object's state at mutated statement, and reveals potential software bugs masked in equivalent mutants.

An Efficient Approach for Distributed Regression Testing of Object Oriented Programs [46]

As the software system grows, it becomes more and more difficult to test the entire system.

Testing minor changes in the software became difficult.

Regression testing was introduced to overcome this issue by testing only the changed portion of the system.

This study presents a novel approach in distributed regression testing of object oriented software

The approach presented in this paper makes use of data flow and control flow analysis of the source code to aid in distribution of the software during testing.

The test cases are executed in parallel, minimizing the time required for the testing process.

[46]Kumar, K. S., A. Lallu, and Sheena Mathew, "An Efficient Approach for Distributed Regression Testing of Object Oriented Programs." ACM In Proceedings of the International Conference on Interdisciplinary Advances in Applied Computing, p. 33, 2014.

TEST CASE GENERATION FOR OO SOFTWARES

Tools & MethodologiesConcerning

INHERITENCE , POLYMORPHISM etc.

OverviewReference No. Key Concept Description

[22] Proposed for random testing that uses state and callback

Algorithm exploits a combination of contracts and environment bindings to guide the test-case generator toward interesting input

[23] Uses byte code instrumentation to automatically separate code from its environmental dependencies

extended version of EVOSUITE Java test generation tool

[24] Approach concerns about constraints on interfaces, annotations, and reflection

[25] A tool named GenRed is proposed, extended version of Randoop.

GENRED, a tool that utilizes three approaches: input on demand creation and coverage-based method selection techniques

[22]Klein, Casey, Matthew Flatt, and Robert Bruce Findler. "Random Testing for Higher-order, Stateful Programs." ACM Sigplan Notices. Vol. 45. No. 10.,pp.555-566,2010.

[23] Arcuri, Andrea, Gordon Fraser, and Juan Pablo Galeotti. "Automated Unit Test Generation for Classes with Environment Dependencies." Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering, pp. 79-90, 2014.

[24] Islam, Mainul, and Christoph Csallner. "Generating test cases for programs that are coded against interfaces and annotations." ACM Transactions on Software Engineering and Methodology (TOSEM) Issue No. 3, Vol No. 23, pp.21,2014.

[25] Jaygarl, Hojun, Kai-Shin Lu, and Carl K. Chang. "GenRed: A tool for generating and reducing object-oriented test cases." IEEE 34th Annual Computer Software and Applications Conference, pp. 127-136., 2010.

OverviewReferenc

e No. Key Concept Description

[26] Approach differentiates between feasible and unfeasible test cases

Study employs evolutionary algorithms for generating and evolving test cases for the structural unit testing of object oriented Java programs.

[27] The proposed approach detects all related variables in uncovered branches and uses a fitness function to give method sequences an evaluation value to reduce candidate methods.

Method sequence generation to get desired object states is challenging. Seeker is one of the state-of-the-art implementation to generate test cases with method sequences. However, the technique generates method sequences to change a value of only a single variable. This study is extension to Seeker; while concerning multiple objects

[28] aPET is a test case generation tool for a distributed asynchronous language based on concurrent objects.

The system receives as input a program, a selection of methods to be tested, and a set of parameters that include a selection of a coverage criterion.

[26] Gupta, Nirmal Kumar, and Mukesh Kumar Rohil. "Improving GA based automated test data generation technique for object oriented software." IEEE 3rd International Conference In Advance Computing (IACC), pp. 249-253, 2013.

[27]Takamatsu, Hiroki, Haruhiko Sato, Satoshi Oyama, and Masahito Kurihara. "Method sequence generation for multiple object states using dynamic symbolic execution.” IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 3632-3636, 2014.

[28]Albert, Elvira, Puri Arenas, Miguel Gómez-Zamalloa, and Peter YH Wong. "aPET: a test case generation tool for concurrent objects.“ACM In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, pp. 595-598, 2013.

OverviewReferenc

e No. Key Concept Description

[29] Approach suited for inherited class hierarchies (ICH) and Non-public methods (NPM)

Study presents an approach to generate test cases for OO software via integrating evolutionary testing with reinforcement learning.

[30] T3i is an automated unit-testing tool to test Java classes.

T3i generates test-cases in the form of sequences of calls to the methods of the target class

[29] He, Wei, Ruilian Zhao, and Qunxiong Zhu. "Integrating Evolutionary Testing with Reinforcement Learning for Automated Test Generation of Object-Oriented Software." Chinese Journal of Electronics, Issue 1 Vol No 24, 38-45, 2015.

[30] Prasetya, I. S. "T3i: A tool for generating and querying test suites for java." ACM, In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, pp. 950-953,2015.

Data Flow Based Test Case Generation Algorithm for Object Oriented Integration Testing

Issue addressed:Test case generation of object oriented integration testing.Summary: A novel approach for test case generation of object oriented integration testing has been proposed.

Search space is reduced by selecting the coupling methods that are directly involved in integration.

OO Integration testing involves interaction of several objects, for proper testing these objects must be in desired state to achieve the required coverage .

Coupling based criteria introduces the coupling relationship between object oriented components. They have used this information in selecting coupling methods involved in integration testing and data

flow criteria to generate the method sequence for desired object state.

[13] Waheed, Shahzada Zeeshan, and Usman Qamar. "Data flow based test case generation algorithm for object oriented integration testing.“, 6th IEEE International Conference on Software Engineering and Service Science (ICSESS), pp. 423-427, 2015.


Proposed Methodology detailsThey used coupling based relation in method sequence generation for object oriented programs. There are four types of coupling identified in [14] 1. Parameter coupling: This type of coupling occurs when method of one class passes the object of another

class as arguments 2. Shared data coupling: This type of coupling relation exists when multiple classes objects are working on

the same shared data.3. Global Coupling: When A and B both classes share the same global reference.4. Inheritance Coupling: When one class inherits the properties form another class.

Proposed algorithm uses the coupling relations to generate the variable length method sequence to test the integration of different components.[13]


[14] Jin, Zhenyi, and A. Jefferson Offutt. "Coupling-based criteria for integration testing." Software Testing Verification and Reliability, Vol No. 8, pp.133-154,1998.


Proposed Methodology details[13] They used flow coverage criteria. Each DU path is considered in selecting method sequence to generate

the desired state for integration testing.

Instead to generating the method sequence for branch coverage we use coupling based data flow coverage for test case generation.

It generates a Tree that contains coupling method as root and all the possible def-use paths can be represented by the sub nodes of that tree.

Tree representation helps in generating the test cases with variable length. Length of the test case depends upon DU-Paths.


Various Test Case Generation Techniques for Object Oriented System

Different approaches of Test Case Generation are categorized as given:Scenario based Test Case Generation: In Scenario based test case generation test scenarios are used for generating test cases. Baikunt Narayan Biswal has given a novel approach for scenario based test case

Generation [15]. Approach deals with test adequacy criteria for complex transactions or Events, scenario

based testing gives best results. Test case generation UML Activity diagrams presented by Kim are also based on

concurrency in Activity Diagram where multiple systems interact with each other.

Model based Test Case Generation: In model based testing, the testing begins at design phase. Early detection of faults can be achieved by using this approach further reducing time, cost

and efforts of the developer to a large extent. Automatic Test case generation using Unified Modeling Language (UML) state diagram by P.samuel and A.K.Bothra and Rajib Mall published on the basis of Model Based Test Case generation.

[15]Baikuntha Narayan Biswal, Pragyan Nanda, Durga Prasad Mohapatra “A Novel Approach for Scenario- Based Test

Case Generation” IEEE International Conference On Information Technology, pp. 244-247,2008.

Various Test Case Generation Techniques for Object Oriented System

Genetic based Test Case Generation: In Genetic based test case generation technique, the test cases are generated using Genetic Algorithm.

Improving GA based Automated Test Data Generation Technique For Object Oriented Software [16] by Nirmal Kumar Gupta, Mukesh Kumar Rohil.

The proposed strategy shows that genetic algorithms are useful in reducing the number of unfeasible test cases by generating test cases for object oriented software.

In [17] a hybrid approach of generating test cases using sequence diagram with genetic algorithm as been propose

Sequence diagram shows the method call dependencies that exist among the methods that potentially appear in a method call sequence, which is good for integration testing..

[16] Nirmal Kumar Gupta, Mukesh Kumar Rohil, “ImprovingGA based Automated Test Data Generation Technique For Object Oriented Software”, IEEE 3rd International Advance Computing Conference (IACC),pp.249-253, 2013 .

[17] Mahesh Shirole, Rajeev Kumar, “A Hybrid Genetic Algorithm Based Test Case Generation Using Sequence Diagrams”, Springer Verlag Contemporary Computing Communications in Computer and Information Science, Vol94, pp 53-63, 2010

aPET: A Test Case Generation Tool for Concurrent Objects[28]

aPET, a test case generation tool for a distributed asynchronous language based on concurrent objects.

Concurrent ObjectsConcurrent objects are actors which communicate via asynchronous method callsFor Example “o!m();” denotes an asynchronous call from the this object to object o.

Each concurrent object (both this and o in this case) allows at most one active task to execute within the object.

The synchronization between the caller and the called methods can be optionally performed when the result is necessary by means of future variables.

Working Methodology The system receives as input a program, a selection of methods to be tested, and a set of

parameters that include a selection of a coverage criterion. It yields as output a set of test cases which guarantee that the selected coverage criterion

is achieved. The information yield by aPET can be relevant to spot bugs during program development

and also to perform regression testing.


Method Sequence Generation for Multiple Object States using Dynamic Symbolic Execution[27]

Software testing in object-oriented programming requires not only test input data, but also method sequences. Method sequences create new instances and change object states as desired

Method sequence generation to get desired object states is challenging. Seeker [31] is one of the state-of-the-art implementation to generate test cases with method

sequences. However, the technique generates method sequences to change a value of only a single

variable. Therefore, it cannot cover the branches that need to modify values of multiple variables. In this study, authors extend the technique in order to cover branches which require

multiple desired object states. The proposed approach detects all related variables in uncovered branches


[31] N. Tillmann and J. Halleux, “Pexwhite box test generation for .net,” in Tests and Proofs, ser. Lecture Notes in Computer Science, B. Beckert and R. Hhnle, Eds. Springer Berlin Heidelberg, vol.0 4966, pp. 134–153,2010..

Improving GA based Automated Test Data Generation Technique for Object Oriented Software[26]

Test automation include lowering the cost of tests and consequently, the cost of whole process of software development

Technique for automation in generating test data but this technique is expensive and cannot be applied properly to programs having complex structures.

Approaches in the area of object-oriented testing are limited in terms of test case feasibility due to call dependences and runtime exceptions.

This study proposes a strategy for evaluating the fitness of both feasible and unfeasible test cases leading to the improvement of evolutionary search by achieving higher coverage and evolving more number of unfeasible test cases into feasible ones.

[26] Gupta, Nirmal Kumar, and Mukesh Kumar Rohil. "Improving GA based automated test data generation technique for object oriented software." IEEE 3rd International Confrence In Advance Computing (IACC), pp. 249-253, 2013.

GenRed: A Tool for Generating and Reducing Object-Oriented Test Cases [25]

An important goal of automatic testing techniques, including random testing is to achieve high code coverage with a minimum set of test cases

To meet this goal, random testing researchers have proposed many techniques to generate test inputs and method call sequences that yield higher code coverage.

Mostly proposed random testing techniques are only suitable for toy systems, and they achieve low code coverage rates while generating too many unnecessary test cases on large-scale software systems

GENRED tool (proposed in this study) that utilizes three approaches:

input on demand creation and coverage-based method selection techniques that enhance Randoop[36]

a state-of-the-art feedback-directed random testing technique a sequence-based reduction technique that removes redundant test cases

without executing them.[36]


[36] C. Pacheco and M. D. Ernst., “Randoop: feedback-directed random testing for Java. GenRed: A Tool for Generating and Reducing Object-Oriented Test Cases” In Object Oriented Programming, Systems, Languages and Applications (OOPSLA), pp100-104, 2007

Generating Test Cases for Programs that Are Coded against Interfaces and Annotations [24]

Other current test case generators cannot handle cases in which the code under test uses multiple interfaces, annotations, or reflection.

To cover such code in an object-oriented setting, this study describes a novel technique for generating test cases and mock classes .

The technique consists of collecting constraints on interfaces, annotations, and reflection, combining them with program constraints collected during dynamic symbolic execution, encoding them in a constraint system

And then solving them with an off-the-shelf constraint solver, and mapping constraint solutions to test cases and custom mock classes.

Approach covered such third-party code with generated mock classes, while competing approaches failed to cover the code and sometimes produced unintended side-effects such as filling the screen with dialog boxes and writing into the file system.[24] Islam, Mainul, and Christoph Csallner. "Generating test cases for programs that are coded against interfaces and

annotations." ACM Transactions on Software Engineering and Methodology (TOSEM) Issue No. 3, Vol No. 23, pp.21,2014.

Automated Unit Test Generation for Classes with Environment Dependencies [23]

When classes interact with their environment, such as the file system, network, user-interactions, etc. This leads to two major problems:

Code that depends on the environment can sometimes not be fully covered simply by generating sequences of calls to a class under test

Even if code that is environment- dependent can be covered, the resulting tests may be unstable, i.e., they would pass when first generated, but then may fail when executed in a different environment.

In this study they applied byte code instrumentation to automatically separate code from its environmental dependencies

And extended the EVOSUITE[32-33] Java test generation tool such that it can explicitly set the state of the environment as part of the sequences of calls it generates.

[23] Arcuri, Andrea, Gordon Fraser, and Juan Pablo Galeotti. "Automated Unit Test Generation for Classes with Environment Dependencies." Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering, pp. 79-90, 2014

[32] G. Fraser and A. Arcuri. EvoSuite: Automatic test suite generation for object-oriented software. In ACM Symposium on the Foundations of Software Engineering (FSE), pages 416–419, 2011

[33]G. Fraser and A. Arcuri. Whole test suite generation. IEEE Transactions on Software Engineering, 39(2):pp276–291, 2013.

An optimized approach to generate object oriented software test case by Colored Petri Net[33]

Attributes of object- oriented software such as inheritance and polymorphism make behavior analysis and test significantly complicated

As the state of the objects may cause faults that cannot be easily revealed with traditional testing techniques

In this study, we propose a new technique for generating the test case by Colored Petri Nets (CPN),

CPN is an extended version of Petri Nets[34] and usually used to system modeling and simulation.

Methodology considers net explosion problem and also our generated Net covers all instances of objects from different classes in the same hierarchy by introducing new algorithm to convert UML State chart to CPN.

[33] Mirzaeian, Esmaeil, Samad Ghaderi Mojaveri, Homayun Motameni, and Ahmad Farahi. "An optimized approach to generate object oriented software test case by Colored Petri Net." In 2nd International Conference on Software Technology and Engineering (ICSTE), 2010, Vol. No 2, pp.2-251, 2010.

[34] H. Watanabe, H. Tokuoka, W. Wu, M. Saeki, "A Technique for Analyzing and Testing Object-Oriented Software Using Colored Petri Nets," apsec,, Fifth Asia-Pacific Software Engineering Conference (APSEC'98), pp.182, 1998

T3i: A Tool for Generating and Querying Test Suites for Java[35]

T3i is an automated unit-testing tool to test Java classes To expose interactions T3i generates test-cases in the form of sequences of calls to the

methods of the target class. This tool treats test suites as first class objects and allows users to e.g. combine, query,

and filter them. With these operations, the user can construct a test suite with specific properties. Queries can be used to check correctness properties. Hoare triples, LTL formulas, and

algebraic equations can be queried. T3i can be used interactively, thus facilitating more exploratory testing, as well as

through a script. The familiar Java syntax can be used to control it, or alternatively one can use the

much lighter Groovy syntax.

[35] Prasetya, I. S. "T3i: A tool for generating and querying test suites for java." In Proceedings of ACM 10th Joint Meeting on Foundations of Software Engineering, pp. 950-953, 2015.

PRIORITIZATION OF TEST CASES


OO Softwares

A Hierarchical Test Case Prioritization Technique for Object Oriented Software

Issue addressed:Test case Prioritization (TCP)Summary:- Every time any change occurs in the software, new test cases are added in addition to the

existing test suite. So there is need to conduct effective regression testing having less number of test cases to reduce cost and time.

A hierarchical test case prioritization technique is proposed wherein various factors have been considered that affect error propagation in the inheritance.

Prioritization of test cases take place at two levels:[18] In the first level the classes are prioritized In the second level the test cases of prioritized classes are ordered.

[18] Chauhan, Naresh, and Harish Kumar, "A hierarchical test case prioritization technique for object oriented software" , IEEE International Conference on Contemporary Computing and Informatics (IC3I),pp. 249-254, 2014.

A Hierarchical Test Case Prioritization Technique for Object Oriented Software[18]

Proposed Methodology:

Firs

t lev

el p

rior

itiza

tion

Seco

nd l

evel

pr

iori

tizat

ion

Prioritize classes according to number of inherited attributes/ methods/ classes/number of descendants

Calculate fault coverage per unit of each test class

Prioritize test class of each case

Apply testing on basis of classes and prioritize use of test classes

Get information about hierarchy of classes

Source Code


Proposed Methodology details[18] First Level Prioritization: The first level prioritization technique prioritizes the classes of object oriented software using inheritance

hierarchy.

In inheritance hierarchy the classes at lower level inherits the properties of classes at upper level. Therefore, the derived classes are dependent on the base classes.

This dependency increases the probability of error propagation through the inheritance hierarchy.

Hence the classes should be tested in such an order that the classes with higher probability of error propagation get tested first.

The probability of error propagation in inheritance hierarchy depends on the number of inherited attributes/methods, level of class in inheritance hierarchy and the number of descendent classes.

The base class should be assigned the highest priority. because if any errors get propagated from this class, will affect the entire hierarchy. So the classes should be ordered in such a way that error propagation can be minimized.


Proposed Methodology details:[18] Second Level Prioritization: The classes prioritized using first level prioritizations are input to the second level prioritization where

the test cases of each individual class are prioritized.

The test cases are prioritized based on fault weight and fault coverage.

The test cases that detect faults which have not been discovered earlier and are more critical are prioritized first.


Prioritizing Test Cases for Regression Testing of Location-Based Services: Metrics, Techniques, and Case Study [48]

Location-based services (LBS) are widely deployed When the implementation of an LBS-enabled service has evolved, regression testing can

be employed to assure the previously established behaviors not having been adversely affected.

Proper test case prioritization helps reveal service anomalies efficiently so that fixes can be scheduled earlier to minimize the nuisance to service consumers.

Study is based on the given observation:

Locations captured in the inputs and the expected outputs of test cases are physically correlated by the LBS-enabled service, and these services heuristically use estimated and imprecise locations for their computations, making these services tend to treat locations in close proximity homogenously.

Study proposes a suite of metrics and initializes them to demonstrate input-guided techniques and point-of-interest (POI) aware test case prioritization techniques, differing by whether the location information in the expected outputs of test cases is used.

[48]Zhai, Ke, Bo Jiang, and W. K. Chan. "Prioritizing test cases for regression testing of location-based services: Metrics, techniques, and case study." IEEE Transactions on Services Computing, Vol no. 1 pp. 54-67, 2014.

Prioritizing Test Cases for Resource Constraint Environments Using Historical Test Case Performance Data

The aim in this study to prioritize test cases during software regression test.

To achieve this, a new equation is presented. The proposed equation considers historical effectiveness of the test cases in fault

detection Each test case’s execution history in regression test and finally the last priority assigned to

the test case. Proposed equation to compute the priority of regression test cases provides two benefits

Faster fault detection High Coverage source code is achieved in less time

[49] Fazlalizadeh, Y., A. Khalilian, M. Abdollahi Azgomi, and S. Parsa. "Prioritizing test cases for resource constraint environments using historical test case performance data." In 2nd IEEE International Conference on Computer Science and Information Technology,(ICCSIT ), pp. 190-195. IEEE, 2009.

A Multi-Objective Technique to Prioritize Test Cases

Several existing test case prioritization techniques leave out the execution cost of test cases and exploit a single objective function i.e. to find test cases with high coverage

A multi-objective test case prioritization technique that determines the ordering of test cases that maximize the number of discovered faults that are both technical and business critical

This technique aims at both early discovering faults and reducing the execution cost of test cases.

Technique automatically recovers links among software artifacts (i.e., requirements Specification, test cases, and source code

A metric-based approach is applied to automatically identify critical and fault-prone portions of software artifacts, thus becoming able to give them more importance during test case prioritization

[37]Marchetto, Alessandro, Mahfuzul Islam, Waseem Asghar, Angelo Susi, and Giuseppe Scanniello. "A Multi Objective Technique to Prioritize Test Cases.“ In IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 42, NO. 10, pp. 918-940, 2016.

Using Partition Information to Prioritize Test Cases for Fault Localization[38]

Fault Localization prioritization aims at reordering existing test cases so that the location of detected faulty components can be identified earlier using certain fault localization techniques

In this paper FLP problem is being addressed using Black Box information derived from partitioning input domain.

This technique does not require test case execution history, that’s the key benefit of their proposed approach as execution history may not be always available

Other Benefits: Faster Localization of faulty statements Reduces the no of test cases that needs to be executed for whole coverage

[38]Zhang, Xiao-Yi, Dave Towey, Tsong Yueh Chen, Zheng Zheng, and Kai-Yuan Cai. "Using Partition Information to Prioritize Test Cases for Fault Localization." In IEEE 39th Annual Computer Software and Applications Conference (COMPSAC), Vol. 2, pp. 121-126. IEEE, 2015.

A Tool for Constrained Pair wise Test Case Generation Using Statistical User Profile Based Prioritization [39]

Pair wise testing is a wildly used approach in order to reduce the size of test suite and take steps to combinatorial testing problems because of an extensively large number of possible combinations between input parameters and values.

Invalid combinations between input parameters and values are possible if constraints are not been handled.

We present a pair wise test generation tool called CPTG, a tool to generate test cases for pair wise testing

User profile guidance is used to select optimal input parameters and values which do not depend on individual tester skills and also providing constraint handling solution between input parameters and values

Proposed tool is valuable in guiding testing with a maximized reliability by testing the most frequently used of the system

[39] Nakornburi, Sompong, and Taratip Suwannasart. "A tool for constrained pairwise test case generation using statistical user profile based prioritization.“, IEEE 13th International Joint Conference on Computer Science and Software Engineering (JCSSE), pp.1-6, 2016.

Prioritizing Test Cases For Regression Testing

Issue addressed:Test case Prioritization Summary:

Rate of fault detection is A measure of how quickly faults are detected within the testing process.

An improved rate of fault detection during testing can provide faster feedback on the system under test and let software engineers begin correcting faults earlier than might otherwise be possible.

The retesting of software is followed by modifications; Prioritization techniques can be done using information gathered from the previous execution of test

cases to obtain test case orderings. Study describes several techniques for using test execution information to prioritize test cases for

regression testing:[19]

Techniques that order test cases based on their total coverage of code components Techniques that order test cases based on their coverage of code components not previously

covered Techniques that order test cases based on their estimated ability to reveal faults in the code

components that they cover.

[19]Gregg Rothermel, Roland H. Untch, Mary Jean Harrold, “Prioritizing Test Cases For Regression Testing”, Ieee Transactions On Software Engineering, Vol.27, NO. 10, pp.929-948, OCTOBER 2001.

Prioritizing Test Cases For Regression Testing []

Several aspects of the test case prioritization problem defined in the study are given below..

Testers may wish to increase the rate of fault detection of a test suite - that is, the likelihood of revealing faults earlier in a run of regression tests using that test suite. .

Testers may wish to increase the coverage of coverable code in the system under test at a faster rate, thus allowing a code coverage criterion to be met earlier in the test process. .

Testers may wish to increase their confidence in the reliability of the system under test at a faster rate.

Testers may wish to increase the rate at which high- risk faults are detected by a test suite, thus locating such faults earlier in the testing process. .

Testers may wish to increase the likelihood of revealing faults related to specific code changes earlier in the regression testing process.


Strategies for Prioritizing Test Cases Generated Through Model-Based Testing Approaches[20]

Issue addressed:Test case Prioritization (TCP)Need of Prioritization:Software testing is expensive and time consuming, especially for complex software. In order to deal with the cost of testing, researchers develop Model-Based Testing (MBT). In MBT, test cases are generated automatically and a drawback is a huge generated test suite.Summary:-TCP techniques may be applied either in a general context, allowing the prioritization even in early stages of the testing process, or in a more specific context, such as regression testing, depending on the input information considered by the techniques. In this work, author proposed a technique based on the Adaptive Random Prioritization strategy, initially proposed by Jiang et al. [], adapted aiming at considering the expertise of the developers and testers about more failure prone regions of the application, exploring related test cases.

[20]Ouriques, João Felipe Silva., "Strategies for prioritizing test cases generated through model-based testing approaches." In Proceedings of IEEE 37th International Conference on Software Engineering, Vol 2, pp.879-882, 2015.

[21] B. Jiang, Z. Zhang, W. K. Chan, and T. H. Tse, “Adaptive random test case prioritization,” in ASE. IEEE Computer Society, pp. 233– 244, 2009.

Polymorphism Testing


OO Softwares

Fragment Class Analysis for Testing of Polymorphism in Java Software [40]

Adequate testing of polymorphism in object-oriented software requires coverage of all possible bindings of receiver classes and target methods at call sites

Tools that measure this coverage need to use class analysis to compute the coverage requirements.

Traditional whole- program class analysis cannot be used when testing partial programs

Study presents a general approach for adapting whole-program class analyses to operate on program fragments.

Study enables the use of whole-program class analyses for testing of polymorphism in partial programs, and identify analyses that compute precise coverage requirements and therefore are good candidates for use in coverage tools.

[40] Rountev, Atanas, Ana Milanova, and Barbara G. Ryder. "Fragment class analysis for testing of polymorphism in Java software." IEEE Transactions on Software Engineering, Issue No. 6, Vol. 30,pp. 372-387, 2004.

An Automated Approach to Inheritance and Polymorphic Testing using a VDM++ Specification [41]

Adequate testing of polymorphism in object-oriented software requires coverage of all possible bindings of receiver classes and target methods at call sites

Tools that measure this coverage need to use class analysis to compute the coverage requirements.

Traditional whole- program class analysis cannot be used when testing partial programs

Study presents a general approach for adapting whole-program class analyses to operate on program fragments.

Study enables the use of whole-program class analyses for testing of polymorphism in partial programs, and identify analyses that compute precise coverage requirements and therefore are good candidates for use in coverage tools.

[41] Nadeem, Aamer, Zafar I. Malik, and Michael R. Lyu., "An Automated Approach to Inheritance and Polymorphic Testing using a VDM++ Specification."IEEE International Multi topic Conference, pp. 224-230, 2006.

The Test Path Generation from State-based Polymorphic Interaction Graph for Object-

Oriented Software [42] In OO software it is particularly hard to detect faults when classes are integrated

because of inheritance, polymorphism and dynamic binding Unified Modeling Language (UML) is not only for software design, but also for

software testing. This study proposes an intermediate test model called the Polymorphism State

SEquence Test Model (PSSETM), which is generated from sequence diagram, class diagram and state-charts for integration testing.

The coverage criteria are also defined for test path generation By the selected coverage criterion, the algorithm of the test path generation is able

to create the test paths.

[42] Wu, Ching-Seh, Chi-Hsin Huang, and Yen-Ting Lee. "The Test Path Generation from State-Based Polymorphic Interaction Graph for Object-Oriented Software." In IEEE Tenth International Conference on Information Technology: New Generations (ITNG), pp. 323-330, 2013.

Inheritance Testing


OO Softwares

Specification Based Testing of Inheritance: A Framework of a Tool [47]

Object Oriented Systems are widely designed using UML, OCL specifications and these systems are to be tested with respect to a given specification.

Unified Modeling Language (UML) is not only for software design, but also for software testing.

This work addresses the problem of testing objects with inherited code For specifying inheritance, class diagrams annotated with OCL specifications and

method call sequences; from these test cases and test oracles are derived.. A tool has been developed to automate the process of testing The tool follows non- intrusive/invasive approach of testing to insure that an error

due to testing does not creep into code.

[47] Boyapati, Chandrasekhar, Sarfraz Khurshid, and Darko Marinov. "Korat: Automated testing based on Java predicates." In ACM SIGSOFT Software Engineering Notes, vol. 27, no. 4, pp. 123-133., 2002.

Integration Testing


OO Softwares

Design of a Tool for Checking Integration Testing Coverage of Object-Oriented Software [42]

Tool consists of 4 parts

o Code instrumentationo Test executiono Data analysiso Test case generation.

[42] Augsornsri, Pachawan, and Taratip Suwannasart. "Design of a tool for checking integration testing coverage of object-oriented software." In IEEE 2013 International Conference on Information Science and Applications (ICISA), pp. 1-4., 2013.

Design of a Tool for Checking Integration Testing Coverage of Object-Oriented Software [42]

Tool consists of 4 parts (details)

o Code instrumentation: In code instrumentation part, the tested source code is instrumented in order to verify which parts of the code are tested when test execution has been finished. The tested source code is read line by line and instrumented.

o Test execution: In this part, the instrumented code is executed after test execution. When test execution has been finishes, our tool will collect values of the counter variables and summarize percentage of class coverage and method coverage and output them to users.

o Data analysis: In case that there are uncovered classes and methods when test execution finishes, users are required to input the system’s sequence diagrams in XML format and use case descriptions to generate test cases in order to increase class coverage and method coverage values

o Test case generation: This section describes the test case generation process. This process can be separated to two sub processes: creating a scenario graph and generating test data.

Automated Test Data Generation for Coupling Based Integration Testing of Object Oriented Programs using Evolutionary Approaches [43]

Integration testing tests the interactions of different components, when they are integrated together in specific application, for the smooth functionality of software system.

Coupling based testing is an integration testing approach that is based upon coupling relationships that exist among different variables across different call sites in functions.

Different types of coupling exist between variables across different call sites. A novel approach for automated test data generation for coupling based integration

testing of object oriented programs using genetic algorithm has been proposed in this paper

Approach takes the coupling path as input, containing different sub paths, and generates the test data using genetic algorithm.

Authors have implemented a prototype tool E-Coup in Java for this approach.

[43]Khan, Shaukat Ali, and Aamer Nadeem. "Automated Test Data Generation for Coupling Based Integration Testing of Object Oriented Programs Using Evolutionary Approaches.”, IEEE Tenth International Conference on Information Technology: New Generations (ITNG), pp. 369-374. 2013

References[1] G.Suganya, S.Neduncheliyan , “A Study of Object Oriented Testing Techniques: Survey and Challenges”, IEEE

international Conference on Innovative Computing Technologies (ICICT), pp. 1-5, 29 March, 2010.



[4] Roger S.Pressman “Software Engineering – A Practitioner’s Approach” McGraw Hill International Edition., 2005

[5] ]http://www.exampler.com/testing-com/ ritings/2-scen.htm

[6] McGregor, John D., and David A.Sykes, “A practical guide to testing object-oriented software.” Addison-Wesley Professional, 2001.

[7] Zhang, Xinglong, Lei Yu, and Xuemei Hou. "A method of metamorphic relations constructing for object-oriented software testing.“, 17th IEEE/ACIS International Conference on. Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD), pp. 399-406, IEEE, 2016.

[8] Chen, Tsong Y., Shing C. Cheung, and Shiu Ming Yiu. "Metamorphic testing: a new approach for generating next test cases." Department of Computer Science, Hong Kong University of Science and Technology, pp. 98-101 1998

[9] ZHAN W, SONG H, "Achievements and Challenges of Metamorphic Testing," in proc. WCSE 2013: IEEE Fourth World Congress on Software Engineering, pp.73-77, 2013.

[11] Chen H Y, CHEN T.Y, "In Black and White: An Integrated Approach to Class-Level Testing of Object-Oriented Programs," ACM Transactions on Software Engineering and Methodology, vol. 10, no. 3, pp. 250-295, Dec. 1998.

References[12] Kumar, KS Vipin, and Sheena Mathew. “Model Based Distributed Testing of Object Oriented Programs." In Procedia

Computer Science, Vol No 46, pp. 859-866, 2015.[13] Waheed, Shahzada Zeeshan, and Usman Qamar. "Data flow based test case generation algorithm for object oriented

integration testing.“, 6th IEEE International Conference on Software Engineering and Service Science (ICSESS), pp. 423-427, 2015.

[14] Jin, Zhenyi, and A. Jefferson Offutt. "Coupling-based criteria for integration testing." Software Testing Verification and Reliability, Vol No. 8, pp.133-154,1998.

[15]Baikuntha Narayan Biswal, Pragyan Nanda, Durga Prasad Mohapatra “A Novel Approach for Scenario- Based Test Case Generation” IEEE International Conference On Information Technology, pp. 244-247,2008.

[16] Nirmal Kumar Gupta, Mukesh Kumar Rohil, “ImprovingGA based Automated Test Data Generation Technique For Object Oriented Software”, IEEE 3rd International Advance Computing Conference (IACC),pp.249-253, 2013 .

[17] Mahesh Shirole, Rajeev Kumar, “A Hybrid Genetic Algorithm Based Test Case Generation Using Sequence Diagrams”, Springer Verlag Contemporary Computing Communications in Computer and Information Science, Vol94, pp 53-63, 2010



[20]Ouriques, João Felipe Silva., "Strategies for prioritizing test cases generated through model-based testing approaches." In Proceedings of IEEE 37th International Conference on Software Engineering, Vol 2, pp.879-882, 2015.

[21] B. Jiang, Z. Zhang, W. K. Chan, and T. H. Tse, “Adaptive random test case prioritization,” in ASE. IEEE Computer Society, pp. 233– 244, 2009.

References[22]Klein, Casey, Matthew Flatt, and Robert Bruce Findler. "Random Testing for Higher-order, Stateful Programs." ACM

Sigplan Notices. Vol. 45. No. 10.,pp.555-566,2010.[23] Arcuri, Andrea, Gordon Fraser, and Juan Pablo Galeotti. "Automated Unit Test Generation for Classes with

Environment Dependencies." Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering, pp. 79-90, 2014.

[24] Islam, Mainul, and Christoph Csallner. "Generating test cases for programs that are coded against interfaces and annotations." ACM Transactions on Software Engineering and Methodology (TOSEM) Issue No. 3, Vol No. 23, pp.21,2014.


[26] Gupta, Nirmal Kumar, and Mukesh Kumar Rohil. "Improving GA based automated test data generation technique for object oriented software." IEEE 3rd International Conference In Advance Computing (IACC), pp. 249-253, 2013.



[29] He, Wei, Ruilian Zhao, and Qunxiong Zhu. "Integrating Evolutionary Testing with Reinforcement Learning for Automated Test Generation of Object-Oriented Software." Chinese Journal of Electronics, Issue 1 Vol No 24, 38-45, 2015.

[30] Prasetya, I. S. "T3i: A tool for generating and querying test suites for java." ACM, In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, pp. 950-953,2015.

[31] N. Tillmann and J. Halleux, “Pexwhite box test generation for .net,” in Tests and Proofs, ser. Lecture Notes in Computer Science, B. Beckert and R. Hhnle, Eds. Springer Berlin Heidelberg, vol.0 4966, pp. 134–153,2010.

[32] G. Fraser and A. Arcuri. EvoSuite: Automatic test suite generation for object-oriented software. In ACM Symposium on the Foundations of Software Engineering (FSE), pages 416–419, 2011

[33]G. Fraser and A. Arcuri. Whole test suite generation. IEEE Transactions on Software Engineering, 39(2):pp276–291, 2013.

References[34] H. Watanabe, H. Tokuoka, W. Wu, M. Saeki, "A Technique for Analyzing and Testing Object-Oriented Software Using

Colored Petri Nets," apsec,, Fifth Asia-Pacific Software Engineering Conference (APSEC'98), pp.182, 1998[35] Prasetya, I. S. "T3i: A tool for generating and querying test suites for java." In Proceedings of ACM 10th Joint Meeting

on Foundations of Software Engineering, pp. 950-953, 2015.[36] C. Pacheco and M. D. Ernst., “Randoop: feedback-directed random testing for Java. GenRed: A Tool for Generating

and Reducing Object-Oriented Test Cases” In Object Oriented Programming, Systems, Languages and Applications (OOPSLA), pp100-104, 2007.

[37]Marchetto, Alessandro, Mahfuzul Islam, Waseem Asghar, Angelo Susi, and Giuseppe Scanniello. "A Multi Objective Technique to Prioritize Test Cases.“ In IEEE Transactions On Software Engineering, VOL. 42, NO. 10, pp. 918-940, 2016.

[38]Zhang, Xiao-Yi, Dave Towey, Tsong Yueh Chen, Zheng Zheng, and Kai-Yuan Cai. "Using Partition Information to Prioritize Test Cases for Fault Localization." In IEEE 39th Annual Computer Software and Applications Conference (COMPSAC), Vol. 2, pp. 121-126. IEEE, 2015.

[39] Nakornburi, Sompong, and Taratip Suwannasart. "A tool for constrained pairwise test case generation using statistical user profile based prioritization.“, IEEE 13th International Joint Conference on Computer Science and Software Engineering (JCSSE), pp.1-6, 2016.

[40] Rountev, Atanas, Ana Milanova, and Barbara G. Ryder. "Fragment class analysis for testing of polymorphism in Java software." IEEE Transactions on Software Engineering, Issue No. 6, Vol. 30,pp. 372-387, 2004.

[41] Nadeem, Aamer, Zafar I. Malik, and Michael R. Lyu., "An Automated Approach to Inheritance and Polymorphic Testing using a VDM++ Specification."IEEE International Multi topic Conference, pp. 224-230, 2006.

[42] Augsornsri, Pachawan, and Taratip Suwannasart. "Design of a tool for checking integration testing coverage of object-oriented software." In IEEE 2013 International Conference on Information Science and Applications (ICISA), pp. 1-4., 2013.

[43]Khan, Shaukat Ali, and Aamer Nadeem. "Automated Test Data Generation for Coupling Based Integration Testing of Object Oriented Programs Using Evolutionary Approaches.”, IEEE Tenth International Conference on Information Technology: New Generations (ITNG), pp. 369-374. 2013

[44] Louzaoui, Khadija, Khalid Benlhachmi, and Jihane Alami Chentoufi. "Conformity testing by optimal constraints for object oriented programs." I4th IEEE International Colloquium on Information Science and Technology (CiSt), , pp. 188-193, 2016.

References[45] Bashir, Muhammad Bilal, and Aamer Nadeem. "A fitness function for evolutionary mutation testing of object-oriented

programs." In IEEE 9th International Conference on Emerging Technologies (ICET), pp. 1-6, 2013

[46]Kumar, K. S., A. Lallu, and Sheena Mathew, "An Efficient Approach for Distributed Regression Testing of Object Oriented Programs." ACM In Proceedings of the International Conference on Interdisciplinary Advances in Applied Computing, p. 33, 2014.

[47] Boyapati, Chandrasekhar, Sarfraz Khurshid, and Darko Marinov. "Korat: Automated testing based on Java predicates." In ACM SIGSOFT Software Engineering Notes, vol. 27, no. 4, pp. 123-133., 2002.

[48]Zhai, Ke, Bo Jiang, and W. K. Chan. "Prioritizing test cases for regression testing of location-based services: Metrics, techniques, and case study." IEEE Transactions on Services Computing, Vol no. 1 pp. 54-67, 2014.

[49]Fazlalizadeh, Y., A. Khalilian, M. Abdollahi Azgomi, and S. Parsa. "Prioritizing test cases for resource constraint environments using historical test case performance data." In 2nd IEEE International Conference on Computer Science and Information Technology,(ICCSIT ), pp. 190-195. IEEE, 2009.