AUTOMATED UNIT TEST GENERATION DURING SOFTWARE...

AUTOMATED UNIT TEST GENERATION DURING

SOFTWARE DEVELOPMENT

José Miguel Rojas j.rojas@sheffield.ac.uk

Joint work with Gordon Fraser and Andrea Arcuri

A Controlled Experiment and Think-aloud Observations

ISSTA 2015

“Testing is a widespread validation approach in industry, but it is still largely ad hoc,

expensive, and unpredictably effective.”“Software Testing Research: Achievements, Challenges, Dreams,”

A. Bertolino. Future of Software Engineering. IEEE . 2007.

“Testing is a widespread validation approach in industry, but it is still largely ad hoc,

expensive, and unpredictably effective.”

“Test case generation has a strong impact on the effectiveness and efficiency of testing.”

“…one of the most active research topics in software testing for several decades, resulting in many different approaches and tools.”

“Software Testing Research: Achievements, Challenges, Dreams,” A. Bertolino. Future of Software Engineering. IEEE . 2007.

”An orchestrated survey of methodologies for automated software test case generation,” S. Anand, E. K. Burke, T. Y. Chen, J. Clark, M.B. Cohen, W. Grieskamp, M.

Harman, M.J. Harrold, P. McMinn. J. Systems and Software. Elsevier. 2013.

BACK IN ISSTA 2013…

“Does automated white-box test generation really help software testers?,” G. Fraser, M. Staats, P. McMinn, A. Arcuri and F. Padberg

BACK IN ISSTA 2013…

“Does automated white-box test generation really help software testers?,” G. Fraser, M. Staats, P. McMinn, A. Arcuri and F. Padberg

ARE UNIT TEST GENERATION TOOLS HELPFUL TO DEVELOPERS WHILE THEY ARE CODING?

CODE COVERAGE

TIME SPENT ONTESTING

CODE COVERAGE

TIME SPENT ONTESTING

IMPLEMENTATION QUALITY

CONTROLLED EXPERIMENT

Golden Implementation and Test Suite

Class TemplateGolden Implementation and Test Suite

Class Template Implementation and Test Suite

Manual

1 hourClass Template Implementation and Test Suite

Manual

DOES USING EVOSUITE DURING SOFTWARE DEVELOPMENT LEAD TO TEST SUITES WITH HIGHER CODE COVERAGE?

FilterIterator FixedOrderComparator ListPopulation PredicatedMap

39% 41%

Assisted Manual

CODE COVERAGEparticipants’ test suites run on their own implementations

CODE COVERAGETi

Category AxisFilterIterator FixedOrderComparator ListPopulation PredicatedMap

Assisted Manual

Times coverage was checked

ListPopulation

Time (min)0 10 20 30 40 50 60

ManualEvoSuiteAssisted

CODE COVERAGEparticipant’s test suites run on their own implementations

ListPopulation

Time (min)0 10 20 30 40 50 60

ListPopulation

Time (min)0 10 20 30 40 50 60

ListPopulation

Time (min)0 10 20 30 40 50 60

28%21%

42%37%35%

Assisted Manual

CODE COVERAGEparticipants’ test suites run on golden implementations

FilterIterator

0%25%50%75%

Time (min)0 10 20 30 40 50 60

Assisted Manual EvoSuite-generated

CODE COVERAGEparticipant’s test suites run on golden implementations, over time

ListPopulation

0%25%50%75%

Time (min)0 10 20 30 40 50 60

FixedOrderComparator

0%25%50%75%

Time (min)0 10 20 30 40 50 60

PredicatedMapBr

0%25%50%75%

Time (min)0 10 20 30 40 50 60

FilterIterator

0%25%50%75%

Time (min)0 10 20 30 40 50 60

Assisted Manual EvoSuite-generated

CODE COVERAGEparticipant’s test suites run on golden implementations, over time

ListPopulation

0%25%50%75%

Time (min)0 10 20 30 40 50 60

FixedOrderComparator

0%25%50%75%

Time (min)0 10 20 30 40 50 60

PredicatedMapBr

0%25%50%75%

Time (min)0 10 20 30 40 50 60

Coverage can be higher when using EvoSuite, depending on how the generated tests are used.

DOES USING EVOSUITE DURING SOFTWARE DEVELOPMENT LEAD TO

DEVELOPERS SPENDING MORE OR LESS TIME ON TESTING?

TESTING EFFORT

12.913.7

8.27.27.8

Assisted Manual

Number of test runs

TESTING EFFORT

Assisted Manual

Minutes spent on testing

TESTING EFFORT

Assisted Manual

Minutes spent on testing

Using EvoSuite reduces the time spent on testing.

DOES USING EVOSUITE DURING SOFTWARE DEVELOPMENT LEAD TO

SOFTWARE WITH FEWER BUGS?

IMPLEMENTATION QUALITYGolden test suites run on participants’ implementations

f Fail

5.34.3

4.26.1

Assisted Manual

IMPLEMENTATION QUALITYGolden test suites run on participants’ implementations

f Fail

5.34.3

4.26.1

Assisted Manual

Using EvoSuite during development did not lead to to better implementations.

DOES SPENDING MORE TIME WITH EVOSUITE AND ITS TESTS LEAD TO

BETTER IMPLEMENTATIONS?

PRODUCTIVITYTime spent with EvoSuite

elatio

f fail

-0.490.02

-0.22-0.29

0.350.32

Number of runsTime spent on tests

PRODUCTIVITYTime spent with EvoSuite

elatio

f fail

-0.490.02

-0.22-0.29

0.350.32

Number of runsTime spent on tests

Implementation quality improves the more timedevelopers spend with EvoSuite-generated tests.

Using automated unit test generation does impact developers’ productivity,

but…

…how to make the most outof unit test generation tools?

Using automated unit test generation does impact developers’ productivity,

but…

THINK ALOUD OBSERVATIONS

J. Hughes and S. Parkes, “Trends in the use of verbal protocol analysis in software engineering research,” Behaviour and Information Technology, vol. 22, no. 2, pp. 127–140, 2003.

K. A. Ericsson and H. A. Simon, Protocol Analysis: Verbal Reports as Data (revised edition). MIT Press, 1993.

Subject

Observer

Subject

2 hoursClass Template Implementation and Test Suite

RESULTS

Images: http://jozef89.deviantart.com/

3 6 2 2 5

15 20 23 7 10

94% | 92% 97% | 72% 94% | 95% 100% | 94% 100% | 83%

FixedOrderComparator ListPopulationFilterIterator PredicatedMap PredicatedMap

RESULTS

3 6 2 2 5

15 20 23 7 10

94% 97% 94% | 95% 100% 100%

RESULTS

3 6 2 2 5

15 20 23 7 10

94% 97% | 72% 94% 100% 100%

LESSONS LEARNED

LESSONS LEARNED• There are different approaches to testing and test generation tools

should be adaptable to them

• Developers’ behaviour is often not driven by code coverage

• Readability of generated unit tests is paramount

• Integration into development environments must be improved

• Education/Best practices: Developers do not know how tobest use automated test generation tools!

“Coverage is easy to assess because it isa number, while readability is a very non-

tangible property…

… What is readable to me may not bereadable to you. It is readable to me justbecause I spent the last hour and a half

doing this.”

… What is readable to me may not bereadable to you. It is readable to me justbecause I spent the last hour and a half

doing this.”—Participant 5

j.rojas@sheffield.ac.uk

www.evosuite.org/study-2014/

j.rojas@sheffield.ac.uk

AUTOMATED UNIT TEST GENERATION DURING SOFTWARE...

Documents

Sameer Shende, Allen D. Malony {sameer,malony}@cs.uoregon.edu Computer & Information Science Department Computational Science Institute University of Oregon

Allen D. Malony, Sameer Shende, Alan Morris {malony,sameer,amorris}@cs.uoregon.edu Department of Computer and Information Science Performance Research

S3D: Comparing Performance of XT3+XT4 with XT4 Sameer Shende tau-team@cs.uoregon.edu

TAU Performance Toolkit (WOMPAT 2004 OpenMP Lab) Sameer Shende, Allen D. Malony University of Oregon {sameer, malony}@cs.uoregon.edu

Allen D. Malony, Sameer Shende, Robert Bell malony@cs.uoregon.edu Department of Computer and Information Science Computational Science Institute, NeuroInformatics

Profiling S3D on Cray XT3 using TAU Sameer Shende tau-team@cs.uoregon.edu

Kai Li, Allen D. Malony, Robert Bell, Sameer Shende {likai,malony,bertie,sameer}@cs.uoregon.edu Department of Computer and Information Science Computational

Logical Relations - cs.uoregon.edu

TAU Performance System tau-team@cs.uoregon.edu

Aroon Nataraj, Matthew Sottile, Alan Morris, Allen D. Malony, Sameer Shende { anataraj, matt, amorris, malony, shende}@cs.uoregon.edu

a new approach to software design analysis - MIT CSAILpeople.csail.mit.edu/dnj/talks/issta15/issta-15.pdf · 2015. 7. 17. · “if you change a style’s format, then all paragraphs

a new approach to software design analysisissta2015.cs.uoregon.edu/slides/jackson-keynote-issta15.pdf · rules & searches (Apple Mail) › two ways to specify set of messages labels

Performance Observation Sameer Shende and Allen D. Malony {sameer,malony} @ cs.uoregon.edu

Allen D. Malony, Sameer Shende, Li Li, Kevin Huck {malony,sameer,lili,khuck}@cs.uoregon.edu Department of Computer and Information Science Performance

Allen D. Malony, Kevin Huck {malony,khuck}@cs.uoregon.edu Department of Computer and Information Science Performance

lect1 - cs.uoregon.edu€¦ · Title: lect1.dvi Created Date: 5/17/2011 3:26:52 PM

Scalability Study of S3D using TAU Sameer Shende tau-team@cs.uoregon.edu

Janice E. Cuny, Ph.D., Department of Computer and Information Science, University of Oregon cuny@cs.uoregon.edu Judith B. Cushing, Ph.D., Department of

Performance Tools for GPU-Powered Scalable Heterogeneous Systems Allen D. Malony, Scott Biersdorff, Sameer Shende {malony,scott,sameer}@cs.uoregon.edu

Performance Tools BOF, SC’07 5:30pm – 7pm, Tuesday, A9 Sameer S. Shende sameer@cs.uoregon.edu Performance Research Laboratory University