Feedback-Directed Random Test Generation

Automatic Testing & ValidationCSI5118

By Wan Bo

Introduction

• Random Testing VS System Testing

– No certain conclusion which is better

– Advantages of random testing for OO

– Normal random test generation is unguided.

• Purpose of this work:

– random generation of unit tests for object-oriented programs.

– a sequence of method calls for unit test.

– random selecting or previous computed object as parameters for each method call.

• Benefits of guided generation:– only sensible to build a legal sequence of method

calls– Speed up generate process and reduce redundant

sequences.– Example:• Double a = sqrt(-1);• Double b = log(a);• With feedback, such call sequence will not be

extended.

• RANDOOP- random tester for object-oriented programs.

– A software developed by this group– It can generate random test cases (more details

later)– Example:

Methodology

• Method Sequence

– a method sequence, or simple sequence, is a sequence of method calls

– Each call named S.i which means the value returned by the ith previous method call in sequence S

• Extending Sequence– extension operation: takes zero or more

sequences and produces a new sequence.– operator extend(m, seqs, vals)• m: is a method with formal parameters of type T1; : : : ;

Tk.• seqs: is a list of sequences.• vals: is a primitive values v1 : T1; : : : ; vk : Tk, or it is the

return value s:i of the i-th method call for a sequence s appearing in seqs.

• Example:

• Feedback-directed Generation– build sequences incrementally– as soon as a sequence is built, execute to ensure

non-redundant and legal objects (filters & contracts)

– GenerateSequences(classes, contracts, filters, timeLimit)• classes: a list of classes for which to create sequences• contracts: a list of contracts• filters: a list of filters• timeLimit: a time limit to stop generation process

• Generate Sequence

– Step1 Create new Sequence– Step2 Discard duplication– Step3 Execute new sequence– Step4 Check contracts– Step 5 added in test sequence set.

• Boolean vector– a boolean flag s:i.extensible. (whither it can be

used as input in new method call)– Be used in filters (explain later)

• m(T1 ...Tk) at line 5

– Use reflection to load dll (being tested), store all method infos and classes infos into collection. Select randomly.

– RandomMethodOrConstructorUniform(),return a reflecton.runtimemethodinfo

• randomSeqsAndVals(T1...Tk)– Types is from methodinfo– The length of sequence is decided by limited time – When a new method call created, put in plan,

when time out, write in cs file.• Parameters– Primitive type, select a primitive value form fixed

pool of values– Reference type • Use one already exist in sequence (in plan).• New one• Null

• Primitive Value Pool:

• Extend Sequence• The method is explained in previous slide

• Discard duplicates– Check whither equivalent sequence was already

created in previous– Non error sequence set– Error sequence set

• Execute sequence– Execute calls in sequence one by one. And check

contracts after each all.– If conflict with contract, set violated be true

• Default Contracts

• Filtering– setExtensibleFlags (newSeq; flters; ~o) line19

• Function: determine which values of a sequence are extensible and should be used as inputs to new method calls

– Equality.• uses the equals() method to determine if the resulting

object has been created before.

– Null.• set newSeq:i.extensible to false iff the corresponding

object is null. Such as void method

– Exceptions.• If exception occurs, even it does not conflict with

contract, still set extendable false

Evaluation

JFP RP JPFU RPU0

0.2

0.4

0.6

0.8

1

1.2

BinTreeBHeapFibHeapTreeMap

Block Coverage

Block Coverage Running Time

JFP RP JFPu Rpu0

5

10

15

20

25

BinTreeBheapFibHeapTreeMap

• Test common libs – REDUCE similar with Junit and Nunit– Used feedback-RANDOOP

• Open Source:• For Java:– http://code.google.com/p/randoop/

• For .Net:– http://randoop.codeplex.com/

• RUNDOOP2 for C# published 09/08/2010

• Main structure:

• Randoop– Load target dll by reflection, analysis and generate

config file (xml)• RandoopBare– According to config file, generate test cases. – Staticmanager calculates figures.– Generate log

• Example 1– Test target .Net System.Data.dll– Right now just can test dlls in C:\WINDOWS\

Microsoft.NET\Framework\v2.0.50727– Randoop needs use csc.exe to compile test cases

• Test Case:

• My test cases:• Lib- RandoopTestLib– Addition– Subtract– Arithmetic

• Subtract

• Put dll into C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727

• For Int: -10 -3 -2 -1 0 1 2 3 10

• For Long: -10 -3 -2 -1 0 1 2 3 10

• For String and Object

I can not find it.

• The test case