Test Blueprints

Test Blueprints

Adrian Lienhard, Tudor Gîrba,Orla Greevy and Oscar Nierstrasz

Software Composition GroupUniversity of Bern, Switzerland

Thanks for the support:

Exposing Side Effects in Execution Traces to Suport Writing Unit Tests

CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : [email protected]

Unit Testing

...unit tests are important!

...write tests first!

Everybody knows


Unit Testing

...unit tests are important!

...write tests first!

Fair enough, but what if ...the code exists already?...and you don’t understand it well?

Everybody knows


Unit Testing

Yet, writing unit tests for legacy systems is important.

“Tests are your life insurance”


Challenges of Testing Legacy Code

1. Selecting a unit to test choose unit of appropriate complexity

2. Creating a fixture instantiate and set up graph of objects

3. Executing the unit under test stimulate the fixture

4. Verifying the expected behavior assert side effects and return value


Idea

Use example runs of the program to extract test scenarios.

Perform a dynamic analysis to extract a test blueprint that shows:

- how to create a fixture for a selected unit- how to execute the unit- how to verify the expected behavior


Program InstrumentationExecution of existing tests

Analysis

object flow +execution trace data

coverage data

Example program execution

Execution Trace Test Blueprint

Tool


Test Blueprint

ASTChecker>>declareVariableNode:

FunctionScope>>addTemp:

TempVar class>>new

...initialization...

TempVar>>name:

TempVar>>scope:

KeyedSet>>add:

...library code...

...

...E

xe

cu

tion

Un

it

Test BlueprintExecution Trace


Test Blueprint

Legend

existing reference

new reference (side effect)

:Class existing instance

:Class new instance


Creating the Fixture

Legend

existing reference



:Class new instance

1 fscope := FunctionScope new.

2 varname := ’x’.

Unit Test for addTemp: method


Executing the Unit under Test

Legend

existing reference



:Class new instance



3 var := fscope addTemp: name.



Verifying Expected Behavior

Legend

existing reference



:Class new instance



3 var := fscope addTemp: name.

4 self assert: var class = TempVar.

5 self assert: var name = varname.

6 self assert: var scope = fscope.

7 self assert: (

fscope tempVars includes: var).



Blueprint Examples

InstanceScope>>newMethodScope

IRBuilder>>add: FunctionScope>>lookupVar:

NonClosureScopeFixer>>acceptVarNode:

Program instrumentation

Execution of existing tests

Analysis (Section 6)

object flow +execution trace data

coverage data

Example program execution

Execution Trace Test Blueprint


Implementation

Aliases are created when an object is:

‣ instantiated

‣stored into or read from a field

‣stored into or read from an array

‣passed as argument/return value

dynamic model static model

Alias

Instance

*

0..1

*0..1

Method* 1

Class1*

*

1

creator

Activation

parent

receiver

Execution-

Unit

root

children

createdAliases

* 1

activations

*

1

Object Flow meta-model

The flow of an object in the system is tracked by the parent-child relationship of its Aliases


Implementation

Imported references = used state (->fixture)Exported references = side effects (->assertions)

Importedreferences

Exportedreferences

Execution Unit

object_1object_2object_3

t

w r

w r w

w

r


Initial Case Study I

- One developer produced 12 tests in 2h- Average fixture size: 5- Average side effects: 4

How straightforward is it to use the proposed tool? Target system: Insurance broker application (520 classes, 6 years old)

- Test Blueprint worked surprisingly well ...yet, sporadically resorted to consulting source code

- Insufficient support for selecting execution units

Observations


Initial Case Study II

- 72 identical- 5 additional- 12 missed ==>

How do tests written using our approach differ compared to conventional tests written by an expert? Content Management System (377 classes)

Study: selected 14 unit tests, removed all 84 assertions and then systematically rewrote them using our tool

verification that special objects are left unmodified

our approach does not detect unmodified objects that are worthwhile to verify!

Resulting assertions


Conclusions

Yes, we can write unit test for not well known code...by exploiting runtime information

Test Blueprints tell us...how the fixture and assertions have to look like

Difficulties are how to...select good execution units in the trace...initialize objects to bring them into the desired state


Conclusions

Yes, we can write unit test for not well known code...by exploiting runtime information

Test Blueprints tell us...how the fixture and assertions have to look like

Difficulties are how to...select good execution units in the trace...initialize objects to bring them into the desired state

Questions?

Technology

Test Blueprints