Firewall for regression testing Tor Stålhane According to White, Jaber, Robinson and Rajlich

Firewall for regression testing

Tor StålhaneAccording to

White, Jaber, Robinson and Rajlich

Why fire walls in regression testingMost regression tests are large and time-consuming to run. The concept of a “fire wall” is used to reduce the set of classes – or components – that need to be tested. A fire wall in regression testing separates the classes that depend on the class that is changed from the rest. There are two central concepts – dependency and encoding

Simple fire wall rules Rules for the fire wall in object-oriented systems:1.Given two successive versions of an OO-system, identify the classes that have changed2.If a changed class is part of an inheritance hierarchy we must also consider descendants of the changed class as changed3.For each changed class, identify all classes that send a message to the changed class or receive messages from a changed class and include them inside the fire wall

Simple fire wall example

The extended fire wall The extended fire wall is identified by the three rules for constructing a simple fire wall plus4.Identify all data paths to and from a modified class5.Find all external classes in scope of the modified class and include them in the extended fire wallIn order to identify an external class, we need some definitions.

Dependency The main concept to understand when we apply the fire wall idea, is the idea of dependency. The main question is:Is the component dealing with the plain values of the inputorMust the component account for how those inputs were generated

Definitions – 1

Let C be a component with inputs {Ii}, outputs {Oj} and post-conditions {Qk}

If Q {Qk} and I {Ii} so that Q depends on I and on a value generated by a component C1 that is connected to I through a data flow through one or more intermediary component C2,…Cm then

•I is an encoded input•C is an external component in scope of C2 and visa versa

Definitions – 2 No component C1 => I is a plain input, i.e. not encodedAll inputs of C are plain => C is an I/O component C is external in scope of D and there is no message sent from C to D => there is a hidden dependency between C and D.

c c1 cc2c1

I I

I is a plain inputI is an encode input

cm

…

Dependency example – 1 Component 1 generates the key k, which is converted to an integer in component 2. Component 3 converts the integer to a hash address A. Component 4 uses the hash address to do a table look-up to generate the value f(k).

Dependency example – 1

The post condition of component 4 depends on k, generated by component 1 => input A is encoded by component 2.Component 4 is an external in scope of component 2 and visa versa. r = f(k) but k is changed by component 2

Hidden dependency

A’s post-condition is dependent on the value “colour” that is •Generated in class I•Encoded as an integer in class A•Decoded in class B Thus class B is external in scope of A. The dependency between A and B is hidden since no message is sent between A and B

Global variables

Here, the output v3 from component 3 is used to modify the global variable g which is used as input to component 4.

Component 4 is external and dependent on variable v1 and is activated by a message from component 3 to component 4.

Extended fire wall – example

Simpel and Extended fire wall – 1 In order to test the concepts of Simple and Extended Fire wall, the authors tested several builds of a TelComm system. This system had already been tested by the developers using their standard test suite. •Common – found using both TFW and EFW: 8 errors•Found only using EFW: 4 errors. Two of these had not been found by the company’s test suite.

Simpel and Extended fire wall – 2

Simpel and Extended fire wall – 3 In the following table, total time for testing is the sum of •Analysis time – identifying the fire wall•Testing time is time needed for

– test setup– test execution– analysis of the test results.

EFW testing requires approximately•20 – 30% more tests•40 – 50% more person hours

Simpel and Extended fire wall – 4

• TFW: 46 person hours, 8 errors => 5.7 person hours per error found

• EFW: 58 person hours, 12 errors => 4.8 person hours per error found

Simpel and Extended fire wall – 5

For TFW some testes were reused from the company’s original test suiteAll extra tests needed for EFW were written from scratch.

Simpel and Extended fire wall – 6

New system – 13 common errors and 3 errors found only by EFW

Simpel and Extended fire wall – 7

• TFW: 93 person hours, 13 errors => 7.2 person hours per error found

• EFW: 120 person hours, 16 errors => 7.5 person hours per error found

Conclusion • Extended Fire Wall testing (EFW) finds more

errors that Simple Fire Wall (FWT) testing• The cost in person hours per error found is

approximately the same – 5 to 8 person hours all in all per error

• Both EFW and FWT – should only be used to test updates and changes– assumes that the initial test suite and the tested

software are of a high quality.