Test Coverage with SOTAMichael Hildebrandt

HUMBOLDT UNIVERSITY BERLINDEPARTMENT OF COMPUTER SCIENCECHAIR OF SOFTWARE ENGINEERING

DAAD Workshop Neum, 2009 2

motivation◦ testing is expensive

purposes of SOTA◦static program analysis◦structure oriented program test

approach◦source code parsing◦ instrumenting

workflow◦source code preparation◦ test execution◦evaluation

agenda

DAAD Workshop Neum, 2009 3

given: function overlaps◦parameter:

(time) interval A (startA, endA) (time) interval B (startB, endB)

◦ result: true if A and B overlap, otherwise false

How many test cases exist?

motivationSoftware Engineering II, Prof. Dr. Holger Schlingloff

DAAD Workshop Neum, 2009 4

1. test cases for overlapping intervals2. test cases for non-overlapping intervals3. test cases for A is before B and vice versa4. test cases for A B and B A5. test cases for the same starting/ending point

of A and B6. test case for A = B

motivation (cont.)

DAAD Workshop Neum, 2009 5

7. test cases for intervals consisting of one point8. test cases for identical intervals consisting of

point9. test cases for intervals containing integer /

real numbers10. test cases for invalid intervals11. test cases for intervals with values at the

borders of number range (maxint, maxreal)

motivation (cont.)

DAAD Workshop Neum, 2009 6

about 20 test cases for this simple example imagine a project with 1.5 MLOC

conclusion:testing every possible input is expensive or even impossible

problem:Which subset of the possible test cases has the highest probability to detect the most possible failures?

possible solution:structure oriented tests (using SOTA)

motivation (cont.)

DAAD Workshop Neum, 2009 7

Structure Oriented Test and Analysis

static program analysis◦metrics ◦control flow graph (CFG)

structure oriented tests◦evaluation of test cases by code coverage◦development of additional test cases

SOTA

author: Ronny Treyße(diploma thesis)

DAAD Workshop Neum, 2009 8

calculated by parsing the source code provides several metrics◦lines of code◦number of statements, branches, atoms, conditions, …◦cyclomatic complexity◦essential complexity

static program analysis: metrics

DAAD Workshop Neum, 2009 9

static program analysis: CFG public static int fac(int n) { int result = 1;

if (n == 0 | n == 1) {

return result;

} else {

for (int i = n; i >= 2; i--) { result *= i; } return result; }}

DAAD Workshop Neum, 2009 10

calculated by instrumenting and test execution

structure oriented tests

FEECC0C1

function entries and exits coveragestatement coveragebranch coverage

C2MMCCMCDCC3

simple condition coverageminimal multiple condition coveragemodified condition decision coveragemultiple condition coverage

MBIBI

modified boundary interior coverageboundary interior coverage

DAAD Workshop Neum, 2009 12

instrumentation◦enrichment of the original source code by adding logging statements breaking up conditions, transforming loops, …

◦source code grows bigger (up to ten times)

logging information during test execution◦values of condition, atoms◦used function entries / exits and branches of control

structures (reconstruction of the control flow)

approach

DAAD Workshop Neum, 2009 14

level 0◦ no instrumentation◦ to mark structures for no instrumentation

level 1◦ all entries and exists of a method, branching control structures logged◦ all coverage measures based on the control flow computable

level 2◦ additional to level 1 the assignment of every atom is logged◦ all coverage measures computable

level 3◦ complete instrumentation: additional to level 2 the execution of every

statement is logged◦ allows control flow analysis of abnormal terminating programs

instrumentation: levels

DAAD Workshop Neum, 2009 15

instrumentation (cont.)if (n == 0 | n == 1) { return result;}

DAAD Workshop Neum, 2009 16

instrumentation (cont.)

for (int i = n; i >= 2; i--) { result *= i;}

DAAD Workshop Neum, 2009 17

3 modes of usage◦manual program testing

pre procession: SOTA test execution: by hand post procession: SOTA

◦ testing with an external test system (e.g. ATOSj) pre procession: SOTA test execution: external test system post procession: SOTA

◦ testing with an automatic test system usage of non-gui functionality by SOTA library accessible via ASTManager object or command line

modes of usage

DAAD Workshop Neum, 2009 18

phase 1: preparation◦ read in and parsing the source code◦configure the instrumentation◦ instrumentation of source code

phase 2: test execution◦compiling of the instrumented source code◦ test execution including the logging of information

phase 3: test evaluation◦ restoration of the original code◦ read In the log files◦calculation of the coverage measure◦visualization of the results◦creation of report

workflow

DAAD Workshop Neum, 2009 19

Questions ? Comments?

Thank you.