T.Y. Chen Swinburne University of Technology, Australia

1

T.Y. Chen

Swinburne University of Technology, Australia

T.H. Tse and Zhiquan Zhou

The University of Hong Kong

Semi-Proving: an Integrated Method Based on Global Symbolic Evaluation and Metamorphic Testing

(speaker)

2

Presentation Outline

Conventional Program Testing and Proving Metamorphic Testing Our method: Semi-Proving Summary.

3

Conventional Program Testing and Proving Metamorphic Testing Our method: Semi-Proving Summary.


4

Conventional Program Testing and Proving

Given a bijective function f ;

A Program: F_Sort (a1, a2, ..., an), n 2

Output: (a1’, a2’, ..., an’), such that

1. (a1’, a2’, ..., an’) is a permutation of (a1, a2, ..., an)

2. f (a1’) f (a2’) ... f (an’).

Given a bijective function f ;

A Program: F_Sort (a1, a2, ..., an), n 2

Output: (a1’, a2’, ..., an’), such that

1. (a1’, a2’, ..., an’) is a permutation of (a1, a2, ..., an)

2. f (a1’) f (a2’) ... f (an’).

5


Testing

1. Design test cases: e.g. (2, 6, 3) for n=3

2. Run: F_Sort (2, 6, 3) = (6, 3, 2)

3. Check: f (6) < f (3) < f (2) ?

1. Design test cases: e.g. (2, 6, 3) for n=3

2. Run: F_Sort (2, 6, 3) = (6, 3, 2)

3. Check: f (6) < f (3) < f (2) ?

6


Proving correctness

1. F_Sort terminates for any valid input;

2. The output is correct.

1. F_Sort terminates for any valid input;

2. The output is correct.

7


Proving properties

F_Sort (a1, a2, ..., an) = (a1’, a2’, ..., an’) F_Sort (a1, a2, ..., an) = (a1’, a2’, ..., an’)

Permutation.

8

Metamorphic Testing

Metamorphic Testing

Employing relationships between different executions

Fact: different permutations will produce same output

F_Sort (a1, a2, a3)

Fact: different permutations will produce same output

F_Sort (a1, a2, a3) F_Sort (a3, a1, a2) = “ Metamorphic Relation ” ·

9

Metamorphic Testing

Metamorphic Test Cases: {(2, 6, 3), (3, 2, 6)}Metamorphic Test Cases: {(2, 6, 3), (3, 2, 6)}

Metamorphic Testing:

1. F_Sort (2, 6, 3) = (6, 3, 2)

Metamorphic Testing:

1. F_Sort (2, 6, 3) = (6, 3, 2)

No matter whether an oracle is available or not;Very useful when the oracle cannot be found.

2. F_Sort (3, 2, 6) = (6, 3, 2)| |

PASS

10

Metamorphic Testing

Metamorphic Test Cases: {(2, 6, 3), (3, 2, 6)}Metamorphic Test Cases: {(2, 6, 3), (3, 2, 6)}

Metamorphic Testing: 1. F_Sort (2, 6, 3) = (6, 3, 2)

Metamorphic Testing: 1. F_Sort (2, 6, 3) = (6, 3, 2)

2. F_Sort (3, 2, 6) = (3, 6, 2) Failure.| |

11

Conventional Program Testing and Proving Metamorphic Testing Semi-Proving: Verifying Metamorphic

Relations Summary.


12

Semi-Proving: Verifying Metamorphic Relations

Objective:

If the program does not satisfy a metamorphic relation on some inputs, locate these inputs;

Otherwise prove the satisfaction of the metamorphic relation over all inputs.

13

Why called “Semi”?

Proving necessary properties, which may not be sufficient for program correctness

Characteristics of Semi-Proving

Multiple symbolic executions

Testing and proving.


14

double GetMid (double x1, double x2, double x3){ double mid;

mid = x3;if (x2 < x3)

if (x1 < x2)mid = x2;

else {if (x1 < x3)

mid = x1;}

elseif (x1 > x2)

mid = x2;else if (x1 > x3)

mid = x1; return mid;

}


mid = x3;if (x2 < x3)

if (x1 < x2)mid = x2;

else {if (x1 < x3)

mid = x1;}

elseif (x1 > x2)



}


15

Specification

“GetMid (X, Y, Z)” returns the median of (X, Y, Z)

E.g. GetMid (3, 4, 1): “3”.


16

Verifying “GetMid” by Semi-Proving

Identify a Metamorphic Relation

GetMid ( X, Y, Z ) = GetMid ( permute(X, Y, Z) )


any numbers any permutation

Purpose: to verify

17

Basic concepts

Transposition

• simple permutation that exchanges two elements

(1, 2, 3)

(1, 2, 3)

......... 1

(1, 2, 3) (1, 3, 2) ......... 2

(2, 1, 3)


18

A tuple (1, 2, 3)

A permutation (2, 3, 1)

(1, 2, 3)

A tuple (1, 2, 3)

A permutation (2, 3, 1)

(1, 2, 3) (2, 3, 1)1 (2, 1, 3) 2

Basic concepts

Composition of Transpositions


19

Result from Group Theory

Any permutation of (X, Y, Z) can be achieved by compositions of transpositions (X, Z, Y) and (

Y, X, Z).


20


Purpose


Only need to verify:

Any permutation.

• GetMid (X, Y, Z) = GetMid (X, Z, Y)

• GetMid (X, Y, Z) = GetMid (Y, X, Z)

21


Purpose


Only need to verify:

• GetMid (X, Y, Z) = GetMid (X, Z, Y)

• GetMid (X, Y, Z) = GetMid (Y, X, Z)

22

Global Symbolic Evaluation on GetMid (X, Y, Z)

Execute all the possible paths.


23


mid = x3;if (x2 < x3)

if (x1 < x2)mid = x2;

else {if (x1 < x3)

mid = x1;}

elseif (x1 > x2)



}


mid = x3;if (x2 < x3)

if (x1 < x2)mid = x2;

else {if (x1 < x3)

mid = x1;}

elseif (x1 > x2)



}


24

C1: (Y X < Z) OR (Z < X Y)

Path Conditions C2: (X < Y < Z) OR (Z Y < X)

C3: (Y < Z X) OR (X Z Y)


X when C1 is trueGetMid (X, Y, Z) = Y when C2 is true

Z when C3 is true

25


?GetMid (X, Z, Y)

?X when C1 is true

GetMid (X, Y, Z) = Y when C2 is trueZ when C3 is true

26

C4: (Z X < Y) OR (Y < X Z)

C5: (X < Z < Y) OR (Y Z < X)

C6: (Z < Y X) OR (X Y Z)

PASS


?GetMid (X, Z, Y)

?X when C4 is true

= Z when C5 is true

Y when C6 is true


Z when C3 is true

27

? ?


Z when C3 is true

X when C4 is true

= Z when C5 is true

Y when C6 is true

Contradiction

PASS

C1: (Y X < Z) OR (Z < X Y) &


GetMid (X, Z, Y)?

C4: (Z X < Y) OR (Y < X Z)

C5: (X < Z < Y) OR (Y Z < X)

C6: (Z < Y X) OR (X Y Z)

28

? ?

C4: (Z X < Y) OR (Y < X Z)

C5: (X < Z < Y) OR (Y Z < X)

C6: (Z < Y X) OR (X Y Z)


Z when C3 is true

X when C4 is true

= Z when C5 is true

Y when C6 is true

C1: (Y <= X < Z) OR (Z < X <= Y) &

X=Y<Z OR Z<Y=X


?GetMid (X, Z, Y)

29

? ? ?

C4: (Z X < Y) OR (Y < X Z)

C5: (X < Z < Y) OR (Y Z < X)

C6: (Z < Y X) OR (X Y Z)


Z when C3 is true

X when C4 is true

= Z when C5 is true

Y when C6 is true

C1: (Y <= X < Z) OR (Z < X <= Y) &

Yes. X=Y

PASS

X=Y<Z OR Z<Y=X


GetMid (X, Z, Y)

30

?


Z when C3 is true


GetMid (X, Z, Y)

verified

31

?


Z when C3 is true


Conclusion

GetMid (X, Z, Y)

32

?


Z when C3 is true


Conclusion

GetMid (X, Z, Y)

33


Z when C3 is true


Conclusion

GetMid (X, Z, Y)

34


Z when C3 is true


Conclusion

GetMid (X, Z, Y)

Composition of transpositions

GetMid (X, Y, Z) = GetMid ( Permute(X, Y, Z) )

GetMid (Y, X, Z)

Any Any.

35

Detecting Program Faults ·

Semi-Proving: Detecting Program Faults

36


mid = x3;if (x2 < x3)

if (x1 < x2)mid = x2;

else {if (x1 < x3)

mid = x1;}

elseif (x1 > x2)



}


mid = x3;if (x2 < x3)

if (x1 < x2)mid = x2;

else {if (x1 < x3)

mid = x1;}

elseif (x1 > x2)



}

37

Verify: GetMid (X, Y, Z) = GetMid (X, Z, Y)


| |

X

when Y X < Z

?| |

Y

when (Z < Y X ) OR (Y Z AND X Z) AND

38



| |

X

when Y X < Z

?| |

Y


(Y=X<Z) OR (Y<X<Z)

39



| |

X

when Y X < Z

?| |

Y


(Y=X<Z) OR (Y<X<Z)

?

failure

Failure-causing inputCan identify all the

failure-causing inputs.

40

Summary A proving technique: all the paths A testing technique:

failure-causing inputs selected path(s)

Characteristics Metamorphic relations Multiple symbolic executions Employing global symbolic evaluation and constraint

solving.

41

Questions are welcome

Documents

T.Y. Chen Swinburne University of Technology, Australia