1ISSTA 2002, Rome, Italy

Investigating the Use of Analysis Contracts to Support Fault Isolation in Object-Oriented Code

Lionel Briand, Yvan Labiche, Hong Sun

Software Quality Engineering Laboratory

http://www.sce.carleton.ca/Squall

Systems and Computer Engineering

Carleton University

Ottawa, Canada

2ISSTA 2002, Rome, Italy

Fault Isolation in OO Programs

• Diagnosis of failure is time-consuming in OO software because of the distribution of functionality and the complexity of methods interactions

• How can Analysis contracts (invariant, pre and post-conditions) be (re)used to instrument the code with assertions so as to help the isolation of faults?

What is an Analysis contract? How can we measure the benefits of using contracts for fault isolation?

3ISSTA 2002, Rome, Italy

Related Work

• [Meyer 92]: Design by Contract

• [Rosenblum 95]: Most effective assertions in C programs

• [Voas 95]: Testing with assertions

– Needs very large numbers of assertions and executions

• [Binder 96]: Suggests the use of Built-in Test support to help testing

• [Baudry 01]: use of contracts to locate faults in classes

– Class level testing

– Unclear definition of contracts

– Unclear measurement of the benefits (diagnosability)

Carefully designed case studies are needed Precise, operational measure for diagnosability

4ISSTA 2002, Rome, Italy

Analysis Contracts

• Contracts defined during Analysis: – Using OCL in the context of UML

– Only for problem domain classes

– Class invariants, method pre and post-conditions

• Elements used in contractsElements Invariant Precondition Post-condition

Attribute X X X

Input parameter X X

Output parameter X

Return value X

Association X X X

5ISSTA 2002, Rome, Italy

Additional Issues

• Oracles are expensive parts of the test drivers. Can they be substituted with assertions instrumenting contracts?

• Can contract assertions help detect additional failures? (I.e., improve observability over oracles alone)

• Why do certain faults remain undetected when using contract assertions?

• What are the issues related to instrumenting contracts in Java systems?

• How does the level of precision of contracts affect the results?

6ISSTA 2002, Rome, Italy

Level of Precision in Analysis Contracts

• Problem: Contract instrumentation has a cost.• Question: Can we simplify the instrumentation?• Constraint: Be systematic.• Three levels of detail in post-conditions:

– Highest (as defined during Analysis)if A1 then B1else if A2 then B2 else B3

– Intermediate (check the condition for the standard situation)if A1 then B1else B2 or B3

– Lowest (ignore the conditions, check only the possible results)B1 or B2 or B3

7ISSTA 2002, Rome, Italy

Diagnosability measure

obj2

ePre-condition orinvariant violated

Assertion violated

Post-condition or invariant violated

obj2

ePre-condition orinvariant violated

Assertion violated

Post-condition or invariant violated

obj1

a

obj2

b

obj3

cd

ef

faulty statement

exception raised and detection in operation e

obj1

a

obj2

b

obj3

cd

ef

faulty statement

exception raised and detection in operation e

Exception raised Methods analyzed Measure

at e’s entry a, b, c 3

during e, before the call to f e, a, b, c 4

during e, after the call to f e, f, a, b, c 5

8ISSTA 2002, Rome, Italy

Contract Instrumentation

• Instrumentation effort depends on:– The language used for writing analysis contracts (i.e., OCL)– The tool and language used for the instrumentation

• programming language dependent

• Translate OCL contracts to code Javadoc assertions• Instrument the code based on Javadoc assertions• Instrumentation issues:

– Access to attribute values => requires the addition of ‘get’ methods– Collections => requires to write search algorithms in assertions– Delegation => where to place assertions?– Contracts using methods that read data from a user or a device =>

requires to use assertions in the body of the method and split postcondition assertions

9ISSTA 2002, Rome, Italy

Experimental Setting

• Case study: an Automated Teller Machine– 21 classes, 2200 LOCs– with Analysis contracts (defined using OCL)

• Seeding of faults: 112 faults (OO and non-OO)– 17 mutation operators (Java mutation operators by Kim et al)– Faults seeded in 11 classes (non-GUI, non-device interface, core

classes)

• Test cases: category-partition• Execution:

– 5 different ATM case studies: without contracts (only oracles), with contracts at three different levels of details, with contracts and oracles.

– Diagnosability measure for each mutant in each test execution that kills it.

10ISSTA 2002, Rome, Italy

Results

• 112 mutants– 99 mutants killed by oracles– 84 mutants killed by contracts (5 equivalent mutants killed)

• Using contracts (at any level of detail) significantly improves diagnosability (one order of magnitude) Significant effort savings during debugging

• Using simple contracts is sufficient

# of mutants considered Average diagnosability

Oracle only 79 13.3

Contracts Highest 79 1.35

Intermediate 75 1.36

Lowest 74 1.37

11ISSTA 2002, Rome, Italy

Results

0.00

5.00

10.00

15.00

20.00

25.00

30.00

35.00

0 10 20 30 40 50 60 70 80

Mutant (killed)

Dia

gnos

abilit

y

Oracle

Contract

12ISSTA 2002, Rome, Italy

Conclusion

• Instrumented Analysis contracts are useful for fault isolation– Significant effort savings during debugging

– Reuse of contracts defined during UML/OCL Analysis

• Other results (in technical report)– When combined with oracles, contracts help detect additional

mutants (+5, 104/112 mutants detected)

– Contracts are good substitute to hard-coded oracles for 84% of mutants

Experiments needed in realistic contexts with actual software developers

Better instrumentation tools (supporting OCL)

13ISSTA 2002, Rome, Italy

Investigating the Use of Analysis Contracts to Support Fault Isolation in Object-Oriented Code

Lionel Briand, Yvan Labiche, Hong Sun

Software Quality Engineering Laboratory

http://www.sce.carleton.ca/Squall

Systems and Computer Engineering

Carleton University

Ottawa, Canada

14ISSTA 2002, Rome, Italy

Analysis Contract: Example

context: Bank::initiateWithdrawl(cardNumber:int, PIN:int, serialNumber:int, from:int, amount:Money, newBalance:Money, availableBalance:Money):intpre: serialNumber > 0 and from = CHECKING or from = SAVINGS or from = MONEY_MARKETpost:if not self.card->exists(_cardNumber = cardNumber) then

result = Status.UNKNOWN_CARDelse

if PIN <> self._currentTransactionCard._PIN thenresult = Status.INVALID_PIN

else if not self._currentTransactionAccount._available then

result = Status.NO_SUCH_ACCOUNTelse

if from = SAVINGS then result = Status.CANT_WITHDRAW_FROM_ACCOUNT

else if self._currenttransactionAccount._availableBalance.less(amount) then

result = Status.INSUFFICIENT_AVAILABLE_BALANCEelse

if (MAXIMUM_WITHDRAWL_AMOUNT_PER_DAY.less(Money.add(self._currentTransactionCard. _withdrawlsToday, amount)) then result = Status.DAILY_WITHDRAWL_LIMIT_EXCEEDED

elseresult = Status.SUCCESS and _currentTransactionAmount.equals(amount) andnewBalance.getValue()=self._currentTransactionAcount._currentBalance.getValue()–amount.getValue() and availableBalance.getValue()=self._currentTransactionAccount. _availableBalance .getValue() – amount.getValue()

endif