Integrating Formal ProgramVerification with Testing

Cyrille Comar, Johannes Kanig and Yannick Moy

Integrating Formal Program

Verification with Testing

Cyrille Comar, Johannes Kanig and Yannick Moy

Integrating Formal Program

Verification with

Cyrille Comar, Johannes Kanig and Yannick Moy

Testing

Motivation

• Cost of testing greater than cost of development

• 10% increase each year for avionics software (Boeing META Project)

• Uneven repartition:

Cost of testing

Se-ries120%

80%

80% of effort!

• Uneven quality: 80% of errors traced to 20% of code (NASA Software Safety Guidebook)

• Need to reduce and focus the cost of testing

Formal methods […] might be the primary source of evidence for the satisfaction of many of the objectives concerned with development and verification.

2011: Formal Methods Supplement (DO-333)

DO-178C: formal methods can replace testing

• Myth 4: Formal methods require highly trained mathematicians

• Myth 5: Formal methods increase the cost of development

• Myth 6: Formal methods are unacceptable to users

• Myth 7: Formal methods are not used on real, large-scale software

(Anthony Hall, Praxis Systems, 1990)

Myths of formal methods

Since 2001, Airbus has been integrating several tool supported formal verification techniques into the development process of avionics software products.

2009: Formal Verification of Avionics Software Products (Souyris, Wiels, Delmas, Delseny)

Practice of formal methods

4%

16%

80%

Cost of verification

80% of testing effort

Hi-Lite goal: using formal verification first, then testing…

80% of formal effort

… to reduce and focus the cost of verification

20%

80%

20%

80%

testing

formal

Proof + Test

Programming Contracts

{P}C{Q} Hoare logic (1969)

logic contractsfor proofs

SPARK (1987)

executable contractsfor tests

Eiffel DbC (1986)

Hi-Lite: executable annotation language???

Project

Ada 2012

Testing vs. Formal Verification

RQ

P

PQ

P calls Q

prove pre of Qassume post of Q

assume pre of Qprove post of Q

PQ

P calls Q

use Q codecover P constructs

actual body of Qor stub…

global soundness argument:all functions proved all assumptions justified

local exhaustivity argument:each function covered enough behaviors explored

Combining tests and proofs

verification combining tests and proofs should be

AT LEAST AS GOOD ASverification based on tests only

PQ

P calls QP is tested

Q is provedQ calls P

How so we justify assumptions made during proof?

) …

Caution: contracts are not only pre/post!

data dependences

parameters not aliased

parameters initialized

strong typing

Combination 1: tested calls proved

PQ

P calls QP is tested

Q is proved

during testing:check that

precondition of Q is respected

assumption for proof:precondition of Q

is respected

Combination 2: proved calls tested

PQ

P is tested

Q is provedQ calls P

during testing:check that

postcondition of P is respected

assumption for proof:postcondition of P

is respected

Testing + Formal Verification

RQ

P

global soundness argument:- proof: assumptions proved- test: assumptions tested

tested

proved

proved

local exhaustivity argument:- test: function covered- proof: by nature of proof

Testing must check additional propertiesDone by compiler instrumentation

GNAT toolsuite

GNAT compiler

GNATtest unit testing

GNATprove unit proof

executable

aggregatedverification

results

Conclusion

• Soundness

• Applicability to the code

• Usability by normal engineers on normal computers

• Improve on classical methods

• Certifiability

Airbus 5 “must-have” of formal methods

current work

Benefits of openness

.org

• public: meeting minutes technical work 69 members

• private: management partner code

• announcements• meeting slides• articles / docs

• all code• dev docs• user docs

external collaborations with industry and academia

Project Partners

www.open-do.org/projects/hi-lite