Hiding the Formalism in Formal Methods

Hiding the Formalism in Hiding the Formalism in Formal MethodsFormal Methods

Lori A. ClarkeLori A. ClarkeLaboratory for Advanced Software Engineering ResearchLaboratory for Advanced Software Engineering Research

(LASER) (LASER) University of Massachusetts, AmherstUniversity of Massachusetts, Amherst

http://laser.cs.umass.edu/http://laser.cs.umass.edu/

Work done in collaboration with George S. Avrunin, Jamieson M. Cobleigh, Rachel Cobleigh, Work done in collaboration with George S. Avrunin, Jamieson M. Cobleigh, Rachel Cobleigh, Heather M. Conboy, Matthew B. Dwyer, Gleb Naumovich, & Leon J. OsterweilHeather M. Conboy, Matthew B. Dwyer, Gleb Naumovich, & Leon J. Osterweil

Model CheckingModel Checking• Includes a wide range of approaches for determining if finite models Includes a wide range of approaches for determining if finite models

of systems are consistent with specified propertiesof systems are consistent with specified properties• E.g., SPIN, SMV,FLAVERSE.g., SPIN, SMV,FLAVERS

• Verifies properties about system behaviorVerifies properties about system behavior• Successfully applied to hardware, software, and various modeling Successfully applied to hardware, software, and various modeling

languages (BPEL, Petri Nets, UML, Little-JIL)languages (BPEL, Petri Nets, UML, Little-JIL)

• Seeks a middle ground between testing and theorem provingSeeks a middle ground between testing and theorem proving• Testing requires “executable” semantics and only provides selective Testing requires “executable” semantics and only provides selective

resultsresults

• Theorem proving can deal with a wider range of properties, but usually Theorem proving can deal with a wider range of properties, but usually requires more mathematical expertiserequires more mathematical expertise

Model checking is still not widely Model checking is still not widely appliedapplied

• State Explosion ProblemState Explosion Problem• The cost of analysis can be exponential in the size of the The cost of analysis can be exponential in the size of the

system being analyzedsystem being analyzed

• Restricted to small systemsRestricted to small systems

• Even with extensive optimizations, creating a concise, but Even with extensive optimizations, creating a concise, but valid, model requires considerable insightvalid, model requires considerable insight

• Specifying properties is difficultSpecifying properties is difficult• Notations are cumbersomeNotations are cumbersome

• Many details must be taken into considerationMany details must be taken into consideration

Two Approaches for increasing Two Approaches for increasing acceptanceacceptance

• Automatically produce the model that is the Automatically produce the model that is the basis for analysisbasis for analysis• Produce a very concise, but conservative modelProduce a very concise, but conservative model

• Incrementally add precisionIncrementally add precision

• Provide natural language Interfaces for creating Provide natural language Interfaces for creating properties properties

OutlineOutline

• FLAVERS approach for automatically creating FLAVERS approach for automatically creating and improving the model and improving the model • Checking PropertiesChecking Properties

• Improving PrecisionImproving Precision

• Experimental ResultsExperimental Results

• PROPEL approach for elucidating propertiesPROPEL approach for elucidating properties• Question TreeQuestion Tree

• Disciplined Natural LanguageDisciplined Natural Language

• Finite-state automaton Finite-state automaton

FLAVERSFLAVERS

• FLFLow ow AAnalysis for nalysis for VERVERification of ification of SSystemsystems

• Verifies properties about concurrent and sequential Verifies properties about concurrent and sequential systemssystems

• Properties are represented as finite state automataProperties are represented as finite state automata• Checked using an efficient state propagation algorithmChecked using an efficient state propagation algorithm

• Uses an abstract, event-based graph model of the Uses an abstract, event-based graph model of the systemsystem• Imprecise, but conservativeImprecise, but conservative

• Precision can be improved incrementallyPrecision can be improved incrementally

Models for Concurrent SystemsModels for Concurrent Systems

• One model for a concurrent system is a One model for a concurrent system is a reachability graphreachability graph

• Represents all of the states a concurrent system Represents all of the states a concurrent system may reachmay reach• Location within each taskLocation within each task

• Values of variablesValues of variables

Reachability GraphReachability Graph

task body t1 is begintask body t1 is begin u;u; t2.send_synch;t2.send_synch; v;v; w;w;end t1;end t1;

task t2 body is begintask t2 body is begin x;x; t1.rec_ synch;t1.rec_ synch; y;y; z;z;end t2;end t2;

bb,,bb

u,u,bb

u,xu,x

bb,x,x

ss,,ss

ss,y,yv,v,ss

w,w,ss v,yv,y

w,yw,y

ee,,ee

ss,z,z

v,zv,z

w,zw,z

Trace Flow Graph (TFG)Trace Flow Graph (TFG)

• A TFG represents control flow through a A TFG represents control flow through a concurrent systemconcurrent system• Built from Control Flow Graphs for the tasks in the Built from Control Flow Graphs for the tasks in the

systemsystem

• Nodes and edges are added to represent concurrency Nodes and edges are added to represent concurrency

`̀`̀

TFG ConstructionTFG Construction

xx

yy

uu

vv

ww

synchsynch rec_ synchrec_ synchsend_synchsend_synch

task body t1 is begintask body t1 is begin u;u; t2.send_synch;t2.send_synch; v;v; w;w;end t1;end t1;

task t2 body is begintask t2 body is begin x;x; t1.rec_synch;t1.rec_synch; y;y; z;z;end t2;end t2;

zz

xx

yy

uu

vv

ww

synchsynch

zz

bb,,bb

u,u,bb

u,xu,x

bb,x,x

ss,,ss

ss,y,yv,v,ss

w,w,ss v,yv,y

w,yw,y

ee,,ee

ss,z,z

v,zv,z

w,zw,z

u

b,b

u,b

synchs,s

vv,s

w

w,s

z

w,z

e,e

u,xx

w,y

y

Feasible PathsFeasible Paths

xx

yy

uu

vv

ww

synchsynch

zz

bb,,bb

u,u,bb

u,xu,x

bb,x,x

ss,,ss

ss,y,yv,v,ss

w,w,ss v,yv,y

w,yw,y

ee,,ee

ss,z,z

v,zv,z

w,zw,z

Infeasible PathsInfeasible Paths

synch

u

b,b

u,b

Elevator PropertyElevator Property

The elevator does not moveThe elevator does not movewhile its doors are open.while its doors are open.

LL(P) is the set of all strings(P) is the set of all stringsaccepted by Paccepted by P

11

22

33

closeclose openopen

movemove

closeclosemovemoveopenopen

openopen

closeclosemovemove

Control Flow Graph (CFG)Control Flow Graph (CFG)

• A CFG G is <N, nA CFG G is <N, ninitialinitial, n, nfinalfinal, E> , E>

• Associate events with nodesAssociate events with nodes• G G is the alphabet of Gis the alphabet of G

• LL(G) is the language of G(G) is the language of G

• The set of all strings in (The set of all strings in (GG)) that occur on paths from that occur on paths from

the initial node to the final nodethe initial node to the final node

• CFG is alphabet refinedCFG is alphabet refined• Remove nodes that do not affect the property being Remove nodes that do not affect the property being

verifiedverified

Simple Sequential ExampleSimple Sequential Example

……1:1: if (stopped) thenif (stopped) then2:2: open;open;

end if;end if;……

3:3: if (stopped) thenif (stopped) then4:4: close;close;


5:5: move;move;……

1: if1: if

2: open2: open

3: if3: if

4: close4: close

5: move5: move

Proving PropertiesProving Properties

• Given a CFG G and a property PGiven a CFG G and a property P• Alphabet refine G with respect to Alphabet refine G with respect to PP

• Need to show Need to show LL(G) (G) LL(P)(P)

• Use data-flow analysis to propagate states of P Use data-flow analysis to propagate states of P to the nodes of Gto the nodes of G

• Worst-case cost is O((NWorst-case cost is O((NGG))2 2 S SPP))

State PropagationState Propagation

2: open2: open

4: close4: close

5: move5: move

Worklist: 2, 3Worklist: 2, 3

{1}{1}

{2}{2}

{1}{1}

{1,2}{1,2}

{1,3}{1,3}

, 4, 5, 4, 5

11

22

33

closeclose openopen

movemove


openopen

closeclosemovemove

3: if3: if

1: if1: if


1

22

3

closeclose openopen

movemove


openopen

closeclosemovemove


{1}{1}

{2}{2}

{1}{1}

{1,2}{1,2}

{{11,,33}}

, 4, 5, 4, 5

2: open2: open

4: close4: close

5: move5: move

3: if3: if

1: if1: if


1: if

2: open

3: if

4: close4: close

5: move

{1}{1}

{2}{2}

{1}{1}

{1,2}{1,2}

{1,{1,33}}

11

22

3

closeclose openopen

movemove


openopen

closeclosemovemove


1: if

2: open

3: if

4: close4: close

5: move

……1:1: if (stopped) thenif (stopped) then2:2: open;open;


3:3: if (stopped) thenif (stopped) then4:4: close;close;



Boolean Variable ConstraintBoolean Variable Constraint

== is a predicate== is a predicate= is assignment= is assignment

S==tS==tS=tS=t

S==tS==tS=tS=t

S==tS==t

S==fS==fS=fS=f

S==fS==f

S==tS==tS=tS=t

S==fS==fS=fS=f

S==fS==fS=fS=f

S=fS=f

S=tS=t

uu

fftt

vv

Boolean Variable ConstraintBoolean Variable Constraint

== is a predicate== is a predicate= is assignment= is assignment

S==tS==tS=tS=t

S==tS==tS=tS=t

S==tS==t

S==fS==fS=fS=f

S==fS==f

S==tS==tS=tS=t

S==fS==fS=fS=f

S==fS==fS=fS=f

S=fS=f

S=tS=t

uu

fft

v

Improving PrecisionImproving Precision

• Use constraints to improve precisionUse constraints to improve precision• Represented as FSAsRepresented as FSAs

• Given a CFG G, a property P, and constraints CGiven a CFG G, a property P, and constraints C11,,

…,C…,Cnn

• Alphabet refine G with respect to (Alphabet refine G with respect to (P P C1C1 … … CnCn))

• Want (Want (LL(G) (G) LL(C(C11) ) …… LL(C(Cnn)) )) LL(P)(P)

• Worst-case cost is O(NWorst-case cost is O(NGG2 2 S SP P S SC1 C1 …… S SCnCn))

Elevator RevisitedElevator Revisited

1: if

2: S==t

5: if5: if

9: move9: move

4: S==f

3: open3: open

6: S==t6: S==t 8: S==f8: S==f

7: close7: close

……1,2,4:1,2,4: if (stopped) thenif (stopped) then3:3: open; open;


5,6,8:5,6,8: if (stopped) thenif (stopped) then7:7: close; close;



, 6, 8, 6, 8, 5, 5, 3, 3


2: S==t2: S==t

1: if1: if

5: if5: if

9: move9: move

4: S==f4: S==f

3: open3: open

6: S==t6: S==t 8: S==f8: S==f

7: close7: close

11

22

33

closeclose openopen

movemovecloseclosemovemoveopenopen

openopen

closeclosemovemove

tt

vv

S==tS==t

ff

uuS==fS==f

S==fS==f S==tS==t

S==tS==t S==fS==f

S==fS==fS==tS==t

<2,t>,<1,v><2,t>,<1,v>


<1,u><1,u>

<1,t><1,t>

<2,t><2,t>

<1,f><1,f>

<2,t>,<1,f><2,t>,<1,f>

1: if

5: if

, 6, 8, 6, 8, 5, 5, 3, 3


2: S==t2: S==t

9: move9: move

4: S==f

3: open3: open

6: S==t 8: S==f8: S==f

7: close7: close

<2,t>,<2,t>,<1,v><1,v>


<1,u><1,u>

<1,t><1,t>

<2,t><2,t>

<1,f><1,f>

<2,t>,<1,f><2,t>,<1,f>

11

22

33

closeclose openopen


openopen

closeclosemovemove

tt

vv

S==tS==t

ff

uuS==fS==f

S==fS==f S==tS==t

S==tS==t S==fS==f

S==fS==fS==tS==t

1: if1: if

5: if5: if

, 6, 8, 6, 8, 5, 5, 3, 3


2: S==t2: S==t

9: move9: move

4: S==f4: S==f

3: open3: open

6: S==t6: S==t 8: S==f8: S==f

7: close7: close

<2,t>,<1,v><2,t>,<1,v>


<1,u><1,u>

<1,t><1,t>

<2,t><2,t>

<1,f><1,f>

<2,t>,<1,f><2,t>,<1,f>

<1,t><1,t>

<2,v>,<1,f><2,v>,<1,f>

<1,t>,<1,f><1,t>,<1,f>

11

22

33

closeclose openopen


openopen

closeclosemovemove

tt

vv

S==tS==t

ff

uuS==fS==f

S==fS==f S==tS==t

S==tS==t S==fS==f

S==fS==fS==tS==t

, 7, 9, 7, 9

1: if1: if

5: if5: if

, 6, 8, 6, 8, 5, 5, 3, 3


2: S==t2: S==t

9: move9: move

4: S==f4: S==f

3: open3: open

6: S==t6: S==t 8: S==f8: S==f

7: close7: close

<2,t>,<1,v><2,t>,<1,v>


<1,u><1,u>

<1,t><1,t>

<2,t><2,t>

<1,f><1,f>

<2,t>,<1,f><2,t>,<1,f>

<1,t><1,t>

<2,v>,<1,f><2,v>,<1,f>

<1,t>,<1,f><1,t>,<1,f>

1

22

33

closeclose openopen


openopen

closeclosemovemove

t

vv

S==tS==t

f

uuS==fS==f

S==fS==f S==tS==t

S==tS==t S==fS==f

S==fS==fS==tS==t

, 7, 9, 7, 9

Automatically Add Constraints Automatically Add Constraints as Neededas Needed

• Variable Automata - model variables that impact Variable Automata - model variables that impact important predicatesimportant predicates

• Task Automata - model control flow of selective Task Automata - model control flow of selective taskstasks

xx

yy

uu

vv

ww

synchsynch

zz

bb,,bb

u,u,bb

u,xu,x

bb,x,x

ss,,ss

ss,y,yv,v,ss

w,w,ss v,yv,y

w,yw,y

ee,,ee

ss,z,z

v,zv,z

w,zw,z

Infeasible PathsInfeasible Paths

synch

u

b,b

u,b

Experimental ResultsExperimental Results

• Evaluate how FLAVERS performance scales as Evaluate how FLAVERS performance scales as program size increasesprogram size increases• TimeTime

• MemoryMemory

• Number of constraintsNumber of constraints

Example: ChironExample: Chiron

• User interface system developed at UC IrvineUser interface system developed at UC Irvine• Uses event-based notificationUses event-based notification

• Scaled by increasing the number of listened for events Scaled by increasing the number of listened for events • Lines of codeLines of code

• 2 events 2 events 259 259

• 53 events53 events 3,5573,557

• Proved several properties about ChironProved several properties about Chiron(Avrunin, Corbett, Dwyer, Pasareanu, Siegel)(Avrunin, Corbett, Dwyer, Pasareanu, Siegel)

• p07 - If listener1 registers for event1 before listener2, then p07 - If listener1 registers for event1 before listener2, then listener1 will be notified of event1 before listener2listener1 will be notified of event1 before listener2

• p09 - The program never terminates while a listener is listening p09 - The program never terminates while a listener is listening for an eventfor an event

p07 Comparison (Original)

0.1

1

10

100

1000

10000

0 5 10 15 20 25 30 35 40 45 50 55

Events

Time (s)

INCA

Spin

SMV

NuSMV

Native Spin

FLAVERS

p07 Comparison (Decomposed)

0.1

1

10

100

1000

10000

100000

0 10 20 30 40 50 60 70 80 90 100

Events

Time (s)

INCA

Spin

SMV

NuSMV

Native Spin

FLAVERS

p09 Comparison (Original)

0.1

1

10

100

1000

10000

0 5 10 15 20 25 30 35 40 45 50 55

Events

Time (s)

INCA

Spin

SMV

NuSMV

Native Spin

FLAVERS

FLAVERS Times

1

10

100

1000

10000

0 5 10 15 20 25 30 35 40 45 50 55

Events

Time (s)

p01p02p03p04p05p06p07p08p09

Observations about Using Observations about Using ConstraintsConstraints

• In our experimentsIn our experiments• For the vast majority of programs and properties, the For the vast majority of programs and properties, the

constraints needed to verify a property for the smallest constraints needed to verify a property for the smallest configuration of the system are sufficient to verify the configuration of the system are sufficient to verify the property for larger configurationsproperty for larger configurations

• Never needed more than 4 constraintsNever needed more than 4 constraints• Have not tried to find the minimal number of constraintsHave not tried to find the minimal number of constraints

• Vast majority of constraints could be determined Vast majority of constraints could be determined automatically using simple heuristicsautomatically using simple heuristics

• A useful modeling approachA useful modeling approach• Can model aspects of the environment Can model aspects of the environment • Can model malicious behaviorCan model malicious behavior

OutlineOutline

• FLAVERS approach for automatically creating FLAVERS approach for automatically creating and improving the model and improving the model • Checking PropertiesChecking Properties

• Improving PrecisionImproving Precision

• Experimental ResultsExperimental Results

• PROPEL approach for elucidating propertiesPROPEL approach for elucidating properties• Question TreeQuestion Tree

• Disciplined Natural LanguageDisciplined Natural Language

• Finite-state automaton Finite-state automaton

Property SpecificationsProperty Specifications

• A property focuses on describing one particular A property focuses on describing one particular aspect of system behavioraspect of system behavior• Even with such focus, it can still be difficult to write a Even with such focus, it can still be difficult to write a

property correctlyproperty correctly

• A property should be precise and accessibleA property should be precise and accessible• precise enough to support unambiguous precise enough to support unambiguous

communication and automated analysescommunication and automated analyses

• accessible enough to be readily understoodaccessible enough to be readily understood

What is the problem?What is the problem?

Specifications need to be…Specifications need to be…

• Accessible Accessible

• RigorousRigorous

• Precise/accurate Precise/accurate

• ConsistentConsistent

• AnalyzableAnalyzable

natural language is accessible

““the nurse must verify the patient's the nurse must verify the patient's identifying information before starting to identifying information before starting to transfuse a unit of blood product into the transfuse a unit of blood product into the patientpatient””

..but it is neither rigorous nor precise Is the nurse allowed to verify the Is the nurse allowed to verify the

patient's identity more than patient's identity more than once before starting to transfuse once before starting to transfuse blood?blood?

Does the nurse have to verify the Does the nurse have to verify the patient's identity again before patient's identity again before transfusing the next unit of blood?transfusing the next unit of blood?

temporal logic is rigorous

Linear Temporal Logic (LTL):

☐ ☐¬order_received ∨¬order_received ∨

♢ ♢(order_received ∧(order_received ∧

(☐¬blood_transfused ∨(☐¬blood_transfused ∨

(¬blood_transfused U (¬blood_transfused U

identity_verified)))identity_verified)))

…but not accessible to most practitioners

What is the problem?What is the problem?

Specifications need to be…Specifications need to be…

• Accessible Accessible

• RigorousRigorous

• Precise/accurate Precise/accurate

• ConsistentConsistent

• AnalyzableAnalyzable

Our ApproachOur Approach• Provides property templates that explicitly shows optionsProvides property templates that explicitly shows options

• Extends property patterns Extends property patterns (Dwyer, Avrunin, & Corbett 1998; 1999)(Dwyer, Avrunin, & Corbett 1998; 1999)

• Provides multiple views of the propertyProvides multiple views of the property• Views chosen to support precision, accessibility, and user Views chosen to support precision, accessibility, and user

guidanceguidance• User can work with one or more of the viewsUser can work with one or more of the views

• Changes made in one view are reflected in the othersChanges made in one view are reflected in the others• Formal view is rigorous enough to support verificationFormal view is rigorous enough to support verification

• Implemented prototype tool, PROPELImplemented prototype tool, PROPEL• PROPPROPerty erty ElElucidation ucidation

Propel TemplatesPropel Templates

Global

Before end

After start

Between start and end

SCOPES

BEHAVIORS

Response A results in B

Precedence A enables B

Absence A never occurs

Existence A must occur

Name Name Intent

Question Tree ViewQuestion Tree View

• Developed to help users select the appropriate Developed to help users select the appropriate pattern templatespattern templates• One tree for scope and one for behaviorOne tree for scope and one for behavior

• Found to also be useful for resolving detailed Found to also be useful for resolving detailed optionsoptions

Example PropertyExample Property

The The patient’s identification must be verifiedpatient’s identification must be verified prior prior to to transfusing each unit of blood producttransfusing each unit of blood product..

•Must identify events of primary interestMust identify events of primary interest• One or two eventsOne or two events

views use the actual parameter names if they are providedviews use the actual parameter names if they are provided

EVENT: transfuse-blood

EVENT: verify-patient-ID

Question Tree ViewQuestion Tree View

How many events of primary interest are there?How many events of primary interest are there? One: event One: event verify-patient-IDverify-patient-ID

Two: events Two: events verify-patient-IDverify-patient-ID and and transfuse-bloodtransfuse-blood

After verify-patient-ID occurs, transfuse-blood is required to occur

transfuse-blood cannot occur until after verify-patient-ID has occurred

Precedence FSA TemplatePrecedence FSA Template

transfuse-bloodtransfuse-blood ororverify-patient-IDverify-patient-ID oror(verify-patient-(verify-patient-ID,ID, transfuse-blood)transfuse-blood) oror

verify-patient-IDverify-patient-ID

transfuse-transfuse-bloodblood

(verify-patient-(verify-patient-ID,ID, transfuse-transfuse-blood)blood)

verify-patient-ID

transfuse-bloodtransfuse-blood ororverify-patient-IDverify-patient-ID oror(verify-patient-(verify-patient-ID,ID, transfuse-blood)transfuse-blood)

Precedence FSA TemplatePrecedence FSA Template

transfuse-bloodtransfuse-blood ororverify-patient-IDverify-patient-ID oror(verify-patient-(verify-patient-ID,ID, transfuse-blood)transfuse-blood) oror

verify-patient-IDverify-patient-ID



verify-patient-ID

transfuse-bloodtransfuse-blood ororverify-patient-IDverify-patient-ID oror(verify-patient-(verify-patient-ID,ID, transfuse-blood)transfuse-blood)

Precedence DNL TemplatePrecedence DNL Template



Example BehaviorExample Behavior

transfuse-blood cannot occur unless verify-patient-ID has already occurred.

It is acceptable for verify-patient-ID to not occur, but if it does not occur then transfuse-blood can never occur. Even if verify-patient-ID does occur, transfuse-blood is not required to occur.

Before the first verify-patient-ID occurs, the events in the alphabet of this property, other than transfuse-blood, can occur any number of times.

After verify-patient-ID occurs and before the first subsequent transfuse-blood occurs: •the events in the alphabet of this property, including verify-patient-ID but not transfuse-blood, can occur any number of times.

After the first subsequent transfuse-blood occurs:•the events in the alphabet of this property, other than verify-patient-ID or transfuse-blood, can occur any number of times;

•neither verify-patient-ID nor transfuse-blood can occur again.

(verify-patient-(verify-patient-ID,ID, transfuse-blood)transfuse-blood)



verify-patient-ID

transfuse-bloodtransfuse-blood

Observations about Specifying Observations about Specifying PropertiesProperties

• Have used PROPEL (and FLAVERS) in the UMass Have used PROPEL (and FLAVERS) in the UMass Medical Safety ProjectMedical Safety Project• 3 in-depth case studies3 in-depth case studies

• Blood transfusion, Chemotherapy, Emergency Room FlowBlood transfusion, Chemotherapy, Emergency Room Flow

• Found (both) to be very effectiveFound (both) to be very effective

• Needed to add exceptional behavior to PROPELNeeded to add exceptional behavior to PROPEL• Users frequently overlook the impact of exceptional behaviorUsers frequently overlook the impact of exceptional behavior

• Both computer scientists and non-computer scientists Both computer scientists and non-computer scientists have found the approach useful have found the approach useful

ConclusionsConclusions

• Many areas for future researchMany areas for future research• Automatic model generationAutomatic model generation• Property specifications Property specifications • Counter example generationCounter example generation• OptimizationsOptimizations• Compositional analysisCompositional analysis

• Believe that model checking will become a standard Believe that model checking will become a standard technique in IDEstechnique in IDEs• Already available as a “hidden” optimization technique for some Already available as a “hidden” optimization technique for some

programming languages programming languages • Predefined properties Predefined properties

• Has tremendous potential in Model Based Development Has tremendous potential in Model Based Development • Properties endure, perhaps with refinement, as models are Properties endure, perhaps with refinement, as models are

elaboratedelaborated

Documents

Hiding the Formalism in Formal Methods