Easier and More Informative Vacuity Checks

Hana Chockler and Ofer StrichmanIBM Research Technion

Technion -Israel Instituteof Technology

not at the same time

no free lunch .

IBM HRL

2

Preliminaries

The players: s.t. M ²

affects in M if M 2 [ <- false].

If 9 s.t. does not affect M, then is satisfied vacuously in M. [BBER01,KV99]

= G(req -> ack)

M

M ² [ack <- false]

An LTL formula: A structure: MA subformula of

T

“satisfies vacuously” =

“satisfies from the wrong reasons”

IBM HRL

3

Main Results

Cheap vacuity checks: Redundancy of properties Vacuity without design Enhancing vacuity information

Mutual vacuity: Maximal set of literals that can be replaced with

false without falsifying the specification (complexity results)

Iterative algorithm to find this set Approximation algorithm Responsibility

more expensive,more information

IBM HRL

4

Cheap-Checks (1): Redundancy

Redundancy of properties: is redundant w.r.t a set of properties if

Each check is easy (there is no system).

The result of a sequential redundancy check is sensitive to the order of checking.

Example:

either pRq or Gq are redundant, depending on the order.

IBM HRL

5

Complexity:

Problem Min-Redundancy: find a minimum subset of that is equivalent to . Complexity: -hard The corresponding decision problem is 2-hard

Naïve algorithm checks all subsets. Can still be feasible in practice because: can be small, and each check is easy.

Cheap-Checks (1): Redundancy

log n queries to oracle

IBM HRL

6

An approximation algorithm(results are sensitive to the order of checks):

Remove-redundancy(){For each 2

if

then = n }

Cheap-Checks (1): Redundancy

Some possible improvements: start with checking all sets of the most likely size,

start by removing longest formulas, etc.

IBM HRL

7

Vacuity without design: Checking vacuity of a property w.r.t the set of properties:

Assume ² . Then vacuity without design implies vacuity in the design:

Cheap-Checks (2): vacuity without design

We save a costly model checking

and vacuity check in the

design!

IBM HRL

8

The algorithm:

Vacuity-without-design(){For each property 2

For each subformula in if ² [Ã false] then

à [à false]}

Note that the order of vacuity checks does not matter: the substitution does not remove behaviors from .

Cheap-Checks (2): vacuity without design

Thus, the checkis polynomial!

IBM HRL

9

Cheap-Checks (3): more info

Suppose that: ²v Let be a subformula of such that M ² [ Ã false]

After “cheap check 2”, we know that 2v Let be the counterexample (an interesting witness).

Perhaps the bug is in the model: demonstrates an interesting behavior that should be added to .

So… we can suggest to enhance

Example: = G(req ->F ack) = { :req } 2 ackà false] = {} ¢ req ¢ ack ¢ {}

Mreq

ack

Constructing a new spec from the

new behavior – future work

satisfies vacuously

IBM HRL

12

Mutual Vacuity

Several subformulas can be replaced with false at the same time without falsifying the property. Vacuity does not check that.

Why is it important ? Next slide...

IBM HRL

13

Mutual Vacuity

Why is it important ?

A result of fixing one vacuity problem, may mask a bigger vacuity problem. That’s because vacuity is not monotonic:

there may be , s.t.: ²[ Ã false], ² [ Ã false], but 2 [ Ã false, Ã false].

Example: AG(a Ç b Ç c) ² AG(a Ç b Ç false) ² AG(false Ç false Ç c) 2 AG(false)

a b a b a b...

c c c c c c...’s single trace

IBM HRL

14

Mutual vacuity and Responsibility

We can ask ‘what is the largest set of subformulas of that can be replaced with false simultaneously without falsifying ?’ We refer to this problem as MAX-VACUOUS (defined in

[CG04]). We will discuss its complexity and how to compute it…

But we can also try to measure the ‘importance’ of each subformula in This will give us fine-grain vacuity. The motivation: the added information (beyond what can be

given to us by MAX-VACUOUS) can be useful for debugging the model.

To quantify `importance’, we will use the notion of responsibility (defined in [CH04]).

IBM HRL

15

Mutual Vacuity

In the discussion that follows we check vacuity and responsibility only for literals (and not for general subformulas). The justification:

[CG04] Theorem: Given a CTL* formula , A subformula of can be replaced with false iff all literals of can be replaced with false.

Example: let = GX(a :b) 8 . ² [ à false] $ ² [a à false, b à true]

IBM HRL

16

Mutual Vacuity

(Decision) Problem MAX-VACUOUS: is there a subset of k literals of that can be replaced with false without falsifying in M ? Complexity: NP-complete By a reduction from CLIQUE

(Computing) Problem MAX-VACUOUS: find a maximum subset of literals of that can be replaced with false without falsifying in M. Complexity: - complete for the size of the

maximum subset. (By a reduction from MAX-CLIQUE-SIZE)

the complexity above refers to the size, not to the

actual subset; to find the actual subset we need

an FNP oracle

IBM HRL

17

Mutual Vacuity (2)

Preprocessing for the algorithm: Construct a Buchi automaton B: for : Convert to a conjunctive Buchi automaton CB:

Case-split on disjunctions in the edge labels; add an edge for each case

s0

s2

s2

s0

s2

s2

XF((p1 Æ p2)Ç q)

Buchi Conjunctive Buchi

IBM HRL

18

Example

We know that M ² thus, no path satisfies CB: But replacing literals with false (and thus labels on edges with

true) creates counterexamples

Example: = p U (q U r); : = :p R (:q R :r);

s0

s2

s1 s3

s4

s5 s6

CB:

IBM HRL

19

Example

LetAccepting path: s0 s1 s4 s5 s6 …

Eliminating set (of occurrences): {p,q}

Put back either p or q in order to get rid of this trace

s6

s0

s2

s1 s3

s4

s5

s0

s1

s4

s5 s6

IBM HRL

20

Example

Suppose we choose to return p:

s0

s2

s1 s3

s4

s5 s6

Example

IBM HRL

21

Example - problems

LetAccepting path: s0 s3 s6 s6 s6 …

Eliminating set (of occurrences): {q}

Now we have to return q in order to remove the red trace

s0

s2

s1 s3

s4

s5 s6

s0

s3

s6

Example

IBM HRL

22

Example (Conclusions)Too bad…

We did not find any vacuous satisfaction because we first chose p, rather than q.Wrong order monotonic increase in ‘allegedly required’ literals

suboptimal vacuity detection.What we need:

Given a set of literal sets (e.g. {{p,q},{q}})

Choose the minimum set S of literals that intersects all sets (e.g. {q})

can be replaced with 8s S. s à false]

This is a minimum hitting set problem!The algorithm constructs the minimum set of literals that blocks all “bad”

transitions in an iterative way based on counter-example traces

The number of iterations is bounded by

exp(|L|)

Can be approximated using known

approximation algorithms for min_hitting_set

IBM HRL

23

Algorithm

literal-set Max_Vacuous(CB: ) {

literal-set S = ;set-of-literal-sets CE = ;literal-set L = the literals of CB:

while (true) {

CB’: = CB: [8l 2 LnS. l à true];

if M £ CB’: is empty then return S;

Let ce be the counterexample projected to L;Let elim-set(ce) be the set of literals that

can eliminate ce;

CE = CE [ elim-set(ce); // minimal set of literals that intersect all sets in CE

S = minimum-hitting-set(CE); } }

Algorithm

IBM HRL

24

Responsibility

When M ² Ã falsewe say that is vacuous in M due to

The check is: is there a counterfactual dependence between and in If no – then causes vacuity.

We would like to quantify ’s importance when there is no counterfactual dependence. Counterfactual dependence (=not vacuous): responsibility

= 1. What happens if there is no counterfactual

dependence, but there is some influence?

IBM HRL

25

Responsibility

Let be a formula in positive normal form s.t. ² Let l be a literal in The degree of responsibility of l, dr(l, ,M) of l in the

satisfaction of in M is 1/(k+1) if k is the smallest number for which there exists a subset S =

{l1…l} of ’s literals that maintains:l S² [l à false, … , l à false]

2 [l à false, … , l à false, l à false]

In other words, k is the size of the minimal subset of literals of that need to be replaced with false in order to make the value of in M depend on l.

IBM HRL

26

Examples:

AG(a Ç b Ç c)responsibility of a,b,c = ½.

AG(a Ç b Ç c); all states in M satisfy a,c, some satisfy b.responsibility of a,c: ½, of b: 0.

Responsibility

M (has a single trace)

a,c

a,b,c

a,c

a,c

a,b,c

a,c b,c

IBM HRL

27

Responsibility: usability

Complexity of computing the responsibility of a literal: FPNPlog(n) – complete, n = Reduction from CLIQUE-SIZE.

After examining responsibility and debugging the model, we simply would like to get a shorter formula in which all signals have responsibility 1.

Replacing any maximal subset of literals with false without changing the satisfiability of , makes all remaining signals ‘fully responsible’, i.e., for all , responsibility() = 1.

that is, it eliminates vacuity

IBM HRL

28

Experimental Results

We applied our easy vacuity checks to specifications of a real hardware block (from European project PROSYD).

The hardware block implements a producer, a consumer, and a data receiver.

The set of 17 properties describes its correct behavior. The results:

9 properties out of 17 are redundant. After removing the redundant properties, 1 property is

vacuous with respect to others.

IBM HRL

29

Can this work be used in RuleBase?

The most appropriate place for easy (preliminary) vacuity checks is the “property visualization tool”

Removing redundant properties and eliminating vacuity without design increases performance of model checking

If vacuity without design is not found, the witnesses can be saved for future information to the user in case of vacuous pass in the model

Writing “assumes” iteratively (to eliminate false counterexamples) can lead to redundant assumptions – checkable with redundancy and vacuity without design

Mutual vacuity – is probably too expensive ... but the approximate algorithm might be efficient