Online Model-based Testing and Runtime Monitoring of the ...€¦ · Offline Model-based Testing and Runtime Monitoring of the Sensor Voting Module Paolo Arcaini Angelo Gargantini

Offline Model-based Testing and

Runtime Monitoring of the Sensor Voting Module

Paolo Arcaini Angelo Gargantini Elvinia Riccobene

Università of Bergamo- Italy

Università di Milano - ItalyTolouse, ABZ 2014

ABZ 2014 Angelo Gargantini - MBT and runtime monitoring SVM

Motivation

• For safety critical components, formal verification and validation of models must be combined with the validation of the implementation. • confidence that the system has been implemented as specified, i.e., its

implementation conforms to its requirements.

• regardless the correctness of the model (guaranteed by formal verification, simulation and so on), the implemented system must be validated itself.

• Even when formal refinement and/or program formal verification are applied

“Who would want to fly in an airplane with software proved correct, but not tested?”

Ed Brinksma in his 2009 keynote at the Dutch Testing Day and Testcom/FATES


General scenario


Model-basedtesting

Runtime Monitoring

requirements

Formal specification

Implementation

Models used for test data generation and verdicts computation (oracle)In off-line, test suites are derived from the model and saved.

During runtime, checking whether a runof a system satisfies or violates a givencorrectness property

Conforms to ?

Proposed validation process


ASM model

Java

Req.

1.modelling

5. Offline testing6. Runtime monitoring

3. Implementing

4. Link Implementation and specification

2 Model validation

Applied to the sensor voting module

1. ASM model for SVM

• Using AsmetaL• http://asmeta.sourceforge.net/

• function channel represents the signals coming from the three channelsdynamic monitored channel: Channel −> Boolean

• function validCh keeps track if each channel is still validdynamic controlled validCh: Channel −> Boolean

• the output is sensor and its validity is simply defined as a derived function validdynamic controlled sensor: Boolean

derived valid: Boolean

• RULES to set the value and the validity


2. Model validation

1. Simulation• User guided

• Scenario guided• “all the sensors stay valid, the value is what expected”

• “sensor1 becomes invalid and after some steps also sensor2 fails”

2. Formal Property Verification • By model checking

• Once the sensor becomes invalid, then it will always remain invalid:• CTLSPEC AG( not(valid) implies AG (not(valid)))

• There exists a path in which the sensor eventually becomes invalid:• CTLSPEC EF (not(valid))

• There exists a path in which the sensor always remains valid:• CTLSPEC EG (valid)


We found at least one errorin our initial specification

3. Write the Java implementation

• The ASM and the Java implementation have been developed independently• once we have agreed upon the interface

• A method to pass the current values of the channels

• A way to get the sensor value and validity

• More freedom (thanks to Java)• Use of generics

• System.out / log4j

• The Java class can contain further methods not interested bythe spec


4. Linking a Java class with its ASM model

• We reuse the approach CoMA

Conformance Monitoring by ASMs• Paolo Arcaini and Angelo Gargantini and Elvinia Riccobene CoMA:

Conformance Monitoring of Java programs by Abstract State Machinesin 2nd International Conference on RUNTIME VERIFICATION RV 2011

• Based on the use of Java annotation (@)

• The implementation itself contains the definition of the links to the specification

• The link should be maintained during software evolution


4. Linking a Java class with its ASM model

• Given a Java class C and its ASM model 𝐴𝑠𝑚𝐶

• 1. Link the class its specification

@Asm("SensorVotingModule.asm")

class Sensor {

….

}

2. Link the state

Java state Abstract State

3. Link the execution

Java methods Rules

Java ClassC

AsmSpecification

𝐴𝑠𝑚𝐶

link

@Asm


4b. Java observed state to ASM signature• Java observed state

OS(C) consists of all public fields, and pure non void public methods of the class C the user wants to observe

𝑙𝑖𝑛𝑘: 𝑂𝑆 𝐶 → 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛𝑠(𝐴𝑠𝑚𝐶)

For the voting module:class Sensor {…@MethodToFunction(“sensor")public boolean getValue(){ return value;}

@MethodToFunction("valid")public boolean isValid(){ return sensorValid;}

Java ClassC

AsmSpecification

𝐴𝑠𝑚𝐶

@FieldToFunction@MethodToFunction

Signature

OBSERVED STATE


class Sensor{….@StartMonitoringpublic Sensor() { …}@RunSteppublic void computeSensorValue() {…}

}

4c Java methods to ASM execution

• the execution of the Java code must be linked with an execution(i.e., a run) of the ASM.

• Identify Java changing methodsCM(C)

is the set of methods of C whose execution is responsible for changing values of OS(C) and that the user wants to observe

Java ClassC

AsmSpecification

𝐴𝑠𝑚𝐶

@StartMonitoring@RunStep

Rules

Changing Methods


5. Offline testing

• Test generation

Using model checking (Spin) (ATGT) and several coverage criteria to generate “abstract” tests

• Test concretization

The test are “adapted” to concrete test execution


ASM model

Java

TESTS

Test generation

Test concretization

5a. Test generation

• ATGT exploits the counter example generation of the model checker SPIN

• ATGT has derived 38 test predicates • 20 for the BRC and 18 for the URC

• Each test predicate represents a test goal or test requirement

• For example

valid = false : the sensor becomes invalid

• ATGT produces • 38 test sequences

• 11 test sequences if tests reduction is applied

• They are enough to cover all the test goals


−−−− state 0 −−−−−−− controlled −−valid = true−− monitored −−channel(ONE) = falsechannel(TWO) = falsechannel(THREE) = true−−−− state 1 −−−−−−− controlled −−sensor = falsevalid = true

Example of test sequence

5b. Test concretization

• Test are translated to JUnit code

• Deriving a concrete Java test• A sequence of method calls

• with suitable checks (i.e., asserts)

• novel technique


The test concretization leverages the linking between the Java class and the ASM and the definitions of state conformance and step conformance of point 3

ATS - ABSTRACT TEST SEQUENCE Junit @Test

Test translation to JUnit

• @StartMonitoring: create a new Object

• For each state of the ATS, call the method annotated with @RunStep (+ parameters)

• Build the oracle by exploiting the annotations @FieldToFunction and @MethodToFunction.


@Test

public void test() {

// state 0

Sensor sut = new Sensor ();

assertEquals(true, sut.isValid());

sut.computeSensorValue(false, false, true);

// state 1

assertEquals(false, sut.getValue());

assertEquals(true, sut.isValid());

}

−−−− state 0 −−−−−−− controlled −−valid = true−− monitored −−channel(ONE) = falsechannel(TWO) = falsechannel(THREE) = true−−−− state 1 −−−−−−− controlled −−sensor = falsevalid = true

6. Runtime Monitoring (CoMa)

1. During the execution of the Java object of class C

2. an observer evaluates when the observed state is changed• Using AspectJ

3. makes the abstract ASM to perform a machine step• Using AsmetaL simulator

4. An analyzer evaluates the step conformance between the Java execution and the ASM simulation


ASM model

𝑂𝐶 - Java OBSERVER

SIMULATOR

ANALYZER

1 23 4

OK/FAIL

Runtime conformance

Runtime conformance 𝑂𝐶 is runtime conformant to its 𝐴𝑠𝑚𝐶 if:

1. no specification invariant of 𝐴𝑠𝑚𝐶 is ever violated.

2. the initial state of the computation of 𝑂𝐶 conforms to the initial state of the computation of 𝐴𝑠𝑚𝐶

3. every observed change step conforms with the step of 𝐴𝑠𝑚𝐶

𝐴𝑠𝑚𝐶

𝑂𝐶

Non changing methods steps are not checked

@StartMonitoring @RunStep

conf conf

@RunStep

computation (states)


Experimental result – Sensor Voting

• MBT: 38 Junit tests

• CoMA: we have simulated the environment by instantiating 10 times a new sensor and computing 10 times the sensor value passing three random values as inputs for the three channels.

• Evaluation• Code coverage by EclEmma: 100% in both cases

• Mutation score by PIT 57 killed mutants over 74 in both cases

• Both techniques are equivalent regarding detecting faults inserted by the standard PIT mutation operators.

• CoMA has a great runtime overhead (ASM simulator must run in parallel)


When runtime monitoring is better?

• we have simulated a delayed short circuit fault that causes isValid to return true after 5 times it is called.

int nvCount = 0;

boolean isValid() { return valid | nvCount++ > 5;}

• The tests produced from the specification do not detect this fault

• the rule coverage of the specification does not imply the coverage of this faulty behavior in the implementation.

• However, monitoring the code with CoMA exposes the failure by any run in which valid becomes false and isValid is called at least 5 times.

• In general, unspecified anomalous behaviors of the implementation are better detected by runtime monitoring than by MBT


Conclusions

• A complete ASM-based methodology for the validation of code

• Guidelines/methodology for:• Linking the code and its specification

• Deriving test cases from models

• Performing runtime monitoring

• Lesson learned• Case study very useful for students/practitioners

• Importance of a linking (for both MBT and runtime, but also future use)

• This can limit the use of Java (skeleton generation could be useful)

• Future work: compare w.r.t other approaches like Junit tests written by hand/ JML - Design by Contract/ test generation


-

Documents

Online Model-based Testing and Runtime Monitoring of the ...€¦ · Offline Model-based Testing and Runtime Monitoring of the Sensor Voting Module Paolo Arcaini Angelo Gargantini