Embed Size (px)
Citation preview
Formal Conformance Testing of Systemswith Refused Inputs and Forbidden ActionsIgor Burdonov,Alexander Kossatchev,Victor KuliaminISP RAS, Moscow
Outline
IntroductionFormal Testing with LTSes and ioco
Proposed conformance relation LTS Completion Conclusion
Formal Conformance Testing
Requirements
System under Test
Specification
Test Suite
Implementation
Model Test Suite
Real World Model World
conforms ??? conforms formally
passes formallypassesderivation
modeling
translation
!!!
Labeled Transition Systems
Model world – LTSes States Transitions Labels
Inputs ?a, ?b Outputs !x, !y Internal transitions τ
Initial state
?a
!x
?b
!y
τ
ioco Conformance Relation
Testing abilities Ability to provide inputs Ability to observe outputs Ability to observe quiescence
I ioco S for each δ-trace (trace with quiescence) σ of S
an output is possible in I after σ only if its is possible in S quiescence is possible in I after σ only if it is possible in S
?a
!x
?b
δ
Examples
?a
!x
?bδ
!y
τ
S
δ
δ?a
!x
?bδ
?b
?a
I1
δ
!y
?a
δ
?a
!x
?bδ
?b
?a
I2
δ
!y
?a
?a
?b
?b
δ
?a
I3
!y
?a
δ
I1 ioco S I2 ioco S I3 ioco S
τ
?
More Subtle Abilities
Testing abilities Ability to provide inputs Ability to observe outputs Ability to observe quiescence
Constraints Implementation should be
input-enabled Tester has full control over action
firing of implementation - synchronous testing( parallel composition semantics )
?a
!x
?b
?b
?a
?a
!y
I
?b
?a
?b
τ
!bT
Practical Considerations
Some implementations are notinput-enabledLaunch buttonMemory allocation
Testing in almost cases isasynchronousEspecially for distributed systems
free(void *ptr)
“ptr should be earlier returned by malloc(), calloc(), realloc()” (POSIX)
Environment
Asynchronous Testing
System under Test
Tester
Real World Model World
Model Tester
Implementation
Context
T
I
CI || C
Compositional Conformance
We may try to check that(I || C) ioco (S || C)
But ioco interacts with || in a bad way! Unexpected conformance
(I || C) ioco (S || C) while I ioco S
Unexpected non-conformance (I || C) ioco (S || C) while I ioco S
Bad Examples
?a
!y
?b
S
!x ?a
!x
I1
!x
C – input and output queues
I1 ioco S
(I1 || C) ioco (S || C)
?a
!y
I2
!x
?b
?b !z
I2 ioco S
(I2 || C) ioco (S || C)
Possible Ways out
ioco is asymmetric – I should be input-enabled, while S should not ioco is preserved when S is input-enabled
Consider input-enabled specs onlyToo narrow
Replace S with S’ such thatI I ioco S I ioco S’ ( S ~ioco S’)and S’ is input-enabled
Consider not input-enabled implementations and ioco analogue for them
Consider context-specific conformance relationsQueues already considered, but others are also needed
Outline
Introduction Proposed conformance relation
Extension of ioco for non-input-enabled implementations
LTS Completion Conclusion
Meaning of Undefined Inputs
ForbiddenShould not be provided
RefusedCan be provided and its refusal can be observed
IgnoredCan be provided, does nothing
should be specified
Proposed Model
LTS with forbidden actions and refused inputs Additional elements
Forbidden action γ Refused inputs {?a}, {?b}
βγδ-tracesContain inputs, outputs, δ, input refusals, γ
γ can only be the last symbol
?a
!y
?b
!x
τ
{?a,?b}, δ
{?a}
{?b}
δ
?b
?a
?a,?b
?a,?b
?b
γ ?a
Safe Traces
βγδ-traces that cannot cause forbidden action to occur
?b
?b
?a
!x
τ
?a
γ
?aτ
!y
τ
γ
?b
Safe βγδ-traces:
?b
!x
?aδ?b{?b}?a
!x?aδ{?a}
δ
{?b}
δ, {?a}!x
Safety of Testing
Test should not destroy implementation Safety hypothesis
(weakening of input-enabledness hypothesis)Each safe βγδ-trace of S, which is also βγδ-trace of I, can be safely extended in I by each symbol safe after this trace in S
Such I is called safe for S
iocoβγδ Conformance Relation
Testing abilities Ability to provide inputs Ability to observe outputs Ability to observe quiescence Ability to observe input refusal
Constraints Implementation is safe for specification Tester has full control over action firing of
implementation (synchronous testing)
iocoβγδ Conformance Relation
I iocoβγδ S I is safe for S and
for each safe βγδ-trace σ of S an output safe in S after σ is possible in I after σ only if its is
possible in S quiescence safe in S after σ is possible in I after σ only if it is
possible in S an input safe in S after σ is possible in Ι after σ only if it is
possible in S an input refusal safe in S after σ is possible in I after σ only
if it is possible in S
Examples
?a
,?b
γ
S I2I1
I1 iocoβγδ S I3 iocoβγδ SI2 is not
safe for S
!y
!x?a
!y
?b
!x
τ
{?a,?b}, δ
{?a}
{?b}
δ
?b
γ ?a?b
?a
?b
γ
!y
!x
?b
?b
{?b}
{?a}
{?a}
!y
δ
{?b}
{?a,?b}, δ {?a,?b}, δ
δ
?a
γ
I3
{?a,?b}, δ
?a
?b !y
!y
?b
?b
{?b}
{?a}
{?a}
!y
δ
I4 iocoβγδ S
I4
δ
I5
I5 iocoβγδ S
!x ?b
I6
I6 iocoβγδ S
{?b}{?a} {?a} {?a}
Test Case Derivation
Very similar to ioco Differences
We should escape forbidden action
– Use safe traces onlyWe should test input refusals
– Try to provide an input and observe refusal as a deadlock
ExamplesS
?a
!y
?b
!x
τ
{?a}
{?b}
δ
?b
γ?a
{?a,?b}, δ
T1
θ
!a
?y?x,θ
!b
?x,?y!b
θ
!a
θ
θ θ
θ
?x,?y
θ
θ
!a
!b
T2
!x
?y,θ
?x,?y!b
θ θ
θ
?x,?y
θ
θ
!a
!b
Outline
Introduction Proposed conformance relation LTS Completion
Trying to force composition to preserve conformance
Conclusion
Completion Operations
ioco is preserved when S is input-enabled Replace S with S’ such that
I I ioco S I ioco S’ ( S ~ioco S’)and S’ is input-enabled
Then,I ioco S I ioco S’ C (I || C) ioco (S’ || C)
S’ is more correct form of S
– it can be used in any context
Demonic completion Ξ
Tretmans et al.
LTS
May add more conforming implementations
Undefined inputs
τAll inputs
All outputs ?a!x !yτ
S I
I ioco S?a
?a
!y
?a
Ξ(S)
?a ?a
τ!x ?a!yI ioco Ξ(S)
Basic Completion
States Bc(S) = δ-traces of S For each δ-trace σ of S add the following transitions
R(σ) means all δ-traces obtained from σ by deleting some δ-s Add σ σ<?a> marked with ?a
if μ R(σ) μ<?a> – δ-trace of S Add σ σ<!x> marked with !x
if μ R(σ) μ<!x> – δ-trace of S Add internal σ σ<δ>
if μ R(σ) μ<δ> – δ-trace of S and σ does not end with δ Add σ σ<!error> marked with !error
if μ R(σ) μ cannot be extended with any output, nor with δ
?a!x !yτ
?a
Proposed Completions
Bc(LTS)
Undefined inputs
τAll inputs
All outputs
Bc(LTS)
Undefined inputs
γ
?a!x !yτ
SΔ(S)
?a ?a
τ!x ?a!y
?a
γ
Γ(S) Δ
Γ
For each I and S in ioco domainI ioco S I iocoβγδ Δ(S) I iocoβγδ Γ(S)
Outline
Introduction Proposed conformance relation LTS Completion Conclusion
Summary
ioco
iocoβγδ
Extension for non-input-enabled
implementations
Completion –construction of equivalent
input-enabled spec
Δ, Γ
?
Announcement
Notation U – non-empty subsets of reachable states of S AU and z is safe in A Az – states reachable from A by z and τ AU and z is unsafe in A Az = γ AU and z is refusal set Az – maximal stable subset of A that for each
refusal in z it exists in each element of the subset States of F(S) – U, U×{outputs, δ}, γ Transitions
γ γ marked with γ If for symbol z Az is nonempty A Az and (A, δ) Az marked with z If !y is safe and exists in Az where z is refusal set internal A (Az,!y)
and (Az,!y) (Az)!y marked with !y If each !y is unsafe or not exists in nonempty Az where z is refusal set
internal A (Az, δ) If for ?a A?a is nonempty (A, !y) A?a marked with ?a
Practical Implications
Not any specs are written correctly for use in component-based systems
The transformation rules can serve for specification adjustment
The rules can be rephrased into characteristics of correctly written specs
Thank you!
