Embed Size (px)
DESCRIPTION
"Software Testing is the process of executing a program or system with the intent of finding errors.", Myers, 1979. The most important activity in this process is designing the required set of effective test cases. Thus, the problem is narrowed down to determining the exact number of required test cases and increasing their effectiveness.Project FoX is a production ready tool developed in Java, which offers Java developers the opportunity to leverage the proven theories and concepts of formal testing using generalized state automata (X-Machines) as a theoretical model of computation. The formal testing strategy FoX is applying, is proven to generate a complete test set that ensures the correctness of the implementation with respect to the specification.FoX enhances a novel testing process that is fully automated, ranging from complete test set generation, to test preparation and execution. This method can be applied to any Java based software system, regardless of its underlying technologies. Utilizing a formal approach will provide unambiguous test cases which are objective and not subjective to the tester's experience and intuition.The formal testing strategy provides functional testing that tests not only for the desired system behaviour (the system does what it should) but also tests that the system has no undesired behaviour (the system does not do anything it should not do).This short presentation will strive to give the audience an overview of the formal testing methodology and a demonstration of the tool (FoX). It will also showcase a real-life demo of the project FoX, applied to a Java SE application and will discuss how the methodology can be applied to any Java EE or ME application.Anyone with a software engineering background will be able to easily follow the talk and understand the benefits which this process offers to modern day software engineering.
Citation preview
PROJECT FOXA TOOL THAT OFFERS AUTOMATED TESTING USING A FORMAL
APPROACH
Ivo Neskovic
CITY College Thessaloniki, an International Faculty of the University of Sheffield
462
2
AGENDA
> SOFTWARE ENGINEERING: WHAT COULD GO WRONG?
> FORMAL METHODS
> PROJECT FOX
> CASE STUDY: THE BUFFER SYSTEM
> CONCLUSIONS AND FUTURE WORK
> BIBLIOGRAPHY
3
THE PROBLEM OF SOFTWARE ENGINEERING
> Faulty systems are a common notion nowadays.
> SE is an engineering discipline, yet lacking the engineering formality.
> Subjective and informal testing.
> Impossible to prove that the system:
– Does what it is supposed to do.
– Does not do what it is not supposed to do.
> Needs structured and precise system designs.
4
FORMAL METHODS
> The applied mathematics of computer systems engineering, used to specify and model the behaviour of a system and mathematically verify that the system design and implementation satisfy functional and safety properties.
> Specification Languages:
– Abstract State Machines
– Generalized State Machines
– Communicating Sequential Processes
– Specification and Description Language
– Petri Nets
– Temporal Logic of Actions
– B and event-B method
– Z
5
FORMAL METHODS AT A TRIAL
> Benefits:
– Specification may be used as a basis for proving the presence or lack of certain properties in the design and by inference in the developed system.
– Mathematical proof of correctness (Theorem proving).
– Model checking (Proving desired properties in a design).
– Formal Testing.
> Used mainly for safety critical systems such as aerospace engineering.
> Criticism:
– Expensive and time consuming approach (though questionable).
– Lack of tooling support.
6
INCORPORATING FORMAL METHODS IN THE DEVELOPMENT CYCLE
7
PROJECT FOX
> Produce the complete set of test cases from a formal specification.
> Execute the tests on the systems implementation.
> Locate errors and non-equivalences and report them to the user.
> Developed in Java for Java.
> Compatible with Java Standard Edition, Enterprise Edition, Mobile Edition.
> Can be extend to work in conjunction with popular Java frameworks.
> Operates on compiled bytecode with the addition of a few specific annotations.
> Utilizes the test drivers of JUnit.
> FoX provides a bridge between regular Java developers and the benefits of complete positive and negative testing, proven to find all faults.
8
USING PROJECT FOX
> Two artefacts necessary:
– Formal specification of the system.
– The system's implementation.
9
BUFFER CASE STUDY – DESCRIPTION
> Simple buffer in a factory.
> Accepts parts, any parts.
> Parts have a name and an ID.
> The buffer has a capacity of 2.
> The buffer can be empty, partially full or completely full.
> Supports adding and removing items.
> If the capacity is reached, no additional items can be placed in the buffer unless an item is removed first.
10
BUFFER CASE STUDY – FORMAL SPECIFICATION> Modeled as a Generalized State
Machine (stream X-Machine).
> A theoretical model of computing, pioneered by Samuel Eilenberg in 1974 (X-Machine).
> Separates flow control from processing.
> Flow control is abstracted to a level suitable for representation as a finite state machine.
> Complex data structures are modeled as an infinite memory.
> Able to model both static (data) and dynamic (control) parts of a system.
11
BUFFER CASE STUDY – FORMAL SPECIFICATION (cont.)> Simple buffer in a factory.
< xMachine name = " Buffer " >
> The buffer can be empty, partially full or completely full.
< states >
< state initialState = " true " > empty </ state >
< state > non_empty </ state >
< state > full </ state >
</ states >
12
BUFFER CASE STUDY – FORMAL SPECIFICATION (cont.)> Accepts parts, any parts.
< input name = " part " ref = " BufferObject " / >
> The buffer has a capacity of 2.
< types >
< builtInType name = " capacity " type = " integer " / >
< builtInType name = " buffer " type = " set: BufferObject " / >
</ types >
< memory >
< memoryBlock ref = " buffer " initialValue = " null " / >
< memoryBlock ref = " capacity " initialValue = " 2 " / >
</ memory >
13
BUFFER CASE STUDY – FORMAL SPECIFICATION (cont.)> Parts have a name and an ID.
< types >
< complexType name = " ItemType " >
< attributes >
< builtInType name = " type " type = " string " / >
</ attributes >
</ complexType >
< complexType name = " BufferObject " >
< attributes >
< complexType name = " type " ref = " ItemType " / >
< builtInType name = " itemId " type = " integer " / >
</ attributes >
</ complexType >
< /type >
14
BUFFER CASE STUDY – FORMAL SPECIFICATION (cont.)> Supports adding and removing items.
< functions >
< function name = " add_part " >
< guard >
!buffer. contains ( part ) && buffer . size () + 1 < capacity . value ()
</ guard >
< body > buffer . add ( part ) ; </ body >
< output > Part Added </ output >
</ function >
...
</ functions >
< transitions >
< transition >
< startingState > empty </ startingState >
< appliedFunction > add_part </ appliedFunction >
< endingState > non_empty </ endingState >
</ transition >
...
</ transitions>
15
BUFFER CASE STUDY – IMPLEMENTATIONpublic class ItemType {
private String type;
public ItemType(String type) {
this.type = type;
}
}
public class BufferObject {
private int itemId;
private ItemType type;
public BufferObject(int itemId, ItemType type) {
this.itemId = itemId;
this.type = type;
}
}
16
BUFFER CASE STUDY – IMPLEMENTATION
> @Xmachine annotating the class representing the system modeled with the specification.
> XMachineModel – a class representing the model, containing a number of useful helper methods.
@XMachine(inputType = "BufferObject",
sampleInputs = {
"integer: 10, ItemType: (string:Box)",
"integer: 17, ItemType: (string:HeavyBox)",
"integer: 25, ItemType: (string:ReallyHeavyBox)",
"integer: 20, ItemType: (string:Dragon)",
"integer: 17, ItemType: (string:Planeswalker)",
"integer: 187, ItemType: (string:Nekrataal)",
"integer: 23, ItemType: (string:Michael Jordan)"
})
public class Buffer extends XMachineModel {
17
BUFFER CASE STUDY – IMPLEMENTATION
> @XMMemoryBlock – a field level annotation, associating Java data structures with their specification equivalents
@XMMemoryBlock(name = "buffer")
private List<BufferObject> buffer;
@XMMemoryBlock(name = "capacity")
private int capacity;
public Buffer() {
super("Buffer");
buffer = new LinkedList<BufferObject>();
capacity = 2;
}
18
BUFFER CASE STUDY – IMPLEMENTATION
> @XMFunction – a method level annotation, referencing the modeled functions implementations.
> reportOutcome( outcome: String) – one of the many helper methods of the XmachineModel class.
@XMFunction(name = "add_part")
public void addPart(BufferObject part) {
if (!buffer.contains(part) && buffer.size() + 1 <
capacity) {
buffer.add(part);
reportOutcome("Part Added");
}
}
19
BUFFER CASE STUDY – EXECUTING FOX
20
BUFFER CASE STUDY – EXECUTING FOX (implanted error)if (!buffer.contains(part) && buffer.size() + 1 <
capacity) {
buffer.add(part);
capacity++;
reportOutcome("Part Added");
}
21
BUFFER CASE STUDY – GENERATED TEST CASES> Tests report the sequence of inputs used for the specific scenario, the sequence
of expected outputs and the actual output.
> Outcome is reported to the user via the usual JUnit red / green notifications.
<tests> <test testID=”1”> <input>[ itemId: 187 type: Nekrataal ]</input> <expectedOutput>[ Part Added ]</expectedOutput> <output>[ Part Added ]</output> </test> <test testID=”2”> <input>[ itemId: 17 type: Planeswalker, itemId: 20 type: Dragon]</input> <expectedOutput> [ Part Added, Part Added – Become Full ] </expectedOutput> <output>[ Part Added, Part Added – Become Full ]</output> </test></tests>
22
CONCLUSIONS AND FUTURE WORK
> FoX enables developers to leverage the already proven theories for formal testing.
> Provides a fully automated testing process, ranging from complete test set generation (satisfying some design for test conditions), to test preparation and execution.
> Operates on any Java based software system, being transparent to it's underlining technologies.
> Provides complete positive and complete negative testing.
> Nest steps:
– Thorough evaluation.
– An additional tool to make the specification step easier and closer to the developer, aiming to “hide” the formality as much as possible.
– NetBeans and Eclipse integration.
– A standalone X-Machine IDE providing additional related functionalities.
– Branch out to other languages and frameworks (eg. C# and .NET).
23
BIBLIOGRAPHY
> S. Eilenberg, Automate, Languages and Machines, Vol. A. Academic Press, London, 1974.
> M. Holcombe, “X-Machines as a basis for dynamic system specification,” Software Engineering Journal, vol. 3(2), pp. 69-76, 1988.
> F. Ipate and M. Holcombe, “Specification and Testing using Generalized Machines: a Presentation and a Case Study,” Softw. Test. Verif. Reliab, vol. 8, pp. 61-81, 1998.
> M. Holcombe and F. Ipate, Correct Systems: Building a Business Process Solution. Springer, Applied Computing Series, November 1998.
> G. Eleftherakis and A. Cowling, “An Agile Formal Development Methodology,” in 1st South Eastern European workshop on Formal Methods (SEEFM 03), (Thessaloniki), pp. 36-47, Nov. 2002. Agile Formal Methods: Practical, Rigorous Methods for a changing world.
> P. Kefalas, G. Eleftherakis, and E. Kehris, “Communicating X-Machines: a practical approach for formal and modular specification of large systems,” Information and Software Technology, vol. 45, pp. 269-280, Apr. 2003.
