Presentation Of Mbt Tools

Muhammad Husnain & Muhammad WaMuhammad Husnain & Muhammad Waseemseem

11

Model Based Testing Model Based Testing Tools Tools

International Islamic UniversityInternational Islamic UniversityDepartment of Computer ScienceDepartment of Computer SciencePresenters:Presenters:Muhammad HusnainMuhammad Husnain && Muhammad WaseemMuhammad Waseem

22Muhammad Husnain & Muhammad WaMuhammad Husnain & Muhammad Waseemseem

OutlineOutlineIntroductionIntroduction What is MBTWhat is MBT

How mostly MBTT WorksHow mostly MBTT Works

Benefits of MBTTBenefits of MBTT

Problems of MBTTProblems of MBTT

Current State Tools in IndustryCurrent State Tools in Industry

Software Tools for MBTSoftware Tools for MBT

ConclusionConclusion

ReferencesReferences


ModelModel Based TestingBased Testing

One way to generate test casesOne way to generate test cases automatically is automatically is “model-based testing” where a model of the system is “model-based testing” where a model of the system is used for test case generation. used for test case generation.


ModelModel Based TestingBased Testing

““Model-based testing is a testing technique where the Model-based testing is a testing technique where the runtime behaviorruntime behavior of an implementation under test is of an implementation under test is checked againstchecked against predictions made by a formal predictions made by a formal specification, orspecification, or model model.”.” - Colin Campbell, Microsoft - Colin Campbell, Microsoft ResearchResearch

In other words:In other words: A model describes how a system should behave in A model describes how a system should behave in

response to an action.response to an action. Supply the action and see if the system responds as Supply the action and see if the system responds as

you expect.you expect.


ModelsModels

Simply, a model of software is a depiction of its behavior. Behavior Simply, a model of software is a depiction of its behavior. Behavior can be described in termscan be described in terms of the of the

input sequences accepted by the system, input sequences accepted by the system, the actions, the actions, conditions, conditions, output logic, or theoutput logic, or the flow of data through the application’s flow of data through the application’s

modules and routines.modules and routines.

BBy y UUsing sing MModeling odeling a shareable, reusable, precise description of the a shareable, reusable, precise description of the systemsystem is acquired. is acquired.


Models İn SW TestingModels İn SW Testing

Finite State MachinesFinite State Machines

StatechartsStatecharts

UMLUML

Markov ChainsMarkov Chains

GrammarsGrammars


77

How Mostly MBTT How Mostly MBTT Works?Works?


How MBT Works?How MBT Works?

The model is usually an The model is usually an abstract, partial presentation abstract, partial presentation of the system under test's of the system under test's desired behavior. desired behavior. The test cases derived from The test cases derived from this model are functional this model are functional tests on the same level of tests on the same level of abstraction as the model. abstraction as the model. These test cases are These test cases are collectively known as the collectively known as the abstract test suiteabstract test suite


How MBT Works? Contd.How MBT Works? Contd.

The abstract test suite cannot The abstract test suite cannot be directly executed against be directly executed against the system under test the system under test because it is on the wrong because it is on the wrong level of abstraction. level of abstraction. Therefore an Therefore an executable test executable test suitesuite must be derived from must be derived from the abstract test suite that the abstract test suite that can communicate with the can communicate with the system under test. system under test. This is done by mapping the This is done by mapping the abstract test cases to abstract test cases to concrete test cases suitable concrete test cases suitable for execution. for execution.


How MBT Works? ContHow MBT Works? Cont

Model

Implementation

Test OracleTest Cases

Run Providesactual results for

Provides expected results forGenerates

PassFail


1111

Problems of MBTProblems of MBT


Problems of MBTProblems of MBT


Limitations of Executable UML methodology:Limitations of Executable UML methodology:

relative immaturityrelative immaturity

limited availability of toolslimited availability of tools

lack of a concrete standard syntax for lack of a concrete standard syntax for action languages. action languages.

We believe that these We believe that these shortcomings will be shortcomings will be

solved as the technology solved as the technology matures.matures.


Motivation behind MBTTMotivation behind MBTT

Testing is expensiveTesting is expensive 30-50% of development costs30-50% of development costs

Poor testing is VERY expensivePoor testing is VERY expensive Down timeDown time Maintenance costsMaintenance costs ReworkRework Law suitsLaw suits

Model Based Testing promisesModel Based Testing promises Increased effectiveness of testingIncreased effectiveness of testing Similar or decreased costsSimilar or decreased costs Reuse of design artifactsReuse of design artifacts


1515

Benefits of MBTBenefits of MBT


Benefits of MBT Contd.Benefits of MBT Contd.

CostCost A typical test engineer A typical test engineer

Earns €41 500 ($50 000 USD) per yearEarns €41 500 ($50 000 USD) per year

Works 40 to 50 hours per weekWorks 40 to 50 hours per week A typical test machineA typical test machine

Costs €250 ($300 USD) per year to buy and Costs €250 ($300 USD) per year to buy and operateoperate

Can work 100 hours per weekCan work 100 hours per week


Benefits of MBTBenefits of MBTShorter schedules, lower cost, and better qualityShorter schedules, lower cost, and better qualityEnhanced communication between developers and testersEnhanced communication between developers and testersEarly exposure of ambiguities in specification and designEarly exposure of ambiguities in specification and designCapability to automatically generate many non-repetitive and useful Capability to automatically generate many non-repetitive and useful teststestsTest harness to automatically run generated testsTest harness to automatically run generated testsEases the updating of test suites for changed requirementsEases the updating of test suites for changed requirementsCapability to evaluate regression test suitesCapability to evaluate regression test suitesCapability to assess software qualityCapability to assess software quality

These benefits all require an These benefits all require an initial investment in tools and initial investment in tools and

training.training.


1818

Current State of MBT in Current State of MBT in IndustryIndustry


Current State of MBT in IndustryCurrent State of MBT in Industry

IBM ResearchIBM Research Most projects were done by PhD HoldersMost projects were done by PhD Holders Very Little was successful (Just in the Lab, Very Little was successful (Just in the Lab,

never actually went into market)never actually went into market) Too Complicated to use and analyzeToo Complicated to use and analyze

Microsoft ResearchMicrosoft Research The Spec# programming system is a new The Spec# programming system is a new

attempt at a more cost effective way to attempt at a more cost effective way to develop and maintain high-quality software develop and maintain high-quality software


Current State of MBT in IndustryCurrent State of MBT in Industry

Spec# is a computer language which includes Spec# is a computer language which includes and extends C#and extends C# Adds pre/post conditions, contractsAdds pre/post conditions, contracts Adds high-level data types with convenient notationsAdds high-level data types with convenient notations Adds logical quantifiers like FORALL and EXISTSAdds logical quantifiers like FORALL and EXISTS

A Spec# model is just a program! A Spec# model is just a program! You can run it like a C# programYou can run it like a C# program It can call framework codeIt can call framework code You can explore it (run all of its possible behavior) You can explore it (run all of its possible behavior)

which is how we do MBT with Spec#which is how we do MBT with Spec#


Current State of MBT in Industry Current State of MBT in Industry (Spec Explorer)(Spec Explorer)

A model exploration and testing tool for .NET: A model exploration and testing tool for .NET: Authoring of models in Word Authoring of models in Word Model exploration (generating FSM)Model exploration (generating FSM) FSM visualizationFSM visualization FSM traversal and test suite generationFSM traversal and test suite generation Automatic implementation bindingAutomatic implementation binding Online test-suite execution and offline test-Online test-suite execution and offline test-

suite code generationsuite code generation


2222



Tools we are going to discuss:Tools we are going to discuss:

Conformiq Test GeneratorConformiq Test Generator

Leirios Test GeneratorLeirios Test Generator

Reactis TesterReactis Tester

TorXTorX

AsmLAsmL

MaTeLoMaTeLo


SpecTestSpecTestTOSTERTOSTERBridge PointBridge PointiUMLiteiUMLiteProsasimProsasimPathMATEPathMATE

Tools we are going to discuss:Tools we are going to discuss:



Conformiq Test GeneratorConformiq Test Generator is a model- is a model-based testing tool using based testing tool using UMLUML state charts state charts that represent testing strategies that represent testing strategies


Leirios Test GeneratorLeirios Test Generator

Leirios Test Generator is a model-based Leirios Test Generator is a model-based testing tool that generates tests testing tool that generates tests automatically from deterministic system automatically from deterministic system specifications specifications


Reactis TesterReactis Tester

Reactis Tester is another model-based Reactis Tester is another model-based testing tool that focuses on control testing tool that focuses on control systems systems

Works with MATLABWorks with MATLAB

Tool downloaded AND availableTool downloaded AND available


AsmLAsmL

AsmL Test Tool can generate tests directly AsmL Test Tool can generate tests directly from an AsmL model from an AsmL model


AsmL is the Abstract State Machine Language. AsmL is the Abstract State Machine Language. It is an executable specification language based It is an executable specification language based on the theory of on the theory of Abstract State MachinesAbstract State Machines. The . The current version, AsmL 2 (AsmL for current version, AsmL 2 (AsmL for Microsoft .NET), is embedded into Microsoft Microsoft .NET), is embedded into Microsoft Word and Microsoft Visual Studio.NET. It uses Word and Microsoft Visual Studio.NET. It uses XML and Word for literate specifications. XML and Word for literate specifications. It is fully interoperable with other .NET It is fully interoperable with other .NET languages. AsmL generates .NET assemblies languages. AsmL generates .NET assemblies which can either be executed from the command which can either be executed from the command line, linked with other .NET assemblies, or line, linked with other .NET assemblies, or packaged as COM components. packaged as COM components.

AsmLAsmL


TOSTERTOSTER

TOSTERTOSTER ((TThe he OObject-oriented bject-oriented SSoftware oftware TTesting esting EnvironmentEnvironment))

a test generation and execution system a test generation and execution system produced by the Warsaw University of produced by the Warsaw University of Technology. It incorporates technology for Technology. It incorporates technology for mapping the information in UML diagrams to the mapping the information in UML diagrams to the source code of an application. It also generates source code of an application. It also generates and runs test cases based on expected results and runs test cases based on expected results derived from the UML state diagrams. There derived from the UML state diagrams. There appear to be two test generation algorithms, but appear to be two test generation algorithms, but no explicit testing directives.no explicit testing directives.


SpecTestSpecTest

SpecTestSpecTest is an automatic test case is an automatic test case generator from George Mason University. generator from George Mason University. It accepts models written in SCR or UML, It accepts models written in SCR or UML, and generates test cases based on a and generates test cases based on a choice of two coverage criteria, with a choice of two coverage criteria, with a further two planned for implementation.further two planned for implementation.


iUMLiteiUMLite

www.kc.comwww.kc.com

iUML Modeler that providing users with the iUML Modeler that providing users with the ability to build complete and consistent ability to build complete and consistent executable UML models.executable UML models.

iUML Simulator that providing users with iUML Simulator that providing users with the ability to execute, test and debug the ability to execute, test and debug these executable UML models.these executable UML models.


Limitations of Limitations of iUMLiteiUMLite free version: free version:

Capacity restrictions are placed on the Capacity restrictions are placed on the models that can be constructed andmodels that can be constructed and

executed, specifically a maximum of:executed, specifically a maximum of:

--) 3 Versions per Domain--) 3 Versions per Domain

--) 15 Classes per Domain version--) 15 Classes per Domain version

cont……..cont……..


Documentations can only be produced in HTML formatDocumentations can only be produced in HTML formatThe import/export facilities for moving models and The import/export facilities for moving models and components between databases are not available.components between databases are not available.Only one user at a time may access a given database.Only one user at a time may access a given database.The code generator provided can only be used for the The code generator provided can only be used for the purposes of executing the models in the iUML Simulator. purposes of executing the models in the iUML Simulator. It cannot be used to deliver target software.It cannot be used to deliver target software.Similarly, iUMLite does not come with the iCCG Similarly, iUMLite does not come with the iCCG Configurable Code Generation capability.Configurable Code Generation capability.The freely available copy of their tool is not capable of The freely available copy of their tool is not capable of any code generation though.any code generation though.

Limitations of Limitations of iUMLiteiUMLite free version: free version:


How to get LicenseHow to get License

It is not greatly easy to acquire an evaluation It is not greatly easy to acquire an evaluation version of this tool. In order to evaluate the version of this tool. In order to evaluate the software, a form has to be filled from the software, a form has to be filled from the company’s web-site. After filling and submitting company’s web-site. After filling and submitting the form, the company will again enter in contact the form, the company will again enter in contact with the prospective evaluator in order to ask with the prospective evaluator in order to ask further questions. The license key given is tied to further questions. The license key given is tied to the serial number of the hard disk, therefore it the serial number of the hard disk, therefore it can’t be transferred to another computer.can’t be transferred to another computer.


TestMasterTestMaster

TestMaster is an automated test program TestMaster is an automated test program generator that includes a GUI generator that includes a GUI development environment that is ideal for development environment that is ideal for building models at both abstract and building models at both abstract and implementation dependent levels.implementation dependent levels.

http://www.teradyne.com/prods/sst/http://www.teradyne.com/prods/sst/product_center/sstprod.htmlproduct_center/sstprod.html

http://www.dnaent.comhttp://www.dnaent.com


MaTeLoMaTeLo

Usage Model Editor Usage Model Editor is a first MaTeLo module which is a first MaTeLo module which let you create a project, design and edit a usage model let you create a project, design and edit a usage model of the system under test. of the system under test.

Testor Testor is the second MaTeLo module which let you is the second MaTeLo module which let you automatically generate test cases by browsing the automatically generate test cases by browsing the usage model and following one path or another by usage model and following one path or another by randomly selecting transitions according to their randomly selecting transitions according to their occurring probability. Testor finally can help you occurring probability. Testor finally can help you manage the test campaign and get some information manage the test campaign and get some information about it such as coverage, system reliability or Mean about it such as coverage, system reliability or Mean Time To Failure. Time To Failure.


What is a usage model? What is a usage model? A usage model represents the interactions between a system and its users..

A MaTeLo usage model is an extended Markov chain for test and validation purposes. It’s a state diagram where a transition from a state to another is related to a system input and its activation frequency in the current state.

A chain is made of 2 specific states: Invoke and Terminate. Those states usually represent the system start and stop.

Chains can be nested, in order to get a hierarchical model.


USAGE MODEL STRUCTURE

When to perform the MaTeLo model?

Ideally, the best moment to design your usage model is when you’ll have written about 2/3 of your system’s specifications.

Eventually, you can update your model just before the test campaign to ensure that your model consistency with the specification at this time.


Cont…Cont…

What’s the usage model use?

The usage model lets you generate test sequences made of stimuli to perform on the system as well as the results to expect from the system. A model design stands for a test plan design


How to model?

The most useful entry documents for MaTeLo modeling are the system’s functional specifications. Ideally, this specification will be made of Use Cases, which will dramatically reduce the system understanding effort.

modeling should consider the main validation aims, in order to better focus on those aims. Such aims could be: reliability, performance, robustness … A chain should not be too sequential (wrong model) and avoid redundancy.


Elementary Modeling Rules: Elementary Modeling Rules:

There can only be one transition going out the A There can only be one transition going out the A state and entering the B state. (except for state and entering the B state. (except for conditional transitions) conditional transitions)

No transition should enter the Invoke state, and No transition should enter the Invoke state, and no transition should go out from the Terminate no transition should go out from the Terminate state. state.

A sub-chain respects the same design rules as A sub-chain respects the same design rules as the main chain.the main chain.


Profiles Management

It is possible to use one or more usage profiles . Profiles make it possible to model various behaviors of SUT and thus obtain different tests according to the usage frequencies allocated to the transitions according to the selected profile.


Testor

Testor is the second module of the MaTeLo system dedicated to the automatic generation of the Test Plans and to the Test Campaign Analysis.


Typical use of Testor

1.Use Testor to open a test project created by UM Editor.

2. Use the first panel to do the Model Checking and request a MC reporting, to analyze the parameters (stationary distribution, test case length and visiting probability).


Cont…Cont…

3.Use the second panel to generate a test suite. It’s possible to generate:

Most probable test suite

Random test suite

Limit test suite

Minimum test suite (only if your model does not include asynchronous or conditional elements)


Cont..Cont..

4. Use the third panel to enter manually or import the test execution for each system version of the SUT. Debugging test: not described by a test suite, debugging test is a simple test like “launching the application” Most probable test: one test for the most probable usage of the SUTRandom Test (xN campaign) Limit Test Minimum test Adding test: additional test, no generate by MaTeLo


Conformiq

Conformiq Qtronic is a tool for both online testing of systems and for offline generation of test scriptsIt can use the model to directly test a running system or to generate test scripts that can be independently executed afterwards.Generated test scripts can be stored in version control system tests can be sent to colleagues or to execute them independently


4949

Are You Feeling Like ThisAre You Feeling Like This

OK….OK….Let us Finish……….Let us Finish……….


5050




Modeling is not easyModeling is not easy What details do you leave out?What details do you leave out? How do you check the model?How do you check the model? There are no recipes for successThere are no recipes for success

What needs to happenWhat needs to happen Easier notationEasier notation Incremental approachIncremental approach Pilots and examplesPilots and examples


QuestionsQuestions

Does tool cover stress testing?Does tool cover stress testing?Does Tool cover how user friendly the system is Does Tool cover how user friendly the system is or it only covers functionality?or it only covers functionality?How do we guarantee that the generated How do we guarantee that the generated expected outputs and inputs are enough and expected outputs and inputs are enough and cover all the cases? cover all the cases? How to make for the errors and exceptions that How to make for the errors and exceptions that may occur due to running the program on may occur due to running the program on different environments and different OSs?different environments and different OSs?How is automated generation of inputs and How is automated generation of inputs and outputs work when an actual human interaction outputs work when an actual human interaction is required as part of the system testing?is required as part of the system testing?


5353




1.1. http://www.goldpractices.com/index.phphttp://www.goldpractices.com/index.php2.2. http://en.wikipedia.org/wiki/Model-based_testinghttp://en.wikipedia.org/wiki/Model-based_testing3.3. http://blogs.msdn.com/nihitk/articles/144664.aspxhttp://blogs.msdn.com/nihitk/articles/144664.aspx4.4. http://research.microsoft.com/specsharp/http://research.microsoft.com/specsharp/5.5. http://react.cs.uni-sb.de/mbt2006/talks/mbt4http://react.cs.uni-sb.de/mbt2006/talks/mbt4massesmasses.pdf.pdf6.6. http://react.cs.uni-sb.de/mbt2006/talks/http://react.cs.uni-sb.de/mbt2006/talks/

ModelBasedTestingSoberEvaluation.pdfModelBasedTestingSoberEvaluation.pdf7.7. http://aetgweb.argreenhouse.com/papers/1999-icse.pdfhttp://aetgweb.argreenhouse.com/papers/1999-icse.pdf8.8. www.agedis.dewww.agedis.de9.9. Colin Campbell, Wolfgang Grieskamp, Colin Campbell, Wolfgang Grieskamp,

Lev Nachmanson, Wolfram Schulte, Nikolai Tillmann, Margus Lev Nachmanson, Wolfram Schulte, Nikolai Tillmann, Margus Veanes “Spec Explorer:Veanes “Spec Explorer:An Integrated Environment for Model-Based Testing.” An Integrated Environment for Model-Based Testing.” Foundations of Software Engineering Microsoft Research, Foundations of Software Engineering Microsoft Research, RedmondRedmond



9.9. EXECUTABLE UML by Selo Sulistyo and Warsun NajibEXECUTABLE UML by Selo Sulistyo and Warsun Najib

10.10. ““Towards Executable Aspect-Oriented UML models” By Towards Executable Aspect-Oriented UML models” By “Lidia Fuentes and Pablo Sánchez” ,March 12-13, 2007.“Lidia Fuentes and Pablo Sánchez” ,March 12-13, 2007.

11.11. http://www.eclipse.org/uml2/http://www.eclipse.org/uml2/

12.12. http://www.magicdraw.comhttp://www.magicdraw.com

13.13. ““Model Driven TestingModel Driven Testing“ By Brian Berger, “ By Brian Berger, Majdi Abuelbassal, and Mohammad Majdi Abuelbassal, and Mohammad Hossain, 1997Hossain, 1997

14.14. VP-UML User’s Guide (Part 1)VP-UML User’s Guide (Part 1)


5656

ThanksThanks

Documents

Presentation Of Mbt Tools