2012.11 AutomatedSoftwareTestingMagazine November2012

8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012

1/44

......

MAGAZIN

OCT/NOV 2012$8.95

How To Make theMost of Model-based Testing

PRACTICAL TIPS AND CASESIN MODEL-BASED TEST

IMPLEMENTATION

CROSSING OVER:

Implement Cross-browser Testing in 4 Simple Steps

Job RotationINVOLVING USERS IN

AUTOMATOR TASKS ANDVICE VERSA

Randomness& Rotations

Randomness& Rotations

SOFTWARETESTINGAUTOMATEDAUTOMATED

AnAUTOMATEDTESTINGINSTITUTEPublication - www.automatedtestinginstitute.com

Random TestsSENDING RANDOM INPUTSTO THE SYSTEM UNDERTEST


2/44

Free Demo

& Trialwww.gotosca.com/testkit

TOSCA puts the "A" in Agile

AGILE processes are strictly risk-oriented

TOSCA risk-based approach

is the ideal complement

AGILE requires early test automation

TOSCA OneView (UI/Non-UI)

is the perfect solution

AGILE is based on specification by example

TOSCA helps provide the right test cases

Ask us how TOSCA can empower youto increase your test automationcoverage up to 85% while minimizingyour maintenance effort up to 80%.

One Solution For GUI, non-GUI, Manual,

Automated, X-Browser and Mobile.

Passion for Testing


3/44

AUTOMATEDSOFTWARETESTINGOct/Nov 2012, Volume 4, Issue 3

Automated Software Testing Magazinewww.automatedtestinginstitute.comOct/Nov 2012

CONTENTS

COLUMNS& DEPARTMENTSEDITORIAL 4

AUTHORS AND EVENTS 6 MOBILE HOSTING & AUTOMATION

SOLUTIONS ARE HOT! 41

I BLOG TO U 36Randomize and Rotate

A Host of Choices in Mobile

Meeting the challenges posed by system dynamics withsome dynamics of our own.

A look at commercial mobile hosting and automatiosolutions.

Read featured blog posts from the web.

Learn about AST authors and upcoming events.

OPEN SOURCERY 12Randys Favorite Free and Cheap Tools

An update to a previous article on free and cheap toolswith information shared at the TestKIT 2012 Conference.

GO ON A RETWEET 38Read featured microblog posts from the web.

RANDOMNESS& ROTATIONSGiven the highly dynamic nature of software test applications, its imperative that our approaches not become to

static. This issue focuses on improving test automation implementation and application quality through increase

data and tests randomness and job rotations.

FEATURESRANDOM TESTS 14Random Test is a special test type that has been proven to be a very good complement to ordinary functional tests. Read this article to learn ho

this technique may prove useful, particularly for systems with high criticality such as marketplace systems. By Lars Wahlberg

HOW TO MAKE THE MOST OF MODEL-BASED TESTING 20Model-based testing (MBT) has the potential to resolve many of the challenges of test automation including maintainability, cost contr

and keeping pace with development. Read this article to obtain a workable MBT workow. By Stefan Mohacsi and Armin Be

TESTKIT TIP 8Crossing OverBy Goran Begic

Implement cross-browser testing in 4 simple steps.

JOB ROTATION: THE ANSWER TO QUALITY AUTOMATED TESTING? 30

Automated testing could benet greatly from more involvement by the front-end user if we could gure out how to get around the facthat the front-end user is often not very technical. This article provides a possible way around this hurdle. By Ursula Friede


4/444 Automated Software Testing Magazine www.automatedtestinginstitute.com Oct/Nov 2012

Often test automators push automated test

scripts into combat with the assumption

that the application is simply going to

let the script win. We assume that data

never gets corrupted, scripts never need

to be maintained, unexpected events never

happen, and each test objective has only a

single path. For this reason, scripts often fail

easily, require excessive maintenance, and

repeatedly overlook simple defects.

Application behavior is dynamic, so its

time that we train our automated scripts to be

more dynamic via the techniques provided inthe automation martial arts known simply as

Taekwondo-mation! Taekwondo-mation

comprises relatively simple techniques that

can be employed without the implementation

of an overly complex framework.

This began a four-part series of articles

I wrote a few years ago entitled

Taekwondo-mation - Training Your

Scripts for Dynamic Combat. A

few months ago, I met a guy that was

familiar with some of my writings, and

specically expressed his partiality formy Taekwondo-mation series. He also

indicated how much hed like for me

to continue the series with some new

articles. Aside from being attered that

some of my writings have made a lasting

impact, I was also intrigued by this

impact. It made me begin to question

why, out of all of the works that Ive

done, did the gentleman specically

bring up Taekwondo-mation? The more

I thought about it, the more I began to

come to the conclusion that it has todo with the fact that it addressed the

dynamic nature of software. And since

software has only gotten moredynamic

since Taekwondo-mation, and promises

to continue in that direction, the articles

are in a sense, timeless. Change is the

only constant when it comes to software,

so people are constantly looking for

information that will help them to deal

effectively with it. So, I will denitely

consider writing some more of my own

articles in the future that present more

aspects of the test automation martial

arts, but in the meantime, this issue of

the AST Magazine has enlisted some

other automation martial artists to

present their spin on effectively handling

software and automation dynamics.

The rst feature entitled Random

Tests by Lars Wahlberg describes how

one team introduced a new automation

approach, whose name is equivalent

to the fedatures title, in an effort to

supplement testing in a marketplace

systems environment where even

thousands of functional tests werent

sufcient for validating the application.

Following behind that feature is an even

more in-depth look at a similar approach

known as model-based testing (MBT),

which is a type of automated testing

approach that dynamically generates

and executes tests in real-time. This

feature article, entitled How to Make

the Most of Model-based Testing,is coauthored by Stefan Mohacsi and

Armin Beer, and presents a workow

for MBT that considers the traceability

between requirements, the test model,

the GUI, and the test cases. Finally,

Ursula Friede presents an article

entitled Job Rotation: The answer

to quality automated testing? In this

article, Friede provides an approach for

improving quality and cross-training

resources in a cost effective manner.

Randomize and Rotateby Dion Johnson

Editorial

It made me begin to question why, out

of all of the works that Ive done, did

the gentleman specically bring up

Taekwondo-mation?


5/44

Celebrating Excellence in the

Discipline of Software Test

Automation

Winners Now Announced!

www.atihonors.automatedtestinginstitute.com



Managing Editor

Dion Johnson

Contributing Editors

Donna VanceEdward Torrie

Director of Marketing and Events

Christine Johnson

A PUBLICATION OF THE AUTOMATED

TESTING INSTITUTE

CONTACT US

AST Magazine

[email protected]

ATI Online Reference

[email protected]

Stefan Mohacsi studied computerscience at the Technical University of Vienna.

In 1997, he joined Siemens and became project

manager of the research project IDATG. Today,

Stefan is senior consultant for model-based

testing at Atos. In addition, he is a member of the

Austrian Testing Board and has held numerous

lectures at international test conferences. Stefans

email address is [email protected].

Whos In This Issue?

Goran Begic is a senior productmarketing manager at SmartBear Software. He

brings 13+ years of professional experience

with various code and design verication tools

ranging from manual testing and test automation

to static analysis and formal verication. Goran

has extensive experience with integratingdevelopment and testing tools into end-user

processes including agile and Model-Based Design. He is an avid customer

advocate and a proponent of visual thinking. Goran holds a masters degree

in electrical engineering from the University of Zagreb, Croatia. Connect

with him via Twitter @gbegicw.

Authors and Events

AUTOMATEDSOFTWARETESTING

ATI and Partner Events

First Quarter 2013

TestKIT ONDEMANDVirtual Conferencehttp://testkitconference.com

November, 2012

AST Magazine Special Editionhttp://astmagazine.automatedtestinginstitute.com/

Ursula Friede has 10 years ofexperience testing software. She worked as

a performance specialist and automated test

analyst, but is currently on maternity leave.

Friede has contributed a chapter to the new test

automation book by Mark Fewster and DorothyGraham and she has a publication about Case

based reasoning in a German IT book.

The Automated Software Testing (AST) Magazine is an

Automated Testing Institute (ATI) publication.

For more information regarding the magazine visit

http://www.astmagazine.automatedtestinginstitute.com

Armin Beer has been working in thearea of test management and test automation for

about 20 years. He is currently an independent

consultant for the test management group at

the social insurance institution BVA. He is alsoparticipating in the research project SoftNet of the

Technical University of Graz and lecturing at the

University of Applied Sciences in Vienna and the

Technical University of Graz. Armins email address is [email protected].

Lars Wahlberghas been working withtesting of marketplace systems for about 10 years

in roles such as test engineer, test manager/leader,

and test strategist. Before that, during the 1990s,

he worked with testing of ight-control systems

for military aircrafts and has a MSc degree in

aerospace. He also teaches ISTQB test foundation

courses in Stockholm with Hans Schaefer.


7/44www.automatedtestinginstitute.comOct/Nov 2012 Automated Software Testing Magazine

Are You ContributingContent Yet?

The Automated Testing Institute relies heavily on the automated testing

community in order to deliver up-to-date and relevant content. Thats why

weve made it even easier for you to contribute content directly to the ATI

Online Reference! Register and let your voice be heard today!

CommunityComments Box

As a registered user you can submit content directly to the site, providing you w

content control and the ability to network with like minded individuals.

>>COMMUNITYCOMMENTSBOX- This comments box, available on the home pagethe site, provides an opportunity for users to post micro comments in real time.

>>ANNOUNCEMENTS& BLOGPOSTS- If you have interesting tool announcementsyou have a concept that youd like to blog about, submit a post directly to the A

Online Reference today. At ATI, you have a community of individuals who would lo

to hear what you have to say. Your site prole will include a list of your submit

articles.

>> AUTOMATION EVENTS - Do you know about a cool automated testing meetwebinar or conference? Let the rest of us know about it by posting it on the ATI s

Add the date, time and venue so people will know where to go and when to be the

Announcements &

Blog Posts

Automation

Events Learn more today at http//www.about.automatedtestinginstitute.com



Crossing OverImplement Cross-browser Testing in 4 Simple Steps

TestKIT Tip

Differences in web browsers may cause an application that

behaves correctly in one browser to behave incorrectly in a

different browser or browser version. The structures of different

vendors browsers are different as are their user interfaces and

inner object hierarchies often even for different versions of

the same browser. Application developers must therefore take

these differences into account and so too must the tests that

verify web applications. Otherwise a test recorded in a version

of Internet Explorer will most likely not work right out of the

box in a version of Mozilla Firefox.

These differences are more than skin deep. Tests that only

generate a series of screen-shots of applications running

in various browsers and operating systems can only verify

that everything displays correctly. They cant verify how

the program logic itself behaves or reacts to user actions

in different browsers. Testing the logic means testing an

applications various objects, methods and properties, either

directly (i.e., with conditional logic), with generic object

mapping, or with a combination.

Testing how program logic performs in multiple browsers

requires four steps, each of which corresponds to one of the

reasons the same application might behave differently in a

different browser or browser version. If we assume that thetest knows how to get a list of installed browsers from the

Windows registry and launch them programmatically, these

four steps are:

1. Test non-application-related differences

2. Test application-related differences (GUI)

3. Test application-related differences (non-GUI)

4. Test use case differences

In these examples we will use JScript, although the same

principles apply using any object oriented scripting languageor (as covered in step 4), a keyword based testing tool.

STEP1: TESTNON-APPLICATION-RELATEDDIFFERENCES

Some browser actions are unrelated to how a web application

is constructed. If the test encounters one of these actions

and has not taken that possibility into account, the test will

probably crash or report erroneous results. An example of

this type of behavior is Internet Explorers check for active

content. If a page contains active content the Internet Explorer

browsers may display a notication message that asks the user

TestKIT is the name used by ATIfor describing ones testing toolkit.A TestKIT is lled with knowledge,information and tools that go withus wherever we go, allowing ourprojects and organizations toquickly reap the benets of thepractical elements that weveamassed. This section provides tipsto add to your TestKIT.

By Goran Begic

Figure 1: StockTrader Website



if the content is permitted, whereas the Mozilla browsers do

not notify the user in the same situation.

Another example is the update modal dialog that appears when

Firefox launches if it detects that a software update to the

browser is available for download and installation. This dialog

does not appear in either Internet Explorer or Chrome. The test

must rst determine which browsers methods and propertiesare currently available. For example, if the possibilities are

just a version of IE or Firefox, you can simply test for Firefox

since anything else will be IE:

Depending on your tools, you can either use browser-specic

names for objects or else use wrappers to provide commonnaming to access equivalent objects that have different names

in different browsers. In general, conditional logic is a lot

easier to code, and that code tends to be much more reliable,

if you use generic (i.e., non-browser-specic) methods and

properties. Thats why TestComplete provides a unied object

model (used here) with generic objects mapped to browser-

specic objects.

In the code above, aqStringis an object, whose method, Find,

looks for occurrences of specied strings. It works with the

Page wrapper object that has methods and properties that

are common for all on-screen objects and methods across

browsers. It adds a number of specic methods and properties

(e.g., browser.FindChild, modalDlg) that let you navigate

through web pages in a browser, search for elements of a web

page and perform other operations.

The isMozillafunction enables you to build another function,

PrepareForBrowser , that tests (in the case of Firefox only)whether the update notice is displayed:

If this is Firefox and the value Firefox Update is found, the

test re-launches the browser (which will install the update

automatically) and the test can then continue.

TestKIT T

Figure 2: Discussion Forum Page

Because different

browsers have

different GUIs, the

way they interact

with users will be

different, even to

accomplish the

same task.



STEP2: TESTAPPLICATION-RELATEDDIFFERENCES(GUI)

Because different browsers have different GUIs, the way they

interact with users will be different, even to accomplish the

same task. For example, the location and program hierarchy of

the Addressedit box is different for each browser. So opening

a page by entering the desired URL via the user interface isGUI-specic and requires separate instructions for each GUI

each of which would need to be tested. But, again, by using

a Pagewrapper method in this case, Page.ToUrl you can

open each supported browser with a single script instruction.

Or suppose you want to record a click over the Backbutton,

this will also be a GUI-specic action and you would need

separate instructions for each GUI. However, if you refer to

the engines methods, then only two code branches will be

needed.

STEP3: TESTAPPLICATION-RELATEDDIFFERENCE(NON-

GUI)

Other aspects of an application are not GUI-related (hence,

beyond the scope of the screenshot type tests mentioned

earlier). A test, for example, might need to verify that links on

a web page actually open the correct pages or that a request for

a browser modal dialog is successful.

Here is a test script portion that accounts for differences in

how various browsers respond to an applications request for

a modal dialog. The code uses conditional logic to identify

the current browser. It then asks if the object, modalDlg,

exists and, depending on the answer, writes either a success

or fail message to the test log. As in the previous test steps,

its an example of how using generic objects across browsers

streamlines test coding.

The code takes into account the following browser specics:

in Firefox version 3 and earlier the modal dialogs have the

JavaScript Application caption and are Window objects;

in Firefox 4, 5 and 6 modal dialogs have the same caption

but are vbox XUL objects; in Internet Explorer modal dialogs

are entitled as Explorer User Prompt. Due to Microsofts

Loosely Coupled IE (LCIE) concept, the modal dialogs in

Internet Explorer 8 and 9 are invoked in the helper tab process.

Hence, we need to search for the dialog object within the entire

Sys object rather than only within the browser process.

STEP4: TESTUSECASEDIFFERENCES

Some differences in how applications behave in different

browser only show up because users behave differently in

different browsers. For example, the user might always click

on a certain button when using Internet Explorer, but not click

on it when using Chrome. Suppose we want to verify the value

of Cash Balance in the StockTrader website (Figure 1)

always equals $89,984.05 in all browsers. But for IE only, we

also want to verify that clicking on the Discussion Forum

button loads that page (Figure 2). Further, we want to test that

the Ask a Question box is enabled once the page loads.

To show how the same general approach conditional logic with

generic object naming applied works in a keyword-driven test

tool with no manual scripting, this time well use TestComplete 9.Figure 3: Recorded Keyword Web Test



Figure 3 shows the test, which TestComplete has constructed

automatically by recording user actions in the applications

GUI: entering a user name and password, clicking the

submit button, clicking the log out button and closing the

browser window. The test is browser-independent because

TestComplete employs a unied object model for web

browsers and pages. The Browser node contains Page objectsthat provide browser-independent access to the pages (e.g.,

pageHomePage) opened in that browser.

To test that the content of Cash Balance is correct, the

TestComplete user inserts a property checkpoint after the

pageHomePage loads. (The user simply drags and drops an

Insert Checkpoint icon on top of the appropriate item on

the web page, and another insert Checkpoint icon at the

appropriate step in the test itself. This action opens a dialog

panel where the user enters the value to be check for.) Note

that the generic property contentText, automatically maps to

either innerText or textContent, the browser-specic propertyfor text on a web page in Firefox or Chrome, respectively.

Figure 4 shows the same test, this time with conditional logic

(indented) that checks whether the browser is IE and, if so,

clicks on the Discussion Forum Button, and waits for the

page to load. A property checkpoint inserted here then checks

whether the Ask a question box on the forum page is

enabled. No other changes were made in order to include this

special IE case in the test.

So, is a cross-browser test of a web application ever really

complete even if you cover all four steps covered here?Since technology and user requirements keep changing, test

completeness is probably more of a goal than an actual result.

How far you get toward that goal, however, depends on the

scope of your testing to begin with whether it does, in fact,

address all four reasons why applications behave differently in

different browsers. It also depends on whether you incorporate

strategies like generic object mapping for addressing as many

potential differences as fast as you can.

Figure 4: Recorded Keyword Test with IE Special Case

How far you get

toward that goal,

however, dependson the scope of your

testing to begin with

whether it does,

in fact, address

all four reasons

why applications

behave differently

in different

browsers. It alsodepends on whether

you incorporate

strategies like

generic object

mapping for

addressing as many

potential differences

as fast as you can.

TestKIT T



Randys Favorite Free andCheap Tools

The above quote is from an article

that Randy Rice wrote for the June

2010 issue (Volume 2, Issue 2) of

the Automated Software Testing Magazine

entitled, Free and Cheap Test Automation.

In that article, Randy Rice not only describes

the importance of and process for getting

started with free and cheap tools, he also

identies and describes some of his favorites.

In his new tutorial at the TestKIT 2012

conference, Randy provided an updated list

of tools in the following categories:

Test Design Test Execution

Test Frameworks Performance Testing and

Monitoring

Test Management Test Data Test Evaluation

Defect Tracking Special Applications

This article identies some of the execution

tools Randys list focuses on.

CUBICTEST

CubicTest is a graphical Eclipse plug-in for

writing Selenium and Watir tests. It makes

web tests faster and easier to write, and

provides abstractions to make tests more

robust and reusable.

CubicTests test editor is centered around

pages/states and transitions between these

pages/states. The model is intuitive for

both Ajax and traditional web applications

and supports most user interaction types.

CubicTest features an innovative test recorderand test runner based on Selenium RC which

are fully integrated with the graphical test

editor. Tests can also run standalone from

Maven 2.

BADBOY

Badboy is a powerful tool designed to aid in

testing and development of complex dynamic

applications. Badboy makes web testing and

development easier with dozens of features

including a simple yet comprehensive

capture/replay interface, powerful load

testing support, detailed reports, graphs and

much more!

CUCUMBER

Cucumber is a tool that executes plain-text

functional descriptions as automated tests.

The language that Cucumber understands is

called Gherkin.

While Cucumber can be thought of as atesting tool, the intent of the tool is tosupport Behavior Driven Development

(BDD). This means that the tests are typicallywritten before anything else and veried bybusiness analysts, domain experts, and othernon-technical stakeholders. The productioncode is then written outside-in, to make thestories pass.

Cucumber itself is written in Ruby, but itcan still be used to test other code writtenin Ruby and/or other languages includingbut not limited to Java, C# and Python.Cucumber does require minimal use of theRuby programming language, but Ruby is arelatively simple language.

SIKULI

Sikuli is an open source visual technology to

automate and test graphical user interfaces

(GUIs) using images (screenshots).

Sikuli includes Sikuli Script, a visual scripting

API for Jython, and Sikuli IDE, an integrated

development environment for writing visual

scripts with screenshots.

You can programmatically control a web

page, a Windows/Linux/Mac OS X desktop

application, or even an iPhone or Android

application running in a simulator or viaVirtual Network Computing (VNC).

ICUTEST

ICUTest is a free GUI Unit test tool for

Windows Presentation Foundation.

Test Generator, included with ICUTest, is a

tool that automatically creates GUI unit tests

for an application. Just specify your assembly

(executable or DLL) and Test Generator will

inspect your assembly for UI components and

generate unit tests automatically.

Open Sourcery

As a tester, I have always found it helpful to have my own toolkit that I understand andcan take with me on projects. If my client has other tools, or does not want me use my owntools, thats fine. I still like to be prepared with my own tools that I can use in a pinch if theopportunity arises.

The problem is that I dont have the same deep pockets as some companies do. So, I lookfor affordable ways to get the job done. Sometimes my clients dont have the budgets formajor commercial test tools, either. In that case, I can offer other alternatives that add to my

value as a consultant.

Randy Rice



ATILocal Chapter

Program

ATI- Meeting Local Needs In Test Automation

ATIs Local Chapter Program is established to help better

facilitate the grassroots, global discussion around test

automation. In addition, the chapter program seeks to

provide a local based from which the needs of automation

practitioners may be met.

ENHANCE THE

AWARENESS OF TEST

AUTOMATIONas a discipline

that, like other disciplines,requires continuous education

and the attainment of a standard

set of skills

OFFER TRAINING AND

EVENTSfor participation by

people in specifc areas around

the world

HELP PROVIDE

COMPREHENSIVE, YET

READILY AVAILABLE

RESOURCESthat will aid

people in becoming more

knowledgeable and equipped to

handle tasks related to testing

and test automation

Start a Local Chapter TodayEmail contact(at)automatedtestinginstitute.com to learn more

www.automatedtestinginstitute.com



M arketplace SystemFor context, a short introduction to marketplace systemsis necessary. Figure 2 is a simple diagram of a marketplace

system illustrating the trading of a nancial instrument on a

market in an order book. Examples of instruments are stocks,

bonds, indexes and commodities etc. A marketplace system

supports automatic matching of incoming orders in real time

and determines trades and order events (orders are cancelled,

updated or deleted).

An order put into the order book is either a bid (buy) or an ask

(sell). There are several types of orders, including:

Limit Order: A limit order is an order with a price. It willbe traded at the specied price or better.

Market Order: Is a buy or sell order with no price. It

will be executed at best available price.

Iceberg Order:Is an order where only a part of the order

is visible. When that quantity has been traded another

portion is put into the market.

All-Or-None Order (AoN):The order will only trade if

the total volume can be executed.

Additional order types include: Combination Orders, Linked

Orders, Fill-Or-Kill (FoK), Fall-And-Kill (Fak), Stop Loss,

Pegged, Quotes etc.

Random Testis a

special test type thathas been proven

to be a very good

complement to ordinary functional

tests for testing marketplace

systems. Figure 1 below shows an

overview of the concept. The main

idea is to send random input to the

system under test (SUT). A (Test)

Oracle listens to the output and

(but not necessarily) the randominput. The Oracle can then decide if

the actual output deviates from the

expected output.

By

LARSWAHLBER

G

it is time to analyze why andhow random tests can be used

It is a fundamental

test principle that you

cannot test everything.

Figure 1: RANDOMTESTCONCEPT

TestOracle

RandomInput Output



NDOMTESTS

Actors,

Oracles and

Analysis of

Random

Tests in a

Marketplace

System



The following notation will be used to

describe an order:

Bid/Ask quantity @ price

For example, a Limit Bid order to buy

200 shares for a stock at price 21.2

could be written as: Bid 200 @ 21.2.

An order book is represented as a table

with two columns, containing the bid

in the leftmost column, and the ask inthe rightmost column. For example, a

stock order book for companyXYZ may

appear as follows:

BID ASK

200 @ 21.2 300 @ 22.3

Now, lets walk through a sample order

event ow. Given the above order book,

if a newBid 200 @ 23.0order is entered,

in most markets, a Trade 200 @ 22.3will occur, with the resulting order book

appearing as follows:

BID ASK

200 @ 21.2 100 @ 22.3

Trading clients sometimes subscribe to

an order event ow to build up a Shadow

order book of the real order book in

the marketplace system. In the above

example, a client would have receivedthe following order events:

Insert: Bid 200 @ 21.2, ID=1

Insert: Ask 300 @ 22.3, ID=2

Insert: Bid 200 @ 23.0, ID=3

Cancel: ID=3

Update: ID=2, 100 @ 22.3

In some marketplace systems the Insert

and the immediate Cancelevents of the

order with theID=3are not published to

the clients (to save bandwidth).

With this brief introduction to marketplace

systems behind us were ready analyze

why and how random tests can be used.

R andom Test forMarketplace Systems

Marketplace systems are very often

considered critical, thus customers and

authorities require these systems to be

subjected to extensive testing. One of

the major challenges faced by those

responsible for testing these systems,

however, is the explosion of order book

situations that need to be veried. So,

while it is a fundamental test principle

that you cannot test everything, the

criticality and importance of correct

functionality for marketplace systems

often still results in thousands offunctional tests (2). Even thousands of

tests may not cover all pertinent areas,

though, which is why random tests are

an excellent complement.

The framework used on my project for

automated execution of random tests

for trading simulations was the same

framework wed already implemented

for automated functional tests; with

a few simple modications. It used

the same API libraries that were usedby the customers for development of

trading clients (i.e. tests were done with

transactions, not via a client GUI). For

more information about the specic

technical makeup of this framework,

read chapter 18 of the Experiences

of Test Automation book listed in the

References section. This article focuses

primarily on the logical design and use

of two major elements of a random test

framework: actors and oracles.

Actor DesignActors are dened as a set of actions that

can be performed on the system. While

it is the frameworks responsibility

to select actors during a simulation,

the actors are responsible for how thatsystem is actually exercised. Some of

the actors that we created included:

Trading Actor:Enters modify,

suspend/activate, and cancel limit,

Fak, FoK, market orders.

Iceberg Actor:Enters modify,

suspend/activate, and cancel

Iceberg orders.

Trading Control Actor:Suspends/

Releases order books, ush orders,

trade halt order books, suspendsother actors, etc.

State Actor:Changes trading

states between pre-trading,

continuous trading, post trading,

market closed, etc.

A probability factor was assigned to

each action belonging to an actor to

increase the likelihood of some actions

occurring. These probability settings

were congurable, however, and could

therefore be easily adjusted.

O racle DesignAn oracle is a component that reviews

the output from a test and determines,

based on that tests input, what the

expected output should be and whether

or not the actual output matches what

was expected.

There are many different approaches tothe design of the Test Oracle depending

on how advanced the situation requires

the random tests to be. Hoffman (1)

suggests the following types of Oracles:

1. No Oracle: Can only detect

crashes of the SUT, but is

simple in design.

2. True Oracle:An independentcopy of the system. Detects all

deviations, but is expensive to build.

Figure 2: MARKETPLACESYSTEM

Orders Trades OrderEvents



3. Consistency:Veries current

run results with a previous run.

Cannot be used to test new

functionality.

4. Self-Referential: Embeds

answer data in the input data.

5. Heuristic:Veries some

outputs. Can miss systematic

errors.

In addition to the types described by

Hoffman, lets add the following:

6. Fuzzy: Can detect that a

deviation may have occurred.

The most used oracle design principle

by us was the No Oracle and Heuristic

Oracle.

The No Oracle design is simple and

nds severe defects. Some defects leadto a crash or controlled stop (critical

assertion failure), which also ends the

simulation. After each simulation the log

les were scanned for critical errors and

exceptions. It was a cheap and easy way

to nd complicated and severe defects.

Different strategies were used for the

Heuristic Oracle. Simple checks on the

output included:

Prices for Trades are always

positive (for stocks)

Prices of Trades must always

be within the range of the

min/max prices of orders sent

into the system

Traded quantity is always

positive

Traded quantity can never be

more than the total quantity

sent into the system (the

lowest from either bidor askside)

A slightly more complicated oracle is

one that listens to the order event ow

and builds up a Shadow order book. If

any of the following situations occurs

something is clearly wrong:

Insertion of an order ID that

has already been inserted

Update of an order ID that

has never been inserted

Cancel of an order ID that

has never been inserted or has

already been cancelled

Self-referential Oracles were used to

verify:

An order is always traded

at the limit price or better

(exceptions are market

orders)

The order can never

participate in a trade with

a larger quantity than was

inserted.

Some attempts were made on a True

Oracle. In the development of one of the

trading systems, a copy of the matching

engine (core logic) was made in Java

and tested against the production code

implemented In C. The Java engine

was used to nd defects early in the

development project and was later

abandoned as the complexity increased.

S imulationsOur simulations used the actors and

oracles we designed to implement

random tests. For a typical simulation,

a mix of the various actor types was

chosen in advance and the simulation

was started with a dened random seed,

to ensure no simulation was identical.

During the simulation, the following set

of steps were continuously performed:

1. The framework selected an actor.

2. Based on the dened

probabilities an action was

selected and executed.

3. An oracle was triggered to verify

the results.

It was possible to run the simulation

either as a single session/thread or asmultiple sessions/threads. In a single

session/thread simulation, each selected

action was performed one after another

in a round robin scheme. This made

it easier to reproduce the sequence

of actions performed by the actors.

The advantage of a multiple sessions/

threads simulation is that a higher load

could be achieved, but it increased the

difculty associated with reproducing

the sequence of actions.

A nalysis of resultsThe simulation results were measured

using mostly defects found vs.

transactions sent into the system; a

transaction being dened as an inserted,

updated or cancelled order. Around one

million transactions (1 MTxN) were sent

each day in production. A test goal for

the simulations was to have at least 20

RandomTests



simulated trading days with no new defects. A typical graph

that was always included in the test report of a new release

to production is shown in Figure 3. The simulation typically

found between one and ve severe defects that were not found

with ordinary functional tests. If a defect was found, it was

immediately corrected and the simulations continued until the

reliability goal was reached.

In one release, an actor was incorrectly designed so it did

not exercise a newly implemented order type. As a result of

the oversight, the real production system went down within

an hour of deploying the new release of the software. Upon

correcting the actor, the simulation was re-executed and found

the problem after just 15 minutes of simulation time! After this

incident we improved the test process by:

Conducting a more extensive review of actor and

oracle code

Analyzing code coverage after running simulations Using fault injection to see if the actors & oracles

could trigger and nd expected errors.

The lesson learned was that simulations (or random tests) are

an important complement to traditional testing techniques for

critical systems.

A dvanced ConceptsSome advanced random test techniques we plan on

implementing in the future include:

Oracle using constraint programming techniques: The

oracle using constraint programming succeeded in modeling

a large part of order types (but not all). It showed that

techniques used in, for example, the aerospace industry

could cost effectively be applied to the nancial industry.

During the design of this almost true oracle, a fault in the

requirements was also found! For more information read

reference 3 identied at the end of the article.

Fuzzy Oracle using Neural Networks:The oracle had the

ability to learn normal behavior of the system using

neural networks. The Oracle could then warn when

something deviated too much. A simplied model of a

marketplace system using z-transforms is shown in Figure 4.

Fuzzy Oracle using Machine Learning:This oracle type uses

the same approach as the Neural Network type but uses

Machine Learning techniques (5).

S ummaryRandom tests are a very good complement to ordinary

functional automated tests. If a framework for automatic

functional tests is in place, it is a small and easy task with a

high payoff to simply add random tests. This article is based

on real tests performed since 1999.

R eferences1. Hoffman, Douglas. Using Test Oracles in Automation.

Software Test Automation Conference Spring 2003.

Presentation

2. Fewster, Mark, and Dorothy Graham. Experiences of

Test Automation. Boston: Addison Wesley, 2012. Print

3. Lozano, Roberto C. Constraint Programming for

Random Testing of a Trading System. MA Thesis.

KTH Royal Institute of Technology, Stockholm. http://

riunet.upv.es/bitstream/handle/10251/8928/roberto_

report_upv.pdf. Web. 28 January 2010

4. Bckmark, Torbjrn. Neronnnts baserad agent. MA

Thesis. http://www.addq.se/document/Referensjobb/

Random-Neuron.pdf. Web. 5 October 2001

5. Stenman, Peter. Learning the transaction ow at an

Exchange. MA Thesis. 2002

Figure 4: Z-TRANSFORMMODEL(4)

Figure 3: SIMULATIONRESULTSGRAPH



Training Thats Process Focused, Yet Hands On

Software Test Automation Trainingwww.training.automatedtestinginstitute .com

Public and Virtual Trainin Available

Come

participate in a set

of test automation courses

that address both fundamental

and advanced concepts from a

theoretical and hands on perspective.

These courses focus on topics such

as test scripting concepts, automated

framework creation, ROI calculations

and more. In addition, these courses

may be used to prepare for the

TABOK Certification

exam.

PUBLIC COURSES

Software Test Automation Foundations

Automated Test Development & ScriptingDesigning an Automated Test Framework

Advanced Automated Test Framework Development

Mobile Application Testing & Tools

VIRTUAL COURSES

Automated Test Development & Scripting

Designing an Automated Test Framework

Advanced Automated Test Framework Development

Mobile Application Testing & Tools


20/44


21/442Automated Software Testing Magazine

By Stefan Mohacsi and Armin Beer

Though the introduction of test automation has its

benets in promoting regression testing, we also

observed some weaknesses concerning the creation

and maintenance of the test cases:

Automated test cases are designed by hand, often

without taking specic test case design methods into

account.

The implementation of automated test scripts requires

advanced programming skills.

The maintenance of automated test cases after changes

to the SUT (Software Under Test) is difcult and time-

consuming.

Model-based testing (MBT) has the potential to resolve these

issues. However, it is often still regarded as too academic and

therefore as not suitable for testing in a real-world project

where resource, cost and time constraints apply.

The most common way of getting a test model for MBT is to

reuse a system model that has been created for development.

However, this method has serious disadvantages:

If both the source code and the test cases are generated

from the same model, no deviations will be found.

Defects in the system model will remain undetected.

The system model is created with a focus on

development, not on testing. Thus, it includes a wealth

of information that is irrelevant to testing while missing

other data that is required.

Many models in notations like UML lack the level of

formality required for automated test case generation,

e.g., because information is contained in plain text

annotations. UML extensions can mitigate this problem,

but add a further level of complexity to the model.

Test automation based on tools like HP QUICKTEST

PROFESSIONAL is well established and successfully usedin many software organizations, but...

www.automatedtestinginstitute.comOctober 2012



For all these reasons, it is recommended

to create a separate model focused on

the purposes of testing. It should only

contain information relevant to testing

and be just detailed enough to generate

test cases and test data. If the notation

used for MBT is understandable for

system testers, they are more likely to

use it. Most system testers nowadays

are familiar with use cases, activity

diagrams and Boolean expressions.

Systematic test design techniques like

equivalence partitioning, boundary value

analysis, and cause-effect analysis are

often not used exhaustively enough by

the testers. MBT fosters the application

of these techniques and as a consequence

improves the capability of the test cases

to detect specic categories of defects.

Despite the initial effort needed to create

a test model and integrate MBT into the

test process, the improved effectiveness

and maintainability of the test cases will

be appreciated by all stakeholders.

This article is structured as follows:

In the next section we present the

recommended workow and give

practical tips for the efcient usage of

MBT. Subsequently, we introduce the

MBT tool Atos TEMPPO Designer, and

show how it supports the MBT workow

using a practical example. Finally, we

give a conclusion and an outlook on our

future work.

Workow and

Practical Tips

for Model-based

Testing

In this section we describe the steps

required to successfully introduce MBT

into your project and also provide a

number of practical tips. But before

reading on, you should be aware of

the rst pre-requisite for applying

test automation in general, and MBT

in particular: a well-dened software

development process that guarantees the

adequate management of project tasks;

requirements; test artifacts; and change

requests. If the process in place is not

at the required maturity level, there is

a strong probability that MBT will not

work. Fast and reliable communication

channels between the testers and the

rest of the project team are an essential

feature of such processes (see Figure 1).

Ensure Testability of

Software Requirements

While requirements engineering is

usually not considered a part of the test

process, it is, nonetheless, crucial for

the subsequent testing phases. There

are some points that have to be taken

into account already at this early stage

to ensure the success of the testing

activities, especially if MBT is going to

be applied.

Testability comprises two aspects:

1. Ensuring that the input documents

for the test process have the

required quality level

2. Ensuring that the system under

test fullls all technical pre-

requisites for test execution

Roles: Requirements Engineer, Test

Manager, Test Analyst, Test Automation

Specialist

Input: User Requirements

Output: SW Requirements, Testability

Analysis Report

Practical Tips:

Involve test experts in early

reviews of requirement and

design specications in order to

avoid testability problems later. A pre-requisite for MBT is that

the requirements are detailed and

clear enough that a formal model

can be derived from them.

Try to use activity diagrams, state

diagrams, usage scenarios etc.,

instead of plain text.

Make sure to plan interfaces

that can later be used by test

automation tools to access the

system under test.

Figure 1: MBT-related activities in relation to the ISTQB test process


23/44www.automatedtestinginstitute.comOct/Nov 2012 2Automated Software Testing Magazine

Defne Test Structure

Based on the requirements, a test

structure consisting of test packages and

test cases is derived. This should be done

with a professional test-management

tool that provides convenient interfaces

to the other tools involved in the test

process (requirements management,model-based test design, test execution,

defect management, etc.)

Roles: Test Analyst

Input: SW Requirements

Output: Test Structure

Practical Tips:

Dont only focus on the required

functions, also plan tests for the

non-functional requirements.

Decide for which test casesmodel-based design shall be

used. Typical candidates are tests

for requirements that can easily

be formalized as a model. On the

other hand, for some requirements

MBT might not make sense.

In addition, dont forget to plan

some experience-based tests in

addition to the systematic tests. A

test model is a good thing but it

cannot replace human experience

and intuition!

Create Model-based

Test Design

In contrast to an ordinary test project

where test cases are designed e.g., in

a tabular format, a model-based test

project requires the creation of a test

model. This model depicts the system

behavior from a testing perspective and

contains all scenarios, constraints, and

dependencies that are relevant to testing.

Modeling can require considerableeffort, especially if the input documents

do not have the required level of

detail and frequent clarications are

required. On the other hand, this is an

excellent opportunity to detect defects

and inconsistencies at an early stage

even before the implementation phase.

Therefore, this initial investment helps

to avoid serious problems in later stages

of the project and greatly enhances the

maintainability of the tests. Nonetheless,

it has to be checked beforehand when the

ROI will be reached and if MBT makes

sense for a certain test item.

Roles: Test Analyst, Requirements

Engineer, Software Designer

Input: SW Requirements, Test Structure

Output: Test Model

Practical Tips:

Avoid reusing a model that

has also been used for code

generation. If the code and

the tests are generated from

the same model, no deviations

will be found! Try to create a

specic model from the testing

perspective instead.

Creating a test model is a

challenging task that requiresmany different skills (test design,

abstraction, understanding

of requirements, technical

background). Dont expect end

users to perform this task but

employ specialists instead!

Start modeling at an abstract

level, then add more detail step

by step

Make sure that requirements

engineers and software designers

are available to clarify any

questions from the test designers.

Pay attention to the reusability

and maintainability of the

model components, e.g., use

parameterizable building blocks

Link Test Model to

Executable Test Scripts

If the test execution shall be done

automatically, the abstract test model

has to be linked to a test script containing

concrete instructions for a test executiontool. A possible way of doing that is

to implement an adapter or script for

each keyword that is used in the model.

However, this requires considerable

effort and advanced programming skills.

A much better alternative is to use

an MBT tool that can import GUI

information and generate executable

scripts automatically. In this case, each

step in the model only has to be assigned

to the affected GUI element.

Roles: Test Analyst, Test Automation

Specialist

Input: Test Model

Output: Detailed Test Model/Test Script

for each Keyword

Practical Tips:

If using a professional test

generator that generates complete

scripts (e.g., TEMPPO Designer),

record GUI information and

assign steps to GUI elements.

If using a generator that only

produces abstract keywords,

implement an adapter or a script

for each keyword.

Prepare Test DataDepending on the requirements it has

to be decided whether test data can be

imported from an existing source, has to

be created by hand or will be generated

automatically. In the latter case, some

MBT tools provide the possibility to

specify data dependencies as part of the

model and to produce data accordingly.

Roles: Test Analyst

Input: SW Requirements, Test Structure

Output: Test Data

Practical Tips:

Try to cover both test data and test

sequence-related aspects in your

model (e.g., dene equivalence

partitions for test data, activity

diagrams for sequences). Finally,

connect the two aspects by

specifying which data is used in

which step of the sequence.

Simple test data design methods

like equivalence partitioning and

boundary value analysis should

only be applied for data elds that

do not have any dependencies

with others. For all other cases,

methods like cause/effect analysis

or CECIL should be applied.

Generate Test Cases

Based on the test model, test cases can be

generated automatically. Depending on

ModelBased



the model type and the generation tool,

the user can choose between various

generation strategies and coverage

criteria. While many tools can only

generate abstract test cases consisting

of keywords or instructions for manual

testing, others can produce complete,

executable scripts.

Roles: Test Analyst

Input: Detailed Test Model-/-Test Script

for each Keyword

Output: Executable Test Scripts

Practical Tips:

Choose test coverage based on

model type and criticality of the

associated requirements.

Combine different test methods

to increase defect detectionpotential. Use random generation

in addition to systematic methods.

The integration should make it

possible to transfer the generated

test cases directly to the test

management tool.

Execute Test Cases

In some projects, the test generator

produces textual instructions for manual

testing. However, more frequently, the

test execution is performed automaticallyby a tool. In both cases, appropriate

interfaces for sending test commands to

the system under test and verifying the

results have to be available.

Roles: Tester, Test Analyst

Input: Executable Test Scripts

Output: Test Protocols, Defect Reports

Practical Tips:

Dont expect test automation to

work immediately -- its ratheran iterative process. Verify ifthe generated test cases work asexpected (GUI object recognition,timing, etc.). Correct the modelwhere necessary until the tests arerunning smoothly.

Test scripts derived from a modelusually require that the systemunder test is in a well-denedstate at the beginning of each testand behaves in a predictable way.For instance, alterations in the

database or unforeseen pop-upmessages may cause the scripts tofail if they are not considered inthe model.

In cases of deviation from the

expected behavior, it has to bedetermined whether the problemwas really caused by a bug in thesoftware or on the contrary by a

defect in the test model.

Create Test Report

The test protocols created during the

test run have to be transformed into

a concise test report that informs the

responsible managers about the current

project status.

Roles: Test Manager

Input: Test Protocols, Defect Reports

Output: Test Report

Practical Tips:

For MBT it is important thatstatistics of the model coverageare included (e.g., the percentageof states, transitions, or scenariosthat have been covered by thetests).

Many test management tools cangenerate such reports, providedthat they have access to all therequired data. Again, a good

integration with other tools is

important.

Maintain Test Model

For each new version that has to be

tested, the tests have to be adapted and

repeated. Updating each single test script

by hand is impractical as it can require

an enormous effort. On the other hand,

updating a model consisting of reusable

components is much less tedious, since

only a few parts have to be changed

before generating perfectly updated test

scripts.

Roles: Test Analyst

Input: Test Model, Change Requests

Output: Updated Test Model

Practical Tips:

It is essential that the test team

is informed about all changes in

the system under test. Even small

modications that are invisible to

the user can cause an automatedtest run to fail.

The effort invested into a well-

structured model design will pay

off now. If designed properly,

only some parts of the model

have to be adapted. Afterwards,

all test cases can be updated

automatically by re-generating

them.

Model-based

Testing with Atos

TEMPPO Designer

While it is relatively easy to nd tools

for automated test execution, there

are signicantly fewer that focus on

model-based test generation. They can

be categorized according to the method

used for test generation: data-oriented

tools and sequence-oriented tools. A tool

that unites both features and can also

Figure 2: Model-Based Testing with TEMPPO Designer



be easily integrated into a projects tool

framework is Atos TEMPPO Designer;

this tool was previously called IDATG

(Integrating Design and Automated Test

case Generation).

When the research project IDATG started

in 1997, the aim was to develop a simple

tool for facilitating the maintenance of

SilkTest and WinRunner scripts for

GUI testing. Over the years, IDATG

was steadily expanded to support an

increasing number of test methods and

output formats, as well as testing via

non-GUI interfaces. In 2004, IDATG

became part of the test framework of

the European Space Agency ESA (this

success story can be found in Dorothy

Grahams new book Experiences of

Test Automation [Graham et al. 2012]).

Interfaces for Model

Creation

Many MBT tools do not have a model

editor of their own, but instead rely on

other sources from which models areimported. However, reusing models

that have not been created with a focus

on testing has several disadvantages.

Quite often such models are not formal

enough (e.g. plain text annotations) or

they contain a lot of information that

is irrelevant to testing while missing

important data. To overcome these

problems, TEMPPO Designer has a

built-in model editor that provides

an independent way of creating test

models. It uses a specic notation that

is simple to learn, and it focuses on the

necessities of testing (see Figure 4).

Test sequences are represented as task

ow diagrams that consist of simple test

steps (blue) and parametrizable building

blocks (yellow) that represent reusable

sub-sequences of steps. For GUI testing,

steps can be assigned to GUI objects

using the built-in GUI Spy.

Apart from common testing methods

like equivalence partitioning, a number

of innovative test strategies have

Fig ure 3: Sample Application

Figure 4: Task Flows in TEMPPO Des igner

ModelBased



been developed for the tool. These

include CECIL (Cause-Effect Coverage

Incorporating Linear boundaries [Beer &

Mohacsi 2008]) and a hybrid algorithm for

random testing (Mohacsi & Wallner 2010).

Interfaces for Test

Execution

A major drawback of many testgenerators is that they only produce

abstract test cases that still require

manual completion, for instance

by implementing a script for each

keyword. A distinguishing feature of

TEMPPO Designer is that its output

are complete, executable scripts for a

variety of popular test execution tools

including HP QuickTest Professional

and the Micro Focus tools SilkTest

and TestPartner. However, the tool can

also be used for producing test cases thatare executed manually or over a non-

GUI interface.

This direct integration between test

generation and execution has proven to

be a considerable advantage in that it

delivers a signicant reduction in test

maintenance costs; i.e., instead of having

to update each single test script for every

new version of the system under test, it

usually sufces to change a few building

blocks inside TEMPPO Designer and let

it generate new test cases. A case study

from an ESA project has shown that the

additional effort for introducing model-

based test automation paid off after

only four test repetitions (Graham et al.

2012).

Interfaces for Test

Management

Integration of test generation with

test and requirements management is

equally important. As its new name

suggests, TEMPPO Designer is a part of

the TEMPPO suite that also includes the

test management tool TEMPPO Test

Manager. Information can be passed

in both directions: Test structures and

requirements created in Test Manager

are passed to Designer. After the testmodel has been created, the generated

test cases are passed back to TEMPPO

Test Manager which ensures proper

versioning, traceability, execution

planning, and reporting.

There is also an interface for HP Quality

Center that allows the user to import

TEMPPO Test Designer test cases via

Excel.

Practical

Example

We are going to demonstrate the most

relevant steps of the MBT workow

using a simple application for managing

tennis players and tournaments. While

this is a hypothetical example, it includes

many experiences from real projects.

It also introduces the usage of the Atos

TEMPPO tool suite, and shows how to

efciently use it for model-based test

design and generation.

Ensure Testability of

Software Requirements

In the rst application design, the

use of the GUI library FlashyGUI

was envisaged by the designer whohad found it on the Internet and was

eager to try it out. Fortunately, the test

automation specialist was involved in

the design review and pointed out that

this technology was not supported by

any test automation tool on the market,

thus making automation impossible.

Finally, it was decided to use a standard

Win32 GUI instead.

Defne Test Structure

Once the requirement and designspecications had been reviewed, the

test analyst began creating a tree-like

test structure in TEMPPO Test Manager.

He started with typical usage scenarios

like Create Player, then added tests

for non-functional requirements (e.g.,

performance of the search function) and

error cases (e.g., assigning a male player

to a female tournament). Based on his

ample experience from past projects he

also planned a few scenarios simulating

unusual user behavior.

Create Model-based

Test Design

Using TEMPPO Designer, the test

analyst began to break down each

scenario of the test structure into a

rough sequence of test steps. While

doing this, he identied recurring

steps like Edit Player and dened

Figure 5: Excerpt from a Generated Script in QTP



them as re-usable building blocks with

parameters. These rough building blocks

were further rened, for instance, Edit

Player consisted of the blocks Edit

Personal Data, Edit Career Data, and

Edit Ranking. This renement was

iteratively repeated until ending up with

single-user actions like entering the rst

name or clicking the OK button.

The user action represented by each

step could be entered in a generic

event language, for instance Click

or Input (Serena). However, instead

of concrete texts, mostly references

to parameters were used to make

the model more exible: Input (#@

EditPersonalData:FirstName#)

Semantic conditions were also

considered in the model. For instance,a middle initial was only entered if

the corresponding parameter was not

empty. This could simply be specied in

the form of a Boolean expression: #@

EditPersonalData:Initial# != .

Also, the expected results and reactions

of the system were specied. This could

easily be done with events like Verify

(CAPTION, Player data has been

saved.).

However, the specication documentson which the model was based seemed

to be incomplete. For instance, it was not

clear how the system should react if a

man was added to a female tournament.

Maybe the designer had assumed that

the programmer would surely consider

this obvious case and it was therefore

unnecessary to mention it. Fortunately,

the test analyst could contact the designer

directly and ask him. It turned out that

this case had simply been forgotten in

the design and that the programmer, who

had no interest in tennis at all, hadnt

thought of it either. Since this defect

had been found in an early stage of the

development, it could quickly be xed

and there was little harm done. However,

if it had found its way into the nished

product, this embarrassing mistake

could have damaged the companys

reputation.

Link Test Model to

Executable Test Scripts

Once the GUI of the tennis application

had been implemented, its window

could be recorded with the built-in

GUI Spy of TEMPPO Designer. Each

blue test step was assigned to a certain

object in the screenshot, for instance,

EnterFirstName to the input eld

FirstName. Since the GUI Spy did

not only capture the appearance but also

the technical details of the GUI objects,

no further information was needed toproduce a complete object repository for

the test execution tool (in this case, HP

QuickTest Professional).

Also, the generic events like Input would

later be translated automatically into the

tool-specic language (for QuickTest,

Input would be translated into the Set

command). Only a few steps involved

more complex actions that could not be

expressed in the generic language but

had to be dened in QuickTest code. An

example was checking the alphabetical

order of the player list, a verication that

had to be implemented as a QuickTest

function. Apart from this, no other

action was required to link the model to

the nalized test scripts.

Prepare Test Data

Most information about the tennis

players was imported from an Excel

sheet showing the current ATP ranking.

In addition, equivalence partitioningand boundary analysis were applied

to generate test data, thus covering

all relevant cases for elds like the

Prize Money. In more complex cases,

dependencies were specied, e.g., that

the start date of a tournament must

lie before its end date. Finally, it was

specied which data was used in each

test step simply by inserting references

to the data tables.

Generate Test CasesOnce the rst version of the model wascompleted, our test analyst was able

to start the script generation. He chose

This direct integration between test generation

and execution has proven to be a considerableadvantage in that it delivers a significant reduction

in test maintenance costs

ModelBased



a combination of graph-oriented tests

covering all step connections, data-

oriented tests covering all tennis players,

and a few random sequences each

consisting of 1000 steps. The output

was a complete set of test scripts and

an object repository for HP QuickTest

Professional. Also, information about

the generated test scripts was transferred

to TEMPPO Test Manager.

Execute Test Cases

Via TEMPPO Test Manager, QuickTest

was opened to start a rst trial run of

the scripts. In the course of this it was

noted that a few scripts failed because

sometimes a timeout occurred when

the tennis application connected to its

database. The test analyst had to insert a

higher delay value in the correspondingstep of the test model and re-generate the

scripts.

In the next trial run it turned out that the

tree control displaying the tournaments

had changed its ID and could thus not be

found any more by QuickTest. The test

analyst solved this by removing the ID

from the set of properties that was used

for object identication. Again, this was

done in the model, not in QuickTest

itself. Finally, after re-generating

the QuickTest les, the tests did runsmoothly.

Once the test analyst decided that the

scripts were ready, the tester could

begin his task. He ran the entire test

suite and evaluated the test protocols

that were produced by QuickTest.

Several deviations from the expected

behavior were found, for instance, it

was possible to enter a negative value

for the prize money. One of the random

sequences even caused the application

to crash when it tried to assign Serena

Williams and her sister Venus to the

same tournament.

Create Test Report

The test results and defect reports were

then comprised by the test manager into a

test report using the reporting capabilities

of TEMPPO Test Manager. The test

manager also included information

about the model coverage, especially the

percentage of covered scenarios, step

connections, and data records. This gave

him an excellent position in the next

meeting with the customer who wanted

to know exactly about the test progress

and was impressed by the accurate data.

Maintain Test Model

Every two weeks, a new version of

the tennis application had to be tested.

Since there were more than 500 test

cases, adapting them by hand for each

version would have been a nightmare.

But thanks to the model-based testing

approach, the test analyst only had

to adapt a few building blocks in the

model and re-generate the scripts. Even

when suddenly a major change of the

rules for Grand Slam tournaments wasannounced by the International Tennis

Federation, it sufced to adapt a few

steps and conditions.

Conclusion

In this article, we presented a workow

for MBT that considers the traceability

between requirements, the test model,

the GUI, and the test cases. The

workow is supported by the Atos

TEMPPO framework which has theadvantage of using a simple model

notation specically designed for the

purposes of testing. Also, instead of

generating only abstract keywords from

the model, the tool produces complete,

executable test scripts for tools like HP

QuickTest Professional (QTP).

The generation of test cases is

systematically increasing the

effectiveness of nding relevant defects

in the SUT. Thus, the condence in

the product quality is increased and

the number of unplanned releases and

hot xes can be reduced signicantly.

The effort required for dening and

maintaining the test cases can be limited

to a reasonable level.

Applying MBT using TEMPPO

Designer has become a major focus of

Atos. A global Competence Center for

MBT has recently been established with

the task of promoting the topic within

the company. Our future plans include

a closer coupling of TEMPPO Designer

with other test tools. The newest version

already supports the import of object

repositories from QTP. The next step will

be the import of recorded actions that

can then be converted into parametrized,

reusable building blocks.

Acknowledgment

The research herein is partially

conducted within the competence

network Softnet Austria II (www.soft-

net.at, COMET K-Projekt) and funded

by the Austrian Federal Ministry of

Economy, Family and Youth (bmwfj),

the province of Styria, the Steirische

Wirtschaftsfrderungsgesellschaft mbH.(SFG), and the city of Vienna in terms of

the center for innovation and technology

(ZIT).

Some parts of this work are based on

an article we published in the March

2012 edition of the Testing Experience

magazine.

References

[Graham et al. 2012] D. Grahamand M. Fewster, Experiences of Test

Automation, Chapter 9, p. 155-175,

Addison-Wesley, 2012.

[Beer & Mohacsi 2008] A. Beer

and S. Mohacsi, Efcient Test

Data Generation for Variables

with Complex Dependencies,

Proceedings of the IEEE ICST,

Lillehammer, Norway, 2008.

[Mohacsi & Wallner 2010] S.

Mohacsi and J. Wallner, A Hybrid

Approach to Model-Based Random

Testing, Proceedings of the Second

International Conference on

Advances in System Testing and

Validation Lifecycle, 2010, pp. 10-

15, 22.-27. August 2010.



CrowdamationCROWDSOURCEDTESTAUTOMATION

Its OK To Follow the CrowdInquire at [email protected]


30/44



We all want to enjoy the benets

offered by automated testing, but

somehow the workload, project

stress or other parameters interfere, and weare not able to turn our perfect plan into

practice.

This article, based on practical experiences

in insurance companies in Germany and

the United Kingdom, provides one possible

approach to addressing these challenges to

test automation.

An alternativeapproach

Sometimes with automated testing one gets

carried away with optimizing the code for

the test tool or nding more efcient ways

to run a script, but we should never lose sight

of the aim of testing itself. That aim is to

improve the quality of the software, and to

minimize the number of defects found in the

live environment.

With this in mind, our automated testing may

benet greatly from more involvement by the

front-end user.

In a perfect world an automated test analyst

is provided with manual test cases that may

be used as a basis for test automation. All

the test cases are quality checked and can beused 1:1. In my experiences, the real world

presents a different situation. There often

only exists a rough idea as to which tests

should be automated, and this idea has to be

used as the basis for building the automated

tests step-by-step. For this reason, knowledge

from the front-end user is important to the test

automation process. In Dorothy Grahams

and Mark Fewsters book Test Automation,

I described how the lack of user knowledge

causes several show stopping mistakes in

test automation. Following is an excerpt from

my chapter in the book:

Typical beginner mistake: we just assumed allpeople are working with the same screen resolution.This, of course, was not the case. Automated testcases passed, because we had on our screen thescreen resolution we assumed everyone else inthe world has. But the real user was not able tonavigate to some pages, because a different screenresolution caused graphical items to be drawn offscreen and therefore become invisible to the user.

Very funny was the fact that the submit buttonwas missing on a specifc screen resolution. Well,not really funny for the user world and this wasdefnitely a lesson learnt.

Even though test automationhas developed a lot over the last years,

we are still at the beginning of an

exciting journey.

What goes wrong in modern day test automation

implementations? Where is the room for

improvement? Are we just wasting our time? These

are questions we need answered. Developing

effective test automation packages is a complex

process. There are very good best-practice guides out

there, many good tools, different kinds of models,

and still our test automation pack is not at its best.



Where is the front-end user

usually involved?

1. Acceptance testing performed by

the customer is known as a User

Acceptance Test (UAT). A UAT

is composed of a selection of

high priority tests and is often animportant nal step for ensuring an

application is ready to be rolled out.

I have worked for several insurance

companies and a UAT was usually

performed at the end of every release.

If major problems are found during

a UAT, it can be extremely costly to

the organization. Hence, involving a

user for the rst time at this stage is

way too late.

2. User stories are created at the

beginning of the developmentprocess, and are a good attempt

to involve the front-end user very

early. Generally there is room for

improvement in terms of the quality

of user stories and they tend to require

many reviews and amendments. User

stories are often documented by a

test analyst representing a front-end

user, however, which means that

the user may not actually be getting

very involved early in the process.

This also means that pertinent steps

in the workow unknown by the test

analyst may fail to make it into the

user stories.

Approach

A new and maybe unusual approach

to getting the front-end user involved

earlier in the testing process is job

rotation. Wikipedia quotes job rotation

as follows:

Job rotation is a management technique

that assigns trainees to various jobs and

departments over a period of a few years.

Surveys show that an increasing number

of companies are using job rotation to train

employees. There are both positive and

negative effects involved with job rotation

that need to be taken into consideration whena company makes the decision to utilize this

technique.

One of the main reasons job rotation has

not been employed for automated testing

is that workers are specialized due

to technology, and automated testing

requires highly skilled workers. For

example, a claims manager from a claims

department in an insurance company

cannot program an automated test

script without some extensive training.

Therefore, getting a non-technicalperson involved in an inherently

technical task such as test automation

could be extremely expensive, unless

done using the job rotation approach.

The job rotation scenario well be using

in this article rst examines a front-

end user conducting test automation

tasks, then examines a test automator

conducting front-end user tasks.

Front-end User JobRotation Scenario

Lets look for example at a front-

end user - i.e. claims manager in an

insurance company - who is rotated

onto an automated test team for a couple

of weeks. The training for coding and

automated test execution takes too long,

and is therefore not a suitable task. A

more tting task is reviewing old test

automation packs. Usually there are alot of meetings and reviews involved in

making sure the pack gets altered and

updated correctly. Although reviewing

TASKTIME-

CONSUMINGFACTOR

CAN TIME BEREDUCED

WITH MORE

CAN IT BEEXECUTEDBY A JOB

ROTATOR?

Automate test

casesYes No No

Review old test

packsYes Yes Yes

Running test cases No No Yes

Analyzing script

error messagesYes No No

Review manual

test cases and

prepare for auto-

mation

Yes Yes Yes

Writing user guides Yes No No

TABLE1: TASKSOFANAUTOMATEDTESTER

...a different perspective can help

make the process more effective.



these test packs does involve minimal

training, the overall reduction in time

that results from having a business

person closely involved will be worth it.

Running test cases is not very time

consuming, but it is also a task which

can be taken over by a front-end user.

Reviewing manual test cases and

preparing them for automated test is

another ideal task. The front-end user

can verify the test steps and prepare

the test scenario so

Documents

2012.11 AutomatedSoftwareTestingMagazine November2012