E s s e n t i a l f o r s o f t w a r e t e s t e r sTE TERSUBSCRIBE

It’s FREE for testers

Simulation virtualization

June 2013 v2.0 number 21£ 4 ¤ 5/

Including articles by:

Howard Osbornee-Testing

Huw Price and Llyr JonesGrid-Tools

Vu LamQASymphony

Stephen JohnsonROQ IT

Doug Lawson

Bogdan BerezaVictO

A bunch of developersCoverity

THIS ISSUE OF PROFESSIONAL TESTERIS SPONSORED BY

Continuoustesting and delivery

Continuoustesting and delivery

Including articles by:

Bogdan BerezaVictO

Roy de KleijnPolteq

Mark Lehky

Jessica Schiffmann Prism Informatics

Eric M.S.P. VeithWilhelm Büchner Hochschule/TU Bergakademie Freiberg

18 PT - June 2013 - professionaltester.com

Continuous testing and delivery

Multiple versions of applications have always been bad news for testing. Many PT readers will remem-ber from experience the nightmare caused by variation between web clients: platforms, browsers, plugins etc, all in various releases. Test design was shrunk by having to repeat test execution across a large number of combinations: important tests simply could not be done. The many disgrace-fully simple failures that occurred at the time were no coincidence.

The browser war was won by standards. Users and support technicians learned that using a more standards-compliant browser gave them a better chance of achieving what they wanted on a wider range of sites. That in turn incentivized businesses to make their sites more compliant. Good browsers like FireFox and Safari rose up the priority list. Bad ones like Internet Explorer 5 and 6 fell down it, so their subsequent versions became more compliant. At that point most of the problem disappeared.

That is well because since then, while the number of important browsers has remained constant, platforms have proliferated in the form of mobile devices. Browser and platform are coupled tightly: the behaviour of browser A for mobile phone B can be very different to its ver-sion for tablet C. Thanks to standards, it is feasible to develop and test adequately a single version of a web application to work well on most clients. Even so, there are often two versions, “desktop” and “mobile”, in order to improve the experi-ence of those with small screens and slow connections. Thus testing effort is split in two (perhaps less than two where the two versions share many components). That’s not good, but it’s better than having to split it into several fragments as was common at the height of the browser war.

Here we go againIf all mobile applications were web appli-cations, cross-client web and mobile development would now be in a strong position. Of course they are not: native apps have gained preference. There are many reasons, good and bad, for this which I will not attempt to analyse here. Some people consider building native apps to be retrograde: for example PhoneGap (http://phonegap.com) “wraps” web applications, converting them to native mobile apps: this idea may catch on.

Cross purposesby Bogdan Bereza

Bogdan Bereza on regression testing without human intervention

Complete automation requires central programmatic control

Agile is going places – are you?

0179

3/PDS/A

D/081

2

© BCS, The Chartered Institute for IT, is the business name of The British Computer Society (Registered charity no. 292786) 2012

BCS, The Chartered Institute for IT, offers the leading agile testing certification for software testers –Certified Agile Tester.

bcs.org/agiletester

The Certified Agile Tester scheme is a trademark of iSQI.

01487_pds_ad_hp_cat_proftest_ma_Layout 1 10/08/2012 14:27 Page 1

Event Loading...

London Autumn 2013

This highly anticipated event now returns to London on 24th October 2013.

Register by 5th July for your early bird discount!

Just visit: testexpo.co.uk

20 PT - June 2013 - professionaltester.com

Continuous testing and delivery

Until and unless it does it is necessary once again to develop for different propri-etary client platforms. That means testers are once again faced with multiple ver-sions of their test items, and they won’t go away any time soon. The economic forces around users’ choice of device are more powerful and complex than those affecting free browser software. There is no incen-tive for standardization and no chance of one platform achieving dominance.

Worse still, the most business-critical apps are hybrids, in the sense that they have multiple web and mobile clients sharing the same server/mainframe ser-vices and data. Web cross-client failures were usually of presentation; native apps have the dangerous potential to cause organization-wide failure. And this is in continuous delivery environments where they are being changed very frequently with high possibility of regression.

Continuous cross-client hybrid app testingSo testing needs to become continuous: the growing and evolving regression suite should be executing automatically and always, to assure against defects introduced by constant change to the test items made by automated integra-tion and deployment.

Although it is very simple, the test script shown in figure 1 could if executed in time have prevented a serious failure: a less-than-one-line change to code prevented the app, when running on one specific and superseded version of the mobile device OS, from accessing the account balance. The rarely-executed code that should have trapped that event also failed. An incor-rect balance, the last one retrieved, was displayed and transactions which should have been prevented were allowed.

So how can a script like this, requiring execution and verification on multiple devices, be fully automated? The dif-ferent approaches to functional testing of mobile apps taken by different tools suggest some possibilities. As so often in test automation, the key difference

between them is how they recognize interface objects.

The device executes the testMost or possibly all tools that do this require the device to be jailbroken or rooted: because of the obvious disad-vantages of doing that I won’t consider them here.

It is possible “legitimate” tools which are themselves apps might emerge, possibly supported by a change of policy by the device manufacturers of the kind that is easier to imagine happening on, say, Android than on iOS. If it does happen, whether they can be used for our current purpose will depend on (i) at what level they intrude on the app under test and (ii) whether and how they communicate with the cross-client “master” script so that it can control execution of the test, monitor its progress and verify its outcome.

Remote control the deviceIt is possible to execute tests on a device from a workstation, using various “desktop sharing” systems. The simplicity of this method is both its strength and its weakness. The recorded script is easy to read and can be controlled and verified by the master script as we require, but it is not robust: and because the only information available about what the application is doing is what is in the pixel image displayed on the device screen, modifying, reusing and maintaining it are visual and therefore very manual tasks.

Remote control the appThe most successful tools such as Ranorex control the app, not the device. This requires its code to be instrumented, a transparent process which makes the app able to automate itself: no other app is needed. User actions on the device are

captured and documented by Ranorex Recorder running on a workstation exactly as those performed on any other platform and can then be augmented with additional actions and validations, managed, maintained and reused in all the same ways, with object recognition accomplished in the most appropriate of multiple available ways.

Continuous delivery aspires to auto-mate as many deployment operations as possible. That is not done simply by recording them and playing them back: programmatic control is needed. Continuous testing requires exactly the same. “Easy” approaches designed to appeal to people who cannot understand simple code are not easy except in dem-onstrations which are always done using trivial example apps. In real testing of real apps they quickly become difficult or impossible. Continuous testing of mobile apps requires not ease, but power: that is what Ranorex offers

Frequent PT contributor Bogdan Bereza is a testing consultant, speaker and trainer, proprietor of VictO (see http://victo.eu) and co-ordinator of the inaugural Good Requirements conference taking place in Warsaw on 3rd and 4th October 2013 (see http://conference.wymagania.org.pl/english.html) A free trial of Ranorex, including all the mobile app testing features described here, is available from http://ranorex.com

Access test account A via the web application

Transfer the entire balance of test account A to test account B

[repeat for all mobile devices and OS versions currently supported]

Access test account A via the mobile application and verify that its balance is displayed as zero

[end repeat]

Figure 1: cross-client test script