Upload
ledung
View
224
Download
0
Embed Size (px)
Citation preview
Increase Your Mobile Test Automation ROI
Make your Mobile Test Automation more mature and dependable. Take your framework to an all new level of robustness by executing your tests on real devices and at the same time increase your ROI
This white paper should help you in understanding the nuances of mobile test automation, its challenges and how to derive more from your test automation effort without having to compromise on cost or quality
About the Author Aniket Deshpande
Pune, India Master in Business Administration (Information Technology)
Aniket Deshpande is a Test Architect at AFour Technologies. He has around 10 years of experience in testing varied applications across different platforms including mobile applications and has formulated testing strategies of various mobile applications. As a Test Architect, he has led teams in diverse domains like Social Engineering, Health & Fitness, E-Commerce, etc. His primary interests lie in conceptualizing & implementing mobile automation frameworks and environments and setting up mobile device labs.
2
Introduction
Mobile Application testing can be a tricky proposition. With the plethora of devices,
operating systems and their versions, further compound the problem. As gatekeepers
of quality, we QAs cannot be satisfied with testing only on simulators and emulators. It
is always risky not to test on real devices for various reasons, above all being, end
users would not be using the application on simulators but on real world devices.
Hence, your mobile test automation should also sign-off the build on real devices.
Should you choose to run your automation on real devices, which devices would you
narrow down on, from the ever-growing list of mobile devices?
Using your short-listed devices, you would need to manage your automation testing
cycles across these devices cleverly to be able to derive more from your investment.
For planned automation initiatives, choosing the right tool is very important for
maximizing returns on your automation effort. You do not want your tool to throw you
ugly surprises later in your framework development efforts.
Integrating your automation runs with the build cycles gives you fantastic ROI as you
may have already realized with your normal web and desktop automation frameworks.
Setting up such a process on your mobile automation will not only fetch optimal
returns, but also make your automation a truly hands-off process.
This White Paper will give you insights into how we achieved increased ROI at minimal
cost while implementing a mobile automation framework for one of our clients. The
solution includes choosing a tool, automating the tests, setting up of a mobile device
lab for test execution and integrating the automation runs with the build process.
Introduction | Trends | Problem | Solution | Benefits | Enhancements | Conclusion | References
“It is always risky not to test on real devices”
“Which devices would you
choose?”
“Need to manage your automation testing cycles across these devices”
“Choosing the right tool is
very important”
“Integrating your automation runs with the build cycles”
3
Trends
More mobile apps are being downloaded than ever before and the number is
increasing yearly. The year 2013 has seen an increase of more than 50% in apps
being downloaded from the respective Stores1. By the end of year 2013, there
were more smartphone owners in the world than PC owners. One in every 5
people now owns some kind of a smartphone2.
This further goes on to show the demand for mobile apps is ever increasing and
there are more smartphone users than PC users. This would mean enterprises
would want to quickly reach out to users on the mobile platforms than traditional
websites approach and would want faster time-to-market, which in turn would
demand process innovations in QA cycles.
Businesses with mobile apps have a customer conversion rate of more than 5% as
against less than 1% for businesses with no mobile interface3.
Gartner says that, by 2015, around 80% of all application development activities
will be for smartphones & tablets. And having a mobile application, systems &
devices are “one of three top CIO priorities”4.
All this points to a huge upsurge in mobile application development.
Many of our customers have release cycles as small as 1 week. The reason being
the ease of submitting the app to the Store and application upgrade on the user’s
device.
As QAs we need to be ahead of the curve and make sure we are prepared to
handle such demanding timelines. More than technological challenges, there is a
need for upheaval in the QA process itself, which can help in making the process
sleeker and agile.
Introduction | Trends | Problem | Solution | Benefits | Enhancements | Conclusion | References
“One in every 5 people now own some kind of a smart phone”
“Enterprises want faster time-to-market”
Gartner: “By 2015, 80% of all application development activities will be for smartphones & tablets”
4
Introduction | Trends | Problem | Solution | Benefits | Enhancements | Conclusion | References
About the App:
The mobile application under test is a
Health & Fitness application, which is
freely downloadable from App Store (iOS)
and Play Store (Android). The application
has implemented WebViews for user
interactions and REST API calls for server
communication.
Problem
1. Tool Selection:
This, by far, was the most vital of all our
activities for implementing the solution.
We had to find a tool that we could use
across both iOS and Android platforms,
because using 2 different tools is just not
the right way to go. Among other things,
maintaining 2 separate code bases and
technologies would be an overhead later
in the automation cycles.
Also, we wanted a tool that would
seamlessly work with a CI server so that
we could integrate it with the
development build cycles, which would
ensure a truly hands-off automation
execution.
The tool should also have rich reporting.
We could trade-off this feature by
implementing our own reporting module,
but an in-built reporting engine is always
more preferable than a custom one.
And importantly, the tool should work
smoothly on real devices and not only on
emulators & simulators.
Reasons for running on real devices:
The end-user downloading the
application from the Store, will be
using the application on an actual
device and not on a simulator.
Features like GPS, for location
services, can be tested reliably on a
real device than on a simulator.
Simulators & emulators use the host
machine’s hardware and processor
architecture, which will be different
than the actual device.
RAM, Disc space, Battery usage, etc.
can cause performance issues on a
device but not on simulators.
We have had many instances on
other projects where we found a bug
on the device but not on the
simulator or vice-versa, which made
the test results unreliable.
Using a device, we could test
applications on cellular data
bandwidth as against wireless/LAN
in case of simulators, which is
obviously always reachable and
faster than cellular data.
Most importantly, for Android
devices, the operating systems
released by OEMs contain many
customizations on middleware and
drivers, which may impact the
application.
You cannot install other applications
on simulators, like browsers, or if
your application uses social media,
then dependency on the social app
needs to be tested. This cannot be
done on a simulator.
2. Mobile Device Lab:
The reasons mentioned above justified
setting up an in-house Mobile Device
Lab. We could have used the services of
a third-party device lab like
DeviceAnywhere, but it would mean
sharing data with third-party vendors.
Also, with such device labs, the devices
are not exclusive and are shared. You
always run the risk of a rogue application
sitting on the device listening to your
activities.
Since these shared devices are cleaned &
reset after use, running automation on it
would require you to install and create all
pre-requisites every time you want to
trigger an automation run.
Another issue was, if you find a bug on
that device and if later, the developer
wants to reproduce it, then the device
may no longer be available or would have
been reset.
3. Continuous Integration:
We are working with a team in diverse
locations.
Setting up a continuous integration server
would make the automation execution
truly hands-off. You have various options
of scheduling or triggering your
automation runs based on events that
happen in your build cycles. Moreover, it
mandates that your environments are
standardized and encourages the team to
automate the pre and post tasks as well.
This approach also helps in managing your
test cycles & reporting. If the triggers /
schedule is setup cleverly, you do not
have to wait for a QA to fire the runs. It
becomes a 24/7 system.
Stitching It All Together:
The above challenges are inter-related
with each one having an impact on the
other. The UI automation tool selection,
device lab design and CI tooling & strategy
all need to be tied up together to create a
tight harness.
Lets see how the solution was
implemented.
5
Introduction | Trends | Problem | Solution | Benefits | Enhancements | Conclusion | References
Solution
1. Tool Selection:
Looking at the different problems and the importance of choosing the right tool, a detailed PoC
(Proof of Concept) was carried out. The litmus test of the tool was automating 3 end-to-end test
cases. We identified these test cases based on the complexity of the flow and the various controls
encountered on the screen. For the PoC we considered, Calabash, Ranorex, Appium, Robotium &
Sikuli. We evaluated these tools on the following features:
Platform Support: Support for both iOS & Android was mandatory. Maintaining separate code
bases for both platforms will be an overhead in the long run. Since WebViews are used
throughout the application, application flows across both platforms are similar. We could
take advantage of this by writing a common code base for both platforms.
The above criterion allowed us to focus our attention on Calabash & Appium.
Tool Cost: Both Calabash & Appium are Free & Open Source tools. Other tools like EggPlant
were also on the list, but the cost was too high to be considered as an option.
Test Cases Automated: The ability of the tool to automate the 3 identified test cases. We
found that Appium was suspect when it came to automating WebViews, while Calabash was
quite stable. Also, Appium did not fully support iOS7 at the time of the PoC.
Ease of coding: Calabash uses Ruby+Gherkin, while Appium uses Java. It is easier to add new
test cases to Calabash due to Gherkin BDD (Behavior Driven Development) implementation,
but on the other hand, it is easier to find Java resources than Ruby ones. Hence, both scored
average in our rating.
Code reusability: Both support modular framework design hence code becomes highly
reusable. Sikuli and Ranorex have image processing engines for object identification. This
makes the code highly unreliable and unstructured.
Tool Support: Both are open source and have ample support on various forums. Upgrades for
new operating systems were also fairly quickly available.
Source Code Requirement: For iOS, Calabash adds its own library to the application source
code and then builds and deploys the application. This is a huge negative for Calabash if your
client is not comfortable sharing the source code. Appium requires just the UDIDs of the
Apple device be included in the build. Both however, do not require source code for Android
automation.
Reporting: Calabash has its own reporting mechanism while Appium uses TestNG. Both have
powerful reporting and Exception handling. They both conformed to our reporting
requirements (Error logging, screenshots, run summary, etc.)
Execution on real devices: Both Appium & Calabash can run automation on real devices. The
process of device detection & test execution for both was quite simple and easily
configurable. The execution speed was comparable to that of simulators.
CI integration: Both have strong command-line execution processes and can be easily
integrated into a CI server. We tried firing automation runs on a device for both Calabash &
Appium through a Jenkins job and both ran successfully.
Calabash Appium Others
Calabash Appium
Calabash Appium
Calabash Appium
Calabash Appium
Calabash Appium
Calabash Appium
Calabash Appium
Calabash Appium
Calabash Appium
6
Introduction | Trends | Problem | Solution | Benefits | Enhancements | Conclusion | References
Both Appium & Calabash scored relatively equally on our PoC. In the end, the inability of Appium to support WebViews & the client agreeing to share the iOS source code tilted the decision in Calabash’s favor.
You can find out more about the tool at http://calaba.sh
The Framework:
Next, we had to create an automation framework that will leverage the advantages of the Calabash tool, foremost being, able to create and maintain a common code base for both iOS and Android automation. Although, both applications load the same WebViews, the method of referencing objects for both are a little different. We created an object repository where we maintained the object reference of each object for both Android & iOS. It looks something like this:
Android Repository:
emailId=#email
password=#password
loginButton=#login
iOS Repository:
emailId=WebView css:'#email'
password=WebView css:'#password'
loginButton=WebView css:'#login'
In the code, we reference the object name:
def Login(email,passwd)
if elementExists ($ids[‘emailId’],30)
typeText($ids[‘emailId’], email)
end
if elementExists($ids[‘password’],30)
typeText($ids[‘password’], passwd)
end
if elementExists($ids[‘loginButton’],30)
touch($ids[‘loginButton’])
end
end
Exception handling is done in the above code to capture failures and log them.
Before execution begins, the respective Object Repository is automatically loaded based on the execution environment (iOS / Android).
Finally, the test cases are written in Gherkin (BDD):
Scenario: Login with valid credentials
Given I open the App and login with "[email protected]","spiderman"
And I touch Profile tab
Then I see that the screen changes to profile screen
And step definitions are written in Ruby:
Given /^I open the App and login with "([^\"]*)","([^\"]*)"$/ do |email, password|
homePage = Home.new
homePage.checkWelcomeScreen()
homePage.Login(email, password)
end
By designing such a framework, we were able to create & maintain a single code base for both
Android & iOS.
7
Introduction | Trends | Problem | Solution | Benefits | Enhancements | Conclusion | References
2. Mobile Device Lab:
To support the application testing we had to test on multiple devices. Testing on iPhones is quite
straightforward since their devices are standardized and common throughout the world. But when
it comes to Android there are a wide range of devices, resolutions & OEMs we need to consider.
To finalize the list of devices that we would use for testing, we sought statistics about the active
users of the application on both Stores. We found the top 5 devices & resolutions that had users
actively using the application. Apart from the 4 iPhones (4, 4s, 5 & 5s), we also procured the top 5
Android devices as per the statistics, to be part of our mobile lab. Collectively, they made up
around 80% of the customer base.
We created a separate physical space for these devices in our server room with dedicated power
supply and Wi-Fi support. By using services like VNC, we were able to remotely connect and
control these devices over LAN. We also setup a VPN for this lab for our client to connect to our
network and access these devices over VNC.
Now that the remote access problem was resolved, we had to solve the automation problem,
where we had to trigger automation runs remotely on these devices. Calabash had very good
support for this. We setup a separate Windows machine for the Android devices and a Mac Mini
for the iOS devices. Using Calabash’s command-line arguments, we could trigger automation runs
on specific devices using the device’s UDIDs and IP addresses.
The dedicated Mobile Device Lab
at AFour Technologies
8
Introduction | Trends | Problem | Solution | Benefits | Enhancements | Conclusion | References
3. Continuous Integration:
Having an automated job that will build and deploy applications on the mobile, trigger automation
runs, email notifications, etc. would really help in achieving a 24/7 service. You would not need a
person to trigger off this complicated set of steps and everything would be a hands-off process.
These are just some of the benefits that a CI server offers.
We decided to use Jenkins as our CI Server where we would create jobs, which automate the
entire run execution process.
Common automation execution steps would include:
Picking up the specified “.apk” file (for Android) from a common repository
Or, pickup a specific iOS code base from a repository
Build the code and/or deploy the application on the specified device
Pickup the automation code from the repository
Trigger the automation
Email the execution report to the stakeholders
We created various parameterized jobs in Jenkins and configured all the above steps in it. By
starting the job, all the above steps would be carried out and the final execution report is emailed
to the specified people.
Now, our client has integrated this into their build server. So, whenever new code is checked into
their object repository, the Jenkins jobs are triggered through their integration server using the
very useful Jenkins REST API calls. They can even specify the specific application build number
against which they wish to fire the automation and the rest is taken care by Jenkins which would
also notify them when the automation run finishes by emailing the execution report.
The Windows machine and the Mac PC are configured as Jenkins slaves and the jobs have been
configured to run from these machines.
Jenkins
9
Introduction | Trends | Problem | Solution | Benefits | Enhancements | Conclusion | References
Benefits
1. Common Code Base
By choosing Calabash that can be run
across both platforms (Android & iOS), we
reduced our coding & code maintenance
efforts by nearly 40%. And we will keep
reaping its rewards in the future as we
add more test cases or update existing
application flows.
The same automation engineer can write
scripts for both Android & iOS.
2. Automation Transparency
Now everyone in the chain knows exactly
what test cases are being executed and
the test data being used, because
Calabash mandates you to write your test
cases in Gherkin (Given-When-Then)
format which are easily readable and can
be understood by everyone involved in
the software development chain. Anyone
can easily change the test case and test
data or even add new automated test
cases to the test suite.
We, as QAs, have started writing all our
test cases in a standardized Gherkin
format instead of the old & traditional
test case writing format. Eventually, we
can just pick these tests and plug them
into the framework. Again, saving us
precious time & effort of re-writing
automation scripts.
3. Reporting
With Calabash’s rich HTML reporting
system with screenshots and activity logs,
our automation engineers can spend their
valuable time in writing quality code
rather than worrying about how should
the error be reported. On encountering
an exception, Calabash will automatically
report the error with screenshots and
activity logs.
4. Device Security
Since these devices are physically located
in our secure server room, physical access
is restricted, making the lab secure with
specific access rights to authorized
personnel, thus making the devices
immune to unscrupulous software or
malware installations. Remote access to
the devices is audited and activity log is
maintained.
Data and software being accessed on the
devices are limited to the device making
client data & other collaterals secure.
5. Dedicated Access
All the stakeholders in the software
development cycle have dedicated 24/7
access to the devices making bug
reproduction and testing more
dependable and convenient. We seldom
require resetting the devices since they
are dedicated and we can securely install
third-party application (like Social Apps)
to check compatibility.
6. Testing on Real Devices
Not testing on real devices is always a
risky proposition as already discussed in
this paper. Our testing results are more
reliable since we test on devices. We can
report issues in the software encountered
for low bandwidth, security, performance,
etc. We can also reliably sign-off on builds
with the knowledge that builds are not
misbehaving when installed on specific
OEM operating systems.
7. Continuous Testing
Since the Jenkins environment is
integrated with the client’s build &
integration environment, automation runs
on real devices can be kicked off
immediately without any manual
intervention or waiting for the QA team to
start the process.
Abstracting the mobile environment layer
under Jenkins provided the client with a
more user-friendly access to the mobile
lab.
By automating the entire pre & post steps
required for automation runs, we have
been able to save 1 full resource day
spent in running automation daily on all 9
devices.
8. Scalability & Standardization
The Mobile Device Lab mandates a
particular process for adding devices to
the lab. Such a process standardizes the
devices and their inclusion into the device
lab. It also makes the lab easily scalable.
Common processes can be set for tasks
such as resetting devices, installing /
uninstalling software on the devices,
software or OS upgrade, etc.
We can now quickly scale up our mobile
lab by just duplicating the VMs & Jenkins
jobs and they become immediately
available for automation execution.
Parallel execution of test suites can be
easily configured and fired under Jenkins.
Although parallel execution-ability of the
test suite depends largely on the design of
the framework and test data, if done
correctly, it helps in drastically reducing
the time required to get the test results.
10
Introduction | Trends | Problem | Solution | Benefits | Enhancements | Conclusion | References
Future Planned Enhancements
Code Coverage Statistics
Using tools, we can find the code coverage statistics of an automation run. These statistics can
be a useful indicator for the impact of the automation run. Using tools like Emma for Android
and internal XCode configurations in iOS, we can get a detailed report of the code that was
traversed during an automation run. We can then tweak or add test scenarios to increase this
coverage.
Suite Execution Controller
A controller that will execute customized test suites where the user should be able to provide
details of the tests to run and the Jenkins job should execute only the selected test cases.
Enhanced Reporting Dashboard
Create a dashboard where the user can view current and past execution reports. Implement
live reporting where the user can see the results of the test being executed. Create a
mechanism where the user can select test cases and choose to execute them from the
Reporting Dashboard itself, which will internally call the Jenkins API for job execution.
Page Load Statistics
Creating a simple logging mechanism, which will record the page load time during automation
execution. This should provide us valuable information over various builds about the
degradation of page load times among other things. This should also help us in finding
performance bottlenecks.
11
Introduction | Trends | Problem | Solution | Benefits | Enhancements | Conclusion | References
Conclusion
The customer saw great benefit in the solution that we provided them. They could now integrate
their development builds with testing and get an immediate view on the health of the build to take
a go/no-go decision.
In the end, it’s the entire turnkey solution that counts and that is what we have been successful in
achieving. It was an interesting challenge to execute since there was a merging of multiple
technologies and platforms. Watching the automation get executed on devices, which gets
automatically triggered from a remote build completion was pure pleasure.
There were many small challenges in the way and, for us, to work around those challenges made
this a memorable solution to work on.
The key to realizing the various benefits was to understand the pain-points of the customer and
the QA team and then, design a foolproof solution that would satisfy all the needs. This solution
has been able to address many such problems.
12
Introduction | Trends | Problem | Solution | Benefits | Enhancements | Conclusion | References
References
1 - http://www.gartner.com/newsroom/id/2592315 2 - http://www.businessinsider.in/One-In-Every-5-People-In-The-World-Own-A-Smartphone-One-In-Every-17-Own-A-Tablet-CHART/articleshow/24353354.cms 3 - https://econsultancy.com/blog/62656-45-of-businesses-still-don-t-have-a-mobile-site-or-app-report 4 - http://www.cioinsight.com/c/a/Enterprise-Apps/Mobile-Application-Development-A-Top-CIO-Priority-895960/