June 2015
Wipro & Experitest Co-webinar:
Perfect Your Mobile App with Load Testing and Test Automation
Guy Arieli
CTO
Experitest
Sudheer Mohan
Director - Mobility
Certification &
Automation
Wipro Technologies
Speakers
June 2015
Agenda
1
2
3
4
Non Functional Testing for Mobile
Demo- See Test Integration with LoadRunner
Case Studies- Major Bank and US Retailer
Challenges in Mobile Load Testing
6
5
Mobility Automation
Wipro’s Automation Framework
© 2015 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL4
Non Functional Testing
Performance issues
( Slow , freeze, crash)
Heavy mobile resource utilization
Uses heavy network bandwidth
Accessibility compliance
Low application ratings
Unsatisfied customers &
Potential loss of Business
Enterprise challenges Platforms &
platform versions
Device
Manufactures
Real world
conditionsService
Providers
© 2015 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL5
Wipro Non-Functional test solution
Mobile
AppsMobile
Platform
Mobile
network
App profiling
Interruptions
App response time
Exploratory testing
Accessibility testing
Network condition
Handover situations
Data transfer
Sensitive data validation
Simulate mobile user load*
Load testing*
Stress testing*
Security testing
Load testing *
Stress testing *
Security testing
Enterprise
Services
Focused non-functional testing* Covered as part of SeeTest load testing solutions
3rd Party
Integration
Social media
Crash analytic
tools
Analytic tools
Mobile users uses your application
more extensively
Mobile generate more load
The number of mobile users
already extended the number of
desktop user.
Thin-Thick Client
Mobile
applications are
replacing Web
Application
“The mobile
enterprise business
application market is
projected to double
by 2018”
Gartner
Mobile App is relatively
more complex
The Market shifts from
web to Mobile
Mobile generate
more load
Challenges in Mobile Load Testing
MobileWeb
JSon over http, JSon over TCP,
XML-RPC, Soap, REST, RMI …Single Protocol- HTTPProtocol
Json, XML, Free Text …HTMLData
Both synchronized and non-
synchronized
Synchronized –
request / responseCommunication
Multiple threadsSingle threaded appConcurrency
Unlimited methodsUnified security –
browser basedSecurity
multiple requests
Single request / single
response -
Submit / next page
Transaction
Thin-Thick Client
Mobile apps are relatively complex
Mobile load create bigger
effect on your servers
Mobile App is relatively
more complex
The Market shifts
from web to Mobile
Mobile generate
more load
Challenges in Mobile Load Testing
Mobile load create bigger effect on your servers
One of the main
cause to load on your
server is number of
concurrent session
Mobile session takes
longer time to be
processed due to
mobile network
conditions- result in
over load on your
servers
Same number of user
mobile compare to
desktop can result in
over 20% more load
on your servers.
Traditional Load
Testing is Synthetic
Mobile load create bigger
effect on your servers
Mobile App is relatively
more complex
The Market shifts
from web to Mobile
Mobile generate
more load
Challenges in Mobile Load Testing
Traditional Load Testing is Synthetic
In traditional load testing the application under test is being removed and a synthetic network traffic is being generated
Typical request can have dozen or even hundreds of fields. One of the big challenges of load engineer is to
understand the nature of all of those fields
How does your application traffic
look like?
Your application open TCP connection to
the server (SYN-> SYN,ACK<- ACK->)
Your application send a request.
The server send a response.
Your application is closing the connection.
What would you find in your
application request?
Key value property data structure include
information like:
Request information
Request parameters
Session ID
Transaction ID
syntactic load is Risky
Traditional Load
Testing is Synthetic
Mobile load create bigger
effect on your servers
Mobile App is relatively
more complex
The Market shifts from
web to Mobile
Mobile generate
more load
Challenges in Mobile Load Testing
syntactic load is Risky
2 type
of correlations:
Why
do I need to
modify the
request
fields?
User parameters to request fields
Server response to user request
To simulate different user inputs i.e different user name / password or different UI
form population
To support internal server protocol - a very common server protocol scenario is that after
the login request, the server will return a session ID. The client need to use this session ID
in any request that is been perform.
syntactic load is Risky
It will create no effect on the server
The server will ignore this field
The server will ignore the entire request
The request will cause a server internal error
With Synthetic load you don't really know what will happen, it is a trial and error process
Main
Risk
What will
happen if I will
modify the
request field?
Hard to validate
syntactic load
syntactic
load is Risky
Traditional Load
Testing is Synthetic
Mobile load create bigger
effect on your servers
Mobile App is relatively
more complex
The Market shifts
from web to Mobile
Mobile generate
more load
Challenges in Mobile Load Testing
validate syntactic load
Performance teams need to adopt methodologies to validate that the generated load is valid.
Few approaches can be:
Database / UI validation to verify that the transaction generated the expected
effect
Looking at the server log files and find transaction completion log messages.
Monitor the server vital to verify the load effect the server CPU, memory, number
of threads…
Hard to validate
syntactic load
syntactic
load is Risky
Traditional Load
Testing is Synthetic
Mobile load create bigger
effect on your servers
Mobile App is relatively
more complex
The Market shifts
from web to Mobile
Mobile generate
more load
Challenges in Mobile Load Testing
As you generating the synthetic load you would like to generate few interaction with the servers based on real
devices running your real application.
Monitoring your load using real device transaction
Provide real
transaction
response time.
Provide validation
to the syntactic
load effect.
BenefitsProvide response
time reference
(when executed
without the
syntactic load)
End-to-End Product Suite - Manual, Automation, Monitoring, Load, Network Virtualization, Manager
• Productivity tool for Manual testers
• Report with screenshots of each step
• Used in conjunction with SeeTestCloud
• Allow remote debugging
• Device Lab located onsite (in the customer VPN)
• Devices can be reserved and accessed by any remote user
• Used in conjunction with SeeTestAutomation/Manual for testing of remote devices
SeeTestManual
SeeTestCloud
• Automation tool
• Same test runs on different devices
• Plugs into all testing & CI environments:
SeeTestAutomation
SeeTestNetworkVirtualization
• Device performance under varying network conditions
• Simulating mobile network bandwidth latency and reception while on the move
SeeTestLoad
• Real device generated mobile load
• Full integration to LoadRunner, Jmeter, etc.
Mobile Add-On for UFT• Full & complete integration into HP UFT:
• Record a test, edit it, run it, and view results
• Integration of object repository and more…
SeeTestManager
Free
Tool!
N
L
G
U
Mobile Add-On for LoadRunner• Full integration to LoadRunner through
SeeTestAutomation
© 2015 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL26
More releases in a year
Parallel Tracks- Phone/ Tablet/ Web
Less time to test
Numerous Devices/ Platforms/ Versions
Not just UI Testing- Services/ Backend
End to End Testing
Why is automation a must?
In Mobility- Automation is the Key!!
© 2015 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL27
1. Early Automation Support Agile & BDD projects
Faster Testing by in- sprint automation
Optimize testing cost by expanding
automation coverage
2. Platform Agnostic Multi platform and device support
Reusable libraries
Jump start automated testing in new
projects
Reduce testing costs3. End to End Automation Services & Back end validation
Test in various environments by
pointing to different systems
behind the scenes
High quality apps
4. Unattended Testing
Parallel Execution
DevOps adaptation
Fully Automated
Integration with Test Management
& CI solutions
Faster test execution
Shorter test cycles & more release support
Wipro Automation
Framework
Wipro Automation Solutions Overview
5.
© 2015 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL28
5
Early Automation Approach
64
2
Application Flow
Objects in each screen
Automatable scenarios
Analyze
Wireframes
Develop Framework
Build reusable functions/ pseudo scripts
Object repository mapping table ( Key->Value pair)
Develop
Automation scripts
Map actual Objects to
Object repository
Execution Ready
Dry Run
Ensure modified scripts
are running fine
Re Engineer
Do modifications to the
scripts to rectify the issues
found in dry run and make it
stable
Execution
Start test case execution on
test devices
Dev-ops or CI/CD
In-sprint automation
Accelerated Testing
1 3
© 2015 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL29
Automated
Build
Checks
in CodeAutomated
Deployment
Automated
Reporting
Automated
Testing
Un-Attended Testing
DevOps support
Quick testing
Integration with existing test management solutions
© 2015 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL30
Demo
Build on Java, leveraging SeeTest
Supports keyword driven and data driven approach
Supports parallel execution for faster testing
Easy to configure and use even by a manual tester
Supports early automation approach
Supports integration with 3rd party test management tools
Supports un-attended testing
Framework Highlights
• Configure the environments via environment sheet
• Write the test cases using Excel based template
using keywords
• Demonstrate un-attended test execution using
Jenkins by automatic deployment of build
• Demonstrate parallel execution using SeeTest on
one Android and iOS device
• Integration with HP ALM – Updating results
• Demonstrate standalone report
Demo – In brief
© 2015 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL31
Case in Point: Complete Mobile Automation- Leading Bank in US
Testing of Bank’s mobile application on
iOS and Android devices
A flexible and highly reusable
automation framework which supports
unattended testing and integration with
existing test management tool (HP
ALM)
Backend mainframe validation for
transactions from mobile devices
Reduce testing cycles and improved
ROI for mobile automation
Up to 40%* effort savings with automation in
comparison with manual testing
Continuous integration using Jenkins and
enhanced reporting capability by automated results
being updated in HP ALM
Reduce testing cycles using early automation and
parallel execution
Shift-left automation script development by
improving ROI for mobile automation
Automated backend validation for each transaction
Easily extendable to other OS platforms if needed
Leveraged and customized Wipro mobile
automation framework which supports:
Platform agonistic scripting
Data driven
Highly reusable
Parallel automation execution
Leveraged SeeTest partnership to build a
Mainframe validation component and
incorporated in to Wipro framework
Integrated Wipro automation framework with
Jenkins and HP ALM for un attended testing with
SeeTest as the automation tool
Implemented in-sprint automation using Wipro
automation framework
Implemented Parallel automation test execution
using SeeTest for faster testing cycles
Problem StatementWipro Solution
Benefits!
* Based on 60% automation and early automation approach
© 2015 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL32
Case in Point: Automation Productivity Enhancement- US Retailer
Testing on iOS and Android devices
High automation script reusability across
customer mobile applications
Build an automation framework which is
easy to use and allows non-technical
testers to do automation
Faster testing cycles and improved ROI
for mobile automation
Enhanced automated test reports based
on customer requirement
Reduce overall testing cost by reduce as
much as commercial tools
Flexibility to run automation across
various test environments
Jump start automation by enabling non technical
testers to do automation using a simple excel
spreadsheet
Improved automation development productivity
using excel based framework
Flexible and highly reusable automation framework
which is completely configurable via excel sheets
Shift-left automation script development by
improving ROI for mobile automation
Developed a Java based automation framework
which allows writing of test cases using Excel
sheets which enables even non-technical testers
to do automation
Core technical team to write new keyword
functions
Framework supports parallel execution by
configuring the devices in the configuration file
Custom made reports for holistic view of test
execution across all devices in a single report
Integrated with Jenkins for Continuous Integration
support
Implemented Parallel automation test execution
using SeeTest for faster testing cycles
Problem StatementWipro Solution
Benefits!
© 2015 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL33
Case in Point: Enhancing Customer Experience for a Leading US
Retailer
Reduce production defects reported by
end users ( Performance issues, app
freezing , no-product found)
Improve app rating of existing mobile
applications
Ensure seamless customer satisfaction
Ensure latest/upcoming features &
technologies like iBeacon, barcode
scanning , iwatch, etc.
Improved app ratings from 3 to 4.5 (iOS) & 1
to 4.3 (Android)
Rated as one of the fastest growing retail
mobile apps
Stable application under real world scenarios
Tested app under various network conditions
and identified the real world issues faced by
real user
Conducted usability testing to analyze the
application from an end user perspective and
identified the usability issues
Conducted app profiling to find out the
resource utilization of mobile application
Conducted network monitoring to ensure
optimum network data transfer
Conducted field/in-store testing to ensure
seamless end user experience
Simulate network load and backend server
load and identified the issues faced by user
in peak load times
Conducted app comparisons to identify the
issues and benchmark the app against other
competitors
Problem StatementWipro Solution
Benefits!