Embed Size (px)
Citation preview
W5 Session 4/15/2015 11:00 AM
"Continuous Mobile Testing for
Critical Business Apps"
Presented by:
Steven Winter
FIS Mobile
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073 888-‐268-‐8770 ·∙ 904-‐278-‐0524 ·∙ [email protected] ·∙ www.sqe.com
Steven Winter
FIS Mobile Steven Winter loves building teams that break big things! As the director of quality for FIS Mobile, Steven is responsible for the complete quality ecosystem for America’s leading provider of mobile financial services and the innovative power behind the Starbucks Card Mobile. With nineteen years of experience in quality, Steven has tested through a wide array of technologies from web, firmware, hardware, cameras, phones, desktop, data—and everything in between. He has continually pushed the testing edge and delivered high performance, world-class QA operations at Broderbund, First Data, Leapfrog, PureDigital/Flip, mFoundry, and now FIS Mobile.
4/8/15
1
CONTINUOUS MOBILE TESTING FOR CRITICAL BUSINESS APPS
Steven Winter Director of Quality, FIS Mobile
Speaker Bio
Director of Quality for FIS Mobile responsible for the complete quality ecosystem for
America’s leading provider of mobile financial services, including over 1,200 banks with
over 2,000 apps in market, and the innovative power behind the nations first and most
successful mobile payments initiative, the Starbucks Card Mobile. With 19 years of
experience in quality, Steven has successfully delivered high performance, world-class
QA operations to a wide array of companies like Broderbund, First Data, Leapfrog,
and PureDigital/Flip, mFoundry, and now FIS Mobile.
Steven Winter Director of Quality FIS Mobile
4/8/15
2
• World’s largest banking and
payments provider
• Nation’s largest mobile banking
and payments provider
• > 3000 Mobile Apps in market
• +1,600 banks, +29MM end users
• Most successful payments app
• Most successful banking app
• Award-winning mobile wallet
FIS Mobile
FIS Mobile earned the 2013 Best in Class Mobile Banking Vendor Overall from Javelin Strategy & Research, the group’s highest award for mobile banking vendors.
AGENDA: KEY TAKEAWAYS
• Challenges of being an early leader in mobile quality
• Core requirements of a new approach
• Mobile test automation capabilities
• Role of continuous integration in mobile delivery
• Real benefits in speed and quality
4/8/15
3
MASSMOBILE
ADOPTION
INCREASINGDEVICE
DIVERSITY
RAPIDCLIENT
GROWTH
CONSUMERDRIVEN
INNOVATION
INCREASINGSECURITYEXPLOITS
INCREASEDTESTING
COMPLEXITY
NEEDMORE
DEVICES
UNABLE TO FINISH
TESTING IN 1 SPRINT
THOUSANDS OF APPS TOSUPPORT IN
MARKET
SCALABILITY AND SPEEDPRESSURE
CA
USE
S
EFFE
CTS
What were our big challenges?
What are challenges of being an early leader in mobile quality?
OPERATIONAL TECHNICAL
• Immaturity of mobile test automation
• Deciding what to automate
• Regression “Tsunami”
• MDM – Mobile Device Management
• Staff
• ROI expectations
• Diverging delivery teams
• Mobile testing is difficult
• Device compatibility matrix
• Lack of robust test tools
• Reliable test results
• Configurable script framework
• Technical debt
4/8/15
4
Reducing the Regression Tsunami
Efficient Scripting Architecture
Single, configurable scripts base used by all teams
C o r e P r o d u c t
I n t e g r a t i o n a n d D e p l o y m e n t
H o s t e d S e r v i c e s
UNIT API DEVICE
4/8/15
5
What are the core requirements of a new approach?
ü Vision, Goals, Metrics
ü Robust Script Architecture
ü “Divide and Conquer” Approach
ü Real Devices
ü Need the Right Tools for the Job
ü Immediate Results to the Developers
ü Tight Control of Test Noise
How do we test?
Test Framework Tools/Platform Primary Usage
Unit • jUnit • Fast, code-level test writing and execution • Granular testing from unit test level upwards
Server API • Java • Apache HTTP Client • Web services testing of server API
Client UI
• EggPlant • TouchTest • Calabash • Selenium (SWAP)
• Requirements-based testing of UI flow • Testing against real devices and emulators
Performance • CloudTest • Jmeter/BlazeMeter/New Relic
• In-depth performance runs • Targeted small-scale tests executed frequently
Security
• Fortify • IBM AppScan • NowSecure • ViaLabs (in-house tool)
• Static code analysis • Dynamic server analysis • External scans done by vendors • Internal scans done by security tester
4/8/15
6
FIS Mobile CI System
SVN JENKINS
Build Deploy Test Results Report
STEP 1 STEP 2 STEP 3 STEP 4 STEP 5
What is the current state of mobile test automation capabilities?
Object Recognition
On-device Performance Telemetry (RUM)
Enhanced Mobile Performance Testing
Maturing Test Platforms and Support of CI
Device Clouds
1
3
5
2
4
4/8/15
7
What is the role of continuous integration in mobile?
Cease dependence on mass
inspection to achieve quality.
Improve the process and build
quality into the product in the
first place.
- W. Edwards Deming
“
”
What is the role of continuous integration in mobile?
• Essential to compete in time-to-market and scalability
• Represents a huge paradigm shift
• Provides faster, more specific test results so developers
can work smarter
• Identifies integration issues sooner
• Enables continual quality
4/8/15
8
What are the real benefits in speed and quality?
• 250 hours of manual tests = just over 1 hour every night
• Nightly Regression Tests instead of waiting until feature complete – 4 months to overnight
• 25% reduction in time to market for production deployments
• Earlier bug fixes
• Declining field failures (aka “Escapes”)
• Lots of ancillary tools as a result of developing test and CI frameworks.
One tool alone saved 40+ hours a week!
SUMMARY
• Mobile test technology is catching up
• Very real benefits from mobile CI
• We are freeing the humans!
