WHITE PAPER

Executive SummaryMMobile devices are changing the way we plug in and work/play. Technological innovations thriv-ing on a humongous developer community base across different mobile platforms, has fur-ther leveraged its effortless ease of use and por-tability to the maximum. The numbers are stag-gering: Over 850,000 mobile applications available on Google Play alone, with 48 billion-plus downloads as of Q1 2013.

Just like supply creates its own demand, it also sets its own benchmarks for quality, reliability and efficiency. With so many mobile applica-tions and services vying for users’ attention, quality and performance of these mobile apps and services can make or mar their success, and thus determine their share of the market pie. The The cocooned world of testing mobile applica-tions therefore faces a constant challenge to rec-reate and reinvent itself. Newer methods and techniques have to be harnessed to ensure high quality and performance of mobile applications.

Although methods and techniques used for testing mobile applications are almost similar to desktop and web applications, they come with

their own set of challenges. This whitepaper discusses the challenges of testing mobile applications and Idexcel’s ap-proach to address these challenges.

Challenges in Testing Mobile apps & Services

Challenge 1: Hardware and Platform Fragmentation: TThe continuing emergence of new Operating Systems (OS) and their newer versions and up-grades, coupled with multiple manufacturers and different device specications, deliver com-plex device and platform fragmentation. The challenge is to deploy a mobile application or service on a much larger number of different devices, operating on different platforms, plat-form versions, display congurations, network options, power and memory requirements, touch screens and more. All these directly impact testing and compel organizations to invest heavily on testing their applications, on hundreds of devices in the market with different congurations.

Testing Mobile Apps & Services: Challenges and Solutions

Idexcel

WHITE PAPER

Challenge 2: Mobile Services and Mobile Enterprise Appli-cations

For the last fteen years, mobile phone apps have been dominated by native apps that run locally on the device itself and use the device capabilities. However with the introduction of new device capabilities, more processing power and memory on mobile devices, many internet service providers and enterprises are developing clieclients for mobile devices as well. Of late, a ma-jority of mobile services and enterprise applica-tions are multi-tier architecture, with the code running on the device itself as the front end client, to data and services supplied by more tra-ditional middle tier and data center back ends. Also developers need to integrate their applications with different aggregators, billing system, ERP, multiple 3rd party services, authentication systems, provisioning servers, device manage-ment servers, through web services protocol.

This complex architecture demands the mobile application testers to come up with more effec-tive approaches to perform comprehensive test-ing of all tiers of the application architecture and the integrated systems. Setting up of test envi-ronments with all back end and integrated sys-tems, entails large cost and complexity for testing of mobile applications and services.

Challenge 3: Software Updates

Application developers are tasked to manage mobile enterprise apps and mobile services, right from deployment and through the lifetime of a mobile service, until decommissioned. This involves periodic updates of the entire gamut of application’s backend, integrated systems, mobile applications, and third party services.

MMobile Application developers are required to run regression testing to ensure existing ser-vices are not negatively impacted, when changes are introduced in scenarios such as:

• New systems are integrated in the backend• Patches are applied in the backend• A new updated software is deployed • A new version of a Mobile app is delivered over the air.

TThe challenge for test managers is to explore ways to optimize the cost and effort required for running multiple regression cycles in each of the above scenarios.

Challenge 4: Mobile Application Distribution Fragmentation

AApplication developers need to consider a wide number of choices when distributing a mobile application. There are a number of application distribution channels such as app stores pro-vided by device manufacturers, operators, plat-form providers, OEM Preload, direct download-ing; each with varying policies, restrictions and regulations. Sometimes different versions of the application that match the specications of dif-ferent application distribution channels may need to be created, before developers can submit their application for distribution. This will have a direct impact on testing, including test case design, and test execution effort.

Challenge 5: Application Performance

A slew of mobile applicA slew of mobile applications, mobile enterprise applications, and mobile websites have become major channels for conducting business, im-proving employee efficiency, communications and reaching consumers. Customers and em-ployees routinely expect optimum performance by their applications which ought to be always “on” and responsive. Any break or drop in the app’s performance can be devastating, leading to revenue loss, brand damage, a dip in em-ployee productivity and customer satisfaction. Hence mobile applications and services require rigorous performance testing.

Idexcel

WHITE PAPER

There is not much to distinguish traditional per-formance testing from mobile performance test-ing. However, there are a few key differences and challenges in mobile performance testing, such as:• Simulate mobile networks such as 3G, 4G, Edge, and compare how different factors such as bandwidth, latency, packet loss and signal qual-ity, impact user experience.• Simulate load from a wide range of devices from different manufacturers, operating systems and hardware capabilities• Simulate realistic user scenarios, such as a mixed population of users accessing applica-tions with a variety of bandwidths.• Identify the most appropriate settings for realistic tests is difficult. For instance, combining technologies used such as 3G, Wi, LTE by differ-ent users and the signal quality among other things.

Idexcel’s Approach to address the challenges in Mobile Test-ing

In order to address the challenges in testing mobile applications outlined above, Idexcel has adopted a two pronged approach – develop-ment of a process framework called “mFrame-work” and a device hub called “mLab”,complete with test automation tools.

. Levering our strong end to end mobile testing expertise and practices, the framework, and the device hub, we have successfully addressed the challenges in testing mobile applications.

TThe mFramework is an end to end mobile test-ing framework consisting of processes, check-lists, best practices and automated testing tools for testing mobile applications. The framework ensures quality across all the phases of a mobile application’s testing lifecycle, right from concept to decommissioning of the application. This framework addresses not only the testing of mobile clients, but also test automation includ-ing mobile client, middleware, backend and other systems integrated with the core services. The framework can also be successfully deployed for testing other aspects of the mobile applications ecosystem such as performance, security, compatibility, usability, application store conformance and operator conformance.

IdIdexcel’s mLab is a veritable mobile device hub with different versions of platforms, form fac-tors, memory constraints, manufacturers, con-nectivity, conguration and display density. The device hub is fully integrated with commercial and open source test automation tools to meet the needs of both functional and performance auautomation. For ease of use and collaboration between teams, Idexcel’s mLab can be accessed remotely across any geography through a web interface.

Idexcel

Figure 1: Idexcel mFramework and mLAB

WHITE PAPER

The key testing areas covered by the mFrame-work and mLab are described below.

Compatibility TestingA major hurdle for organizations is how to tamp down the costs incurred in two of the major challenges in testing mobile applications – device and platform fragmentations and soft-ware updates. With Compatibility testing using the mFramework and mLab, Idexcel has been remarkably able to keep the costs of compatibil-ity testing down for our clients, while compre-hensively testing for platform fragmentations and software updates.

Once we receive a mobile application for test-ing, we work closely with customers to under-stand the application’s target market, its archi-tecture, technology platforms, versions of plat-forms supported, device types, and form factor. All the data and information thus gathered is then analyzed and distilled and then put gether to create device metrics for compatibility testing. Our Mobile test experts have created a unique device feature data base to identify the metrics easily and quickly.

We closely work with our customers to choose the main use cases for the compatibility test suite, based on business logic, rather than run-ning the entire test suite on all devices combina-tions. The emphasis is on automating the com-patibility test suite, so organizations can run re-gression for compatibility testing during diffent phases of the application lifecycle, with minimum resources and cost, and without any compromise on quality.

Using idexcel’s mLab, a device hub and a set of open source mobile automation tools that we specialize in, our customers’ bottom line is im-proved, as it allows them to signicantly cut down the CAPEX incurred on procuring hun-dreds of devices and the cost of licensing tools for automation of compatibility testing. We also provide exible billing patterns with an offshore and onsite mix, with billing models ranging from xed price, to price per test case.

End to end Integration and Functional TestingBacked Backed by extensive experience in large scale enterprise product testing and mobile testing, the mFramework optimally meets the chal-lenges of testing mobile services and mobile enterprise apps, characterized by a multi-tier architecture. mFramework works as a primer in managing the test environment and test data ththroughout the application development and service lifecycle. Idexcel’s test experts, with di-verse technology and domain skills in database, servers, mobile clients, web, and desktop cli-ents, have dened the end-to-end test strategy, test environment and test data management of mobile applications with multi-tier architecture. These technology experts harness their knowl-edge in automation to automate use cases, seamlessly switching the context between mobile, web, and desktop, and also ensure that not only the mobile clients are tested during re-gression, but also all aspects of back end inte-gration points

AppStore CerticationTTo combat the challenge from fragmentation of application distribution, our mobile test experts have created a conformance checklist that is at the core of the mFramework. The checklist is in sync with UI guidelines and policies of indi-vidual Appstores and serve as thumb rules for creating test cases, for any application. The gground rules prescribed in the checklist enables our test engineers to spot or detect any devia-tions from the checklist early in testing, before organizations can submit their application to Appstores for certication and publishing. It thus reduces the risk of an application getting rejected and the eventuality of reworking which may critically delay time to market.

Overall, with its strong end-to-end mobile test-ing practices, Idexcel’s mLab and mFramework not only comprehensively address the chal-lenges of testing mobile applications, but also ensures faster time to market, higher quality, while keeping the costs lower, and reducing the

Idexcel

WHITE PAPER

CAPEX spend by the customer. . When mFramework and mLab are combined with idexcel’s testing center of excellence, our customers derive tried, tested and trusted ben-ets, such as:• Faster time to market• Reduced CAPEX spend by using mLAB• Lesser application testing and deployment cost • Reduced cost of testing through effective test automation • Offshore and onsite mix with exible billing type ranging from xed billing to variable billing per test case.

PPerformance TestingTTo address the challenge of mobile application performance testing, Idexcel’s mLab has inte-grated open source and commercially available mobile performance testing tools, with capabil-ity to perform end-to-end performance testing of mobile applications. We have both on prem-ises and cloud based load generators, coupled with software simulators for network emula-tions. We can realistically simulate mobile net-works such as 3G, 4G and LTE, with bandwidth constraints, latency, packet loss, client response time and the way the server is loaded. Our per-formance experts work closely with customers, to create a performance test plan and desigperformance test cases and simulation. The test cases are run in multiple iterations and results provided to the customers’ performance engi-neering team, along with detailed performance analysis that aids in performance tuning.

Idexcel

WHITE PAPER

About the Author

Harsha B N works as a Test Architect in the Mobility division of Idexcel. He has twelve years of ex-perience in development and testing mobile applications. Prior to joining Idexcel Harsha worked with Nokia for eight years in various capacities as Program Manager, Chief Test Engineer, Project Manager working on OTA infrastructure development, Mobile Payments services, S60 SDK.

About Idexcel:Idexcel is an innovative provider of IT Products & Services focused on emerging technologies. We help world leading companies build efficiencies and stronger businesses. With more than 15 years into existence Idexcel’s main focus is client satisfaction and technology innovation. Our industry expertise and a global, collaborative workforce forms the backbone of our services. We offer high degree of skills in Enterprise Applications, Cloud Services, Data-warehousing, Big Data, Analytic, QA & Testing Services, IT consulting and Staffing. Idexcel product line includes: NDS, ERP, and Cync - a revolutionary credit monitoring application for the manufacturing and nancial management. For moFor more information log on to www.idexcel.com Global Headquarters

459 Herndon Parkway Suite 11Herndon, VA 20170Tel: 703-230-2600Fax: 703-467-0218Email: inquiry@idexcel.com

India Operations

“Crystal Plaza” 9, 10 ,11Bhuvanappa Layout, Hosur RoadBengaluru – 560 029KarnatakaTel: +91-80-2550 8830Email: inquiEmail: inquiry@idexcel.com

Idexcel