Market Guide for Mobile Application Testing Servicesdocshare01.docshare.tips/files/31233/312334486.pdf · 2016. 6. 3. · DevOps methodologies, requiring organizations to reskill

This research note is restricted to the personal use of [email protected]


G00273008

Market Guide for Mobile Application TestingServicesPublished: 29 October 2015

Analyst(s): Susanne Matson, Maritess Sobejana

Improving customer experience in a digital economy is critical to success,driving organizations to invest in mobile application development. Sourcingmanagers must understand the complexity of the mobile testingrequirements to ensure service providers can meet business expectations.

Key Findings■ The rapid developments of mobile and other digital technologies is necessitating agile and

DevOps methodologies, requiring organizations to reskill and maintain the relevant skills in-house. However, lack of available skills in the market is driving higher usage of external serviceproviders since there are multiple skills needed for mobile development and testing — plus therequirements constantly change.

■ Mobile testing services are inherently different to traditional application testing services, as theyrequire added complexity, such as the testing of multiplatforms, networks, OS, as well asdifferent devices. User experience is also key in mobility as it may generate unique requirementsfor the testing process as well as considerations for testing by geography and security testing.

■ Lack of forward planning and engaging sourcing as well as IT, proactively drives the business tobypass quality test planning to meet time-to-market needs. This results in higher risk with a lackof testing metrics in the contract in order to hold service providers accountable.

RecommendationsSourcing managers should:

■ Include experience in agile and iterative development methods as key selection criteria whenchoosing mobile application testing service providers, since it requires a different skill level ofresources than traditional testing services. Due to high collaboration, a nearshore or on-sitepresence is necessary to be fully effective.

■ Document the business requirements — understanding the bimodal aspect of the developmentand testing required. Since mobile application testing is part of both Mode 1 and Mode 2,



addressing both the need for stability as well as agility. This may impact the selection criteria aswell as the provider experience desired.

■ Engage with the business early in order to develop an efficient testing strategy. Avoid risk ofbusiness bypassing in accessing these applications without sufficient planning for testingquality — in lieu of speed.

Market DefinitionGartner's 2015 CIO survey revealed that currently, mobile ranked as the fifth highest priority for CIOspend. Naturally, ensuing testing demands must follow. However, there is lack of awareness byclients on the complexity of testing required in mobile development, as well as how it differs from

traditional application testing, so it makes it difficult to select the right service provider.1

Mobile testing services are different, often requiring more complex than traditional applicationtesting services since they require new skill sets and considerations — such as the testing ofmultiplatforms, networks, OS as well as different devices. These services are needed to test thefunction, performance, compatibility and so forth, of a mobile application — inclusive of bothmanual and automated tests. Therefore, internal testing organizations need to consider mobileapplication testing service providers as part of their quality strategy.

User experience is key in mobility by creating unique requirements for the testing process. Withdigital business and customer experience, applications and data is increasingly available and visibleto external constituents — especially customers. Thus, quality, reliability, security and adaptabilityneeded for external use of systems is far beyond internal usage — and this is especially true withmobile applications. The competitive landscape of mobile application testing service providerscomprises a variety of traditional and nontraditional players including:

■ Full service IT service providers

■ Pure-play testing or mobile testing service providers

■ Digital that offers mobile testing

■ Testing specialists who specialize in mobile testing entirely in the cloud

■ Crowdtesting providers

Buyers of mobility testing services range from business leaders, sourcing managers, applicationdevelopment managers and testing managers. Services to support clients' mobile application,quality control and quality assurance to ensure the high quality demanded by consumers, clients,stakeholders and end users are highly requested by buyers today. This research includes all testconsultancy, advisory, implementation, execution and management services related to mobile apps,we also examine how it functions on various devices.

For the purpose of this Market Guide, we do not include crowdtesting service providers. However,we recognize that this can be effectively used for mobile testing and these providers are covered in

Page 2 of 33 Gartner, Inc. | G00273008



"Market Guide for Crowdsourced Application Testing Services." Additionally, testing productcompanies and providers are covered in "Market Guide for Mobile Application Testing."

Market DirectionApplication development and business managers (sales, marketing operations that include mobileapplications) are facing pressure to ensure the consistent high quality and user experience of mobileapplications. Mobile is one of the key technologies encompassed within digital business initiatives;where demands of the business include shorter time to market, keeping costs low in often complextesting environments along with maintaining the high levels of quality, are top of mind for businessleaders. Financial losses, legal issues and irreparable brand image would be a high price to pay ifmobile application fails to deliver the outcomes expected.

Since mobile applications drive digital business and success is determined by user experience, theyare generally led and adopted by business units rather than the IT department. IT and sourcingtherefore, needs to be fully engaged and be knowledgeable about the requirements of differentbusiness units and their needs in order to deliver high-quality mobile apps. Thus, this entailsworking closely with the business units on mobile testing services demands — which iscontinuously challenging traditional IT management strategies — as it requires agile and flexibleorganizations.

This market is developing rapidly. Currently, it is still fairly fragmented and unstructured, whichmeans more potential confusion for buyers. The need to support these applications is quite new sothe capabilities to manage the required testing are not often present within the organizationsinternally. Anticipating further market maturity, there will be many tactical relationships forming tomeet intense demand.

Mobile Application Testing Tools Market Will Drive Use of External Testing Services

The mobile tools market is expected to rapidly grow with new entrants continuing to appear withvarying product offerings that will influence the services market over the subsequent five-yearperiod (see "Market Guide for Mobile Application Testing"). The reality is that before the market evenreaches maturity, the initial confusion and strong ramp-up in demand is mixed with the need foressentially new skills and capabilities. Looking even further into the future, mobile technologies (andhow they are integrated) are likely to continue evolving rapidly, and will be influenced by othertechnologies such as the Internet of Things (IoT), beacons, wearables, smartwatches and machinelearning. These aspects will drive increased reliance on service providers. Typically, this is the casewith any emerging market.

Lack of Digital Skills Is a Major Incentive to Consider Mobile Application TestingServices

The investments IT service providers make into digital technologies and in continuously trainingtheir staff is a key driver for buying mobile application services. Since digital skills such as mobileapplication development and testing require updated products and apps, it needs ongoing

Gartner, Inc. | G00273008 Page 3 of 33



investment to keep talent and technology refreshed frequently — simply due to the sheer speed ofinnovation.

Test automation skills are becoming more essential alongside digital business. The speed withwhich development of digital technologies occur exposes a new reality for organizations, wherebythey will not be able to reskill or retrain fast enough — there are simply too many skills required thatgo through constant changes. Speed is also accelerating, so the time to market for a mobile projectcan be a matter of days or weeks, as opposed to legacy applications in the traditional sense.Organizations need to make a choice about whether they are willing or able to maintain all theseskills in-house. The market broadly suffers from a shortage of various specialized mobile applicationtesting skills, which proves a significant challenge — even for service providers. However, becausethis is their business, they make focused investments in these areas placing them in a betterposition to train, maintain and acquire the right skills in comparison with internal testingorganizations.

Bimodal IT Will Drive the Need for Agility

Mobile technologies will evolve rapidly and principles such as bimodal IT will help to facilitateenterprise agility. These constant changes, both at the technology and consumer level, placessignificant demands on organizations, which make defining a mobile strategy difficult. Gartner hascovered the area of bimodal in research noting that there are two modes to consider — Mode 1 andMode 2 — which, when combined, serve to address both the stability and the agility requirementswithin organizations.

As organizations develop bimodal IT capabilities, they will need to form application developmentteams with different objectives, processes, metrics, disciplines, and consequently, varying needs inorder to support both Mode 1 and Mode 2 projects. In mobile application development, Mode 1refers to a stable infrastructure (API, mobile app integration layer) where mobile apps can beintegrated with back-end systems; Mode 2 refers to the front-end client of the app (app UI and userexperience [UX]) that the user interacts with.

Mode 2 includes more emerging areas of interest to the organization in terms of new businessdevelopment with quicker turnaround times, agile approaches, new service methods and deliveringfront-end applications. Therefore, a different approach to the various providers needed in thesourcing mix is likely to be necessary.

We often find providers moving into offering specialized services or modes of delivery, as well asmore flexibility and ease of use. Application development and sourcing managers need to be awareof the new opportunities and providers in this emerging market, as well as know how best to workwith these providers and also obtain services that cater for the varying needs of the organization.

Mobility itself is also constantly changing, at both the technology level and the societal level, withthe resulting demands on organizations. This fluid environment makes defining a mobile strategycomplex. How can the conflicting needs of enterprise stability and rapid response be resolved? Theuse of a bimodal strategy — one mode addressing stability and the other agility — allowsorganizations to maximize development efficiency and respect information control policies, yetaddressing the urgency of volatile, competitive, seasonal and unplanned requirements.




Adoption of Agile Methodologies Require More Automation

Developing bimodal IT capabilities demand that IT organizations begin to deal with agile methodsfor application development and delivery. Agile methodologies include frequent meetings betweenthe development team and application team, with much shorter cycles for updates and revisions.This results in the need for increased levels of automation and tooling support, to achieve therequired speed of delivery. However, automation requires different tools and frameworks as well as adifferent skill set to deliver manual execution. Moreover, initial cost for automation is high —licensing cost, training, creation of scripts, maintenance, and so on. The automation toolsdeveloped for traditional testing services might not work for mobile application testing, so providersare building new or extending the tools they already have, to increase the level of automation —leading to accelerated testing, increased efficiency and reliability of testing. Automation tools arefrequently requested by clients, so creating a portfolio of strong offerings can prove to be thedifferentiator for providers. Resources provided by the service provider must also be able to work inan agile, dynamic and fluid development environment. This is a completely different way of workingwith service providers. Shorter sprints and shorter cycles, as well as a much more collaborativeworking methodology mean different requirements of service providers. With an agile approach, it isnot best practice to use service providers for all of the work — it helps to have internal qualitysponsors that will oversee the projects. Effective engagement is dependent upon collaborationbetween business leaders, application developers and testers. Working together, especially on thesame site, often allows for face-to-face collaboration that reduces misinterpretations ofrequirements and enables testing during development. Having project managers on the team whocan lead and interpret results is essential.

Market AnalysisWith the expansion of new technologies such as mobile being further integrated into organizationsthat are forming digital business opportunities, there are multiple challenges with organizationsneeding to ensure that their mobile applications are of good quality.

Testing Mobile Applications Is Far More Complex Than Traditional ApplicationTesting

The inherent complexities in testing mobile applications are compounding the challenges tomaintain quality assurance. When testing mobile applications, the process must cover multipleplatforms, multiple networks of varying stability, different OS including frequent OS upgrades,including the different devices the apps are running on (devices like Samsung Electronics, Nokia,Apple and HTC). All these options and combinations require a different view. A mobile application ispart of a system, for example, mobile payment that may involve the application of a Near FieldCommunication (NFC) reader and a point of sale (POS) system. Furthermore, a prominent testingrequirement in the mobile applications space is user experience. Mobile is increasingly a way fororganizations to communicate with clients, win business in new areas or reach various stakeholders— so user experience is paramount. Therefore, usability and flexibility of mobile applications is topof the agenda. In the future, the user experience will likely become even more central, with thedevices and services less so. Further considerations such as security (like testing the robustness of




data protection solutions) are potentially even more important for mobile applications, since user-facing components of systems will move increasingly outside of the firewall. In addition,geographical and language aspects are elements that also require contemplation in the testing ofmobile applications.

To effectively test mobile applications, clients are using various methods, such as creatingcompetency centers for critical platforms (iOS, Android), carriers (Verizon, BT), as well as securityand digital design. Clients also use mobile testing service providers to complement internalcapabilities and/or crowdsource critical missing competencies.

Fundamental Changes in Engagement Models and Pricing

Due to the nature of mobile applications — in terms of shorter time to market, higher speed, smallerprojects and more collaborative, agile work — it can be more effective to use a combination ofservices, techniques and providers. The RFI and RFP processes for traditional application testingprojects are often too long, therefore quick decisions frequently need to be made. Also, flexibility incontracting is likely more important when negotiating with mobile testing pure-play firms or digitalagencies. For guidance on some of the key criteria that can be used when selecting a mobileapplication testing service provider, see "How to Choose a Mobile Application Testing ServiceProvider."

An increasing number of smaller, more nimble providers are moving into this space and can often bevery effective as part of a sourcing strategy. These providers come from varying backgrounds andcan be good, flexible, quick, cost-effective and skillful additions. However, note the variety ofprovider and carefully consider which type of provider would make the most suitable fit and whatthey specialize in.

The use of on-site or onshore resources as part of the sourcing mix is also essential to ensuresuccess. Agile and DevOps methodologies (with the speed generally needed in these projects),have a positive effect on service providers by winning deals that have sufficient resources closer tothe client. As service providers and clients become more experienced by working on agile projects,offshore usage may be increased over time to control costs. As a result, we see service providerscreating more delivery centers closer to the client (if needed) and testing professionals on-site,embedded within project teams with utilization of offshore resources.

Cloud is a model frequently considered for mobile application testing. Here devices can be run onmultiple systems in the cloud and tested by creating a Web-based mobile environment on asimulator to access the mobile app. Public cloud is of interest to many clients that do not mind thepotential lack of control or privacy issues, while data-privacy sensitive clients will prefer the privatecloud. The cloud can be easily accessed; it is cost-efficient and scalable, fast and effective.

The flexibility in pricing and scalability that testing-as-a-service (TaaS) represents is frequently usedfor mobile application testing and as long as it is clear what the client is paying for, this can prove tobe a good alternative.




Multiple and Varied Partnerships

Mobile testing service providers partner extensively with mobile testing product companies such asExperitest, Keynote, Mobile Labs, Neotys, Perfecto Mobile and TestPlant. Service providers utilizethese partnerships to leverage infrastructure, expertise and automation capabilities from thesevendors in order to offer a full end-to-end mobile testing service solution. There are many newproduct companies in the marketplace and therefore each service provider can have a different setof partners. Consolidation is anticipated to be rife in the mobile testing tools market, which meansmultiple changes in ecosystems are expected in services.

Many service providers are also developing tools using open-source environments, which reducescost to end users due to savings on licensing costs of utilizing tools from commercial tool vendors.

Representative VendorsThe vendors listed in this Market Guide do not imply an exhaustive list. This section is intended toprovide more understanding of the market and its offerings.

In Table 1, we provide a quick overview across a selection of providers that offer mobile applicationtesting services. Following this table is a short profile of some of these service providers. For thepurpose of shared characteristics, they are grouped by:

■ Global full-service providers

■ Regional midtier, full or subset IT service providers

■ Pure-play testing service providers

■ Mobile service providers

Each group is preceded by a short explanation of the characteristics of the type of provider.

The purpose of this research is to show just a small representative number of providers in eachgroup. Please note that this list is not exhaustive. If providers are not listed within a particular groupthis does not signify that they don't necessarily provide these services as well. So, for exampleAccenture, Capgemini Group, CGI, CSC, Deloitte, HCL, NTT Data, SQS and Wipro all offer theseservices and appear in Table 1, but they may not be detailed in the short profile list that follows.




Table 1. Overview of Mobile Application-Testing Service Providers

Provider/Website Category Head-quar-ters

Client Segments Industries (Primary Focus as Indicated byProvider)

Partnerships

Accenture(www.accenture.com)

Global full-service provid-er

Ireland Primary focus isover 10,000 em-ployees

■ Telecommunications

■ Financial services

■ Healthcare

■ High tech

■ Public sector

■ Perfecto Mobile

■ Selenium

■ HP

Altimetrik(www.altimetrik.com)

Regional midt-ier full or sub-set IT serviceprovider

U.S. Primary focus isorganizations withmore than 1,000employees

■ Automotive

■ Healthcare

■ Banking

■ High tech

■ Industrial machinery

■ Life sciences

■ Professional services

■ Retail


■ Travel and transportation

■ Xamarin

■ Appurify

■ DeviceAnywhere

Atos(www.atos.com)


France Primary focus is2,500 employeesand above

■ Automotive

■ Banking

■ Consumer products

■ HP

■ Veracode

■ Perfecto Mobile


https://www.accenture.com/

http://www.altimetrik.com/

http://www.atos.com/





Partnerships

■ Media


■ Retail

■ Sports and entertainment


b-acceptance(www.b-accept-ance.com)

Pure-play test-ing serviceprovider

France Focuses on allsegments


■ Healthcare

■ High tech

■ Media


■ Retail and wholesale distribution

■ Ranorex

■ TestLink

■ HP Quality Center

Capgemini(www.capgemini.com)


France Primary focus is2,500 employeesand above

■ Banking

■ Insurance

■ Communications

■ Other financial services

■ Public sector

■ HP

■ Perfecto Mobile

■ DeviceAnywhere

■ Appium

■ Mobile Labs

CGI(www.cgi.com)


Canada Primary focus is2,500 employeesand above


■ Communications

■ Public sector

■ HP

■ Perfecto Mobile

■ Keynote


http://www.b-acceptance.com/


https://www.capgemini.com/

http://www.cgi.com/en





Partnerships

■ Healthcare

■ Energy

Cigniti Technologies(www.cigniti.com)


India Primary focus isorganizations be-tween 1,000 to2,500 employees

■ Banking

■ High tech

■ Insurance


■ Retail


■ Experitest (SeeTest)

■ Ranorex

■ TestPlant

■ Neotys

■ Perfecto Mobile

■ Mobile Labs

■ HP

■ Soasta

Cognizant(www.cognizant.com)


U.S. Primary focus is2,500 employeesand above

■ Automotive

■ Banking


■ Healthcare

■ Higher education

■ High tech

■ Insurance

■ Life science

■ Media

■ Experitest

■ Mobile Labs

■ Perfecto Mobile

■ Applause


http://www.cigniti.com/

http://www.cognizant.com/





Partnerships

■ Retail




■ Utilities

CresTech SoftwareSystems(www.crestechglo-bal.com)


India NA ■ Banking

■ Insurance

■ Media and entertainment


■ Travel

■ Amazon Web Serv-ices

■ Armorize Technolo-gies

■ IBM

■ Orane Consulting

CSC(www.csc.com)



■ Healthcare

■ Insurance

■ Banking

■ High tech

■ Public services

■ Manufacturing

■ Transportation

■ Retail

■ Perfecto Mobile

■ Soasta


http://crestechglobal.com/


http://www.csc.com/





Partnerships

Deloitte(www.deloitte.com)




■ Public

■ Healthcare

■ Manufacturing

■ Communications

■ Mobile Labs

■ Perfecto Mobile

Epam(www.epam.com)



■ Banking


■ Healthcare

■ High tech

■ Life sciences

■ Media

■ Retail



■ Experitest

■ Xamarin

■ Perfecto Mobile

■ Selenium (opensource)

■ Appium (open source)

HP(www.hp.com)


U.S. Focuses acrossseveral ranges

■ Healthcare

■ Public sector

■ Communications

■ Media and entertainment

Perfecto Mobile


http://www2.deloitte.com/global/en.html?icid=site_selector_global

http://www.epam.com/

http://www8.hp.com/us/en/home.html





Partnerships

■ Transportation

IBM(www.ibm.com)



■ Banking


■ Healthcare

■ Insurance

■ Media

■ Retail




■ Perfecto Mobile

■ IBM Rational

■ DeviceAnywhere

InfoStretch(www.infostretch.com)

Mobile serviceprovider

U.S. NA NA ■ Kony

■ Perfecto Mobile

■ Sauce Labs

■ Soasta

■ Mobile Labs

■ PhoneGap

■ Atos

■ Veracode


http://www.ibm.com/

http://www.infostretch.com/





Partnerships

■ Xamarin

Infosys(www.infosys.com)


India Primary focus is2,500 employeesand above

■ Automotive

■ Banking


■ Healthcare

■ High tech


■ Insurance

■ Life sciences

■ Oil and gas


■ Public sector

■ Retail



■ Utilities

■ Experitest

■ Mobile Labs

■ HP

■ Perfecto Mobile

■ Jamo Solutions

■ Sauce Labs

Kualitatem(www.kualitatem.com)


U.S. Primary focus isless than 500 em-ployees

■ Banking

■ Healthcare

■ HP

■ IBM

■ Selenium


http://www.infosys.com/

http://www.kualitatem.com/





Partnerships

■ Media

NTT Data(www.nttdata.com)


Japan Evenly spreadfrom organizationswith 500 employ-ees, but most or-ganizations havemore than 10,000employees

■ Healthcare


■ Insurance


■ Perfecto Mobile

■ DeviceAnywhere

■ Jamo Solutions

Olenick & Associates(www.olenick.com)


U.S. NA ■ Banking

■ Capital markets

■ Healthcare

■ Utilities

■ HP

■ Microsoft

■ IBM

■ Soasta

■ Perfecto Mobile

■ Tahoe Partners

QASource(www.qasource.com)


U.S. Primary focus isup to 2,500 em-ployees

■ Banking


■ Engineering

■ Construction and operations

■ Healthcare


■ High tech

■ Insurance

QASource is tool-agnostic, al-though it works with multiple part-ners


http://www.nttdata.com/

http://www.olenick.com/

https://www.qasource.com/





Partnerships

■ Life sciences

■ Media


■ Retail

QA Mentor(www.qamentor.com)


U.S. Primary focus isless than 500 em-ployees

■ Banking


■ Engineering

■ Construction and operations

■ Healthcare


■ High tech

■ Insurance

■ Media

■ Retail


■ Perfecto Mobile

■ Neotys NeoLoad

■ Selenium

QualiTest(www.quali-testgroup.com)


U.S. Ranges across allsizes

■ Healthcare


■ Communications

■ Perfecto Mobile

■ TestPlant

■ QualiSystems


http://www.qamentor.com/

http://www.qualitestgroup.com/






Partnerships

■ Manufacturing ■ Keynote Mobile Test-ing

RapidValue Solutions(www.rapidvaluesolu-tions.com)


U.S. Primary focus isorganizations witheither less than500 or between1,000 and 2,500employees

■ Banking


■ High tech

■ Insurance

■ Media

■ Oil and gas


■ Retail


■ Appium

■ Perfecto Mobile

■ SeeTest

■ BlazeMeter

■ SoapUI

SLK Software(www.slk-group.com)


India Primary focus isorganizations withmore than 1,000employees

■ Automotive

■ Banking


■ Insurance

■ Retail

■ Utilities

■ Wholesale distribution

■ Perfecto Mobile

■ Appium

■ HP UFT


http://www.rapidvaluesolutions.com/


http://www.slk-group.com/





Partnerships

StarDust(www.stardust-test-ing.com)


France Primary focus isorganizations withmore than 2,500employees

■ Banking

■ Healthcare

■ Insurance

■ Life sciences

■ Retail


■ Wholesale distribution

NA

SQS(www.sqs.com)


Germa-ny

Primary focus is2,500 employeesand above

■ Banking

■ Other financial services

■ Insurance

■ Automotive

■ High tech

■ Services

■ Other manufacturing

Perfecto Mobile

Syntel(www.syntelinc.com)



Banking ■ Kony

■ Microsoft

■ Salesforce

■ Experitest


http://www.stardust-testing.com/

http://www.stardust-testing.com/

http://www.sqs.com/en-group/index.php

http://www.syntelinc.com/





Partnerships

■ CA Technologies

■ Selenium

Tata ConsultancyServices (TCS)(www.tcs.com)



■ Banking

■ Healthcare

■ Insurance

■ Retail



■ Perfecto Mobile

■ Experitest

■ HP

■ Mobile Labs

■ TestPlant

■ Optimyth Software

■ Tricentis

■ Soasta

Tech Mahindra(www.techmahin-dra.com)



■ Banking


■ Healthcare




■ Experitest

■ Perfecto Mobile

■ HP

Testing4Success(www.testing4suc-cess.com)


Canada NA NA NA


http://www.tcs.com/Pages/default.aspx

http://www.techmahindra.com/pages/default.aspx


http://www.testing4success.com/






Partnerships

Virtusa(www.virtusa.com)


U.S. Primary focus isorganizations withover 10,000 em-ployees

■ Automotive

■ Banking

■ Healthcare

■ Insurance

■ Media


■ Experitest

■ HP Mobile Center

■ Appium

Wipro(www.wipro.com)



■ Retail

■ Banking

■ Insurance

■ Healthcare

■ Mobile Labs

■ Perfecto Mobile

ZenQ(www.zenq.com)


India Primary focus isorganizations withless than 500 em-ployees

■ Banking


■ Healthcare

■ Insurance

■ Retail

■ Appium

■ Silk Mobile

■ pCloudy

■ Keynote

NA = not applicable

Source: Gartner (October 2015)


http://www.virtusa.com/

http://www.wipro.com/

http://zenq.com/



Global Full-Service Providers

Full service providers offer various IT services including testing services practice with the mobileapplication testing capabilities as a subset of that practice. Mobile testing is an area of strongdemand in the market and most of the full-service providers of a certain size offer mobile testingservices. (See Table 1 for further examples of providers.) Buyers can consider using these providersfor larger, complex or global deals.

Atos

www.atos.com

Atos is a global service provider that has a testing practice with over 8,000 resources. Atos also hasa focused mobile application testing practice with just under 800 resources. Atos has most of itsregional capability based in Europe (U.K., Belgium, the Netherlands and Luxembourg [Benelux],Spain, France), although an increasing part of its revenue stems from North America and variousother countries. More strategic focus has been placed on the Atos testing practice over the pastcouple of years, with increased investment into the practice, its rebranding into an independent unitand a raised profile within the organization. Mobile testing is one of the areas of innovation that theprovider stated it wanted to focus on from 2015 onwards, as it leverages its global strategicpartnership with InfoStretch (a mobile application service provider that is detailed in the MobileService Providers section, it delivers mobile application testing services). Atos has a mobile testingcenter of excellence (COE) in India and specialized smart mobility delivery centers. Atos mobiletesting services include end-to-end automated mobile testing solutions, as well as context-drivenmobility services.

Cognizant

www.cognizant.com

Cognizant is a global service provider with a large testing practice, which includes mobileapplication testing services. Cognizant has an India-centric delivery model, with most of its revenueoriginating in its home market of North America but also offers services in multiple other regions andcountries.

Cognizant continues to move further into the areas of digital and mobile by leveraging itstechnology labs, assets, intellectual property (IP) and partner ecosystem. Currently, the provider has10 global labs focusing on mobile activities located in the U.S., China, India, U.K. and a few othercountries in Europe and Asia/Pacific. Cognizant offers an end-to-end mobile testing solutionleveraging internal developed IP, including automation for mobile testing via its "mint" platform and"BlueSim," a solution for connected device virtualization using Bluetooth technology.

Examples of testing services offered include functional testing, automation testing, field networktesting, user experience testing, nonfunctional testing, security, cross browser/platform capabilitytesting and application validation.


http://www.atos.com/

http://www.cognizant.com/



IBM

www.ibm.com

IBM is a full service global provider with various service areas, including testing and mobileapplication testing services. IBM is a global provider that offers local support as well as globaldelivery centers. Mobile application testing is one of the current strategic focus areas of IBM. It usesits portfolio of applications to support clients — from consulting and engagement through to IBM'smobile test factory. IBM has developed mobile test facilities across India, Europe, U.S. and Asia/Pacific, where it performs testing on multiple browsers, devices and platforms, including offeringssuch as functional, performance and network testing services. IBM leverages its Combinatorial TestDesign (CTD) solution for reducing mobile test cases in the mobile test centers and it has alsoworked with one of its partners, eggPlant, to develop a mobile automation solution.

An example of a recent initiative within the portfolio of mobile testing services includes its AppleGarage, in partnership with Apple. The Apple Garage, which includes testing automation, integratesIBM and Apple tools for continuous testing built on agile practices.

Infosys

www.infosys.com

Infosys is a full service global provider with a testing practice, as well as mobile testing capabilities.Infosys includes just under 700 resources in its testing practice that are dedicated to mobileapplication testing services. Infosys has extensive offshore centers which it leverages for clientsprimarily based in North America, Australia and Europe — although it can also offer on-site ornearshore resources where required. Within the service offerings of digital assurance, it offersmobility quality assurance (QA) services, including mobile device testing, mobile app QA and mobileQA consulting. These services comprise offerings such as functionality, platform integration,performance, localization, interoperability, usability, security, UX benchmarking, network andautomation testing services or provide support around creating a mobile test strategy. Within itsmobile testing practice, Infosys has invested to accelerate and improve its effectiveness on clientengagements by creating various tools and frameworks. These include a mobile QA defectrepository tools, Mobile Scenario Testing Rapid Prototyping Engine (Mobile STRiPE) for earlyvalidation; mobile application performance monitoring (mAPM) tool for device side performancemonitoring; network impact testing solution (iNITS) for testing mobile applications under varyingnetwork conditions; a mobile platform specific validation framework as well as "iMobiLoad," aperformance QA tool and "CARES" a user experience test framework. Infosys also includes theoffering, Test Unit (TU)-based charging model, and another offering, integrated mobility-testing labon cloud, which can be accessed from customer locations worldwide.

Tata Consultancy Services

www.tcs.com

Tata Consultancy Services (TCS) is a global full service provider. It has one of the largest testingpractices in the market and its portfolio also includes mobile application testing services withapproximately 1,600 resources. TCS has recently scaled up and reinvented its digital assurance


http://www.ibm.com/

http://www.infosys.com/

http://www.tcs.com/Pages/default.aspx



offerings — including mobile — by introducing specialization skills into its practice for mobiletesting, investing in a customer collaboration center at Santa Clara and a customer experiencecenter at Chennai Lab, and also building partnerships.

TCS incubates startups and partnerships in the mobile ecosystem and uses its dedicated mobiletesting labs in India and U.S. to train its workforce, support delivery, handle proof of concept (POC)programs and actually show customers how they test and assure in this digital space. Services aredelivered in traditional models as well as mobile-testing-as-a-service.

TCS has created mobility frameworks, other tools and IP to support delivery in this practice. MobileReusable Automation Framework for Testing (mRAFT) is a framework for mobile test automation."iBrowse" provides automated cross-browser compatibility testing and result verification acrossmobile browsers, it groups test cases based on business functions while "TCS Hy5 Test," which ispart of the TCS Hy5 Suite, enables the development, testing and deployment of applications on avariety of mobile platforms through automation.

TCS offers assurance services on various mobile platforms, devices and applications, which includeservices such as network/environment virtualization, automation, performance, security,accessibility, functional and nonfunctional testing on and off the cloud.

TCS primary focus is end-to-end mobility assurance engagements. Typically, TCS's engagementsspan all aspects of assuring the mobile business, although it only addresses certain parts of themobile application testing life cycle in certain engagements.

Tech Mahindra

www.techmahindra.com

Tech Mahindra is a full service provider with a mobile application testing practice.

This company uses both on-site delivery along with its mainly Indian global delivery centers tosupport its clients. Most of Tech Mahindra's revenue originates in North America but it also has alarge client base in some countries in EMEA. Its mobile application testing practice in Europe is, infact, its largest in terms of revenue. Mobile application testing services operate in a center ofexcellence with 60 dedicated resources. The company has developed an automation framework,"Mobile Application Testing Ecosystem" (MATE) that focuses on cloud-based automation-testingservices. There is a dedicated lab available for testing with an inventory of mobile devices. Thepractice has capabilities for functional, automation, localization, user experience, compliance,performance, security and cloud enabled testing.

Regional Midtier, Full or Subset IT Service Providers

These are smaller service providers with testing service capabilities including mobile. They offerservices in one region, one country and/or with a more specialized services portfolio. This group ofproviders can be a good option both for small and midsize organizations looking for a local deal, butsome of these providers are also large enough to manage those bigger deals.





Altimetrik

www.altimetrik.com

Altimetrik was founded in 2012 and is part of the Vattikuti Ventures. Altimetrik provides businesstransformation and technology solutions for enterprises in areas of digital, connected solutions andconsumer technologies through five capability centers in the following areas, user experience anddesign, prototyping, application development and integration, information engineering andVisualization, quality engineering and automation and DevOps. The provider employs 240 of itsengineers in its mobile application testing practice and draws most of its revenue from NorthAmerica, despite its presence in multiple other regions.

Its Quality Engineering Automation Capability Center, is where its mobile testing service resides,using its delivery centers in India and Uruguay to deliver mobile testing solutions. Altimetrik has acloud-based mobile device lab called "Appcentric," a framework that connects to an array of realmobile devices enabling remote collaboration and test automation. Altimetrik has also developed anautomation framework enabling scriptless, domain-specific test automation called "Framework forEarly Automated Testing" (FEAT). Altimetrik provides engagement models including on-premises,cloud hosted and subscription-led offerings.

Epam

www.epam.com

Epam is an IT service provider with application development as its core competency.

The company has around 13,600 employees with 2,200 of those based in its testing practice, wheremobile application testing services is part of the portfolio. Most of its consultants are based inEastern Europe although it has been increasing its footprint in the North America and Asia/Pacificregions, as part of its efforts to make the company more global. Meanwhile approximately 50% ofits revenue is derived from the U.S. with the rest originating from Europe. A large portion of itsrevenue is in mobile application testing, stemming from North America and Western Europe. Bothits testing practice and automation are areas of growth — and its main focus. It has offeringsincluding mobile test automation and mobile performance testing. Epam offers mobile testingservices as an end-to-end product, a quality engineering service that includes automation solutions,specialized security testing, mobile performance, usability and user experience assessment.Competency centers, established methodologies, tools and accelerators support these offerings.Epam leverages a unified automation framework, which is reusable between Web and mobile in itstesting practice.

Olenick & Associates

www.olenick.com

Established in 1998, Olenick & Associates concentrates on offerings within software development,including testing. The company primarily focuses on North America with delivery centers in LatinAmerica and Northern Ireland. Olenick & Associates has created a mobile test lab that it can use totest applications across multiple mobile devices. It provides functional testing services to clients as


http://www.altimetrik.com/

http://www.epam.com/

http://www.olenick.com/



well as manual testing and test automation. For test automation, it utilizes open-source functionaltesting tools; and for mobile user experience testing, the provider has created an internalframework.

RapidValue Solutions

www.rapidvaluesolutions.com

RapidValue Solutions is a provider focusing on the digital transformation for the enterprise alongthree dimensions — mobile, omnichannel and cloud. Having an end-to-end approach is the primarybusiness, but the provider has established a dedicated sales team bringing in projects specificallyfor mobile application testing services. RapidValue Solutions employs around 400 resources in total,of which, 72 are based in the mobile testing practice. The company has offices in the U.S. andIndia; it is looking to expand in the European market by opening its first office in the U.K. Thecompany follows the scrum development model where the testing team is part of the developmentteam during the entire project development stages. It also has a dedicated mobile application-testing center (DMTC) and a dedicated lab with devices, simulators and automation testing tools.The testing portfolio of RapidValue Solutions includes test automation, functional and useracceptance, performance, usability and security testing.

SLK Software

www.slk-group.com

Part of the SLK Group, SLK Software is a service provider with around 1,500 employees. SLKSoftware offers mobile application testing as part of an overall testing service, as well as a discretemobile application testing services practice with 50 resources. The company offers testing servicesthrough different engagement models including test automation and performance. SLK Softwareprovides TaaS with an outcome-based pricing model, leveraging its own automation solution and itsown private mobile cloud solution.

StarDust

www.stardust-testing.com

StarDust engages with its clients throughout the process of creating and launching their digitalsolutions, including mobile and websites. The mobile application testing practice comprises 60resources and most of its revenue originates from North America with some clients in EMEA also.The company provides testing of mobile applications — such as bug tracking or checking if themobile website is responsive to different types of media and devices. It also operates upstream ofthe production, both in the design and development phases of project. StarDust works with clientsensuring that they comply with current technical Web and mobile standards as well as meeting anyfunctional requirements. The company provides the following types of testing services — functional,performance, security, location, user experience and automation.



http://www.slk-group.com/

http://www.stardust-testing.com/?lang=en



Syntel

www.syntelinc.com

Syntel is a full service provider including mobile application testing services in its portfolio ofservices. Syntel serves clients through its mobile test center of excellence (COE) where it also offerscloud-enabled mobility testing services. Syntel has just over 24,000 employees with 1,300resources in mobile application testing services. Syntel has developed adaptable testmethodologies for mobile applications and offers both manual and automated testing. Syntel has aprivate cloud mobility-testing lab where it leverages its mobility test automation framework (MTAF).It is built on top of open stack where vendor-specific products can create mobile test automationsuites. It also includes lab-as-a-service (LaaS) into its engagement model providing access to itsprivate lab. Syntel offerings include functional application testing services, performance testing, testautomation services, user experience testing and security.

Syntel's test lab has multiple devices, emulators and tools. One example is the in-house developedcross-browser, Test Automation Framework xWAT for testing browser compatibility on both iOS andAndroid platforms.

Virtusa

www.virtusa.com

Virtusa is an IT services company providing IT consulting, technology and outsourcing services,with just under 10,000 resources in total. The company was founded in 1996 with offices andtechnology centers throughout the U.S., Europe and Asia. Virtusa uses its global delivery model toprovide services to its clients, which includes testing services. In testing, the company has 148dedicated resources focusing on mobile application testing services where it utilizes its mobility laband test automation frameworks to serve clients. Mobile testing services fits into its managedservice test COE; it also offers a mobile subscription service and a TaaS test factory service formobile automation and execution.

Virtusa leverages its test automation platform "centroid," enabling end-to-end digital testautomation.

Pure-Play Testing Service Providers

Pure-play testing service providers gain the vast majority of their revenue from testing services.These firms can have various specializations, but all include mobile application testing within theirtesting portfolio. Buyers can consider these providers for specialist skills, smaller projects, country-specific knowledge or for their flexible and nimble approach.

b-acceptance

www.b-acceptance.com

Founded in 2011, b-acceptance is a pure-play testing firm with 35 resources focused on mobileapplication testing services; mainly focusing on the digital world (media, m-commerce, e-


http://www.syntelinc.com/

http://www.virtusa.com/




commerce). The company's approach concentrates on the validation of business processes.Consequently, b-acceptance offers services executing test strategy techniques, design scenarios,recovery of data, test execution and reporting of test campaigns. "TRA Ready Mobile" is a testingoffering that is preconfigured and immediately available to validate clients' mobile projects.

Cigniti Technologies

www.cigniti.com

Cigniti Technologies is a pure-play testing service provider with a mobile application testingpractice. Cigniti Technologies employs 250 resources that are specialized on mobile applicationtesting services. Cigniti Technologies offers mobile testing to its clients across U.S., U.K., Australia,New Zealand, Asia and the Middle East. Lately, this company has grown rapidly, with a valueproposition primarily centered on IP tools and accelerators, but also frameworks, test COE andinvestments in global enterprise labs and partnerships. Cigniti Technologies uses its mobile testCOE to combine best practices for its mobile practice and it has just invested in building a next-generation mobility lab with cloud capabilities. The company's mobility testing services includemobile test automation, mobile performance benchmarking, customer experience and digitaltesting, mobile apps testing, functional testing, compatibility testing and security testing for bothenterprises and independent software vendors (ISVs).

CresTech Software Systems

www.crestechglobal.com

CresTech Software Systems is a pure-play testing service provider with a mobile application testingservices practice. The company has over 300 consultants spread across its three delivery centers inIndia and North America. CresTech Software Systems has developed a testing platform, "Mobo-Sure," covering all aspects of mobile application quality, including functional, automation,performance and security testing across various platforms, devices and carriers. Through thisplatform, the company offers mobile application testing services including device compatibilitytesting, automation and performance as well as security testing services.

CresTech Software Systems has set up a mobile testing lab; enabling multidevice testing that alsoincludes a patented platform-agnostic test automation framework in its portfolio. CresTech SoftwareSystems has also developed a reusable test asset library.

Kualitatem

www.kualitatem.com

Kualitatem is an independent software testing and information systems pure-play company withoffices in the U.S. and United Arab Emirates (UAE). It provides services for Web, mobile anddesktop applications and its mobile application testing services practice includes 35 resources intotal. Kualitatem has developed a platform for mobile application testing covering key testing types


http://www.cigniti.com/


http://www.kualitatem.com/



such as functional, security and performance, compatibility, usability and localization testing. Thecompany has a lab where it provides access to customers while testing using real-time devices.

QA Mentor

www.qamentor.com

QA Mentor is a pure-play testing services company established in 2010. Mobile testing is part of its"Lab Compatibility Services," which is also offered separately to clients. Although all 106 companyresources are able to support mobile testing projects, there are 28 dedicated resources focused onjust mobile testing. Most of the mobile practice revenue is generated in North America. Despite this,revenue is also derived from Europe (France, Germany, Ukraine) and Russia, as well as India. QAMentor uses its global delivery centers to provide services to clients of varying size and scale usingits own methodology. The company offers mobile testing services along with both functional andnonfunctional areas of testing. An e-commerce platform is what the company considers to be oneof its key "sweet spots."

QASource

www.qasource.com

QASource employs around 550 QA engineers. QASource works with clients in North America withdelivery provided by its Indian delivery center. This pure-play testing service provider offers mobileapplication testing services as part of its portfolio. Depending on client requirements and growth,the number of mobile testers fluctuates slightly, but around 240 resources are dedicated to themobile application testing practice. QASource has some local presence in the shape of projectmanagers in North America but the rest of its team is based in India. QASource has a mobile devicelab of over 150 real devices utilized by its experienced test engineers.

QualiTest Group

www.qualitestgroup.com

QualiTest Group is the second largest pure-play testing service provider globally. Most of its revenueoriginates from its operations in North America, Israel, U.K., the Netherlands, India and Australia.The total application testing resources of QualiTest Group comprise 2,000 resources — and withinthis practice it also offers mobile testing services. Mobile test automation is a differentiator and itprovides coverage on all major devices and platforms.

The company also uses mobile test labs where they can provide access to environments fordistributed testing. QualiTest Group aims for flexibility with tools that can be generated quickly toget projects running as well as by offering on-demand services. QualiTest Group provides testingthrough the life cycle of consulting to managed services as well as crowdtesting services. Servicesinclude tests such as test process improvement, security, accessibility and load testing.


http://www.qamentor.com/

https://www.qasource.com/




Testing4Success

www.testing4success.com

Testing4Success is a pure-play testing service provider focusing on usability testing services. Thecompany ensures quality on Web, mobile or desktop applications. It has built QA test labscontaining hardware, software and tools to create multiple test environment. Testing4Success offersmobile-application testing services, including testing on real devices. The services offered byTesting4Success include functional testing, compatibility testing and usability testing.

ZenQ

www.zenq.com

Created in 2003, ZenQ is a pure-play testing services company with around 300 resources, with 75residing in the mobile application testing services practice. ZenQ works with clients in the U.S.,U.K., Canada and other parts of Europe (via its offices in the U.S. and India). It organizes its mobileapplication-testing offering along functional, performance, security and automation testing services.It has COEs for major testing functional areas. ZenQ has developed expertise across mobile OSplatforms, offers device-agnostic testing, testing on mobile simulators and mobile platforms, andtesting automation using commercial and open-source tools.

Mobile Service Providers

Mobile service providers have mobility as their core focus. Buyers can consider these providers fortheir specialist mobile skills and focus, often using modern approaches.

InfoStretch

www.infostretch.com

InfoStretch is a pure-play mobile testing provider of next-generation mobile application life cycleservices, ranging from design and development to testing and sustenance. The company employsaround 700 resources and is focused on the U.S. market, but has also expanded into the Europeanmarketplace. Since 2004, it has been offering solutions and services to enterprises in the mobileecosystem. The company uses its two mobile development centers to support delivery as well as atest lab with access to various OS and devices. It develops accelerators on top of commercialexisting tools or open-source tools and has developed automated tests for mobile handset devicetesting. The range of services offered by the company includes functional, nonfunctional,performance testing, usability testing, user experience, language testing and security testingservices.

Market RecommendationsMobile testing is no longer simply about the functionality of an application. It must cover all theaspects of platforms, networks and devices in order to meet user expectations, which require very



http://zenq.com/

http://www.infostretch.com/



specific skills from providers. Evaluate mobile testing service providers based on all the aspects ofcomplex mobile testing requirements such as infrastructure, skills, tools, performance, usability andend-to-end quality approach and strategy.

In addition, understand the specific demands of the organization and the objectives they are tryingto achieve, which need to account for aspects such as variations in the development methods andprocesses that must be supported (waterfall, agile, mixed), support of bimodal, partnerships andavailability of tools, accelerators and IP.




Acronym Key and Glossary Terms

API application programming interface

Benelux Belgium, the Netherlands and Luxembourg

CIO chief information officer

COE center of excellence

EMEA Europe, the Middle East and Africa

iOS iPhone operating system

IoT Internet of Things

IP intellectual property

ISV independent software vendor

LaaS lab-as-a-service

MTAF mobility test automation framework

NFC Near Field Communication

OS operating system

POC proof of concept

POS point of sale

QA quality assurance

RFI request for information

RFP request for proposal

TaaS testing-as-a-service

TCS Tata Consultancy Services

UAE United Arab Emirates

UI user interface

UX user experience




Gartner Recommended ReadingSome documents may not be available as part of your current Gartner subscription.

"Market Guide for Crowdsourced Application Testing Services"

"Market Guide for Mobile Application Testing"

"How to Choose a Mobile Application Testing Service Provider"

"Magic Quadrant for Application Testing Services, Worldwide"

"Essential Metrics for Outsourcing Testing Services"

Evidence

"Flipping to Digital Leadership: The 2015 CIO Agenda"

1 Gartner analysts received 47 inquiries on mobile application testing services between September2013 through September 2014, with an increase between September 2014 through September2015 of 64.




GARTNER HEADQUARTERS

Corporate Headquarters56 Top Gallant RoadStamford, CT 06902-7700USA+1 203 964 0096

Regional HeadquartersAUSTRALIABRAZILJAPANUNITED KINGDOM

For a complete list of worldwide locations,visit http://www.gartner.com/technology/about.jsp

© 2015 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. or its affiliates. Thispublication may not be reproduced or distributed in any form without Gartner’s prior written permission. If you are authorized to accessthis publication, your use of it is subject to the Usage Guidelines for Gartner Services posted on gartner.com. The information containedin this publication has been obtained from sources believed to be reliable. Gartner disclaims all warranties as to the accuracy,completeness or adequacy of such information and shall have no liability for errors, omissions or inadequacies in such information. Thispublication consists of the opinions of Gartner’s research organization and should not be construed as statements of fact. The opinionsexpressed herein are subject to change without notice. Although Gartner research may include a discussion of related legal issues,Gartner does not provide legal advice or services and its research should not be construed or used as such. Gartner is a public company,and its shareholders may include firms and funds that have financial interests in entities covered in Gartner research. Gartner’s Board ofDirectors may include senior managers of these firms or funds. Gartner research is produced independently by its research organizationwithout input or influence from these firms, funds or their managers. For further information on the independence and integrity of Gartnerresearch, see “Guiding Principles on Independence and Objectivity.”


http://www.gartner.com/technology/about.jsp

http://www.gartner.com/technology/about/policies/usage_guidelines.jsp

http://www.gartner.com/technology/about/ombudsman/omb_guide2.jsp

Documents

Market Guide for Mobile Application Testing Servicesdocshare01.docshare.tips/files/31233/312334486.pdf · 2016. 6. 3. · DevOps methodologies, requiring organizations to reskill