Embed Size (px)
Citation preview
Rethinking Test Automation: The Case for Moving Beyond the User Interface Rapid development models are forcing quality teams to balance speed with coverage. To enable both effective and efficient testing in this environment, businesses need to replace conventional UI-based automation techniques with more holistic approaches.
Executive Summary With digital disruption and a fiercely competi-tive business landscape, many organizations are forced to balance frequent software upgrades and new feature additions with the ability to support all possible platforms and accelerate time-to- market.1 As pressures mount, many businesses are adopting lean development methods that offer complete lifecycle automation, from envi-ronment setup through build and deployment. These efforts make systems more complex, spawning a higher incidence rate of product quality issues due to insufficient testing.2
Conventional user interface (UI)-based automation approaches cannot handle the needs of these rapid development models, as they are unable to test components or subsystems from the get-go that contain business logic beyond the UI. For speed and coverage, it is important to look beyond automating the UI layer. This whitepaper focuses on unconventional methods and techniques that enable more effective and efficient test automation, early on in the process.
Limitations of UI-based AutomationFocusing test automation on functional behavior at the application UI is an easier option for test strategists, given the abundance of commercial and open source tools and implementation frameworks. In our experience, however, limiting testing to the UI layer may be inadequate due to the following factors:
• Dependence on UI readiness: In most applica-tions, the UI is delivered later in the lifecycle. Waiting for the UI to be ready would mean introducing automation at a much later point in the application lifecycle. Minor UI changes almost always prompt object definitions to be updated frequently, leading to limited reuse and defeating the very purpose of automation.
• Incomplete strategy: For complex applica-tions, in which business logic resides in multiple tiers, database objects and other components, it can be a challenge to cover all application controls and subsequent code interactions.
cognizant 20-20 insights | august 2016
• Cognizant 20-20 Insights
cognizant 20-20 insights 2
• Lack of focus on the business layer: Some components, or a subsystems, containing business logic may need to be tested indepen-dently from the UI-based tests.
• The need for faster turnaround: In projects based on Agile practices and projects that follow weekly/biweekly release cycles, fixing and turning around automated tests is chal-lenging due to shorter testing windows. It may be time-consuming to fix multiple test failures due to UI changes.
Automation beyond the UI is immune to UI changes, resulting in reduced script maintenance efforts. This approach also enables testing of complicated business rules or validating compo-nent-level behavior independent of the UI, thus ensuring a complete testing strategy. Usually, testing at the middle tier or data level is faster than UI tests and provides faster feedback on application behavior. This also acts as an enabler for early testing in upstream environments.
Exploring Alternative Test Automation ApproachesThe conventional strategy is to test end-to-end (i.e., as an end-user would do when using the application UI manually) or, more efficiently, with UI automation. Applications with multi-tiered architectures communicate using APIs, Web services and messaging middleware via the
enterprise messaging bus. The UI (Web browsers, client applications) takes care of end-user inter-actions and presentation.
However, most business rules and functional com-plexities typically reside in the middle tier (see Figure 1). This creates an opportunity to validate business rules and functionality, enhancing coverage at the middle tier. Since communication between the interacting components, or interfac-ing layers, happens through message transac-tions, this creates an opportunity to:
• Simulate component or application behav-ior: Input messages directly into a system, thereby simulating end-user action and reduc-ing UI dependency.
• Launch processes: Make a call to trigger API processing data or rules or, in turn, calling another API.
• Validate business functionality: Make a call to a service or API, parse messages and verify message content and/or querying status in the database for additional confirmation.
We propose the following solution:
• Analyze the technical stack: Study the appli-cation platform, services and message types, protocols and message formats, API available for interactions and database available for conducting additional validations.
A Typical Multi-Tiered Application Architecture
Message Transactions
Message Transactions
Message Transactions
Message Transactions
Interfaces Interfaces Interfaces Interfaces Interfaces
Database Servers Legacy Applications
& MainframesExternal
ApplicationsE-mail Servers JMS. IBM MQ, etc.
MessageTransactions
Database User Interface
Client Systems,Devices & Users
MessageTransactions
MiddleTier
Business Rules Layer
Logic Layer
Load Balancer
ENTERPRISE SERVICE LAYER
Figure 1
• Identify tests for non-UI automation: Ana-lyze the testing flow to identify the functional-ities, components or flows that can be validat-ed using the API or services. Also, identify the inputs for the tests and output to be validated. Any dependencies, such as test data and other third-party services, should be marked for con-sideration in the design phase.
• Evaluate design and tool fitment: Commer-cial tools, such as CA Technologies’ DevTest, IBM’s Rational Integration Tester, SmartBear Software’s SoapUI NG Pro, etc., may be evalu-ated for fit from a technology and design per-spective. This approach should be governed by the costs involved; while these tools are feature-rich, it can be more economical to de-velop a client in Java or native technology. The objective should be to build a scalable solution to support all service or API testing needs within the portfolio.
• Develop solution constituents: Using a framework with reusable components that has multi-protocol and communication support can help reduce redundancies, as well as en-able socialization and know-how. Building an architectural prototype will help establish the concept and encourage stakeholder feedback.
The solution design should provide for segregated test data to facilitate data changes during releases. The solution should run tests individually or in a batch, with single-command execution, potentially run in a continuous inte-gration3 pipeline, to ensure DevOps4 readiness in the development environment (duly supported by stubs or virtualization of services or third-party application behavior, in certain cases). This way, testing is automated using the interface layer rather than the UI tier, which is much slower.5
cognizant 20-20 insights 3
Quick Take
We helped a large investment bank improve its approach to testing and speed defect management. The bank’s cash equities portfolio consisted of regional order entry applications, a fix engine, order crossing and other applications that communicated with order management systems and trade reporting applications using a custom protocol based on TCP/IP. The appli-cations followed weekly or fortnightly release cycles.
Although the bank employed test automation through the UI, a tighter maintenance window for automation scripts resulted in the need for more timely feedback. A custom server-side automation solution was developed to simulate trade entry, replay fix messages and check trade status in the
order management system, as well as verify trade reporting status across various suites. A single test case could be executed in a few seconds that enabled the execution of thousands of tests within a few hours. This resulted in a drastic reduction of cycle time across the portfolio, as well as the ability to run thousands of tests on demand.
More than 60,000 tests a year could be run to certify the quality of software through server-side and UI automation. For applications with mature development processes, the automated suites were integrated in a continuous integra-tion pipeline, providing continuous feedback on quality and reducing the defect detection and defect fixing lifecycle.
Automating Testing at a Large Investment Bank
3cognizant 20-20 insights
cognizant 20-20 insights 4
Potential benefits of our approach include:
• Greater requirement stability and traceabil-ity: Tests break less frequently as interfaces are well-defined and business process inputs and outputs undergo less change and deliver stable responses, hence bringing stability to the automation bed. These characteristics make the automation robust and ready to shift left6 (test execution) from the testing phase to the development phase of the SDLC.
• Early and faster automation via parallel and multiple testing: Techniques such as or-thogonal array and combinatorial testing can be achieved only if tests can be performed be-
yond the UI. This is because UI-level validation prevents the code coverage from being exer-cised on edge conditions.
• Rapid development support: The feedback obtained from the API or service tests can be used to fine-tune the functional behavior at the UI layer, ensuring a better customer expe-rience. These tests are well-suited for Agile/DevOps orchestration because each test runs in a few seconds, and is not susceptible to UI changes. Hence, it works well in continuous integration execution mode and can be made ready for DevOps.
Quick Take
A large commercial bank in the U.S. sought to automate its trade order management and settlement systems.
The upstream applications book and manage trades throughout the lifecycle. The trades are then routed using RESTful Web services to an application server, and then to a Temenos banking application. In addition to existing UI-based automation with HP Unified Functional Testing, we recommended using service-based automation for faster feedback and setup of a continuous feedback mechanism using a continuous integra-tion methodology (see Figure 2).
A Java-based solution was used in the services layer to inject trades into the booking applica-tion and further verify the state of the trade throughout the lifecycle and its final status in the trade database. Approximately 100 tests are run in roughly 45 minutes and provide quick feedback for different trade types. The entire test cycle span was reduced to one day for both service and UI tests. The automated test could be readily integrated with TeamCity for continuous integration.
Saving Test Time and Effort for a Large Commercial Bank
Beyond UI: Continuous Testing Across the Lifecycle
CYCLE TIME
UI-based automation using HP Unified
Functional Testing
API/SERVICES/MESSAGES
Automation introduced late in lifecycle
Service-based automation introduced early in lifecycle
DownstreamCommunication
UpstreamCommunication
Upstream Communication
DownstreamCommunication
Validation of Transactions & Message Communication
USER INTERFACE
Trade OrderManagement System
Trade SettlementSystem
Database
DOWNSTREAM APPLICATIONS
Figure 2
cognizant 20-20 insights 5
Looking ForwardTesters usually employ black box testing techniques, with a range of input conditions, using the UI and verifying outputs. This leaves the core business logic residing in the API layer to be tested by developers, many of whom are not well schooled in the art and science of quality assurance.
The right approach for maximizing test coverage is to create a balance between UI and services or API automation. This approach provides more timely feedback and an ability to test early in the development process, whereas UI automation aims to validate logic in the UI layer and integra-tion layer and mimics the actual end user of the system.
Tests at the non-UI layer will also increase code covered at the branch level, providing an option to test more edge conditions. Our experience has been that automation coverage can comprise service and server-side/API automation to a range of 60% to 70%, and UI automation to a range of 30% to 40%. However, a measured approach is to implement impact-based testing techniques that measure and map tests that cover all appli-cation code and suggest gaps where lines of code are not covered.7
With the adoption of cloud and the advent of microservices architecture,8 API and services-based testing becomes inevitable. QA organiza-tions need to ascertain the quality of components early and independently in the development lifecycle before releasing them for further con-sumption by related components or subsystems.
Footnotes1 “Modernize Application Development to Succeed as a Digital Business,” Gartner, Inc, March 2016, https://
www.gartner.com/doc/3270018/modernize-application-development-succeed-digital.
2 Donald Firesmith, “Common Testing Problems: Pitfalls to Prevent and Mitigate,” Carnegie Mellon University, April 2013, https://insights.sei.cmu.edu/sei_blog/2013/04/common-testing-problems-pitfalls-to-prevent-and-mitigate.html.
3 “Continuous Integration,” Wikipedia, https://en.wikipedia.org/wiki/Continuous_integration.
4 “DevOps,” Gartner, http://www.gartner.com/it-glossary/devops/.
5 Brandon Getty, “4 Advantages of API Testing,” QASource, August, 2014, https://blog.qasource.com/4-advantages-of-api-testing/. The author elaborates the time effectiveness of automated API tests over con-ventional graphical user interface-based automation.
6 “Shift left” is an approach to software testing in which testing occurs early (to find defects early) in the SDLC lifecycle as opposed to limiting testing to just the testing phase of the SDLC. For more information, see https://insights.sei.cmu.edu/sei_blog/2015/03/four-types-of-shift-left-testing.html.
7 Steve Cornett, “What Is Wrong with Statement Coverage,” Bullseye Testing Technology, http://www.bullseye.com/statementCoverage.html. The author explains the risks and misconceptions of a commonly used code coverage metric. Also see “A Brief Discussion of Code Coverage Types,” Jason Rudolph’s blog, June 10, 2008, http://jasonrudolph.com/blog/2008/06/10/a-brief-discussion-of-code-coverage-types/.
8 Microservices architecture is an approach to developing software and enterprise applications with the ability to scale, covering a range of applications (Web, mobile) on multiple devices and platforms running in a cloud or hybrid ecosystem, and moving away from simple, monolithic Web applications. For more infor-mation, see https://en.wikipedia.org/wiki/Microservices and Mark Russinovich, “Microservices: An Appli-cation Revolution Powered by the Cloud,” Microsoft Azure, March 17, 2016, https://azure.microsoft.com/en-us/blog/microservices-an-application-revolution-powered-by-the-cloud/.
About Cognizant
Cognizant (NASDAQ: CTSH) is a leading provider of information technology, consulting, and business process services, dedicated to helping the world’s leading companies build stronger businesses. Head-quartered in Teaneck, New Jersey (U.S.), Cognizant combines a passion for client satisfaction, technol-ogy innovation, deep industry and business process expertise, and a global, collaborative workforce that embodies the future of work. With over 100 development and delivery centers worldwide and approxi-mately 233,000 employees as of March 31, 2016, Cognizant is a member of the NASDAQ-100, the S&P 500, the Forbes Global 2000, and the Fortune 500 and is ranked among the top performing and fastest growing companies in the world. Visit us online at www.cognizant.com or follow us on Twitter: Cognizant.
World Headquarters500 Frank W. Burr Blvd.Teaneck, NJ 07666 USAPhone: +1 201 801 0233Fax: +1 201 801 0243Toll Free: +1 888 937 3277Email: [email protected]
European Headquarters1 Kingdom StreetPaddington CentralLondon W2 6BDPhone: +44 (0) 20 7297 7600Fax: +44 (0) 20 7121 0102Email: [email protected]
India Operations Headquarters#5/535, Old Mahabalipuram RoadOkkiyam Pettai, ThoraipakkamChennai, 600 096 IndiaPhone: +91 (0) 44 4209 6000Fax: +91 (0) 44 4209 6060Email: [email protected]
© Copyright 2016, Cognizant. All rights reserved. No part of this document may be reproduced, stored in a retrieval system, transmitted in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the express written permission from Cognizant. The information contained herein is subject to change without notice. All other trademarks mentioned herein are the property of their respective owners.
About the AuthorNandan Shinde is Associate Director of Projects with Cognizant’s Quality Engineering and Assurance business unit and is part of the company’s Technology Center of Excellence. Currently a test automation architect, he previously worked as a test automation expert, project lead, automation strategist and program manager over a period of 11 years with Cognizant’s banking and finance services clients. His responsibilities pivot around building and socializing automation frameworks, as well as implementing solutions – from front-end, middle-tier and data automation strategies – to leading automation program for large banking customers. He handles automation consulting engagements across the vertical industries that Cognizant serves. He can be reached at [email protected].
Codex 2192
