Gowrishankar Sundararajan
QA Manager
Tata Consultancy Services, Canada
Automated Acceptance testing by Developers &
Automated Functional Testing by Testers
Overview on Traditional Agile Testing Approach
Comparison of Traditional Vs New Proposed Approach
Recommended New Agile Approach with Benefits
Executive Summary
2Gowrishankar Sundararajan Nordic Testing Days 2016
Case Study – An implementation overview with KPI’s
Introduction to Agile
Introduction
Frequent delivery of deployable business value
Frequent changes in requirements
Close collaboration between entities
Prioritization of work is done by the
stakeholders on the basis of requirements and
focus is then paid on the highest risk issues as
the work progresses
Customers/business owners make business
decisions, developers make technical decisions
Agile Flow
3Gowrishankar Sundararajan Nordic Testing Days 2016
Traditional Agile – An Overview
Test Design
Day 1 Day 2 - 8 Day 9- 13 Day 14-15 Day 1
Coding Defect Fix ClosurePlanning
Test Execution ClosurePlanning
QA Plan day wise
Day 1 – Previous Sprint Handover to UAT/Business and
Scope finalization for new Sprint
Day 2-8 – Test Design phase in parallel with Coding
Day 9-13 – Test Execution – Functional Testing
Day 14-15 – Test Execution – Regression Testing for the
current sprint
In a normal 3 weeks Sprint window, below is the QA plan/duration
for each testing Phase80%
Design
Execution
Sprint Duration
Pla
nn
ing
Week1 Week2 Week3
Test
Co
mp
leti
on
%
100%
Closure
4Gowrishankar Sundararajan Nordic Testing Days 2016
Drawbacks/Key Issues faced in Traditional Approach
Comparison
Issues
Satisfaction
• On day 9, when testing team starts
with Smoke testing there is always
greater chance to identify
‘Blocker’ defects.
• Due to blocker defects, a day or
two will be lost which makes
shorter duration for testing team
to complete test execution in 5
days.
• Sometimes, even regression for
the current sprint release
cannot be completed.
• In Many Cases, regression of
prior releases are ignored which
will have higher impact in UAT.
• Finally, test suite preparation of
regression & End-End testing
involves considerable effort,
cost.
• Due to time constraint, few items needs to be removed
from the current sprint.
• Ultimately, Product team is unhappy with the
items not getting delivered which was promised.
5Gowrishankar Sundararajan Nordic Testing Days 2016
The Solution – New Best in Class Approach
QA Plan day wise
Day 2 – 4 : Testers develop the core automated script and test data for Critical path to satisfy Smoke/Sanity Criteria and
hands it to Dev team to run the script before sending the new build to QA
Day 5 – 8 : Testers add scripts to cover all possible test scenarios
Day 9 – 11 : Functional Test Execution using Automation Scripts
Day 12 – 13: In-sprint Defect Resting & Regression Test Execution
Day 14 – 15: Regression for Prior sprints
6
Day 1 Day 2 - 4Day
9 - 11 Day 14-15 Day1
Coding Defect Fix ClosurePlanning
Test Design Test Execution ClosurePlanning
Day 5 - 8
Core ScriptAll possible
Combination
Day
12 - 13
Automated Script Exec
In-sprint Regression
Prior sprint Regression
Sanity Test
Gowrishankar Sundararajan Nordic Testing Days 2016
New Best in Class Approach vs. Traditional Approach – Comparison Chart
Week1 Week2 Week3
Test
Co
mp
leti
on
%
80%
Pla
nn
ing Sprint
DesignSprint
ExecutionClosure
Sprint Duration
Week1 Week2 Week3
Test
Co
mp
leti
on
%
Pla
nn
ing Sprint
DesignSprint
Execution
Closure
Core Regression
Traditional Approach New Approach
Acceptance test
7Gowrishankar Sundararajan Nordic Testing Days 2016
Sprint Duration
8
The Solution – New Best in Class Approach
• Early Automation – Start automation early in SDLC from smoke/Sanity testing rather
than waiting for functional testing
• Testers will no longer perform Smoke/Sanity testing
• Testing team develops automated smoke/sanity test Scripts
• Developers executes automated smoke test on new build before handover to QA
Automation
Smoke/
Sanity Test
• Functional testing is no longer manual
• Maximized Automation
• Scripts reuse for Regression/End-End testing
Functional
Required Change in Approach
Gowrishankar Sundararajan Nordic Testing Days 2016
Benefits with New Approach
9
•
Defect Leakage
Cost Saving
Customer Satisfaction
Coverage
• Cost saving due to
automated smoke,
functional testing
• Automation ROI for
regression and end to
end testing is much
faster
• Zero Showstopper
Leakage from Dev
to SIT to UAT
• Reduced Major
defects leakage
from SIT to UAT
• Increased coverage
by ensuing room for
in-sprint regression
• Ensure coverage for
core regression
• Deliver what is
promised for each
• More confidence on
delivered product
• Improved overall
customer satisfaction
Key
Benefits
Gowrishankar Sundararajan Nordic Testing Days 2016
Case Study – Testing Support for a Leading Canadian Money Transfer Payment Network Provider
Client Profile and Background
Canada's largest money transfer Payment network and Gateway provider.
The client does the services similar to VISA, PAYPAL etc. and carries out 98% of money
transfer across people belonging to same/different FI’s with in Canada.
Key Features
Sending Money across FI’s/Credit Unions
Requesting Money across FI’s/Credit Unions
Project Information
No of resources
7 (1 On : 6 Off)
Product Usage
Money Transfer across FI’s
Technologies
Java, Xml, Batches, Web
Automation Tool
SOAP SONAR
Duration
Jan 2013 – Dec 2015
Tracking / Listing the transactions
Consolidated Reports
10Gowrishankar Sundararajan Nordic Testing Days 2016
Case Study – Testing Support for a Leading Canadian Money Transfer Payment Network Provider
11Gowrishankar Sundararajan Nordic Testing Days 2016
Scope of Work
• Current feature is built on Java Technology (Struts/JDBC) -> System heavy and affecting its performance.
• Rewrite of existing application + new features on Spring/Hibernate feature.
• No change in the XML Structure that is being used across FI’s for Sending Money.
Regression Testing of Old Features Functional Testing of New Features Integration Testing of Both Features
Project Intro
• Web banking customers of participating FI -> electronic payments to anyone else -> web banking
customer of a participating FI.
• Current functionality one-time, immediate money transfer.
• Future Release includes Requesting Money, Auto-deposit etc.
• The system uses an application service provider model (XML’s) with an interface to the FI’s banking site
Case Study – Testing Support for a Leading Canadian Money Transfer Payment Network Provider
Types of Testing
Smoke/Acceptance Testing
System/Functional Testing
System Integration Testing
(End to End)
Regression Testing
Test Automation
User Acceptance Testing
Beta Testing
Performance Testing
12Gowrishankar Sundararajan Nordic Testing Days 2016
Customer
Banks /
Credit
Union
Gateway
Banks /
Credit
Union
Customer
Send Money Receive Money
Send
Request
Receive Request
Receive
Money
Fulfill Request
Customer
Info update
Flow Diagram
Work Load Split-Up
13
MODULES USAGE SPLIT % FUNCTIONAL TESTING EFFORT SPLIT %
Though our application uses web services only by 50%, actual functional testing effort to test other modules
in turn requires to execute web services again. Overall Functional testing effort alone was more than 70%
with web services module
Gowrishankar Sundararajan Nordic Testing Days 2016
Case Study – Key KPI’s to Prove
ProductivityDEFECT LEAKAGE TO UAT Productivity ProductivityDEFECT LEAKAGE BY SPRINT
On an average, we had 4 – 5 ‘Blocker’ and 24
‘Major’ Defects leakage to UAT from SIT before
implementing new approach. Later, we only had
5 ‘Major’ and Max of 1 ‘Blocker’ defect.
From the above graph, it is clear that there is drastic
reduction in the number of both ‘Blocker’ and ‘Major’
defects from Sprint 9 (after implementing new approach).
**First 3 Sprints count is ignored to understand application and taking
into consideration as ‘Learning Curve’
14Gowrishankar Sundararajan Nordic Testing Days 2016
Case Study – Key KPI’s to Prove
ProductivityProductivityREGRESSION DEFECTS LEAKAGE TO UAT ProductivityREGRESSION DEFECTS IDENTIFIED IN SIT
From above graph, it is clear that there is Zero
defect leakage (both Blocker & Major) to UAT from
Sprint 11. The reason we had it in Sprint 9 & 10 is
that regression coverage was done gradually.
From the above graph, it is clear that
a) Regression Defects are caught in SIT phase.
b) Considerable reduction in number of defects
leaked from Dev to SIT.
15Gowrishankar Sundararajan Nordic Testing Days 2016
Case Study – Key KPI’s to Prove
ProductivityProductivityProductivityTEST EXECUTION COVERAGE GAPS IN EXISTING SYSTEMProductivityQA DAYS LOST
QA days lost due to
showstopper issuesAfter implementing new approach,
all the possible negative
flows/paths is covered. This helped
us in identifying few gaps in
existing system, which helped
business to fix this issues before
rolling out new features. This will in
turn save number of tickets opened
to support team and their effort in
future.
Approach
Type
Days
Lost
Traditional 4 PD’s
New 0 PD’s
Exit criteria of any testing team is to ensure 100% coverage is
covered. But due to ‘Blocker’ defects, we always try to cover High
and Medium Priority test cases. Hence, even some Low priority test
cases will result in ‘Major’ UAT defect.
16Gowrishankar Sundararajan Nordic Testing Days 2016
How easy to get this approach implemented in other Testing Model
No Hassle
Most of the interactions across
applications happens thru web
services, interfaces/ESB. And in
recent days, after introduction of
Component Integration
testing this approach is easier
to implement and get the
benefits out of it. Though we
implemented specifically for
XML’s, same approach can be
used for web and other
technologies.
17Gowrishankar Sundararajan Nordic Testing Days 2016
Questions?
18