An Automation Framework That Really Works
ByBasivi Reddy Junna
QA Practice Manager and Sr. Automation Architect
Thursday, May 3,2012
An Automation Framework That Really Works - Basivi Reddy Junna
2
Almost anything that you would
ever want to do in
programming can be done
with Functions
An Automation Framework That Really Works - Basivi Reddy Junna
3
Traditional approach to
automation
Why automation
frameworks at all?
Disadvantages of
automation frameworks
Available frameworks of
automation
Data Driven Framework
Keyword Driven framework
Modular Framework
Functional Decomposition Framework• Conceptual overview• Components• Workflow• Driver
Hybrid Framework
Q&A
Introduction
4
Create Tests
• Record the test or Program it manually
Enhance Tests
• Synchronization• Parameterization• Checkpoints
Execute Tests
• One by one• Batch mode
Analyze and Report
• Analyze Results• Report and Manage Defects
Traditional Approach to Automation
An Automation Framework That Really Works - Basivi Reddy Junna
Create
Enhanc
e
Execute
Analyze
and Repor
t
5
Traditional Approach to Automation Continued…
An Automation Framework That Really Works - Basivi Reddy Junna
Pros
• Best approach for automating few tests• Quick and easy to implement
Cons• In the absence of any common pattern or structure, reusability of automation code is very limited
• Standardization is a challenge• Difficult to maintain as tests are standalone and each test is a
separate point of maintenance• As the number of tests increase, coverage analysis becomes
difficult and test suite may quickly go out of control• Difficult to consolidate and analyze results
6
Standard wish list expected to be fulfilled by automation
An Automation Framework That Really Works - Basivi Reddy Junna
Test Development
Promote reusability of assets
Provide rapid test building capabilities
Even manual testers should be able to create automated tests
Test Data Management
Test data management should be a breeze.
Data coverage analysis should be real simple
Use the data generated by one test in others
Handle non-reusable sets of test data without manual intervention
Test Execution
Execution setup should be centralized.
Even non technical people should be able to setup and run automated tests.
Run tests in multiple environments concurrently
Run a batch of tests with just click of a button
Schedule tests to run over night
Recover from any errors and continue with other tests in the batch
Re-run tests that did not run completely in previous run. Re-run them from the point where they left off.
Re-run tests that have one or more failed checkpoints or assertions in previous run
Test Results and Reporting
Notify designated people when tests fail to execute
Create summarized and detailed reports with results of execution.
Keep history of results.
Compare results of two or more test runs
Maintenance
Fewer points of maintenance
Quickly identify maintenance points when application changes
Maintenance should be super simple
7
Why Use Automation Frameworks At All?
An Automation Framework That Really Works - Basivi Reddy Junna
RobustDo not break easily. Recover from errors for un-interrupted executionCentralized Execution Controland unattended execution.
ScalableCan easily be up-scaled in terms of number and size of tests and yet retain control
FlexibleWith little modifications assets can be leveraged to create new tests and cover more scenarios
MaintainableEasy to maintain - Few maintenance points due code re-use
ReusableAssets are reusable and provide rapid
test building capabilities
8
Difficult to implement
Availability of skilled resources is an issue
Require more up front investment than traditional approach
Over engineering trap
Wrong selection of framework
Get obsessed with framework
Disadvantages of Automation Frameworks
An Automation Framework That Really Works - Basivi Reddy Junna
9
Data Driven Framework
Key word Driven Framework
Modular Framework
Functional Decomposition Framework
Hybrid Framework
Available Frameworks of Automation
An Automation Framework That Really Works - Basivi Reddy Junna
10
Data Driven Tests:
• Parameterized tests are commonly referred to as data driven tests• Parameterization: Fixed input values for test steps are replaced with variables and variables are supplied
with values during runtime• Same steps and logic are repeated across multiple test data sets
Data Driven Framework:
• Will have all features of data driven tests• Logic is controlled by test data. In other words script’s control flow and other behavior is dictated by test data• Suitable for large data intensive applications with limited variations in functional flows and simple to moderately complex business rules• Usually made up of few but very large test scripts whose behavior is controlled by data populated in test data repository• Test maintenance is mainly taken care by updating test data
Data Driven Framework
An Automation Framework That Really Works - Basivi Reddy Junna
11
Data Driven Framework - Example
An Automation Framework That Really Works - Basivi Reddy Junna
Test
Id TestProd_Id TestAddress_Id TestPmt_Id
1 2 1 2
2 3 2 1
Id Product_Id Action
1 10 Search
2 3 Add
2 4 Remove
Tests
Test_ProductsId Product_Name
1 Kindle Fire
2 iPod Charger
3 Google Nexus
4 Blackberry
ProductsId Address_Id Action
1 3 Add
2 1 Edit
Test_Addresses
Id Address City
1 … Chicago
2 … Chicago
3 70 York… Lombard
Addresses
InputTest Id
12
Pros
• Limited numbers of tests to maintain• Script maintenance is largely handled through test data updates
Cons
• Complex to implement: Test data repository may become as complex as the application data model itself• Not suitable for applications with heavy variations in functionalities and flows• Tests scripts are too big to fail
Data Driven Framework
An Automation Framework That Really Works - Basivi Reddy Junna
13
Testing activities can be divided into basic actions called keywords – Click, Select, Enter etc
Keyword(s) => Function(s)
Test scripts are written using key words
Keywords (functions) are invoked by providing as inputs - data and application objects
Automated scripts are automatically generated behind the scene, while you are creating manual scripts using keywords
Keyword Driven Framework
An Automation Framework That Really Works - Basivi Reddy Junna
14
Pros:
• You create manual tests and have automated scripts generated behind the scene automatically• Even manual testers can create automated tests without any knowledge or experience of automation.• Two birds with one shot - Produces highly structured manual tests so you can test manually in case if
automation is NOT working for any reason
Cons:
• Debugging is very tedious• Heavy upfront investment• Suitable for small and medium applications only• Difficult to handle dynamic objects• Grouping steps and controlling iterations is extremely difficult• Complex test flows cannot be specified using keywords
Keyword Driven Framework
An Automation Framework That Really Works - Basivi Reddy Junna
15
Divide the applications into logically identifiable sections called modules • Re-usable• Data Driven – Modules behave according to data• Perform multiple functions and validations• Handle errors• Report results
Develop automation scripts for the modules
Modules characteristics:
Actual tests are created by calling the reusable modules in a required sequence
Can be treated as an additional layer of abstraction on the top of functional decompositions framework
Modular Framework
An Automation Framework That Really Works - Basivi Reddy Junna
16
Pros:
• Large number of tests are created by relatively less number of modules. Hence, fewer points of maintenance
• Provides much higher level of abstraction than Keyword driven or functional decomposition frameworks making it easy for users to understand the test flows.
Cons:
• Best described as half data driven and half functional decomposition framework. It has disadvantages of both the frameworks but not the advantages of Functional Decomposition Framework
• Restricted maneuverability - Not suitable for application with wide variety of functionalities, twists and turns(flows). Integration of modules into actual tests and still conforming to navigational rules of the application is difficult to achieve
• Each module performs multiple functions. Selective execution of functions is difficult to achieve• Combination of higher level of abstraction and underlying data driven framework make it difficult to understand the test data being passed to and consumed by tests
Modular Framework
An Automation Framework That Really Works - Basivi Reddy Junna
17
Divide the application into logically identifiable functions
Create functions using scripting language of the tool
Make the functions data driven using parameters
Create actual tests using the functions
Suitable for medium to highly complex applications
Can handle wide variety of functionalities
Capable of being data driven
Functional Decomposition Framework
An Automation Framework That Really Works - Basivi Reddy Junna
18
Functional Decomposition Framework: Example Test 1
An Automation Framework That Really Works - Basivi Reddy Junna
Launch Application
Login
Search Products
Add To Cart
Edit Cart
Checkout
Shipping Address
Payment Method
Place Order
Logout
Test 1Launch Application
Login
Search Products
Add To Cart
Edit Cart
Checkout
Shipping Address
Payment Method
Place Order
Logout
Launch Application
Login
Search Products
Add To Cart
Edit Cart
Checkout
Select Shipping Address
Select Payment Method
Place Order
Logout
URL
UID and PWD
Criteria & Product Info
19
Functional Decomposition Framework: Example Test 2
An Automation Framework That Really Works - Basivi Reddy Junna
Launch Application
Login
Search Products
Add To Cart
Edit Cart
Checkout
Shipping Address
Payment Method
Place Order
Logout
Test 2Launch Application
Login
Search Products
Add To Cart
Edit Cart
Checkout
Shipping Address
Payment Method
Place Order
Logout
Launch Application
Login
Search Products
Add To CartEdit Cart
Checkout
Select Shipping Address
Select Payment Method
Place Order
Logout
URL
UID and PWD
Criteria & Product Info
Quantity
Capture Order Number
20
Test Data Repository
Business Functions
Common Functions
Generic Functions
Framework Functions
Functions for
Web Applications
FunctionsJAVA
Applications
Test1Function 1Function 2
…Function n
Test2Function 1Function 5
…Function n
Test3Function 6
Function 10…
Function n
DriverTest1Test2Test3
…Test n
Functional Decomposition Framework – Conceptual Overview
Functions for
Mainframe Applications
Etc…
21
Business Functions: These are re-usable functions that perform specific tasks by reading data from test data repository. All functions are data driven. These functions are specific to the application under test. Using common functions (explained later) will significantly accelerate the development of business functions• Validate the inputs• Verify execution status of previous functions in the test• Verify start status• Implement synchronization and perform validations• Recover from errors• Report results
Characteristics:
Components of the Functional Decomposition Framework
An Automation Framework That Really Works - Basivi Reddy Junna
22
Common Functions: These functions are independent of any business domain/application and are commonly used. There are three sub categories of common functions: 1. Generic functions (string functions, date functions etc) 2. Framework functions (Reporting, configuration, error handling functions etc) 3. Environment specific functions such web functions, Power builder functions etc. These functions are mostly called by business functions
Automated Tests: Automated Tests are created by simply calling a series of re-usable business functions in required sequence using parameters to read appropriate data from test data repository.
Test Data Repository: A place to store , organize and manipulate test data. It is usually made up of any one or combination of flat files, excel workbooks, relational databases. This is also where results of the automated test execution will be stored. The idea is to have a centralized location for controlling test data and for reporting.
Driver: It is a batch test which calls other automated tests. It will have error handling/recovery capability where if one test in the batch fails to run due to any error, that test will be ignored and execution control will move to next test in the batch.
Components of the Functional Decomposition Framework
An Automation Framework That Really Works - Basivi Reddy Junna
23
Appendix A - Structure of Business Functions
An Automation Framework That Really Works - Basivi Reddy Junna
<Function Name>(Arg1,Arg2,...)‘Verify if the test failed in previous functionIf STATUS <>"PASS" Then
Exit FunctionEnd If'Validate parameters passed to the function
'for example you verify if they are null. or in some cases verify if they are numbers or not etc'If IsNull(Arg1) or IsNull(Arg2)... Then
'Perform appropriate action based on whether any argument is mandatory for the function to continue'Report about missing or incorrect argument and Exit Action
'End IfValidate Start Status
'Describe in what state the application should be (screen name and short description) when this function starts'Create Expected Result (ER) strings for all conditions this function seeks to verify
'Expected Result1=“…"'Expected Result2=“…“
write code to verify each Expected Results‘Wherever necessary implement synchronization, parameterization, Error handling, Insert output values and
checkpoints'Capture Actual Result for each Expected Result ; Compare Expected and Actual Result'Report Pass if they match or Fail if they don't along with expected and actual results‘While performing above steps recover from errors if any and also call Common functions as needed
<End Function>
24
Example Setup
An Automation Framework That Really Works - Basivi Reddy Junna
Setup file - Setup IDs sheet
Setup file - Setup Details sheet
25
Example Setup
An Automation Framework That Really Works - Basivi Reddy Junna
Setup file – Run Setup IDs sheet Setup file – Tests Sheet
Setup file – Run Tests Sheet
26
Sample functions for a fictitious E-Commerce Application
An Automation Framework That Really Works - Basivi Reddy Junna
27
Functional Decomposition Framework Sample Test and Results
An Automation Framework That Really Works - Basivi Reddy Junna
28
Workflow of the Functional Decomposition Framework
An Automation Framework That Really Works - Basivi Reddy Junna
29
Functional Decomposition Framework - Driver
30
Functional Decomposition Framework – Run Test
31
Workflow of the Functional Decomposition Framework – Test Data Repository Considerations
An Automation Framework That Really Works - Basivi Reddy Junna
Systems/ Applications
With UI
No UI (Headless)
32
Workflow of the Functional Decomposition Framework – Test Data Repository Considerations
An Automation Framework That Really Works - Basivi Reddy Junna
UI Based Applications
Free form text type
Text box
Combo box
Text area
Single Select Type
Single select list view
Single select tree view
Radio button group
Dropdown list
Multi select type
Multi select list view
Multi select tree view
Checkbox
33
Workflow of the Functional Decomposition Framework – Test Data Repository Considerations
An Automation Framework That Really Works - Basivi Reddy Junna
Non UI Based /Faceless Applications
Single values
Set of values
34
Workflow of the Functional Decomposition Framework – Test Data Repository Design Recommendations
An Automation Framework That Really Works - Basivi Reddy Junna
Data Tables Static Data Tables
Contain static data . Examples: list of states, product categories like furniture, electronics etc
1. For each set of static data create a separate data table and assign unique Id for each row
2. Use these IDs delimited by ; inside dynamic tables if the function needs to handle multi select type fields or a set of
values as input3. Use the static data values directly inside the dynamic tables
if the need is to handle single select type fields or a single value as input
Dynamic Data TablesMade up of:
1. Free form text, numbers and dates 2. Different permutations and combinations of data from
static tables
35
Test Data Repository Design - Sample
An Automation Framework That Really Works - Basivi Reddy Junna
Table to store values captured from Application
36
Pros:
• More flexible than other frameworks• Better maneuverability means test flows involving complex twists and turns can be created easily• Better control over test data consumed in your tests• The building blocks (functions) are not too granular, not too high level and present fewer points maintenance• One of the best frameworks suitable for automation in agile environments
Cons:
• Becomes complex when the test gets big - as the test size increases, so will the number of function calls and failure points, debug points
• When a test becomes really big, test flow is not as clearly evident as in the of modular framework
Functional Decomposition Framework
An Automation Framework That Really Works Basivi Reddy Junna
37
Hybrid Framework(s)
An Automation Framework That Really Works - Basivi Reddy Junna
Keyword + Functional Decomposition + Modular Framework
Keyword + Functional Decomposition Framework
Functional Decomposition + Modular Framework
Keyword + Modular Framework
Do not copy or distribute. Proprietary and Confidential to Scalar USA Inc.
38
Q&A
???
An Automation Framework That Really Works - Basivi Reddy Junna
Do not copy or distribute. Proprietary and Confidential to Scalar USA Inc.
39
Thank you!!!
An Automation Framework That Really Works - Basivi Reddy Junna