Upload
dr-ganesh-iyer
View
508
Download
0
Embed Size (px)
Citation preview
Functional &
Non-functional Test
AutomationDr Ganesh Neelakanta Iyer
QA Architect
GANESHNIYER
http://ganeshniyer.com
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.2
About Me
GANESHNIYER
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.3
Quick Poll
Do you have some test automation in place for your
testing?
Do you have required functional test automation in place?
Do you have test automation for functional and non-functional
(performance, load, scalability, security, …) pieces of testing?
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.7
Agenda
Goal of successful automation
Types of test automation frameworks
Functional Test automation
Non-Functional test automation – Major types
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.8
Goals of Successful Automation
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.9
The Goal: Successful Automation
Suggestions:
• Automate the ‘Right’ Test Cases
• Test Early / Test Often
• Define Quality Test Data
• Design Automated Tests that can resist change
• Selecting the right Automation Tool
Graphic from qatestlab.com
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.10
Automate the ‘Right’ Test Cases
• Repetitively Run Tests (aka Smoke
or Acceptance tests)
• Data intensive tests
• Frequently Used Functionality
• High Risk, Business-critical
functionality
• Tests that are difficult to run
manually
• Tests that are run on differing
hardware/software platforms and
configurations
• Tests that take a lot of effort or
time
Graphic from qatestlab.com
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.11
Test Early / Test Often
Start as early in the process as possible
– Could start with Automated Unit testing
– Gradually build your Automation Suite
The more you test, the more bugs you find.
The earlier bugs are found the cheaper they are to fix!
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.12
Define Quality Test Data
With good data…
– Writing and Maintaining Automated tests is easier.
– With Data driven tests you can run more iterations and find more bugs!
– Can extend existing Automated tests as new features are developed – without having to edit the actual automated test itself.
– Stay proactive in updating/defining data.
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.13
Automation Approach
Keep Automated tests small and focused
– Easy to debug
– Easy to manage
– Can be reused and shared
These can then be grouped into a larger Automated test suite organized by functional area.
With the right framework, users can create large/complex Automated tests that are also easy to maintain!
Graphic from checkpointech.com
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.14
Design Automated Tests Resistant to Change
If possible, avoid using screen coordinates.
Componentize your test scripts
Ensure that your development team uses unique names for each control and
object!
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.15
How long do Automated Tests Last?
At some point, the application will change and the script will break.
• If you are using GUI Capture/Playback a major revision to the UI may break your tests.
If the script breaks before the automation effort was repaid, it is better off as a manual
test.
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.16
Selecting the ‘‘Right” Automation Tool
Can be used by business users
No programming skills required
Simple
Automate 60-70% of core
business processes in a few days
Rapid return on investment
Fast
When underlying applications change,
automation continues to work
Simple maintenance of your
business process portfolio
Resilient
True end-to-end business process validation
covering all major technologies and platforms
Complete
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.17
Return On Investment (ROI)
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.18
Classic ROI Calculation
ROI = BENEFIT/COST
Automation Cost = Price Of HW + Price of SW + Development Cost + Maintenance Cost + Execution Cost
Manual Testing Cost = Development Cost + Maintenance Cost + Execution Cost
ROI = (Manual Testing Cost - Automation Cost)/Automation Cost
Looks right, isn’t it?
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.19
Problems with Classic ROI Calculation
You can’t compare Automated Testing and Manual Testing. There are not the same
You can’t compare cost of multiple execution of automated tests vs. manual tests. You would never dream of executing that many test cases manually.
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.20
Automation Real ROI
ROI value IS NOT the value of Automation vs. Cost of executing
these tests manually
Automation ROI value IS the benefit of this type of testing, and it
can be:
Reducing Time to Market
Increased Test Efficiency (Productivity)
Increased Test Effectiveness
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.21
Types of test automation frameworks
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.22
Types of Test Automation Frameworks
Modular Data-Driven
Keyword–Driven Model-Based
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.23
Modular framework
The Modular framework is the natural progression from Record-and-
Playback
The modular framework seeks to minimize this repetition of code by
grouping similar actions into “modules” (e.g.: login)
Test Data is in a script
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.24
Data-Driven/Keyword Driven
These frameworks are similar in that the data is separated from the test
script
The script is just a "driver" or delivery mechanism for the data.
The difference:
• In keyword-driven testing, the navigation data and test data are contained in the data
source
• In data-driven testing, only test data is contained in the data source.
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.25
Model-Based testing
Model-based testing is software testing in which test cases are derived
in whole or in part from a model that describes some (usually functional)
aspects of the system.
Model-based testing for complex software systems is still an evolving
field.
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.26
Functional test automation
Typically what we speak about is all about functional
automation
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.28
Functional Testing
Web
Telerik Test
Studio
Sahi
Selenium
Sikuli
Web Mobile
Desktop Backend
Mobile
Mobile Test
Studio
Telerik
DeviceCloud
Appium
Desktop
QTP
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.29
Testing Mobile Apps in CloudTelerik DeviceCloud
>300
29
27
Devices
Manufacturers
OS versions
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.30
Compatibility Testing; Web, Device, OS, DB,….
Automation:
Telerik Test
Studio
Telerik
DeviceCloud
Sahi
Selenium
Compatibility
Testing
Accessibility
testing
Globalization and
localization testing
Compatibility
under different
situations
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.31
Non-Functional test automation –
Major types
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.32
Major Non-functional test dimensions in a Nut-shell
Non-functional
Security
API Connectivity
PerformanceResilience
Integration
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.33
Security
Security
Traversal
vulnerability
User access/ roles
Identity federation
management
Multi-tenant
penetration
User Access /
Roles
Web UI Security
Multi-Tenant
DBVulnerabilities
Automation:
Veracode
Websecurify
ZAP tool
Scripts
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.34
Security at various stages
PlanningArchitecture and Design
Implementation Level
Whiteboxtesting
Blackboxtesting
Reviews
Code Doc Log Regional Security Compliance Ruleset System Config Network Sniffing File Integrity Checking
PO/
PM
QA
Dev Architect
/ Lead
Dev QA QA
Dev QA QADev Architect
/ LeadDev Architect
/ Lead
QA
Different components of each aspect- what needs to be tested/assured
Security protection methods
Authentication
Access Control
Cryptography
Firewall, Anti-virus etc
Secured Infrastructure
Web Security
URL Manipulation
SQL Injection, Other injection
attacks
Session fixation, XSS
API Security Testing
Cloud Security
Multi-tenant penetration (for
ISVs)
Data protection in Cloud
Cloud Identity Management
Cloud Infrastructure security
Database Security
Reliability and Integrity
DB Disclosure
Security in data mining
Data security
Vulnerabilities and attacks
Network, Memory-related, Metadata
injection
Traversal vulnerability
DoS/DDoS attacks
Encryption in networks/browsers
Programmed Injection
Privacy, Legal issues and
Ethics
Web privacy
Law and Compliance
Email security
Ethical issues
Emerging Technologies and Security
Aspects
Mobile App
Mobile Web
IoT sensors
Information Security
Security Issues within the code
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.36
How we plan?– Four types of activitiesHigh-level test approach and test execution frequency
QA Reviews
• Doc Review
• Code Review
Static Analysis (Whitebox Testing)
•Analyze source for security vulnerabilities typically at the programming and/or testing software life cycle (SLC) phases
Service Level Dynamic Testing
• Test the run time services exposed by applications for various security testing
Business Level Dynamic Testing
•Application Runtime analysis, E2E business use cases
To happen regularly on a scheduled manner
For short releases Once in the release
For long releases, 2-3 times during the release cycle
Sanity Suite to run on Continuous Integration
Automate the cases and run once in a sprint
Automate and run once in a sprint
Use tools for scanning 1-3 times based on release cycle
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.38
API Invocation
API
Invocation
Connectivity and
invocation
API load
API security
Multi-tenancy
Automation:
SoapUI
Node.js
PySys
In-house
Frameworks
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.39
High Availability (HA) and Performance Testing
Automation:
Telerik Test
Studio
NeoLoad
JMeter
HA &
Performance
Time to deploy
Multi-tenancy
Latency
Connectivity and
reliability with
3rd parties
Reliability and
availability
HA/Performance
TestingAvailability
24 x 7 –
Endurance
Reliability
Latency
Stress
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.40
24 x 7 Testing
VMs / Build Integration Machines
ANT ScriptTests Reports using
Kendo UI controls
Decides
parallelism
HTTP POST reqBatch
Script
Decides
frequency
Logs
SUT
Uses any test environments and leverage existing tests
Batch Script
Decides the frequency
ANT Script
We configure #Threads,
#Tests to run in parallel
Reports
Shown using Keno UI controls / widgets
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.41
Resilience and Failover
Ability of a server, network, storage
system, or an entire data center, to
continue operating even when there has
been an equipment failure, power
outage or other disruption
Software error
Hardware error
Human error
Natural Disaster
Picture Credit: http://imperfectspirituality.com/wp-content/uploads/2012/08/Moving-
mountains-with-resilience.jpg
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.42
Resilience and Failover
Resilience
Making one
instance down
Making multiple
instances down
Disaster Recovery
Application
Interface
(UI / API)
IaaS
Logs
ELB
HA Instance 2Stubs for
3rd Parties
HA Instance 1
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.43
Integration
Integration
P2P Integration
System
Integration
End-to-End
Integration
SSO
Customer
Scenarios
Picture Credit: http://goo.gl/wgNKi3
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.44
A Typical Integration Scenario in Progress
OpenEdge DB
OData
OData REST
Service
Telerik
Mobile
QA Dashboard App
DMZ REST
Wrapper
REST Service
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.45
Typical End-to-End Scenario
RegistrationLand on product
dashboard
Click on subscription
details
Choose a plan
Checkout process
Generate invoice
Product usage
Log-out
LDAP Community
Export Compliance,
Marketing
Marketing,
Billing,
Support
Billing,
Invoicing
LDAP
Billing
Metering
Billing
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.46
Integration Test Challenges and Approaches
Multiple systems behave differently
No automated deletion of test data in some systems
Unpredictable delays in updating various systems
Different types of environments for testing
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.47
Integration Test Automation With Cloud
Telerik Test Studio as the UI framework, C# as the Runtime framework
Output
Repository
Test
Manager
Test
Logger
Run Time Test
Framework
API Calls
UI
Framework
Stubs for
3rd Parties
Test Repository Test suite
Libraries (APIs)
Product
1
Product
2
Product
3
There are much more…
Internationalization Localization
Hot-Upgrade Multi-tenancy
Backward Compatibility
Limits
Migration Installation
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.51
Major Test Dimensions in a Nut Shell
Security PerformanceResilience &
FailoverCompatibility
API
ConnectivityIntegration
Traversal
vulnerabilityTime to deploy
Making one
instance downAccessibility
Connectivity and
invocationP2P Integration
User access/ roles Multi-tenancyMaking multiple
instances down
Globalization and
localizationAPI load
System
Integration
Identity federation
managementLatency Disaster Recovery
Compatibility
under different
situations
API securityEnd-to-End
Integration
Multi-tenant
penetration
Connectivity and
reliability with
3rd parties
Multi-tenancy SSO
Reliability and
availability
Customer
Scenarios
© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.52
Automation Frameworks
Telerik Test Studio, Telerik DeviceCloud
Sahi / Selenium scripts
Telerik Test Studio for Load / Performance Testing
Neoload, JMeter
SOAP UI / Node.js / PySys
Telerik Test Studio (with C#)
TestNG + Java + Sahi combined approachIntegration
Functional
(Web/Mobile UI)
Performance
API Invocation
Veracode, Websecurify, ZAP ToolSecurity