53
Functional & Non-functional Test Automation Dr Ganesh Neelakanta Iyer QA Architect GANESHNIYER [email protected] http://ganeshniyer.com

Functional and Non-functional Test automation

Embed Size (px)

Citation preview

Page 1: Functional and Non-functional Test automation

Functional &

Non-functional Test

AutomationDr Ganesh Neelakanta Iyer

QA Architect

GANESHNIYER

[email protected]

http://ganeshniyer.com

Page 2: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.2

About Me

GANESHNIYER

Page 3: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.3

Quick Poll

Page 4: Functional and Non-functional Test automation

Do you have some test automation in place for your

testing?

Page 5: Functional and Non-functional Test automation

Do you have required functional test automation in place?

Page 6: Functional and Non-functional Test automation

Do you have test automation for functional and non-functional

(performance, load, scalability, security, …) pieces of testing?

Page 7: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.7

Agenda

Goal of successful automation

Types of test automation frameworks

Functional Test automation

Non-Functional test automation – Major types

Page 8: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.8

Goals of Successful Automation

Page 9: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.9

The Goal: Successful Automation

Suggestions:

• Automate the ‘Right’ Test Cases

• Test Early / Test Often

• Define Quality Test Data

• Design Automated Tests that can resist change

• Selecting the right Automation Tool

Graphic from qatestlab.com

Page 10: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.10

Automate the ‘Right’ Test Cases

• Repetitively Run Tests (aka Smoke

or Acceptance tests)

• Data intensive tests

• Frequently Used Functionality

• High Risk, Business-critical

functionality

• Tests that are difficult to run

manually

• Tests that are run on differing

hardware/software platforms and

configurations

• Tests that take a lot of effort or

time

Graphic from qatestlab.com

Page 11: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.11

Test Early / Test Often

Start as early in the process as possible

– Could start with Automated Unit testing

– Gradually build your Automation Suite

The more you test, the more bugs you find.

The earlier bugs are found the cheaper they are to fix!

Page 12: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.12

Define Quality Test Data

With good data…

– Writing and Maintaining Automated tests is easier.

– With Data driven tests you can run more iterations and find more bugs!

– Can extend existing Automated tests as new features are developed – without having to edit the actual automated test itself.

– Stay proactive in updating/defining data.

Page 13: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.13

Automation Approach

Keep Automated tests small and focused

– Easy to debug

– Easy to manage

– Can be reused and shared

These can then be grouped into a larger Automated test suite organized by functional area.

With the right framework, users can create large/complex Automated tests that are also easy to maintain!

Graphic from checkpointech.com

Page 14: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.14

Design Automated Tests Resistant to Change

If possible, avoid using screen coordinates.

Componentize your test scripts

Ensure that your development team uses unique names for each control and

object!

Page 15: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.15

How long do Automated Tests Last?

At some point, the application will change and the script will break.

• If you are using GUI Capture/Playback a major revision to the UI may break your tests.

If the script breaks before the automation effort was repaid, it is better off as a manual

test.

Page 16: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.16

Selecting the ‘‘Right” Automation Tool

Can be used by business users

No programming skills required

Simple

Automate 60-70% of core

business processes in a few days

Rapid return on investment

Fast

When underlying applications change,

automation continues to work

Simple maintenance of your

business process portfolio

Resilient

True end-to-end business process validation

covering all major technologies and platforms

Complete

Page 17: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.17

Return On Investment (ROI)

Page 18: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.18

Classic ROI Calculation

ROI = BENEFIT/COST

Automation Cost = Price Of HW + Price of SW + Development Cost + Maintenance Cost + Execution Cost

Manual Testing Cost = Development Cost + Maintenance Cost + Execution Cost

ROI = (Manual Testing Cost - Automation Cost)/Automation Cost

Looks right, isn’t it?

Page 19: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.19

Problems with Classic ROI Calculation

You can’t compare Automated Testing and Manual Testing. There are not the same

You can’t compare cost of multiple execution of automated tests vs. manual tests. You would never dream of executing that many test cases manually.

Page 20: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.20

Automation Real ROI

ROI value IS NOT the value of Automation vs. Cost of executing

these tests manually

Automation ROI value IS the benefit of this type of testing, and it

can be:

Reducing Time to Market

Increased Test Efficiency (Productivity)

Increased Test Effectiveness

Page 21: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.21

Types of test automation frameworks

Page 22: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.22

Types of Test Automation Frameworks

Modular Data-Driven

Keyword–Driven Model-Based

Page 23: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.23

Modular framework

The Modular framework is the natural progression from Record-and-

Playback

The modular framework seeks to minimize this repetition of code by

grouping similar actions into “modules” (e.g.: login)

Test Data is in a script

Page 24: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.24

Data-Driven/Keyword Driven

These frameworks are similar in that the data is separated from the test

script

The script is just a "driver" or delivery mechanism for the data.

The difference:

• In keyword-driven testing, the navigation data and test data are contained in the data

source

• In data-driven testing, only test data is contained in the data source.

Page 25: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.25

Model-Based testing

Model-based testing is software testing in which test cases are derived

in whole or in part from a model that describes some (usually functional)

aspects of the system.

Model-based testing for complex software systems is still an evolving

field.

Page 26: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.26

Functional test automation

Page 27: Functional and Non-functional Test automation

Typically what we speak about is all about functional

automation

Page 28: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.28

Functional Testing

Web

Telerik Test

Studio

Sahi

Selenium

Sikuli

Web Mobile

Desktop Backend

Mobile

Mobile Test

Studio

Telerik

DeviceCloud

Appium

Desktop

QTP

Page 29: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.29

Testing Mobile Apps in CloudTelerik DeviceCloud

>300

29

27

Devices

Manufacturers

OS versions

Page 30: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.30

Compatibility Testing; Web, Device, OS, DB,….

Automation:

Telerik Test

Studio

Telerik

DeviceCloud

Sahi

Selenium

Compatibility

Testing

Accessibility

testing

Globalization and

localization testing

Compatibility

under different

situations

Page 31: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.31

Non-Functional test automation –

Major types

Page 32: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.32

Major Non-functional test dimensions in a Nut-shell

Non-functional

Security

API Connectivity

PerformanceResilience

Integration

Page 33: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.33

Security

Security

Traversal

vulnerability

User access/ roles

Identity federation

management

Multi-tenant

penetration

User Access /

Roles

Web UI Security

Multi-Tenant

DBVulnerabilities

Automation:

Veracode

Websecurify

ZAP tool

Scripts

Page 34: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.34

Security at various stages

PlanningArchitecture and Design

Implementation Level

Whiteboxtesting

Blackboxtesting

Reviews

Code Doc Log Regional Security Compliance Ruleset System Config Network Sniffing File Integrity Checking

PO/

PM

QA

Dev Architect

/ Lead

Dev QA QA

Dev QA QADev Architect

/ LeadDev Architect

/ Lead

QA

Page 35: Functional and Non-functional Test automation

Different components of each aspect- what needs to be tested/assured

Security protection methods

Authentication

Access Control

Cryptography

Firewall, Anti-virus etc

Secured Infrastructure

Web Security

URL Manipulation

SQL Injection, Other injection

attacks

Session fixation, XSS

API Security Testing

Cloud Security

Multi-tenant penetration (for

ISVs)

Data protection in Cloud

Cloud Identity Management

Cloud Infrastructure security

Database Security

Reliability and Integrity

DB Disclosure

Security in data mining

Data security

Vulnerabilities and attacks

Network, Memory-related, Metadata

injection

Traversal vulnerability

DoS/DDoS attacks

Encryption in networks/browsers

Programmed Injection

Privacy, Legal issues and

Ethics

Web privacy

Law and Compliance

Email security

Ethical issues

Emerging Technologies and Security

Aspects

Mobile App

Mobile Web

IoT sensors

Information Security

Security Issues within the code

Page 36: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.36

How we plan?– Four types of activitiesHigh-level test approach and test execution frequency

QA Reviews

• Doc Review

• Code Review

Static Analysis (Whitebox Testing)

•Analyze source for security vulnerabilities typically at the programming and/or testing software life cycle (SLC) phases

Service Level Dynamic Testing

• Test the run time services exposed by applications for various security testing

Business Level Dynamic Testing

•Application Runtime analysis, E2E business use cases

To happen regularly on a scheduled manner

For short releases Once in the release

For long releases, 2-3 times during the release cycle

Sanity Suite to run on Continuous Integration

Automate the cases and run once in a sprint

Automate and run once in a sprint

Use tools for scanning 1-3 times based on release cycle

Page 37: Functional and Non-functional Test automation
Page 38: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.38

API Invocation

API

Invocation

Connectivity and

invocation

API load

API security

Multi-tenancy

Automation:

SoapUI

Node.js

PySys

In-house

Frameworks

Page 39: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.39

High Availability (HA) and Performance Testing

Automation:

Telerik Test

Studio

NeoLoad

JMeter

HA &

Performance

Time to deploy

Multi-tenancy

Latency

Connectivity and

reliability with

3rd parties

Reliability and

availability

HA/Performance

TestingAvailability

24 x 7 –

Endurance

Reliability

Latency

Stress

Page 40: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.40

24 x 7 Testing

VMs / Build Integration Machines

ANT ScriptTests Reports using

Kendo UI controls

Decides

parallelism

HTTP POST reqBatch

Script

Decides

frequency

Logs

SUT

Uses any test environments and leverage existing tests

Batch Script

Decides the frequency

ANT Script

We configure #Threads,

#Tests to run in parallel

Reports

Shown using Keno UI controls / widgets

Page 41: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.41

Resilience and Failover

Ability of a server, network, storage

system, or an entire data center, to

continue operating even when there has

been an equipment failure, power

outage or other disruption

Software error

Hardware error

Human error

Natural Disaster

Picture Credit: http://imperfectspirituality.com/wp-content/uploads/2012/08/Moving-

mountains-with-resilience.jpg

Page 42: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.42

Resilience and Failover

Resilience

Making one

instance down

Making multiple

instances down

Disaster Recovery

Application

Interface

(UI / API)

IaaS

Logs

ELB

HA Instance 2Stubs for

3rd Parties

HA Instance 1

Page 43: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.43

Integration

Integration

P2P Integration

System

Integration

End-to-End

Integration

SSO

Customer

Scenarios

Picture Credit: http://goo.gl/wgNKi3

Page 44: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.44

A Typical Integration Scenario in Progress

OpenEdge DB

OData

OData REST

Service

Telerik

Mobile

QA Dashboard App

DMZ REST

Wrapper

REST Service

Page 45: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.45

Typical End-to-End Scenario

RegistrationLand on product

dashboard

Click on subscription

details

Choose a plan

Checkout process

Generate invoice

Product usage

Log-out

LDAP Community

Export Compliance,

Marketing

Marketing,

Billing,

Support

Billing,

Invoicing

LDAP

Billing

Metering

Billing

Page 46: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.46

Integration Test Challenges and Approaches

Multiple systems behave differently

No automated deletion of test data in some systems

Unpredictable delays in updating various systems

Different types of environments for testing

Page 47: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.47

Integration Test Automation With Cloud

Telerik Test Studio as the UI framework, C# as the Runtime framework

Output

Repository

Test

Manager

Test

Logger

Run Time Test

Framework

API Calls

UI

Framework

Stubs for

3rd Parties

Test Repository Test suite

Libraries (APIs)

Product

1

Product

2

Product

3

Page 48: Functional and Non-functional Test automation

There are much more…

Page 49: Functional and Non-functional Test automation

Internationalization Localization

Hot-Upgrade Multi-tenancy

Page 50: Functional and Non-functional Test automation

Backward Compatibility

Limits

Migration Installation

Page 51: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.51

Major Test Dimensions in a Nut Shell

Security PerformanceResilience &

FailoverCompatibility

API

ConnectivityIntegration

Traversal

vulnerabilityTime to deploy

Making one

instance downAccessibility

Connectivity and

invocationP2P Integration

User access/ roles Multi-tenancyMaking multiple

instances down

Globalization and

localizationAPI load

System

Integration

Identity federation

managementLatency Disaster Recovery

Compatibility

under different

situations

API securityEnd-to-End

Integration

Multi-tenant

penetration

Connectivity and

reliability with

3rd parties

Multi-tenancy SSO

Reliability and

availability

Customer

Scenarios

Page 52: Functional and Non-functional Test automation

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.52

Automation Frameworks

Telerik Test Studio, Telerik DeviceCloud

Sahi / Selenium scripts

Telerik Test Studio for Load / Performance Testing

Neoload, JMeter

SOAP UI / Node.js / PySys

Telerik Test Studio (with C#)

TestNG + Java + Sahi combined approachIntegration

Functional

(Web/Mobile UI)

Performance

API Invocation

Veracode, Websecurify, ZAP ToolSecurity

Page 53: Functional and Non-functional Test automation

[email protected]

http://ganeshniyer.com

GANESHNIYER