Practical Software Testing Tools

Software Testing ToolsAn overview

As per OU Syllabus

Dr Ganesh Neelakanta IyerAnad Adike

© 2015 Progress Software Corporation. All rights reserved.3

Manual vs Automated testing


Manual Testing vs Automated Testing

Manual software testing• performed by a human sitting in front of a computer carefully going through application

screens, trying various usage and input combinations, comparing the results to the expected behavior and recording their observations

• Manual tests are repeated often during development cycles for source code changes and other situations like multiple operating environments and hardware configurations

Automation Testing (using a tool)• An automated testing tool is able to playback pre-recorded and predefined actions, compare

the results to the expected behavior and report the success or failure of these manual tests to a test engineer

• Once automated tests are created they can easily be repeated and they can be extended to perform tasks impossible with manual testing

• Because of this, savvy managers have found that automated software testing is an essential component of successful development projects


Automation

Functional Automation (Black box testing)• Windows applications – Winrunner, QTP• Web applications – Silk Test, Selenium, Telerik Test Studio• Regression Testing – Same set of tools

Performance/Load Testing• Load Runner, Jmeter

Security Testing (black box and White box)• Veracode, ZAP

Functional Testing

Overview of WinRunner, TSL, SilkTestDemo of Selenium


WinRunner


WinRunner

Designed to help customers save testing time and effort by automating the manual testing process

Manual process• Perform operations by hand, • visually check results, and • log results by hand

Automated process: • Create a test script that will perform the same operations as a Human operator• check the same information, and • create a summary report showing the test status


Recording Modes

• Two Modes• Context-sensitive mode• Analog mode

• Tests can combine both recording modes

• Context-Sensitive is the default mode

• Switch between modes using same record key (F2)


Context-Sensitive Mode• Object-based

• Unaffected by minor UI changes

• Maintainable (readable/editable)

• Generally used with GUI applications

• Portable script


Context-Sensitive Mode

set_window ("Save As");edit_set(“File Name”,“output14”);

output14

Set focus to the window

Set the edit field content

Button press

button_press(“OK”);


Analog Mode• Position-dependent

• Works with any application

• UI changes force test script changes

• Usually drives tests with • mouse, • keyboard and • other such manual user inputs

• Less maintainable

y

x


Analog Mode

move_locator_track (1);

mouse drag

mtype (" <T55> <kLeft>-<kLeft>+"); mouse click

output14

type (" <t3>output14" );

keyboard

move_locator_track (2); mtype (" <T35><kLeft>-<kLeft>+ ");

timing


Recording Modes Context-Sensitive mode statements can be recorded or

programmed• record: button_press, win_activate• program: list_get_num_items, edit_get_text• recommended for most situations due to greater robustness

Analog mode statements are rarely programmed, mostly recorded and edited• record: move_locator, type, mtype• program: move_locator_abs, move_locator_rel, click• Analog statements are useful for literally describing the keyboard, mouse, and mouse button

input of the user


Run Modes Debug

• debug is good to use while the test script is being debugged• these test results are overwritten with each new run

Verify• Corresponds to actual results• Generally used when executing testing sessions where results need to be stored

Update• Corresponds to “expected” results. Expected results are the benchmarks used to verify

test results• Test runs in “Update” mode generate the expected results for future runs to compare

back against• These test results become the expected results for subsequent test runs in “Verify”

mode


Synchronization Enhances a test script to ensure reliable replay

• accounts for delays in order to prevent the automated script from running faster than the tested application

• critical for successful test automation implementation• among the main reasons why record-n-playback is not reliable

In Context-Sensitive mode• Examples: (operations)

– wait for a window to appear– wait for a bitmap to refresh– wait for an object property– wait for a specific amount of time

In Analog mode• Examples: (operations)

– wait for a window bitmap to appear / refresh– wait for a specific amount of time


Window Synchronization

invoke_application(“Notepad”,””, “c:\\temp”, SW_SHOW);set_window (”Login”, 10);

edit_set(“User ID:”, “guest”);edit_set(“Password:”, “mercury”);

button_press(“OK”);set_window Waits for the specified window to appear onscreen. If the

window appears before the timeout, the script immediately proceeds to the next line.


Synchronization Controls


Test Script Language


TSL (Test Script Language) TSL is a C-like language

• High-level proprietary programming language designed for test automation• procedural language

Full programming support• variables, arrays, functions• regular expressions• control flow, decision logic, looping


Built-in TSL Functions TSL provides a comprehensive library of hundreds of built-in functions to

simplify test creation• window/object functions• environment functions• reporting functions• database query functions• file/spreadsheet functions• Win32 functions

WinRunner provides a graphical function browser to assist you • Function Generator


Language Syntax

*** Same syntax as in standard C ***


Variables Basic Rules

• do not need to be declared / defined• specific data types are not explicitly

defined• case sensitive• first letter must be a character or

underscore• cannot be a reserved word• by default all variables are local

(static)• can also be public and/or const

Arrays• single dimension: cust[1], cust[2],

cust[3]• multi-dimension: address[1,1],

address[1,2]• Can be indexed with number

– address[1], address[2]• Can be indexed with strings

(associative)– address[“John”], address[“Mary”]


Operators Math

+ - * / ^ % ++ -- Logical

&& || ! Relational

== != >= <= > < Assignment

= += -= *= /= ^= %= Concatenation

&


Test Verification Enhancing a test script to verify data onscreen

• check objects’ values / states• check images• check text• check the database

Context-Sensitive verification

Analog verification


Exception/Recovery Handling (*)

Unexpected errors during replay• unlike error-handling, these can appear at any time when running a script

WinRunner provides a mechanism to trap and handle exceptions• popup exceptions

– popup windows

• object exceptions– object property value changes

• TSL exceptions– TSL error codes


Functions and Libraries simplifies building test frameworks

• application-specific functions• general-purpose functions• greater modularity

can be stored in a script• compiled module (function library)• can be loaded as part of startup or initialization script and available globally

facilitates data-driven testing• data-driven testing is where data retrieved externally from the test being executed drives

the test rather than using hard-coded data within each test case. Using application specific custom functions and scripts helps further the benefits of data-driven testing.


Architecture and use of Silk Test


What is SilkTest ?

A powerful tool for running automated test cases on the front end

• A tool for testing Web based applications across different browsers

• Very very stupid – you have to tell it everything in its own language (4test)

• Inflexible when it comes to interpreting your commands• It cannot guess at what you mean• It requires a certain syntax with certain words

– This can be quite maddening


What can you do with it ?

Mostly anything you can do in a Web browser, you can do with SilkTest• Enter text in text fields• Click buttons• Check boxes• Get files• Clear your cache• etc.


Using a browser

Though SilkTest is its own product, you still have to use a browser to access your application

You have to tell SilkTest what kind of browser you are going to use for your testing so that it knows what it is looking at.

However, you can only run SilkTest on one instance of a browser at a time• One IE browser going at a time


Architecture

Project .vtp (for maintenance) Testplan .pln (organizing the tests) Test Frame* .inc (AUT Window details) Scripts .t (test scripts) Data driven tests .g.t (scripts with many data) Include file .inc Suite .s (for batch runs) Result .res


Test Process

Manage requirements using SilkCentral Test Manager Manage test plans and cases using Silk test Associate test plans/cases to requirements using test manager Declare GUI declarations using SilkTest Develop scripts for atomic activities first so that they can be

reused later Develop scripts using SilkTest and run them Track defects using Defect Tracker


Selenium


Overview of Selenium

Selenium is a robust set of tools that supports rapid development of test automation for web-based applications.

Selenium provides a rich set of testing functions specifically geared to the needs of testing of a web application.

Selenium operations are highly flexible, allowing many options for locating UI elements and comparing expected test results against actual application behavior.

Supports Cross Browser Testing. The Selenium tests can be run on multiple browsers. Allows scripting in several languages like Java, C#, PHP and Python. Assertion statements provide an efficient way of comparing expected and actual

results. Inbuilt reporting mechanism.


Demo

Performance and Load Testing

Overview of LoadRunnerDemo of JMeter


Why Load Test An Application

Does the application respond quickly enough for the intended users?

Will the application handle the expected user load and beyond?

Will the application handle the number of transactions required by the business?

Is the application stable under expected and unexpected user loads?


Functional vs. Load Web Testing

Functional test

Load test

Do business processes function properly after

implementation?Functionality

EXAMPLEOBJECTIVE

Will 2,000 concurrent hits crash the server?

Stability

Is response time acceptable according to specifications?

Performance

Do business processes function properly under

heavy load?

Functionality under load

EXAMPLEOBJECTIVE


Types of Performance Testing

Find out whether the system can handle the expected load upon deployment under real-world conditions.

Load Testing

Find the application's breaking point. Apply testing that measures whether the application's environment is properly configured to handle expected or

potentially unexpected high transaction volumes.

Stress Testing

Find the stability of the system with respect to handling large amounts of data over extended time periods.Volume Testing

Find the behavior and performance of each tier. Component

Testing


Objectives of Performance Testing

How long does it take to complete a task?Application Response Time

Reliability

Configuration Sizing

Capacity Planning

Acceptance

Bottleneck Identification

Regression

Product Evaluation

How Stable is the system under a heavy work load?

Which configuration provides the best performance level?

At what point does degradation in performance occur?

Is the system stable enough to go into Production?

What is the cause of degradation in performance?

Does the new version of Software adversely affect response time?

What is the best server for 100 users?


Testers

Load Generation System Under Test

Do you have the testing resources?• Testing personnel• Client machines

How do you synchronize users?How do you collect and analyze results?How do you achieve test repeatability?

Analysis?123.20

All of you, click the GO button

again

Manual Testing Is Problematic

Web server Database server

Coordinator


Use of LoadRunner


System Under Test

The LoadRunner Solution

Web server Database server

Load Generation

Vuser host

Overcomes resource limitations

• Meaningful results with analysis tools• Repeats tests with scripted actions

AnalysisController

• Runs many Vusers on few machines• Replaces testers with “Virtual Users”

• Controller manages the Vusers


Load Testing Tools Available

LoadRunner : HP (Formerly Mercury Interactive) e-Load :Emprix Silk Performer : Borland (Seague) QALoad : Compuware Rational Performance Tester : IBM Rational Web Load : Radview Neo Load : Neotys Open STA : Open Source Jmeter Telerik Test Studio


Introduction to Load Runner• Load Runner is a Mercury Interactive Tool that predicts performance and

behavior of the system

• By creating lots of load, you can see how the system reacts at peak levels or with simultaneous Users

• To test the application, LoadRunner emulates an environment where multiple users work concurrently. While the application is under load, LoadRunner accurately measures and analyzes the system performance, and its functionality


Supporting Environments• Application Deployment Solution - The Citrix protocol. • Client/Server - MS SQL, ODBC, Oracle Web Applications 11i and

DNS protocols. • Custom - C templates, Visual Basic templates, Java templates,

Javascript, and VBScript • Distributed Components - COM/DCOM, Corba-Java, and Rmi-Java

protocols. • E-Business - FTP, LDAP, Palm, Web (HTTP/HTML), Web Services• Enerprise Java Beans -EJB Testing and RMI-Java protocols. • ERP/CRM - Baan, Oracle NCA, Peoplesoft 8, SAP-Web, SAPGUI


Supporting Environments• Legacy

Terminal Emulation (RTE). • Mailing Services

Internet Messaging (IMAP), MS Exchange (MAPI), POP3, and SMTP. • Streaming

MediaPlayer and RealPlayer protocols. • Wireless

i-Mode, VoiceXML, and WAP protocols.

Internet/ Intranet

Web ServersClients App. Servers Database Server


Supporting Environments

• Platforms• NT, 2000, XP• Sun• HP• IBM• Linux


The LoadRunner Solution

Virtual User Generator

Generates load and collects test results

Creates Scripts as one Single User.

Compiles and displays test results with

graphical and statistical tools

LoadRunner Controller

LoadRunner Analysis


LoadRunner Terminology

• Scenarios• Using LoadRunner, you divide your application performance testing requirements into

scenarios.• A scenario defines the events that occur during each testing sessions.• For example, a scenario defines and controls the number of users to emulate, the

actions that they perform, and the machines on which they run their emulations. • Vusers

• In a scenario, LoadRunner replaces human users with virtual users or Vusers.• When you run a scenario, Vusers emulate the actions of human users—submitting

input to the server. • A scenario can contain tens, hundreds, or even thousands of Vusers.



• Vuser Scripts• The actions that a Vuser performs during the scenario are described in a Vuser

script.• When you run a scenario, each Vuser executes a Vuser script. Vuser scripts

include functions that measure and record the performance of the server during the scenario.

• Transactions• To measure the performance of the server, you define transactions. • Transactions measure the time that it takes for the server to respond to tasks

submitted by Vusers.



• Rendezvous Points• You insert rendezvous points into Vuser scripts to emulate heavy user load on the server. • Rendezvous points instruct multiple Vusers to perform tasks at exactly the same time. • For example, to emulate peak load on the bank server, you insert a rendezvous point to

instruct 100 Vusers to simultaneously deposit cash into their accounts.

• Controller• You use the LoadRunner Controller to manage and maintain your scenarios. • Using the Controller, you control all the Vusers in a scenario from a single workstation.



• Hosts• When you execute a scenario, the LoadRunner Controller distributes each Vuser in the

scenario to a host. • The host is the machine that executes the Vuser script, enabling the Vuser to emulate

the actions of a human user.

• Performance Analysis• Vuser scripts include functions that measure and record system performance during

load-testing sessions. • During a scenario run, you can monitor the network and server resources. • Following a scenario run, you can view performance analysis data in reports and

graphs.


LoadRunner Components

ControllerVuGen

Analysis

LoadRunner

Tuning


How LoadRunner Works ?


TuneSystem Based

on Analysis

LoadRunner Expert Workflow“The Big Picture”

AnalyzeSystem Under Load

Phase 5

LoadRunnerV U G E N

LoadRunnerC O N T R O L L E R &

A N A L Y S I S

RunScenarios

Phase 4

CreateScenarios

Phase 3

Create Web Virtual Users

Phase 2Phase 1

Plan Load Test


Use of JMeter


Introduction to JMeter

Definition • JMeter is an Apache Jakarta project that can be used as a load testing tool for analyzing and

measuring the performance of a variety of services, with a focus on web applications.

Why ?• Free • JMeter can be used as a unit test tool for JDBC database connection, FTP, LDAP,

WebServices,J MS, HTTP and generic TCP connections. • JMeter can also be configured as a monitor, although this is typically considered an ad-hoc

solution in lieu of advanced monitoring solutions.


Load test Process

System Analysis Creating Virtual User Scripts Defining User Behavior Creating a Load Test Scenario Running Load Test Scenario Analyzing Results


System Analysis

Enable customers to convert their goals and requirements into a successful test script.

you can: -Create your own test Script consistently - Careful UI design allows you to create your own test Easily


Creating Virtual User Scripts

- Tester must emulate the real user by driving the real application as a client. - JMeter support this by adding thread group element, this tell JMeter:- the number of users you want to simulate, - how often the users send requests - how many requests they send.- What request (FTP Request, HTTP Request, and JDBC request). - validate that your application is returning the results you expect.


Defining User Behavior

JMeter allows you to simulate human actions more closely - By controlling how long JMeter engine delays between each sample - So, you can define the way that the script runs


Creating a Load Test Scenario

JMeter offers support for this step by:

Assign scripts to individual virtual user Tester can define any number of virtual users needed to run the tests Allows user to simulate concurrent connections to server application.By

enables user to create multiple threads ( virtual user) executing different test plan.

Increasing the number of virtual users in a controlled fashion.


Running Load Test Scenario

With JMeter you can run your tests in a very easy way


Analyzing Results

JMeter offers support for this step by:

displaying the data visually (Graph Results) . save data in file. allows user to see one multiple views of the data displays the response from the server. shows the URL of each sample taken . listeners will show different sets of data. it can send email based on test results.


Analyzing Results

Showing detailed performance results that can be easily understood and analyzed to quickly pinpoint the root cause of problems.


Demo of JMeter


Souce code Testing Utilities

http://sqa.fyicenter.com/FAQ/Testing-Tools/Other_Source_Code_Testing_Tools.html




Why source code testing?

Code coverage• To determine what proportion of your project's code is actually being tested by coded tests

such as unit tests• code coverage is a measure used to describe the degree to which the source code of a

program is tested by a particular test suite. • A program with high code coverage has been more thoroughly tested and has a lower

chance of containing software bugs than a program with low code coverage

Enables in white box testing• method of testing software that tests internal structures or workings of an application, as

opposed to its functionality (i.e. black-box testing)• also known as clear box testing, glass box testing, transparent box testing, and

structural testing


Advantages of source code testing

Enables to test better

Enables to understand uncovered test scenarios

Ensuring messages for various scenarios are correct• Error messages• Notifications


Source Code Testing Utilities

Splint / LCLint (Lint - static source code analysis tool)Splint (formerly LCLint) is a rewrite of the famous Unix lint utility. It is a static source code analyser, that checks your C program source code for bugs and potential problems. This static source code analysis tool requires a certain amount of configuration before you can use it productively. There are versions for a number of operating systems, including Win32, Linux and other Unix systems.

Ftncheck (Lint utility / Static Source Code Analysis Utility for Fortran)ftncheck, a static source code analysis utility, checks your Fortran programs for semantic errors, leaks and potential bugs. For example, you can use ftncheck to scan your program for the use of uninitialized variables and undeclared variables.

Jlint: Java Program Checker (Lint)Like the lint utility which checks C programs, jlint is a static source code analysis tool that scrutinizes your Java source code for suspicious constructs that might indicate a program bug. Although the program is shareware, it is distributed with source code without any restrictions (and hence is listed here).


http://www.splint.org/



http://www.thefreecountry.com/sourcecode/cpp.shtml

http://www.dsm.fordham.edu/~ftnchek/

http://www.dsm.fordham.edu/~ftnchek/

http://www.thefreecountry.com/compilers/fortran.shtml

http://www.ispras.ru/~knizhnik/jlint/ReadMe.htm

http://www.ispras.ru/~knizhnik/jlint/ReadMe.htm

http://www.thefreecountry.com/compilers/cpp.shtml

http://www.thefreecountry.com/compilers/java.shtml



Future of Software Testing and

Test Engineer


PAST


PAST

PRESENT

PRESENT

FUTURE


What does this mean from a Testing perspective?

Seamless Integration

Sky is the limit for a Software Test Engineer• Internet of Things• Big Data and Analytics• Social• Security and Privacy• Open Hardware


Dimensions of a Quality Analyst

Technical QA

Consultant

Analyst

Coaching

DevOps


Career Opportunities as Software Quality (Test) Engineer

Every software needs to be QAed

Familiarize with Popular tools• Web UI Testing Selenium• Performance Testing JMeter • Security Testing ZAP


ACM Membership Benefits

Computing Information Resources: Learning Center with resources for lifelong learning

• Online courses targeted toward essential IT skills and popular certifications• Online books from Books24x7®, Safari®, Morgan Kaufmann and Syngress• Webinars on hot topics, presented by today's innovators

The option to subscribe to the full ACM Digital Library, which includes over 2 million pages of text A full year (12 issues) electronic subscription to Communications of the ACM A full-year (4 issues) subscription to XRDS: Crossroads, ACM's Student Magazine Complimentary e-mentoring services from MentorNet A full-year electronic subscription to monthly MemberNet and quarterly Student Quick Takes, ACM's

member newsletters. A full-year subscription to TechNews, ACM's tri-weekly email IT news digest A full-year subscription to CareerNews, ACM's twice monthly email career news digest acmqueue website featuring articles, commentary, blogs, roundtables, case studies, mutlimedia and more Access to Ubiquity, ACM's IT opinion magazine and forum, and eLearn, ACM's magazine dedicated to

distance education

http://learning.acm.org/

http://dl.acm.org/

http://cacm.acm.org/

http://xrds.acm.org/

http://www.acm.org/membership/student/mentornet

http://membernet.acm.org/

http://www.acm.org/membership/student/sqt

http://technews.acm.org/

http://www.acm.org/membership/careernews/careernews

http://queue.acm.org/

http://www.acm.org/ubiquity

http://www.elearnmag.org/

Engineering

Practical Software Testing Tools