04 black box testing

BLACK B

OX TESTI

NG

G RAMESH

GOPALASWAMY_RAMESH@YAHOO.COM

TODAY’S AGENDABLACK BOX TESTING

What is Black Box Testing

Differences between Black Box Testing and White Box Testing

Different techniques of Black Box Testing

2

WHAT IS BLACK BOX TESTING

Black Box Testing is done without the knowledge of the internals of the system under test

3

LOCK AND KEY EXAMPLE OF BLACK BOX TESTING

4

CHARACTERISTICS OF BLACK BOX TESTINGDone based on requirements

Addresses (Should address) stated as well as implied requirements

Encompasses the end user perspective

Checks for valid and invalid conditions / inputs

May or may not know the technology aspects of the product

5

DIFFERENCES BETWEEN WHITE BOX AND BLACK BOX TESTING

Black Box Testing White Box Testing

No access to program code Has access to program code

Requires external perspective

Requires knowledge of program code

Set of techniques applicable to all other phases of testing

Typically applies only to unit testing, where code is involved

6

PRINCIPLES OF BLACK BOX TESTING

It is not possible to exhaustively test a product

Choose tests so that We can test “as much of the external functionality as possible” Uncover “as many defects as possible” Use “as short time as possible”

7

Have methodologies that choose tests that have a higher likelihood of uncovering new defects

TECHNIQUES / METHODOLOGIES OF BLACK BOX TESTINGRequirements Based TestingPositive and Negative TestingBoundary Value AnalysisDecision TablesEquivalence PartitioningState Based TestingCompatibility TestingUser Documentation TestingDomain Testing (leads to ad hoc testing)

8

GENERAL FORMAT OF DISCUSSION OF TECHNIQUESPresent some reasoning where applicable

List out one or two examples

Walk through the examples

Summarize the process for using the technique

9

REQUIREMENTS BASED TESTING

Done to ensure all requirements in SRS are tested

Difference between implicit and explicit requirements

Review requirements first to ensure they are consistent, correct, complete and testable

Review enables translation of (some of) implied requirements to stated requirements

A reviewed SRS tabulates requirements, along with a requirements id and a priority

This is the genesis of a Requirements Traceability Matrix

10

REQUIREMENTS TRACEABILITY MATRIX –GENERAL FORMAT<REFER TEMPLATE>

11

ZOOMING IN ON TESTING PART OF RTM

Fig 4.5

Test Condition: Different ways of testing a requirement (different types of mapping)

Test Case: Different conditions / scenarios for a given requirement

Phase of testing – helps scheduling

Importance of documenting expected results

12

POSITIVE AND NEGATIVE TESTING

Positive testing to check that the product does what it is supposed to

Behaves correctly when given right inputs Maps to a specific requirement “Coverage” is defined better

Negative Testing to show that the product does not fail when given unexpected inputs

Try to break the system No direct mapping to a specific requirement “Coverage” more challenging

13

BOUNDARY VALUE ANALYSIS

Most defects come up near “boundaries”

Reasons from a White Box perspective Programmers tentativeness in using the right relational

operator (< or <= ?) Multiple ways of implementing loops Confusing array subscripts

Reasons from a Black Box perspective Unclear requirements Ambiguous or “it depends” mindset!

14

BOUNDARY CONDITIONS – EXAMPLE 1:DATABASE BUFFERSDatabase starts with a pre-allocating number of buffers for

caching

Buffers filled up as needed

If full, buffers freed on a FIFO basis

15

16

…

Case i: All buffers free for use

…

Case ii: After inserting two buffers and still having free buffers

…

Case iii: After inserting the last available buffer: No free buffers

…

Case iv: No free buffers and new buffer coming in: First buffer needs freeing

Cases MostLikely to Find defects

EXAMPLE 2:VOLUME DISCOUNT CALCULATIONS

17

Product pricing depends on the quantity procured

Based on pre-defined “slabs”

BOUNDARY VALUE ANALYSIS (BVA):SUMMARY OF PROCESSLook for any kind of gradation or discontinuity in data

values which affect computation – the discontinuities are the boundary values, which require thorough testing.

Look for any internal limits like limits on resources (like the example of buffers given above). The behavior of the product at these limits should also be the subject of boundary value testing.

Also include in the list of boundary values, documented limits on hardware resources. For example, if it is documented that a product will run with minimum 4MB of RAM, make sure you include test cases for the minimum RAM (i.e., 4MB in this case).

The examples given above discuss boundary conditions for input data – the same analysis needs to be done for output variables also.

18

DECISION TABLES

A program’s behavior characterized by several decision variables

Each decision variable specifies a boolean condition

The distinct combinations of these decision variables leads to different scenarios

Each scenario occupies a row in the decision table and the row also has expected results

One representative data point from each scenario needs to be tested

19

EXAMPLE OF DECISION TABLE

20

SUMMARY OF DECISION TABLE

Identify the decision variables.Identify the possible values of each of the decision variables.Enumerate the combinations of the allowed values of each of

the variables.Identify the cases when values assumed by a variable (or by

sets of variables) are immaterial for a given combination of other input variables. Represent such variables by the Don’t Care symbol.

For each combination of values of decision variables (appropriately minimized with the Don’t Care scenarios), list out the action or expected result.

Form a table, listing on each but the last column a decision variable each. On the last column, list the action item for the combination of variables in that row (including Don’t Cares, as appropriate)

21

EQUIVALENCE PARTITIONING (EVP)

Generalization of BVA / Decision Table

Divide the (potentially infinite) set of values into a set of equivalence classes or partitions

One element of the class can act as a representative for the entire class

Results of the test for that one element extrapolated to all the other elements

22

EXAMPLE OF EVP

23

SUMMARY OF PROCESS FOR EVP

Choose criteria for doing the equivalence partitioning (range, list of values, etc.)

Identify the valid equivalence classes, based on the above criteria (number of ranges allowed values, etc.)

Select a sample data from that partitionWrite the expected result based on the requirements

givenIdentify special values, if any, and include it in the tableCheck to have expected results for all the cases preparedIn case, expected result is not clear for any particular

test case, mark appropriately and escalate for corrective actions. If you cannot answer a question, or find an inappropriate answer, consider whether you want to record this issue on your log.

24

STATE OR GRAPH BASED TESTING

Useful for Language Processors or compilers Workflow modeling Dataflow modeling

25

26

1SELECT

2*

3

<col n

ame>

4,

<col name>5

FROM

FROM

6<table name>

7

,

<ta

ble

nam

e>

WHERE

…

Example of Graph Based Testing for a SQL command

PROCESS FOR LANGUAGE PROCESSORS TESTING USING GRAPH METHODIdentify the grammar for the scenario. In the

above example, we have represented the diagram as a State Machine. In some cases, the scenario can be a Context Free Grammar, which may require a more sophisticated representation of a “State Diagram”.

Design test cases corresponding to each valid state-input combination.

Design test cases corresponding to most common invalid combinations of state-input.

27

28

RequestorApprovingAuthority Finance

Vendor

Received Item

AvailableOff the shelf

Buy & Expense(Transaction: Expense Report)

Requisition Approval

Budget Update &

Purchase Order

Delivery

Note

Payment

Item D

elivery

WORKFLOW MODELING

EmployeeDesires Leave

VerifyEligibility

Leave ApplicationForm

ManagerEnsure

Feasibility

Reject

Ineligible

Eligible

Feasible

Not Feasible

Approve

29

GENERAL CHARACTERISTICS OF GRAPH BASED TESTINGThe application can be characterized by a set of states

The data values (screens, mouse clicks, etc) that cause the transition from one state to another is well understood

The methods of processing within each state to process the

input received is also well understood

30

COMPATIBILITY TESTING

To ensure that the product works consistently with infrastructure components

Could be parameters of hardware, OS, network…

Concept of a compatibility matrix

Backward compatibility and forward compatibility

31

USER DOCUMENTATION TESTING

To check if what is stated in the document is available in the product

To check if what is there in the product is available in the document

Documents include user guides, installation guides, setup guides, read me file, software release notes and on-line help

32

BENEFITS OF DOCUMENTATION TESTING

User documentation testing aids in highlighting problems that were overlooked during reviews

High quality user documentation ensures the consistency of documentation and product, thus minimizing possible defects reported by customers.

Results in less difficult support calls. Contributes to better customer satisfaction and better

morale of support staff.New programmers and testers who join a project group

can use the documentation to learn the external functionality of the product.

Customers need less training and can proceed more quickly to advanced training and product usage

Reduced training costs for customers

33

DOMAIN TESTING

34

YES

WHITEBOXTESTING

NO

Knowledge of Program CodeAvailable?

Knowledge of Requirements Specifications Available?

NO

YES

BLACKBOXTESTING

DOMAINTESTING

DOMAIN TESTING

Have “domain people” perform tests by using the software

Business flow determines the test, not “logic” or “steps”

Tests what the users do on a typical business day

Captures the typical problems faced by users (not necessarily captured in SRS)

35

WHEN TO USE WHAT…

When you want to test scenarios that have… The most effective black box testing technique is likely to be…

Output values are dictated by certain conditions depending upon values of input variables

Decision Tables

Input values being in ranges, with each range exhibiting a particular functionality

Boundary Value Analysis

Input values being divided into classes (like ranges, list of values, etc), with each class exhibiting a particular functionality

Equivalence Partitioning

Positive and negative testing

Workflows, process flows or language processors Graph based testing

To ensure that requirements are tested and met properly Requirements Based Testing

To test using the domain expertise rather than the product specification

Domain Testing

To ensure that the documentation is consistent with the product

Documentation Testing

36

TODAY’S AGENDA (RECAP)BLACK BOX TESTING

What is Black Box Testing

Differences between Black Box Testing and White Box Testing

Different techniques of Black Box Testing

37