CEN 5070 - Software V & V Unit Testing Concepts

CEN 5070 - Software V & V

Unit Testing Concepts

2/2000 Unit Test Concepts 2

Purpose

This module presents the basic concepts of black-box and white-box testing for unit testing. A systematic approach is shown for deriving functional, boundary and white-box test cases.

"… arbitrarily selected test set ... results in inefficient testing, leaving

some functions untested while performing redundant testing of others."Darlene Mackay, Quality Consultants Unlimited


Agenda

• What Is Unit Testing

• Black-Box Testing

• White-Box Testing

• Putting It All Together


WHAT IS UNIT TESTING?

• Executing a software element to determine whether it meets its specification

• Executing a software element to discover defects or anomalies

• Inspecting software element code to discover defects or anomalies.


WHAT IS A UNIT?

• Named software element

• Separately invokable

• Performs single function

• Examples• Subprogram or script• Field with validation• Database stored procedure• Java class method


SPRAE: A Model for Testing Practice

• Specification -- basis for software testing

• Premeditation -- testing requires planning, forethought

• Repeatability -- process, results independent of tester

• Accountability -- testing artifacts maintained

• Economy in the use of human, time and computing resources


A TESTING LIFECYCLE

Analysis

Design

Implementation

Execution

Evaluation

Specification

Test Strategy/Plan

Test Script, Data, Driver

Defect DataProblem Reports

Test Results

Test Cases


Agenda

• What is Unit Testing





BLACK-BOX TESTING

• Testing based on the specification rather than the implementation.

• Specification defines the expected response(s) to stimuli

SoftwareunderTest

Stimuli Response(s)


BLACK-BOX TECHNIQUES

• Functional testing -- tests the behavior of the software.

• Boundary testing -- tests behavior at the lower/upper bounds of input values

• Random testing -- tests using randomly generated stimuli (load testing)

• Intuitive (ad hoc) testing -- error guessing


FUNCTIONAL TEST DESIGN METHODOLOGY

• Specification

• Identify behaviors

• Develop test cases

• Write test script


EXAMPLE A(1) Specification

• Compute pay for an hourly employee, given the number of hours worked and the hourly pay rate. Compute overtime at 1.5 times hourly rate for hours in excess of 40.

SoftwareunderTest

Hours Pay

Rate


EXAMPLE A(2) Identify Behaviors

• Case 1: No overtime (Hours <= 40)• Expect Pay = Hours * Rate

• Case 2: Overtime (Hours > 40)• Expect Pay = 40*Rate+1.5*Rate*(Hours - 40)


EXAMPLE A(3) Create Test Cases

• Case 1: No overtime (Hours <= 40)• Use Rate = 10, Hours = 30

• Expect Pay = Hours * Rate = 300

• Case 2: Overtime (Hours > 40)• Use Rate = 10, Hours = 50

• Expect Pay = 40*Rate+1.5*Rate*(Hours - 40)

= 550


EXAMPLE A(4) Write Test Script

StepStimuli Expected Response

Hours Rate Pay =

1

2

30

50 10

10 300

550


A MORE COMPLEX EXAMPLE (B)

• Increased number of behaviors

• Use of decision table to document

behaviors

• Test case generation from decision table


EXAMPLE B(1) Specification

• Compute pay for employee, given the number of hours worked and the hourly pay rate. For hourly employees (rate < 30), compute overtime at 1.5 times hourly rate for hours in excess of 40. Salaried employees (rate >= 30) are paid for exactly 40 hours.


EXAMPLE B(2) Identify Behaviors

• Case 1: Hourly AND No overtime • (Rate < 30) & (Hours <= 40)• Expect Pay = Hours * Rate

• Case 2: Hourly AND Overtime • (Rate < 30) & (Hours > 40)• Expect Pay = 40*Rate+1.5*Rate*(Hours - 40)

• Case 3: Salaried (Rate >= 30) • Expect Pay = 40 * Rate


DECISION TABLE

Condition

c1: Rate < 30 | Y Y N N

c2: Hours <= 40 | Y N Y N

Action

a1: Pay = Straight time | X

a2: Pay = Overtime | X

a3: Pay = Professional | X X

Columns defineBehaviors


EXAMPLE B(3) Create Test Cases

• One test case per column of decision table

• Case 1: Hourly, No Overtime

• Case 2: Hourly, Overtime

• Case 3: Salaried, No Extra Hours

• Case 4: Salaried, Extra Hours

• Order the test cases by column


EXAMPLE B(4) Write Test Script


Hours Rate Pay =

1

2

30

50 10

10 300

550

3 30 40 1600

4 50 40 1600


RULES -- DECISION TABLES

Condition

c1: Rate < 30 | Y Y N N

c2: Hours <= 40 | Y N Y N

Action

a1: Pay = Straight time | X

a2: Pay = Overtime | X

a3: Pay = Professional | X X Use X to select action(s)

Elementaryconditions

Use 'Y', 'N', '-' or space


Your Turn -- Problem P1(1) Specification

• Compute the dosage of drug X for patient, given the patient's Age and Weight. For patients 12 and under, the dosage is 1 pill. For patients over 65, the dosage is 2 pills. For all other patients, the dosage is 2 pills plus an extra pill for each 50 pounds above 120. The drug can not be given to patients over 300 pounds or over the age of 80.


Your Turn(2a) Identify Behaviors

CaseExpected

Stimulus Description #Pills

1

2

3

4

5

6


Your Turn (2b) Decision Table

c1: Age <= 12 |

c2: Age > 65 |

c3: Age > 80 |

c4: Weight > 300 |

c5: Weight > 120 |

a1: Pills = 0 |

a2: Pills = 1 |

a3: Pills = 2 |

a4: Pills = 2+(W-120)/50 |


Your Turn(3) Create Test Cases

Case 1 2 3 4 5 6 7 8 9

Age ___ ___ ___ ___ ___ ___ ___ ___ ___

Weight ___ ___ ___ ___ ___ ___ ___ ___ ___

Pills ___ ___ ___ ___ ___ ___ ___ ___ ___


Your Turn(4) Write Test Script

StepStimuli Pills=

Age

1

2

3

4

5

6

Weight StepStimuli Pills= Age

7

8

9

10

12

Weight

11


SCALING UP

The heart of the approach is to use a decision table as a thinking tool. The most critical task in this process is to identify all the stimuli and responses. When there are many logical combinations of stimuli, the decision table can become large, indicating that the unit is probably too complex.


IDENTIFYING BEHAVIORApproaches

• Work backwards• Identify each response• Identify conditions that provoke response• Identify separate stimuli

• Work forward• Identify each stimulus• Identify how stimulus influences what unit does• Specify the response• Treat stimuli combinations


IDENTIFYING STIMULI

• Arguments passed upon invocation• Interactive user inputs• Internal, secondary data

• global or class variables

• External data (sources)• file or database status variables• file or database data

• Exceptions


IT PAYS TO BE A GOOD STIMULUS DETECTIVE

• Failure to identify stimuli results in an incomplete, possibly misleading test case

• The search for stimuli exposes• interface assumptions -- a major source of

integration problems• incomplete design of unit• inadequate provision for exception handling


IDENTIFYING RESPONSES

• Arguments/Results passed back on exit• Interactive user outputs• Internal, secondary data

• updated global or class variables

• External data (sinks)• output file or database status variables• output file or database data

• Exceptions


IT PAYS TO BE A GOOD RESPONSE DETECTIVE

• Failure to identify responses results in • incomplete understanding of the software

under test• shallow test cases• incomplete expected results• incomplete test "success" verification --

certain effects not checked

• To test, one must know all the effects


A SKETCHING TOOL Black-Box Schematic

Stimulus Type Response Type

SoftwareunderTest

Argument

Inputs

Globals

Database

Exception

Argument

Outputs

Globals

Database

Exception


BEFORE CONTINUTINGMuch of the discussion so far involves how to identify what software does. We have introduced thinking tools for systematically capturing our findings. These thought processes and tools can be used anywhere in the lifecycle, e.g., in software design!

One Stone for Two Birds!!


BOUNDARY TESTING DESIGN METHODOLOGY

• Specification

• Identify elementary boundary conditions

• Identify boundary points

• Generate boundary test cases

• Update test script (add boundary cases).


(1) Specification

• Compute pay for an hourly employee, given the number of hours worked and the hourly pay rate. Compute overtime at 1.5 times hourly rate for hours in excess of 40.

SoftwareunderTest

Hours Pay

Rate


(2) Identify Boundary Conditions

• Condition 1 (bc1): Hours <= 40

• Observations:• Condition taken directly from decision table

• No boundary conditions for Rate


(3) Identify Boundary Points

• bc1 (Hrs <= 40) Boundary Points• Point 1: AT the boundary: Hours = 40

• Point 2: Just INside: Hours = 39

• Point 3: Just OUTside: Hours = 41

• Observations:• Inclusive inequalities have 3 boundary points


(3a) BP Generalization

• bc x > y has TWO boundary points• bp1: Just INside: x = y + precision

• bp2: Just OUTside: x = y

• bc x = y has THREE boundary points:• bp1: OUTlo: x = y - precision

• bp2: OUThi: x = y + precision

• bp3: AT: x = y


(3b) BP Generalization

• bc x != y has THREE boundary points:• bp1: INlo: x = y - precision

• bp2: INhi: x = y + precision

• bp3: OUT: x = y


(4) Generate Test Cases

• Combine Hours boundary points with Rate• Case 1 (AT): Hours = 40, Rate = 10, Pay=400

• Case 2 (IN): Hours = 39, Rate = 10, Pay=390

• Case 3: (OUT): Hours = 41, Rate=10, Pay=415

• Observations:• Test each boundary point individually

• Then consider pair-wise boundary points


(5) Update Test Script


Hours Rate Pay =

1

2

30

50 10

10 300

550

3 40 10 400

4 39 10 390

5 41 10 415


Your TurnBoundary Testing

c1: Age <= 12 | Y N N

c2: Age > 65 | Y N N

c3: Age > 80 | Y

c4: Weight > 300 | Y N N N N

c5: Weight > 120 | N Y

a1: Pills = 0 | X X

a2: Pills = 1 | X

a3: Pills = 2 | X X

a4: Pills = 2+(W-120)/50 | X

Decision Table:


Your Turn (2-3) Boundary Conditions/Points

Boundary Condition Boundary Point (BP)AT IN

bc1:

OUT

bc2:

bc3:

bc4:

bc5:


Your Turn(4) Generate Boundary Test Cases

• To create a test case, you must pair an Age with a Weight• Weight boundary point + NOMINAL Age

• Age boundary point + NOMINAL Weight

• OUT Age + OUT Weight A nominal value is one that is not close to a boundary point. For simplicity, use the same nominal value in all test cases.


Your Turn(4) Boundary-Nominal Test CasesCondition btc BC Weight Age Expect

Weight>300 1 IN 301 21 0

2 OUT 300 21

Age <= 12 3 IN 220 11

4 AT 12

5 OUT 13

Age > 65 6 IN

7 OUT

Age > 80 9 IN

10 OUT

Weight>120 11 IN

12 OUT


Your Turn(4) Out-Out Boundary Test Cases

Condition OUT BP

Weight Age btc Weight Age Expect

>300 <=12 13

>65 14

>80 15

>120 <=12 16

>65 17

>80 18


OBSERVATIONS

• Functional testing defines a minimal number of test cases

• Boundary testing adds a large number of test cases, but are EASY to create

• Boundary testing finds lots of errors!


BOUNDARIES EXIST FOR

• Optional fields: (present, missing).

• Variable length data: null string, max length

• Database tables: empty

• Searching: first/last position

• File: closed, empty


RANDOM TESTING

• Beyond scope of this course

• Generally used to bombard software with inputs

• No effort to identify expected results

• Appropriate for automated load testing, where concern is for capacity/volume.


INTUITIVE (AD HOC) TESTING

• Most common type of informal testing• Often, no specification!!

• No scripts

• Not repeatable

• Not systematic

• Very effective

• Does not guarantee thorough testing


INTUITIVE TESTING

• Ad hoc, exploratory testing, ideal for destructive testing• Goal is to break the software via unexpected

stimuli or sequences of stimuli

• Benefits• Flushes out holes in the specification .. What

really should happen when the input is X? • Forces treatment of error/exception handling


MAIN POINTSBLACK-BOX TESTING

• Black-box = spec-based testing• Tests intentions• Key knowledge = stimuli and responses• Decision table organizes S/R conditions• Boundary testing flushes coding errors• Black-box test case design can drive

software design -- same issues addressed


Agenda






WHITE-BOX TESTING

SoftwareunderTest

Stimuli Response(s)

Testing to ensure that software does not do what is not supposed to do. Test ALL of it!

Tick -- Tick -- TickNo-Surprise Software!!


WHITE-BOX TESTING

• Focus is thorough execution of program elements during the testing process.

• Warning: Tests only what is built, not what was intended!

SoftwareunderTest

Stimuli Response(s)


WHITE-BOX TESTING

• Concept of coverage. Numeric measure of thoroughness of testing, relative to • Statements• Branches• Conditions• Paths


CONTROL FLOW GRAPH

• Defines the flow of control through unit.

1

2 4

5

3


CONTROL FLOW GRAPH TERMINOLOGY

• 5 NODES• sequential blocks of code

terminated by a branch

• 3 PATHS:• [1,2,3,5], [1,2,5], [1,4,5]

• 2 BRANCHES• 1 and 2 are decision nodes

1

2 4

5

3


CONTROL FLOW GRAPH COVERAGE

• One test case forces execution of one path (red)

• Paths are determined by branches (decision nodes)

• A thorough test set forces execution of all paths (red, green, blue).

1

2 4

5

3


COVERAGE LEVELS (%)

• Statement -- a statement has been executed at least once during testing

• Branch -- each outcome of a branch has been performed at least once during testing

• Path -- a path through the code has been executed at least once during during testing

• Condition -- a condition has evaluated to true and to false at least once during testing


CONTROL FLOW GRAPH COVERAGE MEASUREMENT

• For 2 test cases (red, green)• Node (statement) cov = 4/5• Branch cov = 1/2 [2]• Path cov = 2/3

• Acceptable coverage levels• Statement cov = 90%• Branch cov = 80%• Path cov = 70%

1

2 4

5

3


BRANCH vs CONDITION COVERAGE

• Code example 1

2 4

• 100% Branch coverage [1] • (x=0,y=2), (x=1,y=2) [TT,FT]

• But not 100% Condition coverage • Need case TF (x=0,y=1)

if (x<1 && y>1) x = x + y;

else y = y - x;


THE PROBLEM WITH COMPOUND CONDITIONS

• Makes complex logic appear simpler than it really is

• Test cases may be omitted• Logic results in 3 paths, not 2!!

1

2 3

if (x<1){if (y>1) x=x+y; else y=y-x;}

else y=y-x;

1

2 5

3 4


THE PROBLEM WITH PATH COVERAGE

• Not all paths are feasible• No test case can force path [1,2,3,4,5], since

consecutive decisions mutually exclusive.

if (x<1)

y=2;

if (x >= 1)

y=3;

z=y;

1

2

4

3

5


Measuring Path Testing Difficulty

• McCabe metric -- logical code complexity • Formula: 1 + #decisions in control flow graph

• Test Significance: #basis paths through code

• Design use: complexity of code

• Test use: min #test cases for 100% path coverage

• McCabe measures test (development) difficulty


How To Design White Box Tests

• Test cases must execute different paths• Decision tables

• Rows -- elementary conditions in the code• Columns -- combinations of conditions in the code• Column based test case forces flow through

different logic paths in the code• Decision table built from code reflects what was

built versus intended (ie, from the spec)• Decision analysis for white-box testing.


WHITE-BOX TESTING TOOLS

• Tools instrument source code and gathers coverage data when tests • Compiled languages• Script languages -- coming to market

• Some provide test case design assistance• List of paths not covered• Data conditions to force branch/path• Graphic depiction of graph coverage


Problem P1Code (v1) & Decision Table

if (Age>80 || Weight>300) return 0; if (Age <= 12) return 1;if (Age > 65) return 2;if (Weight < 120) return 2else return 2+(Weight-120)/50;

Age>80 | Y N N N N N

Weight>300 | - Y N N N N

Age<=12 | Y N N N

Age>65 | Y N N

Weight<120 | Y N

Pills = | 0 0 1 2 2 C

------------------------------

Note: C: 2+(Weight/120)/50

McCabe = 6


Problem P1Code (v2) & Decision Table

if (Age>80 || Weight>300)return 0; if (Age <= 12) return 1;if (Age > 65 || (Age<=65 && Weight<120)) return 2;return 2+(Weight-120)/50;

Age>80 | Y N N N N N N

Weight>300 | - Y N N N N N

Age<=12 | Y N N N

Age>65 | Y N N

Weight<120 | - Y N Pills = | 0 0 0 1 2 2 C

McCabe = 7


Your Turn -- White-Box Testing(1) Construct Decision Table

pills=0;if (Age < 80 && Weight <300){ pills=1; if (Age >= 65) pills=2; else if (Age > 12) pills=2+(Weight-120)/50;}return pills;

____________ |

____________ |

____________ |

____________ |

____________ |

Pills = | - - - - - - -

---------------------------



Your Turn -- P1(2) Derive White-Box Test Cases

Case 1 2 3 4 5 6 7 8 9

Age ___ ___ ___ ___ ___ ___ ___ ___ ___

Weight ___ ___ ___ ___ ___ ___ ___ ___ ___

Pills ___ ___ ___ ___ ___ ___ ___ ___ ___


OBSERVATIONS -- WHITE-BOX TEST CASES

• Code may be incomplete with respect to input combinations from the specification

• Decision table constructed from code is simpler -- subset of black-box table

• Claim: black-box test cases from decision table force coverage of logic• Unless the code implements the wrong (a

different) function


MAIN POINTSWHITE-BOX TESTING

• White-box = logic testing• Limitation: can't tell what's missing• Don't forget exceptions -- throwing,

catching, propagating (debugger)• Perform decision analysis of code• Coverage tools help.• Use black-box test cases.


Agenda






PUTTING IT ALL TOGETHER

• Test design is a systematic process • whether you use decision tables or not, you

must understand the factors influencing and influenced by the behavior of the unit

• The sooner you design test cases, the better• Test design is more crucial than running tests• Reveals assumptions, omissions and errors

• Use test cases during design and coding inspections


PUTTING IT ALL TOGETHER

• It's okay to ask "What happens when …?

• Look for test design patterns • Expect certain type units to be tested in a

similar way• A way to identify best practices• Efficiency without cutting corners

• View test design as a "design product", not just a test product!!


Functional/Boundary Testing Worksheets

• Stored in Domino• Master template

• Stimulus-Response Analysis• Decision Table• Functional Test Cases• Boundary Test Design

• Boundary Point Analysis• Boundary Test Cases


SOLUTIONS TO EXERCISES


Your Turn (Dosage Problem)(2a) Identify Behaviors

CaseExpected

Stimulus Description #Pills

1

2

3

4

5

6

Any Age, Weight>300

Age > 80, any Weight

Age <= 12, Weight <= 300

Age > 65, Weight <= 300

Age 13-65, Weight 120-300

Age 13-65, Weight < 120

0

0

1

2

2+(W-120)/502


Your Turn(2b) Decision Table #1

c1: Age <= 12 | Y Y N N N

c2: Age > 65 | Y Y N N N

c3: Age > 80 | Y Y

c4: Weight > 300 | N Y N Y N Y N N Y


a1: Pills = 0 | X X X X X

a2: Pills = 1 | X

a3: Pills = 2 | X X

a4: Pills = 2+(W-120)/50 | X


Your Turn (2b) Decision Table #2

c1: Age <= 12 | Y N N

c2: Age > 65 | Y N N

c3: Age > 80 | Y

c4: Weight > 300 | Y N N N N


a1: Pills = 0 | X X

a2: Pills = 1 | X

a3: Pills = 2 | X X

a4: Pills = 2+(W-120)/50 | X


Your Turn(3a) Create Test Cases (testset1)

Case 1 2 3 4 5 6 7 8 9

Age 8 12 67 67 81 85 15 28 28

Weight 40 305 180 360 120 315 100 220 320

Pills 1 0 2 0 0 0 2 4 0

•Is more better?

•Every combination of stimuli is tested.

•Each column specifies at least one condition per variable


Your Turn(3b) Create Test Cases (testset2)

Case 1 2 3 4 5 6

Age 20 10 70 83 13 30

Weight 310 70 160 150 115 220

Pills 0 1 2 0 2 4

•Is fewer better?

•Not every combination of stimuli is tested. •Test case generation easier when each column specifies

at least one condition per variable


Your Turn(4) Write Test Script (testset1)

StepStimuli Pills=

Age

1

2

3

4

5

6

Weight StepStimuli Pills= Age

7

8

9

10

12

Weight

11

8

12

67

81

85

67

28

28

15

315

120360

180

305

40 100

220

320 0

4

2 1

0

2

0

0

0


Your Turn(2) Identify Boundary Conditions

• Condition 1 (bc1): Age <= 12

• Condition 2 (bc2): Age > 65

• Condition 3 (bc3): Age > 80

• Condition 4 (bc4): Weight > 300

• Condition 5 (bc5): Weight > 120


Your Turn(3) Identify Boundary Points

Boundary Condition Boundary PointAT IN

bc1: Age <= 12

OUT

bc2: Age > 65

bc3: Age > 80

bc4: Weight > 300

bc5: Weight > 120

12 11

66

81

301

121

13

65

80

300

120


Your Turn(4) Boundary-Nominal Test Cases

Condition btc BC Weight Age Expect

Weight>300 1 IN 301 21 0

2 OUT 300 21 5

Weight>120 3 IN 121 21 2

4 OUT 120 21 2

Age <= 12 5 AT 220 12 1

6 IN 220 11 1

7 OUT 220 13 4

Age>65 8 IN 220 66 2

9 OUT 220 65 4

Age>80 10 IN 220 81 0

11 OUT 220 80 2


Your Turn(4) Out-Out Boundary Test Cases

Condition OUT B-point

Weight Age btc Weight Age Expect

>300 <=12 12 300 13 5

>65 13 300 65 5

>80 14 300 80 5

>120 <=12 15 120 13 2

>65 16 120 65 2

>80 17 120 80 2


Your Turn -- White-Box Testing(1) Construct Decision Table

pills=0;if (Age < 80 && Weight <300){ pills=1; if (Age >= 65) pills=2; else if (Age > 12) pills=2+(Weight-120)/50;}return pills;

| Age < 80 | N - Y Y Y

| Weight<300 | - N Y Y Y

| Age >= 65 | N Y N

| Age > 12 | N - Y| Pills = | 0 0 1 2 C--------------



Your Turn(2) Derive White-Box Test Cases

Case 1 2 3 4 5

Age 83 24 11 68 35

Weight 97 310 85 225 108

Pills 0 0 1 2 2

| Age < 80 | N - Y Y Y

| Weight<300 | - N Y Y Y

| Age >= 65 | N Y N

| Age > 12 | N Y YPills = | 0 0 1 2 C

Documents

CEN 5070 - Software V & V Unit Testing Concepts