William E. Perry

Effective Methods for Software Testing

Third Edition

01_598376 ffirs.qxp 3/3/06 11:04 PM Page iii

File AttachmentC1.jpg

01_598376 ffirs.qxp 3/3/06 11:04 PM Page ii

Effective Methods for Software Testing

Third Edition

01_598376 ffirs.qxp 3/3/06 11:04 PM Page i

01_598376 ffirs.qxp 3/3/06 11:04 PM Page ii

William E. Perry

Effective Methods for Software Testing

Third Edition

01_598376 ffirs.qxp 3/3/06 11:04 PM Page iii

Effective Methods for Software Testing, Third EditionPublished byWiley Publishing, Inc.10475 Crosspoint BoulevardIndianapolis, IN 46256www.wiley.com

Copyright 2006 by Wiley Publishing, Inc., Indianapolis, Indiana

Published simultaneously in Canada

ISBN-13: 978-0-7645-9837-1ISBN-10: 0-7645-9837-6

Manufactured in the United States of America

10 9 8 7 6 5 4 3 2 1

3MA/QV/QU/QW/IN

No part of this publication may be reproduced, stored in a retrieval system or transmittedin any form or by any means, electronic, mechanical, photocopying, recording, scanning orotherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copy-right Act, without either the prior written permission of the Publisher, or authorizationthrough payment of the appropriate per-copy fee to the Copyright Clearance Center, 222Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 646-8600. Requests to thePublisher for permission should be addressed to the Legal Department, Wiley Publishing,Inc., 10475 Crosspoint Blvd., Indianapolis, IN 46256, (317) 572-3447, fax (317) 572-4355, oronline at http://www.wiley.com/go/permissions.

Limit of Liability/Disclaimer of Warranty: The publisher and the author make no repre-sentations or warranties with respect to the accuracy or completeness of the contents of thiswork and specifically disclaim all warranties, including without limitation warranties of fit-ness for a particular purpose. No warranty may be created or extended by sales or promo-tional materials. The advice and strategies contained herein may not be suitable for everysituation. This work is sold with the understanding that the publisher is not engaged in ren-dering legal, accounting, or other professional services. If professional assistance isrequired, the services of a competent professional person should be sought. Neither thepublisher nor the author shall be liable for damages arising herefrom. The fact that an orga-nization or Website is referred to in this work as a citation and/or a potential source of fur-ther information does not mean that the author or the publisher endorses the informationthe organization or Website may provide or recommendations it may make. Further, read-ers should be aware that Internet Websites listed in this work may have changed or disap-peared between when this work was written and when it is read.

For general information on our other products and services or to obtain technical support,please contact our Customer Care Department within the U.S. at (800) 762-2974, outside theU.S. at (317) 572-3993 or fax (317) 572-4002.

Library of Congress Control Number: 2005036216

Trademarks: Wiley and related trade dress are registered trademarks of Wiley Publishing,Inc., in the United States and other countries, and may not be used without written permis-sion. All other trademarks are the property of their respective owners. Wiley Publishing,Inc., is not associated with any product or vendor mentioned in this book.

Wiley also publishes its books in a variety of electronic formats. Some content that appearsin print may not be available in electronic books.

01_598376 ffirs.qxp 3/3/06 11:04 PM Page iv

www.wiley.com

This book is dedicated to my wife Cynthia, who for many years has beentesting my ability to live in accordance with our marriage vows. Shetaught me that testing is a lifelong process, that testing is necessary to

ensure that you are meeting your objectives, and that testing can be funif it is performed correctly. Thank you, Cynthia. What you have taught me

is incorporated into many of the concepts in this book.

01_598376 ffirs.qxp 3/3/06 11:04 PM Page v

William E. Perry holds degrees from Clarkson University, University of Rochester, andRochester Institute of Technology. Bill also holds the following professional certifica-tions: CPA (Certified Public Accountant), CIA (Certified Internal Auditor), CISA (Cer-tified Information Services Auditor), CSQA (Certified Software Quality Analyst), andCSTE (Certified Software Tester). He has been an examiner for the Malcolm BaldrigeNational Quality Award, and served on standards committees for NIST (NationalInstitute of Standards and Technology), IEEE (Institute of Electrical and ElectronicsEngineers), AICPA (American Institute of Certified Public Accountants) and ISACA(Information Systems Audit and Control Association).

In 1980, Bill founded the Quality Assurance Institute (QAI), a professional associa-tion for testers. QAI offers professional certification for Quality Assurance, SoftwareTesting, Software Project Leaders and Business Analyst Professional. More than 27,000individuals have been certified since the inception of the program.

Bill has authored more than 50 books, many published by John Wiley & Sons. Herecently founded the Internal Control Institute (ICI). ICI and St. Petersburg Collegerecently formed the Internal Control Center of Excellence to share best internal controlpractices, hold conferences on emerging internal control practices, and to offer e-learningcourses and a professional certification in internal control.

About the Author

vi

01_598376 ffirs.qxp 3/3/06 11:04 PM Page vi

Credits

vii

Executive EditorRobert Elliott

Production EditorFelicia Robinson

Editorial ManagerMary Beth Wakefield

Production ManagerTim Tate

Vice President and Executive GroupPublisherRichard Swadley

Vice President and Executive PublisherJoseph B. Wikert

Project CoordinatorMichael Kruzil

Graphics and Production SpecialistsCarrie FosterMary J. GillotLauren GoddardDenny HagerJoyce HaughyeyStephanie D. JumperRashell Smith

Quality Control TechniciansJohn GreenoughBrian H. Walls

Proofreading and IndexingTechbooks

01_598376 ffirs.qxp 3/3/06 11:04 PM Page vii

01_598376 ffirs.qxp 3/3/06 11:04 PM Page viii

Introduction xxv

Part I Assessing Testing Capabilities and Competencies 1

Chapter 1 Assessing Capabilities, Staff Competency, and User Satisfaction 3The Three-Step Process to Becoming a World-Class Testing

Organization 3Step 1: Define a World-Class Software Testing Model 5

Customizing the World-Class Model for Your Organization 7Step 2: Develop Baselines for Your Organization 8

Assessment 1: Assessing the Test Environment 8Implementation Procedures 9Verifying the Assessment 13

Assessment 2: Assessing the Capabilities of Your Existing Test Processes 13

Assessment 3: Assessing the Competency of Your Testers 14Implementation Procedures 14Verifying the Assessment 16

Step 3: Develop an Improvement Plan 16Summary 18

Part II Building a Software Testing Environment 35

Chapter 2 Creating an Environment Supportive of Software Testing 37Minimizing Risks 38

Risk Appetite for Software Quality 38Risks Associated with Implementing Specifications 39

Faulty Software Design 39Data Problems 39

Contents

ix

02_598376 ftoc.qxp 3/3/06 11:05 PM Page ix

Risks Associated with Not Meeting Customer Needs 40Developing a Role for Software Testers 43

Writing a Policy for Software Testing 45Criteria for a Testing Policy 45Methods for Establishing a Testing Policy 46

Economics of Testing 47TestingAn Organizational Issue 50Management Support for Software Testing 50Building a Structured Approach to Software Testing 51

Requirements 54Design 54Program 55Test 55Installation 55Maintenance 55

Developing a Test Strategy 56Use Work Paper 2-1 58Use Work Paper 2-2 58

Summary 60

Chapter 3 Building the Software Testing Process 63Software Testing Guidelines 63

Guideline #1: Testing Should Reduce Software Development Risk 64

Guideline #2: Testing Should Be Performed Effectively 65Guideline #3: Testing Should Uncover Defects 65

Defects Versus Failures 65Why Are Defects Hard to Find? 66

Guideline #4: Testing Should Be Performed Using Business Logic 67

Guideline #5: Testing Should Occur Throughout the Development Life Cycle 68

Guideline #6: Testing Should Test Both Function and Structure 69Why Use Both Testing Methods? 69Structural and Functional Tests Using Verification and

Validation Techniques 69Workbench Concept 71

Testing That Parallels the Software Development Process 72Customizing the Software-Testing Process 74

Determining the Test Strategy Objectives 74Determining the Type of Development Project 75Determining the Type of Software System 76Determining the Project Scope 77Identifying the Software Risks 77Determining When Testing Should Occur 79Defining the System Test Plan Standard 79

x Contents

02_598376 ftoc.qxp 3/3/06 11:05 PM Page x

Defining the Unit Test Plan Standard 83Converting Testing Strategy to Testing Tactics 83

Process Preparation Checklist 86Summary 86

Chapter 4 Selecting and Installing Software Testing Tools 103Integrating Tools into the Testers Work Processes 103Tools Available for Testing Software 104Selecting and Using Test Tools 108

Matching the Tool to Its Use 109Selecting a Tool Appropriate to Its Life Cycle Phase 109Matching the Tool to the Testers Skill Level 111Selecting an Affordable Tool 114

Training Testers in Tool Usage 116Appointing Tool Managers 117

Prerequisites to Creating a Tool Manager Position 118Selecting a Tool Manager 118Assigning the Tool Manager Duties 119Limiting the Tool Managers Tenure 120

Summary 120

Chapter 5 Building Software Tester Competency 125What Is a Common Body of Knowledge? 125Who Is Responsible for the Software Testers Competency? 126How Is Personal Competency Used in Job Performance? 126

Using the 2006 CSTE CBOK 127Developing a Training Curriculum 128

Using the CBOK to Build an Effective Testing Team 129Summary 131

Part III The Seven-Step Testing Process 151

Chapter 6 Overview of the Software Testing Process 153Advantages of Following a Process 153The Cost of Computer Testing 154

Quantifying the Cost of Removing Defects 155Reducing the Cost of Testing 156

The Seven-Step Software Testing Process 156Objectives of the Seven-Step Process 159Customizing the Seven-Step Process 160Managing the Seven-Step Process 161Using the Testers Workbench with the Seven-Step Process 162

Workbench Skills 163Summary 164

Chapter 7 Step 1: Organizing for Testing 165Objective 165Workbench 166Input 167

Contents xi

02_598376 ftoc.qxp 3/3/06 11:05 PM Page xi

Do Procedures 167Task 1: Appoint the Test Manager 167Task 2: Define the Scope of Testing 168Task 3: Appoint the Test Team 168

Internal Team Approach 169External Team Approach 170Non-IT Team Approach 170Combination Team Approach 170

Task 4: Verify the Development Documentation 171Development Phases 171Measuring Project Documentation Needs 174Determining What Documents Must Be Produced 175Determining the Completeness of Individual Documents 179Determining Documentation Timeliness 180

Task 5: Validate the Test Estimate and Project Status Reporting Process 181

Validating the Test Estimate 182Testing the Validity of the Software Cost Estimate 185Calculating the Project Status Using a Point System 189

Check Procedures 200Output 200Summary 200

Chapter 8 Step 2: Developing the Test Plan 209Overview 209Objective 210Concerns 210Workbench 211Input 212Do Procedures 212

Task 1: Profile the Software Project 212Conducting a Walkthrough of the Customer/User Area 212Developing a Profile of the Software Project 213

Task 2: Understand the Project Risks 215Task 3: Select a Testing Technique 222

Structural System Testing Techniques 223Functional System Testing Techniques 229

Task 4: Plan Unit Testing and Analysis 235Functional Testing and Analysis 236Structural Testing and Analysis 238Error-Oriented Testing and Analysis 240Managerial Aspects of Unit Testing and Analysis 243

Task 5: Build the Test Plan 244Setting Test Objectives 245Developing a Test Matrix 245Defining Test Administration 250Writing the Test Plan 251

xii Contents

02_598376 ftoc.qxp 3/3/06 11:05 PM Page xii

Task 6: Inspect the Test Plan 254Inspection Concerns 255Products/Deliverables to Inspect 256Formal Inspection Roles 256Formal Inspection Defect Classification 258Inspection Procedures 259

Check Procedures 262Output 262Guidelines 262Summary 263

Chapter 9 Step 3: Verification Testing 291Overview 292Objective 293Concerns 294Workbench 294Input 296

The Requirements Phase 296The Design Phase 296The Programming Phase 297

Do Procedures 298Task 1: Test During the Requirements Phase 298

Requirements Phase Test Factors 299Preparing a Risk Matrix 302Performing a Test Factor Analysis 310Conducting a Requirements Walkthrough 312Performing Requirements Tracing 314Ensuring Requirements Are Testable 315

Task 2: Test During the Design Phase 316Scoring Success Factors 316Analyzing Test Factors 318Conducting a Design Review 320Inspecting Design Deliverables 322

Task 3: Test During the Programming Phase 323Desk Debugging the Program 325Performing Programming Phase Test Factor Analysis 326Conducting a Peer Review 328

Check Procedures 330Output 331Guidelines 331Summary 332

Chapter 10 Step 4: Validation Testing 409Overview 409Objective 410Concerns 410Workbench 410Input 411

Contents xiii

02_598376 ftoc.qxp 3/3/06 11:05 PM Page xiii