Testing – Why We Do It Badly2

Session 2Session 2 11

Testing – Why We Do Testing – Why We Do It BadlyIt Badly

Alan E. Devney, Jr. PMPAlan E. Devney, Jr. PMP

FusionSoft LLCFusionSoft LLC


Quickly About me!Quickly About me!

Senior Project Manager – Just like Senior Project Manager – Just like most of you!most of you!

Notable customers: Walt Disney Notable customers: Walt Disney World Company, Sega, Duke World Company, Sega, Duke Energy.Energy.

Currently a Project Manager at Currently a Project Manager at FusionSoft LLC.FusionSoft LLC.


If I do not test, I can If I do not test, I can still believe that it still believe that it works!works!


Project Management Project Management ProverbsProverbs

Project Management is just Project Management is just management with a fancy name.management with a fancy name.

We do not know what the We do not know what the expectations are but we are expectations are but we are pretty sure you are not meeting pretty sure you are not meeting them.them.


The Final FrontierThe Final Frontier

Testing is the Last Element of a Testing is the Last Element of a Project’s Success that is still being Project’s Success that is still being defined at its core. defined at its core.

Terms are inconsistent in the Terms are inconsistent in the industry. Is it a tomato or a tamato?industry. Is it a tomato or a tamato?

A large portion of Project A large portion of Project Professionals are still learning how Professionals are still learning how and what it means to test effectively.and what it means to test effectively.


PeoplePeople


Sponsor SupportSponsor Support

We spent so much time on We spent so much time on requirements, design, use cases, requirements, design, use cases, test cases, pillow cases…why do test cases, pillow cases…why do we need to do so much testing?we need to do so much testing?

‘‘We told you what we wanted, We told you what we wanted, didn’t you use our requirements didn’t you use our requirements to build what we asked?’to build what we asked?’


IT Management IT Management SupportSupport We have not done it for the last 20 We have not done it for the last 20

years why start now?years why start now? Lack of understanding of new Lack of understanding of new

technology.technology. Production Support can clean it up.Production Support can clean it up. We must be on time, on scope, and We must be on time, on scope, and

on budget for me to get my bonus.on budget for me to get my bonus.


Lack of Core Business Lack of Core Business CommitmentCommitment Business Users have a full time job Business Users have a full time job

already.already. Business Users do not see the need for Business Users do not see the need for

the new system.the new system. Business Users do not want to test at a Business Users do not want to test at a

detailed level.detailed level. Implementing new systems is not their Implementing new systems is not their

job!job! I am not really sure what we do here!I am not really sure what we do here!


ProcessProcess


Declining Time to TestDeclining Time to Test

Where are the areas that can be Where are the areas that can be reduced with the least amount of Risk?reduced with the least amount of Risk?

Communication of overall Risk Communication of overall Risk associated with cutting testing corners.associated with cutting testing corners.

Communication with Production Support Communication with Production Support Management.Management.

Ultimately it is a Business decision to go Ultimately it is a Business decision to go live. live.


Lack of Detailed Lack of Detailed RequirementsRequirements If we can not say what we are If we can not say what we are

building we can not clearly define building we can not clearly define how to test it.how to test it.

This allows us to test and believe we This allows us to test and believe we are OK because we did not know are OK because we did not know what we were building in the first what we were building in the first place.place.

The most quoted reason for project The most quoted reason for project failure. failure.


Lack of Proper ToolsLack of Proper Tools

Testing 100,000 lines of test scripts Testing 100,000 lines of test scripts can not be managed using Excel.can not be managed using Excel.

The right tool for the right job. Small The right tool for the right job. Small projects small process, larger projects small process, larger projects require greater addition to projects require greater addition to detail.detail.

Suggestion: Mercury Quality CenterSuggestion: Mercury Quality Center


Testing Role DefinitionTesting Role Definition

Who is doing what?Who is doing what? Are they the right person, Are they the right person,

discipline for the job?discipline for the job? How does my business align with How does my business align with

your business?your business? Estimating level of effort. Can one Estimating level of effort. Can one

person run 30,000 lines to test person run 30,000 lines to test scripts in a month?scripts in a month?


Lack of MeasurementsLack of Measurements Where are we anyway?Where are we anyway? How will senior managers know where the How will senior managers know where the

project stands?project stands? How will the PM know where the project How will the PM know where the project

stands?stands? Does not need to be complex just clear.Does not need to be complex just clear. ExamplesExamples

– What is a Critical Defect?What is a Critical Defect?– Number of ScriptsNumber of Scripts– Estimated Time to Test Estimated Time to Test – Estimated DollarsEstimated Dollars


Lack of Defect Lack of Defect Management ProcessManagement Process How are defects managed?How are defects managed? Does the defect need to be Does the defect need to be

addressed?addressed? Who corrects the defect?Who corrects the defect? Who re-tests the defect?Who re-tests the defect? Who closes the defect?Who closes the defect? SOX?SOX?


Doing it Doing it Right!Right!


Plan!!!Plan!!!


How to Plan Like a Tester

• Mirror image of production environment hardware, protocols, configuration, etc Mirror image of production environment hardware, protocols, configuration, etc.

• Existing & New Infrastructure Requirements• Other components that need to run concurrently

during test• Business Scenarios• Server profiles• Workloads• Data Population• Metrics -Decisions about what is acceptable or

“good enough”


360 Degree Testing360 Degree Testing

TechnologyTechnology Business ProcessBusiness Process ApplicationApplication DataData LocationLocation OrganizationOrganization

Computer Science Corporation – Six Domains of ChangeComputer Science Corporation – Six Domains of Change


What Makes Good What Makes Good TestingTesting Where does Testing Begin?Where does Testing Begin?

– Scope DefinitionScope Definition– Business Process DefinitionBusiness Process Definition– Requirements DefinitionRequirements Definition– DesignDesign– Test Cases/ScriptsTest Cases/Scripts– Acceptance Criteria- Get it in Acceptance Criteria- Get it in

Writing!Writing!


What solid testing is…

Getting acceptable “good enough" metrics (measurements) for:– Correctness of function: is the basic

workflow working and flowing?– Data integrity: Is the right data where it

should be?– User Interface: Security, Likeable, Good

error messages– Documentation: End user and administrator– Environmental Factors: Network traffic,

server busy times, patches– Usage (load): When and how much? Where?


Acceptance Criteria Acceptance Criteria define, as

briefly as possible, the rules the client will use to determine whether the work has been successfully completed.

Know what “Done” looks like specifically. This ties directly to “Measurements.”


UAT Facilitated UAT Facilitated TestingTesting Super Users working directly with Super Users working directly with

Business Testers.Business Testers. Mixes Training and TestingMixes Training and Testing Keeps testing moving! Keeps testing moving!


Good Testing Practice Good Testing Practice Should:Should: Ensure that tests are being planned Ensure that tests are being planned

to cover all relevant aspects of an to cover all relevant aspects of an equipment or computer system, and equipment or computer system, and that the test is being executed and that the test is being executed and documented well enough to enable documented well enough to enable tracing of the test, the test results, tracing of the test, the test results, the handling of deviations and the the handling of deviations and the responsible persons for each responsible persons for each activity. activity. Source: The Good Automated Manufacturing Practice Forum (GAMP 3)


Effective testing is Effective testing is essential for achieving essential for achieving successful business successful business outcomes.outcomes.


QuestionsQuestions