Test Management for Busy People

TD Half-day Tutorials

5/6/2014 8:30:00 AM

Test Management for Busy

People

Presented by:

Lloyd Roden

Lloyd Roden Consultancy

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Lloyd Roden Lloyd Roden Consultancy

With more than twenty-eight years in the software industry, Lloyd Roden has worked as a developer, test analyst, and test manager for many different organizations. Lloyd was a consultant/partner with Grove Consultants for twelve years. In 2011 he created Lloyd Roden Consultancy, an independent UK-based training and consultancy company specializing in software testing. Lloyd’s passion is to enthuse, excite, and inspire people in the area of software testing. He has spoken at conferences worldwide including STAREAST, STARWEST, Better Software, EuroSTAR, AsiaSTAR, and Special Interest Groups in software testing in several countries. In 2004, he won the European Testing Excellence award.

Test Management for Busy People

A half-day workshop

Written by Lloyd Roden www.lloydrodenconsultancy.com Version 1_0 © Lloyd Roden

LRC110815

Copyright Notice

It is strictly prohibited to reproduce any of these materials in any form without prior written permission from Lloyd Roden Consultancy. This includes photocopying, scanning and the use of any recording devices.

© Lloyd Roden TMBP140308

Test Management for Busy People Workshop-1

Test Management for Busy People

1

Contents

Introduction

Doing more with less

Building an elite test team

Does it matter – does it matter much

2



Busyness is both good and bad

!   busy good !   work is exciting, challenging and enjoyable !   the day goes quickly !   job satisfaction

!   busy bad !   too much work to cope with !   finding stress levels are constantly high !   family suffer

are you currently “busy good” or “busy bad”?

3

Modern life wants more time…but we have less time

4



Contents

Introduction




5

#1: Challenge redundant documentation

!   are these test documents really needed? !   test plans !   written test reports !   test logs !   detailed test scripts !   test strategy !   test retrospective

!   are these documents used or read? !   how do you know? !   what feedback do you receive? !   are they kept up to date?

if you produce documents that are not used or considered worthwhile then don’t produce them

you have 86,000

seconds in a day How will you spend them?

Don’t waste them?

6



#2: Eliminate waste

!   concept of eliminating waste is good !   examples

!   meetings !   emails !   duplication of effort !   documents not being used !   maintenance of the tests taking longer

than the test execution !   however…

!   sometimes cutting back too much causes more problems

7

Wastage example 1: meetings

10 people @ 1 hour meeting = 10 hours •  1 person arrives 6 minutes late……………………………… •  4 people receive texts and respond taking 2 minutes…….. •  1 person gets a call and leaves room for 15 minutes……... •  6 people on laptops responding to emails…………………..

….1 hour ….80 minutes …. 25 minutes …. 3 hours

total: 6 hours wasted = 60% inefficiency suggestion: conduct meetings with etiquette

!   no laptops !   no phones !   no interruptions !   start on time

8



Wastage example 2: bugs raised in error (or re-raised)

Percentage defects re-raised

Version A

TOTAL 31/96 = 33%

Percentage defects raised in error

Version A

TOTAL 23/96 = 24%

N.B: figures are based on 1 day to raise and investigate. And a cost to the company of 250€ per day

31 days

23 days

total wastage

13500€

54 days

(6% of the project)

9

#3: Don’t count test cases…

…without knowing the quality of them ! !   quantity without quality is meaningless

!   example, in the last hour of testing: !   tester 1 has run 30 test cases !   tester 2 has run 300 test cases !   tester 3 has run 2 test cases !   tester 4 has run 0 test cases

!   know the quality of the tests !   test the tests regularly !   review the tests !   check on coverage

the only number with meaning is

zero

10



#4: Provide some slack

!   maximum velocity does not equal maximum productivity August 2010: productivity in the United States unexpectedly decreased in the second quarter

after employers expanded the workweek by the most in four years. Employees output

decreased by 0.9% per hour of output

7

4

5 2

6 1

8 3

9

efficiency is improved by 11.1% !

BUT we cannot play the game… productivity is reduced…to zero "

source: Tom DeMarco, “Slack”

7

4

5 2

6 1

8 3

11.1% spare capacity Car engines work at

their optimum efficiency (mpg) at 56mph

Management’s decision:

to fill spare capacity to increase productivity

Productivity declines unexpectedly as US workweek lengthens

11

People are not a fungible resource

money is fungible:

net result: still have £100

but people are not fungible:

net result: at least 30% loss of productivity “on average there will be a net loss of 15% productivity due to task switching”

bank 1

£100

bank 1

£30

bank 2

£50

bank 3

£20

100% on project A

30% on project A

50% on project B

20% on project C

therefore be careful of the saying “women can multi-task and men can’t”

decide to divide money into 3 banks

management decide to divide your time into 3

projects

12



Case study: Personal Assistants – what a waste!

!   remember PAs: the manager’s ultimate resource

from this assigned to one manager available all the time able to do anything anytime cheaper resource BUT only 60% productive

net result: was not able to photocopy document for manager so they had to

photocopy 30 copies…and this is efficient and cost effective?

to this reduce PAs by 40% shared resource 100% productive

13

Companies who have introduced “slack”

Gore 10% dabble time 3M 15% think time Google 20% anytime Atlessian 20% Fedex days

14



when would an electric paintbrush be beneficial? when would a manual paintbrush be beneficial?

manual versus automated

but using both

requires skill and manual

intervention

#5: Tools are to “support” testing

15

Use tools wisely

!   tools can help increase productivity and staff morale !   using tools requires new skills !   tools can automate tedious/boring tasks !   tools are less likely to make mistakes !   some tools can find bugs quicker !   tools can help with testing that cannot be achieved

manually !   tools can decrease productivity and staff morale

!   tools do not make good testers !   too much automation can lead to poor testers !   a fool with a tool is still a fool !   automating rubbish produces faster rubbish !   tools are software and have bugs in them

16



#6: Take control of your day

!   learn to say “yes” to yourself occasionally !   saying “yes” to others ultimately means saying “no” to you

!   don’t allow others people to control your day all the time !   this leads to frustration

!   learn to control unplanned activities !   think before reacting

!   set yourself daily goals and achieve them !   John, Susan, Nora and George

17

#7: Less scripted more exploratory testing

a problem with scripted testing A

B 18



Dangers of detailed scripts

!   reasons to write detailed scripts !   describes detailed tests for inexperienced testers !   automation !   provide an audit trail

!   risks !   boring to write…boring to run !   more time spent writing than running tests !   maintenance nightmare !   false confidence in the quality of the

software !   your best testers will probably leave

suggestion: introduce exploratory testing sessions and limit the number of detailed scripts produced

19

!   tests are prioritised against the product risks

!   highest priority risks are mitigated first !   testers will use appropriate tools and

techniques to mitigate the risks efficiently

!   re-prioritisation of the tests will be initiated by the re-prioritisation of the risks

!   product risks will be monitored and reported to stakeholders on a regular basis

#8: Risk based testing is a must

20



Determining the “cut off”

Product Risk Probability Impact Exposure Risk Value Poor response times High High 9 90 Security violation High High 9 90 Tax calculation incorrect Medium High 8 80 Holiday pay not included High Medium 7 70 Bank Transfer interface problems High Medium 7 70 User interface may not be liked Low High 6 30 New employees not paid Medium Medium 5 25 Unable to delete accounts Low Medium 3 3 Reports are unaligned Medium Low 2 2

provides a priority list and a focus for testing

21

#9: Personal daily retrospectives

!   take time to reflect !   15 minutes at the end of the day

!   what did you do well !   what could you improve on !   what did people appreciate

!   take time to change !   to be the best you can be !   improvement starts with “I”

!   take time to learn !   you are never to old !

true nobility is not being better than anyone else but being better than you used to be

22



#10: When “speed up” really means “slow down”

!   this is only efficient if the workload remains constant

!   if priorities change or workloads increase this might cause unpredicted consequences:

me

John

Sue

Amy

flow of work

e.g. John needs information from me urgently, therefore my workload/priorities change to fulfill the expectations. The net effect is that Sue becomes less busy, but she knows the “hurry up” mantra which to her means “stay busy”, so is Sue is becoming less busy, how does she show others that she is busy?

Sue slows down!! 23

Contents

Introduction




24



#11: Create an “elite” team of testers

!   a team that knows, understands and can apply… !   systematic test design techniques !   exploratory testing !   bug hunting !   flexibility !   creativity

!   a team that will want to !   collaborate !   increase their knowledge and expertise !   adapt to the circumstance

lead by example, coach, mentor and work with your testers

a team whose “whole” is greater than the sum of

its parts

25

#12: Use the Tester’s style analysis questionnaire

things to note !   there is no right or wrong answer !   try not to think too much !   it helps us to assess our strengths

!   can be used with other psychometric questionnaires (e.g. Belbin, Myers/Briggs, 16PF)

!   two parts !   answer questionnaire and !   plot the values on the graph

26



The questionnaire - how to complete (x axis)

Friendly Approachable Casual Open Unstructured Social Intuitive Random Warm Perceptive

Formal Retiring Business Like Guarded Organised Introvert Logical Focused Cool Insensitive

X - Axis

x

x x x

x x

x x

x x

627

The questionnaire - how to complete (y axis)

To the point Challenging Quick Insistent Lively Impatient Adventurous Confronting Competitive Strong Minded

Indirect Accepting Leisurely Thoughtful Relaxed Patient Cautious Receptive Co-operative Analytical

Y - Axis

x

x x

x x

x x x

x x

228



Plot your score on the grid

X (6,2)

29

The Model

The Pioneer

The Facilitator

The Pragmatist

The Analyst

30



#13: Test the testers/tests

testing is quality control on development

BUT…how good are your tester/tests? Who is performing quality control on the testers?

developers bug seed, plant 100 bugs and see how many are found

provide an application for new testers with known bugs and see how good they are at finding them

we deserve the best…mediocracy is not an option 31

Areas where this idea can be used

!   testing your regression tests !   manual or automated !   discover how effective they are

!   testing the outsourced teams !   let’s wake up! !   discover whether they are cost effective

!   testing new testers wanting to join the team !   aim to recruit the best !   quality gate into testing

32



#14: learn to be a weather person

weather people… !   predict what is likely to happen using

trends and forecasting !   understand statistics and probability !   use historical data to substantiate

their synopsis !   understand correlation between

systems !   are becoming more accurate !   don’t always get it right

!   not always liked…but it is not their fault!

!   but we still use them to plan and implement our activities

however…

33

Learn to predict the “projects” weather

!   to predict what is likely to happen at an early stage

!   to use basic statistics and probability

!   to gather and analyse historical data

!   understand the correlation between graphs

!   to become more accurate

BUT: you wont always get it right, you wont always be liked, but it’s not your fault

0

50

100

150

200

250

1 2 3 4 5 6 7 8 9 10

0

10

20

30

40

50

1 2 3 4 5 6 7 8 9 10

34



#15: Tailor your test documentation…. and use modern technology

don’t be boring make it exclusive

challenge quality and quantity of what is produced exploit modern

technology

35

Make it relevant and make it personal

!   providing information is a major aspect of testing and must be… !   accurate !   timely !   predictive !   factual !   objective !   personal !   relevant

CEO’s Dashboard

Project Manager’s Dashboard

One size doesn’t fit all 36



#16: Understand data correlation before reporting

…plus basic data significance example 1: defects found

example 2: estimated versus example 3 effectiveness of testing actual quality

good or bad?

what about test effectiveness after 1 month of live Project 1: DDP = 85% Project 2 : DDP = 60% Project 3 : DDP = 90% Project 4 : DDP = 10% conclusion: reward team 3, punish team 4? why don’t we have a look at the numbers

good or bad?

37

#17: Learn the tools of the trade

…and become skilled at what you do !   test techniques

!   state transition, pairwise, domain analysis etc. !   the system

!   application, architecture etc. !   test tools

!   know what is available !   practice makes perfect

!   test on a regular basis 38



#18: Stand out and be different

what have these stores got in common?

what have these stores got in common?

39

Create a unique selling point…

!   what makes you/your team different from others… !   that are cheaper? !   that are faster?

!   your USP is so that you thrive…not just survive !   example USPs

!   recognising and eliminating waste !   being able to find those “nasty” bugs quickly !   measuring and reporting the value of testing !   having a positive attitude at work !   collaborating always !   constantly improving efficiency !   estimating quality… 40



#19: Apprentice train

!   benefits for the apprentice !   learn from others

!   hear (forget) !   see (remember) !   do (understand)

!   fast track !   learning curve is shorter

!   benefits for the mentor !   rewarding to see progress !   improve ourselves

!   learn from the apprentice !   invest time in people

!   greatest contribution we can make

41

Let’s reminisce…

!   think back to the first time you were apprentice trained !   cooking with your mum? !   home improvement with your dad? !   washing up (or nowadays - loading

the dishwasher!) !   what type of experience was it?

!   good/bad/exciting/non-event

42



what they are !   opportunity to learn and

increase skills !   a safe environment to

make mistakes !   a framework to prove

yourself worthy of the current/next job role

!   an environment to provide constructive feedback

!   to be taken seriously

what they are not !   errand running !   a career trap " !   a guarantee !   an excuse to make lots

of mistakes !   a training course/

certification scheme, all theory and no practice

Apprenticeships: what they are/are not

43

Chief Tester one person is responsible for mentoring all new staff

Hierarchical

you mentor subordinates and you are mentored by your manager

Buddy pairing testers to mentor each other, accountability is encouraged

Coterie

everyone mentors everyone!

Various models to consider

44



“An excellent tester this one is”

!   a “jedi” tester will… !   let themselves be distracted by anomalies

and new ideas !   encourage variability, avoid repeating the

same test twice !   use their confusion as a resource !   know their limits and get to know/use

subject matter experts !   adapt testing to fit the context !   develop resources and tools !   train their minds to be curious, cautious

and critical !   take notes and report results in a useful

and concise way 45

#20: Take time to sharpen your axe

!   don’t procrastinate !   create tools and utilities to make testing more efficient !   share your ideas with others !   understand new methodologies and technologies and how

they will affect you (keeping one step ahead) !   measure your own effectiveness and improve !   create ‘me’ time (kick the leaves) !   try something new !   become a champion…

complacency is only one step ahead of

apathy make it your mission to become the

best at what you do

46



Contents

Introduction




47

#21: Choose your battles

!   you can’t fight every battle… so what is important? !   career progression for testers !   remuneration !   providing slack !   a good working environment !   fairness !   more resources !   advice given is used !   purchasing test tools !   efficiency !   a good test environment !   test estimates being agreed

48



#22: Don’t worry about test estimation…too much

!   understanding the problem with test estimation ! recognising that we can test in less !   provide options using a variety of techniques

Testing 4 weeks

Testing Testing Testing 2 weeks 1 week

49

#23: Estimating quality is a must

fact: estimation is based on 2 aspects

effort + people = schedule half truth for testing:

effort + people + quality = schedule we must also estimate the number and severity of the bugs we are going to find to make our estimates meaningful (a direct correlation)

WARNING: be careful of Brook’s law – 9 women cannot have a baby in 1 month! Some activities cannot be accelerated by adding more resource

50



Estimate, monitor and report quality

how to estimate quality !   guess !   use historical data (previous projects, defect density) !   use models

what to monitor and report !   actual versus expected !   trend analysis and forecasting !   correlation between graphs

0

10

20

30

40

50

1 2 3 4 5 6 7 8 9 10 0

50

100

150

200

250

1 2 3 4 5 6 7 8 9 10

Cycle #

Actual bugs

Masked rate

Observed bugs

Fix Rate

Fixed bugs

Insert rate

Inserted bugs

Remaining bugs

1 10 30% 7 80% 6 20% 1 5

2 5 30% 4 80% 3 20% 1 3

3 3 0% 3 100% 3 0% 0 0

4 0 full regression test

51

#24: Remuneration and career path

!   remuneration must represent their skill ! recognising that testing is a skill and profession !   fairness throughout the organisation

!   define a career structure for testers !   where management is not the only option !   encourages testers to aspire in their career !   essential to recruit and retain good quality testers

functional career model

management technical application

functional non-functional test consultant

test manager

team leaders

senior technician

DBA/env technician

test automator

designer designer

executor executor 52



#25: Demonstrate the ROI for testing

requirements team… has an idea and forms requirements design team… takes (vague) requirements and designs the product development team… uses the design to develop the product test team…? if testing doesn’t provide a good ROI… then it is not a good investment

YES NO

ROI

!   all teams must provide the company with a good ROI, the problem is perception…

ROI must be measured in quantifiable

terms (CSPB,

DDP, DFP, PRDFP etc.)

53

#26: Provide management with feedback on decisions that they make

!   example: 4 weeks of testing:

!   new data obtained:

0

20

40

60

80

100

120

0 1 2 3 4

Number of bugs

Actual

Management want to implement the software, but you suggest a

further week of exploratory testing and a final week of regression

testing due to the high number of bugs that have been found

0

20

40

60

80

100

120

140

0 1 2 3 4 5 6

Number of bugs

Actual

25 new bugs found and fixed…full regression pack run

54



Using data to report back

#  cost of bugs to find and fix in test = 1000€ #  cost of bugs to find and fix in live = 3000€ #  1st month of live operation customers find 10 bugs !   if the old date (4 weeks) was used

!   assume 35 bugs found by customer !   DDP = 100/135 = 74% !   cost of fixing = 35 * 3000€ = 105,000€

!   if the new date (6 weeks) was used !   DDP = 125/135 = 93% !   cost of fixing = 10 * 3000€ = 30,000€

+ 25 * 1000€ = 25,000€ = 55,000€

cost saving 50,000€

DDP increase 19% Therefore a good

decision by management !

55

#27: Make your test environment the strongest link !   often the test environment becomes the

weakest link !   concentrate too much on test cases and

test scripts !   reliant on other people !   often difficult to set up well !   we believe the environment will “just” happen

!   make it the strongest link !   constantly think about your environment !   make it a priority prior to test execution !   learn to recognise environmental issues

and raise these with your managers

56



Bad environments

if your environment is wrong, out-of-date, volatile, not representative, or messy then testing cannot succeed

ignore your test environment and the bugs will breed

57

#28: Is there any value in raising bug reports

!   fact: there is very little value in raising bug reports: !   during development (component/integration tests) !   if you are only concerned in fixing bugs

in an automated environment !   if you are sitting with the developer and they

can fix the bug immediately !   if you are not concerned with learning lessons

or improving the process !   if there is a negative ROI for the time it

takes to raise a bug log !   BUT…it depends on what we mean by “value”

…relative worth, merit or importance

58



your company

The value of our bug logs

negative ROI •  information not used •  no improvements •  takes more time to raise than fix

positive ROI •  help with progress •  study bugs for defect prevention •  testers improve testing skills •  reduce wastage •  data is used wisely 59

#29: Can we “go-live”? Answer is always “yes”

…however

!   Test Manager/Tester’s responsibility !   to provide information to stakeholders to

make informed decisions !   to offer opinions about the quality of the

software that may/may not be agreed with !   to be the quality conscience of the

project !   to highlight warning signs clearly

yes no

60



#30: Believe in yourself…and your abilities

!   low self-esteem is one of the main

contributors to lacking drive and enthusiasm: !   reignite your passion

!   fan the flame before it goes out !   see change as an opportunity rather than a threat !   create good habits

!   66 days to create a good habit !   but only 25 days to create a bad habit

practice looking in the mirror each morning and saying…

“I am great!”

no one on their deathbed said “I wish I had spent more hours at the

office 61

Summary

!   busyness can be both good and bad

!   being busy bad requires action !   introduce various ways of

doing more with less !   build an elite test team that

you can trust and rely on !   there is a need to decide

whether certain aspects are worth being concerned about

!   learn to say “no” to others more so that you can say “yes” to yourself

62

© Lloyd Roden Testers Style Analysis Page 1 of 1

X-Axis

Y-Axis

Friendly

__ Formal __ To the Point __ Indirect __

Approachable

__ Retiring __ Challenging __ Accepting __

Casual

__ Business Like __ Quick __ Leisurely __

Open

__ Guarded __ Insistent __ Thoughtful __

Unstructured

__ Organised __ Lively __ Relaxed __

Sociable

__ Introvert __ Impatient __ Patient __

Intuitive

__ Logical __ Adventurous __ Cautious __

Random

__ Focused __ Confronting __ Receptive __

Warm

__ Cool __ Competitive __ Co-operative __

Perceptive

__ Insensitive __ Strong Minded __ Analytical __

__

__

Grid 10 9 8 7 6 0 1 2 3 4 5 6 7 8 9 10 4 3 2 1 0

Based upon “The Communication Styles Analysis”

Tester’s Style Analysis Questionnaire

Name

Technology

Test Management for Busy People