Introduction to test:What is test, and who is the tester?

•Mattias Lönnqvist•Kuoni Nordic IT•STOCKHOLM, 2013-11-27

| 2

Agenda

• Presentation• Introduction to testing

• Who is the tester?

• Why does it matter?• Test tools• Summary• Q & A

• What is testing?• Why do we test?• Test Phases• Requirements• When do we stop?

• Beizer• Cognitive psychology• Biological differences• Assumptions• Cognitive dissonance

• Is all lost?• <- Briefly!

| 3

Presentation

• Mattias Lönnqvist• QA Test Manager at Kuoni Nordic

IT (Previously: OE IT Online)• Been at Kuoni since June 1st 2012• Been working in IT since 1997 and

with test since 2004.• ISTQB certified at foundation level

and advanced level (test manager)• Worked as a consultant in test for 5

years before going to Kuoni• Will give you a quick background

on testing• Please do interrupt if you have

questions• A nice guy

| 4

What is testing? Introduction

• What testing is NOT• It’s not the quality process

• Only quality measurement, i.e. not the Quality process

• It’s not a guarantee• Products will never be

bugfre

• It’s not a cost• We save more on testing

than the cost

• It’s not a waste of time

• What testing IS• ”Testing is measuring quality”

• William Hetzel 1988

• ”Testing is the execution of software in order to show that it doesn’t work” • Glenford Myers 1979

• Example: The Ariane 5 rocket• Cost: $800 million

• It helps improving quality

Ariane 5's first test flight (Ariane 5 Flight 501) on 4 June 1996 failed, with the rocket self- destructing 37 seconds after launch because of a malfunction in the control software.

A data conversion from 64-bit floating point value to 16-bit signed integer value to be stored in a variable representing horizontal bias caused a processor trap (operand error)

because the floating point value was too large to be represented by a 16-bit signed integer.

| 5

What is testing? From a business perspective

Cost

Time

Quality

BusinessOrientedTesting

Reduced Cost = reduced QualityReduced Time = reduced Quality (or increased cost)Increased Quality =

increased cost/time

| 6

What is testing: The V-model / the W-model

Requirements

Specification

Module Design

Code

Compile code

Create system

Deliver productUATTest requirements

Test Specification

Test Integration

System test

Unit test

Integration test

| 7

Why do we test? Introduction

• We are measuring quality• Testing is gaining knowledge

• To keep our deadlines• We cant keep our deadline if we dont know if it works

• Not loosing money• Ariane 5 rocket: $ 800.000.000 • Heathrow terminal 5: £25.000.000 (failed luggage handling)• Forsmark 2006: 2 (of 4) backup generators didn’t start on power

shortage. Cost unknown.• etc

• To make our customers happy• We need them to have confidence in us

• Errare humanum est• Code is written by humans, and humans make mistakes

• Deadlines causes stress and stress induces errors• Changing development and maintenance environments• Lacking or incorrect requirements

| 8

Why do we test? The numbers

• Standish group 1995 – The Chaos Report• 31,1% of all (software) projects are never completed• 52,7% of all projects cost 189% of estimated cost• In 1995, in the US, 81 billions were spent on projects that were cancelled

or never finished• 59 billions were spent on projects that were finished that year• Completed projects had in average 42% of initial requirements met• 9% of all projects in larger companies were considered succesfull

• The biggest reason for this was incomplete requirements• The numbers are better now, but not good

• Standish report costs money, which is why I use the 95 report• Info-tech research group 2006

• 50% of additional work in software projects is caused by problems with requirements

• 70% of failed projects fails because of problems with requirements• Testing is connected to requirements!

| 9

Why do we test – Requirements

• Verification of requirements• Business specifies requirements• Development transforms requirement into code• Testing verifies code against requirements• Requirements can never cover 100%• Requirements can never be unambiguous• Requirements can be misunderstood

• What is this? [Example on requirements]• 1. Carries objects or people• 2. Low maintenance• 3. Low cost• 4. Versatile• The developers interpret as:• System testers interpret as:• Acceptance testers interpret as:• No! This is what business meant:

Myth: I can’t determine if the code is correctTruth: Yes, you

can!

| 10

Why do we test: the costs

• What are the costs?• Test environment• Test resources• Time spent on testing• Etc

• What are the benefits?• Compare to cost of cancelling a release• Compare to cost of fixing a bug in live environment• Compare to cost of decreased sales• Compare to cost of reduction in Brand value

Source: Barry Boehm

2007

| 11

Why do we test: The importance

• How bad is a bug?• Disaster?

• Titanic, Ariane, Patriot Missile, Landing system, etc• Scale of disaster is proportional to the belief it cannot fail• Sometimes called ”Famous last words”

• Bad?• Word 97 crashed all the time in first release• Software wipes database

• Trivial?• Something is ugly, spelling error, etc.

• Catch 22• If you look for defects and find none you are pleased• Finding errors destroys confidence• Testing is looking for errors you don’t wish to find…

| 12

Why do we test – Summary

• Because Software contains bugs/errors?• To estimate / measure quality in Software?• To spend time between code complete and go live?• To prove that there are no bugs?• Because testing is a part of software development?• Because Errors can cost a lot?• To reduce risk of upset customers?• To reduce risk of loosing business?• To stay the best in the travel business?• Because we like to point out others mistakes?• To be able to determine if the software can go live?

Statement True or False?

TrueFalse

True

TrueTrueTrue

FalseFalse

True

FalseTrue

| 13

The test phases – simplified description

• Developers:• Develop code• Compile• Check in• Unit test• (sometimes) integration test

• Own testers• Smoke test• System test• Regression test• Re-test

• Customer• Acceptence test

• This is the customer sign-off, stating if they are satisfied• No acceptance test = we accept the quality, whatever it is…

| 14

The test phases – One (of many) way to split up the work

• Unit tests• Developers test their new code• Does it compile? Does it run?

• System test• Developers make sure system works after new code is added

• Smoke test• Developers make sure that testers can do their work before handing

over• System Integration Test (SIT)

• Testers make sure the system is working• Testers make sure the system is working with all other systems

• User Acceptance Test (UAT)• Customer (might be internal) makes sure the delivered software matches

their requirements• This phase should only find bugs relating to requirements (mismatch)

• All other bugs should be found in the previous phases!

| 15

Test-phases in a waterfall world (but can be described Agile)

End

Start

Reports

Results

Testing

Specification

Planning

We have a project that needs testing

Create test plan, allocate resources

Create test cases from requirements

Execute test cases

Non-functional requirements

Functional requirements

Report bugs

Create test report

Project delivered

Unclear requirement

Insufficient resources, scope, etc

Retest bugs

Unclear or missing requirements

Requirements

Requirements

Requirements

Requirements

Requirements

Requirements

Requirements

Execute test casesA good test case is:

1. Effective (finds bugs)2. Examplatory (covers several areas)3. Maintainable (easy to change / update / keep current)4. Low cost (easy to use, doesnt take a lot of time)5. Possible to automate (sometimes)

| 16

Requirements – why are they so hard?

• We are dealingwith humans

• Humans interpret

• Humans aredifferent

• We have different• Roles• Interpretations

| 17

When do we stop testing?

Statement about end of testing

True or False?

• We can never test enough• We stop testing when the customer is satisfied• We stop testing when we have proved that the

system has no flaws / errors / bugs• We stop testing when we ”feel” quality is good

enough• Whether we are done depends on …

FalseFalseFalse

False

True• Who decides if exit-criteria is fullfilled• What are our risks• What is our test case coverage• What is our S-curve

| 18

When do we stop: The S-curve

• This example is for a typical project• Applies to all projects• Applies to all models• Answers:

Cum

ulat

ive

Def

ects

Implementation dateRequirements Analysis Design Code Test

Time

High Detection Rate

Low Detection Rate

Risk Decision Pointdeploy now?

Test on or implement now

Undetermined time saving

Law of Diminshing Returns

• Can we release now?

• How close to ”ready for delivery” are we?

• How are we progressing?

| 19

Who is the tester?

• We are dealing with humans• We are controlled by our brains• Cognitive psychology

• How / what we percieve• How / what we see• How / what we hear

• Test Psychology• Understanding how we, as

humans affect testing• We are not machines

| 20

What is a tester?

• Beizers phases of testing (Boris Beizer)• Phase 0 : No difference between testing and debugging

• i.e. Testing is something done by developers only• Phase 1: The purpose of testing is to show that the Software works

• i.e. We can prove that the software is flawless• Phase 2: The purpose of testing is to show that the Software doesn’t

work• i.e. Our purpose is to find mistakes

• Phase 3: Purpose of testing is to reduce risks in software• i.e. We reduce risk of errors in delivered software

• Phase 4: Testing is not an act, it is a mental discipline that results in low-risk software without much testing effort• i.e. testing is a part of everything• This is what exploratory testing focuses on

| 21

Cognitive Psychology

• My focus is on human memory functions• How we learn• How we remember• How we recall information

• Human memory consists of:• Short term memory – here and now

• That is why you should take notes when testing• Or you will forget what the bug was

• (But you may still remember that you found a bug)• Medium term memory – last month

• Repeating copies from short term memory• Long term memory – childhood (etc)

| 22

Human memory – Short term memory

• You can only remember 7 +/- 2 items (5 to 9)• Needs to be actively copied to long term memory• Trivial errors often caused by short term memory overload• Major problem with short term memory

• I forgot what the problem is• • If your brilliant idea isnt copied to long term memory you

loose it (but may come up with it again)• That is why you should take notes when testing• Or you will forget what the bug was

• (But you may still remember that you found a bug)

| 23

Human memory – Medium term memory (working memory)

• Working memory• is more a process than a part of the brain

• Analysis causes tension• The mind must solve the problem to relieve the tension• Result is ”feel good factor”• Major problem with working memory

• ”Feel good factor” undermines correct understanding• Desire to solve leads humans to jump to conclusions• Solution may be wrong• E.g. When finding a work-around to do something

that isnt working the way it should, we may be too pleased with our solution to remember the bug

| 24

Human memory – Long term memory

• Your world• Your childhood• Right and Wrong• Your parents and family

life• Education• Formative years• You never forget anything

in the long term memory• But you might not be

able to recall it

• Affects• The way you view things• Your personality• Your temperament• The way the ”penny

drops”• Familiar concepts

• Easily accepted• New ideas & changes

• Resisted

| 25

Biological differences

• Biological differences in GENERAL• Not all men, and not all women

• Women are better at multitasking• Women have a smaller brain, but a more developed

brain stem (the brains information highway)• Less women has color blindness

• Important if you use color to carry information• Men are better AND worse at math

• Women more in the middle on Bell curve• Some biological differences enhanced by society

• E.g. high pitched voices in Japanese women

| 26

Assumptions

• The brain makes assumptions from stored knowledge

• The brain will find a solution• But it doesnt have to be

correct• The penny drops …

• Do you know what you see?• Or do you see what

you know?

| 27

Assumptions – an example

• Read the following colors and say them loud• Dont think; time is money• We have a deadline

•Ready?

| 28

BLUEREDYELLOWGREENBLACKWHITEDONE…

| 29| 29

Assumptions – Contradicting associations

• Color and word doesn’t match• Asking your brain to associate the wrong

color with the name of the color causes a conflict

• The conflict creates overload that increases due to quick slide change (0,5 seconds)

• Cognitive dissonance (more later)

• Now take a look at the next slide…

| 30

Paris in thethe spring

Raise your hand if you spotted the

error…

| 31

Paris in thethe spring

• Let us look again …• See it?

| 32

Assumptions - Blinded by the grammar rules

• English grammar is a set of rules• You know the rules and apply them by

”motor skill”• Your brain is conditioned to apply the rules,

even if what is present breaks the rules• As a result:

• You may not ”see” what is actually present

| 33

Cognitive dissonance

• Do you believe what you see?

• Or

• Do you see what you believe?

| 34

Cognitive dissonance - example 1(color blindness)

| 35

Cognitive dissonance - example 2 (it shouldn’t move)

| 36

Cognitive dissonance - example 3 (might not work on projector)

| 37

Cognitive dissonance - summary

• About 15% of all males are color blind• Around 1% of all women

• Females can interpret patterns more consistently then men

• If the information is in a color or if it can be interpreted in more than one way …• How do you know that you will interpret it

correctly

| 38

Interpretations

• I am a tester • In what field?• What do you do?• Functional or non-

functional?• Manual or automated?• Etc

• What is obvious to you might not be obvious to other

• The better you know an area, the more informed questions can you ask

| 39

Who is the tester? Summary

• To understand the tester we need to understand

• Cognitive Psychology (memory)• Biological differences• Assumptions• Cognitive dissonance• Interpretations

• Why does it all matter?

Test Psychology(in brief)

| 40

Why does it all matter - Overload

• Maximum efficiency at 7 +/- 2 tasks• Overload affects recall (memory)• You have not only forgotten

• But you cant remember that you forgot• Test items are overlooked• Risk is introduced• Impairs ability to interpret (correctly)• You may not be self aware of overload

| 41

Why does it all matter – Cognitive dissonance

• You’ll remove 30-40% of your own errors• Which is why coders shouldn’t test their

own code• Don’t proof read your own work• Dont design tests for software you have

written yourself• Let others challenge your mind set, your

work and your documents• Welcome this!

| 42

Is all lost ?

• What can I do about this?• Be aware of your limitations• Be aware of overload

• Create your own fortress of solitude:• Set the alarm for 60 minutes and:

• Turn off the phone• Put on head phones• Put up a ”Do not disturb” sign

• Dont run the same test twice• Switch with someone else

| 43

Test tools (in brief)

• Screen dumps• An image says more than 1000 words

• Test case tools• To store test cases for easy access• TFS / Spira / Excel / etc

• With or without Gherkin / Specflow• Bug reporting

• TFS / Spira / Google docs / etc• Methods

• Exploratory • Sessions

| 44

Summary

• Remember that testing is PART of the quality process• Be aware of your limitations

• Both your own and those of others• Dont expect testers to find bugs if they cant focus• Do you know what you see or do you see what you know?

• Utilize domain knowledge (business, end users, etc)• No one knows better than them what the software is supposed

to do!• Not me, not the project manager, not the developers• This is the strength and purpose of UAT!

• Avoid overload!• Find a way to create undisturbed working space

• Humans interpret• If you are uncertain about a requirement, ask …

| 45| 45| 45

Q & A

• Questions?• Please do ask…• There are no stupid questions• Only stupid lecturers …