46
AXIOMS Paul Gerrard THE TESTING OF Advancing Testing Using Axioms

Advancing Testing Using Axioms

Embed Size (px)

DESCRIPTION

This is the talk Paul Gerrard gave at Eurostar 2010.

Citation preview

Page 1: Advancing Testing Using Axioms

AXIOMS

Paul Gerrard

THE

TESTINGOF

Advancing Testing Using Axioms

Page 2: Advancing Testing Using Axioms

Agenda

Axioms – a Brief Introduction Advancing Testing Using Axioms

First Equation of Testing Test Strategy and Approach Testing Improvement A Skills Framework for Testers Quantum Testing

Close

Page 3: Advancing Testing Using Axioms

Test Axioms

Formulated as a context-neutral set of rules for testing systems

They represent the critical thinking processes required to test any system

There are clear opportunities to advance the practice of testing using them

Testers Pocketbook: testers-pocketbook.com

Test Axioms Website test-axioms.com

Page 4: Advancing Testing Using Axioms

How can we use Test Axioms?• Test Axioms are not beginners guides• They can help you to think critically about

testing• They expose flaws in other people’s thinking

and their arguments about testing• They generate some useful by-products• They help you to separate context from values

• Interesting research areas!• First Equation of Testing, Testing Uncertainty

Principle, Quantum Theory, Relativity, Exclusion Principle...

• You can tell I like physics

Page 5: Advancing Testing Using Axioms

16 Proposed Axioms

(in three groups)

Stakeholder, (Test) Design and (Test) Delivery

You have the little book

Page 6: Advancing Testing Using Axioms

Stakeholder

Value

Scope Mgt.

Good-Enough

Stakeholderaxioms

Page 7: Advancing Testing Using Axioms

Testing needs stakeholders (p64)

Summary:Identify and engage the people or organisations that will use and benefit from the test evidence we are to provide

Consequence if ignored or violated:There will be no mandate or any authority for testing. Reports of passes, fails or enquiries have no audience.

Questions: Who are they? Whose interests do

they represent? What evidence do

they want? What do they need it

for? When do they want

it? In what format? How often?

Page 8: Advancing Testing Using Axioms

Test Model

Test Basis

Oracle

Coverage

Prioritis-ation

Fallibility

Design axioms

Page 9: Advancing Testing Using Axioms

Test design is based on models (p68)

Summary:Choose test models to derive tests that are meaningful to stakeholders. Recognise the models’ limitations and the assumptions that the models make

Consequence if ignored or violated:Tests design will be meaningless and not credible to stakeholders.

Questions Are design models available to use as

test models? Are they mandatory? What test models could be used to

derive tests from the Test Basis? Which test models will be used? Are test models to be documented or

are they purely mental models? What are the benefits of using these

models? What simplifying assumptions do

these models make? How will these models contribute to

the delivery of evidence useful to the acceptance decision makers?

How will these models combine to provide sufficient evidence without excessive duplication?

How will the number of tests derived from models be bounded?

Page 10: Advancing Testing Using Axioms

Confidence

Sequenc-ing

Event

Never-Finishe

d

Repeat-Test

Environment

Delivery axioms

Page 11: Advancing Testing Using Axioms

Advancing Testing Using Axioms

Page 12: Advancing Testing Using Axioms

First Equatio

n of Testing

Axioms+ Context

+ Values+ Thinking

=Approac

h

Page 13: Advancing Testing Using Axioms

Why is the equation useful? Separation of Axioms, context, values

and thinking Tools, methodologies, certification,

maturity models promote approaches without reference to your context or values

No thinking is required!

Without a unifying test theory you have no objective way of assessing these products.

Page 14: Advancing Testing Using Axioms

One context, multiple approaches Given context, practitioners can promote

different approaches based on their values Values are preferences or beliefs

Pre-planned v exploratory Predefined v custom process Requirements-driven v goal-based Standard documentation v face-to-face comms.

Some contexts preclude certain practices “No best practices”

Page 15: Advancing Testing Using Axioms

Axioms allow (ensure) different approaches and expose positions

Separating axioms, context and values clarifies positions, for example: ‘Structured’ (certified?) test advocates have little

(useful) to say about Agile contexts Exploratory test advocates have little (useful) to

say about contract/requirements-based acceptance

The disputes between these positions is more about values than practices in context

Is a consultant recommendation best for the stakeholders or the consultant?

Page 16: Advancing Testing Using Axioms

Test Strategy and Approach

Strategy is a thought process not a document

Page 17: Advancing Testing Using Axioms

Contexts of Test Strategy

TestStrategy

Risks

Goals

ConstraintsHuman resourc

e

EnvironmentTimescales

Process(lack of?)

Contract

Culture

Opportunities

User involvement

Automation

De-Duplicatio

n

Early Testing

Skills

Communication

Axioms

Artefacts

Page 18: Advancing Testing Using Axioms

IEEE 829 Test Plan Outline1. Test Plan Identifier2. Introduction3. Test Items4. Features to be Tested5. Features not to be

Tested6. Approach7. Item Pass/Fail Criteria8. Suspension Criteria

and Resumption Requirements

9. Test Deliverables10. Testing Tasks11. Environmental

Needs12. Responsibilities13. Staffing and

Training Needs14. Schedule15. Risks and

Contingencies16. ApprovalsBased on IEEE Standard 829-1998

Page 19: Advancing Testing Using Axioms

I’m no fan of IEEE 829

Used as a strategy checklist Scarily vague (don’t go there)

Used as a documentation template/standard Flexible, not prescriptive, but

encourages copy and edit mentality (documents that no one reads)

But many many testers seek guidance on What to consider in a test strategy Communicating their strategy to

stakeholders and project participants

Page 20: Advancing Testing Using Axioms

IEEE 829 Plan and Axioms Items 1, 2 – Administration Items 3+4+5 – Scope Management, Prioritisation Item 6 – All the Axioms are relevant Items 7+8 – Good-Enough, Value Item 9 – Stakeholder, Value, Confidence Item 10 – All the Axioms are Relevant Item 11 – Environment Item 12 – Stakeholder Item 13 – All the Axioms are Relevant Item 14 – All the Axioms are Relevant Item 15 – Fallibility, Event Item 16 – Stakeholder Axioms

Page 21: Advancing Testing Using Axioms

A Better Test Strategy and Plan

1. Stakeholder Objectives Stakeholder management Goal and risk management Decisions to be made and how

(acceptance) How testing will provide

confidence and be assessed How scope will be determined

2. Design approach Sources of knowledge (bases

and oracles) Sources of uncertainty Models to be used for design

and coverage Prioritisation approach

3. Delivery approach Test sequencing policy Repeat test policies Environment requirements Information delivery approach Incident management approach Execution and end-game

approach

4. Plan (high or low-level) Scope Tasks Responsibilities Schedule Approvals Risks and contingencies

Page 22: Advancing Testing Using Axioms

Testing Improvement

Test process improvement is a waste

of time

Page 23: Advancing Testing Using Axioms
Page 24: Advancing Testing Using Axioms
Page 25: Advancing Testing Using Axioms

Actuallyits 11

(most were not software related)

Page 26: Advancing Testing Using Axioms

The delusion of process models(e.g. CMM)

Google search “CMM” – 22,300,000 “CMM Training” – 48,200 “CMM improves quality” – 74 (BUT really 11 – most of

these have NOTHING to do with software) A Gerrard Consulting client…

CMM level 3 and proud of it (chaotic, hero culture) Hired us to assess their overall s/w process and make

recommendations (quality, time to deliver is slipping) 40+ recommendations, only 7 adopted – they

couldn’t change How on earth did they get through the CMM 3 audit?

Page 27: Advancing Testing Using Axioms

“Test Process Improvement is a Waste of Time”

Using process change to fix cultural or organisational problems is never going to work

Improving test in isolation is never going to work either

Need to look at changing context rather than values…

Page 28: Advancing Testing Using Axioms

Why you are where you are

Context+ Values

+ Thinking=Approa

ch

<- your values<- your context

<- your thinking<- your approach

Page 29: Advancing Testing Using Axioms

Where maturity models come from

Context+ Values

+ Thinking=Approa

ch

<- someone else's

<- someone else's<- someone else's<- someone else's

Page 30: Advancing Testing Using Axioms

Making change happen

Axioms+

Context+ Values

+ Thinking=Approa

ch

<- recognise

<- hard to change

<- could change?<- just do some<- your approach

Page 31: Advancing Testing Using Axioms

Using the axioms and questions Axioms represent the critical things

to think about Associated questions act as

checklists to: Assess your current approach Identify gaps, inconsistencies in current

approach QA your new approach in the future

Axioms represent the WHAT Your approach specifies HOW

Page 32: Advancing Testing Using Axioms

Eight stage change process (after Kotter) Mission Coalition Vision Communication Action Wins Consolidation Anchoring

Changes identified here

If you must use one, this is where your

‘process model’ comes into play

Page 33: Advancing Testing Using Axioms

A Skills framework for testers

Axioms indicate WHAT to think about...

...so the Axioms point to SKILLS

Page 34: Advancing Testing Using Axioms

Test design is based on models (p68)

Summary:

Choose test models to derive tests that are meaningful to stakeholders. Recognise the models’ limitations and the assumptions that the models make.

Consequence if ignored or violated:

Tests design will be meaningless and not credible to stakeholders.

Questions: Are design models available to use

as test models? Are they mandatory? What test models could be used to

derive tests from the Test Basis? Which test models will be used? Are test models to be documented or

are they purely mental models? What are the benefits of using these

models? What simplifying assumptions do

these models make? How will these models contribute to

the delivery of evidence useful to the acceptance decision makers?

How will these models combine to provide sufficient evidence without excessive duplication?

How will the number of tests derived from models be bounded?

Page 35: Advancing Testing Using Axioms

Test design and modelling skills A tester needs to understand:

Test models and how to use them How to select test models from fallible

sources of knowledge How to design test models from fallible

sources of knowledge Significance, authority and precedence of

test models How to use models to communicate The limitations of test models

Familiarity with common modelsIs this all that current certification provides?

Page 36: Advancing Testing Using Axioms

Training and certification must change Intellectual skills and capabilities are

more important than the clerical skills Need to re-focus on:

Testing thought processes (Axioms) Testing Stakeholder relationship management Testing as an information provision service Goal and risk-based testing Real-world examples, not theory

Practical, hands-on, real-world training, exercises and coaching.

Page 37: Advancing Testing Using Axioms

Quantum Testing

If evidence arrives in discrete quanta...

...can we assign a value to it?

Page 38: Advancing Testing Using Axioms

How testing builds confidence As tests are run, every individual test

has some significance Some tests expose failures but

ultimately we want all tests to PASS When all tests pass – the

stakeholders are happy, aren’t they? Can we measure confidence by

counting tests?

Not really...

Page 39: Advancing Testing Using Axioms

The value of a test varies by… Coverage model:

A test could cover one or hundreds of functional conditions, ten thousand program statements or ten

Objective: Criticality of the business goal it examples Criticality of the risk it informs

Precedent: The first end-to-end test pass is significant The 100th variation of a similar test is less significant

Functional dependence: A test of shared functionality used thousands of times per hour could

be much more important than a peripheral feature used once/day Stakeholder:

Are customers tests more or less significant than supplier tests? Context:

The same test run at different times in different environments can have different value.

Page 40: Advancing Testing Using Axioms

Quantum Testing

Only a stakeholder can assign a value to a test (but that is very hard thing to do) – (but see relativity)

Testers can’t quantify value, but could define significance

A test is significant (to stakeholders) if it: Can be related to a meaningful test objective Increases coverage with respect to a meaningful test

model Is considered in an acceptance decision (at any level)

Significance is a Boolean but could be 0 or 1 The number of insignificant tests should be zero.

Page 41: Advancing Testing Using Axioms

Assessing significance

Significance can only be assessed by testers if: Our test objectives, models, coverage goals are

meaningful (to stakeholders) or Testers are authorised to create their own

objectives, measures and coverage goals or Testers are their own stakeholder

Testers need a close, trusting relationship with their stakeholders or authorised autonomy E.g. exploratory testing won’t work if stakeholders

do not allow autonomy Testers should not ‘go it alone’.

Page 42: Advancing Testing Using Axioms

The significance of significance Test coverage models and goals that

generate uniform distributions of tests are inefficient and uninformative We need more and better test models Models that are meaningful in context

Significance varies with context and can be used to explain why e.g. some tests aren’t useful as regression

tests How much testing is enough?

Can never be answered by coverage alone.

Page 43: Advancing Testing Using Axioms

Using significance (as booleans) A test objective could be deemed as met

when a set of significant tests are passed Test objectives can often be aligned with

business goals (or risks) e.g.: Faultless end to end or straight-through

processing in an ERP system Customer order capture, money-in or complete

financial reconciliations without failure Could the business put a value on these

goals being achieved? Perhaps the business case provides this level of

information?

Page 44: Advancing Testing Using Axioms

Our current understanding of test value measurement In terms of a mathematical treatment of

testing we are some way off Compare with:

Understanding of heat and energy in 18th century?

Economics in the early 20th Century? A complete theory of test value may never

be achieved Quantum testing suggests that:

Stakeholders judge value (relatively speaking) Testers judge significance.

Page 45: Advancing Testing Using Axioms

Close

Axioms are context-neutral rules for testing The Equation of Testing

Separates axioms, context, values and thinking We can have sensible conversations about

process Axioms and associated questions provide

context neutral checklists for test strategy, assessment/improvement and skills

QT separates significance from value Could it tell us, “how much testing is enough”? Right now, we can ‘feel the heat’, but not

measure temperature.

Page 46: Advancing Testing Using Axioms

AXIOMS

Thank-You!

THE

TESTINGOF

testaxioms.comtesters-pocketbook.comgerrardconsulting.comuktmf.com