Memes & Fitness Landscapes - analogies of testing with sci evol (2011)

Emerging Topics track

1

Memes andFitness Landscapesdown at the Schools-yard:Analogies of testing context withscientific evolution

Neil ThompsonThompson information Systems Consulting Ltd

©Thompson information Systems Consulting Ltd

v1.0

Slides 15 minutes (not “lightning”, but...),discussion 5 minutes (plus after, whenever!)


2


Four, five, six... schools of software testing?(Updated version) March 2007Copyright © 2003-2007 Bret Pettichord. Permission to reproduce granted with attribution

Emphasis onadapting to the circumstances under which the product is developed and used

Annotations by Neil Thompson after the Bret Pettichord ppt (blue text), the list in Cem Kaner’s blog December 2006 (black text)

Emphasis on analytical methods for assessing the quality of the software, including improvement of testability byimproved precision of specifications and many types of modeling (Control):

Emphasis on standards and processes that enforce or rely heavily on standards

(Test-Driven):emphasis on code-focused testing by programmers

Oblivious /Groucho?

Emphasis on policing developers and acting as “gatekeeper”

Axiomatic?

Factory: Emphasis on reduction of testing tasks to routines that can be automated or delegated to cheap labour

, and other sources! (red text)

Neo-Holistic?(like C-D)


3

Why I want to talk about this

• To understand better why and how different people work and communicate in different ways (eg balance of personal and situational factors)

• To try to broker more interactive and detailed debate (with less emotional confrontation?) between people exhibiting the characteristics of different Schools

• To help pave the way for the evolution of testing into the future, eg using power tools such as Grounded Theory with statistical analysis, Artificial Intelligence concepts such as Bayesian belief networks and genetic algorithms; testing AI itself!

© Thompson information Systems Consulting Ltd


4


Precursor talk to this: “The Science of Software Testing”

System Requirements/Specification/ Design

Note: this is starting with some “traditional” views of testing & science

Test

“Aim to find bugs”

Testresult

Test result =“as aim”?

Experimentresult

Hypothesis

Experiment

Experiment result =

“as aimed”?

Test is“successful”

Falsificationconfirmed

Hypothesisnot yetfalsified

Test is so far“unsuccessful”Product

Part of thecosmos

“Aim tofalsify hypothesis”

Y

N

Y

N

Myers

Popper


5


Other quality-relatedcriteria

This comparison informs the hot topic: testing versus “just” “checking”


“Check”

Expectedresult

Checkresult

Check result =

Expected?

Check“passes”

Check“fails”Product


Test

Wayscould fail

Testresult

Test result =appropriate

?

Quality-related info

Info onqualityissuesProduct

Other oracles

Y

N

Y

N

Sources: Michael Bolton blog and various Context-Driven material


6


A complementary view: Testing as facilitating value flow through SDLC

LEVELS OF DOCUMENTATION,pushed by specifiers

WORKINGSOFTWARE

Accepted

System-tested

Integrated

Unit / Component-tested

FLOW OF FULLY-WORKING SOFTWARE, pulled by

customer demand

Requirements

+ FuncSpec

+ TechnicalDesign

+ Unit / Componentspecifications

+ Test Specifications


7


Value Flow in SDLC as layerswith four characteristics

Levels of system &service integration

Levels of stakeholdersBusiness, Users,Business Analysts,Acceptance Testers

Architects,“independent” testers

Designers,integration testers

+ Businessprocesses

Developers,unit testers

Understanding of problem... Un

ders

tand

ing

of s

olut

ion.

..

Desired quality

Tested (“known”) quality

Levels of

spec-ificationRequirements

Functional &NF specifica-

tions

Technicalspec, Hi-level

design

Detaileddesigns

But, howmeasure &manage?

Remember: not only for waterfall or V-model SDLCs, rather iterative / incremental go down & up through layers of stakeholders, specifications & integrations


8


Managing Value Flow through SDLC using VF ScoreCards

FinancialEfficiencyProductivityOn-time, in budget- Cost of quality

SupplierUpwardmanagement

Informationgathering

• Based on Kaplan & Norton Balanced Business Scorecard and other “quality” concepts

• Value chain ≈ Supply chain:– in the IS SDLC, each participant

should try to ‘manage their supplier’

– for example, development supplies testing(in trad lifecycles, at least!)

– we add supplier viewpoint to the other 5, giving a 6th view of quality

• So, each step in the value chain can manage its inputs, outputs and other stakeholders

CustomerVALIDATIONRisksBenefitsAcceptanceSatisfaction- Complaints

Improve-menteg TPI/TMM…PredictabilityLearningInnovation

ProcessComplianceeg ISO9000Repeatability

- Mistakes

ProductVERIFICATIONRisksTest coverage

- Faults- Failures


9


Value Flow ScoreCards canbe cascaded

Pieces of a jig-saw puzzle!In addition to “measuring” qualityinformation within the SDLC:• can use to align SDLC principles with

higher-level principles from the organisation

Business Analysts Requirements Reviewers

Architects

Acceptance Test Analysts

Func Spec Reviewers

Designers

Acceptance Testers

Sys TestersSys Test Analysts

Tech Design Reviewers

Developers

Int Test Analysts

via pair programming?

Component Test Analysts, Designers & Executers?

AT Designers & Scripters

ST Designers & Scripters

IT Designers, Scripters & Executers

(...but don’t necessarily need all of these!)


10


Using “Metaphysics” & Scienceas analogies to inform Value Flow

Levels of system &service integration

Levels of stakeholders

Business, Users,Business Analysts,Acceptance Testers

Architects,“independent” testers

Designers,integration testers

+ Businessprocesses

Developers,unit testers

Layers of quality

Static values:

• Inorganic

Dynam

ic values?

• Biological

• Social• Intellectual

Layers of science

......

......

...Sc

ale.

......

......

..Physics

Chemistry: InorganicChemistry: Organic

Biology (& systemsthinking)

Social sciences(Philosophy)


11

What was that about layers of Science?

© Thompson information Systems Consulting Ltd Image from http://www.aaas.org/spp/dser/03_Areas/cosmos/perspectives/Essay_Primack_SNAKE.GIF

(Ouroboros: Greek Οὐροβόρος or οὐρηβόρος,

from οὐροβόρος ὄφις "tail-devouring snake”)

• There is a cascade (and approx symmetry!):– Biology depends on Organic Chemistry– Organic chemistry depends on the special properties of

Carbon– Chemical elements in the upper part of the periodic

table come from supernovae– Elements in the lower part of the periodic table come

from ordinary stars– Elements are formed from protons, neutrons, electrons

(Physics)– ... quarks... string theory?? etc

• It just so happens that humans are about equidistant in scale from the smallest things we can measure to the largest

• Then... humans have evolved to use tools, build societies, read, invent computers...

• So, it is possible to think of pan-scientific evolution as a flow of value

• Now, back to software lifecycles...

Sources: Daniel Dennett “Darwin’s Dangerous Idea” “cosmic Ouroboros” (Sheldon Glashow, Primack & Abrams, Rees etc)

Inventions by humans,

eg Social Sciences

http://www.aaas.org/spp/dser/03_Areas/cosmos/perspectives/Essay_Primack_SNAKE.GIF


12


Invert that picture: Testing is reallyat the top, and should diverge fast!

• Each level of progress generates possibilities, which are tested

• Then, each level is a platform which, when established, is easily built uponby “cranes” (without having to worry about the details below)

• After the science levels...• humans made tools, talked and

co-operated• printing gave us another level • now, software is following

exponential growth• So, software testing should surf the

wave of evolution (not flounder in the shallows behind it)

• Kurzweil epochs

5: Bio methods integrated into technology?6: Intelligence into matter/energy patterns?

4: Technology

3: Brains2: Biology

1: Chemistry & Physics

+0: Maths?!

“SINGULARITY”


13


Evolution: first, traditional Darwinian (ie biological)

Image from www.qwickstep.com

http://www.qwickstep.com/


14


Image from www.qwickstep.com

Biological reproduction & evolutionare controlled by Genes

Image from .schools.wikipedia.org

Diversity

Sophist-ication

Replication & Selection

Mutation

GENE




15


Biological Evolution as Sophistication plotted against Diversity

Source: Daniel Dennett “Darwin’s Dangerous Idea”

Diversity

Sophistication


16


But evolution is not smooth?

“Punctuated equilibra” idea originated by Niles Eldredge & Stephen Jay GouldImages from www.wikipedia.org

Sophistication

Diversity“Gradual” Darwinsim

Sophistication

DiversityPunctuated equilibria

“Explosion” in species, eg Cambrian

Spread into new niche,eg Mammals

Mass extinction, eg Dinosaurs

(equilibrium)

(equilibrium)

(equilibrium)

Sophistication

Diversity

Number ofspecies

http://www.wikipedia.org/


17


So... Evolution ofScience overall

Biology

Chemistry

Organic

Inorganic

Physics

Social sciences

• Arguably other sciences have not evolved smoothly either• Sudden advances, akin to punctuated equilibria in biological evolution


18


OK, what’s this got to do with software testing?

Social sciencesTools

Language

Books

Computers

• We have an important and difficult job to do here!

• Social sciences evolution Tipping Points

(Malcolm Gladwell)


19


...and computers are evolving, in both sophistication and diversity, faster thansoftware testing?

Computers1GL

ObjectOrientation

Internet,Mobile devices

ArtificialIntelligence?!

4GL

3GL

2GL

• Are we ready to test AI??


20


How software testing has evolved so far?

Overall periods developed after Gelperin & Hetzel, “The Growth of Software Testing”, 1988 CACM 31 (6) as quoted on Wikipedia

PERIODDEBUGGING (Psychology)DEMONSTRATION (Method)DESTRUCTION (Art)

EVALUATION

PREVENTION (Craft?)

SCHOOL(S)

1957

1976

EXEMPLAR OBJECTIVES SCOPE APPROACH

Pre-

1983

1984

2000

Science?2011

Verify, +maybe Prove, Validate, “Certify”

Weinberg(1961 & 71) Test + Debug Programs Think, Iterate

Hetzel(1972)

Show meetsrequirements

Find bugsMyers(1976 & 79)

Programs

Programs, Sys,Acceptance

+ Walkthroughs, Reviews & Inspections

Kaner et al(1988 & 99)

Experiment &Evolve? Neo-Holistic?

Measurequality

?

Beizer(1984)

+ IntegrationFind bugs,show meetsrequirements,+prevent bugsFind bugs, in serviceof improving quality,for customer needs

Realistic, pragmatic,normal


21


Can we retrofit the Schools to this history?

PERIODDEBUGGING (Psychology)DEMONSTRATION (Method)DESTRUCTION (Art)

EVALUATION

PREVENTION (Craft?)

SCHOOL(S)

1957

1976

EXEMPLAR OBJECTIVES SCOPE “SCHOOL”?

Pre-

1983

1984

2000

Science?2011

Weinberg(1961 & 71) Test + Debug Programs

Hetzel(1972)

Show meetsrequirements

Find bugsMyers(1976 & 79)

Programs

Programs, Sys, Acc

Kaner et al(1988 & 99)

Experiment &Evolve?

Neo-Holistic?

Measurequality

?

Beizer(1984)

+ IntFind bugs,show meetsrequirements,+prevent bugsFind bugs, in serviceof improving quality,for customer needs

Analytic

Standard(Control)

Quality

ContextDriven

“no schools, but...”

?

Agile(Test-Driven)

Factory


22


The Philosophy of Science is also evolving!

Classical

Bayesianism,GroundedTheory...

Logical

• So, perhaps the Philosophy of Software Testing could learn from this, perhaps it’s also evolving?...

PositivismEmpiricism

PopperKuhn

LakatosLaudan

• Again, progress jerky, not smooth• Paradigm shifts akin to punctuated equilibria

Hull


23


Memes as an extension of the Genes concept

Biological evolution

GENES

Theme developed from Daniel Dennett “Darwin’s Dangerous Idea”

Mental, social & cultural evolution

Platforms

Cranes

MEMES


Mutation

Ideas Beliefs PracticesSymbols

Gestures

Speech

Writing

Image from .www.salon.comTaxonomy from www.wikipedia.org

“Otherimitable

phenomena”

Rituals




24


Some candidates for Memes in software testing

Always-considerEffectiveness

EfficiencyRisk management Quality management

Insurance Assurance

V-model: what testing against W-model: quality management

Risks: list & evaluate

Define & detect errors (UT,IT,ST)Give confidence (AT)

Prioritise tests based on risks

Tailor risks & priorities etc to factors

Refine test specifications progressively: Plan based on priorities & constraints Design flexible tests to fit Allow appropriate script format(s) Use synthetic + lifelike data

Allow & assess for coverage changes Document execution & management procedures

Distinguish problems from change requests Prioritise urgency & importance

Distinguish retesting from regression testing

Use handover & acceptance criteria

Define & measure test coverage

Measure progress & problem significance

Be pragmatic over quality targets

Quantify residual risks & confidence

Decide process targets & improve over time

Define & use metrics

Assess where errors originally made

Define & agree roles & responsibilities

Use appropriate skills mix

Use independent system & acceptance testers

Use appropriate techniques & patterns

Plan early, thenrehearse-run,acceptance tests

Use appropriate tools

Optimise efficiency

Source: Neil Thompson STAREast 2003 (not “best practices” but reference points for variation?)


25


An example of a differentsoftware testing “memeplex”

Source: Neil Thompson BCS SIGiST 2002 review of Lessons Learned in Software Testing (Kaner, Bach & Pettichord)

Your career in software testing

The role of the tester

Thinking like

a tester

Testingtechniques

Bug

advocacy

Autom

atingtesting

Documenting testing

Interacting with programmers

Managing the testing project

Managing the testing group

Planning thetestingstrategy


26


Memeplexes and Fitness Landscapes: fixed / flexible?

------- RUGGED &

FIXED? -------

Emphasis on adapting to circumstances under which the product is developed & usedContextDriven

FLEXIBLE?

Standard(Control)

standards and processes

Factorydelegated to cheap labour

reduction of tasks

routines can be automated

Agile(Test-Driven)

code-focused testing by programmers (needs

automation)

Analytic precision of specsanalytical methods

assessing quality

of software

testability

----- many types of modeling -----

Quality acting as “gatekeeper”

policing developers

FinancialEfficiencyProductivityOn-time, in budget

- Cost of quality

SupplierUpwardmanagementInfo from otherlevels of Treble-V model

CustomerVALIDATIONRisksBenefitsAcceptanceSatisfaction- Complaints

Improvement &Infrastructureeg TPI/TMM…PredictabilityLearningInnovation

ProcessComplianceeg ISO9000Repeatability

- Mistakes

ProductVERIFICATIONRisksTest coverage

- Faults- Failures


27


Are these separate species which cannot interbreed?

(Control)

(Test-Driven)

Factory

• ... or is each part of an ecosystem with its suppliers and customers?


28


Shouldn’t (or doesn’t) Context-Drivensubsume practices of other schools *when context is appropriate*?

(Control)

(Test-Driven)

Factory

Neo-Holistic?(or Context-Driven

itself?)


29


Examples of how memeplexes can help ascend peaks of fitness landscapes

precision of specsanalytical methods

assessing quality

of software

testability

----- many types of modeling -----Analytic

FinancialSupplier Customer Improvement &Infrastructure

Process Product

ContextDriven Prioritise tests

based on risks

• Context-Driven thinking could address gaps and rebalance the scorecard; or...• It might prefer its own scorecard

PLUS... OR INSTEAD! Risks: list & evaluate

Quantify residual risks & confidence

Allow & assess for coverage changes

W-model: quality management

Measure progress & problem significance

Assess where errors originally made


30


A Tester’s Taxonomy for Meme generation &transmission


MutationIdeas Beliefs PracticesSymbols

Gestures

Speech

Writing

Image from .www.salon.comTaxonomy from www.wikipedia.org

“Otherimitablephenomena”

Rituals

WEEKENDTESTING

V/W MODEL

IN BAR WITHWORKMATES

CONFERENCETALKS

WORKINGON APROJECT

BLOGS

BOOKS

COURSES

PAPERS

STANDUPMEETINGS

IN BED/PARK/BAR, THINKING

SPECIFICMETHODS & PROCESSES

STANDARDS




31


Speculation on meme variations in software testing

Gestures

Speech

Writing

“Otherimitablephenomena”

Rituals

ContextDriven

AgileFactory

QualityAnalytic(Test-Driven)

Standard(Control)

Favoured Meme vehicles

School groups

PracticesSymbolsFavoured Meme types? IdeasBeliefs

BLOGS

COURSES (Certification)COURSES (Proficiency)

WEEKENDTESTING

BOOKS, PAPERS

STANDUPMEETINGS

V/W MODEL

CONFERENCE TALKS (but not necessarily the same conferences!)

STANDARDS

SPECIFIC METHODS& PROCESSES


32


Not only Evolution, but Emergence: progress along order-chaos edge

Physics

Social sciences

Chemistry

Biology

• For best innovation & progress, need neither too much order nor too much chaos• “Adjacent Possible”• Might this also apply to testing?

ORDER

CHAO

S

ORDER

CHAO

S

ORDER

CHAO

S

ORDER

CHAO

S

Extrapolation from various sources, esp. Stuart Kauffmann, “The Origins of Order”, “Investigations”


33

So... What’s the message, here?

• Regarding schools of software testing – I believe they are a very useful concept but:– please see analogies with species, fitness landscapes & ecosystems– don’t just preach to testers, educate the customers!– are schools really fixed around beliefs, or are they more flexible?

• When you think & communicate, try using the memes framework to better understand what you are building on, where you are innovating, and what you want to achieve, with what audience:– could “old school” people please blog and tweet more?– could Context-Driven people write more books please?– let’s share our experiences on projects

• Software testing is evolving, should continue to evolve:– future jumps could be quite big (“platforms & cranes”)– may be sudden paradigm shifts (cf punctuated equilibria,

Per Bak’s sandpiles)– most fruitful path is on the chaos-order boundary?



34

Next steps already considering• Analyse more specific examples of memes in congenial & hostile

environments – the “Extended Phemotype”!• More analogies of testing with history & philosophy of science –

both for individual strategies and for improvement• Practical uses of Bayesianism to focus testing – already are some?

What can we actually use? What’s coming?• Is there correlation between personality (eg Myers-Briggs, Belbin)

and “membership” of schools of software testing?– Myers-Briggs fixed, Belbin can vary with situation & mood??

• Is the Cynefin construct of any use here? (pronounced “kanavin”?)– quadrants of systems/situations (simple, complicated, complex & chaotic)

may be suitable for different school-type behaviour – or, different responses of Context-Driven

– suggests steps to approach, which vary in usage & sequence: Sense, Categorise, Analyse, Probe, Respond, Just-act



35

Questions to think about / discuss

• Are (some?) people “stuck” in a school because of:– their personalities, – upbringing, – education/inculcation, – deeply-held beliefs?...– the company they keep, or – the jobs they tend to get recruited for?– what their boss wanted yesterday?

• Are the schools themselves evolving – if so, how? Eg...– Context-Driven, from origins to book(s) to blogs to this conference?– Factory school in response to “maturing” of outsourcing / offshoring

market?– current Agile movements?

• Do some memes replicate in spite of not really helping their hosts?



36

Questions to think about / discuss (continued)

• Shouldn’t Context-Driven, “by definition”, embrace practices of all the other schools *where appropriate* (or are other schools bad even in their own context? Are cultural / ethical divisions insoluble?)

• Analogies between testing schools and schisms in science, eg string theory v the others, frequentists v Bayesians

• How should individuals in software testing evolve?• How should software testing be preparing for the future, eg testing

Artificial Intelligence:– what happened to Genetic Algorithms?– what can we do with Grounded Theory?– and (as above) Bayesian?

• What other big innovations could be coming? (see Steven Johnson)

• Any memes about to die out?• Do we want to go beyond testing

only software?



37

Main references & Acknowledgements– Bret Pettichord: “Four Schools” presentation– Dave Gelperin & Bill Hetzel paper, The Growth of Software Testing– testingreferences.com: testingtimeline– Stuart Reid paper, Lines of Innovation in Software Testing – Cem Kaner: blog, and “Software Testing as a Social Science” presentations– James Bach & Michael Bolton blogs, plus Kaner Bach & Pettichord “Lessons Learned...” book– Mike Smith: originating motivation & ideas, then co-development, of Value Flow ScoreCards – plus key

input from Isabel Evans – Robert Pirsig books: Zen & the art of Motorcycle Maintenance + Lila – Sheldon Glashow: cosmic Ouroboros– Charles Darwin books– Richard Dawkins: various books, esp. Extended Phenotype & Climbing Mount Improbable– Daniel Dennett: various books, esp. Darwin’s Dangerous Idea– Susan Blackmore, Robert Aunger, Kate Distin etc: various books on Memes– Matt Heusser, blog post 31 Jul 2009, esp. comments by Laurent Bossavit & James Christie– Peter Godfrey-Smith book: Theory and Reality (Philosophy of Science)– Stuart Kauffman: various books, eg Investigations– Ray Kurzweil book, The Singularity is Near– Jurgen Appelo book & website, Management 3.0– Sharon Bertsch McGrayne book: The Theory That Would Not Die (Bayes)– Dave Snowden, Cynefin



38


• Thanks for listening! • Questions & discussion?