Effective Software Testing for ModernSoftware Development

Agile Tour London ‐ October 2017

Alan Richardson

www.eviltester.com/agilewww.compendiumdev.co.uk@eviltester

@EvilTester 1

Common Questions About ModernSoftware Development, Can you...

Develop Software without Testing?

Develop Software without Testers?

@EvilTester 2

Have we had such bad experiences ofSoftware Testing?

@EvilTester 3

We tend to argue over names anddefinitions

"Agile", "Devops""Automation Pyramid", "ATDD", "BDD", "TDD", "ContinuousIntegration", "Continuous Delivery""Tester", "Test Automator", "SDET"

@EvilTester 4

We argue over roles, responsibilities andapproaches

Who should test on Agile Projects?Should testers code?Can programmers test?

Can we automate all testing?Should we automate through the GUI?

@EvilTester 5

Most of this seems like the wrong focus

@EvilTester 6

We need a customised and tailored process forour organisation.

We build a System Of DevelopmentWe want the System to work effectively in our environmentPeople are part of that System

We Automate as part of that System

@EvilTester 7

Alan Richardson: Software DeveloperI Test Things ﴾Tester﴿

I help people test Better ﴾Consultant﴿I develop software ﴾Programmer﴿I learn to develop software better ﴾Practitioner﴿etc.

@EvilTester 8

"I know that I am not a category. I amnot a thing ‐ a noun. I seem to be a verb,an evolutionary process ‐ an integralfunction of the universe."R. Buckminster FullerI Seem to Be A Verb, 1970

@EvilTester 9

You can... Develop software withouttesting

If you don't want to know if it is good enough.

@EvilTester 10

You Can... Develop Software withoutTesters

If you call the people doing the testing a differentname.

@EvilTester 11

Those are Verbal Hacks

@EvilTester 12

"What is a mathematician ‐ someonewho does mathematics?"Ian StewartThe Magical Maze, 1997

@EvilTester 13

"...a mathematician is someone who seesopportunities for doing mathematicsthat the rest of us miss."Ian StewartThe Magical Maze, 1997

@EvilTester 14

"...a tester is someone who seesopportunities for doing testing that therest of us miss."Paraphrasing: Ian Stewart, The Magical Maze

@EvilTester 15

Software development used to look easywhen it was a long line of specialistprocesses

@EvilTester 16

Modern software development is hardbecause we have to customiseeverything

@EvilTester 17

Have we carried forward too manyconcepts from Legacy SoftwareDevelopment?

@EvilTester 18

Guiding Principle: Regularly ReleaseSoftware That Works

@EvilTester 19

Hard, but worth itBetter Quality ProductSpeed of delivering requirementsMore interesting workBetter quality staff

@EvilTester 20

What is Modern Software Development?A mess of terms ‐ TDD, BDD, ATDD, Agile, Devops, CI, CDA Plethora of Technologies

commonalities ‐ flow, fast, automated execution and assertion,short cycles, more releases, automated deployment pipeline

@EvilTester 21

What is Not Modern SoftwareDevelopment?

Development without testingTools above Concepts

Development without riskAccumulation of Technical Debt

@EvilTester 22

Modern software Development goes fastbecause it has more safety controls

@EvilTester 23

Risk: A Process might only look andsound like Modern SoftwareDevelopment

@EvilTester 24

Historic Software Development calledthe safety control "Testing"

@EvilTester 25

Historic Software Development created aTesting Process around...

Test Phases that related to Development Phases

Test Strategy ‐ AnalysisTest Planning ‐ Design

Test Cases & Scripts ‐ ProgrammingTest Execution ‐ Build and Deployment

... but that was never "Testing", that was the part of the SoftwareDevelopment process that The Test Team performed

@EvilTester 26

Modern Software Development hasdropped phases

Analysis PhaseDesign PhaseProgramming Phase

Testing PhaseDeployment PhaseSupport Phase

@EvilTester 27

We need to separate the concept of"Testing" from...

A Testing PhaseA Formal Testing ProcessAn Outsourced Activity

An Activity that only Testers doetc.

We need to work out what "Testing" really is.

@EvilTester 28

Software Testing is part of The SoftwareDevelopment Process

Testing Process needs to be created tocontextually fit the Development Process.

@EvilTester 29

Modern Software DevelopmentProcesses Build in Safety Controls Basedon Risk

@EvilTester 30

Risk that:we build something the user doesn't want

prioritised backlog, MVP, agreed acceptance criteria,release working implementation regularly, work in smallchunks ﴾stories﴿, review stories when 'done'

we break something as we build new stuffautomate acceptance criteria assertions, automated unittesting, continuous integration

We create a disciplined process to mitigate risk.

@EvilTester 31

Testing has always been related to Risk

@EvilTester 32

But there is still a risk thatthe acceptance criteria are not complete enough

we don't cover edge cases

we only focus on acceptance criteria and ignore technical riskwe don't test integration and interfaceswe don't step back enough to review and test the wholesystemperformance, emergent behaviour, failover, security, data, etc.

@EvilTester 33

Testing Takes a Risk & Uncertainty Focus

@EvilTester 34

What fundamentally is testing?

@EvilTester 35

TDDTest before writing code

testing is modelling, design, specificationtest fails

Write codeRun Test

Assert on specific conditions we agree in advance

Test test scope ﴾review﴿tests document the scope of developmentAny more tests needed?

Done ‐ Exit

@EvilTester 36

A side‐effect of TDD is a set of code...that we can use to continually assert that the conditions we wrotethe code to meet, are still met.

We sometimes refer to that using the legacy term"Regression Testing"

@EvilTester 37

Do we still need to test if we do TDD?Integration risk mitigation?

What about missing  @Test ?What about missing asserts?Just enough to drive code, not always enough to cover all usesof the code

e.g. exceptions, null, bad dataWhat about future unplanned usage?

e.g.  add(int, int)  tested against  1‐100 , used with Integer.MAX_VALUE 

@EvilTester 38

Basic Model of a Testing Process

@EvilTester 39

"Observers are men, animals, ormachines able to learn about theirenvironment and impelled to reducetheir uncertainty about the events whichoccur in it, by dint of learning."Gordon PaskAn approach to Cybernetics, 1961

@EvilTester 40

Despite Observation being an importantpart of the process of Testing, we alsotest to increase our uncertainty.

New RisksNew QuestionsNew Decisions

@EvilTester 41

Automated ATDD, BDDVery often:

this is not Driving Development ﴾DD﴿

this is automated DSL based acceptance condition assertionswe execute against deployed and integrated systemsRequirement Risk Based

We sometimes refer to that using the legacy term"Acceptance Testing"

@EvilTester 42

Do we still need to test if we do ATDD?Very often:

we spend so much time automating ATDD, BDD we don't leaveenough time to test

Automation backlogTest sprints

we spend time maintainingRequirements not complete, examples don't cover full riskprofile

@EvilTester 43

Acceptance TestingTom Demarco, 1978, "Structured Analysis and System Specification"

"The Specification is the Acceptance Test"

...

"Acceptance tests are derived from the Target Document andfrom the Target Document alone"

"Acceptance tests will fall into the following five categories:normal path tests, exception path tests, transient state tests,performance tests, special tests"

@EvilTester 44

Special Tests"It would be naive to think that any system, no matter howcomplex, could be tested completely by the derived test setsas I have explained them."

"...they must be placed explicitly in the Structured Specificationbefore it is considered complete. This will maintain thestandard that all the criteria for acceptance are contained inthe Structured Specification"

Exploratory Testing to Learn

Add what we learn to the Acceptance Criteria and coverage

@EvilTester 45

We still need to test because we makemistakes and we miss things out

Risks

and we might not want to assert on them continuously

@EvilTester 46

Risk: Overlap Between Feeding ModelsTDD, ATDD, BDD:

all use the same models as Testingresult in Automated Execution and Condition Assertion

We want to avoid a lot of overlap.

@EvilTester 47

Automating Vs Testing Vs Developing

@EvilTester 48

Abstraction

"The purpose of abstraction is not to be vague,but to create a new semantic level in which onecan be absolutely precise."

E. W. Djkstra1972 ACM Turing Lecture: 'The Humble Programmer'

@EvilTester 49

No Abstractions ‐ Web Testing

@Testpublic void canCreateAToDoWithNoAbstraction(){ WebDriver driver = new FirefoxDriver(); driver.get(     "http://todomvc.com/architecture‐examples/backbone/");

 int originalNumberOfTodos = driver.findElements(             By.cssSelector("ul#todo‐list li")).size();

 WebElement createTodo; createTodo = driver.findElement(By.id("new‐todo")); createTodo.click(); createTodo.sendKeys("new task"); createTodo.sendKeys(Keys.ENTER); assertThat(driver.findElement(       By.id("filters")).isDisplayed(), is(true));

 int newToDos = driver.findElements(          By.cssSelector("ul#todo‐list li")).size();  assertThat(newToDos, greaterThan(originalNumberOfTodos));}

@EvilTester 50

But there were Abstractions

WebDriver ‐ abstraction of browser

FirefoxDriver ‐ abstraction Firefox ﴾e.g. no version﴿WebElement ‐ abstraction of a DOM elementcreateToDo ‐ variable ‐ named for clarityLocator Abstractions via methods  findElement   findElements 

Manipulation Abstractions  .sendKeys Locator Strategies  By.id  ﴾did not have to write CSS Selectors﴿...

Lots of Abstractions ‐ just no Domain Abstractions

@EvilTester 51

Using Abstractions ‐ Same flow

@Testpublic void canCreateAToDoWithAbstraction(){  TodoMVCUser user =         new TodoMVCUser(driver, new TodoMVCSite());

  user.opensApplication().       and().       createNewToDo("new task");

  ApplicationPageFunctional page =           new ApplicationPageFunctional(driver,           new TodoMVCSite());

  assertThat(page.getCountOfTodoDoItems(), is(1));  assertThat(page.isFooterVisible(), is(true));}

@EvilTester 52

What Abstractions were there?Domain level abstractions

 TodoMVCUser 

User

Intent/Action/Behaviour BasedHigh Level

 ApplicationPageFunctional 

StructuralModels the rendering of the application page

﴾see also Domain Driven Design by Eric Evans﴿

@EvilTester 53

Automating With Abstractions SupportsTesting

regular automated execution with assertions

creation and maintenance by many rolesexploratory testingmodel based automated executionperformance testing

multithreaded bots

potential re‐use in performance/stress tools

@EvilTester 54

Adaptation

"If the people can not adapt to the methods, thenthe methods must be adapted to the people."

E.F. Schumacher,

Small is Beautiful, 1973

@EvilTester 55

We are creating A System ofDevelopmentEverything that applies to the System Under Development appliesto our Development System

Testing as "opportunities for doing testing that the rest of usmiss."Systems have risksTesting Targets Risk

Re‐think testing in terms of RiskAutomate Strategically to Mitigate Risk

Flexible Abstractions

@EvilTester 56

EndAlan Richardson www.compendiumdev.co.uk

Linkedin ‐ @eviltesterTwitter ‐ @eviltesterInstagram ‐ @eviltesterFacebook ‐ @eviltester

Youtube ‐ EvilTesterVideosPinterest ‐ @eviltesterGithub ‐ @eviltesterSlideshare ‐ @eviltester

@EvilTester 57

BIOAlan is a test consultant who enjoys testing at a technical levelusing techniques from psychotherapy and computer science. In hisspare time Alan is currently programming a multi‐user textadventure game and some buggy JavaScript games in the style ofthe Cascade Cassette 50. Alan is the author of the books "Dear EvilTester", "Java For Testers" and "Automating and Testing a REST API".Alan's main website is compendiumdev.co.uk and he blogs atblog.eviltester.com

@EvilTester 58