Flipping the Triangle

8/13/2019 Flipping the Triangle

1/84

Flipping the TrianglePaths to Best, Least-Cost Automated Testing

Patrick Wilson-WelshLisa Crispin

1Thursday, August 7, 2008

Part 1; Part 2.


2/84

Part I

A Triangle of Bricks, Sticks, and Straw



3/84

brief history of manual regression testing


Our experiences of relying solely on manual regression testing to detect and then fix defectshave not been good. Thats why were all in this room together. There are several bad thingsthat happen when most of your testing e!ort is manual, and especially when all of yourregression testing is manual. But we can sum it up in the last chapter of any release:


4/84

were deploying to prod this afternoon


The hidden phase that comes after the final delivery phase, after deployment: the emergencydebug and fix phase, and blamestorming phase. We are here to talk with you about how atransition to a healthy automated testing design can prevent exploding deployments andrelated headaches.


5/84

It takes 4 days to run the smoke tests;Ill get back to you in 4 days.


And from the testers perspective, manual regression testing is Sisyphian. There are neverenough resources for perfect repeatability, total coverage, and it gobbles all the time there is.There is then no time for important activities like Exploratory Testing.


6/84

3 kinds of automated tests


So we are here to talk test automation, and three little kinds of it. Brick tests, Stick tests, andStraw tests. And we are here to talk about the best mix to have, and how it takes time to getto that best mix.


7/84

brief survey:who here has tested what?

who wants to learn what?


But first of all, what testing experiences to people here have? Who in the room is a professional tester?What kinds of testing, manual and automated, have you been trying?Who here is a programmer? What kinds of manual and automated testing have you been trying? Howmuch time do you spend in the debugger?


8/84

Cohns ideal triangle


Here is our goal state, though the heights are not to scale! Mike Cohn, smart agilist, devisedthis notion of a testing pyramid or triangle that places emphasis, and allocates resources, incertain ways.


9/84

TCO

% of automated testing work


At the top of the triangle, we have the tests that are most expensive to maintain. At the bottom, wehave the tests that are least expensive to maintain. The width corresponds to how much of our totaltesting resources budget we want to dedicate to the di!erent kinds of tests. So what are they, exactly?


10/84

GUI

end-2-end,integration, story,

example, acceptance

unit/micro/isolation


At the top, our straw tests are those that test the system as a black box, through its GUI. In thispresentation, well mainly be talking about automated testing solutions for web applications, not rich clientapps, BTW. At the bottom, we have programmer tests: unit tests that isolate bits of behavior. And in themiddle, our stick tests do not go through the GUI, but typically test much more of the system than our unittests do.


11/84

good:leastinvestment

good:

mostinvestment


In our ideal triangle, again, we want most of our time and money for automated tests goinginto the unit tests. And we want the least of it going into the through-the-GUI tests.


12/84

least rework & waste; lowest TCO


In theory, this gives us the lowest Total Cost of Ownership, because it gives us the least amountof rework and waste. It detects the most bugs the cheapest, it drives the best software design, itverifies features most e"ciently, and it is least expensive to extend and maintain. So lets explorethese contentions.


13/84

tales of straw


Lets talk a bit about through-the-GUI tests


14/84

Watir, Selenium, Canoo...

commercial products...


There are several open-source and commercial products for recording through-the-GUItests. Here, well be talking about the ones for web applications, not rich client applications(though such tools obviously do exist).


15/84

Exercise: break into discussion

groups, come up with one goodstory about GUI-style tests.


Lets have folks break into groups of 4 or 5, and spend 8 minutes finding the most interesting,illustrative, or compelling story about GUI tests in the group. Who in each group has the beststory about success or failure, joy or pain? Each group pick one person to tell the best story fromthat group. Then write a 17-syllable headline for your story. Make it compelling. Notice, BTW, that

a haiku has 17 syllables!


16/84

story time


Lets now have each groups elected member read us their storys headline. Well pick thebest one or two to dig into.


17/84

Lisa: tales of Canoo tests.


Lisa Crispin also has a story to tell about settling on Canoo, after some experimentation, totest a web app...


18/84

black box, real-world,functional, slow


So here are the kinds of characteristics that Patrick and Lisa find inherent to GUI tests. They tend touse the entire system as a black box, mimicking real-world behavior, talking to all of the real code andexternal resources and dependencies of the system. They tend to focus on large chunks of behavior.And they tend to be slow


19/84

big. easy to learn to write. alsobrittle and slow. also brittle.


These tests also tend to be big. They tend to be fairly easy to learn to write, at first. Theytend to be slow, again. And they break a lot. It can consume ungodly amounts of timekeeping them running, since, in a web app for example, the tiniest tweak to the GUI canbreak a test.


20/84

brief Selenium IDE demo


Writing a test on the fly for a survey on the http://nerdtests.com/ft_nt2.php site. Selenium is afairly mature open source app for recording tests for web apps, and organizing them into suites.They can also be exported to a number of di!erent languages, and run in at least three di!erentways. By far the best introduction to Selenium is the Firefox Selenium IDE plugin.


21/84

tales of sticks


Now lets focus on the middle layer in our triangle; the tests that are larger than unit tests,but do NOT go through the applications GUI. What are they like?


22/84

Fit, xUnit, FitNesse,

ZiBreve, Concordian...22Thursday, August 7, 2008

So there is a growing handful of open source and commercial tools that fit this small butgrowing automated testing niche.


23/84


groups, come up with one goodstory about non-GUI, e2e tests.


So again, lets find some good stories. First of all, who here as done any of this kind of testing,whether through jUnit, Fit, nUnit, or whatever? Lets break into groups again and pick the beststories about this kind of testing, and write 17-syllable headlines. Haiku optional, butencouraged.


24/84

story time


Lets here a few headlines...


25/84

bypass GUI, less brittle


So what Lisa and Patrick have found to be true about this layer of tests is that partly sincethey bypass the GUI, then tend to be less brittle. They are certainly more brittle than reallygood unit tests, however.


26/84

functional, acceptance,real-worldish, slow, biggish


They do tend to be about chunks of behavior as large as a feature, more than about smaller isolatedbehavior. They again tend to use real external resources, real-ish data. They again tend to be large,and to run slowly. By this we mean many minutes, as opposed to a few seconds.


27/84

about building the right thing,more than building the thing right


One oft-repeated organizing principle about these kinds of tests is that they are best atverifying that we are building the right thing, much more than they are good at verifying thatwe are building the thing right.


28/84

brief FitNesse demo


The original standard web app IDE for automated acceptance testing is FitNesse, built on topof the Fit framework. Lets briefly explore a couple of fit test pages.


29/84

tales of brick


Ah, the good stu!.


30/84

xUnit, TestNG, MbUnit...


These are a few of the usual suspects for true unit testing, isolation testing, programmertesting, micro testing.


31/84


groups, come up with one good

story about xUnit tests.


Once more, lets hear some stories of pain and joy, failure and success with unit tests. Breakinto groups again, and write some haiku.


32/84


33/84

fast, small, pinpoint bugs,

less brittle, hard to learn,single-most valuable software

artifact per system... 33Thursday, August 7, 2008So Patrick and Lisa have found that well-written unit tests, let me repeat that, WELL-WRITTEN unit tests,tend to run really fast (on the order of 10 to 100 per second) because they run entirely in memory. Theytend to pinpoint bugs really well. Then tend to be hard to break. Then are hard to learn to write truly well.And they are the single most important thing you can learn to automate. Without a solid suite of well-written unit tests, its hard to find a software team is that is not basically screwed.


34/84

brief jUnit demo


Lets look at a few unit tests for a small project, and talk a bit about what it means for themto be well written.


35/84

so we want this

GUI


example, acceptance



OK. So this is where we want to be. This, again, is our ideal state.


36/84

but we dont start here

GUI


example, acceptance



But virtually no teams that are new to agility, new to unit testing, new to automated GUItesting, new to Refactoring -- virtually none of these teams get to start here.


37/84

were deploying to prod this afternoon


Most of us started out, basically, here, without much test automation.

b d


38/84

we often start here

bad:most investment

bad:

least investment


And our initial automate testing triangle actually looks more like this. We are throwing toomuch time and money into through-the-GUI automated tests. And we are throwing far toolittle time, money, skill, and discipline into really good unit tests.


39/84


40/84

big. easy to learn to write. alsobrittle and slow. also brittle.


We often start focusing on through-the-GUI automated tests because they are easier, at first,to learn to write, because our programmers dont do testing, because we have dedicatedmanual testers who at first seem like the logical test automators. We tend to fall naturallyinto this pattern: starting with the through-the-GUI tests.


41/84

fast, small, pinpoint bugs,

less brittle, hard to learn...


We tend not to start with unit tests, because they are hard to learn to do well, becauseprogrammers are afraid of them, or because programmers feel that they do not have the timeor permission to write them. I still sometimes hear software professionals claim that thesetests are not worth the e!ort, that they are not valuable enough.


42/84

SRP, small modules, mocking/faking,dependency injection, Refactoring, legacycode rescue, TDD, OO, BDD, CI, Design

Patterns...


To be fair, here are some of the things a programmer must learn to do well, in order toproduce high-value, low-maintenance-cost unit test suites. This is quite a bit to learn.


43/84

Kent Beck, Martin Fowler, Bob Martin,

Gerard Meszaros, Dave Astels, MichaelFeathers, Lasse Koskela, GoF, etc, etc


There are several books, written by several very smart guys, that one must have read in orderto claim to be conversant with the state of the unit testing art. Then on top of that, one mustwrite several thousands of such tests before truly becoming skillful at it.


44/84

hard


This is a hell of a lot of work. Unit testing is really, really, hard for most teams to learn. Thisis especially true when they are already being slogged about by enormous, nasty, untestedlegacy codebases.


45/84

might not yield earliest win


So, starting out with a massive management initiative dictating that all code shall be writtenusing TDD, and we shall achieve and maintain 95% test coverage with our unit tests? Thatmight not work the way you would hope.


46/84

So, what automated

testing choice shouldthe team maketoday?


So, given these patterns and trends, how, for any team, in their particular context, in theirplace along the path toward the ideal triangle, do they decide to do on a particular day,iteration, release.


47/84

Q/A


Questions on material so far?


48/84

30-Minute Break


So, given these patterns and trends, how, for any team, in their particular context, in theirplace along the path toward the ideal triangle, do they decide to do on a particular day,iteration, release.


49/84

Part II

Flipping the Triangle


First, an administrative note. We had originally planned to make this a three-hour session,but discovered during rehearsals that that was an absolutely cruel plan. So instead this will bea 90-minute session, repeated again starting two hours from now, which will be 4:00 PM.Please, if you enjoy this session, Twitter and IM and text all of your friends who are not here,and tell them they MUST attend the second session.


50/84

How about some

principles?


Lets start with a few principles for this journey from upside down triangle to right-side-uptriangle.


51/84

1. Follow the pain.


This is true for most agile transitions. Is it really production defects blowing up badly thatyou want to focus on? And are you constrained in the unit tests you can create just yet? Andare all or most of your tests currently manual? Then it might, in fact, make sense to startwith GUI test suites.

2 Early wins;


52/84

2. Early wins;

low-hanging fruit.


For such teams, GUI tests are often the lowest hanging fruit, a logical starting place (though adreadful ending place).

3. The Whole Team


53/84

owns the transition.


Let there not be fiat testing initiatives from above. No THou Shalt Do TDD or Else directives,nor coverage rate directives. At least, not without the buy-in and input from the entire team.Let the team self-organize -- testers, programmers, BAs, managers -- around where thebiggest pain points are, where the low hanging fruit is, and which steps to take when, whilestill finding a way to meet ridiculous production schedules.

4 P i


54/84

4. Pair programmerswith testers/QA

persons.The parable of the overloaded tester.


There should be no such thing as a group of programmers blocked by testers who cannottest code fast enough. Thats just a version of thats not my job. Here is what you get whenany role or responsibility on the team says thats not my job.


55/84

Moving sticks? Not my job, man.


There should be no such thing as a group of programmers blocked by testers who cannottest code fast enough. Thats just a version of thats not my job. Here is what you get whenany role or responsibility on the team says thats not my job.


56/84

6 Three initiatives:


57/84

6. Three initiatives:

straw, sticks, and bricks.


The team needs to look at these three initiatives as separate. We need separate plans forenough bricks, enough sticks, and just enough straw. We need plans for how to increase thebricks as we scale back the straw.


58/84

7. Learn to makebricks, no matter what.


And no matter what happens, if you dont end up with an absolutely great suite of unit tests-- on the typical non-trivial project, this means thousands of tests than run in a few seconds-- then you wont ever get your automated test costs down to least TCO levels. And yourcode maintain costs, turnover costs, and customer satisfaction costs will all be higher as well.

8. Earn testers freedom


59/84

to do moreExploratory Testing.


What are the best testers really good at? They are good at sni"ng out bugs where otherswould never think to find them. This is instinctual, blink skill (from Malcolm Gladwells bookon deep expertise). Use a great automated testing strategy to buy your testers time to domore and more and more exploratory testing (ET) per iteration.

9. There is such a thing


60/84

as too little courage,skill, trust, respect



61/84

how do teams learn new skills

and tools?


We are talking about culture change here, which involves a group of people unlearning abunch of habits and skills, and learning a bunch of new habits and skills.


62/84


63/84

The fatal pedagogicalerror is to throw answerslike stones at the heads of

those who have not yetasked the questions.

-- Paul Tillich


If I am not asking the question you are answering, then your answer sounds to me like aBowflex commercial in the middle of my late night movie. I wish you would just shut up andgo away, and let me go back to my code in my cubicle.


64/84

so, who enables the questions?


So where do the right questions come from? How do they arise?


65/84

natural leaders


[definition]


66/84

How can I enable the rightquestions?



67/84

continuous learningcourageous experimentationsocializing over evangelizingcommunity-building

encouragement, inspiration

leading by trust & example


what if we started the GUI


68/84

what if we started the GUI

tests with small smoke tests?

what if we started with a Fittest for 1 critical feature per

iteration?

what if all new methods wereno longer than 10 lines?

which parts of the codebasemost deserve legacy rescue, and

how much?



69/84

Exercise: break into discussiongroups, come up with one good

story about objections teammembers have to trying new

initiatives in the straw, stick, or

brick areas.


So again, lets find some good stories.


70/84

story time: how do we meetthe objections


How can the team work together, how can natural leaders work, to en-courage people, to getfolks to try small courageous experiments?

Lets get some ideas out.


71/84

time permitting:

micro Open Space


This crowd has a lot of natural wisdom and hard-won experience around automated testing, Ibet. Are there a half-dozen topics we could have 30-minute Open Space talks around?

[Briefly go over Open Space rules/guidelines...]


72/84

Open Space rules



73/84

Whoever shows up

is the right group.



74/84


75/84

Whenever it starts

is the right time.



76/84

When its over, its over.



77/84

The Law of Two Feet.


* If you feel that you are not contributing or benefiting from a presentation, please feel freeto move on to something else.* Allow the discussion convener to steer his or her topic.


78/84

Getting Started


* Come up with an interesting topic and title for your discussion, related to what we havecovered.* Write it on a sticky, announce it, and put it over here.* If you see topics that have something in common, consider combining them into a singletime slot.

* just drop in to the one that most resonates with you.* Use the law of two feet.

bad:most investment


79/84

get from here

bad:least investment


GUI


80/84

to here


example, acceptance




81/84


82/84

Agile Testing:a Practical Guide for

Agile Testers and Teams

-- Lisa Crispin and Janet Gregory


Agile Testing:a Practical Guide for Agile Testers and Teams

--Lisa Crispin & Janet Gregory


83/84

Fit for Software Development--Rick Mugridge & Ward CunninghamxUnit Test Patterns

--Gerard Meszaros

Test-Driven Development:

By Example--Kent Beck

Refactoring--Martin Fowler

Working Effectively With Legacy Code-- Michael Feathers

Clean Code: a Handbook of AgileSoftware Craftsmanship



84/84

Documents

Flipping the Triangle