What You are Doing Wrong With Automated Testing

Anti-Patterns in Automated Testingby Martin Folkoff and Shawn Faunce

US

Martin Folkoff US CIS Shawn Faunce

Anti-Patterns• “An anti-pattern is just like a pattern, except

that instead of a solution, it gives something that looks superficially like a solution but isn't one.” - Andrew Koenig

Ice CreamCone Recorders

Low Priority

BugsOwnershipMonolith

Coverage=

Quality

Ice CreamCone

99%UI Tests

70%Unit & Integration Tests

20%API Tests

10%UI Tests

ExploratoryTesting

Looks like a good solution• Easy to create• Mimics user interactions• You see something happening

10%Unit & Integration Tests

20%API Tests

70%UI Tests

ExploratoryTesting

Anti-Pattern: When you automated testing strategy relies mostly or exclusively tests at the user interface (UI) layer

But it introduces problems• Maintenance nightmare• No fast feedback• Complex tests, complex debugging

Looks like a good solution• Tools make it quick and easy• No programming required• Leverage manual testers

Anti-Pattern: When you use a screen recorder to build your automated tests instead of one of the automated testing frameworks that relies on programming.

But it introduces problems• Extremely fragile• Dependent on having a UI• No mock objects

Recorders

Looks like a good solution• There are bigger fish to fry• Only happens occasionally• Not worth our attention

Anti-Pattern: When you ignore intermittent low priority bugs that occasionally affects the automated test, but a user might easily overcome or not notice.

But it introduces problems• Lose confidence in your tests• Flaky tests• Build technical debt

Low Priority

Bugs

Looks like a good solution• Avoid managing dependencies• Always run your full test suite• Just keep appending new tests

Anti-Pattern: When you organize your test suite so that all of the tests are designed to run together as one test suite.

But it introduces problems• Slow feedback loop• Longer build times• Encourages sloppy engineering

Monolith

Looks like a good solution• Testers test, and this is a test• Frees up developers• Clear lines of ownership

Anti-Pattern: When you have testers write automated tests, you are transferring the ownership of quality

But it introduces problems• Need programming skills• Quality is not someone else’s job• Creates ownership issues

Ownership

Looks like a good solution• Can’t get any better than 100%• Everything appears to be tested• The dashboard is all green

Anti-Pattern: You achieve 100% test coverage on your code, ergo you have 100% Quality.

But it introduces problems• Are the tests good tests?• Satisfy the coverage tool • Can lead to assertion free testing

Coverage=

Quality

“I expect a high level of coverage. Sometimes managers require one. There’s a subtle difference.”

“People optimize their performance according to how they’re measured”

“If a part of your test suite is weak in a way that coverage can detect, it’s likely also weak in a way coverage can’t detect.”

-- Brian MarickHow to Misuse Code Coveragehttp://www.exampler.com/testing-com/writings/coverage.pdf